Senior Data Pipeline Engineer

This is a fully remote position, open to applicants in United States.

📋 Description

• Design, develop, and maintain ETL/ELT and C#/.NET pipeline services, incorporating SSIS, Airflow, AWS Lambda, enhancer, and message-processing services.

• Efficiently transfer data from source systems into Candid's warehouse and lakehouse.

• Support the ongoing development of Candid's cloud data platform, including Iceberg ingestion and transformation pipelines as well as DynamoDB.

• Optimize query performance using Trino/Starburst.

• Ensure comprehensive monitoring of data quality from end-to-end.

• Analyze production pipeline issues to identify root causes, document findings, and implement fixes through reprocessing.

• Define the scope and integrate new external and regulatory data sources from start to finish.

• Collaborate on schema and architecture decisions, coordinating changes across ingestion and transformation layers.

• Maintain standard operating procedures (SOP) and technical documentation.

• Mentor Associate-level engineers and evaluate their work.

• Engage in the team's release and deployment processes.

• Collaborate with the Data Operations Manager on architectural and roadmap decisions.

• Provide escalation-level support for critical pipeline issues, including after-hours assistance as required.

⛳️ Requirements

• Over 5 years of experience in database engineering, data engineering, or ETL development, demonstrating ownership of production data pipelines.

• Proficient in Transact-SQL (stored procedures, views, indexing).

• Practical experience in C#/.NET development.

• Established expertise in designing ETL processes using SSIS.

• Familiarity with orchestration platforms such as Apache Airflow is preferred.

• Experience with AWS and open table formats (Apache Iceberg, Parquet).

• Knowledge of AWS Lambda, DynamoDB, and Trino/Starburst is advantageous.

• Strong skills in data quality and root cause analysis, with a track record of resolving production data incidents.

• Understanding of source control, CI/CD, and release/deployment processes across Dev/QA/Prod environments.

• Capability to mentor and evaluate the work of other engineers.

• Exceptional written and verbal communication skills.

• Sound business judgment and ability to prioritize independently.

• Ability to thrive in a distributed team environment.

• Willingness to take on additional duties and special projects as needed/requested.

• Sensitivity to and respect for racial, gender, sexual orientation, and cultural differences.

• Commitment to Candid's values: driven, direct, accessible, curious, and inclusive.

• Must be legally authorized to work in the United States without employer sponsorship, now or in the future.

🏝️ Benefits

• Health insurance (medical, dental, vision).

• Retirement contribution with an additional matching option.

• Paid life insurance and accidental death & dismemberment (AD&D).

• Paid time off (PTO, compassionate leave, volunteer, holiday, parental).

• Short-term and long-term disability coverage.

• Pre-tax transit benefits.

• Flexible spending accounts.

• Supplemental insurance options.

• Summer hours.

• Eligible employer for the Public Service Loan Forgiveness (PSLF) program.

• Remote work arrangement.

• In-person attendance at annual all-staff summits and other meetings.