Senior Data Pipeline Engineer
Candid
Summary
Candid is a nonprofit that provides the most comprehensive data and insights about the social sector. The Senior Data Pipeline Engineer is a senior technical role responsible for the day-to-day development and delivery of Candid's data pipelines. This role blends deep SQL and ETL engineering expertise with hands-on experience across modern cloud data platform technologies to design, build, and tune pipelines that move data reliably from source systems into Candid's data warehouse and lakehouse.
Job description
Position Summary
Candid is a nonprofit that provides the most comprehensive data and insights about the social sector. The Senior Data Pipeline Engineer is a senior technical role responsible for the day-to-day development and delivery of Candid's data pipelines. This role blends deep SQL and ETL engineering expertise with hands-on experience across modern cloud data platform technologies to design, build, and tune pipelines that move data reliably from source systems into Candid's data warehouse and lakehouse.
Responsibilities
- Pipeline development: Architect, develop, and maintain ETL/ELT and C#/.NET pipeline services (SSIS, Airflow, AWS Lambda, enhancer and message-processing services) that move data reliably from source systems into Candid's warehouse and lakehouse.
- Cloud platform & performance: Support Candid's evolving cloud data platform, including Iceberg ingestion and transformation pipelines, DynamoDB, and query performance tuning via Trino/Starburst.
- Data quality & incident resolution: Monitor data quality end-to-end and investigate production pipeline issues at the root cause, documenting findings and driving fixes through to reprocessing.
- Source onboarding & schema coordination: Scope and onboard new external and regulatory data sources end-to-end, and contribute to schema and architecture decisions, coordinating changes across ingestion and transformation layers.
- Documentation, mentorship & process: Maintain SOP and technical documentation, mentor and review the work of Associate-level engineers, and participate in the team's release and deployment process.
- Partnership & support: Partner with the Data Operations Manager on architecture and roadmap decisions, and provide escalation-level support for mission-critical pipeline issues, including after-hours as needed.
Requirements
- 5+ years of experience in database engineering, data engineering, or ETL development, with demonstrated ownership of production data pipelines.
- Expert-level Transact-SQL skills (stored procedures, views, indexing) and hands-on C#/.NET development experience.
- Proven experience architecting ETL processes with SSIS; experience with orchestration platforms such as Apache Airflow preferred.
- Experience with AWS and open table formats (Apache Iceberg, Parquet); AWS Lambda, DynamoDB, and Trino/Starburst a plus.
- Strong data quality and root-cause troubleshooting skills, with experience resolving production data incidents.
- Familiarity with source control, CI/CD, and release/deployment processes across Dev/QA/Prod environments.
- Ability to mentor and review the work of other engineers; excellent written and verbal communication.
- Good business judgment, independent prioritization, and the ability to work well in a distributed team.
- Willingness to perform other duties and special projects as needed/requested.
- Sensitivity and respect for racial, gender, sexual orientation, and cultural differences.
- Commitment to Candid's values: driven, direct, accessible, curious, and inclusive.
Requirements
- 5+ years of experience in database engineering, data engineering, or ETL development, with demonstrated ownership of production data pipelines.
- Expert-level Transact-SQL skills (stored procedures, views, indexing).
- Hands-on C#/.NET development experience.
- Proven experience architecting ETL processes with SSIS.
- Experience with AWS and open table formats (Apache Iceberg, Parquet).
- Strong data quality and root-cause troubleshooting skills.
- Experience resolving production data incidents.
- Familiarity with source control, CI/CD, and release/deployment processes across Dev/QA/Prod environments.
- Ability to mentor and review the work of other engineers.
- Excellent written and verbal communication.
- Good business judgment and independent prioritization.
- Ability to work well in a distributed team.
- Legally authorized to work in the United States without employer sponsorship.
Benefits
- Health insurance
- Dental insurance
- Vision insurance
- Retirement contribution
- Retirement match option
- Paid life insurance
- AD&D insurance
- Paid leave time
- Short-term disability
- Long-term disability
- Pre-tax transit
- Flexible spending accounts
- Supplemental insurance
- Summer hours
- PSLF eligible employer
Company overview
Candid is a leading medical device company specializing in clear aligner therapy for orthodontic treatment. Founded in 2017, the company initially operated as a direct-to-consumer provider but has since evolved into a doctor-driven model through its CandidPro platform. Candid develops and manufactures custom-made, clear plastic aligners designed to straighten teeth and correct mild to moderate orthodontic issues. The company leverages advanced 3D printing technology, including partnerships with Carbon's DLS process, to produce highly accurate aligner models at scale. Candid's innovative approach includes a patent-pending treatment planning methodology, AI-powered remote monitoring through its ProMonitoring app, and a comprehensive ecosystem of hardware and software solutions. The company focuses on providing affordable, effective orthodontic treatment with fewer in-office visits, making teeth straightening more accessible to patients. CandidPro aligners are manufactured using BPA-free medical-grade polymers and are designed to treat various dental alignment issues while offering a more comfortable and convenient alternative to traditional braces.