Senior Data Engineer

This is a fully remote position, open to applicants in Virginia.

📋 Description

• Design, develop, and maintain scalable data pipelines and data products that are production-ready using Spark (Python/SQL) within a Databricks environment.

• Oversee the integration and transformation of intricate data from various DoW and federal health systems into dependable, reusable data products.

• Create scalable methods for data ingestion, integration, and exchange, including API-based services and integrations.

• Establish and advocate for reusable data engineering patterns, standards, and best practices.

• Offer technical direction on data architecture, pipeline design, data modeling, integration strategies, and engineering practices.

• Monitor, troubleshoot, and enhance production workflows and data pipelines.

• Define and enforce data validation, quality, and governance practices.

• Resolve complex technical and data integration issues, identify root causes, and implement sustainable solutions.

• Collaborate with engineers, architects, analysts, and customer stakeholders to convert complex data requirements into scalable technical solutions.

• Provide mentorship and technical guidance to fellow engineers.

• Identify opportunities for improving engineering tools, processes, and patterns, and assist in promoting their adoption.

• Assume responsibility for complex technical areas while maintaining engineering quality, consistency, and cohesion as the platform and portfolio of data products expand.

⛳️ Requirements

• Master’s degree in Computer Science, Information Systems, Software Engineering, or a related discipline.

• Over 10 years of experience in data engineering, software engineering, or a related technical field.

• Extensive hands-on experience in designing, building, and managing production data pipelines and data products.

• Advanced skills in Python and SQL.

• Strong background in Apache Spark and distributed data processing.

• Experience with Databricks or similar modern data platforms.

• Proven experience in designing and maintaining ETL/ELT processes for complex, large-scale datasets.

• Proficient in troubleshooting and optimizing complex production data pipelines for performance, reliability, and scalability.

• Familiarity with Git-based development workflows and contemporary software engineering practices.

• Demonstrated ability to provide technical guidance, mentor engineers, and influence engineering practices.

• Excellent client and stakeholder communication skills, capable of translating technical concepts and recommendations for both technical and non-technical audiences.

• Ability to independently navigate ambiguity, identify technical risks, and drive complex engineering challenges to resolution.

• Experience identifying security, privacy, and compliance issues while collaborating with security/compliance/legal stakeholders.

• Must possess and maintain an active Secret clearance.

• Experience with AWS cloud services.

• Background in working with very large datasets, including those containing billions of records.

• Familiarity with Palantir Foundry.

• Experience with GitLab.

• Knowledge of working with Advana or similar DoW data environments.

• Experience handling federal health, financial, or other regulated and sensitive data.

• Proficient in using AI/ML to enhance work efficiency, including automating routine tasks and accelerating development and debugging processes.

🏝️ Benefits

• Competitive base salary.

• Opportunities for variable compensation.

• Health insurance coverage.

• Access to flexible spending accounts.

• Health savings accounts available.

• Retirement savings plans.

• Life and disability insurance programs.

• Paid and unpaid time off from work.