Senior Data Engineer

This is a fully remote position, open to applicants in Virginia.

📋 Description

• Design, develop, and sustain scalable, production-ready data pipelines and products utilizing Spark (Python/SQL) within a Databricks environment.

• Lead the integration and transformation of complex data from various DoW and federal health systems and sources.

• Create scalable data ingestion, integration, and exchange strategies, including API-based integrations and services.

• Establish and advocate for reusable data engineering patterns, standards, and best practices.

• Provide technical expertise on data architecture, pipeline design, data modeling, integration strategies, and engineering methodologies.

• Monitor, troubleshoot, and enhance production workflows and data pipelines.

• Identify opportunities for improvements in performance, reliability, and scalability.

• Define and implement practices for data validation, quality assurance, and governance.

• Resolve complex technical and data integration issues, identify root causes, and implement sustainable solutions.

• Collaborate with engineers, architects, analysts, and customer stakeholders to convert data requirements into scalable technical solutions.

• Mentor fellow engineers and assist teams in overcoming complex technical hurdles.

• Enhance engineering tools, processes, and patterns while promoting their adoption.

• Own intricate technical domains and uphold engineering quality, consistency, and cohesion as the platform and data product portfolio expands.

⛳️ Requirements

• Active Secret clearance / active DoW Secret clearance.

• Master’s degree in Computer Science, Information Systems, Software Engineering, or a related discipline.

• Over 10 years of experience in data engineering, software engineering, or related technical fields.

• Extensive practical experience in designing, building, and operating production data pipelines and products.

• Advanced skills in Python and SQL.

• Strong background in Apache Spark and distributed data processing.

• Familiarity with Databricks or similar contemporary data platforms.

• Experience in designing and maintaining ETL/ELT processes for complex, large-scale datasets.

• Proven ability to integrate data from diverse systems and to create or utilize API-based data integrations.

• Solid understanding of data modeling, architecture, quality, and governance principles.

• Experienced in troubleshooting and optimizing complex production data pipelines for performance, reliability, and scalability.

• Proficient with Git-based development workflows and modern software engineering practices.

• Experience working in terminal/command-line environments.

• Demonstrated ability to provide technical guidance, mentor engineers, and influence engineering practices.

• Strong communication skills with clients and stakeholders.

• Capability to convey technical concepts to both technical and non-technical audiences.

• Ability to independently manage ambiguity, identify technical risks, and steer complex engineering challenges towards resolution.

• Experience in identifying security, privacy, and compliance issues while collaborating with security/compliance/legal stakeholders.

• Familiarity with AWS cloud services.

• Experience handling very large datasets, including those containing billions of records.

• Knowledge of Palantir Foundry.

• Experience with GitLab.

• Familiarity with Advana or similar DoW data environments.

• Experience with federal health, financial, or other regulated and sensitive data.

• Proficient in utilizing AI/ML to expedite work processes, including automating routine tasks, accelerating development, and debugging.

🏝️ Benefits

• Competitive base salary.

• Opportunities for variable compensation.

• Health insurance coverage.

• Flexible spending accounts.

• Health savings accounts.

• Retirement savings plans.

• Life and disability insurance programs.

• Paid and unpaid time off.

• Industry-leading 401k contributions.

• Collaborative and dynamic work environment.