Senior Data Engineer

This is a fully remote position, open to applicants in United States.

📋 Description

• Design, develop, and manage scalable, production-ready data pipelines and data products utilizing Spark (Python/SQL) within a Databricks environment.

• Lead the integration and transformation of complex data from various Department of War (DoW) and federal health systems into dependable, reusable data products.

• Create scalable strategies for data ingestion, integration, and exchange, which includes API-based integrations and services.

• Establish reusable patterns, standards, and best practices in data engineering.

• Offer technical advice regarding data architecture, pipeline design, data modeling, integration strategies, and engineering methodologies.

• Monitor, troubleshoot, and enhance production workflows and data pipelines.

• Define and enforce data validation, quality assurance, and governance practices.

• Resolve intricate technical and data integration issues while promoting sustainable solutions.

• Collaborate with engineers, architects, analysts, and customer stakeholders to devise scalable technical solutions.

• Mentor engineers and assist teams in overcoming complex technical challenges.

• Enhance engineering tools, processes, and methodologies, and promote their adoption.

• Take ownership of complex technical domains and uphold engineering quality as the platform expands.

⛳️ Requirements

• Over 10 years of experience in data engineering, software engineering, or a related technical field.

• Significant hands-on experience in designing, constructing, and operating production data pipelines and data products.

• Advanced expertise in Python and SQL.

• Strong background in Apache Spark and distributed data processing.

• Proficiency in Databricks or comparable modern data platforms.

• Experience in designing and maintaining ETL/ELT processes for intricate, large-scale datasets.

• Familiarity with integrating data across diverse systems and developing or utilizing API-based data integrations.

• Solid understanding of data modeling, data architecture, data quality, and data governance principles.

• Experience in troubleshooting and optimizing complex production data pipelines.

• Knowledge of Git-based development workflows and contemporary software engineering practices.

• Experience in terminal/command-line environments.

• Proven track record of providing technical guidance, mentoring engineers, and influencing engineering practices.

• Excellent communication skills with clients and stakeholders.

• Ability to independently manage ambiguity, identify technical risks, and address complex engineering challenges.

• Experience in identifying security, privacy, and compliance concerns and collaborating with security/compliance/legal stakeholders.

• Candidates must be U.S. Citizens with an active DoW Secret (or higher) clearance.

• Bachelor’s Degree in Computer Science, Engineering, Data Science, or a related technical discipline (preferred).

• Nice-to-have: AWS cloud experience; expertise in very large datasets; familiarity with Palantir Foundry; GitLab; Advana or similar DoW data environments; experience with federal health, financial, or other regulated data; knowledge of AI/ML tools.

🏝️ Benefits

• Flexible PTO and all Federal holidays off.

• Health, dental, and vision insurance options.

• Flexible Spending Account (FSA).

• 401k with employer matching.

• Company-sponsored life insurance, including short- and long-term disability coverage.

• Opportunities for professional development, including training, certifications, and conferences.

• Paid cloud developer accounts.

• Referral bonuses.

• Headquarters office perks, such as parking/metro reimbursement and complimentary nitro coffee & lunches.

• Annual social events including 540 Week, hackathons, charity golf tournaments, and more.

• Access to 540’s Washington Capitals and Nationals tickets.