Lead Data Engineer – PySpark, Palantir Foundry

This is a fully remote position, open to applicants in Washington.

📋 Description

• Provide exceptional client value and maintain high levels of client satisfaction.

• Design, improve, and sustain production-quality data pipelines that facilitate model output aggregation and downstream risk analysis.

• Develop and enhance governance practices for repositories, encompassing branching strategies, pull request standards, merge policies, release tagging, and version control workflows.

• Manage release engineering practices to ensure reproducible, traceable, and auditable production releases.

• Refactor and enhance pipeline code to improve modularity, maintainability, scalability, and documentation quality.

• Create configuration-driven pipeline patterns that span across various environments and releases.

• Assist in testing, validation, benchmarking, and change management for essential data pipelines.

• Collaborate with data scientists, machine learning engineers, data engineers, product stakeholders, and other technical teams.

• Ensure alignment among teams regarding schemas, interfaces, inputs, and delivery expectations.

• Convey complex technical concepts into straightforward updates for both technical and non-technical stakeholders.

• Enhance structure and governance in codebases, repositories, and engineering workflows.

• Contribute to engineering best practices within a regulated, audit-sensitive delivery environment.

⛳️ Requirements

• 10-15+ years of experience in data engineering, data science, machine learning engineering, or related fields, utilizing Python.

• Proven experience in leading technical teams and managing enterprise-scale data initiatives.

• Strong proficiency in PySpark, SQL, and cloud services.

• Capability to enhance, refactor, or stabilize existing codebases and pipeline environments.

• Familiarity with cloud-optimized datasets, efficient partitioning strategies, and large-scale spatial operations.

• Understanding of machine learning model outputs that integrate into downstream data pipelines, platforms, or production systems.

• Experience in highly regulated sectors such as utilities, financial services, healthcare, insurance, or similar fields.

• Track record of designing maintainable, scalable, and well-documented cloud-based data infrastructure or modern data platform environments.

• Expertise in supporting reproducibility, dataset versioning, release traceability, and audit readiness.

• Ability to define expected inputs, outputs, schemas, and interfaces across technical teams.

• Excellent communication skills.

• Detail-oriented and governance-focused approach to engineering.

• Hands-on experience with Git-based workflows, code reviews, branching strategies, and release management.

• Familiarity with Palantir Foundry is highly preferred.

• Experience with GIS technologies and geospatial data platforms.

🏝️ Benefits

• Competitive base salary.

• Performance-based bonuses.

• Additional incentives.

• Opportunities for training and mentorship.

• Project opportunities aimed at career development.

• A supportive, globally connected work environment.