Staff Data Engineer – Core Migrations

This is a fully remote position, open to applicants in United States.

📋 Description

• Lead the discovery and technical due diligence process by thoroughly investigating poorly documented legacy systems (ETL, stored procedures, reporting layers, file feeds), reconstructing the business logic, and capturing it in lineage maps, mapping specifications, and risk analyses utilized by the team.

• Reverse-engineer complex legacy systems using agents; you will be tasked with reverse-engineering complex, undocumented legacy systems (SSIS packages, stored procedures) where business logic is embedded but not documented. You must reconstruct the intent of these systems to create modern, stable equivalents.

• Drive ambiguity to resolution by identifying unknowns early, taking ownership of open questions, and proactively seeking answers from clients, SMEs, and Operations rather than waiting for blockers to be removed.

• Architect and build migration pipelines that transform intricate legacy logic into production-grade Airflow DAGs and Spark jobs on the Machinify platform, ensuring inclusion of edge cases, payer-specific carve-outs, and business-rule exceptions, while overseeing the entire flow from ingestion to reconciliation and steady-state handoff.

• Make and document critical architectural decisions by taking ownership of significant choices (pipeline design, partitioning strategy, validation approach) and providing a clear rationale so others can learn and build upon your work.

• Prove correctness at scale by creating automated reconciliation frameworks that confidently verify that migrated outputs match the source down to the individual row.

• Own the program from start to finish, serving as the sole technical owner from kickoff through go-live and hypercare: define the scope, sequence, and track the work, identify risks in advance, align Operations, Client Success, Platform Engineering, SMEs, and the client, and facilitate UAT through to sign-off.

• Raise the bar for the practice by institutionalizing migration knowledge through the creation of runbooks and retrospectives, mentoring L3/L4 engineers to develop independent capabilities, and transforming recurring migration challenges into scalable, reusable tools.

⛳️ Requirements

• 8+ years of experience as a hands-on Data Engineer or Software Engineer, with a proven track record of independently managing complex, multi-stakeholder technical projects from inception to completion.

• Strong proficiency in Python and SQL, with the ability to navigate complex, unfamiliar legacy code, rather than just greenfield development.

• Deep understanding of Apache Spark, including distributed processing, performance tuning, partitioning, and debugging on a large scale.

• Advanced knowledge of Apache Airflow; capable of designing and authoring production DAG architectures from the ground up.

• Experience with AI-assisted development, actively utilizing coding agents (Claude, Copilot, Cursor, or equivalent) as a force multiplier in daily engineering tasks. Comfortable with prompt-engineering for code generation, debugging, and documentation, with the ability to critically evaluate and trust AI outputs.

• Familiarity with legacy ETL stacks; able to reconstruct intent from code and accurately translate it into a modern stack (e.g., from SSIS, SQL Server stored procedures, T-SQL, or equivalent).

• Experience in data validation and reconciliation at scale, with a background in designing automated frameworks that verify that migrated data aligns with source systems with high confidence.

• Proficient in AWS, particularly with S3, and possess a solid understanding of cloud-native data patterns, including partitioned object storage, secure cross-account access, and cost/performance trade-offs.

• Demonstrated ability to coordinate across engineering, operations, and client-facing teams simultaneously, maintaining velocity without losing focus.

• Strong technical communication skills, capable of producing discovery specifications, architecture documentation, and stakeholder updates that are accurate, concise, and actionable for both technical and non-technical audiences.

• A rigorous, data-first mindset with extensive experience in building automated reconciliation frameworks that ensure 100% data fidelity between legacy and modern systems.

🏝️ Benefits

• Work from anywhere in the US! Machinify is digital-first.

• Comprehensive Medical/Dental/Vision coverage for employees and their families.

• A flexible and trusting environment where you’ll feel empowered to perform at your best.

• Unlimited Flexible Time Off (FTO).

• Competitive salary, equity, and a 401(k) plan with employer matching.