Lead Data Engineer

This is a fully remote position, open to applicants in Maryland.

📋 Description

• Design and execute the foundational DuckDB / Apache Iceberg data platform on AWS

• Define the physical data structure, schema, partitioning, file organization, snapshots, compaction, and optimization strategies for Iceberg tables

• Integrate DuckDB with Amazon S3, Apache Iceberg, and the AWS Glue Catalog

• Establish conventions for bronze, silver, and gold data layers along with reusable engineering patterns

• Lead and mentor Data Engineers, offering technical direction, conducting code reviews, and providing engineering guidance

• Build and sustain analytical datasets that support application services, reporting, analytics, and data science initiatives

• Develop reusable frameworks for incremental loads, change data capture (CDC), schema evolution, late-arriving data, backfills, and historical reconstruction

• Collaborate with the Data Architect to implement a unified canonical data model and source-to-canonical mappings

• Oversee the migration and harmonization of datasets from various source systems

• Implement reconciliation processes to ensure completeness and accuracy between source and target systems

• Establish automated data quality monitoring, operational observability, and testing protocols

• Deploy agentic and AI-enabled data engineering patterns for source analysis, mapping, pipeline generation, troubleshooting, reconciliation, and quality assurance

• Facilitate the integration of structured data with related document and geospatial datasets

• Lead the development of certified data products, analytical datasets, and business metrics

• Optimize DuckDB queries and Iceberg datasets for enhanced performance and efficiency

• Create engineering documentation, operational procedures, and technical handoff materials

⛳️ Requirements

• Bachelor's degree in Computer Science, Data Engineering, Information Systems, Engineering, or a related discipline

• Over 6 years of relevant experience in data engineering or data platforms, including experience in leading or mentoring engineering teams

• Proficient hands-on experience with DuckDB and Apache Iceberg

• Advanced SQL skills and strong experience in Python development

• Experience in building cloud-based analytical platforms utilizing object storage and columnar formats such as Parquet

• In-depth understanding of partitioning, schema evolution, snapshots, compaction, incremental processing, and analytical query optimization

• Proven experience in implementing bronze/silver/gold or similar layered data architectures and transforming heterogeneous enterprise datasets

• Familiarity with AWS data services, particularly Amazon S3 and AWS Glue

• Experience in designing reusable, metadata-driven ingestion and transformation frameworks

• Proven track record in implementing automated data quality, reconciliation, monitoring, and operational controls

• Experience with agentic or AI-enabled data engineering solutions, including the application of agents or AI-assisted workflows for data mapping, pipeline generation, validation, testing, or troubleshooting

• Practical experience with agentic frameworks such as LangChain, LangGraph, or similar frameworks for constructing tool-using or multi-step AI workflows

• Strong skills in troubleshooting, performance tuning, and technical leadership

🏝️ Benefits

• Medical, dental, and vision coverage

• 401k plan with employer contributions

• Paid holidays

• Vacation time

• Tuition reimbursement