Data Engineer

This is a fully remote position, open to applicants in District of Columbia.

📋 Description

• Design, implement, and sustain data governance and architecture functionalities utilizing Databricks Unity Catalog.

• Establish automated data quality assessments, monitoring, and alerts, while managing remediation workflows.

• Integrate data platforms with Microsoft Fabric and provide trusted Gold-layer data products via enterprise data marketplaces.

• Create and uphold reference architectures, data engineering standards, and technical documentation.

• Develop data annotation workflows and labeling interfaces within Databricks.

• Monitor annotation progress, data quality, and dataset versions for machine learning training and inference.

• Build LLM and RAG pipelines, encompassing document ingestion, chunking, embedding generation, and vector search.

• Assist machine learning pipelines through feature engineering, batch and real-time scoring, and data surveillance.

• Sustain MLflow capabilities for experiment tracking and model registry management.

• Design and uphold data classification schemas, taxonomies, and labeling specifications.

• Take ownership of and improve Gold-layer data products, including experience arrays and friction signal aggregates.

• Integrate new data sources, apply business rules, and develop consumption-ready data views and APIs.

• Maintain data dictionaries, product documentation, and metadata.

• Collaborate with cross-functional teams in an Agile development setting to deliver scalable data solutions.

⛳️ Requirements

• U.S. citizenship is required.

• A University Degree or equivalent experience with a minimum of 8 years of relevant experience, or an Advanced Degree in a related field with at least 5 years of experience.

• Over 5 years of experience in data engineering or analytics engineering, ideally using Databricks or a comparable lakehouse platform.

• Proficiency in SQL, Python, and Delta Lake.

• Experience with Unity Catalog, data governance, and data quality management.

• Proven track record in building and maintaining data pipelines for machine learning training and inference.

• Familiarity with data modeling, classification schemas, and taxonomies.

• Extensive knowledge of data warehousing concepts, dimensional modeling, and ETL processes.

• Experience in an Agile development environment while managing multiple priorities.

• Preferred experience with Microsoft Fabric, Delta Sharing, or cross-platform data sharing.

• Familiarity with annotation tools, NLP, and vector databases is preferred.

• Experience with MLflow or similar tools is preferred.

• A background in metrics and KPIs, information architecture, or IT and infrastructure domains is preferred.

🏝️ Benefits

• Healthcare, wellness, retirement, and work/life benefits.

• Parental (including paternal) leave.

• Flexible work schedules.

• Achievement awards.

• Educational assistance.

• Child/adult backup care.

• Medical, dental, and vision insurance.

• Life insurance.

• Short-term disability.

• Long-term disability.

• 401(k) match.

• Flexible spending accounts.

• Employee assistance program.

• Employee Scholar Program.

• Paid time off and holidays.

• Annual short-term and/or long-term incentive compensation programs may be available.