Data Engineering Manager

This is a fully remote position, open to applicants in Virginia.

📋 Description

• Define the technical architecture that includes development tools and methodologies for innovative technology solutions throughout the enterprise.

• Offer strategies for technology, application, and data management that ensure web, portal, and/or analytics systems meet customer expectations and integrate with both internally developed systems and vendor-supplied solutions, whether hosted locally or in the cloud.

• Educate and mentor development teams on new technology tools and methodologies.

• Lead the definition of requirements for the training curriculum for software developers.

• Guide senior technical staff in designing the application and web integration components of new internally developed and procured solutions.

• Design and develop a metadata-driven file ingestion framework on Databricks that aligns with the medallion architecture.

• Oversee the onboarding of datasets using a pattern-based ingestion approach rather than one-off pipelines.

• Define and uphold data contracts that include schema, keys, and data quality standards.

• Implement scalable ingestion patterns that accommodate header, no-header, and schema evolution scenarios.

• Promote a table-driven configuration model to reduce reliance on static YAML-based onboarding.

• Integrate the ingestion framework with governance tools (e.g., DataHub) for lineage tracking, discovery, and ownership management.

• Ensure high data quality, auditability, and visibility through centralized logging and control tables.

• Collaborate with business and source system teams to grasp data semantics and define keys.

• Establish best practices for file ingestion, schema management, and incremental processing.

• Lead vendor teams to ensure that deliverables align with the target architecture and platform standards.

⛳️ Requirements

• A bachelor's degree in computer science (Required).

• 15 years of experience in a complex computing environment may be considered in lieu of a degree.

• 7 years of experience in information technology (Required).

• 4 years or more in project management (Required).

• Over 8 years in data engineering with extensive experience in modern data platforms (Azure preferred).

• Hands-on expertise in Databricks and PySpark for processing large-scale data.

• Strong comprehension of medallion architecture (Bronze / Silver / Gold).

• Experience in designing metadata-driven or configuration-driven data pipelines.

• In-depth knowledge of data ingestion patterns (batch, file-based, incremental loads).

• Strong background in data modeling concepts (keys, SCD, merge strategies).

• Experience in implementing data quality frameworks and validation rules.

• Proficient in SQL and distributed data processing concepts.

• Experience with Azure Data Lake Storage (ADLS) and file-based ingestion methodologies.

• Familiarity with CI/CD, Git, and deployment practices in data engineering.

• Experience with data catalog and governance tools (DataHub, Collibra, Alation, etc.).

• Exposure to customer 360 / MDM / CRM data integration (Salesforce, Dynamics, etc.).

• Experience in healthcare or regulated environments (HIPAA, PHI handling).

• Familiarity with Unity Catalog and data access governance frameworks.

• Experience in integrating data platforms with workflow tools like ServiceNow for access management.

• Exposure to event-driven ingestion patterns (file triggers, streaming, etc.).

• Experience in large-scale migration or modernization initiatives.

• Strong stakeholder management experience working with business, architecture, and vendor teams.

• Experience in large-scale enterprise data platforms.

• Experience in enabling self-service data access and governance.

• Familiarity with API-driven data integration patterns.

🏝️ Benefits

• Medical, Dental, Vision plans.

• Adoption, Fertility, and Surrogacy Reimbursement up to $10,000.

• Paid Time Off and Sick Leave.

• Paid Parental & Family Caregiver Leave.

• Emergency Backup Care.

• Long-Term, Short-Term Disability, and Critical Illness plans.

• Life Insurance.

• 401k/403B with Employer Match.

• Tuition Assistance – $5,250/year and discounted educational opportunities through Guild Education.

• Student Debt Pay Down – $10,000.

• Reimbursement for certifications and free access to complete CEUs and professional development.

• Pet Insurance.

• Legal Resources Plan.

• Colleagues have the opportunity to earn an annual discretionary bonus if established system and employee eligibility criteria are met.