Senior Data Engineer

This is a fully remote position, open to applicants in Michigan.

📋 Description

• Develop and sustain data ingestion pipelines to transfer data from EP’s business systems into Databricks.

• Automate quality assurance checks and source reconciliation to detect missing, incorrect, or duplicate data.

• Create and construct silver and gold data layers.

• Provide governed data products that facilitate reporting, cross-system matching, and identity resolution.

• Oversee daily platform operations, which include job scheduling, monitoring, alerting, data freshness, access management, and cost optimization.

• Maintain runbooks, test recovery procedures, and document knowledge for platform support.

• Manage CI/CD pipelines and deployments across development, testing, and production environments utilizing Azure DevOps.

• Collaborate with EP’s IT vendor regarding Azure infrastructure requirements beyond the data platform.

• Safeguard systems of record, enforce appropriate user access, and verify licensing and platform constraints prior to deployment.

• Work in partnership with business stakeholders to translate requirements into practical, reliable data solutions.

• Design and enforce confidentiality, retention, least-privilege access, and audit controls for sensitive client and matter data.

• Support bounded AI and partner pilots with data masking, evaluation data, and human approval processes.

• Assess the existing Databricks platform, identify areas for improvement, and help shape the evolving data function.

• Collaborate with the Manager of Data Services and Products, as well as business, application, IT, and security stakeholders, on architecture and build-versus-buy decisions.

• Perform additional duties as assigned by management.

⛳️ Requirements

• Bachelor’s or Master’s degree in Computer Science, Engineering, or a related field, or equivalent practical experience.

• 4+ years of experience in the field of data engineering.

• 2+ years of hands-on experience in building and operating Databricks solutions in a production setting.

• Familiarity with multi-task Lakeflow Jobs over Delta tables governed by Unity Catalog.

• Strongly preferred experience in deploying through Declarative Automation Bundles (formerly Databricks Asset Bundles) or equivalent infrastructure-as-code.

• Capability to articulate a production pipeline built, including design choices, failures, and operational challenges.

• Strong communication skills to engage clearly with non-technical partners and relate engineering decisions to business outcomes.

• Proficiency in Python, including experience with testing using pytest and packaging.

• Preferred experience in PySpark.

• Strong expertise in SQL and dimensional modeling, particularly in serving governed datasets to BI tools.

• Experience in building REST API and SaaS ingestion pipelines, covering aspects such as pagination, rate limits, schema drift, and credential rotation.

• Familiarity with Git-based CI/CD workflows.

• Experience with Azure DevOps is a plus.

• Experience utilizing AI-assisted development tools and reviewing, testing, and validating generated code prior to deployment.

• Experience in building Lakeflow pipelines, managed connectors, entity resolution, or record linkage is advantageous.

• Experience managing Azure resources supporting a data platform, ideally as infrastructure-as-code, is a plus.

• All work must be conducted within the United States.

🏝️ Benefits

• An opportunity to engage in challenging, dynamic, and highly rewarding work with an inspiring mission.

• A team of intelligent, creative, and dedicated immigration professionals who value respectful and collaborative working relationships.

• An inclusive firm culture that celebrates diverse perspectives, backgrounds, identities, and experiences.

• People-centered workplace policies and flexible work schedules, including opportunities for remote work.

• Competitive compensation package.

• Health, dental, vision, and life insurance benefits.

• 401(k) retirement savings plan with employer matching.

• Generous paid time off policy.

• Casual dress code for employees working in person at the Troy or Ann Arbor offices.

• Abundant free food for employees working in person at the Troy or Ann Arbor offices.

• Regular opportunities to experience international cuisine and culture for employees working in person at the Troy or Ann Arbor offices.