Data Engineering Manager

This is a fully remote position, open to applicants in United States.

📋 Description

• Provide technical leadership for the Data Engineering team through guidance on architecture, code reviews, mentoring, and adherence to engineering best practices.

• Mentor and develop junior and mid-level Data Engineers.

• Establish engineering standards for software development, testing, CI/CD, documentation, observability, and operational excellence.

• Lead technical design discussions, assess architectural tradeoffs, and promote the adoption of modern engineering practices.

• Act as the technical lead for Penn Foster Group's Databricks Lakehouse platform.

• Design, build, and support scalable enterprise data pipelines utilizing SQL, Python, Apache Spark, and Databricks.

• Define best practices for Databricks development covering Workflows, Repos, Jobs, notebooks, Python libraries, Git integration, cluster policies, SQL Warehouses, and deployment automation.

• Design and optimize Delta Lake architectures employing Medallion patterns, Delta optimization, Liquid Clustering, and Photon.

• Create reusable ingestion frameworks for batch, streaming, CDC, and API-based integration patterns.

• Optimize Spark workloads to enhance performance, scalability, reliability, and cloud cost efficiency.

• Develop trusted semantic data products, Genie Spaces, and Genie Ontologies for Databricks Genie.

• Establish practices for semantic modeling, governed metrics, business metadata, and AI-ready datasets.

• Collaborate with Data Science and MLOps teams on initiatives involving machine learning, generative AI, and advanced analytics.

• Design scalable Lakehouse architectures for analytics, reporting, AI, and operational data products.

• Implement enterprise governance using Unity Catalog, encompassing lineage, security, metadata management, and access controls.

• Advocate for automated testing, monitoring, observability, data quality, and production reliability.

• Serve as the technical escalation point for complex production issues and lead root cause analysis efforts.

• Drive platform modernization initiatives and contribute to technical roadmaps and platform strategy.

• Collaborate with teams from Business Intelligence, Product, Platform Engineering, Security, and MLOps.

• Translate business requirements into technical solutions for reporting, analytics, AI, and strategic decision-making.

⛳️ Requirements

• Bachelor's degree in Computer Science, Engineering, Information Systems, or a related field, or equivalent practical experience.

• Over 8 years of experience in Data Engineering, Analytics Engineering, or Data Platform Engineering.

• At least 3 years of experience leading technical initiatives and mentoring engineering teams.

• Proven track record in developing engineers and promoting engineering excellence and technical standards.

• Strong communication and collaboration skills with both technical and business stakeholders.

• Expert knowledge of the Databricks Lakehouse Platform, including Apache Spark/PySpark, Delta Lake, Unity Catalog, Workflows & Jobs, Repos, SQL Warehouses, Cluster Policies, Serverless Compute, MLflow, Lakehouse Monitoring, Auto Loader, and Delta Live Tables/Lakeflow.

• Proficient in SQL and Python.

• Significant experience with BI tools such as Power BI, Tableau, or Business Objects.

• Strong familiarity with Microsoft Azure, including ADLS Gen2, Microsoft Entra ID, RBAC, networking, and cloud security.

• Experience in implementing Medallion Architecture, dimensional modeling, and domain-oriented data products.

• In-depth understanding of Spark optimization techniques, including Adaptive Query Execution, partitioning, caching, Photon, Liquid Clustering, and Delta optimization.

• Experience in implementing CI/CD pipelines, Git-based workflows, Infrastructure as Code, and automated testing.

• Strong understanding of data governance, metadata management, data quality, observability, security, and compliance.

• Successful completion of a role-specific assessment.

• Successful completion of applicable pre-employment screening.

• Completion of federal employment eligibility verification through Form I-9.

• Preferred: Experience with Databricks Genie, semantic models, AI-ready data products, machine learning/MLOps, Generative AI, dbt, Great Expectations, Databricks Certified Data Engineer Professional, and/or Microsoft Azure certifications.

🏝️ Benefits

• Medical insurance.

• Dental insurance.

• Vision insurance.

• Flexible spending.

• Generous paid time off.

• Sponsored volunteer opportunities.

• 401K with a company match.

• Free access to all of our online programs.

• Remote work arrangement.

• On-camera collaborative work environment.