Senior Data Engineer

This is a fully remote position, open to applicants in United States.

📋 Description

• Design, construct, and enhance the modern cloud data platform and AI strategy for Penn Foster Group.

• Offer technical leadership through architectural guidance, code reviews, mentorship, and best engineering practices.

• Coach and nurture junior and mid-level Data Engineers.

• Set standards for software development, testing, CI/CD, documentation, observability, and operational excellence.

• Create, develop, and maintain scalable enterprise data pipelines utilizing SQL, Python, Apache Spark, and Databricks.

• Define development practices for Databricks including Workflows, Repos, Jobs, notebooks, Python libraries, Git integration, cluster policies, SQL Warehouses, and deployment automation.

• Design and refine Delta Lake architectures employing Medallion patterns, Delta optimization, Liquid Clustering, and Photon.

• Construct reusable ingestion frameworks for batch, streaming, CDC, and API-based integrations.

• Enhance Spark workloads for performance, scalability, reliability, and cost efficiency in the cloud.

• Develop trusted semantic data products, Genie Spaces, and Genie Ontologies for Databricks Genie.

• Establish practices for semantic modeling, governed metrics, business metadata, and datasets that are ready for AI.

• Collaborate with Data Science and MLOps teams on machine learning, generative AI, and advanced analytics projects.

• Design scalable Lakehouse architectures catering to analytics, reporting, AI, and operational data products.

• Implement governance with Unity Catalog, which includes lineage, fine-grained security, metadata management, and access controls.

• Advocate for automated testing, monitoring, observability, data quality, and production reliability.

• Act as the technical escalation point for intricate production issues and perform root cause analysis.

• Propel platform modernization and contribute to technical roadmaps and overall platform strategy.

• Work in collaboration with Business Intelligence, Product, Platform Engineering, Security, Data Governance, and MLOps teams.

• Convert business requirements into technical solutions that support reporting, analytics, AI, and strategic decision-making.

⛳️ Requirements

• A Bachelor's degree in Computer Science, Engineering, Information Systems, or a related discipline, or equivalent practical experience.

• 6+ years of experience in Data Engineering, Analytics Engineering, or Data Platform Engineering.

• 2+ years of experience leading technical initiatives and mentoring engineering teams.

• Proven success in developing engineers while promoting engineering excellence and technical standards.

• Excellent communication and collaboration skills with the ability to influence both technical and business stakeholders.

• Proficient knowledge of the Databricks Lakehouse Platform, including Apache Spark (PySpark), Delta Lake, Unity Catalog, Databricks Workflows & Jobs, Repos, SQL Warehouses, Cluster Policies, Serverless Compute, MLflow, Lakehouse Monitoring, Auto Loader, and Delta Live Tables / Lakeflow.

• In-depth knowledge of SQL and Python.

• Strong experience with BI tools such as Power BI, Tableau, or Business Objects.

• Solid Microsoft Azure experience, encompassing ADLS Gen2, Microsoft Entra ID (Azure AD), RBAC, networking, and cloud security.

• Experience with implementing Medallion Architecture, dimensional modeling, and domain-oriented data products.

• Profound understanding of Spark optimization techniques, including Adaptive Query Execution, partitioning, caching, Photon, Liquid Clustering, and Delta optimization.

• Experience with CI/CD pipelines, Git-based development workflows, Infrastructure as Code, and automated testing.

• Strong grasp of data governance, metadata management, data quality, observability, security, and compliance.

• Must reside and perform work within the United States.

• Must be willing and able to travel as necessary.

• Must fulfill applicable pre-employment screening requirements.

• Must complete federal employment eligibility verification through Form I-9.

• Preferred: experience with implementing Databricks Genie, Genie Spaces, and Genie Ontologies; semantic models and AI-ready data products; enterprise BI platforms; machine learning platforms, MLOps, or Generative AI applications; dbt, Great Expectations, or similar tools; Databricks Certified Data Engineer Professional and/or Microsoft Azure certifications.

🏝️ Benefits

• Medical insurance.

• Dental insurance.

• Vision insurance.

• Flexible spending accounts.

• Generous paid time off.

• Sponsored volunteer opportunities.

• 401(k) with a company match.

• Free access to Penn Foster Group's online programs.

• Remote work arrangement.

• Sponsored travel to team meetings, company events, conferences, and summits as applicable.