Lead Data Engineer

This is a fully remote position, open to applicants in United States.

📋 Description

• Design and sustain batch data pipelines that extract data from Workday (HCM, Finance, Student) and legacy systems into the Bronze layer.

• Create transformation logic across Bronze, Silver, and Gold layers utilizing Spark, Python, and SQL within Databricks.

• Structure data for analytical use, ensuring a balance between performance, reliability, and maintainability.

• Manage and schedule workloads (for example, Databricks Workflows, Azure Data Factory, or Fabric pipelines) and oversee their reliability.

• Establish data quality checks and validation processes within pipelines.

• Act as the senior technical leader for intricate integrations and cross-institution data initiatives (Lead Data Engineer).

• Set and uphold engineering standards, patterns, and best practices.

• Provide mentorship and technical guidance, along with code reviews for fellow engineers.

⛳️ Requirements

• Data Engineer: 3–6 years of experience in developing production data pipelines.

• Lead Data Engineer: 7–10 years of experience, showcasing technical leadership.

• Proficient in Apache Spark, Python, and SQL in a Databricks setting.

• Experience in designing and constructing medallion or layered data architectures (Bronze, Silver, Gold, or equivalent).

• Demonstrated ability to integrate data from various ERP, SIS, and operational source systems into an analytics platform.

• Familiarity with batch pipeline orchestration and scheduling (Databricks Workflows, Azure Data Factory, or similar tools).

• Understanding of data governance, security, and access principles in a shared or multi-tenant environment.

• Strong analytical skills with a continuous emphasis on data quality and reliability.

• Excellent communication and teamwork abilities in a remote working environment.

• Bachelor’s degree in a technical discipline or equivalent practical experience.

🏝️ Benefits

• Competitive salary.

• Comprehensive benefits for full-time employees.

• Opportunities for professional growth and development.

• Flexibility to support work-life balance.

• Initiatives for charitable fundraising.