Lead Data Engineer

This is a fully remote position, open to applicants in Tennessee, +1 more state.

📋 Description

• Guide, mentor, and cultivate the skills of engineers in both the Data Operations and Analytics Engineering teams.

• Foster a culture centered around clear objectives, open feedback, and ongoing development within the team.

• Encourage teamwork and accountability while upholding high standards for quality, efficiency, and continual enhancement.

• Oversee incident management and resolution concerning pipeline failures, cluster challenges, and data quality issues, ensuring comprehensive root cause analysis and preventive measures are implemented.

• Establish and uphold operational standards, runbooks, and on-call protocols for the team.

• Manage and sustain Databricks Workflows and job orchestration, guaranteeing that SLAs are consistently achieved.

• Supervise the design, development, and upkeep of data models and transformations that cater to business intelligence and analytics requirements.

• Define and implement best practices in analytics engineering, including modular transformation patterns, data testing, and code review protocols.

• Collaborate with data analysts and business stakeholders to grasp modeling requirements, ensuring data accuracy, accessibility, and comprehensibility.

⛳️ Requirements

• Over 8 years of hands-on experience with SQL development.

• More than 8 years of experience with relational and non-relational databases, possessing a solid foundation in data modeling, schema design, and query optimization.

• At least 5 years of professional experience in developing scalable solutions utilizing Python or a comparable object-oriented language.

• Proficient in Databricks, Spark, and Delta tables, with a preference for experience in large-scale distributed data processing.

• Practical experience in operating and monitoring data pipelines at scale within a production setting.

• Strong understanding of Lakehouse and Medallion architectures.

• Familiarity with Azure data services (ADLS Gen2, Azure Data Factory, Event Hubs, or equivalents).

• Knowledge of Unity Catalog for data governance, access control, and data lineage.

• Proven track record in designing Data Integration/ETL pipelines using tools like Azure Data Factory or similar.

• Exceptional communication skills, capable of articulating technical concepts and operational updates to both technical and non-technical audiences.

🏝️ Benefits

• Values-driven culture.

• Support from industry leaders in oncology, technology, and finance.

• Opportunity to collaborate with talented individuals.