Lead Data Engineer

This is a fully remote position, open to applicants in United States.

📋 Description

• Act as the senior technical leader for client projects.

• Take ownership of solution architecture, provide guidance during the build process, manage delivery risks, and ensure that the delivered solutions meet established commitments.

• Design and implement scalable and reliable data pipelines and lakehouse architectures on Databricks and various cloud platforms.

• Write and review production-quality code while establishing engineering standards.

• Convert complex client requirements into technical designs, reference architectures, and data models.

• Oversee technical scope and timelines while identifying potential blockers early in the process.

• Collaborate with project managers and client stakeholders to keep projects on schedule.

• Mentor junior and mid-level engineers through pairing, code reviews, and direct feedback.

• Contribute to technical blogs, reference architectures, and internal documentation.

• Establish and uphold DataOps standards, including automated testing, pipeline observability, version control, CI/CD, and documentation.

⛳️ Requirements

• Over 8 years of practical experience in data engineering, focusing on designing and delivering production-grade data platforms.

• Expert knowledge in Apache Spark, including understanding of runtime internals, performance tuning, and optimization techniques.

• Ability to produce clean, production-quality code in Python.

• Proficiency in Scala is a significant advantage.

• Experience in building and deploying solutions on Databricks, including Delta Lake architectures, Unity Catalog governance, and Databricks Workflows.

• Databricks certification is highly desirable.

• Familiarity with at least two major cloud platforms (AWS, Azure, GCP), with substantial expertise in at least one of them.

• Experience with cloud-native services such as AWS Redshift/Glue/S3, Azure Synapse/Data Factory/ADLS, or Google BigQuery/Dataflow/GCS.

• Proven history of leading data engineering projects from start to finish in a client-facing or consulting environment.

• Experience in managing technical scope, stakeholder expectations, and delivery timelines effectively.

• Knowledge of CI/CD for data pipelines, automated testing, observability, and infrastructure-as-code methodologies.

• Experience encompassing ETL/ELT design, data warehousing, lakehouse architecture, and data modeling.

• Strong communication skills, capable of engaging with both technical and non-technical stakeholders.

🏝️ Benefits

• Competitive salary and performance-based bonuses.

• Opportunities for professional development and continuous learning.

• Flexible working hours and remote work options.

• Comprehensive health insurance and wellness programs.

• Collaborative and innovative work environment.