Lead Data Engineer

This is a fully remote position, open to applicants in United States.

📋 Description

• Provide guidance to clients, including data owners, analytics users, and executive stakeholders, on how to convert business inquiries into AI-compatible data architectures.

• Solve intricate data engineering challenges independently across various industries as a member of a small team.

• Design and establish AI-optimized data platforms, which include cloud data warehouses, lakehouses, ETL/ELT pipelines, orchestration jobs, and analytical layers.

• Develop and enhance semantic and analytical layers that support Snowflake Cortex, Databricks Genie, BI platforms, and emerging AI copilots.

• Deliver scalable solutions utilizing Snowflake, Databricks, dbt, Fivetran, and cloud-native orchestration frameworks.

• Create modern ELT/ETL pipelines for structured, semi-structured, and unstructured data.

• Design data models focusing on metrics layers, knowledge graphs, and semantic consistency for AI utilization.

• Write production-ready SQL, Python, and Spark code following Git and CI/CD best practices.

• Utilize AI-assisted techniques for data exploration, quality assurance, schema generation, documentation, lineage, and transformation acceleration.

• Contribute to the AI-driven data engineering and infrastructure practice, including internal accelerators, patterns, and client-ready architectures.

• Collaborate with analytics, data science, and ML teams to implement AI-enabled analytics, features, and inference pipelines.

• Lead project delivery in small teams, support practice and business development, and offer innovative ideas.

• Note that developing machine learning models or algorithms is not part of this role.

⛳️ Requirements

• A degree in Computer Science, Engineering, Mathematics, or equivalent experience is required.

• Proven experience in managing stakeholders and collaborating with clients.

• Excellent written and verbal communication skills are essential.

• Minimum of 5 years of experience working with relational databases and query languages.

• At least 5 years of experience in building production data pipelines for structured, semi-structured, and unstructured data.

• 5+ years of experience in data modeling techniques, such as star schema, entity-relationship, or data vault.

• 5+ years of experience writing clean, maintainable, and robust code in Python, Scala, Java, or similar programming languages.

• Preferred experience of 5+ years with dbt Core/Cloud.

• Experience utilizing AI tools like Codex, Claude, Copilot, Snowflake Cortex Code, and/or Databricks Genie Code to enhance data platform engineering workflows is preferred.

• Capability to independently manage an individual workstream and lead a small team of 1–2 persons.

• Strong understanding of software engineering concepts and best practices.

• DevOps experience is required.

• Experience with cloud data warehouses, particularly Databricks or Snowflake, is essential.

• One or more certifications in Databricks or Snowflake are strongly preferred.

• Familiarity with cloud ETL/ELT tools such as Fivetran, dbt, Matillion, Informatica, or Talend is preferred.

• Experience with cloud platforms like AWS, Azure, or GCP and container technologies such as Docker or Kubernetes is preferred.

• Experience with Apache Spark is desirable.

• Background in preparing data for analytics and adhering to a data science workflow to achieve business outcomes is preferred.

• Consulting experience is strongly preferred.

• Willingness to travel is required.

🏝️ Benefits

• Fully remote work environment.

• Opportunity to work from headquarters in Sandy Springs, GA for applicants located in Atlanta.

• Access to certifications to validate your skills.

• Engage with modern data engineering platforms and tools.

• Opportunity to contribute to internal accelerators, patterns, and client-ready architectures.

• Chance to present innovative ideas and initiatives to the company.