Senior Data Engineer – Full Stack

This is a fully remote position, open to applicants in United States.

📋 Description

• Create and implement Command Line Interface (CLI) tools, scripts, and internal utilities to streamline repetitive tasks across the data platform, such as:

• Orchestrating and executing pipelines

• Managing data governance workflows

• Synchronizing metadata

• Setting up and configuring environments

• Developing test harnesses

• Automating workflows on Databricks, which includes:

• Deploying and scheduling jobs

• Provisioning environments

• Executing MLOps processes utilizing APIs, Terraform, or Databricks SDK

• Establishing and implementing comprehensive testing frameworks:

• Conducting integration tests for pipelines

• Performing end-to-end validation of ETL/ELT workflows

• Testing and validating ML inference workflows

• Enhancing the overall productivity, scalability, and reliability of the data and ML engineering environment

• Designing lightweight internal tools and dashboards with frameworks like React, Streamlit, or similar technologies to:

• Visualize data pipelines and workflows

• Showcase model inference capabilities

• Offer configuration and operational controls

• Facilitate internal productivity tracking and dashboarding

• Collaborate with cross-functional teams to pinpoint automation prospects and apply best practices

⛳️ Requirements

• Over 8 years of experience in Data Engineering, Platform Engineering, or similar roles

• Proficient in Python and scripting for automation and backend development

• Practical experience with the Databricks platform and its ecosystem

• Knowledge of APIs, Terraform, and/or Databricks SDK for automation purposes

• Strong grasp of ETL/ELT pipelines and data platform architecture

• Experience in building testing frameworks for data pipelines and ML workflows

• Familiarity with CLI tool development and system automation

• Understanding of MLOps principles and methodologies

• Experience with contemporary development practices, which include:

• Specification-driven development

• Utilizing coding agents or automation-supported development tools

• Version control and CI/CD pipelines

🏝️ Benefits

• Flexible work arrangements