Senior Data Engineer

This is a fully remote position, open to applicants in Alabama, +1 more state.

📋 Description

• Design, develop, and uphold production data pipelines on Databricks utilizing Python, SQL, Apache Spark, and Delta Lake.

• Construct Bronze-layer ingestion from APIs, relational databases, flat files, cloud storage, and SaaS platforms using dlt and Databricks-native ingestion.

• Create Silver-layer transformations in dbt and Python over Delta Lake to cleanse, standardize, deduplicate, validate, conform, and enrich data.

• Develop Gold-layer data products such as dimensional models, slowly changing dimensions, fact and bridge tables, aggregates, and serving tables.

• Generate curated feature and training tables for ML engineering that are versioned and reproducible.

• Write and maintain data contracts adhering to the Open Data Contract Standard.

• Implement data quality tests for uniqueness, not-null, referential integrity, accepted values, freshness, and business rules.

• Orchestrate ingestion and transformation in Dagster Cloud across development, branch, and production deployments.

• Apply governance through Unity Catalog and manage credentials via Azure Key Vault.

• Execute incremental and merge-based Delta Lake processing while tuning Spark jobs, table layouts, and compute.

• Diagnose production failures, data-quality issues, source-system changes, late-arriving or duplicate data, backfills, and recovery.

• Engage in the on-call rotation and conduct root-cause analysis.

• Construct and manage CI/CD for data assets in Azure DevOps.

• Instrument pipelines for freshness, volume, quality, latency, and cost observability with SLA/SLO alerting.

• Adhere to least-privilege access, secrets management, change management, and audit controls.

• Participate in code reviews and document architecture, runbooks, and data products.

• Collaborate with data architects, analysts, product owners, and business stakeholders to convert requirements into sustainable data solutions.

⛳️ Requirements

• Bachelor’s degree in Computer Science, Information Systems, Engineering, or a related discipline; equivalent practical experience will be considered.

• Over 3 years of experience in building and supporting production data pipelines in a cloud data platform environment.

• Proficient in hands-on Python and SQL.

• Practical experience with Databricks or a similar Spark-based lakehouse, including Delta Lake tables, MERGE, and incremental load patterns.

• Solid understanding of medallion/multi-layer lakehouse design.

• Experience in ingesting data from APIs, relational databases, files, or SaaS applications, including incremental and state-management challenges.

• Working knowledge of dimensional modeling, ELT design patterns, and data quality practices.

• Experience with orchestration and scheduling using Dagster, Databricks Workflows, Airflow, Azure Data Factory, or similar tools.

• Familiarity with Git-based source control, pull request review, automated testing, and CI/CD; preferably Azure DevOps or similar.

• Proven experience troubleshooting production data failures, performance bottlenecks, and source-system changes.

• Experience with pipeline monitoring and alerting, including freshness and quality SLAs.

• Ability to articulate technical designs and trade-offs to both technical and non-technical stakeholders.

• Databricks certification or demonstrated equivalent expertise is preferred.

• Unity Catalog experience is a plus.

• Experience with dbt on Databricks or another Spark transformation framework is preferred.

• Familiarity with Python-based modeling frameworks over Delta Lake, Type 2 history, surrogate keys, and merge strategies is preferred.

• Experience with dlt, Airbyte, Meltano, or Fivetran is advantageous.

• Dagster experience with assets, asset checks, sensors, schedules, and branch deployments is preferred.

🏝️ Benefits

• Comprehensive health, dental, and vision insurance.

• Mental health benefits.

• Employee assistance program.

• Paid time off.

• Paid parental leave.

• Short-term disability.

• Cultural observance day.

• Contributions to healthcare accounts.

• Pension plan.

• 401(k) plan with company matching.

• ProHealth Rewards wellness platform offering cash rewards.

• Annual incentive based on individual and company performance.