Senior/Lead Data Engineer – AI-Native Aftermarket Platform

This is a fully remote position, open to applicants in Colombia.

📋 Description

• Create and develop resilient, idempotent data pipelines from the ground up using a contemporary data stack.

• Design star and snowflake schemas, crafting precise, grain-aware SQL for the development of scalable data marts.

• Produce production-quality, unit-tested Python code at the module level, following robust engineering practices such as type hinting and testing.

• Construct and validate dbt models across staging, intermediate, and mart layers while overseeing the overall project structure.

• Author and deploy jobs utilizing Databricks Asset Bundles (DAB) in accordance with established architectural patterns.

• Execute stringent data quality checks at source, intermediate, and destination layers to avoid unnoticed drops of null values or duplicates.

• Uphold data governance through thorough dbt tests and a strict documentation-at-merge-time policy.

• Securely operate within a multi-repository architecture, employing service principals and ensuring no personal credentials are used in production deployments.

• Conduct cross-repository exposure checks before merging changes that could break schemas.

• Take ownership of data pipelines from start to finish, making essential technical design choices and guiding mid-level engineers through meaningful code reviews.

• Define the overarching technical strategy across core data systems, including modeling standards, branching strategies, observability thresholds, and secret management protocols.

• Serve as a technical leader to remove obstacles for the team and actively engage in hiring panels to expand the engineering organization.

⛳️ Requirements

• Proficiency in SQL and dimensional modeling techniques, including medallion architecture, SCDs, and grain management.

• Demonstrated ability to design idempotent pipelines employing incremental, checkpoint, and replaceWhere strategies.

• Extensive background in production-grade Python engineering, including type hints, pytest, and ruff.

• Strong skills in diagnosing and resolving failing Spark / PySpark jobs using tools such as Spark UI.

• Comprehensive knowledge of Delta Lake features, including MERGE, OPTIMIZE, Z-ORDER, and time travel.

• Practical experience with dbt, covering models, tests, and exposures.

• Experience in authoring and deploying jobs using Databricks Asset Bundles (DAB) and functioning within a Unity Catalog environment.

• Dedication to ensuring data quality through pre-write asserts, schema checks, and maintaining dbt relationship and uniqueness tests.

• Strong commitment to disciplined Git workflows, conventional commits, and meticulous documentation practices.

• Experience provisioning and using Service Principals, GitHub environment secrets, and secret management tools like Azure Key Vault or Databricks secret scopes.

• Excellent written technical communication abilities for PR descriptions and runbooks, with a knack for translating pipeline work into business metrics.

• Proven capability to make decisions in ambiguous situations and balance trade-offs between cost, latency, and reliability.

• Preferred experience in leading technical initiatives, establishing architectural standards, and contributing to interview rubrics.

• Preferred experience in reading or modifying Azure Data Factory (ADF) pipelines and familiarity with Azure Data Lake storage.

• Familiarity with dbt observability tools, such as Elementary, is an advantage.

• Awareness of best practices for PII detection and masking is preferred.

• Strong plus for experience with multi-tenant configuration patterns to onboard new tenants without code changes.

• Preferred proficiency in reading and editing GitHub Actions workflows for Databricks deployment.

• Advantageous ability to make cost-conscious compute decisions, selecting the appropriate cluster shape for each workload.

• Preferred proficiency in AI-assisted development tools like Claude Code for daily tasks and code reviews.

• Valuable experience in writing incident post-mortems and coordinating feature handovers with Data Science teams is a plus.

🏝️ Benefits

• 100% Remote Work: Experience the flexibility of working from anywhere that helps you thrive. All you need is a laptop and a reliable internet connection.

• Highly Competitive USD Pay: Receive an exceptional, market-leading salary in USD that surpasses typical market offerings.

• Paid Time Off: We prioritize your well-being. Our paid time off policies ensure you have the opportunity to relax and recharge when necessary.

• Work with Autonomy: Enjoy the freedom to manage your time effectively as long as the work is completed. Focus on results rather than the clock.

• Collaborate with Top American Companies: Enhance your expertise by working on innovative, high-impact projects with industry-leading U.S. companies.