Senior/Lead Data Engineer – AI-Native Aftermarket Platform

This is a fully remote position, open to applicants in Dominican Republic.

📋 Description

• Create and develop resilient, idempotent data pipelines from the ground up using a modern data stack.

• Design star and snowflake schemas, crafting precise and grain-aware SQL to build scalable data marts.

• Produce high-quality, unit-tested Python code at the module level, adhering to robust engineering practices such as type hinting and testing.

• Construct and validate dbt models across staging, intermediate, and mart layers while overseeing the overall project structure.

• Author and deploy jobs utilizing Databricks Asset Bundles (DAB) in accordance with established architectural patterns.

• Implement stringent data quality checks at source, intermediate, and destination layers to avoid unnoticed nulls or duplicates.

• Uphold data governance through thorough dbt tests and rigorous documentation-at-merge-time practices.

• Operate securely in a multi-repository architecture, employing service principals and ensuring no personal credentials are used in production deployments.

• Conduct cross-repository exposure checks prior to merging any schema-breaking changes.

• Manage data pipelines from start to finish, making crucial technical design choices and mentoring mid-level engineers through in-depth code reviews.

• Establish the overall technical direction across core data systems, including modeling standards, branching strategies, observability thresholds, and secret management protocols.

• Serve as a technical leader to facilitate team progress and actively engage in hiring panels to expand the engineering team.

⛳️ Requirements

• Proficiency in SQL and dimensional modeling techniques, including medallion architecture, SCDs, and grain management.

• Demonstrated ability to design idempotent pipelines using incremental, checkpoint, and replaceWhere strategies.

• Extensive experience in production-grade Python engineering, encompassing type hints, pytest, and ruff.

• Strong problem-solving skills to diagnose and rectify failing Spark / PySpark jobs using tools like Spark UI.

• In-depth knowledge of Delta Lake functionalities such as MERGE, OPTIMIZE, Z-ORDER, and time travel.

• Practical experience with dbt, including models, tests, and exposures.

• Experience in authoring and deploying jobs with Databricks Asset Bundles (DAB) and functioning within a Unity Catalog environment.

• Dedication to data quality through pre-write asserts, schema validations, and maintaining dbt relationship and uniqueness tests.

• Strong commitment to disciplined Git workflows, conventional commits, and meticulous documentation practices.

• Familiarity with provisioning and utilizing Service Principals, GitHub environment secrets, and secret management tools like Azure Key Vault or Databricks secret scopes.

• Excellent written technical communication abilities for PR descriptions and runbooks, with the capacity to translate pipeline work into business metrics.

• Proven capability to make decisions amidst ambiguity and balance trade-offs between cost, latency, and reliability.

• Experience leading technical initiatives, establishing architectural standards, and contributing to interview rubrics is preferred.

• Familiarity with reading or modifying Azure Data Factory (ADF) pipelines and knowledge of Azure Data Lake storage is highly preferred.

• Knowledge of dbt observability tools, such as Elementary, is an advantage.

• Understanding of PII detection and masking best practices is preferred.

• Experience with multi-tenant configuration patterns to onboard new tenants without code changes is a significant plus.

• Proficiency in reading and editing GitHub Actions workflows for Databricks deployment is preferred.

• Ability to make cost-conscious compute decisions by selecting the appropriate cluster shape for each workload is a plus.

• Familiarity with AI-assisted development tools like Claude Code for daily tasks and code reviews is preferred.

• Experience in drafting incident post-mortems and coordinating feature handovers with Data Science teams is an advantage.

🏝️ Benefits

• 100% Remote Work: Enjoy the flexibility to work from wherever you thrive. All you need is a laptop and a stable internet connection.

• Highly Competitive USD Pay: Receive an exceptional, market-leading salary in USD that exceeds typical market offerings.

• Paid Time Off: We prioritize your well-being. Our paid time off policies allow you to relax and recharge as needed.

• Work with Autonomy: Enjoy the liberty to manage your time effectively as long as the work is completed. Focus on outcomes, not hours.

• Collaborate with Top American Companies: Enhance your expertise working on innovative, impactful projects with industry-leading U.S. companies.