Senior/Lead Data Engineer – AI-Native Aftermarket Platform

This is a fully remote position, open to applicants in Brazil.

📋 Description

• Develop and construct robust, idempotent data pipelines from the ground up using a contemporary data stack.

• Create star and snowflake schemas, writing precise, grain-aware SQL to build scalable data marts.

• Produce production-grade, unit-tested Python code at the module level, adhering to strong engineering principles such as type hinting and testing.

• Construct and validate dbt models across staging, intermediate, and mart layers while overseeing the overall project structure.

• Author and deploy jobs utilizing Databricks Asset Bundles (DAB) in accordance with documented architectural standards.

• Enforce rigorous data quality checks at the source, intermediate, and destination layers to prevent unnoticed nulls or duplicates.

• Uphold data governance through comprehensive dbt tests and strict documentation-at-merge-time practices.

• Operate securely within a multi-repository architecture, employing service principals and ensuring no personal credentials are used in production deployments.

• Execute cross-repository exposure checks before merging schema-altering changes.

• Own data pipelines end-to-end, making significant technical design choices and providing mentorship to mid-level engineers through in-depth code reviews.

• Define the overall technical strategy across core data systems, encompassing modeling standards, branching strategies, observability thresholds, and secret management policies.

• Serve as a technical leader to facilitate team progress and actively engage in hiring panels to enhance the engineering organization.

⛳️ Requirements

• Proficiency in SQL and dimensional modeling strategies, including medallion architecture, SCDs, and grain management.

• Demonstrated ability to design idempotent pipelines using incremental, checkpoint, and replaceWhere strategies.

• Significant experience in production-grade Python engineering, including type hints, pytest, and ruff.

• Strong aptitude for diagnosing and resolving failing Spark / PySpark jobs using tools like Spark UI.

• Comprehensive knowledge of Delta Lake features such as MERGE, OPTIMIZE, Z-ORDER, and time travel.

• Practical expertise with dbt, covering models, tests, and exposures.

• Experience in authoring and deploying jobs with Databricks Asset Bundles (DAB) and functioning within a Unity Catalog environment.

• Commitment to maintaining data quality through pre-write asserts, schema checks, and dbt relationship and uniqueness tests.

• Strong adherence to disciplined Git workflows, conventional commits, and rigorous documentation practices.

• Familiarity with provisioning and utilizing Service Principals, GitHub environment secrets, and secret management tools like Azure Key Vault or Databricks secret scopes.

• Excellent written technical communication skills for PR descriptions and runbooks, with the capability to translate pipeline work into business metrics.

• Proven decision-making skills to navigate ambiguity and balance trade-offs among cost, latency, and reliability.

• Preferred experience in leading technical initiatives, establishing architectural standards, and contributing to interview rubrics.

• Preferred familiarity with reading or modifying Azure Data Factory (ADF) pipelines and Azure Data Lake storage.

• Familiarity with dbt observability tools, such as Elementary, is a plus.

• Awareness of best practices for PII detection and masking is preferred.

• Experience with multi-tenant configuration patterns to onboard new tenants without code changes is a strong advantage.

• Preferred ability to read and edit GitHub Actions workflows for Databricks deployment.

• Capability to make cost-effective compute decisions by selecting the appropriate cluster shape for each workload is a plus.

• Preferred proficiency in AI-assisted development tools like Claude Code for daily tasks and code reviews.

• Experience in writing incident post-mortems and coordinating feature handovers with Data Science teams is a plus.

🏝️ Benefits

• 100% Remote Work: Enjoy the flexibility of working from any location that suits you best. All you need is a laptop and a reliable internet connection.

• Highly Competitive USD Pay: Receive outstanding, market-leading compensation in USD that surpasses typical market offerings.

• Paid Time Off: We prioritize your well-being. Our paid time off policies ensure you can take time to relax and rejuvenate when necessary.

• Work with Autonomy: Benefit from the freedom to manage your time as long as you meet your work commitments. Focus on results rather than the clock.

• Work with Top American Companies: Expand your expertise by engaging in innovative, high-impact projects with industry-leading U.S. companies.