Senior Data Engineer

This is a fully remote position, open to applicants in Brazil.

📋 Description

• Design, develop, and manage ingestion pipelines from advertising and business platforms utilizing REST APIs, Delta Sharing, object storage, and event streams.

• Model curated data layers in Delta Lake under Unity Catalog applying medallion architecture, incremental and CDC patterns, data contracts, lineage, and row/column-level governance.

• Construct the AI serving layer using SQL views, Unity Catalog functions, and low-latency stores accessed by agents via MCP behind Azure API Management.

• Take ownership of data quality and health monitoring as code, covering aspects such as freshness, volume, schema drift, and business-rule expectations.

• Set up alerting in Grafana and define clear triage processes.

• Deploy Databricks and Azure resources, jobs, permissions, and manage environment promotions through Terraform and CI/CD.

• Manage performance and cost through warehouse and cluster sizing, partitioning, liquid clustering, and orchestration with Databricks Workflows/Lakeflow.

• Collaborate daily with AI coding agents by drafting precise specifications and reviewing their generated outputs.

• Work in partnership with AI engineers, software engineers, and product teams to transform requirements into data products.

• Implement systems with OpenTelemetry and structured logging.

• Enhance production reliability and reduce latency.

• Document and maintain the codebase while adhering to engineering best practices.

⛳️ Requirements

• Strong foundation in software engineering principles, particularly in Python and advanced SQL.

• Extensive hands-on experience with Databricks, including Spark/PySpark, Delta Lake, Unity Catalog, Workflows, and SQL warehouses.

• Proven skills in performance tuning real Databricks workloads.

• Demonstrated experience in designing production data pipelines and data models.

• Familiarity with medallion or similar layered architectures, incremental processing, CDC, and dimensional modeling.

• Experience in integrating third-party APIs at scale, including aspects like authentication, pagination, throttling, partial failure handling, backfills, idempotency, and schema evolution.

• Knowledge of Azure services, including ADLS, Container Apps, Key Vault, API Management, and Entra ID.

• Proficiency with Terraform and CI/CD pipelines.

• Strong Git skills and code-review practices.

• Experience with data quality tests, expectations, monitoring, and upstream issue detection.

• Practical experience utilizing AI coding assistants/agents in engineering tasks.

• Excellent analytical and problem-solving abilities.

• Bachelor's degree or higher in Computer Science, Information Technology, or a related field, or equivalent practical experience.

🏝️ Benefits

• Equal opportunity employer committed to diversity, inclusion, and belonging.