Senior Data Developer – Databricks

This is a fully remote position, open to applicants in Brazil.

📋 Description

• We are in search of a Senior Data Developer (Databricks) to become a vital member of our team, taking charge of a crucial data platform that underpins business reporting and analytics for our client.

• This role offers the chance to delve deeply into the Databricks ecosystem — from initial data ingestion to a fully modeled dimensional layer — while serving as the technical foundation on which the rest of the team depends for platform expertise.

• The position combines hands-on execution with technical leadership: you will manage and enhance notebooks throughout the complete medallion architecture, design and expand dimensional modeling artifacts that drive downstream business intelligence, and act as the primary reviewer and technical go-to for the team.

• You will function both strategically — by suggesting architectural and process enhancements — and operationally — by troubleshooting production jobs, deploying across environments, and ensuring that documentation remains up to date.

• Delivery & Continuous Improvement: Manage assigned tickets and bugs while continuously seeking improvement opportunities beyond the immediate task — proactively identifying technical debt, refactoring possibilities, and automation gaps instead of limiting contributions to what is assigned.

• Data Pipeline Ownership: Manage and enhance notebooks across the Stage, Bronze, Silver, and Gold layers, from data ingestion through the dimensional model utilized by business intelligence reporting tools.

• Dimensional Modeling: Design and implement dimensional modeling artifacts — including facts, dimensions, and slowly changing dimensions — with a clear grasp of how they enable downstream reporting.

• Workflow Management: Modify data processing jobs and workflows as necessary to accommodate evolving business needs.

• Code Review & Quality: Evaluate pull requests from other developers, ensuring code quality, performance, and architectural consistency across the codebase.

• Environment & Deployment Management: Deploy and promote changes across environments (Dev, QA, UAT, PROD), keeping deployment tracking current, and support environment operations such as restoring environments or tables from another environment or from a specific point in time.

• Production Monitoring & Troubleshooting: Monitor and troubleshoot daily production jobs, investigating failures and performance issues using platform-native diagnostic tools, job logs, and table history.

• Testing & Automation: Maintain and enhance the automated testing pipeline, including CI/CD workflows and the underlying test framework.

• Documentation: Keep technical documentation updated to ensure that institutional knowledge is preserved as the pipelines and connections relied upon by the team evolve.

• Technical Reference & Mentoring: Serve as the primary technical reference for the team on the data platform — the person others consult when they require deep platform expertise — and assist fellow team members with data modeling and development topics.

• Stakeholder Collaboration: Suggest and recommend architectural and process enhancements, working closely with the client's business and technical stakeholders — including the client's data architecture function — to translate requirements into scalable, well-tested data pipelines, while remaining equally comfortable taking direction from client-side technical leadership.

⛳️ Requirements

• Extensive experience in data development, with proven hands-on production experience on the Databricks platform.

• Strong proficiency in PySpark (DataFrame API, Spark SQL, UDFs, window functions) and Databricks SQL (ANSI SQL, MERGE INTO, COPY INTO, CTEs), including performance tuning techniques such as partition pruning, file compaction, skew handling, and query optimization.

• Solid, practical experience with Delta Lake: MERGE/upsert patterns, ACID transactions, time travel, and table maintenance (OPTIMIZE, VACUUM, ZORDER, liquid clustering, Change Data Feed).

• Demonstrated experience implementing Slowly Changing Dimensions (Type 1 and Type 2) and dimensional modeling principles (star schema, fact/dimension design) — not requiring you to have designed a model from scratch, but requiring the mindset to understand and extend one.

• Experience with medallion (or similar layered) architecture in a production data platform, along with Unity Catalog, jobs/workflows, secrets management, and notebook-based development.

• Experience with Git and Azure DevOps (or equivalent) for version control, pull requests, and CI/CD pipelines, in addition to familiarity with Microsoft Azure services (Key Vault, Service Principal/Managed Identity, Data Lake Storage).

• Ability to read and navigate a large, established codebase (400+ notebooks), learning and adhering to existing conventions rather than rewriting them, and to quickly adapt in a business-rule-heavy environment.

• Advanced English (C1 or above) communication skills, with the capability to work directly with US-based client stakeholders, propose technical recommendations, and align with decisions made by client-side technical leadership.

• Nice to Have

• Experience with Databricks Asset Bundles or other Infrastructure-as-Code methods for managing jobs, clusters, and permissions as code.

• Familiarity with Delta Live Tables and Databricks Genie (AI/BI Genie) for natural-language querying and conversational analytics.

• Knowledge of data quality frameworks (e.g., Great Expectations, Soda Core, or custom validation frameworks).

• Experience with pytest and databricks-connect for automated testing of Spark pipelines outside of manual notebook execution.

• Familiarity with Pydantic or similar typed-configuration methods, and experience with schema migration/versioning strategies (e.g., Flyway, Liquibase, or custom frameworks).

• Comfort in using AI-assisted development tools (e.g., GitHub Copilot, Cursor, or similar) to expedite coding, debugging, and code review processes.

🏝️ Benefits

• Health and dental insurance.

• Meal and food allowance.

• Childcare assistance.

• Extended paternity leave.

• Partnerships with gyms and health and wellness professionals via Wellhub (Gympass) TotalPass.

• Profit Sharing and Results Participation (PLR).

• Life insurance.

• Continuous learning platform (CI&T University).

• Discount club.

• Free online platform dedicated to physical, mental, and overall well-being.

• Pregnancy and responsible parenting course.

• Partnerships with online learning platforms.

• Language learning platform.

• And many more!