Senior DevOps / DataOps Engineer – Databricks Platform

This is a fully remote position, open to applicants in Brazil.

📋 Description

• Design, implement, and enhance the CI/CD pipeline for the Databricks platform, facilitating the automatic promotion of artifacts through development, staging, and production environments.

• Standardize the packaging and deployment of workloads utilizing Databricks Asset Bundles, creating reusable templates tailored for various data domains and teams.

• Implement and manage Infrastructure as Code (IaC) with Terraform and the Databricks provider for workspaces, compute policies, service principals, secret scopes, and Unity Catalog objects.

• Define and refine strategies for versioning, branching, code review, and release management utilizing Bitbucket, GitHub, or GitLab.

• Construct and sustain pipelines as code, encompassing deployment, versioning, rollback, and promotion across different environments.

• Establish automated quality gates through unit and integration tests, data contract validation, static analysis, linting, and minimum coverage metrics.

• Automate governance and security controls, including permissions via Unity Catalog, RBAC, cluster and cost policies, environment segregation, secrets management, and auditing processes.

• Advocate for software engineering best practices across platform, migration, and analytics engineering teams.

• Contribute to architectural decisions, technical standards, documentation, and provide technical mentoring.

• Assist in monitoring and troubleshooting pipelines, workloads, and infrastructure using Datadog or comparable tools.

⛳️ Requirements

• Proven experience in DevOps, DataOps, or related areas, with a successful history of building and enhancing comprehensive CI/CD pipelines.

• Familiarity with CI/CD as code using Bitbucket Pipelines, GitHub Actions, or GitLab CI.

• Strong understanding of Git, including Trunk-Based Development, GitFlow, or similar methodologies; proficient in pull requests, code reviews, branch protection, and release versioning.

• Experience with Databricks Asset Bundles, Databricks CLI, and the packaging and deployment of jobs, workflows, and configurations.

• Advanced proficiency in Python, including project structuring and modularization, library development, wheel packaging, and artifact publishing.

• Experience with automated testing, encompassing unit and integration testing using Pytest or similar tools, including mocks, fixtures, and coverage metrics.

• Experience with Terraform, especially in automating and managing resources via the Databricks provider.

• Background in cloud environments, preferably AWS or Azure, with a solid understanding of identity, storage, networking, and managed services.

• Strong SQL skills for validating, analyzing, and troubleshooting data pipelines.

• Nice to have: experience in migrating legacy Data Warehouses to Lakehouse architectures.

• Nice to have: knowledge of Databricks platform administration and support, encompassing workspaces, clusters, jobs, workflows, and compute policies.

• Nice to have: experience with Unity Catalog, data governance, RBAC, secrets management, environment segregation, and automated audit controls.

• Nice to have: experience with observability and monitoring using Datadog or equivalent tools.

• Nice to have: background in data platforms, Data Engineering, Data Warehousing, and the modernization of enterprise environments.

🏝️ Benefits

• Competitive salary and performance-based bonuses.

• Comprehensive health and wellness benefits.

• Opportunities for professional development and continuous learning.

• Flexible working hours and remote work options.

• Collaborative and innovative work environment.