Principal Engineer – Databricks Data Engineer

This is a fully remote position, open to applicants in Mexico.

📋 Description

• Conduct an audit of the existing Databricks workspace state, encompassing environments, clusters, Unity Catalog configuration, access controls, and cost management.

• Implement and strengthen Unity Catalog, focusing on namespace architecture, access controls, cluster policies, and environment segmentation.

• Establish workspace guidelines, naming conventions, access request processes, and onboarding procedures for users and teams.

• Develop data quality frameworks that include SLAs, certification criteria, and models for data ownership.

• Create automation templates for managing cluster lifecycles, cost alerts, and utilization monitoring.

• Design observability dashboards that track platform health, costs, job success rates, and data quality SLA KPIs.

• Oversee the monitoring of core data asset lineage across various workspaces.

• Produce operational runbooks for routine tasks that can be executed by junior or mid-level engineers.

• Facilitate knowledge-transfer sessions with platform engineering teams.

• Deliver final documentation that details architecture decisions, operational standards, outstanding items, and recommendations.

• Assess and enhance Databricks environments, workspaces, clusters, catalogs, permissions, and security setups.

• Establish and enforce governance standards for secure, scalable, and compliant platform operations.

• Spearhead the adoption of Unity Catalog and design data structures, access models, and strategies for environment segregation.

• Set up RBAC policies, workspace standards, naming conventions, and processes for user access.

• Champion data governance initiatives, including data quality frameworks, ownership models, SLAs, lineage, and stewardship.

• Create automation solutions and Infrastructure as Code frameworks.

• Implement practices for cost optimization, monitoring, and operational excellence.

• Design performance and observability dashboards that track platform health, utilization, job success rates, and data quality KPIs.

• Generate operational documentation, runbooks, and support procedures.

• Collaborate with security, compliance, infrastructure, and data engineering teams.

• Provide training, mentoring, and knowledge-transfer sessions.

• Suggest platform enhancements and long-term strategies for governance, scalability, and organizational maturity.

⛳️ Requirements

• Essential skills include: Azure Data Factory, Databricks, Unity Catalog.

• Preferred skills include: Data platform implementation, Google Workspace Administration.

🏝️ Benefits

• Opportunity for remote work.

• Full-time employment.