Data Manager, Databricks

This is a fully remote position, open to applicants in Texas.

📋 Description

• Define the overarching data platform architecture for Lakehouse/EDW environments, which includes reference patterns, technology choices, and integration blueprints.

• Create conceptual, logical, and physical data models tailored for multi-tenant and industry-specific data products.

• Standardize logical data layers, covering ingest/raw, staged/curated, and serving layers.

• Implement data governance, metadata management, cataloging, lineage tracking, data contracts, and classification standards.

• Establish security and compliance measures, involving access management, data masking, encryption, network segmentation, and auditing policies.

• Design for scalability, high availability, disaster recovery, capacity management, and cost efficiency.

• Lead the selection and integration of Databricks, Delta Lake, Delta Live Tables, Fivetran, Azure Data Factory/Data Fabric, orchestration tools, and monitoring solutions.

• Develop and enforce CI/CD frameworks for data artifacts, incorporating testing, automated deployments, and rollback techniques.

• Define patterns for batch and streaming data ingestion, compaction strategies, partitioning schemes, and storage layouts.

• Specify observability standards, including metrics, SLAs, dashboards, logging, tracing, and alerting mechanisms.

• Act as a technical authority and mentor for Data Engineering teams.

• Conduct architecture and code reviews for key components.

• Collaborate with Data Product Owners, Security, Infrastructure, BI, and ML stakeholders to convert requirements into actionable solutions and roadmaps.

• Design, develop, test, and deploy processing modules utilizing Spark, PySpark/Scala, Spark SQL, and stored procedures.

• Construct and enhance data pipelines on Databricks and various database platforms.

• Implement DevOps practices, infrastructure-as-code, CI/CD pipelines, automated testing, and version control systems.

• Diagnose complex data quality, performance, and availability challenges, and drive continuous improvements.

⛳️ Requirements

• Prior experience in an architectural or lead technical role focusing on enterprise data platforms.

• Practical experience with Databricks technologies, including Delta Lake, Unity Catalog, Delta Live Tables, Auto Loader, and Structured Streaming.

• Strong proficiency in Spark, PySpark and/or Scala, Spark SQL, and optimization of distributed jobs.

• Solid understanding of data warehouse and lakehouse design principles.

• Familiarity with Medallion, Lambda, and Kappa architectural patterns.

• Experience integrating SaaS/ETL/connectors such as Fivetran.

• Knowledge of orchestration platforms such as Airflow, Azure Data Factory, and Data Fabric.

• Experience with relational and hybrid databases, including MS SQL Server, PostgreSQL, Oracle, Azure SQL, and AWS RDS/Aurora or their equivalents.

• Proficiency in implementing CI/CD for data pipelines using Azure DevOps, GitHub Actions, Jenkins, or comparable tools.

• Experience in packaging and deploying artifacts like .whl files and containers.

• Familiarity with batch and streaming processing, file compaction, partitioning strategies, and storage optimization.

• Understanding of cloud security measures, IAM/RBAC, encryption, VPC/VNet concepts, and cloud networking.

• Knowledge of observability and monitoring tools such as Prometheus, Grafana, Datadog, or native cloud monitoring solutions.

• Experience in automating CI/CD pipelines and trunk-based development is a plus.

• Advanced proficiency in PySpark is a plus.

• Advanced skills in Spark workflow optimization and orchestration are a plus.

• Databricks, cloud architect, or data certifications are advantageous.

• Ability to identify, troubleshoot, and resolve intricate data-related issues.

• Strong collaborative and communication skills.

• A curious mindset with a readiness to adopt new tools, technologies, and methodologies.

🏝️ Benefits

• Full-time employment.

• Equal opportunity and inclusive workplace environment.

• Commitment to diversity.