Data Architect, AI

This is a fully remote position, open to applicants in Mexico.

📋 Description

• Create comprehensive data architectures utilizing the Databricks Lakehouse Platform.

• Develop dependable, production-ready data pipelines across Bronze, Silver, and Gold layers.

• Diagnose and optimize large-scale Apache Spark jobs.

• Formulate governance frameworks for data and AI resources.

• Implement access controls based on roles, rows, and columns, along with data lineage tracking.

• Collaborate with AWS, Azure, or GCP cloud infrastructures.

• Enhance the performance and cost-efficiency of Databricks clusters, including DBU usage.

• Potentially construct, deploy, and oversee GenAI applications and machine learning models.

• Automate Databricks workflows while supporting streaming data ingestion and downstream business intelligence integration.

⛳️ Requirements

• Proficiency in designing comprehensive data architectures that incorporate Bronze, Silver, and Gold layers.

• Extensive knowledge of distributed computing and the inner workings of Databricks/Spark.

• Capability to troubleshoot and fine-tune large-scale Spark jobs by utilizing caching, partitioning, and broadcast joins.

• Familiarity with Delta Lake's ACID transactions, schema enforcement, time travel, and Z-ordering.

• Skill in designing cohesive governance frameworks for data and AI assets, including role-based access control (RBAC), row-level security, column-level security, and data lineage.

• Strong understanding of the native cloud ecosystems of AWS, Azure, or GCP, encompassing VNet configurations and IAM roles.

• Advanced SQL proficiency.

• Fluency in Python or Scala programming languages.

• Ability to monitor DBU usage, appropriately size serverless and multi-node clusters, and optimize cloud expenditures.

• Proficient in English.

• Databricks certifications, along with experience in GenAI/MLflow, CI/CD, DevOps, streaming data, and business intelligence are advantageous but not mandatory.

🏝️ Benefits

• Competitive salary and performance-based incentives.

• Opportunities for professional development and training.

• Flexible working environment with options for remote work.

• Comprehensive health and wellness benefits.

• Collaborative and innovative workplace culture.