Senior Data & AI Engineer

📋 Description

• Design and execute medallion architecture (bronze/silver/gold) utilizing Delta Lake as the core layer across client environments.

• Construct and enhance scalable data pipelines employing Apache Spark and Lakeflow (Declarative Pipelines, Jobs, and Connect for ingestion).

• Architect and deploy Unity Catalog as the framework that manages data lineage, fine-grained access control (including row/column-level security), Lakehouse Federation, and Delta Sharing.

• Advocate best practices for performance tuning, cost governance, and compute optimization on Databricks.

• Implement and refine Databricks SQL for enterprise analytics, encompassing AI/BI Dashboards and integration with BI tools (Tableau, Power BI, Looker).

• Develop data models that cater to both operational reporting and strategic analytics use cases.

• Collaborate with business stakeholders to convert complex data requirements into dependable, high-performance solutions.

• Create production AI solutions on the Databricks platform utilizing tools such as RAG pipelines with Mosaic Vector Search, agent development with the Mosaic AI Agent Framework, Model Serving endpoints, and external model APIs (OpenAI, Anthropic, Gemini, and more).

• Leverage Mosaic AI, AI/BI Genie, and Databricks Apps to craft intelligent, data-driven applications that provide tangible client value.

• Stay updated on Databricks AI capabilities and proactively introduce emerging patterns to the practice.

• Act as a foundational technical resource for V2's Databricks practice, influencing standards, methodologies, and delivery frameworks.

• Engage hands-on across client projects as a practitioner and technical advisor, translating business challenges into platform solutions.

• Assist in pre-sales and solution efforts: scoping engagements, contributing to proposals, and establishing credibility in client discussions.

• Elevate the wider technical team by leading code reviews, conducting both formal and informal knowledge transfers, and enhancing the skills of those around you.

⛳️ Requirements

• Mastery of the platform. Proven and demonstrable expertise in Databricks, including proficiency in Delta Lake, Unity Catalog, DLT, Databricks SQL, and Spark. Databricks Certifications (particularly Data Engineer Professional and ML Engineer Professional) are highly preferred.

• Comprehensive engineering experience. You have developed pipelines, governed data platforms, and designed Lakehouse architectures in live production settings.

• Experience with AI integration. Practical experience in integrating LLMs, constructing RAG pipelines, and deploying AI-driven solutions using Databricks.

• Consulting or professional services aptitude. You comprehend the dynamics of consulting engagements and possess experience in scoping, client communication, delivering under pressure, and building trust with senior stakeholders.

• Extensive data platform knowledge. Familiarity with cloud platforms (Azure, AWS, or GCP) and adjacent data technologies, such as Snowflake, Salesforce Data Cloud, or dbt, is a plus.

• Cultural compatibility. You thrive in fast-paced, agile environments and enjoy entrepreneurial settings.

🏝️ Benefits

• Health insurance

• Retirement plans

• Paid time off

• Flexible work arrangements

• Professional development