Senior Data & AI Engineer

This is a fully remote position, open to applicants in United States.

📋 Description

• Design and execute medallion architecture (bronze/silver/gold) utilizing Delta Lake as the core layer across various client environments.

• Construct and enhance scalable data pipelines with Apache Spark and Lakeflow (Declarative Pipelines, Jobs, and Connect for ingestion).

• Architect and deploy Unity Catalog as the framework that manages data lineage, fine-grained access control (including row/column-level security), Lakehouse Federation, and Delta Sharing.

• Promote best practices for performance optimization, cost management, and compute efficiency on Databricks.

• Implement and refine Databricks SQL for enterprise analytics, encompassing AI/BI Dashboards and integration with BI tools (Tableau, Power BI, Looker).

• Develop data models that cater to both operational reporting and strategic analytical requirements.

• Collaborate with business stakeholders to convert complex data needs into dependable, high-performing solutions.

• Create production-level AI solutions on the Databricks platform utilizing tools such as RAG pipelines with Mosaic Vector Search, agent development via the Mosaic AI Agent Framework, Model Serving endpoints, and external model APIs (OpenAI, Anthropic, Gemini, and others).

• Utilize Mosaic AI, AI/BI Genie, and Databricks Apps to craft intelligent, data-driven applications that provide real value to clients.

• Keep abreast of Databricks AI capabilities and proactively introduce emerging patterns to the practice.

• Act as a key technical resource for V2's Databricks practice, contributing to the development of standards, methodologies, and delivery frameworks.

• Work across client engagements as a hands-on practitioner and technical advisor, translating business challenges into platform solutions.

• Assist in pre-sales and solutioning activities: scoping engagements, participating in proposals, and establishing credibility in client discussions.

• Enhance the broader technical team by leading code reviews, facilitating formal and informal knowledge transfer, and uplifting the capabilities of those around you.

⛳️ Requirements

• Mastery of the platform. Demonstrable depth in Databricks, with proficiency in Delta Lake, Unity Catalog, DLT, Databricks SQL, and Spark. Databricks Certifications (particularly Data Engineer Professional and ML Engineer Professional) are highly preferred.

• Comprehensive engineering experience. Proven track record of building pipelines, managing data platforms, and designing Lakehouse architectures within production settings.

• Experience in AI integration. Practical knowledge in integrating LLMs, constructing RAG pipelines, and deploying AI-driven solutions using Databricks.

• Consulting or professional services mindset. Familiarity with how consulting engagements operate, along with experience in scoping, client communication, delivering under pressure, and establishing trust with senior stakeholders.

• Wide-ranging data platform knowledge. Background in cloud platforms (Azure, AWS, or GCP) and awareness of related data technologies, such as Snowflake, Salesforce Data Cloud, or dbt, is advantageous.

• Cultural fit. Ability to operate with speed and agility, thriving in fast-paced, entrepreneurial environments.

🏝️ Benefits

• Health insurance

• Retirement plans

• Paid time off

• Flexible work arrangements

• Professional development