Data Engineer, Databricks

This is a fully remote position, open to applicants in Peru.

📋 Description

• Design and construct production data pipelines utilizing Lakeflow Declarative Pipelines, Autoloader, and Structured Streaming, ensuring complete ownership of ingestion, transformation, data quality standards, and CI/CD deployment through Declarative Automation Bundles.

• Architect and develop Lakehouse solutions on Databricks — including medallion architecture, Delta Lake, and Unity Catalog — specifically tailored to meet the analytics, AI, and application requirements of clients.

• Create and sustain Databricks transformation layers — such as DLT pipelines, PySpark notebooks, and dbt — incorporating established data quality constraints and SLAs.

• Design and uphold the foundational data and AI components — including Unity Catalog, Feature Store, MLflow, and Model Serving — that support production machine learning, agent workflows, and AI-driven digital products.

• Collaborate with product and backend engineers to develop data models, APIs, and application data contracts, ensuring that the platform serves the product effectively rather than merely functioning as a warehouse.

• Engage with clients to identify their data-related challenges, formulate data strategies, and implement enduring solutions.

• Modify your approach according to project requirements — at times leading discussions on data architecture with clients, and at other times providing specialized data expertise to internal teams.

• Operate within multi-cloud environments — primarily AWS and Azure — anchoring data platform recommendations on Databricks where it aligns with the client's architecture and objectives.

• Promote data governance through Unity Catalog — focusing on access control, lineage, data quality policies, and compliance — as an integral component of every engagement rather than an afterthought.

• Design data-to-application architectures — encompassing Lakebase-backed services and Databricks Apps — that link governed data to AI workflows, digital products, and user-facing experiences.

• Contribute to the growth of Livefront's Databricks practice — participating in the development of accelerators, internal training, certification initiatives, and Databricks partner go-to-market materials in addition to delivery responsibilities.

⛳️ Requirements

• 3-5 years of experience in data engineering, with a minimum of 2 years in production Databricks environments, ideally in a consulting or client-facing role.

• Strong working knowledge of AWS and Azure cloud services relevant to Databricks deployments — including storage, networking, IAM, and compute — with familiarity in GCP being an advantage.

• Extensive, production-grade expertise in Databricks: Lakeflow Declarative Pipelines, Autoloader, Structured Streaming, Lakeflow Jobs, and Unity Catalog (including fine-grained access control and lineage) — proven through delivered production workloads, not just prototypes.

• Demonstrated experience in designing Lakehouse architectures — covering medallion patterns, Delta Lake table design, partitioning, Z-ordering, and query optimization — at a production scale.

• Practical experience with data pipeline testing, observability, and CI/CD for data — including unit testing, data quality frameworks, and version-controlled deployments using Git and Declarative Automation Bundles.

• High proficiency in SQL and Python, capable of writing clean, efficient, and maintainable code.

• Understanding of data modeling, schema design, and query optimization.

• Excellent communication skills, able to articulate complex data concepts to both technical and non-technical audiences.

• Strong problem-solving capabilities, adept at navigating ambiguous requirements and delivering practical solutions.

• Above-average discipline and organizational skills.

• Clear comfort with critique and peer review within an iterative development context.

• A demonstrated eagerness for personal and professional development.

• A clear passion for and dedication to the craft of data engineering.

🏝️ Benefits

• Work alongside passionate and skilled individuals who are constantly seeking ways to enhance processes.

• Enjoy a workplace where respect, mutual trust, and collaborative spirit are fundamental.

• Collaborate with colleagues who take their responsibilities seriously, yet maintain a sense of humor and know how to relax and have fun.

• Be part of a team renowned for excellence that actively gives back to the community through education, mentorship, and sponsorship.

• Contribute to products and accounts that have a significant impact and reach.

• Share a commitment to detail, quality, and pride in delivering exceptional results.

• Play a key role in establishing a data practice specialization from the ground up — influencing our market approach with Databricks, developing accelerators, and shaping the essence of this work within a digital product company.