Senior Data Engineer, Databricks

This is a fully remote position, open to applicants in United States.

📋 Description

• Design and create production data pipelines utilizing Lakeflow Declarative Pipelines, Autoloader, and Structured Streaming.

• Take ownership of ingestion, transformation, data quality expectations, and CI/CD deployment through Declarative Automation Bundles.

• Architect and implement Databricks Lakehouse solutions using medallion architecture, Delta Lake, and Unity Catalog.

• Develop and maintain DLT pipelines, PySpark notebooks, and dbt transformation layers with data quality constraints and SLAs.

• Design and uphold data and AI foundations utilizing Unity Catalog, Feature Store, MLflow, and Model Serving.

• Collaborate with product and backend engineers on data models, APIs, and application data contracts.

• Consult with clients to identify data challenges, formulate data strategies, and implement sustainable solutions.

• Lead discussions on client data architecture or provide specialized data expertise to internal teams.

• Operate in multi-cloud environments including AWS and Azure.

• Advocate for data governance, encompassing access control, lineage, data quality policies, and compliance.

• Design application architectures that connect data to services backed by Lakebase and Databricks Apps.

• Contribute to Livefront's Databricks practice, accelerators, internal enablement, certification objectives, and partner go-to-market resources.

• Engage in a hiring process that might involve an initial phone interview, video interviews, and a take-home assignment.

⛳️ Requirements

• 7-10 years of experience in data engineering, including a minimum of 5 years in production environments with Databricks.

• Preferably possess experience in consulting or client delivery.

• Familiarity with AWS and Azure services pertinent to Databricks deployments, including storage, networking, IAM, and computing.

• Extensive production-grade expertise in Lakeflow Declarative Pipelines, Autoloader, Structured Streaming, Lakeflow Jobs, and Unity Catalog.

• Experience in designing production-scale Lakehouse architectures, incorporating medallion patterns, Delta Lake table design, partitioning, Z-ordering, and query optimization.

• Proficient in data pipeline testing, observability, and CI/CD practices, including unit testing, data quality frameworks, Git, and Declarative Automation Bundles.

• Strong skills in SQL and Python.

• Understanding of data modeling, schema design, and query optimization techniques.

• Capability to articulate complex data concepts to both technical and non-technical stakeholders.

• Ability to handle ambiguous requirements and deliver practical solutions.

• Demonstrated discipline and personal organization.

• Comfort with receiving critique and participating in peer reviews.

• A strong desire for personal and professional growth.

• A genuine passion for data engineering.

• A brief note about yourself, a summary of your work experience, and links to actively maintained public profiles when applying.

🏝️ Benefits

• Opportunity to collaborate with passionate and talented individuals.

• A respectful, mutually trusting, and egoless collaborative environment.

• Work on impactful products and accounts with significant reach.

• Team reputation for excellence and community engagement through education, mentoring, and sponsorship.

• Opportunity to contribute to the establishment of the Databricks practice from the ground up.

• Quick hiring process featuring video interviews and a take-home exercise lasting up to one week.