
Senior Data Engineer – Databricks
Posted 6 days ago

Posted 6 days ago
This is a fully remote position, open to applicants in Spain.
• Design and implement Lakehouse architectures on Databricks utilizing a Medallion architecture (Bronze/Silver/Gold).
• Develop batch and streaming ingestion and transformation pipelines using PySpark, Spark SQL, and Delta Lake.
• Implement incremental ingestion with Auto Loader and Lakeflow Connect, including Change Data Capture (CDC) and Slowly Changing Dimensions (SCD).
• Construct declarative pipelines using Lakeflow Spark Declarative Pipelines and orchestrate them with Lakeflow Jobs.
• Implement data governance with Unity Catalog, including permissions, lineage, catalogs, schemas, Lakehouse Federation, and Delta Sharing.
• Apply data quality processes, expectations, quality monitoring, and pipelines.
• Optimize the performance and costs of clusters, jobs, serverless environments, and SQL Warehouses.
• Integrate data from databases, files, APIs, and events in cloud environments.
• Participate in migrations from legacy platforms to Databricks.
• Apply best practices for CI/CD and infrastructure as code using Databricks Asset Bundles, Git, and Terraform.
• Support pre-sales processes through reference architectures, estimations, technical proposals, Proof of Concepts (PoCs), and client demonstrations.
• Mentor junior and semi-senior profiles.
• Document architectures and developments.
• More than 5 years of experience in data engineering.
• At least 2-3 years of experience in projects involving Databricks.
• Proficiency in PySpark, Spark SQL, and Delta Lake.
• Advanced knowledge of SQL and Python.
• Experience with Unity Catalog, Lakeflow Jobs, and Lakeflow Spark Declarative Pipelines or DLT.
• Experience with at least one cloud platform; Azure preferred, with experience in ADLS Gen2, Entra ID, Key Vault, and Data Factory; AWS or GCP is a plus.
• Competence in dimensional data modeling, both normalized and denormalized, along with DWH, DL, and DLH architectures.
• Experience with Git and CI/CD pipelines.
• Autonomy, proactivity, client communication skills, and a consultancy orientation.
• Ability to present and defend technical proposals in pre-sales.
• A passion for mentoring and an ability to adapt to various projects.
• Databricks Data Engineer Associate / Professional certifications are desirable.
• Experience with Snowflake, migrations to Databricks, dbt, Terraform, MLflow, Databricks SQL/AI/BI, Mosaic AI, or Vector Search is a plus.
• Permanent contract and stability from day one.
• Innovative projects using modern technology.
• Continuous training and professional development.
• Flexible budget for courses, certifications, and English classes.
• Communities and collaborative learning opportunities.
• 23 business days of vacation per year, plus an additional day for every two years of service, up to a maximum of 30 days.
• Remote work compensation of €562.50/year.
• Flexible compensation for training, meals, transportation, and childcare.
• Provision of PC, chair, monitor, keyboard, mouse, and footrest.
• A library of books and games available in the Granada office.
• Access to WellHub offering discounted sports activities.
• Discounts via IBenefits.
• Financial support for shared hobbies with colleagues.
• Surprise breakfast on your birthday.
• Celebration and support for the birth of children.
• Recognition for sporting challenges.
• Sports groups and cultural events.
• Monthly Civipregunta and Civicerves events covered by the company.
• Christmas lunch.
• Cívica Day for employees and their families.
Highmark Health
Prompt Therapy Solutions Inc
Jones Lang LaSalle Americas, Inc.
Allshore Talent
Get handpicked remote jobs straight to your inbox weekly.