Senior Data Engineer – Remote from Portugal, Selected EU Countries

This is a fully remote position, open to applicants in Portugal.

📋 Description

• Serve as the technical expert for the Client’s Databricks data platform.

• Design, construct, and optimize both batch and streaming data pipelines on Databricks utilizing PySpark, Spark SQL, Delta Lake, and Delta Live Tables.

• Advance the Lakehouse architecture (bronze, silver, and gold layers) to ingest and transform operational, behavioral, and HSE data from various clients and sources.

• Create and manage CI/CD pipelines for data workloads using Databricks Asset Bundles, Git, Azure DevOps, or GitHub Actions.

• Automate data testing, quality assurance, and promotion across development, staging, and production environments.

• Implement and oversee Unity Catalog, including catalogs, permissions, lineage, auditing, and data access policies.

• Define and enforce data governance standards, security measures, data masking, and per-client data isolation to ensure compliance with LGPD and GDPR.

• Monitor pipeline performance, establish SLAs, configure alerts, and lead incident reviews.

• Provision infrastructure as code using Terraform and standardize pipeline templates and best practices.

• Review code, mentor mid-level and junior engineers, and collaborate with product, data science, and consulting teams.

• Document architectural decisions, technical standards, and data engineering best practices.

⛳️ Requirements

• Minimum of 6+ years of experience in similar positions.

• At least 3+ years of experience with Databricks in production settings.

• Extensive hands-on experience with Databricks in production, including PySpark, Spark SQL, Delta Lake, and Delta Live Tables / Lakeflow.

• Solid background in designing and optimizing batch and streaming pipelines and evolving Lakehouse architectures (bronze, silver, gold layers).

• Demonstrated experience in implementing CI/CD for data workloads utilizing Git, Azure DevOps or GitHub Actions, and Databricks Asset Bundles.

• Practical DataOps experience, including automated testing, pipeline versioning, observability, and monitoring.

• Hands-on experience with data governance, preferably using Unity Catalog (access control, lineage, and auditing).

• Experience with Infrastructure as Code using Terraform.

• Strong proficiency in Python, PySpark, and advanced SQL.

• Cloud experience, ideally on Azure (ADLS, Data Factory, Key Vault, Entra ID).

• Familiarity with agentic coding.

• Fluent in English; minimum B2+ level (professional working proficiency).

• Databricks Data Engineer Professional certification or Azure certifications (DP-203 / DP-700) is a plus.

• Experience with multi-tenant SaaS platforms and per-client data isolation is advantageous.

• Knowledge of MLOps practices (MLflow, Feature Store, Model Serving) and building data pipelines for generative AI / RAG is appreciated.

• Experience with Microsoft Purview is valued.

• Familiarity with data quality tools (Great Expectations, Lakehouse Monitoring, dbt) is a plus.

• Experience with streaming technologies (Kafka, Event Hubs, or Structured Streaming) is appreciated.

• Domain knowledge in HSE, industrial operations, or risk management data is advantageous.

• Ability to comprehend requirements and technical solutions.

• Strong communication abilities.

• Critical thinking skills and a commitment to continuous improvement.

• Strong focus on delivery and a commitment to meeting deadlines and goals.

• Must currently and legally reside with a work permit in Portugal, Bulgaria, Croatia, Estonia, Hungary, Latvia, Lithuania, Slovakia, or Romania.

• No visa sponsorship or relocation assistance is available.

🏝️ Benefits

• Fully remote position with occasional, pre-planned visits to Client or FCamara locations in Portugal or Central Europe; regular office visits are not expected.

• No visa sponsorship or relocation support provided.

• Long-term project opportunity.

• Contract model options: employment contract, independent contractor, or B2B.