Senior Data Engineer, Databricks

This is a fully remote position, open to applicants in Portugal.

📋 Description

• Design, develop, and sustain scalable data pipelines utilizing Databricks, PySpark, Python, and SQL.

• Construct and enhance batch and near real-time data processing solutions within the Microsoft Azure ecosystem.

• Create and manage Delta Lake-based data architectures.

• Develop resilient and scalable data products and contemporary data platform components that support strategic business initiatives.

• Contribute to the implementation and automation of CI/CD for data solutions.

• Apply engineering best practices regarding source control, testing, deployment, and monitoring.

• Assist in Infrastructure-as-Code and environment standardization projects.

• Guarantee high standards of data quality, observability, monitoring, security, and operational reliability.

• Collaborate with Product Managers, Architects, Data Engineers, and business stakeholders to achieve high-quality results.

• Engage in technical design discussions and contribute to decisions regarding solution architecture.

• Support platform governance, cost optimization, technical documentation, and knowledge-sharing initiatives.

⛳️ Requirements

• Over 5 years of experience in Data Engineering.

• More than 3 years of hands-on experience with Databricks.

• Demonstrated experience in building and maintaining modern cloud data platforms.

• Experience in Agile delivery environments.

• Proven capability to work effectively within distributed and international teams.

• Professional proficiency in English.

• Strong practical expertise with: Databricks, PySpark, Delta Lake, Databricks SQL, workflow orchestration, and performance tuning.

• Strong proficiency in: Python and SQL.

• Experience with: Microsoft Azure Data Platform services, Azure DevOps, Git-based development, CI/CD implementation, data modeling, and data warehousing concepts.

• Good understanding of: data governance, security and access management, and monitoring and observability.

• Ability to design and deliver scalable, secure, and cost-efficient data solutions.

🏝️ Benefits

• Meal allowance: €10.20/day

• Flexible benefits plan

• Private medical insurance

• 22 days of annual leave, increasing every 3 years (up to 25 days)

• Continuous learning via KLX – Keyrus Learning Experience

• A collaborative, international, and human-centered work environment