Senior Data Engineer, 6-month contract

This is a fully remote position, open to applicants in Poland.

📋 Description

• Collaborate within Agile teams to fulfill the committed backlog according to acceptance criteria.

• Design, construct, and manage ingestion and orchestration pipelines utilizing Azure Data Factory.

• Create ETL/ELT processes in Azure Databricks employing a medallion architecture for analytics, reporting, and AI.

• Lead code reviews and establish engineering standards.

• Engage in estimating, planning, testing, deployment, and documentation activities.

• Influence data architecture choices, such as metric definitions, semantic models, and data-serving AI agents.

• Implement data governance, security measures, and access controls in line with client classification and residency standards.

• Utilize AI-assisted development tools judiciously to expedite delivery while maintaining quality and security.

• Update stakeholders, including management, on progress, outstanding items, and issues.

• Keep abreast of relevant technological advancements.

• Deliver resilient data platforms and AI-driven solutions on Microsoft Azure to support decision-making, autonomous execution, and integrated operations.

• Mentor team members and lead by example.

⛳️ Requirements

• Over 7 years of experience in data engineering or software development.

• More than 3 years of hands-on experience with Azure Databricks or Spark.

• Proven experience in actively mentoring junior team members.

• Proficient in Python, PySpark, and SQL at an expert level.

• Familiarity with Azure Data Factory, Azure Databricks, Delta Lake, Workflows, Unity Catalog, Azure Data Lake Storage, and Azure SQL.

• Experience in dimensional and lakehouse data modeling.

• Background in integrating data through APIs, OData, JDBC/ODBC, and files.

• Knowledge of Azure AI services, including Azure AI Foundry and Azure OpenAI, along with preparing data for AI agents.

• Experience with secure CI/CD and infrastructure-as-code using tools like Terraform, Bicep, or Databricks Asset Bundles.

• Competence across DEV, UAT, and PROD environments using Azure DevOps or GitHub, with a solid understanding of Git branching practices.

• Familiarity with AI-assisted development and reviewing generated code.

• Understanding of data quality, testing, security, and privacy in regulated or restricted-data situations.

• Experience in designing comprehensive data solutions and recommending frameworks and technologies.

• Background in Agile and Kanban methodologies within mixed-discipline teams.

• Self-motivated, organized, and capable of working with minimal supervision.

• Bachelor’s degree in computer science, engineering, or a related field, or equivalent education and experience.

• Preferred: experience integrating enterprise systems.

• Preferred: knowledge of data management and governance concepts, including data quality, cataloging, lineage, master data, and stewardship.

🏝️ Benefits

• Fully remote position.

• 6-month contract.