Senior Data Platform Engineer

This is a fully remote position, open to applicants in Poland.

📋 Description

• Take charge of the Apache Airflow platform, overseeing environments, deployment, upgrades, configuration, connections, variables, pools, and standards for DAGs.

• Transition orchestration from Azure Data Factory to Airflow while advancing the Metadata-Driven Orchestration Framework.

• Guarantee the reliability and observability of Airflow through SLAs, alerting, retries, backfills, health monitoring, and ownership of failures.

• Manage Azure resources utilizing Terraform, which includes Databricks workspaces, Data Lake Storage, Data Factory, Key Vault, networking, and ancillary services.

• Oversee secrets, identity, and access management through Key Vault, service principals, managed identities, Azure RBAC, Entra ID, and Unity Catalog.

• Manage Azure DevOps repositories, CI/CD pipelines, branch policies, multi-environment promotions, deployment validation, and Terraform state management.

• Monitor Azure resource costs and governance through tagging, budgeting, right-sizing, and cost attribution.

• Contribute to the Data Catalog, Data Quality Framework, and overarching platform lineage.

• Enhance observability for debuggable, alert-driven pipelines with defined failure ownership.

• Operate Databricks and Unity Catalog, which includes cluster and job policies, Auto Loader ingestion, as well as workload performance and cost optimization.

• Engage in architecture reviews and contribute to technical decision-making processes.

• Establish and promote engineering standards via code reviews and design assessments.

• Collaborate with Architects, Product Owners, and data engineers to advance the technical maturity of data products.

• Develop and sustain developer tools and automation, incorporating AI-assisted engineering workflows.

• Provide second-line support to the Run/Operations team.

• Diagnose platform incidents, lead root cause analyses, and conduct post-mortems.

• Ensure proper handover, documentation, and operational readiness for platform modifications.

• Enhance runbooks, monitoring protocols, and operational processes.

⛳️ Requirements

• Over 6 years of experience in data engineering, with a substantial focus on platform or infrastructure topics.

• Strong technical proficiency in Python and SQL (advanced), including knowledge of PySpark and Delta Lake semantics.

• Extensive hands-on experience with Apache Airflow in a production environment: DAG design, deployment, operations, upgrades, and orchestration of Databricks and ADF workloads.

• Practical experience with Azure resource management: Key Vault, managed identities, service principals, and secret management.

• Knowledge in access management: Azure RBAC, Entra ID, and Unity Catalog permissions.

• Familiarity with Azure DevOps (repositories, pipelines, service connections, environments) and CI/CD processes for data platforms.

• Experience with Azure Data Lake Storage, networking, security fundamentals, and Azure Data Factory.

• Proficiency in Infrastructure as Code (Terraform) and multi-environment promotion practices.

• Hands-on experience with Databricks and Unity Catalog (compute, jobs, governance).

• Knowledge in observability and monitoring, including alerting and incident management.

• Strong understanding of Data Lakehouse architecture, Medallion architecture, and metadata-driven architecture.

• Awareness of cost management and performance considerations.

• Excellent problem-solving skills and analytical thinking abilities.

• Capability to work effectively in cross-functional teams and influence stakeholders without direct authority.

• A proactive ownership mindset and keen attention to detail.

• Strong communication skills to engage with both technical and business stakeholders.

• Role demands a permanent contract or B2B arrangement.

🏝️ Benefits

• Competitive salary and performance-based bonuses.

• Opportunities for professional development and career advancement.

• Flexible working arrangements to promote work-life balance.

• Comprehensive health, dental, and vision insurance.

• Generous paid time off and holiday policies.