Senior Data Platform Engineer, Microsoft Fabric

This is a fully remote position, open to applicants in Poland.

📋 Description

• Take full ownership of the client's medallion data architecture from start to finish on Microsoft Fabric.

• Design and construct Bronze, canonical Silver, and Gold Delta tables featuring PAN-safe physical layouts.

• Develop Fabric Spark feature pipelines for rolling velocity, merchant-level rolling, hierarchical rollup, and cross-merchant features.

• Create per-flavor-version Silver adapters, unified by name and validated against golden fixtures.

• Establish the idempotent Bronze-to-canonical-Silver MERGE DAG according to Aperia's canonical Fabric internal specification v1.4.

• Create Gold materialization pipelines for transaction-risk and merchant-risk grains, incorporating score-source stamping and idempotent merge semantics.

• Design and implement Direct Lake semantic models for reporting surfaces, performance-testing them against representative data volumes.

• Configure OneLake namespace layouts, workspace catalogs, and relationships for cross-workspace shortcuts.

• Develop idempotent Delta write patterns using delta-rs from Python containers featuring segmented parallel writes and a single atomic commit.

• Assess Fabric capacity, choose F-SKUs, conduct load testing, and make decisions between Reservation versus PAYG.

• Collaborate with the DevOps engineer on Fabric Infrastructure as Code (IaC), including Terraform capacity provisioning and Fabric REST workspace setup.

• Provide consultation on a ClickHouse contingency if Direct Lake latency fails to meet client-tier UI requirements.

• Work alongside a Senior Technical Lead, Senior DevOps engineer, and mid-level developer on platform foundation and risk product delivery.

• Engage in a platform that ingests transaction and merchant data via SFTP and API, tokenizing sensitive PAN data at the ingestion boundary, while feeding DMN rules and Mastercard Brighterion feeds.

⛳️ Requirements

• Over 5 years of experience in data engineering.

• At least 2 years of practical Spark development experience; PySpark is preferred.

• Profound knowledge of Delta Lake internals, including transaction log format, checkpoints, retention settings, VACUUM, OPTIMIZE, deletion vectors, and statistics collection.

• Expertise in Parquet format, covering footers, statistics allowlists, row-group sizing, dictionary encoding, and column indexes.

• Hands-on experience with Microsoft Fabric, including workspaces, capacities, notebooks, lakehouses, semantic models, Direct Lake, and SQL analytics endpoints.

• SQL performance tuning capabilities for OLAP (Fabric) and OLTP (PostgreSQL) workloads.

• Strong Python data engineering skills, including familiarity with pandas, PyArrow, and delta-rs.

• Ability to clearly articulate architecture in writing and work independently on well-defined features.

• Experience with PCI-scoped data platforms.

• Background in financial services or the payments domain.

• Experience in authoring Power BI semantic models using DAX.

• Familiarity with established data-standard governance patterns, including logical model vs physical profile, SCD2 dimensions, natural-key idempotency, and extension attribute stores.

• Knowledge of OneLake shortcuts and cross-workspace lakehouse patterns.

• Familiarity with ClickHouse or a similar OLAP engine.

• Experience with Terraform for Azure data resources.

• Previous experience with fluid or schema-flexible ingestion patterns.

🏝️ Benefits

• Flexible and remote working options.

• Opportunities to work on international projects with leading global clients.

• Travel opportunities related to international projects.

• A non-corporate working environment.

• Access to language classes.

• Participation in knowledge-sharing initiatives.

• Provision of private healthcare and life insurance.

• Availability of a multisport card.

• The availability and terms of individual benefits may vary depending on the chosen form of cooperation.