Senior Data Platform Engineer

This is a fully remote position, open to applicants in Portugal.

📋 Description

• Design and enhance Sword’s streaming lakehouse.

• Construct and manage distributed streaming pipelines that ensure low-latency and highly dependable data transfer.

• Oversee robust workflows that coordinate intricate data movement across various systems.

• Define the platform API interface for both producers and consumers.

• Assess and integrate third-party data platforms.

• Contribute to the development of self-service and agent-friendly interfaces for humans, systems, and AI agents.

• Collaborate with data engineers and analysts on contracts, governance, and data lineage.

• Develop and sustain AI-ready data infrastructure that supports machine learning and AI-driven products.

• Leverage AI coding assistants and large language models to expedite development, automate documentation, and enhance code quality.

• Operate within a regulated environment that adheres to audit, compliance, and governance standards.

⛳️ Requirements

• Demonstrated experience in designing and managing data platforms at scale in production environments.

• Practical experience with contemporary lakehouse table formats; Iceberg is preferred, while Delta Lake or Hudi are also acceptable.

• Knowledge of metadata structures, snapshots, manifests, compaction, copy-on-write, and merge-on-read processes.

• A clear understanding of data catalogs, including REST, Polaris, Glue, Unity, and Hive.

• Familiarity with Snowflake, Starburst, or Databricks from an architectural standpoint.

• Strong proficiency with distributed processing engines; Flink is preferred, but Spark is also acceptable.

• Capability to optimize running jobs and troubleshoot declining pipelines.

• Familiarity with durable execution frameworks such as Temporal, Restate, or similar solutions.

• Experience in building and managing REST or gRPC APIs at scale in a production setting.

• Understanding of API contracts, versioning, retries, rate limiting, and observability practices.

• Solid grasp of Kafka and event-driven architectures, including producers, consumers, partitioning, and delivery semantics.

• Comfort operating in regulated environments where audit, compliance, and data governance are integral to every design process.

• A platform-oriented mindset that prioritizes self-service, API-first systems, and AI agents as users.

• Proficiency in AI at least at the Explorer level (Level 1).

• Bonus: familiarity with Polaris, Nessie, Unity, Prometheus, Grafana, OpenTelemetry, agentic or AI-focused APIs, MCP-style interfaces, HIPAA, FedRAMP, SOC 2, dbt, DataHub, or data contract tools.

🏝️ Benefits

• Variable compensation and private company stock options may be part of the total compensation package.

• Additional benefits are provided beyond the total compensation.

• Country-specific benefits will be detailed separately.