Senior Data Platform Engineer

This is a fully remote position, open to applicants in Alabama, +45 more states.

📋 Description

• Develop, scale, and manage the enterprise data platform.

• Design and implement reusable batch and streaming processing frameworks utilizing Airflow, Spark, and Flink.

• Oversee the Iceberg/Nessie/Trino/StarRocks data lakehouse layer, focusing on governance, query performance, and storage lifecycle management.

• Construct and maintain Kafka streaming infrastructure, including topics, schemas, and connectors.

• Create reusable frameworks for data ingestion, cleansing, and harmonization.

• Develop UI components for platform status, catalogs, job status, and data quality visibility.

• Design APIs that expose ingestion, querying, catalog metadata, and job orchestration capabilities.

• Build MCP servers and interfaces for AI-agent access to platform data, catalogs, and lineage.

• Create tools and guardrails that enable AI agents to perform platform functions with human-in-the-loop controls.

• Collaborate with platform consumers, including the Ontology Platform team, to address API and agentic access requirements.

• Ensure scalability through effective capacity planning, performance tuning, and architectural evolution.

• Design resilient systems that provide fault tolerance, disaster recovery, data consistency, and graceful degradation.

• Establish observability through metrics, logging, tracing, and alerting mechanisms.

• Lead on-call and production support practices, including runbooks and incident response.

• Advocate for agentic software development utilizing AI coding agents.

• Implement testing, CI/CD, infrastructure-as-code, and code and architecture review practices.

• Mentor engineers on platform internals and tools for agentic development.

⛳️ Requirements

• 8-12 years of experience in software/data engineering.

• Extensive hands-on experience in building or operating large-scale data platform infrastructure.

• In-depth, practical experience with several technologies including Apache Airflow, Apache Iceberg, Apache Spark, Apache Flink, Apache Kafka, Nessie, Trino, and StarRocks.

• Strong grasp of lakehouse architecture, table formats, catalog management, query engines, and the trade-offs between batch and streaming.

• Experience in designing and building APIs, with a solid understanding of API governance, versioning, and security.

• Direct experience with or a strong interest and quick aptitude for building MCP servers or similar agent-facing interfaces.

• Practical experience using AI coding agents like Claude Code or GitHub Copilot as a core part of the development workflow.

• Strong fundamentals in distributed systems, performance tuning, capacity planning, fault tolerance, and resilience.

• Experience in building and maintaining production observability through metrics, logging, and tracing.

• Capability to function as an individual contributor with deep technical ownership and to mentor others.

• Familiarity with UI/front-end development is advantageous.

• Experience in financial services or other regulated, data-intensive industries is preferred.

• Contributions to open-source projects within the Airflow/Iceberg/Spark/Flink/Trino ecosystem are preferred.

🏝️ Benefits

• Medical, dental, and vision insurance coverage.

• 401(k) plan with company matching contributions.

• Paid time off.

• Holidays.

• Parental leave.

• Opportunities for professional development reimbursement.