Staff Software Engineer, Data Governance – Foundations

This is a fully remote position, open to applicants in California, +18 more states.

📋 Description

• Convert Instacart's data strategy into a practical multi-year architectural roadmap and synchronize it with leadership.

• Take ownership of the open lakehouse foundation, which encompasses unified table formats, storage governance, and multi-engine computing.

• Propel real-time and streaming infrastructure for Advertising, Fraud detection, and Machine Learning applications.

• Implement LLM/AI tools for development, automation, observability, and cost optimization, while assisting in the integration of AI capabilities into the platform.

• Conduct architecture reviews, mentor senior and staff engineers, influence recruitment, and convey technical trade-offs to both technical and executive audiences.

⛳️ Requirements

• Over 10 years of software engineering experience in constructing and managing production-scale data infrastructure or distributed systems.

• Proficient in modern data lakehouse architectures and open table formats, including Apache Iceberg, Delta Lake, or Hudi.

• Familiar with distributed query and compute engines such as Trino, Spark, or ClickHouse, along with performance optimization and production reliability.

• Experience with event-driven and streaming infrastructure technologies like Kafka or Flink.

• Demonstrated ownership of significant platform transitions or migrations successfully implemented in production.

• Capability to develop cost/benefit and Total Cost of Ownership (TCO) models for infrastructure investments.

• Ability to facilitate alignment through architectural documentation and strategy memos across teams and leadership levels.

• Preferred: knowledge of platform governance controls and compliance frameworks, including SOX, CPRA, or GDPR.

• Preferred: experience in FinOps, managing multi-million-dollar infrastructure budgets, and negotiating vendor contracts.

• Preferred: strong SQL skills along with robust Python or Scala capabilities.

• Preferred: experience with orchestration and data transformation pipelines such as Apache Airflow and dbt.

• Preferred: a Bachelor’s, Master’s, or PhD in a relevant engineering or computer science discipline, or equivalent practical experience.

🏝️ Benefits

• Equity grant for new hires.

• Annual refresh equity grants.

• Flexible work location options, including home, office, or coffee shop.

• Regular in-person events.

• Competitive market compensation and benefits.