This is a fully remote position, open to applicants in United States.

📋 Description

• Design and manage efficient distributed operations for the Lance dataset.

• Create effective indices to facilitate predicate pushdown and enhance query performance in Spark, Ray, or Trino.

• Work on various aspects of the Lance format in Rust, including table formats and data encodings.

• Lead initiatives in the open-source community to integrate the Lance format with systems like Spark, Hive Metastore, Presto, Trino, Ray, and other data infrastructure platforms.

• Operate and enhance the internal data processing infrastructure.

• Advocate for the Lance format within open-source communities and at Big Data conferences.

• Contribute to scalability and performance improvements, integrations within the broader data and AI ecosystem, simplified distributed operations, and enhancements in usability and maintainability.

⛳️ Requirements

• Over 10 years of experience in developing high-performance databases, big data systems, or large-scale data services.

• Profound understanding of the internals of open-source Big Data or AI training systems (such as Hadoop, Spark, Flink, Ray, Iceberg, Delta Lake, Hudi, ClickHouse, Trino, Presto, PyTorch, or JAX).

• Strong expertise in high-performance computing using C++, Java, and/or Scala.

• Experience with Rust or a willingness to learn it.

• Ability to work quickly, independently, and collaborate with a high-caliber team.

• Contributor, committer, or PMC member in Apache or other significant open-source projects.

• Familiarity with Apache Arrow, DataFusion, Parquet, Iceberg, or Delta Lake.

• Proven track record of driving substantial features or integrations in distributed systems.

• Strong presence in the community and a passion for open-source collaboration.