Lead Data Engineer, Contract, Full-Time

This is a fully remote position, open to applicants in India.

📋 Description

• Design and construct scalable data pipelines and infrastructure that support AI and product systems.

• Create and uphold architectures for data ingestion, transformation, and storage.

• Develop and oversee both batch and real-time data pipelines.

• Construct and enhance vector search, retrieval, and machine learning data pipeline systems.

• Ensure the reliability, security, and governance of data.

• Collaborate with AI and backend engineering teams on training, inference, and product features.

• Implement frameworks for monitoring, observability, and data quality.

• Optimize the performance of large-scale datasets and query systems.

• Contribute to decisions regarding technical architecture and long-term data strategy.

• Define the culture, standards, and hiring criteria as the initial data hire.

• Work closely with founders and product leadership to translate data capabilities into product decisions.

⛳️ Requirements

• Over 7 years of professional experience, primarily in dedicated data engineering roles.

• Extensive experience in designing and constructing data pipelines and distributed data systems.

• Proficiency with relational databases; PostgreSQL is preferred, while MySQL or similar options are acceptable.

• Familiarity with NoSQL databases.

• Experience with vector databases tailored for modern AI systems.

• Strong programming skills in Python.

• Ability to make and rationalize architectural decisions.

• Experience in building scalable backend systems.

• Expertise in designing data models and storage architectures.

• Solid understanding of data-processing performance and optimization.

• Highly desirable: Knowledge of Apache Spark, Apache Airflow, Kafka, Elasticsearch, or OpenSearch.

• Highly desirable: Experience with PostgreSQL, MongoDB, Qdrant, Milvus, or pgvector.

• Highly desirable: Familiarity with Pandas or Polars.

• Nice to have: Experience with AI or machine learning platforms.

• Nice to have: Familiarity with stream processing and event-driven architecture.

• Nice to have: Experience with GCP, AWS, or Azure cloud infrastructure.

• Nice to have: Background in high-growth startups or early-stage companies.

🏝️ Benefits

• A genuine community that prioritizes growth and well-being.

• A remote-first work environment.

• Full-time, long-term positions available.

• Opportunity to collaborate with global teams and products.

• Opportunities for personal and professional growth.