Senior Data Engineer – AI/ML

This is a fully remote position, open to applicants in India.

📋 Description

• Create and develop AI/LLM data pipelines that facilitate training, inference, evaluation, embeddings, and retrieval tasks.

• Construct production-quality RAG systems, encompassing ingestion, chunking, embedding generation, indexing, retrieval, reranking, and context building.

• Design AI applications utilizing LLMs, structured outputs, function/tool invocation, and agentic workflows.

• Enhance semantic search and vector retrieval systems.

• Formulate frameworks for LLM evaluation, monitoring, tracing, quality assessment, latency, and cost optimization.

• Architect scalable batch and streaming pipelines leveraging Databricks, Apache Spark, Delta Lake, Snowflake, and Airflow.

• Create data products and platforms that enable AI applications to access structured and unstructured enterprise data.

• Develop reliable ETL/ELT pipelines and enhance large-scale distributed workloads for efficiency and cost-effectiveness.

• Establish best practices for data quality, governance, lineage, security, and observability.

• Collaborate with ML and application engineering teams to transition AI prototypes into production-ready systems.

• Contribute to building the data and AI foundation for smart products and production-ready AI systems.

⛳️ Requirements

• Over 5 years of experience in data engineering, software engineering, distributed systems, or a related discipline.

• Proficient programming abilities in Python and/or Scala/Java, along with advanced SQL skills.

• Practical experience with Databricks, Snowflake, Apache Spark, Delta Lake, and Airflow.

• Strong familiarity with cloud data platforms, particularly Snowflake and/or Databricks.

• Hands-on experience in developing applications utilizing LLMs or Generative AI.

• In-depth knowledge of RAG architectures, embeddings, vector databases, semantic search, and retrieval systems.

• Understanding of LLM concepts such as prompting, structured outputs, tool invocation, and model evaluation.

• Experience in designing scalable, reliable, and observable production data systems.

🏝️ Benefits

• Work remotely (almost) anywhere for up to 20 days each year.

• Company-funded therapy sessions via SpringHealth.

• Company-sponsored subscription to Headspace.

• Annual company-wide week off each year.

• Paid parental leave.

• Generous paid vacation plus time off for your birthday.

• Paid time for volunteer work.

• Development Dollars for personal growth.

• Leadership development opportunities.

• Access to thousands of on-demand e-learning resources.

• Travel discounts.

• Employee Resource Groups.

• Quarterly team retreats.

• Tax optimization options available.

• Comprehensive health insurance.

• Pension fund contributions.