RAG and Evaluation Engineer

This is a fully remote position, open to applicants in United States.

📋 Description

• Take ownership of the knowledge surface, including ingestion pipelines for source code, structured metadata, technical documentation, patches, and other corpora provided by the customer.

• Ensure high retrieval quality through effective chunking, embeddings, hybrid retrieval, reranking, and maintaining freshness.

• Manage the evaluation harness, which includes benchmarks for translation accuracy, dependency-map correctness, and overall agent quality.

• Conduct A/B testing and regression detection across prompts, retrieval, and model modifications.

• Facilitate the feedback loop from production usage back into evaluations and retrieval processes.

• Define the benchmarks for what constitutes “good” for the platform in situations where clarity is lacking, enabling the team to assess the agent's improvement.

• Collaborate with the Agent Engineers during the prompt-and-evaluation iteration cycle.

⛳️ Requirements

• A Bachelor’s degree in Computer Science, Engineering, Information Science, or a related field, accompanied by 4 years of professional software engineering experience; equivalent experience may be considered in lieu of a degree.

• Proven experience in shipping a production RAG system, with quality metrics the candidate can articulate (emphasis on rigor over scale).

• Ability to thrive in a fast-paced, collaborative work environment.

• Practical experience with retrieval pipelines, including ingestion, chunking, embedding, hybrid retrieval, and reranking.

• Strong skills in applied evaluation, including benchmark design, regression detection, and LLM-as-judge patterns.

• Understanding of when BM25 outperforms embeddings and the limitations of both.

• Commitment to measuring all outputs; opinions on chunking are supported by benchmarks.

• Detail-oriented and capable of defining metrics prior to team consensus.

• Proficient in the native use of AI tools, such as parallel agents and models as collaborators.

• Strong programming skills in TypeScript or Python.

• Proven experience in a remote work setting.

🏝️ Benefits

• Opportunity to support high-visibility federal missions in IT and healthcare.

• A culture that emphasizes innovation, growth, collaboration, and quality.

• Access to state-of-the-art tools and technologies.

• Comprehensive benefits package for you and your family.

• A career path that recognizes and rewards ambition and performance.