Senior Evaluation AI Engineer

This is a fully remote position, open to applicants in Mexico, +2 more countries.

📋 Description

• Design, architect, implement, and scale customized automated evaluation pipelines for multi-agent LLM systems.

• Assess intent classification, tool-calling accuracy, state-transition integrity, checkpointer manipulation, and multi-turn memory replay.

• Create advanced domain-specific metrics utilizing G-Eval, LLM-as-a-judge architectures, and deterministic verification techniques.

• Analyze intricate document parsing, such as leases and invoices, along with real estate financial modeling.

• Execute defensive alignment testing, adversarial red-teaming, and prompt-injection mitigation strategies.

• Mitigate hallucination, data leakage, and unauthorized tool execution within corporate cloud environments.

• Develop evaluation datasets, golden regression sets, and synthetic-data pipelines that model the behaviors of property managers, tenants, and brokers.

• Oversee production agent workflows to monitor for behavioral drift, latency regressions, cost inefficiencies, and token utilization.

• Design, create, and sustain scalable evaluation frameworks and pipelines using Python, SQL, APIs, and CI/CD processes.

• Execute hundreds or thousands of evaluation cases across various models, prompts, RAG pipelines, and AI agents.

• Implement automated grading through deterministic validation, reference-based evaluation, human-defined rubrics, LLM-as-a-judge, and human-in-the-loop assessments.

• Define scoring criteria and quality benchmarks for AI applications.

• Collaborate with Engineering, Product, Data, and Quality Engineering teams to ensure AI capabilities are measurable, explainable, reliable, and ready for production.

⛳️ Requirements

• 5 to 7 years of experience in software engineering, quality engineering, test automation, AI engineering, machine learning, or a related technical field.

• Practical experience in evaluating or testing LLM, NLP, machine learning, or AI applications.

• Proficient Python skills and experience in developing automated test or evaluation workflows.

• Experience in creating and maintaining golden datasets, test datasets, or benchmark datasets.

• Familiarity with adversarial testing methodologies.

• Understanding of AI safety, security & privacy, and responsible AI evaluation practices.

• Extensive structural knowledge of LangChain, LangGraph (including sub-graphs and conditional routing), and LangSmith.

• Hands-on experience with LangSmith for tracing, debugging, conducting experiments, managing datasets, and evaluations.

• Strong GitHub experience, including handling repositories, branching, pull requests, code reviews, and CI/CD workflows.

• Familiarity with Claude Code or similar AI-assisted development tools.

• Experience with AWS and Amazon Bedrock.

• Proficiency in REST APIs, JSON, SQL, and contemporary application architectures.

• Comprehensive understanding of LLMs, prompt engineering, embeddings, vector search, RAG, and AI agents.

• Experience with automated testing, regression testing, integration testing, and end-to-end testing.

• Hands-on experience in evaluating LangGraph-based or multi-agent applications at both node and state-transition levels.

• Strong analytical, problem-solving, and communication capabilities.

• Ability to work independently and take responsibility for ambiguous technical challenges.

• Preferred: Experience with Ragas, DeepEval, Arize Phoenix, TruLens, Promptfoo, or similar evaluation tools.

• Preferred: Familiarity with LLM-as-a-judge and human-in-the-loop evaluation methodologies.

• Preferred: Experience with Bedrock Agents, Knowledge Bases for Amazon Bedrock, Lambda, S3, or CloudWatch.

• Preferred: Experience integrating AI evaluations into GitHub Actions.

🏝️ Benefits

• 6-month duration with a strong likelihood of extension based on performance.

• Opportunities for long-term role potential.