AI Evaluation Engineer

This is a fully remote position, open to applicants anywhere in the world.

📋 Description

• Develop and implement automated CI/CD testing pipelines to validate agent performance prior to deployment.

• Create and sustain comprehensive datasets of regression prompts, edge cases, and synthetic scenarios to evaluate agent accuracy, safety, and response time.

• Execute cost-effective, rapid code-based evaluations, including JSON schema checks, regex validation, API status monitoring, and embedding similarity assessments.

• Construct test suites for multi-turn dialogues, assessing context retention, state drift, and safeguards against infinite tool-calling loops.

• Develop adversarial test suites addressing prompt injections, malformed inputs, and unpredictable tool responses to ensure secure sandboxes and safety measures.

• Implement tracing functionalities using OpenTelemetry, LangSmith, Phoenix, or comparable tools.

• Minimize LLM API costs through effective caching and mocking strategies.

⛳️ Requirements

• Proficient in Python or TypeScript, particularly in API testing, headless execution, and asynchronous systems.

• Practical experience in constructing LLM evaluation pipelines or statistical testing frameworks, including confidence intervals or pass rates across multiple runs.

• Familiarity with integrating tests into GitHub Actions or similar CI systems utilizing SDK-level execution.

• Skilled in validating structured outputs, including JSON schema, OpenAPI specifications, and function-calling payloads.

• Knowledge of tracing tools like OpenTelemetry, LangSmith, Arize Phoenix, W&B Weave, or custom logging solutions.

• Capability to independently design and develop an evaluation architecture from the ground up.

• Strong proficiency in written and spoken English.

• Minimum of 3 hours of daily overlap with US Central Time business hours.

• Experience with code execution sandboxes or agent frameworks such as LangChain, LlamaIndex, AutoGen, or CrewAI.

• In-depth understanding of embedding models and vector databases.

• Experience in load testing non-deterministic endpoints.

🏝️ Benefits

• Paid training and additional professional development opportunities.

• Flexible work schedule and preferred working hours.

• Option to work remotely from any location with a reliable internet connection.

• Steady stream of engaging and challenging product-related tasks.

• Annual salary increases based on diligence and strong performance.

• Monthly compensation payment via wire transfer.