AI SDET – Software Development Engineer in Test

This is a fully remote position, open to applicants in Brazil.

📋 Description

• Take ownership of and oversee the Langfuse evaluation harness, which includes the development of "Golden Paths" and adversarial testing scenarios.

• Ensure compliance with regulatory quality gates, such as SR 26-2 and NYDFS Reg 187.

• Conduct shadow mode validation to assess AI performance in comparison to human benchmarks.

• Evaluate for data quality degradation, errors in population segmentation, and the reliability of human-in-the-loop (HITL) checkpoints.

• Confirm that the output standards established by the Designer, including explanation blocks and signal types, are upheld in all evaluation contexts.

• Identify edge cases through thorough skeptical and adversarial testing methodologies.

• Collaborate with engineers to address and resolve logic failures.

⛳️ Requirements

• Advanced expertise in Python 3.11+, Pytest, and automated testing frameworks.

• Proven experience with AI evaluation tools like Langfuse, RAGAS, or comparable systems.

• Familiarity with LLM-as-a-judge paradigms.

• Understanding of model risk management and regulatory compliance testing.

• Experience implementing shadow mode deployment strategies and conducting statistical validation.

• A skeptical and adversarial mindset to uncover edge cases.

• Strong attention to detail concerning regulatory evidence.

• A collaborative approach to working alongside engineers to rectify logic failures.

🏝️ Benefits

• Health and dental insurance.

• Meal and food allowance.

• Childcare assistance.

• Extended paternity leave.

• Partnership with gyms and health and wellness professionals through Wellhub (Gympass) TotalPass.

• Profit Sharing and Results Participation (PLR).

• Life insurance.

• Ongoing learning opportunities via CI&T University.

• Discount club.

• Free online platform dedicated to physical, mental, and overall well-being.

• Courses on pregnancy and responsible parenting.

• Collaborations with online learning platforms.

• Access to a language learning platform.