ML Engineer, Retrieval – Grounded Generation

This is a fully remote position, open to applicants in United States.

📋 Description

• Take ownership of the system test plan encompassing functional, integration, performance, and load testing.

• Convert product acceptance criteria into actionable tests.

• Examine acceptance criteria prior to development, identifying any ambiguities or unverifiable requirements.

• Ensure accountability among engineering, platform, and partner teams regarding acceptance criteria and release evidence.

• Engage daily with engineering while reporting test results independently.

• Organize and manage customer user acceptance testing, tracking findings until resolution.

• Develop and uphold automated test suites that integrate into CI/CD as release gates.

• Set automated coverage standards and report against them.

• Create performance and load testing scenarios for anticipated user loads.

• Sustain regression coverage throughout releases.

• Construct labeled evaluation sets in collaboration with subject-matter experts.

• Design and conduct sampled audits of automated decisions.

• Assess system misses, correct outcomes, and user overrides.

• Document metrics, data, and release evidence.

• Generate evidence independently from model builders.

• Keep track of version inventories across builds, models, configurations, and thresholds.

• Create test reports, evaluation records, and rollback criteria for each release.

• Ensure test environments adequately reflect production settings.

• Maintain the release gate, ensuring no release proceeds without the necessary evidence.

⛳️ Requirements

• A minimum of 5 years in software quality engineering, test automation, or software engineering with significant dedicated test ownership.

• A clear perspective on effective testing methodologies and discernment regarding valuable practices.

• Proficient coding skills in a test-automation language and framework, such as Python with pytest or JavaScript/TypeScript with Playwright, Cypress, or Selenium.

• Capability to comprehend and analyze application code that is under test.

• Regular use of AI-assisted development, with the ability to evaluate the accuracy of AI output.

• Proven experience in building automated test suites and integrating them into CI/CD as release gates.

• History of managing a test plan through its review and acceptance phases.

• Experience in testing against requirements instead of implementation, collaborating with product owners or end users.

• Familiarity with performance and load testing.

• Working knowledge of statistical principles, including sampling design and measurement confidence.

• Experience in measuring system misses, not just successes.

• Willingness to report unfavorable results independently.

• Ability to gate work owned by individuals outside your direct reporting line.

• Strong written documentation skills that are clear and audit-ready.

• Openness to holding and explaining a release gate.

• US Citizenship is required.

• Active US Secret clearance is essential from the start, along with a favorable investigation and CAC eligibility.

• Willingness to travel up to 25% to customer locations, DEFCON AI HQ, and vendor sites.

• Preferred qualifications include active Top Secret clearance; experience in government DevSecOps, RMF, or ATO; formal accreditation/audit evidence; contract/API testing; test data management; test infrastructure; AI-assisted testing; familiarity with tools like JMeter, k6, Locust, or similar; ML/model-output evaluation including groundedness and citation checking; knowledge of model cards or NIST AI RMF; and Section 508 and WCAG verification.

🏝️ Benefits

• Competitive salary, bonus, and equity package.

• 100% employer-covered comprehensive health insurance, including medical, dental, and vision for you and your family.

• Unlimited PTO, subject to manager/management approval.

• Flexible work environment that allows you to manage your workday.

• 14 weeks of fully paid parental leave.

• Career advancement opportunities with potential for rapid growth based on strong performance.

• Access to professional development and ongoing learning opportunities.

• Optional 401K, FSA, and equity incentives.

• Mental health benefits provided through Tara Mind.

• Cost-effective GLP-1 solutions available through Crux.