Model Test and Measurement Engineer

This is a fully remote position, open to applicants in United States.

📋 Description

• Create labeled ground truth for evaluating models.

• Design and execute sampled audits.

• Gate releases based on model versions; manage version inventory, evaluation records, and rollback triggers.

• Conduct drift and override reviews.

• Generate evidence for human oversight, fairness, and disparate effects.

• Maintain independence from build roles and generate evaluation evidence without developing models, approving thresholds, or sanctioning releases based on that evidence.

• Assess workflow improvements, including review time, throughput, backlog movement, and rates of overrides and rework.

• Develop quality control for the evaluation phase concerning sample selection, review units, and fair comparisons across various methods or sources searched.

• Specify measurement events needed from other teams and ensure the review workspace produces workflow events from initial use.

• Collaborate closely with data scientists, AI engineers, and technical leadership.

• Guide release decisions, improvement initiatives, and customer evaluations through independent assessments.

⛳️ Requirements

• Over 5 years of experience in model validation as a primary responsibility, not a secondary task.

• Direct involvement in ground truth construction and sampling design.

• Proficient in Python and SQL.

• Ability to work independently from the teams whose models you assess.

• US Citizenship is required.

• Active US Secret clearance is necessary.

• Elevated personnel security requirements are applicable to certain aspects of this work and will be discussed during the screening process.

• Background in regulated industries or government model risk.

• Experience with fairness, disparate effect testing, and documentation for human oversight.

• Familiarity with NIST AI RMF or comparable methodologies.

• Active Top Secret clearance is preferred.

🏝️ Benefits

• Competitive salary, bonus, and equity package.

• A fully remote work environment focused on results.

• Comprehensive health insurance fully covered by the employer, including medical, dental, and vision for you and your family.

• Unlimited PTO, subject to manager approval.

• Flexible work environment allowing you to manage your own workday.

• 14 weeks of fully paid parental leave.

• Occasional travel to DEFCON AI headquarters, customer locations, and partner facilities as required.