Associate Director, AI Product Engineer – Financial Institution Operations & Technology

This is a fully remote position, open to applicants in United States.

📋 Description

• Establish the essential requirements for each product, assessing how errors may impact customers or operations, and identifying necessary evidence prior to deployment.

• Develop reusable software capable of executing tests, evaluating results, and comparing different versions.

• Collaborate with engineers to replicate failures, determine root causes, and confirm resolutions.

• Contribute product code aimed at enhancing accuracy and control.

• Define pass criteria and scoring guidelines in conjunction with business and risk teams.

• Sustain representative test cases, which include incomplete records and rare, high-impact errors.

• Evaluate factual accuracy, missing information, incorrect additions, and support from referenced sources.

• Validate automated scoring against expert assessments.

• Compare product versions using the same tests and document results based on task and error severity.

• Authenticate extracted facts against original records, including amounts, dates, totals, and citation support.

• Assess when the product should provide answers, request clarification, or refer inquiries to a human.

• Test for fabricated facts, unsupported recommendations, biased outcomes, sensitive information disclosures, and circumventions of safeguards.

• Safeguard client data, limit access and authorized actions, and utilize only sanctioned services.

• Document consistent test results, unresolved issues, and limitations.

• Advise on release decisions, restricted usage, or further development, while supporting independent reviews and client approvals.

• Sample live outputs, monitor error rates and user corrections, and initiate investigations or suspensions when quality standards drop below agreed thresholds.

• Articulate findings, mentor colleagues, develop reusable tests and controls, and assist in proposals, technical estimates, and recruitment.

⛳️ Requirements

• At least 5 years of professional experience, including a minimum of 1 year in AI evaluation.

• Proven ownership of AI evaluations that have influenced actual release decisions, encompassing test design, expert-reviewed answers, scoring, and failure investigation.

• Proficient in Python with experience in creating automated testing software, linking data sources, and documenting repeatable results across product versions.

• Capability to select representative samples, analyze error rates and uncertainties, and differentiate between genuine improvements and weak tests or inconsistent scoring.

• Background in building or enhancing AI products utilized in operations, with practical controls for sensitive data, output accuracy, and human oversight.

• Experience within financial services or other regulated sectors, with the ability to translate business needs into effective tests and controls.

• Strong communication and collaboration skills, with a knack for mentoring colleagues, and a willingness to travel to client locations as required.

• Preferred: Experience in banking, particularly in lending, payments, servicing, collections, or default operations.

• Preferred: Independent evaluation of AI products, verifying automated scoring against expert judgment, or assessing security and fairness in financial institutions.

• Preferred: Evaluation experience in document reviews, summaries, information searches, or automated workflows, including comparisons across AI services and traditional software.

• Preferred: Consulting or product delivery experience, including client discovery, implementation planning, and user adoption.

• Candidates must be able to confirm their legal eligibility to work in the United States without employer sponsorship now or in the future.

🏝️ Benefits

• Equal opportunity employment.

• Reasonable accommodations for essential job functions.

• Travel opportunities to client sites as needed.