Senior Analyst, Model Risk & AI – Gen AI Model Validation

This is a fully remote position, open to applicants in Connecticut.

📋 Description

• Conduct validations on models pertaining to GenAI and agentic AI applications across various functional domains and business lines.

• Verify that model calculations, algorithms, and methodologies are precise and suitable for their intended purposes.

• Create and establish challenger solutions along with testing methodologies for tasks such as summarization, question answering, search, data synthesis, LLM-as-a-judge, and similar activities.

• Examine quantitative and qualitative testing methods to ensure model accuracy, strength, and dependability.

• Evaluate data inputs, assumptions, prompt engineering, and context engineering.

• Develop and implement testing strategies for non-deterministic AI systems, which include variability analysis, confidence intervals, and statistical evaluation techniques.

• Scrutinize modeling components such as inputs, calculations, outputs, conceptual integrity, monitoring and controls, and documentation.

• Identify issues and recommendations, conduct impact assessments, and prepare model validation reports.

• Carry out governance responsibilities related to tracking findings, remediation testing, and validation.

• Improve the GenAI model validation framework by incorporating standardized evaluation metrics and validation instruments.

• Collaborate with Data Science teams to synchronize model risk practices with advancing modeling capabilities and encourage proactive risk management.

• Remain updated on AI/ML, GenAI, agentic systems, regulatory requirements, and departmental initiatives.

• Assist in the evaluation and testing of tools such as VertexAI/Google ADK, LangChain/LangGraph, RAG frameworks, HuggingFace, and OpenAI APIs.

• Evaluate the alignment of solutions with AI Governance standards, Model Risk Management policies, and emerging regulatory demands.

⛳️ Requirements

• An advanced degree (M.S. or Ph.D.) in a relevant area such as Artificial Intelligence, Machine Learning, Computational Science, Engineering, Statistics, Applied Mathematics, Actuarial Science, Computer Science, or Quantitative Economics.

• Over 3 years of industry experience in machine learning or data science, with at least 1 year concentrated on GenAI and/or agentic AI.

• Experience in P&C, Group, Life, or similar insurance products is advantageous.

• Proficient programming and technology skills in Python, SQL, and Git.

• Strong grasp of GenAI principles, including prompt and context engineering, retrieval-augmented generation (RAG), LLM internals, LLM evaluations, and multi-agent systems.

• Familiarity with contemporary cloud and AI tools such as GCP/Vertex AI, Google uAgent Development Kit (ADK), AWS/SageMaker, LangChain/LangGraph, HuggingFace, and OpenAI APIs.

• Capacity to work autonomously with proactive self-directed accountability, meeting deadlines while adjusting to changing priorities.

• Excellent analytical, critical, and investigative thinking abilities.

• Dedication to continuous learning and development in advancing modeling techniques and AI technologies.

• Creative problem-solving skills, innovative thought processes, and a readiness to question established norms.

• Outstanding communication and teamwork skills, with the ability to convey complex technical concepts to non-technical audiences.

🏝️ Benefits

• Short-term or annual bonuses.

• Long-term incentives.

• Immediate recognition for outstanding performance.

• Additional perks and benefits (details not specified).