Director, AI Assisted Solutions

This is a fully remote position, open to applicants in United States.

📋 Description

• Oversee the complete delivery process for assigned College Board assessment program engagements.

• Identify challenges and requirements, and design and develop AI tools, prompts, and workflows.

• Provide functional solutions along with training and adoption strategies.

• Assess AI-generated outputs for construct misalignment, reasoning flaws, biases, and factual inaccuracies.

• Take responsibility for the final quality approval of tools delivered to partner teams.

• Set quality benchmarks and evaluation criteria, including rubrics, benchmarks, and scoring frameworks.

• Determine requirements for system-based solutions that necessitate additional stakeholder input and resources.

• Analyze cultural, staffing, process, and technological gaps that impact scalability.

• Prioritize engagement and develop program roadmaps for AI-driven item development.

• Establish workflow standards and necessary inputs.

• Contribute to enterprise-wide strategies, architectures, and roadmaps for agentic-human hybrid systems.

• Advocate for supported programs in discussions related to cross-program tools and methodologies.

• Create and facilitate training on AI-assisted and agentic workflows.

• Produce process documentation to support responsible tool adoption.

• Define and document impact metrics.

• Maintain the College Board's assessment design and development standards.

• Experiment with cutting-edge AI tools, models, and techniques.

• Share insights and learnings with teams and internal communities of practice.

• Attend and represent the College Board at pertinent external meetings and conferences.

⛳️ Requirements

• A minimum of 7 years of experience in developing standardized assessment items across various types.

• An expert-level understanding of item-writing and construct alignment.

• Extensive subject-matter expertise in Reading Comprehension, Writing, Math, or Science assessments.

• At least 3 years of practical experience with AI-assisted or agentic methodologies, including AIG, RAG, prompt engineering, and context engineering.

• Proven track record of driving process or technology enhancements that lead to measurable outcomes.

• Familiarity with large language models, hallucinations, context-window limitations, and the obsolescence of training data.

• Experience with agentic and multi-agent workflows, tool-calling, and orchestrated prompts.

• Background in vibe coding or rapid prototyping.

• Capability to independently develop functional scripts, tools, or interactive prototypes.

• Experience in designing processes, educating others, and facilitating the adoption of tools, processes, and systems.

• Proficiency in bridging technical and non-technical discussions.

• Demonstrated ability to collaborate across functions.

• Strong organizational and planning skills.

• Experience in implementing processes and procedures, managing project timelines, and maintaining tracking documentation.

• Knowledge of item and test development life cycles, digital item authoring and review tools, and content management/item banking systems.

• Comfort in navigating ambiguity within a fast-paced environment.

• Willingness to travel 6–8 times annually.

• Authorization to work in the United States.

• A genuine passion for enhancing educational and career opportunities through mission-driven work.

• An inquisitive nature and eagerness to explore emerging technologies and AI-driven solutions.

• Strong written and verbal communication skills that are clear and concise.

• A learner’s mindset and dedication to personal growth.

• A commitment to creating impactful outcomes and striving for excellence.

• A collaborative and empathetic approach to teamwork.

• A Master’s degree or equivalent education and experience is strongly preferred.

🏝️ Benefits

• A meaningful career.

• A supportive team environment.

• A comprehensive benefits package designed to promote employee well-being.

• Competitive and fair compensation.

• Location-adjusted salaries.

• Transparent compensation offers based on market data.

• Option for fully remote work.

• Hybrid work option for candidates residing near College Board offices, requiring in-office presence on Tuesdays and Wednesdays.

• Opportunities to travel to relevant professional meetings and conferences.