Director of AI Operations – Governance

This is a fully remote position, open to applicants in United States.

📋 Description

• Assess, benchmark, refine, and modify configurations of machine learning and generative AI models (including LLMs) in production, utilizing practical knowledge of model training, tuning, and evaluation methodologies instead of exclusively depending on vendor documentation.

• Implement applied AI/ML research and cutting-edge techniques to guide decisions regarding build-vs-buy, model selection, and architectural strategies across the AI portfolio.

• Collaborate directly with data science and ML engineering teams on issues related to model performance, drift detection, and the need for retraining or reconfiguration, contributing technical insights rather than merely acting as an intermediary.

• Maintain proficiency in prompt engineering, retrieval-augmented generation, agentic/orchestration frameworks, and the significant differences between generative AI and traditional ML systems, translating those differences into governance and staffing decisions.

• Take ownership of AI sustainment and governance for workplace AI tools, encompassing enablement, purchased solutions, and custom builds, serving as the central operational function for the AI strategy, and developing the team and processes to scale it.

• Oversee all AI-related licenses and entitlements: track usage, optimize allocations, facilitate reallocation, and collaborate with Finance for cost transparency and optimization.

• Analyze production usage patterns and performance to propose roadmap items, enhancements, and deprecations grounded in real-world production dynamics.

• Manage and triage support tickets for workplace AI tools and platforms, driving resolutions across vendors, internal engineering teams, and security partners.

• Conduct ongoing evaluations of AI tools and models, including monitoring for drift, benchmarking performance, and ensuring tools remain effective and aligned with business KPIs.

• Keep updates and security status for AI platforms, coordinating patches, version upgrades, vulnerability remediation, and compliance with Shield AI security policies.

• Coordinate model swaps and configuration adjustments in production, including rollout planning, risk assessment, change control, and post-deployment monitoring.

• Design, maintain, and govern shared prompt libraries, establishing standards for prompt quality, reuse, versioning, and training for both end-users and builders.

• Manage secrets (API keys, credentials, tokens) utilized by AI tools and orchestrators, ensuring secure storage, rotation, and access control in collaboration with Security and IT.

• Define and maintain connectors and extensions (e.g., integrations into SaaS systems, data sources, and workflow tools) to guarantee reliable, secure data access for AI workflows.

• Establish and operate auditability frameworks for AI tools, including logging, traceability of AI-assisted actions, and reporting for compliance and risk management.

• Lead AI governance practices for workplace AI (policies, guardrails, usage standards, approval workflows, exception processes) in collaboration with Security, Legal, and HR.

• Collaborate with business solution and build teams to guarantee their outputs meet sustainment, observability, and governance requirements prior to production deployment.

• Define operational playbooks, SLAs, and incident response procedures for AI systems, including on-call patterns supported by contractors and platform specialists.

⛳️ Requirements

• 15+ years of experience in platform operations, ML/AI operations, DevOps, or SaaS sustainment roles, including substantial experience in leadership and people management, with a proven history of managing production systems in high-stakes environments (defense, aerospace, enterprise SaaS, or similar).

• Direct, hands-on experience in developing, training, fine-tuning, or evaluating machine learning models or generative AI systems — this is a core requirement, not optional. Candidates must be able to discuss model architecture, training/tuning methodologies, and evaluation techniques with credibility.

• Proficient understanding of AI/ML research practices and the capability to apply current research to production decision-making.

• Background in software engineering or data science sufficient to engage deeply with technical teams on model behavior, integration challenges, and system design trade-offs.

• Direct experience with AI platforms or orchestration tools (e.g., LLM providers, RPA/workflow tools like n8n, enterprise SaaS integrations) and their operational management, including at an organizational or strategic level.

• Demonstrated comprehension of the unique technical and operational challenges posed by generative AI as opposed to traditional ML — including differing skill requirements, risk profiles, and market/salary dynamics — and the ability to leverage this distinction for team design and recruitment.

• Strong background in governance, compliance, or security within the context of data-driven or AI systems, including familiarity with audit, logging, and access control best practices.

• Proven ability to manage licenses and optimize costs for SaaS or AI tools at scale, including collaboration with Finance and procurement stakeholders, and managing substantial budgets.

• Practical experience with monitoring and observability stacks (logs, metrics, alerts) and utilizing those signals to influence product roadmaps and operational enhancements.

• Strong technical proficiency across APIs, connectors, and integrations; capable of working closely with engineering teams and vendors to design and maintain extensions.

• Established record of building and leading high-performing teams, including hiring, mentoring, and developing talent across a mix of core staff and contractors.

• Exceptional executive communication skills, with the ability to translate production dynamics and risks into clear recommendations for senior business and technical leaders, including executive stakeholders.

• Experience operating in a hybrid environment of contractors and core team members, with the capability to define processes and standards that scale as the team matures.

🏝️ Benefits

• Competitive salary and performance-based bonuses.

• Comprehensive health, dental, and vision insurance plans.

• Generous vacation and paid time off policies.

• Opportunities for professional development and continuing education.

• Flexible work arrangements and a supportive work environment.