AI Operations Specialist

This is a fully remote position, open to applicants in United States.

📋 Description

• Prioritize initiatives based on their business impact and advocate for valuable projects.

• Collaborate with process owners to map current workflows and determine optimal use of AI and human intervention.

• Develop AI workflows, compose prompts, establish pipelines, uphold quality standards, and oversee post-launch performance.

• Construct test sets, evaluate accuracy, and calibrate LLM judges in alignment with human reviewers.

• Establish criteria for launch readiness of AI features and furnish evidence for go/no-go assessments.

• Convert ambiguous policies into consistently applicable review criteria in collaboration with product, legal, and policy teams.

• Develop labeling guidelines and generate human-verified labels using actual data.

• Execute large-scale labeling with vendor agents and verify outcomes through agreement metrics and spot checks.

• Document reusable solutions and key learnings.

• Ensure methods remain up-to-date as models, tools, and vendor agents evolve, retiring ineffective approaches.

• Manage assigned workstreams comprehensively, from initial setup to quality assurance checks.

⛳️ Requirements

• 5–7 years of experience in operations, program management, analytics, consulting, or applied AI, with a focus on accountability for outcomes.

• Daily user of AI who has successfully created a project with measurable impact, such as a workflow, automation, or analysis.

• Proficient in understanding accuracy, precision, recall, and sampling metrics.

• Experienced in transforming disorganized processes into measurable frameworks.

• Background in quality assurance, including review guidelines, QA, or audits.

• Strong writing skills and the ability to collaborate effectively with business leaders, product managers, and engineers.

• Self-motivated and comfortable working with minimal guidance.

• Must be legally authorized to work in the U.S. without sponsorship.

• Preferred: hands-on experience with SQL, Python, or API integration.

• Preferred: experience in trust and safety, content moderation, annotation operations, or other human-in-the-loop processes at scale.

• Preferred: familiarity with fine-tuning models or calibrating LLM judges.

• Preferred: experience in a two-sided marketplace, travel, or another operations-intensive industry.

• Preferred: experience in establishing intake and prioritization processes for a new team.

🏝️ Benefits

• Benefits (unspecified)