
Senior AI Development Lead
Posted 9 hours ago

Posted 9 hours ago
This is a fully remote position, open to applicants in United States.
β’ Lead the development of AI workflows and features from conception to deployment, establishing quality benchmarks and assessment plans in collaboration with engineering, product, and business stakeholders.
β’ Prioritize initiatives based on business impact and negotiate trade-offs with the manager alongside business and product leaders.
β’ Analyze current processes to identify opportunities for AI implementation and determine where human oversight is essential.
β’ Design and construct AI workflows, which include prompts, pipelines, and tools.
β’ Oversee quality control and monitor any shifts post-launch.
β’ Develop evaluation frameworks, create test sets, establish accuracy and agreement metrics, and align LLM judges with human reviewers.
β’ Collaborate with engineers on architectural and tooling decisions while rigorously assessing estimates and project scope.
β’ Transform ambiguous guidelines into clear and consistent reviewer criteria in partnership with product, Legal, and Policy teams.
β’ Generate labeling guidelines and human-verified labels in conjunction with subject matter experts.
β’ Manage large-scale labeling operations with vendor teams and review outcomes using agreement metrics and spot checks.
β’ Establish launch quality benchmarks for AI features and provide documentation along with go/no-go recommendations.
β’ Create reusable evaluation and workflow components, document insights, and promote adoption across various teams.
β’ Update methods in line with changes in models, tools, and policies, and advise on retiring outdated methodologies.
β’ 8+ years of experience in product or program management, analytics, consulting, or applied AI.
β’ Demonstrated success in guiding technical programs from concept to execution while managing outcomes such as backlog, cycle time, or launch velocity.
β’ Proficient in SQL.
β’ Strong understanding of accuracy, precision, recall, sampling, and reviewer consensus.
β’ Experience in deploying an AI workflow, agent, or automation that is utilized by others, with evidence of measurable before-and-after results.
β’ Technical comprehension sufficient for evaluating engineering estimates, understanding prompts, pipelines, and evaluations, and identifying workarounds.
β’ Proven ability to quantify chaotic processes and assess business results.
β’ Capability to evaluate qualitative feedback and resolve reviewer discrepancies.
β’ Excellent writing skills and ability to collaborate with sales, operations, PMs, and engineers.
β’ Openness to tackle problems head-on and experiment directly.
β’ Must reside in a state where Airbnb, Inc. has a registered entity.
β’ Preferred: hands-on experience with Python and APIs.
β’ Preferred: experience in LLM evaluation.
β’ Preferred: background in experimentation and measurement.
β’ Preferred: experience in trust and safety, content moderation, customer support automation, annotation operations, or similar human-in-the-loop activities at scale.
β’ Preferred: experience in a two-sided marketplace, travel, or other operations-intensive industries.
β’ Eligibility for bonuses.
β’ Eligibility for equity.
β’ Access to benefits.
β’ Employee travel credits.
β’ Opportunities for occasional work at an Airbnb office or participation in offsite events, as coordinated with the manager.
β’ Reasonable accommodations provided throughout the recruitment process.
CVS Health Careers
Sourcegraph
MCA Connect
Lightly
Get handpicked remote jobs straight to your inbox weekly.