Tech AI Data Delivery Manager

This is a fully remote position, open to applicants in India.

📋 Description

• Design and execute AI data initiatives.

• Create hybrid automation by integrating deterministic tools with LLM agents.

• Determine which workflow steps should utilize rules versus models and plan the hand-offs accordingly.

• Establish and operate agents within Claude and the native enterprise agentic environment.

• Set up task specifications, establish data and tool connections, manage schedules, oversee monitoring, and handle failures.

• Develop quality assurance workflows, guidelines, rubrics, and scorecards.

• Calibrate automated checks in relation to human reviewers and create measurable error categories.

• Manage end-to-end data collection and annotation projects, encompassing scoping, vendor coordination, workflow management, tracking, and delivery.

• Create tools for project coordinators.

• Gather, clean, and analyze both production and quality data.

• Generate reports on throughput, rework, error rates, and contributor performance.

• Conduct testing on automation prior to release and establish continuous auditing workflows.

• Document specifications, tickets, and runbooks for coordinators and engineers.

⛳️ Requirements

• Proficient in JavaScript and Python at an intermediate level, capable of building and maintaining production automation.

• Practical experience in building agents within Claude, including managing projects, skills, connectors, and scheduled tasks.

• Competence in developing agents in the native enterprise agentic environment and integrating APIs with existing infrastructure.

• Experience in LLM evaluation, including the design of prompts and rubrics, along with calibration against human judgment.

• Background in data analytics, particularly in the data collection or annotation sector.

• Proven experience in managing projects involving external vendors or contributors and coordinating with multiple stakeholders.

• Capability to make and take ownership of decisions even when faced with incomplete information.

🏝️ Benefits

• Opportunity for remote work.

• Full-time, fixed-term employment for one year with the possibility of extension.