Tech AI Data Delivery Manager

This is a fully remote position, open to applicants in India.

📋 Description

• Lead the design and execution of AI data initiatives.

• Create hybrid automation solutions that integrate deterministic tools with LLM agents.

• Determine which workflow steps should utilize rules versus models and orchestrate hand-offs.

• Establish and manage agents within Claude as well as in the native enterprise agentic environment.

• Configure task specifications, data and tool integrations, schedules, monitoring systems, and failure management protocols.

• Develop workflows for quality assurance, including guidelines, rubrics, and scorecards.

• Align automated checks with human reviewers and translate findings into quantifiable error categories.

• Oversee the end-to-end data collection and annotation processes, covering scoping, vendor management, workflows, tracking, and delivery.

• Create tools to assist project coordinators.

• Gather, clean, and analyze production and quality data.

• Generate reports on throughput, rework, error rates, and contributor performance.

• Conduct automation testing prior to release and establish continuous auditing processes.

• Document specifications, tickets, and runbooks for coordinators and engineers.

⛳️ Requirements

• Proficient in JavaScript and Python, with the capability to develop and maintain production-level automation.

• Practical experience in building agents within Claude, including projects, skills, connectors, and scheduling tasks.

• Competence in constructing agents in the native enterprise agentic environment and interfacing various APIs with the native infrastructure.

• Experience in LLM evaluation, encompassing prompt design, rubric formulation, and calibration against human judgment.

• Background in data analytics, particularly within the data collection or annotation sectors.

• Proven track record in managing projects involving external vendors or contributors alongside multiple stakeholders.

• Capability to make and take ownership of decisions even with incomplete information.

🏝️ Benefits

• Opportunity for remote work.

• Full-time, fixed-term employment for a duration of 1 year, with the potential for extension.