Senior AI Developer

This is a fully remote position, open to applicants in United States.

📋 Description

• Design, develop, and oversee ClickUp's AI evaluation framework, which includes golden datasets, LLM-as-judge, automated scoring systems, regression detection, and quality tracking across AI functionalities.

• Create and refine the foundational agent harness and the AI that excels in evaluations, collaborating with product and engineering teams to establish quality benchmarks.

• Establish self-improvement mechanisms where agents modify other agents or prompts, re-evaluate, and iterate automatically.

• Benchmark new models and manage model selection, routing, and fallback strategies among providers while optimizing quality, latency, and cost.

• Design, architect, and implement scalable AI platform services for the deployment, orchestration, and lifecycle management of LLMs and other AI models.

• Develop and maintain robust APIs and backend systems that integrate AI-powered features into ClickUp's core platform.

• Create infrastructure for model serving, monitoring, logging, and automated evaluation to ensure reliable, high-performance production AI services.

• Promote best practices in AI privacy, security, and compliance, including data anonymization, secure data management, and adherence to regulations.

• Work collaboratively across product, frontend, and data teams to deliver dependable, user-focused AI experiences.

⛳️ Requirements

• Over 5 years of software engineering experience, including at least 2 years focused on building production LLM or ML systems.

• Practical experience in developing LLM evaluation systems in production, encompassing golden datasets, LLM-as-judge, judge calibration, and both offline and online evaluations linked to actual product quality.

• Strong experimental design and statistical analysis skills, involving A/B testing, offline vs. online metrics, and distinguishing genuine regressions from noisy or variable results.

• A proven history of implementing LLMs and AI models in real-world, customer-facing product features.

• Demonstrated experience in integrating and managing multiple LLMs, including benchmarking, routing, and weighing cost, latency, and quality factors.

• In-depth expertise in backend engineering, distributed systems, and cloud-native technologies such as Kubernetes, Docker, and AWS/GCP/Azure.

• Proficient programming skills in Python, Go, TypeScript, or similar backend and AI platform programming languages.

• Familiarity with orchestration frameworks and workflow engines like LangGraph, Airflow, Ray, or similar technologies.

• Understanding of search and retrieval technologies and their application in AI-driven solutions.

• Excellent interpersonal and communication skills, with a proven ability to work effectively in cross-functional teams.

• A strong enthusiasm for keeping pace with advancements in AI and leveraging new technologies to address real-world challenges at scale.

• Nice to have: experience in data science, applied science, or machine learning engineering.

• Nice to have: familiarity with evaluation tools such as Braintrust, LangSmith, Weave, Arize, or proprietary frameworks.

• Nice to have: experience with model routing, AI gateways, or inference optimization.

• Must reside in or be legally permitted to work in the United States; visa sponsorship is not guaranteed and is subject to business requirements.

🏝️ Benefits

• Equity

• 401k

• Health, Dental, and Vision insurance

• Spending accounts

• Life & Disability

• Paid parental leave

• Flexible paid time off

• Enhanced employee assistance program

• Employee wellness stipend

• Professional development stipend