Senior AI Developer

This is a fully remote position, open to applicants in United States.

📋 Description

• Design, develop, and take ownership of ClickUp's AI evaluation system, which encompasses golden datasets, LLM-as-judge, automated scorers, regression detection, and quality monitoring across AI functionalities.

• Create and enhance the core agent harness and the AI to perform effectively in evaluations, collaborating with product and engineering teams to establish quality benchmarks.

• Establish self-improvement mechanisms whereby agents adjust other agents or prompts, re-evaluate, and iterate autonomously.

• Benchmark newly released models and manage model selection, routing, and fallback strategies across providers, ensuring a balance between quality, latency, and cost.

• Architect, design, and implement scalable AI platform services for deployment, orchestration, and lifecycle management of LLMs and other AI models.

• Develop and maintain robust APIs and backend systems to integrate AI-enhanced features into ClickUp's core platform.

• Create infrastructure for model serving, monitoring, logging, and automated evaluation to guarantee reliable, high-performance production AI services.

• Promote best practices in AI privacy, security, and compliance, including data anonymization, secure data handling, and adherence to regulations.

• Work collaboratively with product, frontend, and data teams to provide reliable, user-centric AI experiences.

⛳️ Requirements

• A minimum of 5 years of software engineering experience, including over 2 years in building production LLM or ML systems.

• Practical experience in developing LLM evaluation systems in production, covering golden datasets, LLM-as-judge, judge calibration, and both offline and online evaluations linked to actual product quality.

• Strong skills in experiment design and statistical analysis, including A/B testing, offline versus online metrics, and differentiation of genuine regressions from noisy or variable results.

• Demonstrated success in applying LLMs and AI models to real-world, customer-facing product features.

• Proven experience in integrating and managing multiple LLMs, including benchmarking, routing, and balancing cost, latency, and quality trade-offs.

• In-depth knowledge of backend engineering, distributed systems, and cloud-native technologies such as Kubernetes, Docker, and AWS/GCP/Azure.

• Proficient programming skills in Python, Go, TypeScript, or similar backend and AI platform languages.

• Familiarity with orchestration frameworks and workflow engines such as LangGraph, Airflow, Ray, or similar.

• Understanding of search and retrieval technologies and their incorporation into AI-driven applications.

• Exceptional collaboration and communication abilities, with a history of effective teamwork in cross-functional groups.

• A strong passion for remaining at the cutting edge of AI and leveraging new technologies to address real-world challenges at scale.

• Visa sponsorship is not guaranteed and will depend on business requirements for the specific role.

🏝️ Benefits

• Equity

• 401k

• Health, Dental, and Vision insurance

• Spending accounts

• Life & Disability

• Paid parental leave

• Flexible paid time off

• Enhanced employee assistance program

• Employee wellness stipend

• Professional development stipend