Senior Applied Scientist

This is a fully remote position, open to applicants in Colombia.

📋 Description

• Design and conduct experiments to assess and enhance the quality of applications and agents based on LLM.

• Take ownership of evaluation methodologies, encompassing benchmarks, regression suites, scoring techniques, acceptance thresholds, and both offline and online performance comparisons.

• Collaborate with platform engineering to establish automated evaluation infrastructures, drift and regression notifications, and release gates.

• Develop AI-driven functionalities for Caseware’s comprehensive agent builder.

• Create capabilities for synthetic data generation and evaluation builder.

• Transform domain procedures into task-plus-verifier frameworks.

• Utilize statistical methods and audit-domain expertise to enhance agentic memory.

• Establish and verify criteria for memory promotion and demotion.

• Propel advancements in self-learning and self-improving systems.

• Ensure that memory criteria comply with legal and contractual obligations in collaboration with Security, Legal, and Domain SMEs.

• Design evaluations demonstrating improvements in agent quality with platform usage.

• Convert advances in GenAI into practical, sustainable capabilities.

• Shape technical direction through RFCs and design reviews at the staff level.

• Mentor scientists and engineers.

⛳️ Requirements

• 6+ years of professional experience for senior roles to 8+ years for staff roles in applied science, machine learning, research, or data-intensive engineering.

• Minimum of 2 years experience with production GenAI or LLM-based systems.

• Proven expertise in designing and constructing evaluation frameworks for ML or LLM systems.

• Familiarity with metrics, benchmarks, scoring methodologies, and rigorous experimental design.

• A strong experimental mindset and the capability to derive defensible conclusions from noisy, real-world data.

• Proficient in building and managing production tooling and services, preferably on AWS.

• Solid understanding of GenAI trade-offs, including quality, latency, cost, reliability, and safety.

• Strong foundation in probability and statistics, covering experimental design and significance testing.

• Excellent English communication and collaboration abilities.

• Comfortable working in dynamic environments characterized by ambiguity and shifting requirements.

• PhD or MS in a quantitative discipline is preferred; equivalent industry experience is also considered.

• Previous experience in machine learning.

• Familiarity with AI guardrails, governance, or safety protocols.

• Experience with distributed, SaaS, cloud-native, multi-tenant platforms operating at scale.

• Knowledge in Infrastructure as Code, including CDK, CloudFormation, or Terraform.

• Experience in regulated or compliance-intensive sectors.

• Familiarity with financial audit, accounting, or assurance processes.

🏝️ Benefits

• Indefinite term contract with all legal benefits.

• Prepaid medical coverage.

• Life insurance and funeral assistance.

• Internet allowance.

• Home office stipend.

• Competitive compensation exceeding the market average.

• Fully remote work environment promoting an excellent work-life balance.

• 5 Personal Time Off days annually.

• Sick Leave Top-up covering 100% of salary paid by the employer from Day 3 to 90.

• Recognition Award, providing additional paid time off in acknowledgment of service duration.

• Enhanced vacation benefits starting after 5 years of service.

• Opportunity to work with a rapidly growing global SaaS leader.

• A culture that fosters independence, innovation, trust, and accountability.

• A creative and innovative environment for strategizing future initiatives.

• Access to mentorship from highly experienced professionals.

• Budget allocated for training and development.

• AI-first environment utilizing modern tools, automation, and AI-driven workflows.

• Flexible work arrangements and generous time-off policies.

• Recognition programs, performance bonuses, and pathways for career advancement.

• Engagement in international projects and collaboration with a diverse, global team.