Senior Applied Scientist

This is a fully remote position, open to applicants in Colombia.

📋 Description

• Design and conduct experiments aimed at assessing and enhancing the quality of applications and agents based on LLM technology.

• Establish and manage evaluation methodologies, benchmarks, regression suites, scoring techniques, acceptance criteria, and release-gate signals.

• Collaborate with platform engineering to develop automated comparison infrastructure and set up alerts for drift or regression.

• Create AI-driven tools and functionalities for Caseware’s comprehensive agent builder.

• Develop capabilities for synthetic data generation and evaluation construction.

• Transform domain procedures into task-plus-verifier frameworks.

• Propel applied science in agentic memory, defining and validating criteria for memory promotion and demotion.

• Design self-learning and self-improving systems that enhance performance both offline and online.

• Ensure compliance with legal and contractual memory criteria by working alongside Security, Legal, and Domain Subject Matter Experts (SMEs).

• Create evaluations that clearly demonstrate improvements in agent quality with the use of the platform.

• Convert advancements in GenAI into practical and maintainable capabilities.

• At the staff level, guide technical direction through Requests for Comments (RFCs) and design reviews while mentoring scientists and engineers.

⛳️ Requirements

• 6+ years of experience for senior roles to 8+ years for staff positions in applied science, machine learning, research, or data-intensive engineering.

• At least 2 years of experience working with production GenAI or LLM-based systems.

• Demonstrated experience in designing and constructing evaluation frameworks for ML or LLM systems, encompassing metrics, benchmarks, scoring methods, and rigorous experimental design.

• A strong experimental mindset with the capability to derive defensible conclusions from complex, real-world data.

• Proficiency in building and managing production tools and services, preferably on AWS.

• In-depth understanding of GenAI trade-offs, including quality, latency, cost, reliability, and safety.

• Solid foundation in probability and statistics, particularly in experimental design, significance testing, and reasoning under uncertainty.

• Excellent English communication and collaboration skills.

• Ability to thrive in fast-paced environments characterized by ambiguity and changing requirements.

• A PhD or MS in a quantitative field is preferred; equivalent industry experience will also be considered.

• Previous experience in machine learning.

• Familiarity with AI guardrails, governance, or safety measures.

• Experience with distributed, SaaS, cloud-native, multi-tenant platforms at scale.

• Knowledge of Infrastructure as Code tools, including CDK, CloudFormation, or Terraform.

• Background in regulated or compliance-heavy fields.

• Understanding of financial audit, accounting, or assurance workflows.

• Proficiency in TypeScript, NestJS, Python, AWS EKS, AWS Lambda, AWS Bedrock, AWS AgentCore, AWS OpenSearch, S3 Vectors, AWS Textract, DynamoDB, S3, LangFuse, LangSmith, LangChain, LangGraph, GitHub, GitHub Actions, and Nx Monorepo.

🏝️ Benefits

• Indefinite contract with all legal benefits.

• Prepaid medical coverage.

• Life insurance and funeral assistance.

• Internet allowance.

• Stipend for home office expenses.

• Competitive salary that exceeds the market average.

• Fully remote work setup and a fantastic work-life balance.

• 5 Personal Time Off days each year.

• Sick leave topped up to 100% of salary, covered by the employer from Day 3 to Day 90.

• Recognition Award that provides additional paid time off based on years of service.

• Vacation upgrades beginning at 5 years of service.

• Opportunity to work for a rapidly growing global SaaS leader.

• A culture that fosters independence, innovation, trust, and accountability.

• Open environment for creativity, innovation, and strategic future planning.

• Mentorship from highly experienced professionals.

• Budget allocated for training.

• AI-first workplace with modern tools, automation, and AI-driven methodologies.

• Flexible work options, remote opportunities, and generous time-off policies.

• Recognition programs, performance bonuses, and pathways for career development.

• Participation in international projects and collaboration with a diverse, global team.