Senior Software Engineer II – Applied AI

This is a fully remote position, open to applicants in Washington.

📋 Description

• Lead the establishment of the AI Platform Foundation by designing and managing the essential infrastructure that underpins all Smartsheet AI functionalities. Your focus will be on creating a robust, multi-tenant system that minimizes friction for internal teams, enabling them to effortlessly deploy reliable and scalable AI features.

• Standardize the AI Developer Path by crafting high-level abstractions and 'Golden Path' APIs that democratize AI development within Smartsheet. By shielding product teams from infrastructure complexities, you will facilitate the rapid delivery of intelligent features while ensuring safety and consistency at scale.

• Engineer AI Trust & Safety Systems by implementing critical monitoring and quality assurance frameworks that safeguard Smartsheet customers. Through the development of rigorous evaluation pipelines, you will guarantee that every AI-driven feature meets the stringent standards for safety, data privacy, and deterministic performance expected by our enterprise partners.

• Drive the technical strategy by collaborating with principal engineers to outline the technical roadmap for Smartsheet’s AI infrastructure, making architectural decisions that will influence our approach to AI development for years to come.

⛳️ Requirements

• Over 8 years of experience in software engineering, with a minimum of 2 years of direct experience working with LLMs in production environments.

• Extensive, hands-on experience with prompt engineering and context engineering, demonstrating an understanding of how model behavior shifts based on framing, structure, and input design.

• Strong familiarity with RAG architectures, including chunking strategies, embedding models, retrieval evaluation, and failure diagnosis.

• Experience in building or enhancing LLM evaluation frameworks, having designed scorers and worked with golden datasets while carefully considering evaluation standards.

• Proficient in Python, with comfort in data-intensive environments such as Databricks, Delta tables, or similar technologies.

• Capable of articulating complex quality findings (both written and verbal) to technical and non-technical stakeholders, effectively explaining issues, their significance, and necessary actions without losing engagement.

• Strong cross-functional judgment, knowing when to escalate issues, when to independently resolve them, and how to build credibility across engineering, product, and AI platform teams.

• A tendency towards clarity in ambiguous situations; when faced with unclear failure modes and real trade-offs, you provide structure and a clear perspective rather than waiting for consensus.

• Strong Plus: Previous experience in an Applied AI or LLMOps platform within a product company.

• Familiarity with Kubernetes (EKS/GKE): The industry standard for AI, including skills in managing GPU scheduling, auto-scaling based on token throughput, and utilizing tools like Karpenter for cost-effective node provisioning.

• Proficient in Infrastructure as Code (IaC): Using Terraform, Pulumi, or AWS CDK to provision Vector Databases, SQS queues, and S3 buckets.

• Experience with Vector Databases: Expertise in managing and optimizing Pinecone, Milvus, Weaviate, or Databricks Vector Search.

• Knowledge of AI Gateways: Building or configuring proxies (such as LiteLLM or Kong AI Gateway) to manage rate-limiting, PII masking, and cost-tracking.

• Experience with LLM Observability: Setting up tracing tools like Langfuse, LangSmith, or MLflow to monitor 'Time to First Token' (TTFT) and trace hallucination issues.

• Implementing Model-Based Evaluations: Developing automated scoring systems (such as RAGAS or DeepEval) that utilize an 'LLM-as-a-Judge' to assess production outputs.

🏝️ Benefits

• Employer-subsidized medical, vision, and dental coverage for full-time employees.

• 401k Match to assist you in saving for your future (50% of your contribution up to the first 6% of your eligible pay).

• Monthly stipend to enhance your work and productivity.

• Flexible Time Away Program, along with Sick Time Off.

• US employees are automatically enrolled in Smartsheet-sponsored life insurance, short-term, and long-term disability plans.

• US employees enjoy 12 paid holidays annually.

• Up to 24 weeks of Parental Leave.

• A personal paid Volunteer Day to contribute to our community.

• Opportunities for professional growth and development, including access to Udemy online courses.

• Company-funded perks, including a counseling membership, local retail discounts, and your own personal Smartsheet account.

• Telecommuting options available from any registered location in the U.S. (role-specific).