Senior Software Engineer II – AI Engineering

This is a fully remote position, open to applicants in United States.

📋 Description

• Design and develop production-grade agents and workflows powered by LLM within Smartsheet.

• Make architectural decisions regarding system design, data pipelines, and deployment strategies.

• Create and refine prompts, RAG pipelines, and agent reasoning methodologies.

• Construct evaluation frameworks to assess accuracy, hallucination rates, and performance across different model versions.

• Implement and oversee comprehensive AI systems on AWS, GCP, or Azure.

• Monitor and enhance cloud-based AI systems while responding to incidents.

• Collaborate with Engineering, Product, and cross-functional teams to transform business requirements into technical AI solutions.

• Mentor engineers on applied ML best practices, modern AI development approaches, and production considerations.

• Establish best practices and standards for AI development, model governance, and responsible AI practices.

• Assess emerging AI/LLM tools, frameworks, and methodologies.

• Contribute to both legacy and greenfield full-stack applications.

• Deliver measurable, validated enhancements in the quality of SmartAssist agents.

• Increase evaluation coverage and create a repeatable quality improvement methodology.

⛳️ Requirements

• Over 8 years of software engineering experience, including at least 2 years working directly with LLMs in a production environment.

• In-depth, hands-on experience with prompt engineering and context engineering.

• Solid working knowledge of RAG architectures, including chunking strategies, embedding models, retrieval evaluation, and failure diagnosis.

• Experience in building or enhancing LLM evaluation frameworks, which includes designing scorers and utilizing golden datasets.

• Proficient in agent system design and understanding architectural trade-offs.

• Strong Python programming skills.

• Comfortable navigating data-intensive environments such as Databricks and Delta tables, or their equivalents.

• Ability to articulate complex quality findings to both technical and non-technical stakeholders.

• Strong judgment when working cross-functionally.

• A tendency to seek clarity in ambiguous situations.

• Legally authorized to work in the U.S. on an ongoing basis.

• BS or MS in Computer Science, a related discipline, or equivalent industry experience.

• Strong advantage: experience with MLflow or similar experiment tracking tools.

• Strong advantage: familiarity with CI-integrated evaluation pipelines.

• Strong advantage: experience with multi-agent orchestration frameworks.

• Strong advantage: previous work in an Applied AI or LLMOps role within a product company.

🏝️ Benefits

• Employer-subsidized medical, vision, and dental coverage for full-time employees.

• 401k Match to assist you in saving for your future (50% of your contribution up to the first 6% of your eligible pay).

• Monthly stipend to enhance your work and productivity.

• Flexible Time Away Program, along with Sick Time Off.

• U.S. employees receive automatic coverage under Smartsheet-sponsored life insurance, short-term, and long-term disability plans.

• U.S. employees are entitled to 12 paid holidays annually.

• Up to 24 weeks of Parental Leave.

• A Personal paid Volunteer Day to support community involvement.

• Opportunities for professional growth and development, including access to Udemy online courses.

• Company-funded perks, including a counseling membership, local retail discounts, and a personal Smartsheet account.

• Teleworking options from any registered location in the U.S. (role specific).

• Market-competitive incentive opportunities.