Software Engineer – AI Systems, Go

This is a fully remote position, open to applicants in Arizona, +16 more states.

📋 Description

• Design, develop, test, deploy, and manage production-quality software and backend services, primarily utilizing Go.

• Architect and create AI agents and agent-based systems employing tools, APIs, state, and context for intricate multi-step workflows.

• Design robust workflows that can checkpoint, resume, retry, recover, and continue safely after any failures or interruptions.

• Define responsibilities across models, agents, tools, APIs, data sources, services, and deterministic application logic.

• Engineer integrations across various AI model providers, including routing, failover, rate-limit management, health monitoring, and degradation strategies.

• Establish safeguards such as structured-output validation, error handling, retries, quality gates, evidence validation, and automated recovery.

• Formulate testing strategies for non-deterministic AI behavior using fixtures, recorded/replayed interactions, regression suites, evaluation harnesses, and quality baselines.

• Enhance observability through tracing, metrics, logging, evaluation data, and replayable execution histories.

• Design human-in-the-loop workflows that involve review, approval, intervention, and modifications to workflow states.

• Construct and maintain APIs, backend services, relational data models, job-processing infrastructure, and supporting application components.

• Create secure methods for ingesting and processing user-supplied documents and external content.

• Improve reliability, latency, throughput, scalability, output quality, and execution costs.

• Develop reusable patterns and shared components for agents, model providers, tools, workflows, and output types.

• Take ownership of technical challenges from investigation and architecture through implementation, deployment, monitoring, troubleshooting, and operation in production.

• Collaborate with product stakeholders and domain experts to convert requirements into measurable behaviors and technical solutions.

• Contribute to architectural decisions and engineering standards for AI-powered applications across the University.

• Establish dependable production AI workflows, enhance measurability and quality, minimize latency and costs, and implement foundational AI engineering practices.

⛳️ Requirements

• Significant professional experience in developing and operating production software, with strong expertise in Go (Golang) or proven proficiency in another backend language, with the capacity to quickly become productive in Go.

• Practical experience in delivering an LLM-backed application, AI agent, or agent-based system into production and providing ongoing support post-deployment.

• Comprehensive understanding of AI application architecture, including the interactions between models, agents, tools, APIs, data sources, services, and application logic.

• Proven knowledge in distributed-systems engineering, encompassing concurrency, asynchronous processing, queues, idempotency, retries, partial failures, state management, and recovery.

• Experience in designing systems that maintain reliability even when individual services, providers, or stages in the workflow fail.

• Demonstrated discipline in testing systems that involve non-deterministic behavior.

• Strong experience in designing and using HTTP APIs.

• Familiarity with relational databases, SQL, and persistent application states.

• Experience in building, deploying, monitoring, and troubleshooting backend services within production environments.

• Understanding of software architecture, testing, debugging, observability, and production engineering practices.

• Ability to independently manage ambiguous technical issues from investigation to production implementation.

• Strong analytical judgment with the capacity to balance reliability, quality, performance, complexity, and cost.

• Applicants must be legally authorized to work in the United States.

• Must reside in an eligible state at the time of employment.

• Official transcripts are required prior to hiring.

• Degrees obtained from institutions outside the United States must be evaluated by a recognized credential evaluation service.

• Must be capable of performing each essential duty satisfactorily and meet the specified physical requirements.

• Preferred: experience with AI agent frameworks, RAG, embeddings, vector databases, semantic search, LLM evaluation and observability, judge models, structured-output validation, human-in-the-loop workflows, media pipelines, event-driven architectures, containers, cloud infrastructure, CI/CD, prompt injection, authorization boundaries, data isolation, and security considerations.

🏝️ Benefits

• Health Care Plan (Medical, Dental & Vision)

• Retirement Plan (401k)

• Exciting university events

• Seasonal motivational health and wellness challenges

• Work/Life Balance initiatives

• Onsite wellness program / Staff Chiropractor

• Life Insurance (Basic, Voluntary & AD&D)

• Paid Time Off (Vacation, Sick & Public Holidays)

• Family Leave (Maternity, Paternity)

• Reasonable accommodations may be made to enable individuals with disabilities to perform essential functions.