Senior Backend Engineer

This is a fully remote position, open to applicants in United States.

📋 Description

• Deploy AI solutions into production.

• Develop tools utilizing LLM agents for tasks such as planning, function and tool invocation, multi-step workflows, and establishing guardrails for grant discovery, application drafting, and research support.

• Transform prototypes into robust, observable services that include defined SLAs, rollback and fallback strategies, as well as cost and latency considerations.

• Implement evaluation and observability to ensure our AI remains grounded, safe, and cost-efficient.

• Construct reliable backends.

• Produce high-quality, well-tested code across backend systems and data pipelines that facilitate retrieval and evaluation.

• Contribute to reliability practices including alerts, dashboards, and incident response protocols.

• Collaborate effectively to elevate standards.

• Work alongside Product, Design, and GTM teams on scoping, user experience, and measurement initiatives.

• Conduct experiments (A/B testing, canaries), analyze outcomes, and iterate based on findings.

• Enhance engineering quality through clear, maintainable code, comprehensive tests, documentation, and thoughtful code reviews.

⛳️ Requirements

• Over 7 years of experience in building and deploying production backend systems using Python (FastAPI, Celery, or similar), taking features from prototype to production while implementing real reliability practices such as testing, observability, and rollback strategies.

• Proven hands-on experience in developing LLM features in production, including tool and function invocation, multi-step agent workflows, and the guardrails and evaluations that ensure they are grounded, safe, and cost-effective. This is central to the role.

• Strong foundational knowledge in data management: SQL, schema design, and constructing pipelines that support retrieval and evaluation processes.

• Ability to thrive in a dynamic, fast-paced startup environment with a focus on ownership, speed, quality, and simplicity.

• Nice to have:

• Familiarity with TypeScript and Node, as well as knowledge of Ruby on Rails (our core platform) or a willingness to learn.

• Experience with AWS or GCP, Docker, CI/CD, and observability tools (logs, metrics, traces).

• In-depth understanding of RAG techniques, including document ingestion, chunking and windowing, embeddings, hybrid search (keyword and vector), re-ranking, and grounded citations.

• Experience with re-rankers and cross-encoders, tuning hybrid retrieval, or developing search and recommendation systems.

• Evaluation mindset: skilled in designing evaluation suites (RAG/QA, extraction, summarization) using both automated and human-in-the-loop methods, with familiarity with frameworks such as Ragas, DeepEval, or OpenAI Evals.

• Experience with orchestration frameworks like LangChain, LangGraph, LlamaIndex, Semantic Kernel, or custom orchestration solutions.

🏝️ Benefits

• Comprehensive health, dental, and vision insurance fully covered for employees (50% for dependents).

• Generous paid time off (PTO), including parental leave.

• 401(k) retirement plan.

• Company-provided laptop and home-office stipend.

• Bi-annual company retreats.

• Instrumentl is rapidly evolving, offering continuous challenges and opportunities for growth.