Senior AI Application Engineer

This is a fully remote position, open to applicants in United States.

📋 Description

• Take charge of the complete RAG pipeline, from data ingestion through to response generation.

• Design and implement indexing strategies for Azure AI Search, including schema design, scoring profiles, and performance enhancements.

• Develop and improve chunking and metadata strategies to enhance retrieval relevance and context quality.

• Optimize retrieval pipelines (hybrid search, vector search, filters, and reranking) for precision, speed, and cost-effectiveness.

• Create and manage prompt orchestration frameworks, involving dynamic prompt assembly and context injection.

• Establish robust grounding and citation mechanisms to ensure model outputs are traceable and verifiable.

• Design and implement AI agents and workflows, incorporating tool utilization, planning, and multi-step task execution.

• Integrate agents with both internal and external systems (APIs, databases, SaaS tools) to facilitate real-world actions.

• Define and execute evaluation frameworks (both offline and online) to assess retrieval quality, agent performance, and answer accuracy.

• Design and enforce safety measures (guardrails) to mitigate risks such as hallucinations and ensure policy compliance during generation and agent behavior.

• Collaborate with product, data, and platform teams to embed AI capabilities into applications.

• Monitor system performance and continuously improve based on user feedback and performance metrics.

• Work with business and technology leadership to identify, assess, and prioritize agentic AI opportunities across Flash's operations.

• Translate business challenges into AI system designs—scoping feasibility, defining data requirements, and overseeing delivery.

• Develop repeatable patterns and internal tools to accelerate future AI development within the organization.

• Contribute to the evolving governance and responsible use practices of AI at Flash.

⛳️ Requirements

• Proven experience in building or maintaining RAG systems in production settings.

• Familiarity with hybrid search (keyword + vector) and reranking models.

• Knowledge of evaluation frameworks (e.g., RAGAS, LLM-as-a-judge methodologies).

• Experience in integrating agents with enterprise tools such as CRM, ERP, and ticketing systems.

• Proven ability to optimize systems for scalability, latency, and cost efficiency.

• Strong understanding of search technologies (Azure AI Search, Elasticsearch, or equivalent).

• Hands-on expertise with LLMs, embeddings, and vector databases.

• Experience in designing and deploying AI agents or agentic systems.

• Demonstrated ability to integrate agents with enterprise systems, including ERP, CRM, and ticketing platforms.

• Understanding of information retrieval concepts (ranking, relevance, recall/precision trade-offs).

• Experience with prompt engineering and orchestration tools for LLMs.

• Familiarity with evaluation methodologies for generative AI and agent systems.

• Ability to convert ambiguous business problems into tangible AI system designs.

• Capacity to balance experimentation with production reliability.

• Hands-on experience in building AI agents and workflows, including tool usage, multi-step reasoning, and orchestration across APIs and services.

• Knowledge of designing systems that can plan, retrieve context, take actions, and operate within defined guardrails in production environments.

• Proven track record of building effective and reliable AI agents that accomplish real-world tasks accurately and safely.

• Ability to deliver measurable enhancements in retrieval relevance and overall system performance.

• Previous experience in creating robust guardrails and evaluation systems that identify issues early.

• Experience in building scalable and maintainable architecture integrated into core products.

• Self-motivated individual who thrives on taking ownership.

• Must possess the current and ongoing right to work in the United States of America without restrictions or expirations.

🏝️ Benefits

• Supportive and collaborative work environment.

• Opportunities for both technical and personal growth.

• Emphasis on work-life balance and professional development.