Applied AI Engineer

This is a fully remote position, open to applicants in Canada.

📋 Description

• Develop and sustain regression suites for GIQL query generation, ensuring the accuracy of writeback and tool selection.

• Establish quantifiable quality benchmarks for each action type.

• Conduct structured assessments prior to model updates or prompt modifications, providing recommendations on whether to proceed.

• Monitor quality trends over time to identify degradation before it impacts customers.

• Instrument Gateway actions to gather data on tokens, cache behavior, model performance, latency, and tenant specifics.

• Set up and uphold cost-per-action baselines.

• Achieve cost reduction through techniques such as prompt compression, caching, context pruning, and optimizing routing for smaller models.

• Detect costly customer behaviors, query shapes, failure instances, and retry patterns.

• Track model releases across various providers and assess their performance against Velixo workloads.

• Maintain an understanding of the price/performance frontier.

• Advise on the timing of migrations, whether to wait, or the feasibility of running models concurrently.

• Provide unit cost data to inform credit pricing and plan allowances.

• Analyze the margin impact of suggested pricing modifications alongside the VP of Finance.

• Highlight product decisions that could influence the cost trajectory.

• Deliver key milestones for the first 90 days, including Gateway instrumentation, evaluation suite, cost dashboard, and cost-reduction strategy.

⛳️ Requirements

• Extensive experience with LLM APIs in production settings, beyond demonstrations or notebooks.

• Proficiency in creating evaluation harnesses, with an understanding of the pitfalls of naive approaches.

• Practical experience with tracing and observability platforms like Langfuse.

• Familiarity with evaluation frameworks such as promptfoo.

• Knowledge of token accounting, context window management, and caching strategies.

• Strong analytical skills and comfort with data analysis.

• Ability to articulate trade-offs clearly for a non-technical audience.

• Proficient in Python or TypeScript.

• Preferred: Experience with MCP or similar tool-calling frameworks.

• Preferred: Background in ERP, accounting, or financial systems.

• Preferred: Previous exposure to usage-based or credit-based pricing models.

• Preferred: Experience in a small company setting where you can set your own priorities.

🏝️ Benefits

• Fully remote position.

• Equal opportunity employer dedicated to diversity and inclusion.