Principal AI Engineer

This is a fully remote position, open to applicants in United Kingdom.

📋 Description

• Define the architecture and technical roadmap for LLM and agentic AI systems in the areas of Detection and Response and Exposure and Remediation.

• Lead the model strategy, which includes selecting between frontier or open-source models and making decisions regarding fine-tuning, distillation, and post-training processes.

• Fine-tune, distill, and post-train open-source LLMs utilizing security data, overseeing the process from curation through training, evaluation, and deployment.

• Establish engineering standards for context engineering, inference optimization, and evaluation across various teams.

• Optimize LLM inference at scale through techniques such as speculative decoding, KV-cache management, continuous batching, quantization, and custom kernels.

• Implement evaluations as the critical release gate for every AI system deployed.

• Lead design reviews and address complex technical trade-offs.

• Prototype and develop essential model, training, and inference code in a hands-on manner.

• Mentor Staff and Senior engineers to enhance their skills.

• Influence product and platform investment strategies.

• Create AI systems that drive cybersecurity use cases, including alert triage, threat investigation, automated response, vulnerability prioritization, attack surface risk assessment, and guided remediation.

⛳️ Requirements

• A minimum of 12 years of experience in software or ML engineering.

• At least 2 years of experience building production LLM or agentic AI systems.

• A PhD in Computer Science, Machine Learning, AI, or a related field is preferred, or equivalent expertise demonstrated through industry experience and technical contributions.

• Proven track record of developing scalable, production-grade AI systems utilized across various teams or products.

• In-depth understanding of transformer architecture, attention mechanisms, tokenization, positional encodings, model training, and serving.

• Capability to implement core LLM components from the ground up.

• Expert-level knowledge in PyTorch, including custom model code, training loops, mixed precision, and distributed training techniques such as FSDP and DeepSpeed.

• Hands-on experience with full and parameter-efficient fine-tuning methods, including LoRA and QLoRA.

• Familiarity with distillation methods and preference or RL-based techniques such as DPO and GRPO.

• Experience in context engineering, incorporating retrieval, memory, and context management for long-running agent workflows.

• Comprehensive knowledge of inference optimization techniques, including prefill and decode, KV-cache management, continuous batching, speculative decoding, FlashAttention, and quantization.

• Experience in serving AI workloads at scale using tools like vLLM, SGLang, or TensorRT-LLM.

• Ability to define evaluation strategies and benchmarks for LLM and agent systems.

• Strong foundation in predictive AI and classical ML concepts.

• Solid understanding of distributed systems and Python programming.

• Proven ownership of significant technical outcomes across multiple teams.

• Excellent communication skills for articulating technical trade-offs.

• Ability to foster cross-functional partnerships.

• Skill in aligning customer needs and business priorities with technical decisions.

• Ability to embody the company’s core values.

• Preferred: experience in pre-training or training LLMs from scratch.

• Preferred: experience in writing GPU kernels using CUDA or Triton and profiling GPU workloads.

• Preferred: experience in the security domain, such as detection engineering, SOC/MDR, or vulnerability management.

• Preferred: experience running AI workloads on AWS at scale.

• Preferred: publications, patents, or contributions to open-source AI projects.

🏝️ Benefits

• Flexible/remote work arrangement as indicated by the location options and #LI-Remote.

• Opportunities for mentoring and technical leadership.

• Chance to influence product and platform investments.

• Collaborative and multi-dimensional team environment.

• Career development opportunities through engaging work in cybersecurity and AI.