
Principal AI Engineer
Posted 6 days ago

Posted 6 days ago
This is a fully remote position, open to applicants in California.
• Define the architecture and technical roadmap for LLM and agentic AI systems encompassing Detection and Response as well as Exposure and Remediation.
• Lead the model strategy, determining the appropriate times to utilize frontier or open-source models and when to engage in fine-tuning, distillation, or post-training.
• Fine-tune, distill, and post-train open-source LLMs using security data, managing the entire process from data curation through training, evaluation, and deployment.
• Establish engineering standards for context engineering, inference optimization, and evaluation across various teams.
• Optimize LLM inference on a large scale through speculative decoding, KV-cache management, continuous batching, quantization, and custom kernels.
• Implement evaluations as a prerequisite for the release of every AI system delivered.
• Conduct design reviews and address the most challenging technical trade-offs.
• Prototype and develop essential model, training, and inference code hands-on.
• Mentor Staff and Senior engineers.
• Influence product and platform investment strategies.
• Help define and guide the technical trajectory of LLM and agentic AI systems.
• Enhance engineering standards and create greenfield AI solutions for global cybersecurity clients.
• 12+ years of experience in software or ML engineering.
• 2+ years dedicated to building production LLM or agentic AI systems.
• PhD in Computer Science, Machine Learning, AI, or a related field preferred, or equivalent expertise demonstrated through industry experience and technical contributions.
• Proven history of constructing scalable, production-grade AI systems utilized across various teams or products.
• Strong track record of making a meaningful impact through technical leadership, mentoring, and establishing shared platforms and standards across organizations.
• In-depth understanding of transformer architecture, attention mechanisms, tokenization, positional encodings, and model training and deployment.
• Capability to implement core LLM components from the ground up.
• Expert-level proficiency in PyTorch, including custom model code, training loops, mixed precision, and distributed training techniques such as FSDP and DeepSpeed.
• Practical experience with full and parameter-efficient fine-tuning methods, including LoRA and QLoRA.
• Familiarity with distillation techniques and preference or RL-based methods such as DPO and GRPO.
• Expertise in context engineering, covering retrieval, memory, and context management for long-running agent workflows.
• Comprehensive knowledge of prefill and decode processes, KV-cache management, continuous batching, speculative decoding, FlashAttention, quantization, and large-scale serving.
• Experience with vLLM, SGLang, or TensorRT-LLM.
• Ability to develop evaluation strategies and benchmarks for LLM and agent systems.
• Strong foundations in predictive AI and classical ML.
• Solid understanding of distributed systems and Python fundamentals.
• Proven ownership of significant technical results across multiple teams.
• Proficient in identifying and addressing risks while driving complex challenges to resolution.
• Excellent communication skills for articulating technical trade-offs to engineering, product, and leadership audiences.
• Ability to forge cross-functional partnerships across Engineering, Product, and other teams.
• Skillful in aligning customer needs and business priorities with technical decisions.
• Experience pre-training or training LLMs from the ground up is preferred.
• Familiarity with writing GPU kernels using CUDA or Triton, along with profiling GPU workloads is preferred.
• Experience in the security domain is preferred.
• Experience executing AI workloads on AWS at scale is preferred.
• Publications, patents, or contributions to open-source projects in AI are preferred.
• Variable/incentive compensation (excluded from stated salary range).
• Equity (where applicable/eligible).
• Benefits (where applicable/eligible).
Extend
OhioHealth
Capital One
Software Mind
Get handpicked remote jobs straight to your inbox weekly.