Principal AI Engineer

atRapid7RemoteUS flagCaliforniaFull-timeAI EngineerLead$221k – $299k/year

Posted 6 days ago

This is a fully remote position, open to applicants in California.

📋 Description

• Define the architecture and technical roadmap for LLM and agentic AI systems encompassing Detection and Response as well as Exposure and Remediation.

• Lead the model strategy, determining the appropriate times to utilize frontier or open-source models and when to engage in fine-tuning, distillation, or post-training.

• Fine-tune, distill, and post-train open-source LLMs using security data, managing the entire process from data curation through training, evaluation, and deployment.

• Establish engineering standards for context engineering, inference optimization, and evaluation across various teams.

• Optimize LLM inference on a large scale through speculative decoding, KV-cache management, continuous batching, quantization, and custom kernels.

• Implement evaluations as a prerequisite for the release of every AI system delivered.

• Conduct design reviews and address the most challenging technical trade-offs.

• Prototype and develop essential model, training, and inference code hands-on.

• Mentor Staff and Senior engineers.

• Influence product and platform investment strategies.

• Help define and guide the technical trajectory of LLM and agentic AI systems.

• Enhance engineering standards and create greenfield AI solutions for global cybersecurity clients.


⛳️ Requirements

• 12+ years of experience in software or ML engineering.

• 2+ years dedicated to building production LLM or agentic AI systems.

• PhD in Computer Science, Machine Learning, AI, or a related field preferred, or equivalent expertise demonstrated through industry experience and technical contributions.

• Proven history of constructing scalable, production-grade AI systems utilized across various teams or products.

• Strong track record of making a meaningful impact through technical leadership, mentoring, and establishing shared platforms and standards across organizations.

• In-depth understanding of transformer architecture, attention mechanisms, tokenization, positional encodings, and model training and deployment.

• Capability to implement core LLM components from the ground up.

• Expert-level proficiency in PyTorch, including custom model code, training loops, mixed precision, and distributed training techniques such as FSDP and DeepSpeed.

• Practical experience with full and parameter-efficient fine-tuning methods, including LoRA and QLoRA.

• Familiarity with distillation techniques and preference or RL-based methods such as DPO and GRPO.

• Expertise in context engineering, covering retrieval, memory, and context management for long-running agent workflows.

• Comprehensive knowledge of prefill and decode processes, KV-cache management, continuous batching, speculative decoding, FlashAttention, quantization, and large-scale serving.

• Experience with vLLM, SGLang, or TensorRT-LLM.

• Ability to develop evaluation strategies and benchmarks for LLM and agent systems.

• Strong foundations in predictive AI and classical ML.

• Solid understanding of distributed systems and Python fundamentals.

• Proven ownership of significant technical results across multiple teams.

• Proficient in identifying and addressing risks while driving complex challenges to resolution.

• Excellent communication skills for articulating technical trade-offs to engineering, product, and leadership audiences.

• Ability to forge cross-functional partnerships across Engineering, Product, and other teams.

• Skillful in aligning customer needs and business priorities with technical decisions.

• Experience pre-training or training LLMs from the ground up is preferred.

• Familiarity with writing GPU kernels using CUDA or Triton, along with profiling GPU workloads is preferred.

• Experience in the security domain is preferred.

• Experience executing AI workloads on AWS at scale is preferred.

• Publications, patents, or contributions to open-source projects in AI are preferred.


🏝️ Benefits

• Variable/incentive compensation (excluded from stated salary range).

• Equity (where applicable/eligible).

• Benefits (where applicable/eligible).

People also viewed

Extend6 hours ago

Senior AI Software Engineer, AI Enablement

US flagUnited States OnlyFull-timeAI Engineer$155k – $170k/year
ApplyView job
OhioHealth7 hours ago

Associate Data and AI Engineer

US flagOhio OnlyFull-timeAI Engineer
ApplyView job
Capital One7 hours ago

Staff AI Engineer

US flagUnited States OnlyFull-timeAI Engineer$244.7k – $279.2k/year
ApplyView job
Software Mind7 hours ago

Tech Lead – AI Prototyping, Python, Full Stack

CR flagCosta Rica OnlyFull-timeAI Engineer
ApplyView job
Terac7 hours ago

Software Developer – AI Assistant Integrations

US flagUnited States OnlyFreelanceAI Engineer
ApplyView job
Homebound7 hours ago

Senior/Staff AI Engineer

US flagUnited States OnlyFull-timeAI Engineer$160k – $220k/year
ApplyView job

Never miss a great job!

Get handpicked remote jobs straight to your inbox weekly.

Trusted by 7,400+ designers