Principal Engineer – Agentic Security Operations

This is a fully remote position, open to applicants in California.

📋 Description

• Oversee the design and practical implementation of Agentic Security Operations capabilities within the SOC.

• Define and enhance the reference architecture for governed data services, agent functionalities, workload management, execution environments, and deterministic orchestration.

• Develop reusable functionalities that convert operational intent into tested, reviewable, and auditable security results.

• Set engineering standards for the creation, testing, registration, approval, release, monitoring, and retirement of agents.

• Promote detection-as-code methodologies, encompassing versioning, simulation, deployment, tuning, migration, and rollback.

• Construct governed knowledge and context services that include authoritative documentation, policies, operational history, citations, freshness data, and permission-aware retrieval.

• Design systems for workload identity, least-privilege access, short-lived authorizations, spending controls, approval gates, audit trails, rollback capabilities, and emergency shutdown protocols.

• Establish levels of autonomy for automated, human-approved, and analyst-led tasks.

• Implement agent evaluation and observability through replay, synthetic scenarios, comparisons, policy testing, traces, outcome verification, and regression testing.

• Assess quality, security, reliability, latency, cost, analyst acceptance, and operational impact.

• Create integrations across search, case management, workflow, communications, and response systems.

• Ensure that consequential response actions are deterministic, attributable, reversible, and subject to human oversight.

• Devise models for production reliability, continuity, recovery, and support.

• Lead proofs of concept and architectural evaluations based on measurable criteria.

• Collaborate with teams in enterprise architecture, infrastructure, data, risk, privacy, legal, and compliance.

• Mentor engineers, set technical standards, review designs, and influence the strategy for Threat Detection Engineering.

• Present technical decisions, risks, trade-offs, and investment suggestions to engineering and executive stakeholders.

• Collaborate across Threat Detection and Response groups to convert operational feedback into enhancements.

⛳️ Requirements

• Extensive experience in designing and operating enterprise security, software, data, or distributed systems on a large scale.

• Principal-level technical leadership with practical experience in architecture, prototyping, coding, testing, and production delivery.

• Hands-on experience in delivering production AI-assisted or agentic systems utilizing tools, memory, retrieval, planning, and structured workflows.

• Strong comprehension of security operations, including telemetry management, detection engineering, threat hunting, triage, investigation, response, and automation.

• Experience with high-volume streaming and batch data pipelines, object-based storage, elastic analytics, distributed queries, schema management, and data-quality controls.

• Proficiency in designing modular agents, reusable functionalities, workload management, local and managed execution, and human-in-the-loop workflows.

• Deep understanding of grounding, retrieval, source attribution, context freshness, permission inheritance, and safeguarding against untrusted or manipulated context.

• Experience in evaluating nondeterministic systems through versioned test sets, historical replay, synthetic scenarios, regression testing, and measurable outcome criteria.

• Strong software engineering practices, including source control, automated testing, continuous delivery, infrastructure automation, peer reviews, and rollback.

• Experience in designing secure APIs, event-driven integrations, workload identities, authorization boundaries, secrets management, and auditable action execution.

• Knowledge of AI security risks such as prompt injection, data poisoning, excessive agency, sensitive data exposure, insecure tool usage, and model or workflow drift.

• Strong production engineering experience encompassing observability, service objectives, capacity, failure isolation, continuity, recovery, and operational support.

• Ability to critically evaluate emerging capabilities and validate architectural assumptions through controlled testing.

• Excellent written and verbal communication skills.

• Capability to lead through influence and effectively collaborate across organizational and technical boundaries.

• Hands-on experience with cloud-scale data and analytics platforms that support security operations, including Databricks, Snowflake, or similar technologies.

• Experience in operating and migrating security analytics and detection workloads across modern platforms.

• Strong working knowledge of AWS, Azure, and GCP, including aspects like identity, networking, logging, data protection, and workload-security controls.

• A minimum of 10 years’ experience in cybersecurity or related fields.

• Bachelor’s degree in computer science, Information Technology, Cybersecurity, a related field, or equivalent work experience.

• Preferred certifications such as GASAE, GAIPS, GDSA, GCDA, CEH, OSCP, CISSP, etc.

• Preferred experience in modernizing centralized security analytics environments, knowledge graphs, entity resolution, semantic normalization, retrieval-augmented systems, multi-agent coordination, security-content migration, detection engineering, incident response, threat hunting, adversary emulation, security automation, AI governance, and internal security platforms.

🏝️ Benefits

• Medical insurance

• Dental insurance

• Vision insurance

• 401(k)

• PTO/paid sick leave

• Employee stock purchase plan

• Inclusive workplace culture

• Equal opportunity employment