Technical Lead, Splunk Security Core AI Agentic SOC

This is a fully remote position, open to applicants in California.

📋 Description

• Design and implement components for the shared Security AI runtime, orchestration, evaluation, and applied-quality systems.

• Take ownership of significant features from initial requirements and design through to implementation, testing, integration, deployment, and ongoing enhancements.

• Contribute to the architectural and technical direction through the use of prototypes, evaluations, telemetry, and production evidence.

• Create reusable capabilities for agent execution, governed tool usage, context and memory management, shared APIs, telemetry, evaluation, trace analysis, and experimentation.

• Develop agents and associated systems for grounded reasoning, planning, delegation, human oversight, traceability, and safe failure handling.

• Enhance AI behavior utilizing golden datasets, both offline and online evaluations, production traces, trajectory analysis, regression gates, and experiments involving models, prompts, or tools.

• Design systems compatible with cloud, on-premises, sovereign-cloud, and air-gapped environments.

• Implement identity propagation, authorization, tenant isolation, policy enforcement, privacy measures, auditability, and permission-aware actions.

• Integrate and foster the adoption of shared Cisco and Splunk technologies.

• Utilize contracts, adapters, and configurations to accommodate product and deployment variances.

• Leverage AI tools throughout the software development lifecycle, ensuring human validation and adherence to security, privacy, and intellectual-property standards.

• Balance factors such as AI quality, latency, performance, cost, reliability, security, operability, and delivery.

• Diagnose failures across models, prompts, tools, data, orchestration, services, infrastructure, and product integrations.

• Write secure, maintainable code while contributing to architecture, testing, reviews, and technical documentation.

• Collaborate with product, domain, deployment, quality, release, and support teams.

• Mentor engineers to enhance engineering practices and facilitate knowledge sharing.

⛳️ Requirements

• A minimum of 5 years of experience in developing, testing, and/or operating production software.

• Proficiency in production software engineering using Python and/or Golang.

• A Bachelor’s degree with 8 years of relevant experience, a Master’s degree with 6 years of related experience, or a PhD with 3 years of pertinent experience.

• Experience in developing and operating production AI, agentic, or distributed systems.

• Familiarity with agent runtime, orchestration, shared APIs, context and memory management, evaluation frameworks, experimentation systems, or production tracing.

• Capability to design, implement, debug, and maintain complex production-grade systems.

• Knowledge of LLM and agentic-system concepts, including grounding, retrieval, planning, delegation, tool usage, human oversight, behavioral evaluation, and regression analysis.

• Experience in delivering containerized or cloud-native software across public cloud, on-premises, sovereign cloud, or air-gapped environments.

• Understanding of SOC, SIEM, detections, incidents, investigations, response, identity, authorization, policy, privacy, tenancy, and auditability.

• Experience in evaluating, integrating, or adopting shared technologies within large organizations.

• Commitment to responsible AI-assisted engineering with a focus on validation, code quality, security, privacy, intellectual property, and human accountability.

• Authorization for remote work in the U.S. is required.

🏝️ Benefits

• Medical, dental, and vision insurance.

• 401(k) plan with a Cisco matching contribution.

• Paid parental leave.

• Short- and long-term disability coverage.

• Basic life insurance.

• Potential grants of Cisco restricted stock units, contingent upon continued employment and vesting periods.

• 10 paid holidays each calendar year.

• 1 floating holiday for non-exempt employees.

• Paid day off for birthdays.

• Paid shutdown during the year-end holiday.

• 4 paid personal wellness days.

• 16 days of paid vacation each calendar year for non-exempt employees.

• Flexible vacation time-off program with no set limit for eligible exempt employees.

• 80 hours of sick leave provided on the hire date and each January 1st.

• Up to 80 unused sick-time hours can be carried over.

• Additional paid leave for critical or emergency family matters.

• Optional 10 paid volunteer days per calendar year.

• Annual bonuses for non-sales positions, subject to Cisco policies.

• Performance-based incentive compensation for sales-plan employees.

• Opportunities for professional development and growth.