Principal Software Engineer – Agentic SOC

This is a fully remote position, open to applicants in United States.

📋 Description

• Spearhead the creation of Huntress’s Agentic SOC, utilizing LLM-based agents to comprehensively investigate security signals.

• Oversee the architectural design and technical strategy for agent-led investigations.

• Establish workflows for signal-to-investigation processes, manage investigation states, and define agent tools and outputs, including verdicts, timelines, scope, and evidence.

• Develop agents that emulate the investigative behaviors of proficient analysts.

• Create LLM-driven preprocessing methods for analyst tools and data sources.

• Design trust boundaries, ensure tenant isolation, and implement auditability for environments influenced by attackers and customer data.

• Define and uphold standards for evaluating investigation accuracy, reasoning, and documentation.

• Determine which system actions can be automated and which necessitate human oversight based on performance metrics.

• Design systems for reliability, low latency, cost-efficiency, throughput, and correctness at scale.

• Guarantee that investigations and actions are both traceable and explainable.

• Develop efficient transitions between agents and SOC analysts.

• Prototype and mitigate risks associated with uncertain technical strategies.

• Advocate for the technical perspective of the Agentic SOC to SOC leadership, engineering teams, product managers, and executives.

• Foster consensus on technical decisions and project scope.

• Conduct code reviews, collaborate on challenging issues, mentor engineers, and enhance team practices and tools.

• Collaborate with the engineering manager to support team well-being and provide constructive feedback.

• Utilize AI coding tools responsibly to expedite the development process.

⛳️ Requirements

• Over 15 years of experience in developing complex software products, with considerable time spent as the technical lead on production systems.

• Practical experience in designing, deploying, and managing LLM-based agents in production, including tool utilization, context management, structured output, guardrails, and evaluation processes.

• Knowledge of the potential failure modes of LLM-based agents.

• Experience in developing evaluations for non-deterministic systems, including labeled datasets, offline assessments, and regression gates.

• Proven track record of measuring system quality over time.

• Experience in constructing systems that manage untrusted input, enforce multi-tenant isolation, and generate auditable records.

• Background in building systems that make automated decisions with tangible consequences.

• Discernment regarding what tasks should be automated versus those that should be handled by individuals.

• Interest in understanding the thought processes and workflows of security analysts, along with a willingness to learn from their expertise.

• Extensive backend proficiency in distributed systems, queues, durable workflows, and concurrency.

• Strong capabilities in one or more backend programming languages, with a quick learning curve for new languages.

• Expectation and ability to achieve fluency in Ruby on Rails.

• Familiarity with AWS, Azure, or other public cloud platforms.

• Experience with Postgres and Redis or comparable data storage solutions.

• Experience using AI coding tools such as Claude Code.

• BS or MS in Computer Science or Engineering, or equivalent professional experience.

• Remote work limited to candidates within the United States.

• Bonus: experience in or with a SOC, incident response, threat hunting, or detection engineering.

• Bonus: knowledge of endpoint and identity telemetry and attacker methodologies.

• Bonus: experience in attacking or adversarially testing ML or LLM systems.

• Bonus: familiarity with Ruby on Rails.

🏝️ Benefits

• Fully remote work environment.

• Generous paid time off policy, encompassing vacation, sick leave, and paid holidays.

• 12 weeks of paid parental leave.

• Highly competitive and comprehensive medical, dental, and vision benefits packages.

• 401(k) plan with a 5% contribution from the company, independent of employee contributions.

• Life and disability insurance coverage.

• Stock options available for all full-time employees.

• One-time $500 reimbursement for setting up or enhancing a home office.

• Annual budget for education and professional development support.

• $75 USD/month for digital reimbursement.

• Access to the BetterUp platform for coaching and personal and professional growth opportunities.

• Bonus and equity included in the base compensation package.