SRE Monitoring Platform Software Engineer – Early Career, Temporary

This is a fully remote position, open to applicants in California, +1 more state.

📋 Description

• Contribute to the NeoCloud SRE platform, which is a multi-region system managing a GPU rental fleet across various data centers.

• Construct well-defined components from the design phase through to production code with guidance from senior engineers.

• Deploy code via GitOps and CI/CD release pipelines while adhering to Plugin Framework standards and defined SLOs.

• Develop components for collection and storage, including ingestion, query, storage-path, enrichment, and collection-monitor code.

• Assist in the creation of alerting, correlation, and SLO frameworks; implement and refine default alert rules.

• Create topology and cluster-health services along with collection plugins for Kubernetes, Slurm, Ray, Volcano, Kueue, and KubeRay.

• Aid in the development of remediation actuators, orchestration/workflow components, inspection probes, and job schedulers.

• Instrument services with metrics, logs, and traces utilizing OpenTelemetry.

• Design dashboards and draft actionable on-call runbooks.

• Write unit, integration, and contract tests for the components delivered.

• Engage in chaos and soak tests led by senior engineers.

• Operate the developed systems under supervision and participate in on-call duties as a shadow before assuming primary responsibilities.

• Aim to independently deliver components and manage a sub-context within a 12-month timeframe.

⛳️ Requirements

• 0–2 years of experience in software engineering; recent graduates with strong projects or internships are encouraged to apply.

• Strong foundational knowledge in at least one programming language: Go (preferred), Python, Java, or Rust.

• Capability to write clean, tested, and readable code while articulating design decisions.

• Understanding of data structures, algorithms, concurrency, TCP/HTTP networking, and operating system principles.

• Familiarity with distributed systems concepts including idempotency, retries, back-pressure, caching, and eventual consistency.

• Practical experience with monitoring and observability tools such as Prometheus, Grafana, or Loki; proficiency in writing basic PromQL and instrumenting a service.

• Knowledge of Linux, shell scripting, system logs, and common debugging tools.

• Fundamentals of Kubernetes, including Pods, Services, and Deployments; experience running applications on Kubernetes.

• Understanding of Git and CI fundamentals, including branching, pull requests, and CI pipeline usage.

• Regular practice of writing unit and integration tests.

• Strong verbal and written communication skills in English.

• Curiosity and a desire to learn about GPU/AI infrastructure, AIOps, distributed systems, and observability.

• Nice-to-have: internship or project experience in monitoring/observability, telemetry pipelines, or platform/SRE tooling.

• Nice-to-have: exposure to GPU/AI infrastructure such as DCGM, InfiniBand/RoCE, Kubernetes GPU Operator, Slurm, or Ray.

• Nice-to-have: familiarity with AIOps or ML-adjacent tools.

• Nice-to-have: contributions to open-source observability or cloud-native projects.

🏝️ Benefits

• Mentorship from senior and principal engineers.

• Guided involvement in on-call, starting as a shadow before taking on primary responsibilities.

• Exposure to production-scale observability, GPU/AI infrastructure, AIOps, distributed systems, and cloud-native technologies.

• Opportunity to contribute to greenfield systems utilizing an established Plugin Framework, GitOps pipeline, and SLO framework.

• Equal employment opportunities.