SRE Monitoring Platform Software Engineer

This is a fully remote position, open to applicants in California, +1 more state.

📋 Description

• Develop monitoring and automation components for the NeoCloud SRE platform.

• Create components such as collection-agent, metrics-store, logs-store, traces-store, profiles-store, enrichment-service, and collection-monitor.

• Write code for ingestion, query, and storage paths.

• Contribute to the alert-engine framework, alert correlation, and SLO framework.

• Implement and fine-tune default alert rules.

• Assist in the development of topology, cluster health, and OSS-SRE-tool collection plugins for Kubernetes, Slurm, Ray, Volcano, Kueue, and KubeRay.

• Aid in building components such as remediation, orchestration, workflow, inspection-probe, and job-scheduler.

• Instrument services with metrics, logs, and traces utilizing OpenTelemetry.

• Create dashboards and draft on-call runbooks.

• Develop unit, integration, and contract tests.

• Engage in chaos and soak testing.

• Deploy production code using GitOps and CI/CD pipelines.

• Operate the developed components under the guidance of senior engineers and participate in on-call duties as a shadow before transitioning to primary responsibility.

⛳️ Requirements

• 0-2 years of experience in software engineering; recent graduates with notable projects or internships are encouraged to apply.

• Strong foundation in Go, Python, Java, or Rust; Go is preferred.

• Ability to write clean, tested, and readable code while articulating design decisions.

• Knowledge of data structures, algorithms, concurrency, TCP/HTTP networking, and operating system concepts.

• Understanding of distributed systems concepts, including idempotency, retries, back-pressure, caching, and eventual consistency.

• Practical experience with Prometheus, Grafana, Loki, or similar tools; capability to write basic PromQL and instrument services.

• Familiarity with Linux, shell commands, system logs, and standard debugging tools.

• Basic knowledge of Kubernetes, including Pods, Services, and Deployments; experience running applications on Kubernetes.

• Understanding of Git and CI fundamentals, including branching, pull requests, and utilizing a CI pipeline.

• Consistent practice of writing unit and integration tests.

• Proficient in written and verbal English.

• A strong curiosity and eagerness to learn about GPU/AI infrastructure, AIOps, distributed systems, and production-scale observability.

🏝️ Benefits

• Competitive salary and performance-based bonuses.

• Comprehensive health, dental, and vision insurance.

• Flexible work hours and remote work options.

• Opportunities for professional development and continuous learning.

• Collaborative and inclusive company culture.