WEKA Data Platform Administrator

This is a fully remote position, open to applicants in United States.

📋 Description

• Oversee the daily operations of the WEKA environment, which includes monitoring, maintenance, upgrades, capacity management, configuration, and audits.

• Develop and manage file systems, implement object-storage tiering, create snapshots, set quotas, and configure protocols such as POSIX, NFS, SMB, and S3.

• Ensure a well-documented, healthy, and recoverable operational state is maintained.

• Implement approved changes while confirming scope, classifying risks, validating outcomes, documenting rollbacks, and recording results.

• Monitor for drift, capacity issues, and anomalies, and recommend corrective actions.

• Collaborate with tenants and platform teams to gain insights into storage usage for training, inference, checkpointing, data loading, and scheduling activities.

• Optimize filesystem layout, tiering strategies, snapshot policies, quotas, and quality of service (QoS) for each tenant.

• Validate GPU node clients, BlueField-3 and SR-IOV networking configurations, mount options, cgroups, kubelet CPU policies, WEKA Kubernetes operators, and CSI drivers.

• Facilitate the onboarding of tenants and clients using standardized, verified procedures.

• Set performance baselines and conduct benchmarks and tests.

• Identify scalability limits regarding client counts, process ceilings, NAT configurations, and tenant network routing.

• Develop and maintain operating runbooks for customers, detailing procedures, access, validation, rollback, and recovery processes.

• Automate repetitive tasks using Python, Bash, and Ansible.

• Prepare handover documents and backup briefs.

• Articulate procedures and decisions clearly to customer operators.

• Initiate and update escalation cases, detailing conditions, impacts, evidence, completed actions, and requested next steps.

• Coordinate with Technical Services, GRE, R&D, cloud service providers, network teams, and third-party partners.

• Contribute operational metrics, trends, and risks to weekly status reports and quarterly reviews.

⛳️ Requirements

• More than 10 years of experience administering large-scale enterprise infrastructure, with a strong emphasis on storage solutions (file, object, or block).

• Extensive Linux administration expertise, particularly with RHEL-family and Ubuntu systems, in distributed environments.

• Practical experience with high-speed networking technologies such as Ethernet, InfiniBand, RDMA, DNS, ACLs, and fundamental TCP/IP concepts.

• Familiarity with Kubernetes and containerization, specifically regarding persistent storage, CSI, and operators.

• Experience managing GPU, HPC, or extensive cloud infrastructures for external clients or multiple tenants.

• Proficient in at least one major cloud platform: AWS, Azure, Google Cloud, or Oracle Cloud.

• Strong skills in Python, Bash, or Ansible.

• Experience with monitoring and alerting systems, including Prometheus, Grafana, and log and metrics analysis tools.

• Familiarity with ticketing systems such as Jira and major incident management processes.

• Excellent written and verbal communication abilities.

• Strong technical writing skills.

• Prior experience with WEKA or a similar parallel or distributed file system is a plus.

• Experience operating Slurm or Kubernetes-based AI/ML platforms is a plus.

• Familiarity with NVIDIA GPU cluster networking and DGX-class systems is a plus.

• Experience working collaboratively between support, product, or engineering teams is a plus.

• Knowledge of Confluence, Slack, and other collaboration tools is a plus.

🏝️ Benefits

• Competitive salary and performance-based bonuses.

• Comprehensive health and wellness benefits.

• Opportunities for professional development and continuous learning.

• Flexible work arrangements and a supportive team environment.