Senior Linux Systems Administrator

This is a fully remote position, open to applicants in United States.

📋 Description

• Take charge of RapidFort's infrastructure, which encompasses bare-metal Linux servers in data centers, virtual machines, containers, networks, self-hosted development services, and AWS resources.

• Construct, configure, manage, and enhance Linux systems.

• Handle remotely managed physical servers, VMs, and containers, including storage, capacity planning, and hardware replacement.

• Oversee the operating-system lifecycle, including patch testing, staged deployment, upgrades, exceptions, and end-of-life migrations.

• Ensure consistent configuration and rectify configuration drift.

• Design, manage, and troubleshoot routing, traffic shaping, firewalls, DHCP, DNS, and overlay networks.

• Administer AWS resources, focusing primarily on EC2 and S3.

• Operate self-hosted GitLab, CI runners, build machines, and package mirrors.

• Troubleshoot production issues related to CPU, memory, disk I/O, filesystems, systemd, networking, and service dependencies.

• Utilize logs, metrics, and command-line tools to identify root causes and confirm fixes.

• Enhance monitoring and alerting systems.

• Lead incident responses, conduct incident reviews, and implement corrective actions.

• Collaborate with developers, IT personnel, data-center staff, and hardware vendors, ensuring clear findings and evidence during escalations.

• Develop and sustain Bash and Python automation scripts.

• Manage Ansible roles and playbooks.

• Store configuration and automation scripts in Git while utilizing merge requests, reviews, and automated deployment processes.

• Oversee access control and security protocols, including SSH keys, hardware security keys, network policies, least privilege practices, patching, vulnerability scanning, and supply-chain risk management.

• Execute and verify backups and restores.

• Organize maintenance activities with well-documented procedures, validation steps, and rollback plans.

• Maintain runbooks, configuration records, and troubleshooting documentation; review changes made by fellow engineers.

• Participate in a shared on-call rotation and adhere to scheduled maintenance windows.

⛳️ Requirements

• A minimum of 8 years of relevant hands-on experience managing Linux servers in production environments with root responsibility.

• Familiarity with physical hardware, not limited to cloud instances.

• Comprehensive understanding of Linux processes, services, users and permissions, package management, filesystems, and logging.

• Extensive experience with Debian and various other Linux distributions.

• Understanding of UEFI, GRUB, initramfs, kernel configuration and building, kernel modules including DKMS, and sysctl.

• Experience with cgroups v2, memory management, OOM killer, PSI, strace, perf, iostat, ss, and /proc.

• Proficient in systemd and sysvinit.

• Knowledge of TCP/IP, IPv4/IPv6, subnetting, bridges, VLANs, bonding, NAT, and connectivity troubleshooting.

• Experience with Linux routing, policy routing, traffic shaping using tc, nftables, iptables, DNS, and DHCP.

• Familiarity with WireGuard and NetBird or similar overlay networks.

• Skill in analyzing packet captures using tcpdump or Wireshark.

• In-depth knowledge of ZFS; experience with mdadm RAID, LVM, ext4, XFS, filesystem repair, SSD caching, and handling failing disks.

• Experience with KVM/QEMU or Xen, LXC, Docker, and sufficient Kubernetes knowledge to debug a cluster.

• Proficient in monitoring, alerting, backup, and recovery processes, including restore testing.

• Familiarity with AWS services including EC2, S3, IAM, EBS, and IMDSv2.

• Advanced Git and GitLab administration skills.

• Experience with GitOps, infrastructure as code, OpenTofu or Terraform, and configuration management.

• Competence in writing and maintaining Ansible roles and playbooks or using comparable tools.

• Knowledge of host hardening, least privilege, firewall design, SSH, SSH certificates, FIDO2 hardware keys, GPG, TLS certificates, secrets management, patch management, vulnerability scanning, and supply-chain risk.

• Expertise in Bash scripting and a good working knowledge of Python for systems automation.

• Proven track record of resolving complex technical challenges and executing corrective actions.

• Sound judgment regarding change risk, testing, rollback, and escalation procedures.

• Clear and precise written communication skills in English.

• Ability to work independently within a remote team environment.

• U.S. citizenship is required.

• Preferred: Experience with IPMI, iDRAC, iLO, RAID/HBA controllers, PCIe diagnostics, physical data-center operations, GPU/CUDA infrastructure, Proxmox/VMware, SAN/NAS, monitoring tools, Jenkins/build farms, backup tools, and SOC 2/ISO 27001 infrastructure, as well as familiarity with Debian derivatives and other cloud providers.

🏝️ Benefits

• Equity

• Benefits

• Shared on-call rotation

• Scheduled maintenance windows