Lead, Big Data Administrator

This is a fully remote position, open to applicants in Texas.

📋 Description

• Deploy and oversee large-scale distributed systems.

• Ensure optimal system performance, availability, and scalability.

• Supervise the installation, configuration, and maintenance of Hadoop ecosystems, including HDFS, Yarn, Hive, and Spark.

• Manage Kafka clusters effectively.

• Implement strategies for high availability and disaster recovery in Hadoop and Kafka environments.

• Conduct resource allocation, cluster optimization, performance tuning, and monitoring.

• Orchestrate data pipelines and integrate Kafka producers, consumers, and downstream processing frameworks.

• Safeguard data security through encryption, access control, and auditing mechanisms.

• Troubleshoot and resolve issues related to cluster health, data loss, Kafka offsets, connectivity, performance degradation, and system failures.

• Automate repetitive tasks utilizing Ansible and shell scripting.

• Collaborate closely with development, data engineering, and analytics teams.

• Provide strategic insights on system upgrades and architectural enhancements.

• Stay informed about emerging technologies in big data and streaming.

⛳️ Requirements

• Bachelor's degree in Computer Applications or a related discipline, with eight years of progressive, post-baccalaureate experience in the job offered or eight years of progressive, post-baccalaureate experience in any occupation where the requisite experience was acquired.

• Expertise in HDFS, Yarn, Hive, Spark, and Kafka clusters.

• Experience using ZooKeeper for coordination and synchronization within Hadoop and Kafka clusters.

• Familiarity with Kerberos, LDAP integration, SSL/TLS, Hadoop KMS, Ranger, Atlas, or custom auditing solutions.

• Proficient in using Prometheus and Grafana for cluster monitoring, metrics visualization, and alerts.

• Ability to diagnose and resolve intricate cluster connectivity, performance, and system failure issues.

• Experience in fine-tuning cluster configurations and optimizing job performance through memory, I/O, and replication parameters.

• Proficiency in Bash and Python for automation tasks.

• Telecommuting is available from anywhere in the US.

🏝️ Benefits

• Flexible time off (FTO).

• 14 paid holidays.

• Paid sick time.

• Parental/new child leave.

• Childcare & elder care assistance.

• Adoption assistance.

• Comprehensive health coverage.

• 401(k) plan.

• Tuition assistance.

• Commuter benefits.

• Opportunities for professional development.

• Employee recognition programs.

• Charitable donation matching.

• Health coaching and counseling services.

• Medical coverage.

• Dental coverage.

• Vision coverage.

• Disability coverage.

• Paid time off.