Big Data Lead

This is a fully remote position, open to applicants in India.

📋 Description

• Integrate a hands-on Subject Matter Expert (SME) to utilize big data technologies for addressing intricate data challenges.

• Allocate nearly half of the time to practical coding tasks.

• Engage in extensive text data processing, create event-driven data pipelines, and perform in-memory computations while optimizing for CPU core, network I/O, and disk I/O.

• Leverage cloud-native services available in AWS and GCP.

⛳️ Requirements

• Over 7 years of practical experience in Software Development with an emphasis on big data and extensive data pipelines.

• At least 3 years of experience in developing services and pipelines using Python.

• Proficiency with various data processing systems, including streaming, event, and batch processing (such as Spark and Hadoop/MapReduce).

• Familiarity with at least one NoSQL database like MongoDB, Elasticsearch, or HBase.

• Knowledge of data models, sharding, and data location strategies for distributed data stores within large-scale, high-throughput, and high-availability environments, particularly in relation to non-structured text data processing.

• Experience in managing scalable and highly available systems using AWS or GCP.

• Preferred: Experience with Docker and Kubernetes.

• Exposure to Continuous Integration/Continuous Deployment (CI/CD).

• Understanding of Crawling and Scraping techniques.

🏝️ Benefits

• Fully remote work opportunity.

• Birthday leave policy.

• Flexible remote working arrangements.