Big Data Infrastructure Engineer

This is a fully remote position, open to applicants in Singapore.

📋 Description

• Optimization of Spark / YARN jobs: Assess job resource utilization and pinpoint inefficiencies (such as CPU idling, memory waste, timeouts, and issues with small files).

• Development of job profiling systems: Implement job classification, establish resource baseline modeling, and conduct historical trend analysis.

• Generate optimization reports and encourage business owners to adopt improvements.

• AI-Driven Automation & Tooling: Proficiently utilize AI coding tools (such as Cursor, Copilot, Claude, etc.) to enhance development speed and tool delivery.

• Engage in cost optimization initiatives for EMR / S3: Evaluate high-cost jobs and storage, identify waste, and motivate owners to take corrective actions.

• Create automated operational scripts: Conduct scheduled health checks, establish anomaly alerting, and automate data governance policy deployments.

• Assist in developing internal SaaS tools to streamline and productize repetitive manual tasks.

• Monitoring & Alerting: Enhance and maintain existing monitoring systems (including Prometheus metrics, alerts, and log analysis).

• Contribute to the development of health monitoring for Flink / Spark jobs.

• Support capacity alerting and governance for disk, S3, and other storage layers.

⛳️ Requirements

• Currently pursuing a Bachelor's or Master's degree in Computer Science, Software Engineering, or a related discipline.

• Skilled in Python — capable of independently writing clean and maintainable scripts and tools.

• Must be proficient in AI coding tools (such as Cursor, Copilot, ChatGPT, Claude, etc.) for development and troubleshooting — this is a crucial requirement; please evaluate your experience with AI tools before applying.

• Comfortable working with Linux: executing programs on servers, analyzing logs, and debugging issues.

• Experience with Spark / Hive / Flink, even if only through coursework (preferred).

• Basic knowledge of AWS services (S3, EMR, Athena) (preferred).

• Experience writing Shell scripts or configuring crontab scheduled tasks (preferred).

• Familiarity with Kafka, Prometheus, or any messaging queue / monitoring systems (preferred).

• Completed a full project utilizing AI tools and can clearly articulate how and where they were applied (preferred).

• Any practical project experience: contributions to open source, participation in competitions, lab projects, etc. (preferred).

• Bilingual in English and Mandarin is a plus to facilitate collaboration with overseas partners and stakeholders (preferred).

🏝️ Benefits

• Competitive salary alongside company benefits.

• Flexible work-from-home arrangements (specific details may vary based on the operational needs of the business team).