Data Engineer

This is a fully remote position, open to applicants in Malaysia.

📋 Description

• Design, build, test, deploy, and maintain scalable batch and streaming data pipelines for the Ascend Sandbox analytical platform.

• Create Python, PySpark, and SQL solutions to ingest, transform, and manage large-scale datasets.

• Develop data models and reusable, trusted datasets/data products for analytics, data science, and downstream applications.

• Engineer and optimize AWS-based data solutions utilizing Amazon S3, AWS Glue, Amazon Athena, Amazon Redshift, and Amazon EMR.

• Implement data lake and lakehouse engineering methodologies, including storage formats, partitioning, schema evolution, and storage/query optimization.

• Design resilient ETL/ELT patterns featuring incremental processing, dependency management, idempotency, retries, backfills, and failure recovery.

• Establish workflow orchestration and scheduling using Airflow/MWAA, AWS Step Functions, or AWS Glue Workflows.

• Create automated data quality controls and monitoring to ensure completeness, validity, reconciliation, schema compliance, freshness, pipeline health, and processing SLAs/SLOs.

• Enhance Spark workloads and distributed data processing for improved performance, scalability, and cost-effectiveness.

• Enforce data governance, metadata, lineage, cataloguing, security, and access-control requirements.

• Collaborate with clients, business users, data scientists, and technical teams to translate requirements into scalable solutions and integrations.

• Investigate and resolve production data and pipeline issues, conduct root-cause analysis, and implement reliability enhancements.

• Participate in code reviews, CI/CD, and automated deployment practices while upholding architecture, coding, testing, security, and quality standards.

⛳️ Requirements

• A Degree or Advanced Diploma in Computer Science, Data Engineering, Software Engineering, or a related field is required.

• Minimum of 6 years of experience in data engineering or development of data-intensive systems, including the design, construction, and production support of data pipelines.

• Strong hands-on experience with Python and PySpark for large-scale and distributed data processing.

• Proficient in SQL, including complex transformations, joins, window functions, query optimization, and performance tuning.

• Practical experience in designing and operating scalable ETL/ELT data pipelines, including incremental loads, retries, backfills, dependency management, and failure recovery.

• Experience in data modeling for analytical applications, including dimensional modeling, facts/dimensions, normalization/denormalization, and management of historical data.

• Hands-on experience with AWS data services such as Amazon S3, AWS Glue, Amazon Athena, Amazon Redshift, and Amazon EMR.

• Familiarity with data lake/lakehouse concepts, storage layout, partitioning, schema evolution, and efficient data formats like Parquet.

• Strong understanding of Spark performance engineering, including partitioning, shuffles, joins, data skew, parallelism, memory/resource optimization, and troubleshooting.

• Experience with pipeline orchestration and workflow scheduling using tools such as Airflow/MWAA, AWS Step Functions, AWS Glue Workflows, or similar alternatives.

• Experience in implementing automated data quality checks and data observability.

• Practical knowledge of data governance, metadata management, data cataloguing, lineage, and schema management.

• Practical understanding of AWS data security, including IAM/least privilege, encryption/KMS, secrets management, S3 security, and data access controls.

• Knowledge of cloud cost and performance optimization.

• Experience with Git-based workflows, branching strategies, code reviews, CI/CD pipelines, and automated deployment.

• Experience in writing automated unit, integration, and data pipeline tests for Python/PySpark transformations and data quality.

• Strong skills in testing, debugging, troubleshooting, and root-cause analysis within production data environments.

• Good understanding of Agile, SCRUM, SDLC, and software engineering practices.

• Excellent communication and stakeholder management skills.

• Strong analytical thinking, problem-solving abilities, teamwork, and eagerness to learn, with the capacity to work independently and deliver results under tight deadlines.

🏝️ Benefits

• Hybrid work arrangement: home and office.

• Inclusive paid leave.

• People-first and inclusive workplace.