ETL Modernization Developer

This is a fully remote position, open to applicants in United States.

📋 Description

• Evaluate current ETL/ELT processes, databases, and data warehouses for their readiness for migration.

• Redesign legacy IBM DataStage jobs utilizing Databricks Spark on AWS.

• Create, test, and implement scalable cloud-based data pipelines.

• Conduct parity checks, unit testing, functional testing, UAT, regression testing, and performance testing.

• Verify schemas, data quality, pipeline efficiency, and accuracy of migrations.

• Establish CI/CD processes for automated development, testing, and deployment.

• Assist with migration cutover, production deployment, and post-migration support.

• Coordinate the redeployment of applications and manage end-to-end testing.

• Retire legacy DataStage jobs and associated artifacts.

• Create technical documentation and materials for operational readiness.

• Facilitate knowledge transfer and training sessions for engineering and business teams.

⛳️ Requirements

• Experience in IBM DataStage development.

• Knowledge of ETL/ELT migration and modernization.

• Proficiency in Databricks and Apache Spark on AWS.

• Skills in PySpark, Python, SQL, and Shell scripting.

• Familiarity with Delta Lake, Unity Catalog, and Photon.

• Experience with AWS Glue, Lambda, and Redshift (preferred).

• Understanding of CI/CD and automated ETL testing.

• Proficiency in PyTest and XML/JSON parsing.

• Strong hands-on experience in developing and maintaining IBM DataStage jobs.

• Demonstrated experience in modernizing large-scale ETL/ELT and data warehouse workloads.

• Experience in building Databricks and Spark pipelines within an AWS environment.

• Advanced programming capabilities in PySpark, Python, SQL, and Shell scripting.

• Familiarity with Delta Lake, Unity Catalog, and Databricks Photon.

• Experience using GitLab or Azure DevOps for CI/CD.

• Knowledge of automated testing frameworks and methods for ETL validation.

• Experience in supporting production cutovers, stabilization, and application teams.

• Work authorization: Green Card and US Citizens Only.

• Experience with JIRA and Agile delivery (preferred).

• Knowledge of data governance and cloud security standards (preferred).

• Experience in creating operational documentation and providing technical training (preferred).

🏝️ Benefits

• W2 employment.

• Remote work arrangement.

• 12-month contract duration.