Senior Data Engineer

This is a fully remote position, open to applicants in Jordan.

📋 Description

• Design, implement, and oversee intricate DAGs and SubDAGs within Apache Airflow.

• Utilize Operators, macros, connections, variables, and XCom to ensure efficient task execution.

• Conduct unit testing for workflows in Airflow.

• Perform file operations and facilitate data transfers between HDFS and S3.

• Manage Hive external tables, update table DDL, and execute partition operations.

• Run Hadoop commands and navigate through HDFS.

• Employ Hive on Tez for enhanced query performance.

• Utilize AWS command line tools for operations involving S3 files.

• Implement and manage secrets using Ansible Vault and AWS Secrets Manager.

• Develop and refine data processing tasks utilizing PySpark.

• Use Livy for executing Spark jobs.

• Import and export data between Microsoft SQL Server and HDFS using Sqoop.

• Apply dimensional modeling techniques, define facts and dimensions, and manage surrogate keys.

• Implement Git branching strategies effectively.

• Initiate and review Merge Requests in GitLab.

• Use SSIS for data migration workflows.

• Model data within Snowflake using both external and internal tables.

• Work with views and materialized views in Snowflake.

• Collaborate with cross-functional teams to deliver high-quality data solutions.

• Focus on ensuring data quality, optimizing performance, enhancing security, and maintaining compliance.

⛳️ Requirements

• Bachelor's degree in Computer Science, Information Technology, or a related discipline.

• Proficiency in Python versions 2.7 and 3.6 or higher.

• Experience in unit testing and test-driven development methodologies.

• Familiarity with tools and techniques for secrets management.

• Knowledge of data modeling principles and best practices.

• Strong understanding of Git version control and workflows in GitLab.

• Awareness or knowledge of IT security best practices in alignment with ISO, SOC, or similar standards.

• Experience utilizing SSIS for data migration tasks.

• Familiarity with Snowflake, including external/internal tables, views, and materialized views.

• Familiarity with Hive on Tez.

• Experience working with AWS command line tools and S3 operations.

• Knowledge of Livy for executing Spark jobs.

• Proficient in Apache Airflow, PySpark, Hive, AWS, Snowflake, HDFS, Sqoop, Microsoft SQL Server, and SSIS.

• Excellent problem-solving capabilities and communication skills.

🏝️ Benefits

• Competitive long-term compensation package that includes salary and performance-based bonuses.

• A remote-first organization offering flexibility in work arrangements.

• Opportunities for professional development through both technical and non-technical training programs.

• Global exposure through products that involve collaboration with international teams and clients.

• Attendance at both virtual and in-person international technology conferences.

• Nursery reimbursement benefit available.

• Aspire Wellness Program included.

• Experience in an IT environment that adheres to stringent ISO / SOC security and compliance standards.