Enterprise Data Solutions Engineer

📋 Description

• Design and sustain ETL/ELT pipelines utilizing PySpark and Python within the Databricks environment

• Create and enhance data transformations across the Bronze, Silver, and Gold layers of the medallion architecture

• Produce well-organized, comprehensible, and testable code adhering to team standards and incorporating review feedback

• Diagnose and rectify data pipeline failures, escalating complex issues to senior engineers when necessary

• Contribute to the documentation of pipelines, including data flow diagrams and inline code annotations

• Develop and optimize T-SQL queries, stored procedures, and views in SQL Server

• Assist with data loads and transformations that support the SQL Server data warehouse

• Help identify and resolve query performance issues under the guidance of senior engineers

• Apply fundamental indexing knowledge and comprehend schema design patterns utilized by the team

• Create unit tests and data quality checks for pipelines that are developed or modified

• Implement established data quality frameworks and tools that align with team standards

• Investigate and escalate data anomalies or quality concerns identified in pipelines

• Engage in code reviews — both receiving and providing constructive feedback as skills progress

• Collaborate with team members during sprint planning, standups, and retrospectives

• Clearly communicate blockers and progress to your manager and team

• Work alongside data analysts and other stakeholders to comprehend requirements for assigned tasks

⛳️ Requirements

• 2–4 years of professional experience in data engineering, complemented by a Bachelor's degree

• Strong proficiency in SQL, with the ability to write multi-table joins, window functions, and subqueries

• Practical knowledge of Python for data processing activities

• Familiarity with Databricks, Apache Spark, or PySpark in a professional or academic context

• Understanding of cloud data storage concepts; experience with Azure or ADLS is an advantage

• Knowledge of ETL/ELT principles and data pipeline architecture

• Comfortable using Git for version control and participating in a collaborative branching workflow

🏝️ Benefits

• Remote work

• Opportunities for professional development