Senior Data Engineer

This is a fully remote position, open to applicants in Canada.

📋 Description

• Architect and execute comprehensive data solutions, including data lakes, data warehouses, and analytics platforms.

• Establish scalable, secure, and efficient strategies for data integration and transformation.

• Convert business needs into technical solutions that facilitate analytics, reporting, and AI projects.

• Create data models, ETL/ELT pipelines, and frameworks for managing both structured and unstructured data.

• Offer technical guidance and mentorship to data engineers and developers.

• Advocate for best practices in data governance and management.

• Ensure adherence to data governance, security, and privacy regulations across various platforms.

• Enhance current data architectures and processes to boost performance and reliability.

• Keep abreast of cloud data services and new technologies such as Databricks, Snowflake, Azure Synapse, and AWS Redshift.

• Provide clients with expert advice on architectural decisions and best practices for data modernization.

• Work closely with business stakeholders, data engineers, and analytics teams to ensure solutions align with client objectives.

⛳️ Requirements

• Over 5 years of experience in data architecture, data engineering, or designing analytics solutions.

• Practical experience with data lake and warehouse technologies, including Databricks, Snowflake, Redshift, or Synapse.

• Comprehensive knowledge of data modeling, data integration, and ETL/ELT design principles.

• Proficiency in SQL and familiarity with one or more programming languages such as PySpark/Python or Scala.

• Experience with intricate data transformations and optimization processes within Spark.

• Strong understanding of data governance, security, and privacy best practices.

• Proven experience in designing, implementing, and optimizing large-scale data ingestion pipelines utilizing Databricks Autoloader.

• Hands-on experience in building and managing reliable, self-managing ETL/ELT pipelines using Delta Live Tables.

• Skilled in creating high-throughput, low-latency streaming data ingestion solutions with Apache Kafka, Spark Structured Streaming, and Databricks Streaming.

• Extensive experience in applying and enforcing Medallion architecture within a Databricks context.

• Familiarity with designing and implementing CI/CD pipelines for Databricks workflows, notebooks, and cluster configurations using tools like Azure DevOps, GitHub Actions, or GitLab CI.

• Experience in planning and executing data migration projects from conventional data warehouses to the Databricks Lakehouse.

• Strong knowledge of AWS or Azure data storage, networking, and security principles relevant to Databricks deployments.

• Ability to engage with clients, present technical solutions, and articulate complex concepts effectively.

• Bachelor's or Master's degree in Computer Science, Information Systems, Engineering, or a related discipline.

• Exceptional problem-solving, communication, and teamwork abilities.

🏝️ Benefits

• Supportive and collaborative team atmosphere.

• Recognition as a Great Place to Work.

• Opportunities for skill enhancement and professional development.

• Engaging work that addresses real-world business challenges.

• Opportunities for technical leadership and mentorship.

• Experience with cutting-edge cloud, data, and AI technologies.