Data Engineer

This is a fully remote position, open to applicants in India.

📋 Description

• As part of our team, you will design, construct, and sustain robust data pipelines utilizing Databricks and AWS infrastructure to enhance analytics and reporting capabilities throughout the organization.

• You will design and execute scalable ETL/ELT pipelines employing both batch and streaming methodologies.

• Your role will include building and maintaining ingestion workflows from various sources such as databases, APIs, and event streams.

• You will implement Change Data Capture (CDC), full-load, and incremental ingestion techniques.

• Development and management of data workflows using Apache Airflow for orchestration will be part of your responsibilities.

• You will configure and manage data ingestion connectors utilizing Airbyte.

• Collaborating with Databricks, you will construct and optimize data engineering workloads on the Lakehouse platform.

• Writing and optimizing complex SQL queries will be essential to your role.

• You should possess solid hands-on experience with dbt on Databricks, creating modular and testable dbt models for data transformation.

• Developing and maintaining data models in staging, intermediate, and mart layers, adhering to data warehousing best practices, will be expected.

• You will containerize data services and applications using Docker and EKS.

• Ensuring data quality, observability, and reliability across the data platform is a key responsibility.

• Documentation of pipelines, models, and data dictionaries will be necessary to maintain platform knowledge.

⛳️ Requirements

• We seek a skilled and enthusiastic Data Engineer to join our expanding Data Platform team.

• A minimum of 3 years of professional experience in data engineering is required.

• A strong grasp of data platform architecture, including Lakehouse, Data Warehouse, and Data Lake patterns, is essential.

• You should have hands-on experience with ETL/ELT design patterns, including batch and stream processing.

• Familiarity with ingestion techniques such as full load, incremental, CDC, and event-driven is necessary.

• At least 5 years of hands-on experience as a Database Administrator in production environments is required.

• A solid understanding of database internals, including storage engines, transactions, isolation levels, locking, MVCC, and query planners, is crucial.

• Proven experience in supporting mission-critical OLTP workloads with high availability needs is necessary.

• Strong scripting skills in Bash and/or Python for automation purposes are required.

• Experience in building data pipelines on Databricks, including Delta Live Tables, Jobs, and Notebooks, is expected.

• Proficiency in PySpark or Spark SQL for large-scale data processing is essential.

• Familiarity with Delta Lake concepts such as ACID transactions, time travel, and schema evolution is necessary.

• Proficiency in Apache Airflow for authoring, scheduling, and monitoring DAGs is required.

• Experience using Airbyte for managing source-to-destination data connectors is necessary.

• Hands-on experience with MongoDB administration (self-managed and/or Atlas) is required.

• Strong SQL skills, including query optimization, window functions, CTEs, and complex joins, are essential.

• Practical experience with AWS services, including S3, Lambda, IAM, and CloudWatch, is needed.

🏝️ Benefits

• Humility and kindness: We prioritize humility as a core characteristic in our hiring process, fostering a culture that values lightheartedness and laughter. Kindness is equally important to us, as we strive to create and sustain a diverse and inclusive environment.

• Flexibility: You will have the option to work from home. If you need to adjust your working hours to accommodate family or other commitments, we are supportive, provided you complete your tasks and communicate proactively. Additionally, you can work fully remotely from anywhere globally for one month each year.

• Financial benefits: We offer competitive salaries aligned with market standards and generous paid time off alongside holiday schedules. You will also have access to our flexible benefits scheme for home office equipment or health and fitness expenses. Certain employees at Sleek may be eligible for our employee share ownership plan, allowing them to participate in the growth of our company as we aim for a listing on a prominent stock exchange in the Asia Pacific region.

• Personal growth: At Sleek, you'll be entrusted with significant responsibility and autonomy. We operate at a fast pace, providing opportunities for decision-making, learning from mistakes, and personal development. We also host various internal and external training programs. As we lead the way in utilizing AI within our industry, we are building a regional center of AI excellence. Our goal is for you to leave Sleek as a more well-rounded individual and professional.

• Sleek is proudly a certified B Corp. Since our inception in 2017, we have been dedicated to building Sleek as a force for good. In just over five years, we have joined a community of industry leaders such as Patagonia, Ben & Jerry's, and P&G, who are committed to fostering an inclusive, equitable, and regenerative economy.