Senior Data Engineer – ETL Data Modeling

This is a fully remote position, open to applicants in India.

📋 Description

• Develop ETL/ELT pipelines both within and outside a data warehouse utilizing Python, Pyspark, and Snowflake's Snow SQL.

• Assist in the migration of Redshift DWH to Snowflake.

• Design, implement, and maintain the infrastructure for data warehouses and data lakes using the AWS big data ecosystem, Python, Redshift, Snowflake, Glue/Lake Formation, EMR/Spark/Scala, etc.

• Collaborate with data analysts to enhance value-adding capabilities, including data integrations and transformations, feature modeling, and the application of statistical and machine learning models.

• Engage with Product Managers, Finance, Service Engineering Teams, and Sales Teams on a daily basis to fulfill their analytics needs.

• Execute data quality and governance protocols, including data profiling and validation practices.

• Establish and maintain data governance standards to ensure the quality, integrity, and security of data throughout its lifecycle.

• Utilize open-source technologies to create robust and economical data solutions.

• Design and maintain streaming data pipelines using technologies such as Apache Kafka.

⛳️ Requirements

• A minimum of 5 years of IT experience, with at least 3 years in data integration, ETL/ELT development, and database or data warehouse design.

• Extensive knowledge and experience with distributed systems, streaming systems, and data engineering tools like Kubernetes, Kafka, Airflow, Dagster, etc.

• Experience in data transformation, utilizing ETL/ELT tools and technologies such as AWS Glue, DBT, etc., for handling structured, semi-structured, and unstructured datasets.

• Proficient in ingesting and integrating data from APIs, JDBC, and CDC sources.

• Strong expertise in Python, SQL, relational and non-relational database design, and master data management strategies.

• Proven ability to define, architect, and launch data products, with full ownership through their lifecycle.

• In-depth understanding of Star and Snowflake dimensional modeling.

• Experience with relational databases, including SQL queries, database definitions, and schema design.

• Familiarity with data warehouses, distributed data platforms, and data lakes.

• Strong command of SQL and at least one programming language (e.g., Python, Scala, JS).

• Knowledge of data orchestration tools, like Apache Airflow, with skills in designing and managing complex data workflows.

• Familiarity with agile methodologies, including sprint planning and retrospectives.

• Proficient with version control systems such as Bitbucket/Git.

• Ability to thrive in a fast-paced startup atmosphere and adapt to evolving requirements while managing multiple concurrent projects.

• Excellent verbal and written communication abilities.

• Preferred/bonus skills include experience with Redshift to Snowflake migration.

• Experience with DevOps technologies such as Terraform, CloudFormation, and Kubernetes is advantageous.

• While not essential, experience or knowledge of machine learning techniques is highly desirable, enhancing our data engineering capabilities.

• Familiarity with non-relational databases/data stores (e.g., object storage, document or key-value stores, graph databases, column-family databases).

🏝️ Benefits

• Engaging work environment within a rapidly growing company.

• Experience working in an international setting.

• Enjoy a pleasant workplace with minimal hierarchy.

• Engage in intellectually stimulating work that significantly contributes to client success and scalability.

• Flexible working hours.