Senior Data Engineer

This is a fully remote position, open to applicants in India.

📋 Description

• Directs projects focused on the design, development, and maintenance of a data and analytics platform.

• Efficiently processes, stores, and makes data accessible to analysts and other stakeholders.

• Collaborates with essential business stakeholders, IT professionals, and subject matter experts to strategize, design, and deliver optimal analytics and data science solutions.

• Engages with one or multiple product teams simultaneously.

• Designs and automates the deployment of our distributed system for ingesting and transforming data from various source types (relational, event-based, unstructured).

• Develops and implements a framework to consistently monitor and address data quality and integrity issues.

• Establishes data governance processes and methodologies for managing metadata, access, and retention of data for both internal and external users.

• Guides the design and construction of reliable, efficient, scalable, and high-quality data pipelines with monitoring and alert mechanisms that integrate diverse sources using ETL/ELT tools or scripting languages.

• Creates and implements physical data models to outline the database structure.

• Enhances database performance through effective indexing and table relationships.

• Contributes to the optimization, testing, and troubleshooting of data pipelines.

• Designs, develops, and manages large-scale data storage and processing solutions utilizing various distributed and cloud-based platforms for data storage (e.g., Data Lakes, Hadoop, Hbase, Cassandra, MongoDB, Accumulo, DynamoDB, and others).

• Employs innovative and contemporary tools, techniques, and architectures to automate repetitive and tedious data preparation and integration tasks, minimizing manual errors and boosting productivity.

• Aids in modernizing the data management infrastructure to enhance automation in data integration and management.

• Guarantees the timely execution and success of critical analytics initiatives by leveraging agile development methodologies such as DevOps, Scrum, and Kanban, while also mentoring and developing junior team members.

⛳️ Requirements

• Intermediate experience in a relevant field is essential.

• Familiarity with the latest technologies and trends in data engineering is highly desirable and includes:

• Understanding of complex business systems, industry requirements, and/or data regulations.

• Experience in processing and managing large datasets.

• Design and development experience for a Big Data platform using open-source and third-party tools.

• Proficiency in SPARK, Scala/Java, Map-Reduce, Hive, Hbase, and Kafka or equivalent academic coursework.

• Knowledge of SQL query language.

• Experience with clustered cloud-based compute implementations.

• Expertise in developing applications that require large file movement in a Cloud-based environment and utilizing various data extraction tools and methods.

• Experience in creating analytical solutions.

• Intermediate experience in the following areas is preferred:

• Familiarity with IoT technology.

• Background in Agile software development.

🏝️ Benefits

• Equal employment opportunities for all qualified individuals without discrimination based on race, sex, color, disability, national origin, age, religion, union affiliation, sexual orientation, veteran status, citizenship, gender identity, or any other status protected by law.