Data Engineer II

This is a fully remote position, open to applicants in India.

📋 Description

• **What You Will Do ❓**

• Design, develop, and maintain scalable batch and real-time data pipelines using Maxwell, Kafka, Spark, and dbt to fuel analytics and essential business applications.

• Create and refine data models adhering to Medallion Architecture (Bronze, Silver, Gold) to ensure the delivery of reliable, reusable, and high-quality datasets.

• Construct and manage cloud-native data platforms utilizing S3, Spark, Trino, and BigQuery, ensuring scalability, reliability, and cost-effectiveness.

• Craft robust data ingestion frameworks that utilize CDC (Maxwell), Kafka, and event-driven architectures for supporting near real-time data processing.

• Develop, optimize, and sustain data warehouses and data marts that facilitate fast, reliable reporting and self-service analytics.

• Collaborate closely with Product Managers, Data Analysts, Backend Engineers, and Business stakeholders to translate business needs into scalable data solutions.

• Create reusable dbt models, testing frameworks, and documentation to enhance data quality, governance, and developer efficiency.

• Optimize Spark jobs, Trino queries, and storage configurations for improved performance, reliability, and cost efficiency.

• Manage the complete lifecycle of critical data pipelines, ensuring high availability, monitoring, SLA compliance, and proactive incident management.

• Enhance the core data platform by developing reusable frameworks, automation, CI/CD pipelines, and adhering to engineering best practices.

• Maintain data quality through validation, monitoring, lineage, and observability while implementing best practices for security and governance.

• Empower analytics teams by providing reliable datasets, semantic models, and dashboards that facilitate decision-making through Metabase.

⛳️ Requirements

• **What We're Looking For 🚀**

• Over 4 years of practical experience in designing and constructing scalable data platforms, data lakes, and data warehouses.

• Strong expertise in Spark (Scala, Python) and SQL, with experience in building production-grade data pipelines and distributed data processing applications.

• Hands-on experience with Apache Spark and a thorough understanding of distributed data processing, including performance tuning and optimization.

• Experience in developing batch and streaming data pipelines using technologies such as Kafka, CDC/Maxwell, or comparable event-driven architectures.

• Solid understanding of contemporary data lake architectures, including Medallion Architecture, data modeling, partitioning, and storage optimization.

• Experience with cloud-native data platforms and technologies such as Amazon S3, BigQuery, Trino, or similar analytics engines.

• Extensive experience in designing dimensional models, star schemas, and creating dependable data marts that support analytics and business intelligence.

• Hands-on experience with dbt, including the development of reusable models, implementation of automated testing, and maintenance of documentation.

• Strong knowledge of data quality, observability, lineage, and engineering best practices for building reliable and maintainable data products.

• Experience in optimizing large-scale data pipelines, SQL queries, and distributed processing jobs for performance, scalability, and cost efficiency.

• Familiarity with CI/CD, Git-based development workflows, infrastructure automation, and modern software engineering best practices.

• Excellent problem-solving abilities with the capability to independently manage projects from design to production.

• Strong communication and stakeholder management skills, with experience in collaboration across Product, Engineering, Analytics, and Business teams.

• A genuine passion for constructing scalable data platforms and consistently enhancing developer experience, platform reliability, and operational excellence.

🏝️ Benefits

• **What We Offer You❗** An inclusive and diverse environment: We nurture an inclusive and diverse workplace that values innovation and provides remote work options.

• Competitive compensation: Our compensation packages are highly competitive and may include potential share options for specific roles.

• Personal growth and development: We are dedicated to your personal and professional advancement, offering regular training and an annual learning stipend to support your career growth in a dynamic environment.

• Autonomy and mentorship: You will enjoy a significant degree of autonomy in your role, backed by mentorship and ambitious goals that foster both your success and the company's growth.