Senior Data Engineer

📋 Description

• Create, develop, and sustain scalable data pipelines that ingest, transform, and validate large quantities of data from various sources and channels.

• Enhance the scalability, reliability, and performance of our data pipelines to accommodate rapidly expanding workloads and new data streams.

• Participate in the design and implementation of our Data Lake architecture, facilitating dependable data storage and reuse across teams.

• Oversee and optimize data ingestion processes, including data sourced from web scrapers, third-party vendors, and internal systems.

• Monitor the health of pipelines, investigate issues, and implement enhancements to boost system reliability and observability.

• Aid in the onboarding and integration of new AI channels and data sources into the platform.

• Collaborate with various teams within the organization to ensure that data generated by different systems can be effectively reused for analytics and business intelligence.

• Identify and resolve performance limitations in distributed systems, focusing on rate limiting, concurrency, and throughput challenges.

• Provide guidance to engineering and product teams on data architecture, data quality, and best practices for managing scalable data workflows.

• Continuously assess and enhance our data platform to support the company's rapid growth and evolving product requirements.

⛳️ Requirements

• Extensive experience in building and operating scalable data pipelines in production settings.

• Practical experience with Data Lakes or Data Warehouses (e.g., AWS Athena or similar technologies).

• Expertise in data transformation and modeling.

• Strong experience with AWS.

• Familiarity with Infrastructure-as-Code tools for managing cloud infrastructure.

• Proficiency in Python for data processing and automation tasks.

• Experience working with distributed systems and overseeing large-scale data workflows.

• Proven experience in implementing monitoring, observability, and incident response strategies for data systems.

🏝️ Benefits

• Unique opportunity to be part of a fast-paced startup at the forefront of AI development.

• Engage with challenging and abstract problems while revolutionizing the $300 billion legacy mar-tech industry.

• Become a member of a skilled high-performing team where you will have immediate ownership and impact.

• Experience a genuine meritocracy with significant career advancement potential as the business grows.