Platform Data Engineer

📋 Description

• Design, develop, and uphold **schemas and data models**.

• Enhance table structures, partitioning, indexing, and compression for handling high-volume data.

• Guarantee swift and efficient querying for logs, requests, metrics, and performance traces.

• Manage ingestion pipelines for billions of records.

• Create resilient pipelines for:

• - API logs

• - Model inference logs

• - Error events

• - Usage & integration events

• - GPU & system metrics

• Implement ETL/ELT processes to convert raw data into formats suitable for analytics.

• Ensure the quality, dependability, and real-time accessibility of data sources.

• Develop tools to facilitate large-scale **log analysis**.

• Enable thorough investigation into latency, throughput, errors, and bottlenecks.

• Deliver the raw data foundation for end-to-end inference-time monitoring.

• Assist in debugging production issues utilizing logs and traces.

• Collaborate closely with DevOps, ML, and backend engineering teams.

• Integrate pipelines with monitoring tools (Prometheus, Grafana, Datadog, OpenTelemetry).

• Automate ingestion and cleanup processes.

• Create internal libraries or utilities to aid monitoring and debugging workflows.

• Supply clean data interfaces for the Data Expert (dashboards, monitoring, analytics).

• Support engineering teams by providing the appropriate logs and metrics.

• Contribute to debugging, root cause analysis (RCA), and performance enhancement initiatives.

⛳️ Requirements

• Substantial experience as a **Data Engineer** or a similar position in a production setting.

• Strong grasp of **data pipelines**, streaming versus batch processing, and data modeling.

• Experience with **analytical databases** (ClickHouse is advantageous, but not essential).

• Comfortable analyzing **logs, metrics, and platform data** to comprehend system behavior.

• Familiarity with **event-driven systems, monitoring, and observability principles**.

• Practical mindset: prioritizing usefulness, reliability, and performance over theoretical concepts.

• Ability to work collaboratively across backend, infrastructure, and data profiles.

• Experience in a startup or scale-up environment is a plus.

• Nice to Have:

• Experience with high-throughput or real-time systems.

• Exposure to cost monitoring, performance analytics, or platform observability.

• Background in AI, ML platforms, or data-intensive products.

🏝️ Benefits

• Generous paid time off – vacation, sick days, public holidays.

• Meaningful stock options – share in the upside you create.

• Remote-first setup – work from home wherever we can engage you.

• Flexible hours – manage your schedule outside of core collaboration times.

• Family leave – paid maternity, paternity, and caregiver time.

• Company retreats – biannual gatherings in inspiring locations.