Staff Agentic Data Engineer – Data Platform

This is a fully remote position, open to applicants in United States.

📋 Description

• Take ownership of the architecture and progression of large-scale data processing and analytics systems, encompassing ingestion, normalization, transformation, aggregation, and customer-facing analytics.

• Design and enhance data models that support high-volume analytical workloads, effectively balancing query performance, storage efficiency, maintainability, and scalability.

• Create and refine complex analytical SQL queries across extensive datasets, pinpointing performance bottlenecks and devising scalable solutions.

• Construct distributed and asynchronous job processing systems for the dependable handling of large data volumes across concurrent workloads.

• Develop and adapt Spark, Flink, or similar data-processing workflows for extensive transformation, enrichment, aggregation, and computation tasks.

• Promote the effective use of SingleStore/MemSQL, focusing on schema design, indexing and partitioning strategies, query optimization, and workload design.

• Establish architectural patterns for batch processing, data transformation, job orchestration, failure recovery, idempotency, and reprocessing.

• Troubleshoot complex performance and scalability challenges across databases, distributed processing systems, application services, and infrastructure.

• Collaborate with engineering and product teams to convert ambiguous analytical and data-processing requirements into scalable technical architectures.

• Define technical standards and reusable patterns for data modeling, data processing, and analytical query development.

• Offer technical leadership across engineering initiatives, influencing architecture and mentoring engineers while remaining actively involved in implementation.

• Recognize architectural limitations and spearhead improvements to enhance platform scalability, reliability, and performance.

⛳️ Requirements

• Over 10 years of professional experience in software engineering, data engineering, or data platforms, with substantial experience in designing and managing production data systems.

• Profound expertise in relational and analytical data modeling, including the creation of schemas for large-scale analytical and reporting workloads.

• Expert-level proficiency in SQL, with proven experience in designing, analyzing, and optimizing complex analytical queries over large datasets.

• Extensive experience with SingleStore/MemSQL or equivalent distributed SQL/analytical database technologies.

• Significant practical experience with Spark, Flink, or similar technologies, focusing on the design and optimization of distributed data-processing workloads.

• Strong understanding of distributed data processing, encompassing partitioning, parallelism, data skew, resource utilization, and performance optimization.

• Experience in designing high-throughput job processing systems, addressing scheduling, concurrency management, retries, failure recovery, idempotency, and observability.

• Proficient in designing data pipelines that support batch ingestion, transformation, enrichment, aggregation, and downstream analytics.

• Strong grasp of database performance characteristics, including indexing, partitioning, query execution plans, data distribution, and memory/compute trade-offs.

• Experience in identifying and rectifying performance bottlenecks across application, database, and data-processing layers.

• Proficient in Python, Java, or another production programming language utilized for building data-processing and distributed systems.

• Strong understanding of testing strategies for data-intensive systems, including unit, integration, data-quality, and end-to-end testing.

• Demonstrated ability to independently manage technically ambiguous problems and drive them from investigation to production implementation.

• Proven capability to influence architecture and technical direction across multiple teams without relying solely on organizational authority.

• Ability to articulate complex data architecture and performance trade-offs clearly to engineers, product stakeholders, and technical leadership.

• Efficacious collaboration with distributed teams across various time zones.

• Preferred: Extensive production experience with SingleStore/MemSQL, especially with large-scale analytical workloads.

• Preferred: Advanced experience with pipeline performance tuning, partitioning strategies, memory management, and optimization of large distributed workloads.

• Preferred: Experience in designing platforms that facilitate both high-volume ingestion and low-latency analytical queries.

• Preferred: Experience in building or operating distributed job execution and orchestration systems.

• Preferred: Familiarity with large-scale cybersecurity, vulnerability, asset, telemetry, or event datasets.

• Preferred: Experience with vulnerability scanning technologies across the security stack, including SCA, SAST, DAST, IAST, vulnerability scanning, and container security.

• Preferred: Experience designing systems for incremental processing, backfills, replay/reprocessing, deduplication, and late-arriving data.

• Preferred: Experience with data observability and operational monitoring for large-scale processing workloads.

• Preferred: Experience integrating AI/LLM-driven capabilities with data platforms, including preparing and processing large datasets for AI-assisted analytical workflows.

• Preferred: Experience operating data-intensive applications on Linux and cloud infrastructure.

• Preferred: Familiarity with Agile/Scrum methodologies in a professional engineering environment.

🏝️ Benefits

• Flexible PTO

• 401k company contribution (not a match, a contribution)

• Generous education and training budget

• 100% company-paid healthcare

• Fully remote work