Data Engineer, AI – Distributed Systems

This is a fully remote position, open to applicants in United States.

📋 Description

• Design and manage batch and streaming pipelines that process and enhance high-volume unstructured data.

• Take full ownership of pipeline components, overseeing everything from implementation to production monitoring.

• Create and expand data pathways that support NLP, LLM, and retrieval services.

• Conduct data preparation, embedding generation, and indexing workflows.

• Integrate and optimize search and vector stores for semantic search, clustering, and real-time retrieval.

• Develop and refine microservices and APIs that provide analytics to enterprise clients.

• Produce clean, well-tested, and maintainable code.

• Engage in code reviews, CI/CD processes, and infrastructure-as-code methodologies.

• Troubleshoot production issues and enhance system reliability.

• Collaborate with Data Science, ML, Product, and Security teams to transition ideas from prototype to production.

⛳️ Requirements

• Minimum of 3 years of experience in building and operating data pipelines in a production environment.

• Strong programming proficiency in a JVM language such as Scala, Java, or Kotlin.

• Competence in Python for data manipulation and scripting tasks.

• Practical experience with a distributed processing framework, particularly Apache Spark.

• Familiarity with a streaming platform, ideally Kafka, including knowledge of consumer groups, offsets, partitioning, and lag management.

• Hands-on experience with AWS and comfort using Docker.

• Capability to operate within a Kubernetes environment.

• Experience with workflow orchestrators like Airflow, Prefect, or Dagster.

• Proficient in SQL and knowledgeable in at least one NoSQL or caching layer, such as Redis, MongoDB, or DynamoDB.

• Strong foundation in computer science principles, particularly in data structures and algorithms.

• Ability to assess performance and correctness in distributed systems.

• Excellent written communication skills and the ability to work asynchronously across U.S. time zones.

🏝️ Benefits

• Fully remote work environment.

• Support for learning Scala for candidates who are proficient in Java or Kotlin.

• Opportunity to collaborate with experienced engineers managing systems at scale.