Senior Data Engineer

This is a fully remote position, open to applicants in United States.

📋 Description

• Design, construct, and maintain both batch and streaming data pipelines for structured and unstructured data sourced from databases, APIs, event streams, and files.

• Develop ETL/ELT processes while optimizing for performance, cost, scalability, and reliability.

• Build and manage cloud solutions utilizing AWS services such as EC2, ECS, Lambda, API Gateway, and SQS.

• Create and oversee data workflows that include scheduling, managing dependencies, handling retries, sending alerts, and performing backfills.

• Set up monitoring, logging, and observability systems to detect issues and ensure reliable production operations.

• Design, develop, test, and maintain secure REST APIs and Python-based data services using frameworks like FastAPI or Flask.

• Implement API authentication, authorization, validation, versioning, error handling, rate limiting, and documentation.

• Establish data foundations for machine learning and generative AI, which encompass feature pipelines, training and inference datasets, embeddings, and retrieval-based solutions.

• Integrate AI tools and large language model (LLM) APIs into workflows for tasks such as document processing, enrichment, classification, and summarization.

• Conduct data quality checks, validation, lineage, and documentation while adhering to security, privacy, and compliance standards.

• Lead architecture and code reviews, mentor engineers, and convey technical recommendations to both business and technical stakeholders.

⛳️ Requirements

• 8–10 years of professional experience in data engineering or a related area.

• Proficient in Python and advanced SQL, with a proven track record of delivering large-scale production data pipelines.

• Hands-on experience with at least one workflow orchestration platform and big data processing frameworks.

• Experience in developing and deploying production REST APIs, including aspects of integration, security, and automated testing.

• Familiarity with cloud platforms and data services.

• Understanding of data warehousing, lakehouse architecture, and data modeling principles.

• Working knowledge of AI and machine learning workflows, LLM APIs, embeddings, and vector databases.

• Experience with Git, CI/CD, Docker, and infrastructure as code tools like Terraform.

• Strong problem-solving abilities, communication skills, and stakeholder management capabilities.

• Preferred: Experience in building and operating solutions on AWS.

• Preferred: Experience in delivering retrieval-augmented generation (RAG) or agent-based solutions in production settings.

• Preferred: Familiarity with data governance, cataloging, and lineage tools.

• Preferred: Cloud or data platform certifications.

• Preferred: Experience in the healthcare industry.

• Preferred: Bachelor’s or master’s degree in Computer Science, Engineering, or a related discipline.

🏝️ Benefits

• Competitive salary based on your experience and skills.

• Bonus potential (based on role and is discretionary).

• 401k plans.

• Opportunities for growth and advancement.

• Comprehensive medical, dental, and vision insurance plans.

• Tax-free dependent care account.

• Life insurance, short-term, and long-term disability coverage.

• Excellent PTO policy.

• A great work-life balance environment.

• Strong supportive teams.