Senior Data Engineer

This is a fully remote position, open to applicants in United States.

📋 Description

• Take ownership and manage data pipelines, which includes Tala's change data capture (CDC) and data synchronization infrastructure.

• Guarantee the precise and prompt delivery of data from market-level databases into Snowflake.

• Enhance pipeline performance and scalability, minimize latency, optimize infrastructure costs, and accommodate increasing data volumes.

• Integrate new data sources, tables, and markets while overseeing schema evolution and system compatibility.

• Set service level objectives (SLOs), implement monitoring and alerting, and create automated data quality verification processes.

• Design and uphold unit and integration tests, CI/CD workflows, infrastructure-as-code, and safe deployment methods, including dry runs and blue-green deployments.

• Diagnose production issues, perform root cause analyses, and enhance monitoring, testing, documentation, and operational runbooks.

• Engage in architecture and design discussions, compose and evaluate technical design documents, and provide feedback on code reviews.

• Work collaboratively with Data Science, Analytics, Data Platform, Product, Credit, and Business Development teams.

⛳️ Requirements

• Over 4 years of experience in backend engineering or data engineering, preferably supporting consumer-scale applications.

• A minimum of 2 years of practical experience with Python in a professional setting.

• Demonstrated ability to independently build, manage, and enhance production-grade data pipelines.

• Practical experience with change data capture (CDC) and stream processing technologies such as Kafka, Amazon MSK, AWS DMS, or similar tools.

• Strong expertise in Apache Spark, especially Spark Structured Streaming.

• Familiarity with modern data lakehouse architectures and table formats like Apache Iceberg, Delta Lake, or Apache Hudi, as well as Parquet.

• Solid database experience across relational and analytical systems, including MySQL, PostgreSQL, Amazon RDS, and Snowflake.

• Proficient in AWS cloud infrastructure, ideally including EMR, S3, MSK, DMS, RDS, and EKS.

• Experience with infrastructure-as-code and CI/CD tools such as Terraform, Terragrunt, Helm, and Jenkins.

• Knowledge of pipeline orchestration and job scheduling tools like Airflow or Python-based services operating on Kubernetes.

• Strong grasp of distributed systems, scalable algorithms, and large-scale data processing.

• Ability to articulate complex technical concepts clearly and collaborate effectively with engineering teams and business stakeholders.

• Nice to have: Experience with Trino or Presto.

• Nice to have: Experience with Kubernetes and containerized applications.

• Nice to have: Familiarity with AWS Glue.

• Nice to have: Experience with additional stream processing technologies such as Debezium, Amazon Kinesis, or Apache Flink.

• Nice to have: Experience with other modern data lakehouse technologies like Delta Lake or Apache Hudi.

• Nice to have: Advanced degree in Computer Science, Mathematics, or a related technical field.

🏝️ Benefits

• Emphasis on a remote-first working model.

• Office hubs located in Santa Monica, CA (HQ); Nairobi, Kenya; Mexico City, Mexico; Manila, the Philippines; and Bangalore, India.