Senior Data Engineer

This is a fully remote position, open to applicants in Taiwan, +2 more countries.

📋 Description

• Design, develop, and enhance scalable pipelines for the ingestion, transformation, and integration of high-volume market data, news feeds, and unstructured information.

• Ensure the quality, consistency, and real-time monitoring of data.

• Create processes to standardize and structure the data warehouse for cross-departmental utilization.

• Implement advanced data management methodologies to guarantee scalable, accessible, and efficient data storage solutions.

• Uphold data integrity, security, and performance at scale for trading and research infrastructures.

• Collaborate with research and analytics teams to identify data requirements and establish frameworks for data exploration, analysis, and model development.

• Develop tools for integrating data from various sources.

• Enhance storage, processing, and management for cost efficiency while maintaining a balance between performance and cost control.

• Assess and implement appropriate data engineering technologies while ensuring alignment with current best practices in data management.

⛳️ Requirements

• Strong problem-solving skills and analytical thinking.

• Excellent communication abilities for effective cross-functional collaboration.

• Proficient in creating robust data quality checks for ingested information.

• Experience in detecting anomalies within ingested data.

• Advanced skills in writing complex SQL (and similar) queries and optimizing their performance.

• Proficiency in Python or Java/Scala.

• Experience in constructing and maintaining intricate ETL pipelines using tools such as Apache Airflow, dbt, or custom scripts.

• Solid understanding of dimensional modeling, star/snowflake schemas, and normalization/denormalization principles.

• Proven experience with platforms like Snowflake, Redshift, BigQuery, Synapse, etc.

• Expert knowledge of Apache Spark, Kafka, Flink, or similar technologies.

• Strong comprehension of data security and privacy standards.

• Familiarity with a major cloud platform and its data services, validated by relevant certifications.

• Experience with data quality frameworks such as Great Expectations or Deequ.

• Familiarity with Git/GitHub or similar tools for version control.

• Experience with infrastructure-as-code tools like Terraform or CloudFormation.

• Exposure to containerization and orchestration technologies such as Docker or Kubernetes.

• Understanding of data governance, data lineage, and cataloging tools.

• Practical experience with observability and monitoring tools for data pipelines.

• Knowledge of machine learning pipeline processes.

• Prior experience in a trading or financial services environment.

🏝️ Benefits

• Competitive salary and performance-based bonuses.

• Comprehensive health and wellness benefits.

• Opportunities for professional development and continuous learning.

• Flexible work arrangements and a supportive work environment.