Data Engineer, Salesforce Focus

This is a fully remote position, open to applicants in Colombia, +4 more countries.

📋 Description

• Implement comprehensive migration of critical datasets, legacy data workflows, and orchestrated DAGs to upgraded cloud infrastructure.

• Transform legacy Hive table structures into Apache Iceberg format on AWS S3.

• Reconfigure, validate, and oversee Apache Kafka connections, brokers, and topics across various AWS regions.

• Adapt, model, and optimize data pipelines directly within Snowflake.

• Create, schedule, and troubleshoot Apache Airflow DAGs.

• Address upstream failures, enhance sensor dependencies, and avert pipeline bottlenecks.

• Establish and implement data reconciliation and validation methodologies.

• Ensure zero data loss, schema consistency, and complete data integrity throughout migrations.

• Analyze, profile, and resolve performance challenges across distributed Spark/PySpark tasks, Kafka consumer groups, and AWS storage tiers.

• Collaborate in daily standups and real-time architectural meetings with US engineering leadership.

⛳️ Requirements

• Over 6 years of practical data engineering experience in designing, deploying, and maintaining distributed, production-level data pipelines for petabyte-scale datasets.

• High level of proficiency in Python.

• Advanced SQL capabilities, including complex CTEs, window functions, schema design, and query optimization.

• Proven production experience with Apache Spark / PySpark for both batch and streaming transformations.

• Strong practical experience in authoring and maintaining complex Apache Airflow DAGs.

• Significant hands-on experience in configuring cross-regional event pipelines using Apache Kafka.

• Direct experience in managing AWS S3 data lakes with Apache Iceberg and Parquet formats.

• Practical experience with Snowflake schema modeling, warehouse setup, and query optimization.

• Ability to independently troubleshoot issues related to distributed pipeline execution failures, network latencies, and serialization bottlenecks.

• Capability to work within an enterprise AWS WorkSpace (VDI) environment.

• Must perform a mandatory AWS network health-check verification and provide a passing screenshot during the application process.

• Availability for full-time business hours aligned with US Central Time (CT).

• Proficiency in English required for interviews, technical documentation, architectural evaluations, daily team syncs, and operational communications.

🏝️ Benefits

• 100% remote work.

• Full-time vendor contractor engagement directly with In All Media.

• Time zone compatibility with US Central Time.

• Remote, distributed-team workplace.

• Global collaboration with international clients and teams.