Data Engineer

This is a fully remote position, open to applicants in Colombia.

📋 Description

• Create, develop, and manage scalable data pipelines utilizing Databricks and Apache Spark.

• Design and enhance ETL/ELT processes to acquire, transform, and load data from various source systems.

• Implement Change Data Capture (CDC) strategies to facilitate real-time and near-real-time data synchronization.

• Write clean, efficient, and maintainable Python code for data processing and automation tasks.

• Work closely with data architects and analysts to convert business needs into effective data pipeline designs.

• Monitor, troubleshoot, and enhance the performance, reliability, and cost-effectiveness of data pipelines.

• Ensure data quality, consistency, and integrity throughout the ingestion, transformation, and storage processes.

• Assist in the integration of structured and unstructured data sources into cohesive data platforms.

• Take part in code reviews and advocate for best practices in data engineering within the team.

• Document data pipelines, transformations, and architecture to ensure team visibility and maintainability.

• Collaborate with cross-functional teams to aid in analytics, reporting, and subsequent AI/ML applications.

⛳️ Requirements

• Over 3 years of experience in data engineering or a similar discipline.

• Extensive hands-on experience with Databricks for data processing and analytical tasks.

• Proficient in Apache Spark for large-scale distributed data processing.

• Experience in implementing CDC solutions and tools (e.g., Debezium or comparable technologies).

• Strong programming capabilities in Python for data engineering activities.

• Comprehensive understanding of SQL and relational database principles.

• Proven experience in building and maintaining ETL/ELT pipelines in production settings.

• Knowledge of data modeling principles and data warehousing concepts.

• Excellent analytical and problem-solving abilities.

• Effective communication skills, with the ability to collaborate across technical and business teams.

🏝️ Benefits

• Health insurance.

• Paid time off.

• Flexible work arrangements.

• Opportunities for professional development.