Lead Data Engineer

This is a fully remote position, open to applicants in Colombia.

📋 Description

• Design, construct, and sustain data pipelines and ETL processes utilizing Databricks and Apache Spark.

• Enhance data workflows for optimal performance, scalability, and cost-effectiveness.

• Implement Lakehouse architecture and oversee data ingestion from various sources.

• Collaborate with data scientists and analysts to facilitate advanced analytics and machine learning initiatives.

• Ensure data quality, governance, and security across all data assets.

• Monitor and resolve issues with Databricks clusters, jobs, and workflows.

• Integrate Databricks with cloud services (AWS, Azure, or GCP) and other enterprise systems.

• Document processes, standards, and best practices in data engineering.

⛳️ Requirements

• Practical experience with Databricks, Apache Spark, and PySpark.

• In-depth knowledge of SQL, Python, and data modeling concepts.

• Experience with cloud platforms (AWS, Azure, or GCP) and their associated data services.

• Familiarity with Delta Lake, Lakehouse architecture, and data governance practices.

• Understanding of CI/CD pipelines and DevOps methodologies for data workflows.

🏝️ Benefits

• Health insurance

• Professional development opportunities