Data Engineer, AWS – Databricks

This is a fully remote position, open to applicants in Brazil.

📋 Description

• Design and improve Data Engineering solutions utilizing Databricks and AWS.

• Manage and enhance Unity Catalog functionalities.

• Organize catalogs, schemas, tables, volumes, and various data objects.

• Implement and oversee access controls and permissions through Unity Catalog.

• Assist in defining and executing data security and governance policies.

• Engage with Data Lineage, data discovery, and data traceability initiatives.

• Build and sustain batch data pipelines and, when necessary, streaming pipelines.

• Create transformations using SQL, Python, and PySpark.

• Work with Delta Lake / Delta Tables and Lakehouse architecture.

• Integrate data sources using AWS services and components of the Databricks platform.

• Ensure the quality, reliability, security, and availability of data.

• Aid in establishing architecture standards and best practices for utilizing Databricks.

• Monitor and enhance data workload performance and associated costs.

• Collaborate with teams in Data Engineering, Data Science, Analytics, Architecture, Security, and Governance.

⛳️ Requirements

• Extensive experience with Databricks.

• Strong practical experience with Unity Catalog.

• Familiarity with catalogs and schemas.

• Understanding of tables and views.

• Knowledge of volumes.

• Insight into access controls and privileges.

• Understanding of GRANT/REVOKE commands.

• Knowledge of Data Lineage concepts.

• Familiarity with data asset organization and governance.

• Understanding of integration with identity and security systems.

• Awareness of governance best practices within corporate settings.

• Experience with Delta Lake.

• Knowledge of Apache Spark and/or PySpark.

• Experience with Databricks Workflows/Jobs.

• Understanding of workload optimization and performance tuning.

🏝️ Benefits

• No specific benefits, additional benefits, or extra compensation were provided.