
Senior Data Engineer
Posted 1 day ago

Posted 1 day ago
This is a fully remote position, open to applicants in Brazil.
• Design and enhance the architecture of the platform’s data using Databricks, AWS, and PostgreSQL.
• Create, maintain, and support ETL/ELT pipelines utilizing PySpark, SQL, and AWS services.
• Execute data ingestion processes, including incremental loads and Change Data Capture (CDC).
• Organize and develop the Bronze, Silver, and Gold layers within Delta Lake.
• Model and deliver data that is optimized for utilization in BI tools, with a focus on Power BI.
• Oversee data governance through Unity Catalog, ensuring access control, traceability, and lineage.
• Implement testing protocols and controls for data quality.
• Monitor pipelines, clusters, and databases, identifying failures and opportunities for performance enhancement and cost reduction.
• Address incidents and issues related to data solutions.
• Apply best practices for security, privacy, and compliance with LGPD.
• Organize and maintain CI/CD, code versioning, and infrastructure processes.
• Engage in projects related to the implementation, evolution, and modernization of the data platform.
• Compile technical documentation encompassing architecture, pipelines, integrations, and data models.
• Collaborate with stakeholders to gather requirements and delineate solutions.
• Act as a technical resource for the team by reviewing code, establishing standards, and mentoring less experienced professionals.
• Minimum of 3 years of experience in Data Architecture or Data Engineering.
• Practical experience in developing and supporting ETL/ELT pipelines in a production environment.
• Strong expertise in Databricks, particularly with Delta Lake, Lakehouse architecture, Unity Catalog, and Workflows/Jobs.
• Familiarity with AWS, especially S3, IAM, RDS/Aurora PostgreSQL, and integration and automation services.
• Advanced understanding of PostgreSQL, including data modeling, indexing, query optimization, and extraction/CDC techniques.
• Strong command of SQL and Python.
• Experience in PySpark and distributed processing.
• Knowledge of data modeling for analytical purposes.
• Experience delivering data to BI tools, preferably Power BI.
• Proficient with Git and collaborative version control practices.
• Understanding of data quality, security, and governance, including LGPD compliance.
• Ability to gather and assess requirements in collaboration with business teams.
• Experience in preparing technical and architectural documentation.
• Databricks and/or AWS certifications are advantageous.
• Familiarity with Terraform and Databricks Asset Bundles is a plus.
• Knowledge of CI/CD and DevOps/DataOps methodologies is a plus.
• Experience in FinOps and cost optimization within Databricks and AWS is a plus.
• Background in data platform migration or modernization initiatives is a plus.
• Experience administering Metabase is a plus.
• Day off
• Wellness platform
• Cost allowance
• Flexible working hours
Trax Technologies
Fiserv
Ontrac Solutions
General Dynamics Information Technology
Get handpicked remote jobs straight to your inbox weekly.