Senior Data Engineer – Apache NiFi

This is a fully remote position, open to applicants in Portugal.

📋 Description

• Create, develop, and maintain data pipelines utilizing Apache NiFi to ensure effective data ingestion and transformation.

• Integrate both structured and unstructured data from various sources, including Salesforce, APIs, SFTP, SAP, and CRM, into cloud-based and on-premise databases.

• Safeguard data quality, security, and governance across a variety of sources and destinations.

• Enhance and troubleshoot NiFi workflows to boost performance, reliability, and scalability.

• Collaborate with APIs to retrieve and transmit real-time data for external integrations (e.g., CustomerGauge).

• Oversee and manage NiFi architecture within Kubernetes environments.

• Partner with data scientists, analysts, and engineers to ensure data availability and usability.

• Refine and enhance ETL/ELT processes for extensive data processing.

• Utilize NiFi Registry and Toolkit for version control and to streamline deployments across various environments.

• Write Bash and Python scripts for automation, monitoring, and executing workflows.

• Implement CI/CD pipelines for the deployment of data pipelines and automation of workflows.

• Diagnose and resolve issues related to data pipelines, guaranteeing high availability and reliability.

⛳️ Requirements

• In-depth knowledge of Apache NiFi, including flow design, performance optimization, version control, and security configurations.

• Experience in data ingestion, transformation, and processing for both batch and real-time workflows.

• Strong proficiency in SQL and NoSQL databases, such as PostgreSQL, MySQL, and MongoDB.

• Familiarity with big data technologies, including Apache Kafka, Spark, Hadoop, and HDFS.

• Experience with cloud platforms (AWS, Azure, or GCP) and their data services (e.g., AWS Redshift, Azure Synapse, Google BigQuery).

• Proficient in at least one programming language, such as Python or Java.

• Experience with CI/CD pipelines, version control tools (Git, Jenkins), and automation frameworks.

• Strong problem-solving abilities and capability to work autonomously in a remote setting.

• Excellent communication and stakeholder management skills.

• Proficiency in English.

• Must be based in Europe (Portugal/Spain preferred for permanent positions).

🏝️ Benefits

• Meal allowance: €10.20/day

• Flexible benefits plan

• Private medical insurance

• 22 days of annual leave, increasing every 3 years (up to 25 days)

• Continuous learning through KLX – Keyrus Learning Experience

• A collaborative, international, and human-centered work environment