Data Engineer – Client Data Operations

This is a fully remote position, open to applicants in United States.

📋 Description

• Establish new data feeds from client and vendor systems, which includes SFTP and scheduled file deliveries.

• Recognize source fields, changes, missing data, and inconsistent formats.

• Map incoming data to predefined schemas and tables, identify sensitive fields, and implement approved privacy and access protocols.

• Create validation and matching procedures, which encompass record counts, identity matching or LTID assignment, and CASS/NCOA processing where applicable.

• Document feed inputs, transformations, checks, outputs, and failure protocols.

• Develop and enhance SQL, Python, and Spark pipelines for direct mail and digital clients.

• Configure scheduled jobs, file watchers, transformations, testing, and monitoring.

• Optimize existing pipelines as source files, schemas, tools, and business requirements evolve.

• Conduct daily checks and troubleshoot Databricks workflows, AWS jobs, and associated tools.

• Investigate failures and resolve root causes, including schema drift, faulty source files, expired credentials, broken dependencies, stale data, and vendor-tool complications.

• Enhance alerts, recovery protocols, and runbooks.

• Manage routine audience and mail-drop production, which includes suppressions, address hygiene, count reconciliation, quality control, and vendor file preparation.

• Identify risks to scheduled client drops and propose safe solutions.

• Transfer reliable, documented data to the Senior Analytics Engineer for subsequent modeling, analysis, and reporting.

• Collaborate with campaign, account, and technical teams to clarify requirements and resolve issues.

⛳️ Requirements

• Several years of practical data engineering experience, preferably four or more, encompassing pipelines operated in production and personally resolved failures.

• Comfort in transforming messy or poorly documented source files into dependable production data feeds.

• Strong attention to detail with the ability to reconcile counts, question questionable data, and avert recurring issues.

• Capability to work autonomously, articulate technical issues clearly, and recognize when business-rule or privacy concerns require an owner.

• Proficient in SQL and Python.

• Practical experience with Spark or similar large-scale data processing technologies.

• Experience in production cloud data engineering; familiarity with Databricks and AWS or comparable platforms is essential.

• Knowledge of orchestration, monitoring, schema mapping, file-based integrations, access controls, Git, testing, and operational documentation.

• Diligent handling of personally identifiable or protected health information in regulated data environments.

• Experience with direct mail and/or digital marketing audience data is advantageous, including deduplication, suppressions, CASS/NCOA, AccuZIP, AccuTrace, USPS tracking, or digital marketing sources and APIs.

• Background in healthcare, Medicare, or health plan marketing data; familiarity with SFTP, S3, Parquet, and client-facing reporting feeds is beneficial.

🏝️ Benefits

• .