Senior Data Engineer

This is a fully remote position, open to applicants in United States.

📋 Description

• Design, construct, and sustain well-structured ETL pipelines.

• Operate at an enterprise level, processing terabytes of financial data.

• Ensure data is timely, accurate, and stored within the required contractual timelines.

• Develop and manage ingestion pipelines utilizing AWS Glue.

• Process and ingest large-volume XML filings.

• Store curated zone data in Apache Iceberg tables.

• Prevent duplicate data loads across target destinations.

• Integrate the agency's ETL Common Library into Glue jobs.

• Ensure all ETL jobs generate ETL Load Reports in real-time.

• Implement technical standards for data quality.

• Develop and maintain the semantic layer.

• Deploy ETL resources using CloudFormation templates.

• Create and maintain a comprehensive documentation suite for each dataset.

• Support the operationalization of statistical outputs and derived data products.

⛳️ Requirements

• US Citizenship is mandatory.

• A Bachelor's Degree is required.

• A minimum of 6 years of experience related to the position is essential.

• Strong expertise in AWS Glue (Spark-based), PySpark, and Python (PEP 8) is needed.

• Direct experience in constructing large-scale ETL pipelines on AWS.

• Familiarity with Apache Iceberg, Parquet, ORC, and Avro file formats is required.

• Experience with PostgreSQL, Redshift, and Oracle; familiarity with NoSQL, knowledge bases, and vector stores is necessary.

• Experience with Trino, Athena, and Hive.

• Proficiency in parsing and ingesting large-volume XML datasets with schema evolution handling using PySpark.

• Proficient with CloudFormation, GitHub branching workflows, and CI/CD pipelines.

• Proven ability to produce comprehensive ETL documentation.

🏝️ Benefits

• Flexibility to support personal and professional growth.

• Emphasis on collaboration, continuous learning, and excellence.

• Special incentives for team members residing in qualified HUBZones.