Senior Data Engineer

This is a fully remote position, open to applicants in United States.

📋 Description

• Design, develop, and sustain scalable data pipelines utilizing Spark, Hive, and Airflow.

• Create and implement data processing workflows on Databricks.

• Develop API services to facilitate data access and integration.

• Generate interactive data visualizations and reports using AWS QuickSight.

• Construct infrastructure for extracting, transforming, and loading data from diverse sources using AWS and SQL.

• Monitor and enhance data infrastructure and process performance.

• Create data quality and validation jobs.

• Assemble intricate datasets that fulfill both functional and non-functional business requirements.

• Write unit and integration tests for data-processing code.

• Collaborate with DevOps engineers on Continuous Integration (CI), Continuous Deployment (CD), and Infrastructure as Code (IaC).

• Transform specifications into code and design documentation.

• Conduct code reviews and enhance code quality processes.

• Improve data availability and timeliness through refreshes, tiered storage, and dataset optimization.

• Ensure data security and privacy during storage and transmission.

• Execute other assigned responsibilities.

⛳️ Requirements

• Bachelor’s degree.

• Over 7 years of practical software development experience.

• More than 4 years of experience in building data pipelines using Python, Java, and cloud technologies.

• Demonstrated experience with Spark and Hive for large-scale data processing.

• Ability to obtain and maintain a Public Trust clearance.

• Must reside in the US and have authorization to work in the US.

• All work must be conducted within the US.

• Must have lived in the US for 3 full years out of the last 5 years.

• Experience with Databricks job workflows.

• Strong knowledge of AWS products such as S3, Redshift, RDS, EMR, AWS Glue, AWS Glue DataBrew, Jupyter Notebooks, Athena, QuickSight, and Amazon SNS.

• Familiarity with data transformation, workload management, data structures, dependencies, and metadata processes.

• Experience in data governance for batch and streaming ingestion, curation, and data sharing.

• Proven experience in building and optimizing data pipelines and data systems.

• Familiarity with Cassandra, Postgres, relational, NoSQL, and SQL databases.

• Experience with Airflow, Luigi, and Azkaban.

• Experience with Spark Streaming and Storm.

• Proficiency in Scala, C++, Java, and Python.

• Familiarity with CI/CD pipelines, GitHub Actions, and Terraform IaC.

• Capability to obtain and maintain a Public Trust while living in the United States.

• Experience with Agile methodologies and test-driven development.

🏝️ Benefits

• Equal opportunity employer.

• Reasonable accommodations available for disabled veterans, individuals with disabilities, and individuals with sincerely held religious beliefs.

• Confidential accommodation support.

• Full-time employment compensation range of $89,649.00–$152,404.00.