Data Engineer

This is a fully remote position, open to applicants in Pennsylvania.

📋 Description

• Create, construct, and uphold scalable data pipeline architectures.

• Develop cloud-based data solutions utilizing AWS services such as S3, Glue, Redshift, EMR, Athena, Kinesis, and Lambda.

• Establish and sustain data orchestration workflows with Apache Airflow, AWS Step Functions, or similar tools.

• Extract, transform, integrate, and load data from diverse sources using SQL and big data technologies.

• Design and enhance data environments to support statistical analysis, predictive modeling, machine learning, and research activities.

• Collaborate with data scientists, research analysts, and policy analysts to facilitate reproducible research and data-driven decision-making.

• Identify and rectify data quality concerns through validation, monitoring, and root-cause analysis.

• Automate manual processes to enhance data delivery, infrastructure scalability, and operational efficiency.

• Create dashboards, reports, analytical models, and decision-support tools.

• Develop monitoring, alerting, and logging frameworks for data pipelines.

• Assist with data conversions linked to system modernization and replacement initiatives.

• Ensure adherence to data governance, privacy, security, and Commonwealth policies.

• Partner with executive, data, design, and support teams to resolve technical challenges and enhance data systems.

• Support the Pennsylvania Longitudinal Data System (PALDS) for the Commonwealth of Pennsylvania.

⛳️ Requirements

• A minimum of 5 years of hands-on experience with AWS cloud services, including S3, EC2, Glue, EMR, RDS, Redshift, Athena, Lambda, and DynamoDB.

• At least 5 years of programming/scripting experience in Python, R, Java, C++, or Scala.

• A minimum of 5 years of experience with SQL and relational/NoSQL databases, such as Oracle, SQL Server, PostgreSQL, and Cassandra.

• At least 5 years of experience using Git, version control, and CI/CD pipelines.

• Strong understanding of data modeling, including dimensional modeling, Data Vault, and star/snowflake schemas.

• Experience with AWS Glue Data Catalog, metadata management, and data cataloging.

• Knowledge of data governance, data lineage, and data quality frameworks.

• Experience in designing and integrating REST APIs.

• Practical experience with Hadoop, Spark, and Kafka.

• Familiarity with Apache Airflow, Azkaban, or Luigi.

• Experience with Informatica, MuleSoft, Talend, or TIBCO.

• Knowledge of Storm or Spark Streaming.

• Advanced SQL skills and experience working with large, complex datasets.

• Strong analytical, problem-solving, project management, and organizational abilities.

• Experience in supporting cross-functional teams in a dynamic setting.

• Capacity to manage large, complex, and confidential datasets.

• Work authorization in the USA / visa status information required.

• Availability for a second-round in-person interview in Harrisburg, Pennsylvania for finalists.

🏝️ Benefits

• Remote work arrangement.

• Work hours Monday–Friday, 8:00 AM–4:30 PM.

• Text updates about application status available.

• Message and data rates may apply; option to opt out of text messages.