Big Data Engineer

This is a fully remote position, open to applicants in Virginia.

📋 Description

• AdvanSix is looking for a Big Data Engineer to develop and manage our enterprise Unified Data Layer (UDL).

• Design and implement batch/CDC/streaming pipelines, create curated/semantic layers, and enhance run-state through testing, CI/CD, security, and observability.

• Collaborate closely with the data team and the broader IT organization.

• Create and deliver scalable, secure data pipelines and data models that effectively link operational systems to analytics.

• Construct ingestion pipelines (batch, CDC, streaming) from sources including S/4HANA/DataSphere, PHD/historian, LIMS, TMS, HSE, and others.

• Enforce data contracts, manage schema/versioning, handle SCD, implement partitioning, and optimize performance.

• Create dimensional/semantic models that support certified Power BI datasets and APIs for applications/agents.

• Integrate OT data through OPC UA/MQTT and work with plant controls on change management, signal quality, and downtime scheduling.

• Incorporate data quality rules, unit/integration tests, and validation checks.

• Streamline build/test/deployment processes using Git-based CI/CD.

⛳️ Requirements

• At least 5 years of experience in data engineering, specifically in building production pipelines at scale (batch/CDC/streaming).

• Proficient with the Azure data stack: Databricks or Fabric/Synapse, ADF/Pipelines, ADLS/OneLake, Azure SQL/SQL MI, and Key Vault.

• Strong skills in SQL and Python/PySpark, with a good understanding of Spark Structured Streaming and performance optimization.

• Experience in implementing tests/observability (freshness, schema, expectations) and utilizing Git-based CI/CD.

• Familiar with SAP S/4HANA structures and the semantic modeling in SAP DataSphere.

• Knowledge of OT concepts: historians (PHD/PI), OPC UA/MQTT, event/batch frames, and basics of ISA-95/99.

• Understanding of Power BI consumption (semantic models, RLS) and APIs for downstream AI/ML applications/agents.

🏝️ Benefits

• Competitive salary and performance-based bonuses.

• Comprehensive health, dental, and vision insurance.

• Generous paid time off and holiday schedule.

• Opportunities for professional development and continuous learning.

• Collaborative and inclusive work environment.