Data Engineer

This is a fully remote position, open to applicants in United States.

📋 Description

• Develop, design, and uphold data architectures, ensuring they meet business needs.

• Acquire data, establish dataset processes, and store optimized data effectively.

• Detect issues and inefficiencies, implementing appropriate solutions.

• Identify tasks that can be automated to reduce manual efforts.

• Pinpoint and enhance data bottlenecks, utilizing automation when feasible.

• Establish and oversee data lifecycle policies (including retention, backups/restoration, etc.).

• Possess deep expertise in creating, sustaining, and managing ETL/ELT pipelines.

• Develop, maintain, and manage data transformations.

• Keep documentation current and updated.

• Create, manage, and oversee schedules for data pipelines.

• Monitor the performance of data pipelines.

• Establish, maintain, and manage data quality gates (using Great Expectations) to ensure data integrity.

• Assist AI/ML teams in optimizing feature engineering code.

• Demonstrate proficiency in Spark, Python, Databricks, Data Lake, and SQL.

• Create, manage, and maintain Spark Structured Streaming jobs, including the use of Delta Live Tables and/or DBT.

• Investigate existing data within the data lake to identify the best data sources.

• Create, manage, and maintain ksqlDB and Kafka Streams queries/code for data-driven testing to ensure data quality.

• Update and maintain Python-based data processing scripts running on AWS Lambdas.

• Develop unit tests for all Spark, Python data processing, and Lambda codes.

• Maintain the PCIS Reporting Database data lake, implementing optimizations and ongoing maintenance (performance tuning, etc.).

• Streamline the data processing experience by formalizing how to handle lake data, defining windows, and understanding how window definitions affect data freshness.

⛳️ Requirements

• A minimum of 5 years of IT experience focusing on enterprise data architecture and management, including data flow charts, diagrams, and other technical documentation.

• Proven experience with Databricks, Structured Streaming, Delta Lake concepts, and Delta Live Tables is essential.

• Strong Python development skills are required.

• Familiarity with ETL and ELT tools such as SSIS, Pentaho, and/or Data Migration Services, with the ability to integrate Python as needed.

• Advanced SQL skills (including Joins, Aggregation, Windowing functions, Common Table Expressions, RDBMS schema design, and Postgres performance optimization).

• Proficiency in using Git for version control, covering repository management, branching, merging, and pull requests.

• Active CompTIA Security+ certification is preferred.

🏝️ Benefits

• Candidates must be eligible to obtain and maintain a secret security clearance.

• United States Citizenship is required to meet eligibility criteria for obtaining this type of security clearance.