Senior Data Engineer

This is a fully remote position, open to applicants in United States.

📋 Description

• CTG is looking for a Senior Data Engineer to create, construct, and sustain scalable and efficient data pipelines and systems in accordance with modern data engineering best practices.

• Develop, construct, and uphold scalable data pipelines, ETL/ELT workflows, and data models utilizing Python, Apache Spark (PySpark), Databricks, dbt, SQL (PostgreSQL), and AWS Glue.

• Create and enhance AWS-native data platforms by leveraging AWS Glue, Amazon EMR, Amazon MWAA (Apache Airflow), Lambda, Step Functions, Amazon S3, Redshift, RDS, DMS, and CloudWatch.

• Establish high-performance ingestion, transformation, and orchestration workflows for both structured and semi-structured data using Apache Iceberg, Parquet, ORC, and Avro.

• Design and refine analytical data platforms utilizing Amazon Athena, Trino, Hive, OpenSearch, and enterprise data catalog technologies.

• Integrate internal and external data sources across relational and NoSQL platforms, including PostgreSQL, Oracle, Redshift, GraphDB, and various NoSQL databases.

• Develop AI-driven data solutions using Amazon Bedrock, RAG pipelines, and vector search technologies such as Amazon S3 Vector and OpenSearch vector indexes.

• Create cloud infrastructure utilizing CloudFormation (Infrastructure as Code), GitHub, Harness, and enterprise CI/CD pipelines, while employing SNS, SQS, and EventBridge for event-driven architectures.

• Enhance the reliability, scalability, performance, and maintainability of enterprise data platforms through monitoring, troubleshooting, automation, and continuous optimization.

• Support critical analytics and reporting solutions within extensive AWS-based federal data environments, implementing solutions that adhere to FedRAMP and NIST 800-53 security controls.

• Lead modernization efforts in migrating legacy platforms such as IBM DataStage, Hadoop, RunDeck, and shell-based workflows to cloud-native AWS services.

• Guide junior engineers through technical mentorship, architectural discussions, and code reviews while fostering engineering best practices.

• Collaborate with cross-functional teams in an Agile setting to define requirements, deliver high-quality data solutions, and effectively communicate technical concepts to both technical and non-technical stakeholders.

⛳️ Requirements

• Bachelor’s degree in Computer Science, Engineering, or a related technical discipline.

• 9+ years of professional experience in data engineering or associated fields.

• Extensive hands-on experience with Databricks, Apache Spark (PySpark), Python, SQL (PostgreSQL), and dbt for large-scale data engineering, ETL/ELT development, data transformation, and data modeling.

• Expertise in designing, building, and maintaining AWS-native data platforms using AWS Glue, Amazon EMR, Amazon MWAA (Apache Airflow), AWS Lambda, AWS Step Functions, Amazon S3, Amazon Redshift, Amazon RDS, AWS DMS, and Amazon CloudWatch.

• Proficiency in developing scalable data pipelines, workflow orchestration, and data integration solutions across enterprise environments.

• Experience with modern data lake technologies including Apache Iceberg and data formats such as Parquet, ORC, and Avro.

• Knowledge in designing and optimizing solutions with relational and NoSQL databases including PostgreSQL, Redshift, Oracle, GraphDB, and other NoSQL platforms.

• Ability to build reliable, high-performance data platforms through performance tuning, system optimization, and enterprise-scale ETL/ELT architectures.

• Java development experience and familiarity with modern CI/CD practices using Harness.

• Strong analytical and problem-solving capabilities.

• Experience in agile, iterative software development environments.

• Capacity to swiftly learn and apply new technologies and domain knowledge.

• Excellent written and verbal communication skills, with the capability to explain complex topics to varied audiences.

🏝️ Benefits

• Remote Work (Hybrid roles will be specified in the job post).

• Competitive Compensation Package.

• Medical, Dental, and Vision coverage.

• Life Insurance, Short/Long Term Disability.

• Employee Assistance Program.

• 401(k) with 4% matching.

• Generous PTO vacation policy.

• Annual Continuing Education budget.

• Annual Wellness Budget.

• Bonus Incentive Programs (Employee referrals and performance-based rewards).