Data Engineer II

This is a fully remote position, open to applicants in Texas.

📋 Description

• Assist in assessing and implementing emerging technologies for both batch and streaming data engineering.

• Foster innovation and facilitate the adoption of contemporary data solutions.

• Collaborate effectively with data scientists, architects, developers, and business stakeholders.

• Develop, deploy, and enhance scalable data pipelines and automation processes that empower advanced analytics and deliver business insights.

⛳️ Requirements

• 2-4 years of practical experience in data engineering is essential.

• A Bachelor’s degree in a relevant field or equivalent experience is required.

• Proficient in handling large data sets utilizing Hadoop, HDFS, Spark, Kafka, Pulsar, Flume, or similar distributed systems.

• Skilled in ingesting various source data formats such as JSON, Parquet, CSV, SequenceFile, Cloud Databases, Document Databases like CosmosDB, MQ, and Relational Databases like Oracle.

• Familiarity with Cloud technologies (including Azure, AWS, GCP) and native toolsets such as Azure ARM Templates, Hashicorp Terraform, and AWS Cloud Formation.

• A solid understanding of cloud computing technologies, business drivers, and emerging trends in computing.

• Comprehensive knowledge of Hybrid Cloud Computing, including virtualization technologies, Infrastructure as a Service, Platform as a Service, and Software as a Service delivery models, as well as the competitive landscape.

• Practical experience with Object Storage technologies, including but not limited to Data Lake Storage Gen2, S3, and ADLS.

• Familiarity with Agile development methodologies, including SAFe and Scrum, as well as Application Lifecycle Management.

• Strong background in source control management systems (GIT or Subversion), Code Quality (Sonar), Artifact Repository Managers (Artifactory), and Continuous Integration/Continuous Deployment (Azure DevOps).

• Experience with NoSQL data stores such as CosmosDB and MongoDB.

• Previous experience working with vehicle telemetry and auto insurance data is advantageous.

• Proficient in creating and maintaining ETL processes.

• Knowledgeable about best practices in information technology governance and privacy compliance.

• Experience with Adobe solutions (preferably Adobe Experience Platform) and REST APIs.

• Capable of troubleshooting complex issues and collaborating across teams to fulfill commitments.

• Excellent computer skills and expertise in digital data collection.

• Ability to thrive in an Agile/Scrum team environment.

• Strong interpersonal, verbal, and written communication skills.

• Understanding of big data platforms and architectures, data stream processing pipelines/platforms, data lakes, and data lake houses.

• SQL experience: adept at querying data and extracting actionable insights.

• Familiarity with cloud solutions, specifically Microsoft Azure and Amazon AWS architecture and services.

• Knowledge of GDPR, privacy, and security matters. Familiarity with data management and governance tools like Atlan and Immuta is a plus.

• Proficient in Microsoft Office software, data querying platforms (Databricks is a plus), and statistical programming tools such as Python.

🏝️ Benefits

• 401K matching.

• Bonding leave for new parents (12 weeks, fully paid).

• Tuition assistance.

• Training opportunities.

• GM employee auto discount.

• Community service pay.

• Nine company holidays.