Big Data Engineer, Hadoop, Azure

This is a fully remote position, open to applicants in Pennsylvania.

📋 Description

• Design, develop, and sustain an efficient data pipeline architecture that integrates both structured data sources and big data across on-premise and cloud environments, applicable in both streaming and real-time contexts.

• Create and automate ETL code utilizing scripting languages, ETL tools, and job scheduling software to fulfill all reporting and analytical data requirements.

• Conceptualize and construct dimensional data models to advance data warehouse initiatives.

• Compile extensive, intricate data sets that cater to the analytical requirements of the data science team.

• Evaluate new data sources to gain insights into their availability and data quality.

• Identify, design, and execute internal process enhancements: automating manual tasks, optimizing data pipeline performance, and redesigning infrastructure for improved scalability and information accessibility.

• Engage in requirements gathering sessions to refine technical specifications from business inquiries.

• Collaborate with business partners to productionize, optimize, and expand enterprise analytics.

• Work alongside data architects and modelers to establish data store designs and adhere to best practices.

⛳️ Requirements

• Over five (5) years of experience with traditional and modern Big Data technologies (HDFS, Hadoop, Hive, Pig, Sqoop, Kafka, Apache Spark, HBase, Oozie, NoSQL databases, PostgreSQL, GIT, Python, REST API, Snowflake, etc.).

• At least two (2) years of experience in building data platforms using the Azure stack (Azure Data Factory, Azure DataBricks, etc.).

• Proficient in object-oriented/object function scripting languages such as Python, Java, C++, and Scala.

• Experience in extracting, querying, and joining large data sets at scale.

• Familiarity with utilizing Snowflake to develop data marts with data stored in Azure storage is an advantage.

• Comprehensive understanding of relational, columnar, and NoSQL database architectures along with industry best practices for development.

• Knowledge of dimensional data modeling for the design and construction of data warehouses.

• Exceptional advanced SQL coding and performance tuning capabilities.

• Experience with parsing data formats like XML/JSON and utilizing external APIs.

• Familiarity with agile development methodologies.

• Ability to thrive in a collaborative, team-oriented environment, with strong interpersonal abilities.

• Strong analytical and problem-solving skills, capable of evaluating various technical solutions against original business needs to identify the most cost-effective option.

• Meticulous attention to detail and ability to assess the impact of design changes before implementation.

• Self-motivated, driven, and quick to learn.

• Capability to prioritize and execute tasks effectively in a high-pressure environment.

• Strong customer service orientation.

• Ability to convey and clarify technical information to diverse audiences in a manner that builds rapport and fosters understanding.

• Work experience with geospatial data and spatial analytics is preferred.

🏝️ Benefits

• Competitive salary and performance-based bonuses.

• Comprehensive health, dental, and vision insurance.

• Flexible working hours and remote work options.

• Opportunities for professional development and training.

• Collaborative and inclusive work environment.