Senior Data Engineer

This is a fully remote position, open to applicants in United States.

📋 Description

• Design, develop, and sustain scalable architectures for streaming data utilizing Kafka, MSK, and Kinesis.

• Create real-time data pipelines capable of managing high-volume and high-velocity data streams.

• Implement event-driven architectures and microservices approaches for processing streaming data.

• Develop and refine data streaming topologies to accommodate complex event processing scenarios.

• Architect fault-tolerant streaming systems with adequate error handling and data recovery methodologies.

• Set up, deploy, and oversee Apache Kafka clusters and AWS MSK environments.

• Establish Kafka Connect pipelines for integrating streaming data.

• Formulate optimal strategies for Kafka topic partitioning and replication configurations.

• Monitor and enhance Kafka cluster performance, throughput, and latency.

• Configure Kafka security settings, including SSL/TLS, SASL, and ACLs.

• Manage Kafka Schema Registry to facilitate data serialization and evolution.

• Design and execute solutions using Amazon Kinesis Data Streams and Kinesis Data Firehose.

• Set up Kinesis Analytics applications for processing real-time streams.

• Optimize shard management and auto-scaling configurations for Kinesis.

• Implement strategies for data retention and archival within Kinesis.

• Integrate Kinesis with other AWS services to create comprehensive streaming solutions.

• Develop real-time stream processing applications utilizing Apache Spark Streaming, Kafka Streams, or AWS Lambda.

• Apply complex event processing (CEP) patterns for real-time analytics.

• Construct streaming ETL pipelines that transform data in motion.

• Develop real-time aggregations, windowing operations, and stateful stream processing techniques.

• Enhance streaming query performance and resource utilization.

• Ensure seamless integration between streaming systems and data lakes, data warehouses, and operational databases.

• Implement data lineage and monitoring for streaming data pipelines.

• Establish automated data quality checks and validations for streaming data.

• Manage data serialization formats (Avro, JSON, Protobuf) along with schema evolution.

• Collaborate with data scientists and analysts to ensure that streaming data fulfills analytical requirements.

• Apply Infrastructure as Code (IaC) principles for streaming data platforms using Terraform or CloudFormation.

• Automate the deployment and management of streaming infrastructure via CI/CD pipelines.

• Monitor the health of streaming systems, performance metrics, and alerting mechanisms.

• Develop strategies for disaster recovery and high availability of streaming systems.

• Stay informed about emerging trends in streaming technologies and cloud-native solutions.

• Work alongside data architects, data scientists, and application teams to address streaming data needs.

• Uphold rigorous project governance through daily progress reviews and time tracking.

• Provide technical guidance and mentorship to junior data engineers.

• Convey complex streaming concepts to both technical and non-technical stakeholders.

• Operate with transparency and responsiveness to foster high-performing teams.

⛳️ Requirements

• Over 7 years of experience in the data engineering domain with a strong focus on streaming data.

• Bachelor’s degree in Computer Science, Engineering, or a related STEM discipline.

• Extensive practical experience with Apache Kafka, including cluster management, performance tuning, and associated tools.

• Proven track record with AWS MSK and Amazon Kinesis services in production settings.

• Solid background in real-time data processing and stream analytics.

🏝️ Benefits

• Medical, dental, and vision health insurance.

• Short-term and long-term disability, as well as life insurance.

• 401k plan with company matching.

• Paid time off (PTO) totaling 120 hours, accruable over one year.

• Paid time off for major holidays, amounting to 14 days per year.

• These and other employee benefit offerings are at management’s discretion and may be subject to change at any time.