Senior Data Engineer

This is a fully remote position, open to applicants in Texas.

📋 Description

• Design, develop, and maintain scalable data pipelines utilizing Databricks, Apache Spark, Python, and Scala.

• Create and enhance ETL/ELT processes for data ingestion, transformation, and processing across various data sources.

• Work with Hadoop and its ecosystem to process and manage extensive volumes of both structured and unstructured data.

• Develop, troubleshoot, and enhance Spark applications to boost performance, scalability, and reliability.

• Utilize Control-M to schedule, monitor, and manage batch jobs, workflow dependencies, and data processing pipelines.

• Diagnose job failures, address data pipeline issues, and ensure the timely completion of scheduled workloads.

• Implement data validation, error handling, and quality checks to guarantee data accuracy and consistency.

• Collaborate with data architects, analysts, and cross-functional teams to comprehend requirements and deliver dependable data solutions.

• Adhere to coding standards, testing practices, version control, and deployment procedures.

• Provide support for production deployments, incident resolution, and ongoing maintenance of data engineering solutions.

⛳️ Requirements

• Over 8 years of comprehensive Data Engineering experience.

• Extensive hands-on experience with Databricks and Apache Spark.

• Proficiency in Python and Scala for data processing and application development.

• Practical experience with Hadoop and big data technologies.

• Familiarity with Control-M for batch scheduling, job monitoring, workflow orchestration, and dependency management.

• Strong understanding of ETL/ELT processes, data transformations, and distributed data processing.

• Experience in performance tuning, debugging, troubleshooting, and production support.

• Strong SQL skills and a solid grasp of data management concepts.

• Excellent analytical, problem-solving, and collaboration abilities.

🏝️ Benefits

• Medical

• Dental

• Vision

• Retirement Accounts

• Long and Short Term Disability

• Life Insurance

• HSA/FSA Accounts (USA)

• Discretionary time-off

• Continuous learning and hands-on exposure to emerging technologies

• Opportunities to build expertise in Generative AI, agentic AI, and next-generation data platforms

• Collaboration with global AI, data, engineering, and consulting talent

• Opportunities to lead strategic engagements and develop strategic client partnerships