Senior Director, Software Engineering

This is a fully remote position, open to applicants in Illinois.

📋 Description

• Create, develop, and enhance scalable data workflows utilizing Python, PySpark, and Airflow

• Execute real-time and batch data processing through Spark

• Uphold data quality, governance, and security best practices throughout the entire data lifecycle

• Supervise and automate data systems to guarantee availability, reliability, and performance

• Oversee cloud infrastructure and minimize data-processing expenses

• Establish CI/CD pipelines for dependable data-workflow deployments

• Construct and improve large-scale data-processing pipelines with Apache Spark and PySpark

• Apply data partitioning, caching, and performance optimization for Spark workloads

• Work with both structured and unstructured data to support analytics and machine-learning projects

• Design and manage Airflow DAGs to automate intricate data workflows

• Monitor, diagnose, and enhance job execution and dependencies

• Lead a team of data engineers, providing technical guidance and mentorship

• Advocate for coding standards, version control, and best practices in documentation

• Collaborate directly with clients to grasp business requirements and convey technical concepts

• Work together with engineering, product, and architecture teams to promote scalable and maintainable products

• Partner with Program Management to synchronize delivery schedules and address risks

⛳️ Requirements

• Over 8 years of experience in data engineering

• Proficient in Python, PySpark, Apache Spark, and Apache Airflow

• Strong programming abilities in Python and SQL

• Capability to produce efficient and maintainable code

• Comprehensive understanding of Spark internals, including RDDs, DataFrames, DAG execution, and partitioning

• Experience with Airflow DAGs, scheduling, and managing dependencies

• Familiarity with Git, Docker, Kubernetes, Terraform, and DevOps best practices for CI/CD workflows

• Background in optimizing large-scale data processing

• Experience in leading teams and working within Agile/Scrum frameworks

• Familiarity with common data warehouse and/or data lake platforms

• Knowledge of cloud technologies, including Azure, AWS, or Google Cloud

• Proven experience using generative AI tools for software development, data engineering, system design, testing, and troubleshooting

• Strong understanding of AI-assisted engineering practices, including prompt engineering, code generation, automated testing, and AI-driven productivity workflows

• Ability to recognize opportunities for applying AI solutions to enhance productivity, code quality, and operational efficiency

• Excellent communication and collaboration skills

• Experience with ML Engineering is an advantage

🏝️ Benefits

• Flexible and accommodating work environment

• Paid time off

• Medical insurance

• Dental insurance

• Vision insurance

• 401(k)

• Bonus pay eligibility

• Accessibility accommodations for applicants with disabilities

• Background checks conducted in accordance with local legislation

• Current employer will not be contacted without permission