Senior Data Engineer

📋 Description

• Design and construct scalable organizational data infrastructure and Medallion architecture within a Lakehouse environment.

• Develop resilient, fault-tolerant ETL/ELT applications to facilitate seamless data ingestion, transformation, and distribution for analytics, reporting, and AI workloads.

• Collaborate with various stakeholders and teams to provide technical solutions related to data and support their data infrastructure requirements.

• Investigate and experiment with new use cases, frameworks, and tools to enhance AI capabilities while ensuring data integrity, quality, and reliability.

• Identify and execute infrastructure redesigns to enhance scalability, optimize data delivery, and automate manual workflows.

• Partner with cross-functional teams to comprehend data requirements, create robust data models, and deliver actionable insights.

• Monitor, troubleshoot, and optimize tasks for performance, addressing data pipeline bottlenecks and ensuring cost efficiency.

⛳️ Requirements

• Bachelor’s or master’s degree in computer science, engineering, or a related discipline.

• Over 5 years of demonstrated experience in data engineering, delivering business-critical software solutions for large enterprises with a solid track record of success.

• Proficient in writing ETL/ELT jobs.

• Familiarity with Azure and the Databricks Platform.

• Experience with Python, SQL, and REST APIs.

• Understanding of cloud architecture principles including compute, storage, networks, security, and cost.

• Skilled in utilizing open-source tools and frameworks such as FastAPI, Pydantic, Polars, Pandas, Delta Lake, Docker, and Kubernetes.

• Knowledge of CI/CD, Git, or infrastructure-as-code concepts.

• Strong project management abilities, capable of prioritizing tasks and managing multiple projects concurrently in an Agile environment.

• Understanding of how data engineering integrates with Business Intelligence and reporting tools (Power BI/Tableau).

• Excellent problem-solving and analytical skills.

• Strategic thinker with a strong orientation towards execution.

• Ability to collaborate effectively within cross-functional teams.

• Meticulous attention to detail and commitment to data quality.

🏝️ Benefits

• Healthcare Coverage: Comprehensive medical, dental, and vision plans.

• Time Off and Leave Policies: Generous paid time off (PTO), paid company holidays, and generous parental and family leave.

• Protective Insurances: Life insurance, short- and long-term disability coverage, and accident protection.

• Compensation and Rewards: Competitive salary structures, performance-based incentives, and merit-based compensation reviews.

• Retirement Plans: 401(k) plans with company matching.