Senior Data Engineer

This is a fully remote position, open to applicants in Pennsylvania.

📋 Description

• Create and execute scalable enterprise data architectures utilizing Databricks and AWS platforms.

• Establish data architecture patterns for data ingestion, processing, storage, transformation, serving, and consumption.

• Design and enhance data models, data marts, data warehouses, and enterprise data platforms.

• Develop and enforce data governance, security, quality, lineage, and access-control frameworks.

• Build and refine data processing pipelines using PySpark, SQL, and ETL frameworks.

• Collaborate with Databricks for data engineering, analytics, AI/ML, and data product enablement.

• Create cloud-native and serverless solutions on AWS.

• Architect and integrate solutions leveraging APIs and API-based data integrations.

• Utilize AWS Bedrock to design and implement GenAI and AI-agent-based solutions.

• Offer architectural guidance for AI Agents, Databricks Genie, and enterprise GenAI use cases.

• Design reusable, scalable, and governed data products.

• Conduct SQL query optimization, performance tuning, and data-processing enhancements.

• Collaborate with Data Engineers, Data Scientists, AI/ML Engineers, Product Owners, and business stakeholders.

• Define architecture standards, reusable patterns, technical documentation, and best practices.

• Perform technical reviews and mentor development teams on architecture, performance, scalability, and maintainability.

• Ensure solutions comply with enterprise security, governance, compliance, and cloud architecture standards.

⛳️ Requirements

• Over 8 years of overall experience in Data Engineering / Data Architecture / Cloud Architecture.

• More than 4 years of experience with Databricks and contemporary cloud data platforms.

• Hands-on expertise with AWS and modern data/AI technologies.

• Extensive experience in Databricks architecture and development.

• Strong practical knowledge of the AWS cloud platform and its services.

• In-depth understanding of Enterprise Data Architecture.

• Proficient in PySpark and SQL.

• Experience with AWS Bedrock and GenAI/AI-agent solutions.

• Strong knowledge of data modeling, data governance, DBMS, data warehouses, and data marts.

• Experience in ETL/data pipeline architecture and development.

• Proficient in SQL query optimization and performance tuning.

• Familiarity with designing and utilizing REST/API-based integrations.

• Experience in architecting and enabling enterprise data products.

• Understanding of serverless architecture and cloud-native design patterns.

• Experience with Databricks Genie / Genie Spaces.

• Familiarity with AI Agents / Agentic AI frameworks and architectures.

• Experience with AWS serverless services such as Lambda, API Gateway, Glue, Step Functions, and related services.

• Experience with Delta Lake / Lakehouse architecture.

• Knowledge of Data Catalog, metadata management, lineage, and data quality frameworks.

• Experience with CI/CD and DevOps practices for data and AI platforms.

• Exposure to LLM application architecture, RAG, vector databases, and prompt engineering.

• Experience in large-scale enterprise data transformation or modernization programs.

• Proven experience in designing and delivering enterprise-scale data solutions.

• Experience leading technical discussions and collaborating with cross-functional stakeholders.

🏝️ Benefits

• Employees have the flexibility to work remotely.

• Equal employment opportunities provided without discrimination.

• A welcoming and inclusive work environment.

• Commitment to diversity.