Senior AI Data Engineer

This is a fully remote position, open to applicants in United States.

📋 Description

• Assess and enhance the quality, completeness, and consistency of data across more than 30 databases, applications, platforms, and APIs.

• Create and develop Scorpion's analytical data platform, establishing a reliable source of truth for business and client data.

• Design and maintain scalable data pipelines that effectively transfer data from operational systems to the analytical platform.

• Assist teams in transitioning from direct queries on production databases to utilizing trusted analytical data sources.

• Create data access patterns that allow AI agents and applications to swiftly obtain pertinent client and business information.

• Develop secure, scalable solutions that provide client context and business insights to internal applications and AI-driven experiences.

• Define, monitor, and enhance service level agreements (SLAs) for data freshness, availability, reliability, and performance.

• Collaborate with engineering, product, and data science teams to set data standards, governance practices, and data contracts.

• Continuously enhance the scalability, performance, and reliability of Scorpion's data ecosystem.

⛳️ Requirements

• A Bachelor's degree in Computer Science, Data Engineering, Information Systems, Software Engineering, or a related technical discipline, or equivalent practical experience.

• Over 7 years of data engineering experience, including the design and operation of production-scale analytical data platforms.

• Experience in constructing scalable data pipelines, analytical systems, and data platforms within modern cloud environments.

• Proven experience in integrating and unifying data from multiple systems into reliable, business-ready analytical platforms.

• Background in supporting data access and consumption patterns for applications, analytics, machine learning, or AI-driven solutions.

• Expertise in designing scalable data models that facilitate analytics, reporting, and AI-driven applications.

• Experience in establishing data governance standards, data contracts, and documentation practices across teams.

• Extensive knowledge of Databricks, Delta Lake, and Apache Spark for large-scale data processing, streaming ingestion, and data transformation.

• Familiarity with Snowflake, BigQuery, ClickHouse, Azure Synapse, or similar analytical database technologies.

• Proficient SQL skills, including query optimization, execution planning, and performance tuning.

• Strong Python skills for the development of data pipelines, automation, transformation, and integration with production systems.

• Experience in building and managing data pipelines using dbt, Airflow, Prefect, or similar orchestration and transformation tools.

• Comprehensive understanding of lakehouse architecture, data partitioning strategies, metadata management, and best practices for data security.

🏝️ Benefits

• 100% employer-covered medical, dental, and vision insurance.

• Flexible paid time off to ensure you can rest, relax, and recharge away from work.

• Paid parental leave.

• Reimbursement for cell phone and service expenses.

• Remote office allowance.

• Opportunities for professional development and training courses.

• Regular check-ins with your manager to promote performance and career growth through Lattice.