Senior DataOps Engineer

This is a fully remote position, open to applicants in California, +1 more state.

📋 Description

• Contribute to the design, implementation, automation, maintenance, and operational support of large-scale cloud data platforms utilizing Databricks, AWS cloud services, and Infrastructure as Code methodologies.

• Lead the design, development, optimization, and operational oversight of enterprise-level ETL/ELT pipelines within Databricks.

• Create and sustain scalable batch and streaming data pipelines employing PySpark, Spark SQL, Delta Lake, and Databricks Workflows.

• Configure, optimize, and manage Databricks clusters to ensure performance, scalability, reliability, and cost-effectiveness.

• Collaborate with software development and systems teams to create data integration solutions.

• Design and develop data models using Lucid, Talend, Erwin, and MySQL Workbench.

• Define and improve the enterprise data model.

• Evaluate application data systems for compliance with data governance policies.

• Design and implement Python ETL/ELT infrastructure, automation, and data transformation solutions.

• Create and implement BI dashboards to visualize trends and forecasts.

• Design and deploy highly available, reliable, scalable, and efficient data infrastructure components.

• Develop, train, and deploy ML models.

• Implement monitoring solutions to proactively identify and resolve potential issues.

• Work collaboratively with security teams to adhere to industry standards and compliance requirements.

• Partner with product managers, developers, and business stakeholders to ensure robust and dependable systems.

• Meet with Scout colleagues in person and occasionally travel to company events.

⛳️ Requirements

• Bachelor's degree in computer science, information technology, or a related field, or equivalent professional experience.

• Over 8 years of hands-on experience as a DataOps Engineer in a manufacturing or automotive setting.

• Familiarity with streaming and event-driven architecture.

• Experience in implementing data lakehouse solutions utilizing Databricks.

• Proficient in infrastructure as code practices, specifically with Terraform.

• Skilled in building data pipelines using Python and SQL.

• Experience with AWS data services such as Glue, Kinesis, Firehose, or similar services.

• Proficient with structured, unstructured, and time series databases.

• Comprehensive understanding of cloud data storage solutions including RDS, DynamoDB, DocumentDB, MongoDB, Cassandra, and Influx.

• Several years of experience working with cloud platforms such as AWS and Azure.

• Proven expertise in developing and deploying scalable ML models.

• Practical experience in designing, training, and deploying ML models.

• Strong capability to extract actionable insights using machine learning techniques.

• Ability to utilize machine learning algorithms for forecasting trends and decision-making.

• Excellent problem-solving and troubleshooting abilities.

• Strong communication and collaboration skills.

• Applicants must be 18 years of age or older.

🏝️ Benefits

• Competitive medical, dental, vision, and income protection insurance plans.

• 401(k) program with employer match and immediate vesting.

• 20 days of planned PTO, accrued over time.

• 40 hours of unplanned PTO annually.

• 14 company or floating holidays each year.

• Up to 16 weeks of paid parental leave for biological and adoptive parents of all genders.

• Paid leave for bereavement, jury duty, voting time, or military leave.

• Competitive compensation that promotes physical, mental, and financial wellbeing.

• Opportunity to build products and contribute to the growth of a company from the ground up.