Solutions Architect, Databricks

This is a fully remote position, open to applicants in United States.

📋 Description

• Design and develop scalable architectures for Databricks data lakehouse solutions tailored for federal data initiatives.

• Create and enhance end-to-end data pipelines that meet federal standards for performance, security, and compliance.

• Establish data governance and security measures in accordance with federal regulations, including FedRAMP, FISMA, and NIST.

• Collaborate with federal stakeholders to gather and analyze requirements, translating them into practical data solutions.

• Integrate Databricks with AWS GovCloud, Azure Government, and various other federal cloud services.

• Provide expert knowledge on Databricks, Delta Lake, Databricks SQL, and Apache Spark.

• Optimize Databricks clusters for cost-effectiveness, high availability, and peak performance.

• Facilitate advanced analytics and machine learning workflows utilizing MLflow and associated frameworks.

• Create real-time data processing solutions for critical federal applications.

• Oversee federal data policies and ensure compliance with relevant regulations.

• Guide cross-functional teams and mentor technical teams on best practices for Databricks.

⛳️ Requirements

• 5–10 years of experience in data engineering, data architecture, or distributed data processing.

• A minimum of 3 years of practical experience with Databricks.

• Proficient in Databricks SQL, Delta Lake, and the Lakehouse platform.

• Expertise in Apache Spark and frameworks for distributed data processing.

• Advanced knowledge of programming languages such as Python, Scala, or Java.

• Practical experience with AWS GovCloud, Azure Government, or Google Cloud Public Sector.

• Strong understanding of ETL processes, data workflows, and data transformation pipelines.

• In-depth knowledge of data lakes, data warehouses, data modeling, and schema design.

• Experience in implementing data governance and security measures in regulated settings.

• Familiarity with FISMA, FedRAMP, NIST 800-53, and CMMC.

• Experience in securing federal data architecture, including encryption and access controls.

• Advanced SQL skills for managing and querying large relational datasets.

• Knowledge of NoSQL databases.

• Ability to lead cross-functional teams to deliver intricate data solutions.

• Excellent interpersonal and communication skills to engage effectively with federal stakeholders.

• Experience in training and mentoring technical teams on best practices for Databricks.

• Preferred: Familiarity with CI/CD pipelines and Databricks workflow/notebook deployment.

• Preferred: Experience with Infrastructure-as-Code tools like Terraform, CloudFormation, or ARM templates.

• Preferred: Knowledge of machine learning pipelines and Databricks MLflow.

• Preferred: Experience with TensorFlow, PyTorch, or Scikit-learn.

• Preferred: Familiarity with Kafka, Azure Event Hubs, or AWS Kinesis.

• Preferred: Experience with Hadoop, Flink, or Elasticsearch.

• Preferred: Knowledge of real-time analytics and event-driven architectures.

• Preferred: Capability in preparing technical documentation, reports, and presentations for federal audiences.

• Advanced Databricks certifications are preferred.

• Federal cloud and cybersecurity certifications such as AWS Certified Security – Specialty, CCSP, CISSP, or CISM are preferred.

• Eligibility for Public Trust or higher clearance and prior experience with federal clients are preferred.

🏝️ Benefits

• 401(k) matching

• Dental insurance

• Health insurance

• Paid time off

• Parental leave

• Vision insurance