We are seeking a highly skilled and experienced Databricks SME to lead the architecture, design, and implementation of enterprise-scale Big Data solutions on the Databricks Lakehouse Platform. The ideal candidate will have at least 12 years of hands-on experience architecting big data solutions with Databricks, with proven expertise in migrating data and pipelines from legacy platforms like Alteryx, handling streaming data, and implementing CI/CD practices for data pipelines.
This role is critical for driving data modernization initiatives, ensuring robust data governance, and
enabling scalable data ingestion and transformation across multiple source systems into the Databricks
ecosystem.
Key Responsibilities
• Architect enterprise Big Data solutions on Databricks using Bronze/Silver/Gold (Medallion) layers
with Delta Lake and Delta Live Tables
• Migrate data and pipelines from legacy platforms (Alteryx, Informatica, DataStage, Talend etc) to
Databricks, including source-to-target mapping and data validation
• Design streaming data pipelines using Structured Streaming, Auto Loader, and Delta Live Tables
• Implement CI/CD practices for data pipelines using Git, Azure DevOps, and GitHub Actions

Driving Innovation, Empowering Insights
• Develop data governance framework using Unity Catalog ABAC/RBAC policies and collaborate with
stakeholders to translate requirements into scalable solutions
• Experience with Databricks Workflows, Jobs, and Databricks Asset Bundles for deployment
automation
• Mentor junior team members and provide technical leadership across data engineering projects
Required Skills
• 12+ years architecting Big Data solutions with Databricks Lakehouse Platform
• Proven experience migrating pipelines from Alteryx or similar legacy ETL tools
• Hands-on expertise with Delta Live Tables, Delta Lake, and Medallion architecture
• Strong experience building streaming pipelines with Structured Streaming and Auto Loader
• Demonstrated ability implementing CI/CD for data workloads
• Unity Catalog Data governence framework with ABAC / RBAC policies.
• Proficient in Python, SQL, PySpark, and strong experience in performance tuning of large-scale
distributed pipelines
• Proven experience on at least one major cloud platform — Azure (preferred) or AWS — and its native
data services
Preferred Skills
• Unity Catalog for data governance and lineage
• Data quality tools: Great Expectations, Informatica DQ, Collibra
• Data modeling tools: Erwin, dbt, ER/Studio
• AI/ML experience with GenAI, LLMs, MLOps
• Azure certifications in data engineering
Certifications (Preferred)
• Databricks Certified Data Engineer Associate/Professional
• Microsoft Certified: Azure Data Engineer Associate

Big Data Specialist

Similar Other Jobs

Data Steward 1

Data Center Controls Tech, Data Center Capacity Delivery - Controls

Data Entr Analys

Junior Data Analys

Consultant, Data Visualization & Communications Strategy

Digital & Data Services Supervisor