- We are seeking a highly skilled Databricks Architect to lead the design and implementation of next-generation data platforms built on the Lakehouse paradigm.
- This role goes beyond pipeline development—you will own the Databricks platform architecture end-to-end, driving scalability, governance, performance, and cost optimization across enterprise data ecosystems.
- Databricks Platform Architecture
- Architect and implement enterprise-scale Databricks environments (dev/test/prod)
- Define workspace strategy, cluster policies, and job orchestration frameworks
- Design secure, scalable Lakehouse architecture using Delta Lake
- Data Engineering & Processing
- Build and optimize high-performance data pipelines using Apache Spark (PySpark / Scala)
- Implement Medallion Architecture (Bronze, Silver, Gold)
- Develop batch and real-time streaming pipelines using Structured Streaming
- Governance, Security & Compliance
- Implement fine-grained access control using Unity Catalog
- Define enterprise-wide data governance, lineage, and auditing frameworks
- Ensure compliance with security and regulatory standards
- Performance & Cost Optimization
- Optimize workloads using partitioning, caching, and Photon engine
- Design cost-efficient cluster strategies (autoscaling, spot instances, DBU optimization)
- Monitor and improve query and pipeline performance at scale
- DevOps & Automation
- Implement CI/CD pipelines for Databricks using Azure DevOps, Jenkins, or GitHub Actions
- Enable Infrastructure as Code using Terraform or equivalent
- Standardize reusable frameworks and engineering best practices
- Cloud Integration
- Architect Databricks solutions on AWS / Azure / GCP
- Integrate with cloud-native services (AWS S3/Glue, Azure ADLS/ADF, etc.)
- Must-Have Skills
- Advanced proficiency in SQL and Python
- Hands-on experience with Delta Lake, Medallion Architecture, and Unity Catalog
- Strong experience with at least one cloud platform (AWS / Azure / GCP)
- Experience with CI/CD tools (Azure DevOps, Jenkins, GitHub Actions)
- Knowledge of workflow orchestration tools (Airflow or equivalent)
- Good to Have
- Databricks certifications (Associate / Professional)
- Experience with Databricks SQL & dashboards
- Exposure to ML pipelines (MLflow)
- Experience with Kafka or event-driven architectures
- Domain experience in Finance / Retail / Healthcare
- What Makes You a Great Fit
- Strong architectural mindset with ability to design scalable data platforms
- Deep understanding of performance tuning and cost optimization
- Ability to balance technical depth with business impact
- Excellent communication and stakeholder management skills
Mention you found this on Data First Jobs — it helps us bring you more roles like this.
Data Architect
VySystems
Similar Other Jobs
View all Other jobs→CSZNet Inc.
Data Platform Architect
New
Harrisburg, Pennsylvania (USA)
S.A. Comunale Co., Inc.
Senior Project Manager - Data Centers (Mission Critical | Fire Protection)
New
Barberton, Ohio (USA)
Carta Healthcare
Full-Time STS Clinical Data Abstractor (Hourly) Non-Exempt
New
RemoteUSA
Meta
Data Collection Program Manager
New
Fremont, California (USA)$99,000 - $144,000
Largeton Group
AWS Data Migration Lead
New
St. Louis, Missouri (USA)$60,000 - $60,000
Largeton Group
Sr Data Center Technician
New
Beaverton, Oregon (USA)$40,000 - $50,000
Like this role? Get carefully selected jobs like it, twice a week, straight to your inbox.
Free, no spam. Unsubscribe anytime.