Data First Jobs

VySystems

Data Architect

Contract · In Office · New Jersey (USA)

Posted Aug 12, 2026

  • We are seeking a highly skilled Databricks Architect to lead the design and implementation of next-generation data platforms built on the Lakehouse paradigm.
  • This role goes beyond pipeline development—you will own the Databricks platform architecture end-to-end, driving scalability, governance, performance, and cost optimization across enterprise data ecosystems.
  • Databricks Platform Architecture
  • Architect and implement enterprise-scale Databricks environments (dev/test/prod)
  • Define workspace strategy, cluster policies, and job orchestration frameworks
  • Design secure, scalable Lakehouse architecture using Delta Lake
  • Data Engineering & Processing
  • Build and optimize high-performance data pipelines using Apache Spark (PySpark / Scala)
  • Implement Medallion Architecture (Bronze, Silver, Gold)
  • Develop batch and real-time streaming pipelines using Structured Streaming
  • Governance, Security & Compliance
  • Implement fine-grained access control using Unity Catalog
  • Define enterprise-wide data governance, lineage, and auditing frameworks
  • Ensure compliance with security and regulatory standards
  • Performance & Cost Optimization
  • Optimize workloads using partitioning, caching, and Photon engine
  • Design cost-efficient cluster strategies (autoscaling, spot instances, DBU optimization)
  • Monitor and improve query and pipeline performance at scale
  • DevOps & Automation
  • Implement CI/CD pipelines for Databricks using Azure DevOps, Jenkins, or GitHub Actions
  • Enable Infrastructure as Code using Terraform or equivalent
  • Standardize reusable frameworks and engineering best practices
  • Cloud Integration
  • Architect Databricks solutions on AWS / Azure / GCP
  • Integrate with cloud-native services (AWS S3/Glue, Azure ADLS/ADF, etc.)
  • Must-Have Skills
  • Advanced proficiency in SQL and Python
  • Hands-on experience with Delta Lake, Medallion Architecture, and Unity Catalog
  • Strong experience with at least one cloud platform (AWS / Azure / GCP)
  • Experience with CI/CD tools (Azure DevOps, Jenkins, GitHub Actions)
  • Knowledge of workflow orchestration tools (Airflow or equivalent)
  • Good to Have
  • Databricks certifications (Associate / Professional)
  • Experience with Databricks SQL & dashboards
  • Exposure to ML pipelines (MLflow)
  • Experience with Kafka or event-driven architectures
  • Domain experience in Finance / Retail / Healthcare
  • What Makes You a Great Fit
  • Strong architectural mindset with ability to design scalable data platforms
  • Deep understanding of performance tuning and cost optimization
  • Ability to balance technical depth with business impact
  • Excellent communication and stakeholder management skills

Mention you found this on Data First Jobs — it helps us bring you more roles like this.

Data Architect

VySystems

Like this role? Get carefully selected jobs like it, twice a week, straight to your inbox.

Free, no spam. Unsubscribe anytime.