Senior PySpark & Databricks Data Engineer

• Location: Atlanta, Georgia - Hybrid 3 days/onsite
• Role Summary :
• We are looking for a skilled Data Engineer with strong hands-on experience in PySpark and Databricks to design, build, and maintain scalable data pipelines. The ideal candidate has deep expertise in stream
• processing with Apache Kafka, workflow orchestration with Apache Airflow, data warehousing on Amazon Redshift, and building cloud-native solutions on AWS.

Key Responsibilities

• - Design and develop scalable ETL/ELT pipelines using PySpark on Databricks
• - Build and maintain real-time data streaming pipelines using Apache Kafka
• - Orchestrate and schedule data workflows using Apache Airflow (DAG development, monitoring, and troubleshooting)
• - Manage and optimize data models and queries in Amazon Redshift
• - Architect and deploy data solutions on AWS using services such as S3, Glue, Lambda, EMR, IAM, and CloudWatch
• - Collaborate with analysts and platform teams to deliver high-quality data products
• - Monitor pipeline performance and implement tuning strategies for large-scale data workloads
• - Implement data quality checks, observability, and alerting across pipelines
• - Participate in code reviews and contribute to engineering best practices
• - Document data flows, architecture decisions, and pipeline logic

Required Skills & Experience

• 🔹 PySpark — 3+ years
• DataFrame API, Spark SQL, query optimizations and performance tuning

• 🔹 Databricks — 3+ years
• Notebooks, Jobs, Delta Lake, Unity Catalog

• 🔹 Apache Kafka — 2+ years
• Producers/consumers, Kafka Streams, schema registry

• 🔹 Apache Airflow — 2+ years
• DAG authoring, task dependencies, operators, scheduling

• 🔹 Amazon Redshift — 2+ years
• Data modeling, query tuning, Redshift Spectrum

• 🔹 AWS — 3+ years
• S3, Glue, Lambda, EMR, IAM, CloudWatch, VPC

• 🔹 Python — 4+ years
• Strong scripting and engineering fundamentals

• 🔹 SQL — Strong
• Complex queries, window functions, performance tuning

• ---
• Nice to Have

• - Experience with Delta Lake
• - Familiarity with dbt for transformation layers
• - Knowledge of Databricks Workflows alongside Airflow
• - Exposure to Confluent Platform or AWS MSK (Managed Kafka)
• - Experience with Terraform or AWS CDK for infrastructure-as-code
• - Understanding of data governance, security best practices, and lake house architecture

• ---
• Qualifications

• - Bachelor's or Master's degree in Computer Science, Engineering, or a related field (or equivalent practical experience)
• - 4–7 years of overall experience in data engineering roles
• - Strong problem-solving skills and ability to work independently in an agile environment