Sr Data Engineer

• Role: Senior Data Engineer
• Location: Atlanta, Georgia, United States
• Employment Type: Not specified
• Payout: Competitive, based on experience.

• Role Overview
• The Senior Data Engineer is responsible for designing and building scalable data platforms and AI-ready data pipelines that support real-time analytics, machine learning, and intelligent automation. This role partners with cross-functional teams to transform large-scale IoT and enterprise data into reliable, high-quality datasets that power advanced AI solutions.

• Key Responsibilities
• Design and implement scalable data architectures for processing high-volume IoT sensor and telemetry data.
• Build and maintain data pipelines supporting AI model training, feature engineering, inference, and production workloads.
• Develop and optimize Retrieval-Augmented Generation (RAG) solutions, including vector databases, embedding pipelines, and retrieval mechanisms.
• Lead the architecture and development of scalable data platforms using Databricks.
• Integrate Generative AI capabilities into enterprise data workflows and applications.
• Implement DataOps practices, including CI/CD pipelines, automated testing, data quality validation, and monitoring.
• Collaborate with data scientists and ML engineers to support efficient model training, deployment, and lifecycle management.
• Mentor team members, establish engineering best practices, and optimize data processing for performance, scalability, reliability, and cost.

• Required Skills & Qualifications
• Minimum 5 years of experience building production-grade data pipelines using Databricks for large-scale data processing.
• Strong expertise with PySpark for distributed data processing and transformation.
• Experience implementing Medallion Architecture (Bronze/Silver/Gold), Delta Lake, Delta Live Tables (DLT), and Lakeflow.
• Experience working with Azure, Google Cloud Platform (GCP), and Databricks for AI/ML data workflows.
• Hands-on experience with Event Hub, Kafka, Git, GitHub Actions, Databricks Asset Bundles (DAB), and DataOps practices.
• Experience building RAG applications using vector databases, LangChain, LangGraph, and large language model integrations.
• Knowledge of Apache Spark Streaming, Structured Streaming, MLOps, and AI data pipeline development.
• Familiarity with Docker, Kubernetes, time-series databases, data governance, and Agile/Scrum methodologies.
• Strong analytical, troubleshooting, communication, and collaboration skills.
• Must be legally authorized to work in the United States in compliance with applicable U.S. export control regulations.

• More About the Opportunity
• This opportunity offers the chance to work on cutting-edge AI, machine learning, and industrial IoT solutions while building scalable cloud-based data platforms. You'll collaborate with multidisciplinary teams to develop next-generation data infrastructure supporting advanced analytics and intelligent automation.

• Equal Opportunity Employer
• We hire based on skills and expertise. All qualified candidates are welcome regardless of background, experience, or prior employment history. Applications are reviewed solely on demonstrated technical ability and qualifications.

Apply Now!