Data Engineer (Databricks, Azure, Kafka)

About the Role

We are seeking an experienced Data Engineer to design, develop, and maintain scalable data pipelines using Databricks, Apache Spark, and Azure cloud services. This remote position is suited to a data engineering professional with strong experience in cloud-based data platforms, automation, and event-driven architectures.

You will work with modern data engineering tools to build reliable ETL/ELT workflows, integrate data across platforms, and support AI and generative AI initiatives. The role also involves collaborating with data architects, analysts, and business stakeholders to translate business requirements into secure, efficient, and maintainable technical solutions.

Key Responsibilities

• Design, develop, and optimize scalable data pipelines using Databricks, Apache Spark, and PySpark.
• Build and maintain ETL/ELT workflows that integrate Snowflake and other data platforms.
• Develop and manage Azure-based data solutions using Azure Data Factory (ADF), Azure Data Lake Storage (ADLS), Azure Functions, and Azure Key Vault.
• Implement CI/CD pipelines and infrastructure automation using GitHub Actions, Terraform, and DevOps best practices.
• Build and maintain event-driven architectures using Kafka, including change data capture (CDC) and streaming integrations.
• Support AI and generative AI initiatives involving large language model (LLM) fine-tuning, retrieval-augmented generation (RAG), and prompt engineering.
• Monitor, troubleshoot, and optimize data pipeline performance, reliability, and efficiency.
• Collaborate with data architects, analysts, and business stakeholders to understand requirements and deliver appropriate technical solutions.
• Maintain data quality, security, and governance standards across data pipelines.
• Orchestrate and schedule data workflows using Apache Airflow.

Required Qualifications

• At least eight years of data engineering experience.
• Hands-on experience with Databricks, Apache Spark, PySpark, Python and/or Scala, SQL, and Snowflake.
• Experience with Azure cloud services, including Azure Data Factory, Azure Data Lake Storage, Azure Functions, Azure Key Vault, and Azure DevOps.
• Knowledge of AI and generative AI technologies, including LLM fine-tuning, RAG, and prompt engineering.
• Experience with Apache Airflow, GitHub, GitHub Actions, CI/CD, Terraform, and infrastructure automation.
• Experience with Kafka, event-driven architecture, and CDC or streaming integrations.

Preferred Qualifications

• Bachelor’s or master’s degree in Computer Science, Engineering, or a related field.
• Azure certifications, such as Azure Data Engineer Associate or Azure Solutions Architect.
• Experience working in large-scale enterprise environments or regulated industries, including finance, healthcare, or insurance.
• Strong communication skills and the ability to work directly with client stakeholders.

Work Arrangement & Employment Type

Work location: Remote

Employment type: W-2, C2C, or 1099.