Data Engineer

Overview

We are seeking a Data Engineer to help build and scale a modern cloud data platform. This role will design data pipelines, integrate internal and external data sources, and develop scalable solutions using Azure Databricks, AWS, Python, and SQL. The Data Engineer will collaborate with Data Scientists, Business Analysts, and business stakeholders to transform raw data into reliable, actionable insights.

Responsibilities

• Design, develop, and maintain scalable ETL/ELT pipelines using Azure Databricks, PySpark, and SQL.
• Integrate structured and unstructured data from internal systems, third-party providers, and cloud platforms.
• Design and optimize data models for reporting, business intelligence, analytics, and machine learning.
• Monitor and troubleshoot data pipelines to improve performance, reliability, and scalability.
• Implement data quality controls, security standards, and governance best practices.
• Build and maintain AWS-based data solutions using S3, Glue, Lambda, IAM, and CloudWatch.
• Automate file ingestion into Amazon S3 and orchestrate downstream data processing.
• Develop internal applications, APIs, automation tools, and reusable utilities using Python.
• Prepare high-quality datasets for advanced analytics and AI initiatives.
• Partner with business, analytics, and IT teams to define requirements and deliver enterprise data solutions.
• Support future container-based deployments using ECS, EKS, Kubernetes, or similar technologies.

• Required Skills
• Bachelor’s degree in Computer Science, Information Systems, Data Engineering, Software Engineering, or a related field.
• Three or more years of experience in data engineering or a similar role.
• Hands-on experience with Azure Databricks and AWS.
• Strong Python programming and advanced SQL development skills.
• Experience building and supporting scalable ETL/ELT pipelines.
• Strong understanding of relational databases, data modeling, and data warehousing.
• Experience with Git or another source-control platform.
• Ability to collaborate with technical teams and business stakeholders.

• Desired Skills
• Experience in consumer packaged goods, food and beverage, retail, distribution, or manufacturing.
• Experience working with NielsenIQ, Circana/IRI, SPINS, retailer point-of-sale data, or similar market data providers.
• Experience with PySpark, Spark SQL, Delta Lake, or Azure Data Factory.
• Familiarity with Snowflake, Amazon Athena, or Amazon Redshift.
• Experience with Docker, Kubernetes, ECS, EKS, or other container technologies.