Title: Senior Big Data Engineer (Kubernetes / AWS / Spark)
Location : DC Metro area, NYC Metro area (hybrid, 3x week)
Target Start Date : ASAP
Type: contract
Pay Rate: DOE
We are seeking an experienced Senior Big Data Engineer to help modernize and evolve a large-scale enterprise data platform supporting mission-critical analytics and data processing. This role is ideal for an engineer who enjoys solving complex data challenges, building cloud-native solutions, and working with massive datasets in distributed environments.
You will play a key role in modernizing existing big data infrastructure by migrating traditional data processing workloads to containerized, Kubernetes-based platforms while partnering with engineers across architecture, platform engineering, and data science. This position offers the opportunity to work with modern cloud technologies, AI-assisted development tools, and next-generation data engineering practices.
What You'll Do
Design, develop, and optimize highly scalable big data pipelines using Spark, Python, Scala, SQL, and distributed processing frameworks.
Modernize existing big data workloads by migrating EMR and EC2-based solutions to Kubernetes and cloud-native containerized architectures.
Build, deploy, and support Spark applications running on Kubernetes and Amazon EKS.
Engineer high-performance data solutions capable of processing terabyte- and petabyte-scale datasets.
Optimize data pipelines, Spark jobs, and distributed workloads for performance, scalability, resiliency, and cost efficiency.
Design data ingestion, transformation, and storage solutions that support enterprise analytics and reporting.
Troubleshoot production data pipelines, Kubernetes environments, and distributed processing issues.
Collaborate with architects, software engineers, analysts, and business stakeholders to translate requirements into scalable technical solutions.
Develop automated testing, CI/CD processes, and deployment pipelines to improve software quality and delivery.
Contribute to technical documentation, engineering standards, and solution design throughout the software development lifecycle.
Evaluate and leverage emerging technologies, including AI-assisted development tools, to improve engineering productivity.
Required Qualifications
5 years of experience building enterprise-scale Big Data solutions.
Strong hands-on experience with:
Kubernetes
Apache Spark
AWS cloud services
Python and/or Scala
Advanced SQL
Experience processing and optimizing large-scale distributed data workloads (terabytes or greater).
Strong understanding of Spark architecture, performance tuning, partitioning, resource management, and distributed computing concepts.
Experience designing, deploying, and supporting containerized applications in Kubernetes.
Experience with CI/CD pipelines using tools such as GitLab, Bitbucket, Jenkins, or similar platforms.
Strong problem-solving skills with the ability to optimize algorithms, troubleshoot complex production issues, and improve system performance.
Excellent communication skills with experience collaborating across technical and business teams.
Preferred Qualifications
Experience with Elasticsearch and distributed search technologies.
Experience with Hadoop, Hive, Trino, or similar big data ecosystems.
Experience with Amazon EKS, EMR on EKS, Glue, Athena, Lambda, or other AWS analytics services.
Familiarity with Infrastructure as Code (Terraform or CloudFormation).
Experience with monitoring and observability tools such as Prometheus, Grafana, ELK, or CloudWatch.
AWS, Kubernetes (CKA/CKAD), or Big Data certifications.
Financial services or other highly regulated industry experience.
Modern Engineering Environment
We're looking for engineers who embrace modern software development practices and continuously evaluate new technologies to improve delivery. Experience using AI-assisted engineering tools such as GitHub Copilot, ChatGPT, Claude, Amazon Kiro, Model Context Protocol (MCP), or agentic AI workflows is highly valued.
If you're passionate about cloud-native architecture, Kubernetes, large-scale data engineering, and building the next generation of enterprise data platforms, we'd love to hear from you.

Welcome to ConsultNet, a premier national provider of technology talent and solutions. Our expertise spans across project services, contract-to-hire, direct search, and managed services onshore, nearshore, and hybrid. For over 25 years, we have connected thousands of consultants with meaningful roles through a personal, communication-driven approach, partnering with a diverse client base to build high-performing teams and create lasting impact. Our comprehensive service offerings cover a wide range of technology and engineering positions across key markets nationwide. Learn more at www.consultnet.com .
We champion equality and inclusivity, proudly supporting an Equal Opportunity Employer policy. We welcome applicants regardless of Race, Color, Religion, Sex, Sexual Orientation, Gender Identity, National Origin, Age, Genetic Information, Disability, Protected Veteran Status, or any other status protected by law.

Apply now

Mention you found this on Data First Jobs — it helps us bring you more roles like this.

Senior Big Data Engineer (Kubernetes / AWS / Spark)

ConsultNet Technology Services and Solutions

Apply now