Data Automation Engineer – AWS / Python / PySpark
Location: Remote; quarterly travel to Gaithersburg, MD
Schedule/Shift: Full-time, 40 hours per week, typically Monday–Friday during standard business hours; Eastern or Central Time Zone preferred
Pay: $55–$60 per hour

Position Overview
The Data Automation Engineer designs, develops, and maintains scalable cloud-based data pipelines and automation solutions using Python, SQL, AWS, and Apache Spark/PySpark. This role focuses on production data engineering, cloud data integration, pipeline optimization, and incorporating Generative AI and large language model capabilities into enterprise workflows. The ideal candidate has strong hands-on technical experience building and troubleshooting production data pipelines, integrating diverse data sources, and implementing reliable, secure, and scalable solutions.
Job Responsibilities
Design, develop, and maintain scalable ETL/ELT data pipelines using Python, SQL, AWS, and Apache Spark/PySpark.
Build automated data ingestion, transformation, and integration workflows across cloud platforms.
Develop and support solutions using AWS services such as S3, Glue, Lambda, EMR, Step Functions, and DynamoDB.
Create batch and near-real-time data pipelines using Kafka, Flume, or comparable data ingestion technologies.
Integrate Generative AI and large language model services into Python-based automation workflows.
Develop integrations between AWS, Microsoft Azure, SQL databases, REST APIs, and external data sources.
Troubleshoot complex data-processing issues and optimize pipeline performance, reliability, and scalability.
Implement continuous integration and continuous delivery practices using tools such as GitHub, Jenkins, or Azure DevOps.
Maintain data security, integrity, quality, and compliance with established technical standards.
Collaborate with cross-functional engineering teams and participate in Agile development activities.
Document technical solutions, data workflows, and troubleshooting procedures.
Job Requirements
Minimum 5 years of professional experience in data engineering, data automation, or related software development.
Hands-on experience designing, developing, and troubleshooting production data pipelines.
Strong Python and SQL development skills.
Demonstrated experience building and supporting production ETL/ELT pipelines in AWS environments.
Strong hands-on experience with Apache Spark and PySpark, including practical data transformation and processing.
Experience with multiple AWS data services, such as S3, Glue, Lambda, EMR, or Step Functions.
Working knowledge of Kafka, Flume, or comparable data ingestion technologies.
Practical experience integrating a Generative AI or large language model service into Python-based workflows.
Experience integrating REST APIs and external data sources.
Familiarity with Git, CI/CD practices, and Agile development methodologies.
Strong troubleshooting, analytical, problem-solving, and communication skills.
U.S. citizenship.
Ability to obtain a Public Trust clearance.
Preferred Qualifications
Experience integrating AWS and Microsoft Azure data environments.
Familiarity with Amazon Bedrock, Azure OpenAI, LangChain, or Hugging Face.
Experience with Docker, Kubernetes, or other containerized deployment technologies.
Knowledge of vector databases, Apache Solr, OpenSearch, or retrieval-augmented generation (RAG).
AWS, Azure, or Databricks certifications.
Experience developing secure, scalable data solutions in enterprise environments.
Education
Bachelor's degree in Computer Science, Engineering, or a related technical field.

Data Automation Engineer

Similar Engineering Jobs

AI Field Engineer / Forward Deployed Engineer (FDE)

Data Engineer

Data Visualization Developer

Power BI Developer

Senior Software Engineer / Data Engineer

Data Engineer W2 only