Position Summary
The SDET (Software Development Engineer in Test) will be responsible for ensuring the quality, reliability, and integrity of ETL processes and data integration workflows. The role involves designing and implementing automated tests to validate data transformation and migration, while collaborating with data engineers and developers to identify and resolve data quality issues.
The ideal candidate will have strong expertise in ETL testing, data validation, automation, PySpark, AWS, and Big Data ecosystems to enhance overall data pipeline performance and integrity.
Minimum Qualifications
At least 8 years of IT experience, with a minimum of 5 years of experience in the relevant skills and technologies.
Strong proficiency in ETL Testing and developing automation frameworks and test cases using at least one scripting language.
Strong SQL skills, with experience handling Big Data using ETL tools such as Informatica.
Mandatory hands-on experience with PySpark.
Experience with testing in AWS cloud environments.
Experience with the following AWS services:
S3
SSS
AWS Glue
EMR
Data Catalog
Crawler
SQS
ETL Jobs
EventBridge
Athena
Redshift Spectrum
Experience using AWS commands for running jobs, file processing, and file movements.
Experience with Test Automation and DevOps practices in ETL and cloud environments.
Experience working within a Big Data ecosystem and processing data in a data lakehouse.
Experience providing testing oversight within Agile teams.
Hands-on experience with large-scale test data generation, preparation, validation, and reconciliation, including Big Data and high-volume datasets across complex testing scenarios.
Excellent business communication skills to interact effectively with business and cross-functional teams.
Key Responsibilities
Review requirements, specifications, and technical design documents and provide feedback.
Develop and implement test strategies for complex initiatives.
Estimate, prioritize, plan, and coordinate testing activities.
Create and maintain automation test frameworks.
Design, develop, and execute automated test scripts.
Identify, document, track, and manage defects throughout the testing lifecycle.
Perform troubleshooting, root cause analysis, and problem resolution.
Conduct thorough regression testing after defect resolution.
Develop and implement testing processes for new and existing features to meet client requirements.
Collaborate with cross-functional teams to identify and define system requirements.
Track QA metrics, including defect density, open defect counts, and other quality indicators.
Stay current with emerging testing tools, technologies, and test strategies.
Support and mentor junior team members.
Ensure ETL and data integration workflows meet quality, performance, and data integrity requirements.
Technical Skills
ETL Testing
Test Automation
Automation Framework Development
SQL
PySpark
Informatica
AWS
Amazon S3
AWS Glue
AWS EMR
AWS Data Catalog & Crawler
Amazon SQS
AWS EventBridge
Amazon Athena
Amazon Redshift Spectrum
Big Data
Data Lakehouse
Data Validation & Reconciliation
DevOps
Agile Testing