Sr. AWS Data engineer

Hi

• Job Title: Senior Data Engineer – Scala to PySpark Conversion | AWS
• Location : New Jersey, New York, or Fort Mill SC (Hybrid)
• Please share only 14+ years profiles
• Note: H4 EAD, USC, L2 EAD Only (W2)

Please don't share OPT or H1B and Don't waste your time by share OPT or H1B profiles

Need someone who has started as a Big Data or Scala Developer and now a Senior AWS Data Engineer, with strong PySpark coding with Scala job work.

• Key Responsibilities
• Analyze and understand existing Scala/Spark-based ETL pipelines and data processing applications.
• Convert and migrate Scala Spark code to PySpark, ensuring functional and data processing equivalence.
• Refactor existing Spark jobs to improve performance, scalability, maintainability, and code quality.
• Design, develop, and maintain ETL pipelines using AWS Glue, Glue Studio, and Glue Data Catalog.
• Build and optimize PySpark applications for large-scale data processing and complex transformations.
• Work with AWS S3, Redshift, Athena, Lambda, and Step Functions for data storage, processing, querying, and workflow orchestration.
• Migrate existing Scala-based data pipelines to PySpark while maintaining data accuracy and business logic.
• Perform code reviews and identify opportunities to optimize Spark transformations, joins, partitioning, caching, and resource utilization..
• Ensure data solutions comply with security, governance, and regulatory requirements, preferably within the BFSI domain.
• Participate in Agile/Scrum ceremonies and contribute to technical design and solution discussions.

• Key Skills
• Primary: PySpark, Python, Scala, Apache Spark, AWS Glue
• AWS: S3, Glue, Glue Data Catalog, Redshift, Athena, Lambda, Step Functions, CloudWatch
• Data: ETL/ELT, Data Lakes, Data Warehousing, Data Modeling, Data Quality
• Migration: Scala-to-PySpark Conversion, Spark Modernization, Code Refactoring, Performance Optimization
• Database: SQL, Redshift, Relational Databases
• Methodology: Agile, Scrum, CI/CD