• Design and develop Big Data pipelines using Spark and Scala to process large volumes of raw data stored in Hive/Hadoop.
• Analyze raw Hive tables, source schemas, data dictionaries and source-to-target mappings to understand business data.
• Develop Spark/Scala transformation frameworks to convert raw source data into standardized conformed data views/tables.
• Implement complex business transformation rules including joins, filters, aggregations, derivations and reference-data lookups.
• Standardize data across multiple source systems to create a consistent enterprise/conformed data model.
• Implement incremental processing, partition-based processing and historical data handling for high-volume datasets.
• Develop reusable Spark/Scala components for data enrichment, transformation and validation.
• Implement data-quality controls for completeness, uniqueness, validity, referential integrity and business-rule validation.

Big Data Engineer

Similar Engineering Jobs