Senior Data Engineer

• Role - Data Engineer with Azure Databricks
• Location - Pittsburgh, PA
• Fulltime with Mastech Digital

Description

• Your future duties and responsibilities
• . Design, develop, and maintain scalable data pipelines using Azure Databricks.
• . Build and optimize ETL/ELT workflows leveraging PySpark, SQL, and Delta Lake.
• . Implement and manage Unity Catalog to support enterprise data governance, security, metadata management, and fine-grained access control.
• . Develop cloud native data engineering solutions on the Microsoft Azure platform.
• . Collaborate with architects, business analysts, and cross functional teams to gather and translate business requirements into technical solutions.
• . Optimize Spark jobs and data pipelines for performance, scalability, and cost efficiency.
• . Ensure data quality, reliability, and compliance through validation, monitoring, and best practices.
• . Support production deployments, troubleshooting, and performance tuning.
• . Participate in code reviews, technical documentation, and continuous improvement initiatives.
• . Apply best practices for cloud data engineering, security, and DevOps within Azure environments.

• Required qualifications to be successful in this role
• . Minimum 5 years of hands-on experience with Azure Databricks.
• . Hands on experience with Unity Catalog is mandatory.
• . Strong experience building enterprise scale data engineering solutions on Microsoft Azure.
• . Expert level experience with Apache Spark (PySpark) and SQL.
• . Strong experience with Delta Lake, ETL/ELT development, and data pipeline implementation.
• . Experience with Azure data services such as Azure Data Factory, Azure Data Lake Storage (ADLS Gen2), and related Azure ecosystem services.
• . Strong understanding of data governance, metadata management, security, and access control.
• . Mandatory experience in the Investment/Capital Markets domain.
• . Experience with Git and CI/CD practices.
• . Excellent analytical, communication, and problem solving skills.
• . Experience working in Agile/Scrum delivery environments.

• Preferred Qualifications
• . Databricks Certified Data Engineer (Associate or Professional).
• . Microsoft Azure Data Engineer Associate (DP 203) certification.
• . Experience with Infrastructure as Code (Terraform/Bicep).
• . Experience implementing DevOps practices for Azure Databricks environments.
• . Familiarity with Azure Purview/Microsoft Purview is a plus.