Data Engineer

• Data Engineer
• King of Prussia, PA - 2 days onsite in week
• Fulltime Opportunity
• in-person required.

Must Have Skills:

Big Data | Cloud/On Prem | DevOps | CI/CD |Data Services/APIs/Microservices | DataBricks |(PySpark, Spark SQL, Delta Lake)| Spark | Azure Cloud Services | ETL/ELT.

As a Senior Data Engineer, you will play a key role in driving the design and architecture of data pipelines that consume real-time streaming data from connected medical devices. You will collaborate cross-functionally with business analysts, software engineers, machine learning engineers, and business users to implement technical approaches and infrastructure that support data consumption across the organization.

This is a great opportunity for someone who is passionate about data engineering, thrives in environments where you can take ownership and recommend solutions, and is a self-starter who is open to learning new tools as needed.

Description:

• 5+ years of experience designing, building, and operating big data and real-time streaming pipelines across cloud and on-premises environments
• 5+ years applying DevOps and CI/CD practices to data and analytics workloads
• Production experience building data services, APIs, or microservices for downstream data consumption
• Strong, hands-on experience designing and building production-grade data pipelines on Databricks
• Demonstrated experience with Spark optimization and tuning in Databricks, including performance analysis, partitioning strategies, caching, shuffle optimization, and cost-aware pipeline design
• Strong experience with Azure cloud services for data engineering and streaming workloads
• Experience building data services, APIs, or microservices for data consumption.
• Data quality, validation, and privacy-aware handling for regulated or sensitive data.
• Design and build batch and streaming data pipelines on Azure and Databricks, with primary focus on real-time IoT and telemetry use cases within a Medallion architecture.
• Develop ETL/ELT workflows to ingest, transform, and validate large volumes of structured and unstructured data.
• Build and maintain data services, APIs, and microservices for application, analytics, and ML/AI teams.
• Implement real-time streaming solutions using Azure Event Hubs, Azure Stream Analytics, and related Azure integration patterns, with cost-effective throughput, partitioning, and downstream delivery to Databricks.
• Optimize production Databricks pipelines using PySpark, Spark SQL, and Delta Lake, including Spark tuning for performance, reliability, and cost.
• Troubleshoot and resolve complex pipeline issues across Databricks, Azure, and on-premises systems, including root cause analysis and corrective action.
• Partner with data analysts, software engineers, ML engineers, and business stakeholders to translate requirements into technical designs and delivery priorities.
• Apply data quality, validation, and privacy-first practices, and deliver reliable pipelines through software engineering standards, documentation, testing, and CI/CD.