Senior Data Engineer

• Senior Data Engineer
• Plano, TX | On-site (Fridays from home after 90 days)

We are working with a client that develops and delivers technology solutions used by businesses across the U.S. With a focus on scalable, data-driven products, the company continues to invest in its technology and engineering capabilities.

• They are looking for an experienced Senior Data Engineer to join their team in Plano, Texas. You’ll play a key role in developing and optimizing a modern Azure data platform, working with high-volume datasets and building scalable solutions that support analytics and wider business needs.
• This is an on-site position, requiring you to commute to Plano, TX daily.

Responsibilities

• Design and develop scalable ETL/ELT pipelines using Azure Databricks, PySpark, and Spark SQL
• Build data ingestion workflows across sources including Event Hub, APIs, Azure SQL, and Blob Storage
• Develop and maintain Bronze, Silver, and Gold layers within a medallion architecture
• Implement best practices across Delta Lake and Unity Catalog, including schema evolution, versioning, and governance
• Optimize large-scale data pipelines for performance and cost through partitioning, caching, clustering, and Spark configuration tuning
• Integrate Databricks with ADLS Gen2, Azure Data Factory, Azure Synapse, Event Hub, Key Vault, and Azure DevOps
• Implement secure data access using managed identities, RBAC, and data masking
• Work with data analysts, data scientists, business stakeholders, architects, and cloud teams to deliver trusted datasets and KPIs
• Provide technical guidance and mentorship to junior engineers, including code reviews and architectural input
• Partner with enterprise architecture and information security teams to support data security and privacy requirements
• Implement data quality checks, alerts, and lineage tracking using tools such as Great Expectations, Unity Catalog, and data observability platforms
• Build monitoring, alerting, and recovery processes for mission-critical data pipelines

Qualifications

• Bachelor’s or Master’s degree in Computer Science, Engineering, or a related field
• 5+ years of data engineering experience
• 3+ years of hands-on experience with Azure Databricks and Spark/PySpark
• Strong expertise in Delta Lake, Unity Catalog, Databricks Notebooks, and SQL Warehousing
• Hands-on experience with ADLS Gen2, Event Hub, Azure Data Factory, Key Vault, and Azure Synapse
• Strong programming skills in Python and SQL, with familiarity with Scala
• Solid understanding of OLTP/OLAP data modelling, star and snowflake schemas, and performance tuning
• Experience implementing CI/CD for Databricks using Databricks CLI, Terraform, or Azure DevOps

Additional experience that would be beneficial includes

• Databricks certification, such as Databricks Certified Data Engineer Professional
• Real-time streaming technologies including Structured Streaming, Event Hub, or Kafka
• MLflow or supporting machine learning pipelines within Databricks
• Power BI, Azure Purview, or Microsoft Fabric
• Data governance, lineage, and compliance standards including GDPR, HIPAA, and SOC 2