Data Engineer

Security Clearance Requirements:

Public Trust background investigation required. U.S. citizenship required. Must be stated in your resume. Must be eligible for adjudication.

Overview:

The Senior Data Engineer will design, implement, and maintain the Azure-based data infrastructure supporting a federal oversight and investigative analytics mission focused on fraud detection, audit support, and cross-program data integration.

• Required Qualifications
• 5 or more years maintaining production SQL databases and conducting advanced operations in SQL and T-SQL, including performance tuning, stored procedures, indexing strategies, and complex query development.
• 5 or more years designing, implementing, and maintaining ELT and ETL processes in cloud-based data analytics environments at enterprise scale.
• 3 or more years of hands-on experience in Azure Synapse Analytics and Azure Machine Learning specifically — Synapse workspace experience including Dedicated SQL Pools, Serverless SQL Pools, and Synapse Pipelines required; Azure ML pipeline development and SDK experience (V1 and V2) required.
• 3 or more years of data manipulation in Python with Pandas required; PySpark or Polars experience strongly expected given the scale requirements of this role.
• U.S. citizenship required. Must be stated on your resume. Must be eligible for and able to obtain a federal Public Trust background investigation.

• Key Responsibilities
• Design, implement, and maintain efficient, secure, and flexible data architecture in Azure, with all assets managed via source control.
• Design, implement, and maintain ELT/ETL pipelines in Azure Synapse and Azure Machine Learning (SDK V1 and SDK V2).
• Incorporate source control and CI/CD workflows for all pipelines and data analytics codebases.
• Review and improve existing architecture and pipelines; conduct periodic audits to address bottlenecks, deprecated dependencies, and architecture drift.
• Establish quality controls including error handling, logging mechanisms, and validation checks for all pipelines.
• Optimize ingestion, processing, and storage of diverse datasets including modern columnar formats such as Parquet.
• Develop self-service query and export capabilities for analysts supporting investigations and audits.
• Coordinate with data scientists to ensure architecture efficiently supports machine learning workloads in Azure Machine Learning.
• Author and maintain robust SOPs governing the creation, maintenance, and monitoring of all data pipelines and assets in Azure.
• Normalize entity attributes — addresses, phone numbers, shared identifiers — and develop cross-program lookup tables for investigative analysis.
• Stay current with emerging AI tools relevant to data engineering; contribute to exploratory LLM integration efforts.

• Additional Preferred Qualifications
• Experience implementing pipelines and infrastructure using code-first approaches including Python SDK, CLI, REST APIs, or IaC tooling.
• Demonstrated experience implementing source control via Git or Azure DevOps and CI/CD deployment workflows.
• Familiarity with AI coding assistants and LLM integration patterns.
• Experience with modern columnar storage formats including Parquet.
• Experience in financial crime, inspector general, healthcare fraud, or law enforcement data environments.

• Education and Certifications
• Bachelor's degree in computer science, data engineering, information systems, or a related field — or equivalent professional experience.
• Databricks Certified Data Engineer Associate or Snowflake SnowPro Core are recognized equivalents.