Lead Data Engineer - Fulltime Role
Remote

Job Summary
We are seeking an experienced, technically strong Lead Data Engineer to design, develop, implement, and support enterprise-scale data engineering platforms and integration solutions.
The ideal candidate will bring deep hands-on experience in Python, PySpark, SQL, Apache Spark, Databricks, Snowflake, ETL/ELT, and modern Data Lakehouse architectures, along with the ability to guide engineering teams and drive technical delivery.
The role involves close collaboration with Data Architects, Data Engineering teams, AI/ML teams, business stakeholders, and technology leadership to deliver scalable, reliable, secure, and high-performance data solutions.

Key Responsibilities
1. Data Engineering & Pipeline Development
Design, develop, and maintain scalable batch and real-time data pipelines using Python, PySpark, SQL, Spark, and ETL/ELT technologies.
Build and optimize data ingestion, transformation, integration, and processing workflows across enterprise systems.
Develop and maintain enterprise data warehouses, data lakes, databases, tables, and data processing systems.
Write complex SQL queries and optimize database and pipeline performance.
Troubleshoot production data issues and improve pipeline reliability, efficiency, and quality.
2. Data Architecture & Platform Engineering
Design and implement data structures, schemas, data models, and integration frameworks.
Work with modern cloud data platforms including Databricks, Snowflake, OCI, and similar technologies.
Build and support Delta Lake, Lakehouse, and enterprise data warehouse solutions.
Apply data engineering best practices to ensure scalability, performance, maintainability, and reliability.
Evaluate technical designs, identify potential issues, and recommend improvements.
3. Technical Leadership & Delivery
Provide technical guidance to data engineers and collaborate with Data Architects and engineering teams.
Translate business requirements into technical specifications, implementation plans, and test plans.
Coordinate technical activities across multiple teams and enterprise platforms.
Support implementation, testing, deployment, and production stabilization.
Manage competing technical priorities and ensure timely delivery of engineering solutions.
4. Data Governance, Quality & Security
Collaborate with data governance, data quality, and information security teams.
Implement data quality checks, validation processes, and monitoring frameworks.
Support metadata management, data modeling, and data security requirements.
Ensure pipelines meet applicable governance, security, and certification standards before production deployment.
Ensure compliance with HIPAA regulations and appropriate handling of Protected Health Information (PHI).
5. Analytics, AI & Data Science Enablement
Work closely with analytics, AI/ML, and data science teams to develop reliable datasets and data processing solutions.
Prepare structured and unstructured data for predictive and prescriptive analytics.
Support data transformation and integration requirements for machine learning and AI applications.
Identify opportunities to improve automation, data processing, and analytical capabilities.
6. DevOps & DataOps
Apply Agile, DevOps, and DataOps principles to data engineering development and delivery.
Improve deployment processes, pipeline monitoring, validation, and production reliability.
Support automation of engineering workflows and continuous improvement initiatives.

Required Qualifications
12+ years of relevant IT/data engineering experience, including at least 3 years in data modeling, schema design, and building and maintaining data processing systems.
Strong hands-on programming experience with Python, PySpark, SQL, and Spark.
Extensive experience designing, developing, and optimizing ETL/ELT pipelines.
Experience with Databricks, Snowflake, Delta Lake, data warehouses, or comparable enterprise data platforms.
Strong knowledge of data modeling, database development, data transformation, and large-scale data processing.
Experience troubleshooting end-to-end data pipelines and resolving production data issues.
Understanding of data governance, data quality, metadata, and security controls.
Ability to collaborate with engineering, architecture, analytics, and business teams.
Strong analytical, problem-solving, communication, and technical leadership skills.
Bachelor's degree in Computer Science, Information Technology, or a related discipline preferred.

Lead Data Engineer

Similar Engineering Jobs

Data Engineering & Analytics Senior Manager

AI Data Engineer | Local Only

AWS Data Engineer with AI

Software Engineer, Data Science

AWS Python Data Engineer

Principal Data Engineer