Data Engineer II - WFH

# Senior Data Platform Engineer

## About the Role

Handshake is hiring a Senior Data Platform Engineer to build and operate reliable data infrastructure supporting its career marketplace and AI products. As part of the Data and ML Platform team, you will own systems that orchestrate batch workloads and deliver timely, trustworthy data to engineers, analysts, data scientists, ML teams, and product organizations.

This is a hands-on technical leadership role focused on the technical direction and evolution of Handshake's data infrastructure. You will work across workflow orchestration, data ingestion and pipelines, BigQuery, streaming infrastructure, and cloud foundations while partnering with ML engineers, data scientists, and other engineering teams. The platform will also support AI development, including model and agent workflows, evaluation datasets, and production data access.

## Key Responsibilities

* Lead the technical direction and roadmap for the Airflow and Astronomer platform and own its production operations, including:

• * Deployment
• * DAG packaging
• * Scheduler and worker capacity
• * Upgrades
• * Observability
• * Incident response
• * Build reliable, reusable workflows and paved paths for authoring, testing, deploying, and operating data workflows.
• * Develop reusable operators, CI checks, local and staging environments, and clear operational runbooks.
• * Design and operate streaming and change data capture (CDC) pipelines using technologies such as Pub/Sub, Dataflow/Beam, and Datastream.
• * Build resilient data delivery systems that handle retries, duplicates, schema changes, late events, backfills, and replay.
• * Define meaningful latency, freshness, and reliability targets for data pipelines.
• * Improve the cloud foundation supporting data workloads using Kubernetes, Terraform, IAM, secrets management, CI/CD, and cost-aware capacity management.
• * Lead cross-team technical decisions involving data contracts, interfaces, and operational ownership.
• * Mentor engineers and align application, analytics, ML, and cloud teams around scalable platform approaches.
• * Participate in on-call responsibilities and troubleshoot production failures across data and infrastructure systems.
• * Convert production incidents into durable engineering improvements.
• * Build dependable data and orchestration foundations for AI workloads, including batch inference, evaluation datasets, and agent-facing data.

## Required Qualifications

### Technical Skills

• * Strong software engineering skills in **Python**.
• * Experience building and operating production data systems or distributed systems.
• * Deep hands-on **Apache Airflow** experience beyond DAG development, including scheduling and execution behavior, deployment, scaling, upgrades, monitoring, and debugging.
• * Experience with event-driven or streaming data infrastructure, including a message broker or managed event bus and a stream processing system.
• * Strong understanding of:

• * Change Data Capture (CDC)
• * Delivery guarantees
• * Idempotency
• * Event ordering
• * Schema evolution
• * Recovery and replay
• * Experience with cloud infrastructure and infrastructure as code.
• * Comfortable working with containers, Kubernetes, CI/CD, access controls, and production observability.

### Leadership & Collaboration

• * Demonstrated experience leading ambiguous infrastructure initiatives.
• * Ability to set technical direction and make pragmatic architecture trade-offs.
• * Track record of driving adoption of platform approaches across teams.
• * Clear communication skills and the ability to partner effectively across engineering, analytics, ML, and cloud teams.
• * Experience owning systems through production support and operational incidents.

## Preferred Qualifications

* Experience with GCP services, including:

• * Pub/Sub
• * Dataflow
• * Datastream
• * BigQuery
• * GKE
• * Cloud Storage
• * Experience with Astronomer, Apache Beam, Terraform, Spacelift, Datadog, dbt, Spark, or Dataproc.
• * Experience supporting both batch and low-latency data consumers.
• * Experience with product-facing data services or ML features.
• * Experience supporting ML or AI workloads such as inference pipelines, reproducible evaluations, or governed data access.
• * Experience building infrastructure for model and agent workflows.

## About Handshake

Handshake was founded on the belief that everyone deserves a path to a great career, regardless of where they went to school or who they know. The platform currently supports 25 million job seekers, more than 1 million employers, and 1,600 educational institutions.

In 2025, Handshake launched Handshake AI, working directly with frontier AI lab researchers to create evaluations, publish benchmarks, and advance AI data capabilities. Handshake describes the business as having grown from $0 to approximately $1 billion in run rate and paying approximately $60 million to more than 30,000 individuals every month.

## Benefits & Perks

The following benefits apply to full-time U.S. employees:

• * Equity in a fast-growing company.
• * 401(k) match and competitive compensation.
• * Financial coaching.
• * Paid parental leave.
• * Fertility benefits and parental coaching.
• * Medical, dental, and vision coverage.
• * Mental health support.
• * $500 wellness stipend.
• * $2,000 learning stipend.
• * Ongoing professional development.
• * Internet and commuting support.
• * Free lunch and gym access at the San Francisco office.
• * Flexible PTO.
• * 15 holidays plus 2 flex days.
• * Team outings and referral bonuses.

## Work Arrangement

Handshake provides remote and office-related benefits for its U.S. employees. The provided job description does not specify a single required work location for this position.