Data Engineer

About the Company

• Wynd Labs builds infrastructure that delivers large-scale public web data to companies developing advanced AI models. The team supports Grass, a bandwidth-sharing network that powers a distributed web crawler, and develops data pipelines for ingesting, segmenting, and annotating billions of videos, transcripts, and audio files for AI dataset creation.
• Wynd Labs is a lean, technical team focused on expanding access to public web data and building infrastructure for AI.

About the Role

• Wynd Labs is seeking a Data Engineer to develop, maintain, and optimize large-scale data pipelines and systems that support web data collection and AI dataset creation. This role combines advanced Python development, web scraping, distributed processing, data warehousing, and infrastructure operations.
• You will work with high-volume datasets, distributed workers, analytical databases, and containerized workloads while improving data quality, pipeline reliability, and processing efficiency.

Key Responsibilities

• Maintain, optimize, and troubleshoot database queries and data systems to support efficient data access, processing, and reliability.
• Build, maintain, and improve data pipelines for collecting, processing, transforming, validating, and delivering large-scale datasets.
• Develop, test, and maintain scripts and tools for gathering publicly available web data in accordance with company requirements.
• Support high-volume web scraping and data collection initiatives involving media, metadata, and platform APIs.
• Monitor distributed data pipelines, troubleshoot processing issues, and resolve data quality concerns.
• Improve the reliability, scalability, and performance of data processing workflows across workers and servers.
• Document database queries, pipeline processes, scraping workflows, technical decisions, issues, and resolutions.
• Participate in research and development projects to improve data products and engineering workflows.
• Develop and maintain scalable APIs and supporting data infrastructure.
• Support deployment, monitoring, and scaling of data workloads using containerization and orchestration technologies.

Required Qualifications

• Bachelor's degree or equivalent work experience.
• Advanced Python programming skills, including asynchronous programming, multiprocessing, and production-grade code for long-running data jobs.
• Hands-on experience with high-volume web scraping, including proxies, rate limiting, and anti-bot evasion.
• Experience with platform APIs and large media or metadata datasets, including video platforms or social media.
• Experience designing and operating distributed data pipelines across multiple workers or servers.
• Familiarity with task queues and messaging technologies such as Celery, Kafka, RabbitMQ, or similar tools.
• Practical experience with columnar or analytical data warehouses.
• Ability to write complex analytical queries and work with partitioning strategies and cost-aware cloud warehouse queries.
• Experience with Docker and Kubernetes, including containerized workloads, deployment configurations, and autoscaling.
• Linux server administration and troubleshooting skills, including diagnosing disk I/O, network, and memory performance issues.
• Experience with CI/CD for data workflows using tools such as GitHub Actions or ArgoCD.
• Experience writing scalable APIs.

Preferred Technologies and Experience

• Experience with Databend, ClickHouse, or BigQuery.
• Experience creating Helm charts and Kubernetes manifests.
• Familiarity with distributed scraping and processing infrastructure.
• Experience managing services on Linux servers and troubleshooting performance without relying solely on managed-cloud abstractions.
• Experience with video, audio, transcript, and other large-scale media datasets.

Work Arrangement

• Fully remote position.
• Location listed as New York, New York, United States.
• Remote team environment with a lean, technically focused engineering culture.

Compensation and Benefits

Wynd Labs offers a competitive salary, benefits, and equity package. Specific salary figures and individual benefit details are not provided in the source listing.