Rust Jobs for Rustaceans

Key Responsibilities:

• Building petabyte-scale, high-throughput data processing systems managing hundreds of petabytes to exabytes of data.
• Designing and operating large-scale distributed systems and pipelines processing hundreds of thousands to millions of operations per second.
• Managing workloads across large cloud compute clusters.
• Pre-processing datasets for AI training.
• Building and operating large-scale crawlers, gathering and communicating requirements clearly and concisely.

Requirements:

• Strong engineering skills with a passion for improving different aspects of data and model performance.
• Strong proficiency in at least one compiled language: Rust, Go, C++, or Java.
• Has worked on one or more modalities other than text and demonstrated exceptional work.
• Building bespoke data processing libraries from scratch.
• Designing and implementing distributed systems in Rust.
• Keeping up with state-of-the-art techniques for preparing AI training data.
• Experience with performance optimization of large-scale systems is preferred.
• Organizing and meticulously bookkeeping data across multiple clouds, of multiple modalities, and from many sources.
• Experience with SQL/NoSQL databases, especially columnar databases, is a plus.
• Great debugging skills are a must.
• Must have deep knowledge of how the internet works, including DNS, OSI model, crawler architectures, challenges operating crawlers, and headless browsers.

Tech Stack

• Python
• Rust
• Spark
• Kubernetes