Portfolio · Project 19

Alexander CHERNOV

Principal Engineer · Agentic AI & scientific data systems

← the whole portfolio
19

Massive GPU: a simulated fleet with a real control path

2026 – present · Independent work, unaffiliated with any employer.

Changes to a GPU fleet pass through a gate and a person, and that path was tested by breaking it.

A 3D view of a Kubernetes cluster: 1,125 simulated nodes and 1,000 simulated GPUs beside one real GPU serving a real model. A proposed change goes to a gate under written rules, a person approves what the rules send up, and Git or an evict-only executor carries it out.

The demo as recorded: 3 minutes 26 seconds, captions on screen and no sound. The page plays back a record of the running system in real time: a ledger edit caught, the gate's pod removed, the Git server removed, a pod deleted, and a drain of 45 pods that a person approves. The captions are the page's own, and people and machines are named by aliases. The file is the one archived on Zenodo.

The fleet is simulated on purpose, and the page says so on its first screen. A real Kubernetes (k3s) schedules 50,386 pods; the simulated nodes and the pods on them are real API objects with nothing running in them (KWOK), and the GPUs and their load are a formula published under DCGM's metric names, not a measurement. Beside them stands a real GeForce RTX 4060 Laptop GPU serving Qwen3-4B-AWQ, reached by a LangGraph assistant through llm-d routers.

What is real is the path a change takes. The assistant or an operator proposes it, and the gate rules on it. A drain of a node running 45 pods is too large for the rules to admit alone, so it waits for a person; once approved, the cordon goes through Git, Argo CD syncs it, and the executor evicts the pods. Every step lands in a hash-chained ledger anchored in Git.

The recording includes the faults planted to test that path. A ledger record edited on disk is reported as a broken chain, and the gate refuses new records until it is restored. With the gate's pod removed for 43 seconds, nothing is admitted. With the Git server removed, an admitted change fails to commit, the ledger says so, and the change never reaches the cluster.

The page is a recording, with no server behind it. Its numbers are read from the record, so a number typed into its words fails the build, and it is checked against the watcher's own state at dozens of moments.