tooluniverse.remote_resident package

Resident-worker plumbing shared by GPU-backed remote tools.

Submodules

tooluniverse.remote_resident.admission module

Whether there is room to hold one more provider. This is what the budgets are for.

Every manifest declares [gpu] vram_budget_gb, measured rather than guessed, and STANDARD.md §6 says it is “what a host adds up to decide how many providers may be resident at once”. Twenty-seven providers now declare one and nothing added them up: backend.running() returns {slug: budget} with a docstring saying a scheduler above it reads that, and no caller. This module is the addition.

IT EXISTS BECAUSE THE MACHINE WAS WEDGED, by this project, with the numbers in hand. Four providers’ workers were held, their RSS summed to about 12 GiB, and a fifth provider’s 28 GiB smoke was started on top. The host stopped responding. Two things made that possible and both are requirements here:

1. The sum of what THIS process holds is not what the machine holds. The workers that turned 12 GiB into a wedge were boltzgen workers from a different checkout entirely, started by a session that had no reason to announce them. backend.running() is blind in exactly the direction that caused the harm, so the holdings come from machine_holdings – the process table, which sees every checkout’s workers and counts two processes of one provider as two holders. The two gates then cover different unknowns: the declared sum protects against peaks that have not happened yet, and the live reading protects against everything on this machine that is not a resident worker at all.

2. On unified memory the two budgets are one budget. nvidia-smi –query-gpu=memory.total answers [N/A] on this GB10 because device memory IS system memory: 121 GiB for the host and the card together. So capacity is MemTotal, a VRAM budget is spent from the same pool as a page cache, and a provider’s budget cannot be checked against a device total that does not exist. STANDARD.md already recorded that this breaks a kit’s own arithmetic; it breaks a scheduler’s the same way one level up.

THE LIVE GATE ASKS A NARROWER QUESTION WHEN IT HAS CO-TENANTS, which is the second thing this project got wrong before measuring it. The first guard gated on MemAvailable and waved a worker through at 60.4 GiB available with 5 GiB free; it died with CUDA error: out of memory, because the other 56 was reclaimable page cache that a device allocation did not get to reclaim in time. Changing it to MemFree then made it too strict: a bounded probe allocated to 24 GiB starting from 18.7 GiB free, with Cached falling 72.2 -> 63.4 while free held near 4 – reclaim works. Both readings are true and they answer different questions, so which one gates depends on whether anything else is competing for the reclaim: MemFree when there are co-tenants, because losing a race to one is what the OOM was, and MemAvailable when there are none, because then reclaim is not a race.

THE LIMIT OF THAT EVIDENCE, stated because the rule extrapolates past it: the uncontended probe reached 24 GiB starting from 18.7 GiB free. An uncontended admission of more than that is trusting the kernel’s own estimate of what it can reclaim, which is exactly what MemAvailable is, and which has not been falsified here – but it has only been confirmed once and at one size. A host that wants a harder floor passes a larger reserve_gb; the honest default is not to invent a bound nobody measured.

A refusal here names the numbers and does not round them, per §8.

exception tooluniverse.remote_resident.admission.AdmissionRefused

Bases: RuntimeError

There is not room to hold this provider. The message carries the numbers.

class tooluniverse.remote_resident.admission.Verdict

Bases: object

The decision, the reason, and every number it was made from.

admit: bool

reason: str

slug: str = ''

budget_gb: float | None = None

capacity_gib: float | None = None

capacity_source: str = ''

declared_held_gb: float = 0.0

free_gib: float | None = None

available_gib: float | None = None

gate: str = ''

co_tenants: int = 0

unknown_budgets: Tuple[str, ...]

raise_if_refused()

__init__(admit, reason, slug='', budget_gb=None, capacity_gib=None, capacity_source='', declared_held_gb=0.0, free_gib=None, available_gib=None, gate='', co_tenants=0, unknown_budgets=<factory>)

tooluniverse.remote_resident.admission.meminfo_gib()

MemTotal, MemFree and MemAvailable in GiB, from /proc/meminfo.

tooluniverse.remote_resident.admission.capacity_gib()

The size of the one pool a budget is spent from, and where the figure came from.

On unified memory there is no device total to ask for – nvidia-smi returns [N/A] – and the honest capacity is MemTotal, because a VRAM allocation and a page cache come out of the same 121 GiB. On a discrete card the device total is the right ceiling for a VRAM budget and MemTotal is not.

Returns (None, reason) when neither can be established, which is a thing a caller must handle rather than a zero it can compare against.

tooluniverse.remote_resident.admission.pool_free_gib(timeout=15.0)

How much of that one pool is free right now, and where the figure came from.

The mirror of capacity_gib, for the same reason and across the same three machines. A discrete card knows what it has free and that is the answer. On unified memory nvidia-smi returns [N/A] for memory.free exactly as it does for memory.total, and the honest reading is MemAvailable, because the model’s allocation and the page cache come out of one 121 GiB pool.

A host with no NVIDIA driver at all gets (None, reason) rather than its free RAM: a machine with no device is not a machine with an empty device, and reporting host RAM as free VRAM would admit a model onto hardware that cannot hold it. None means DO NOT REASON ABOUT MEMORY, which a caller has to handle as its own case rather than compare against zero.

tooluniverse.remote_resident.admission.machine_holdings(*, ps_output=None, budget_of=None)

Every resident worker on this machine, not only this process’s.

THIS IS THE HOLE THE REST OF THIS MODULE WAS WRITTEN AROUND. backend.running() knows what this interpreter started; the workers that wedged this host were started by a session in another worktree of this same repository, and the whole mistake was not knowing they were there. The process table does know: a resident worker is a *_resident_worker.py invoked with –socket, and its own path names the provider and the checkout.

Scanning sockets instead would undercount, because the socket path is one per slug per uid: two checkouts running the same provider are one socket and two processes, and the second process is the one that turns a comfortable sum into a wedge. A stale socket also outlives its worker – there is one in that directory as this is written – so a socket is evidence of a past worker and a process is evidence of a present one.

Keys are slug for a holder in this tree and slug@tree for one elsewhere, so a restart can release its own budget without releasing someone else’s. Values are the DECLARED budget from the local manifest, because that is what a budget is for; a slug this checkout does not have returns None, which decide refuses on rather than treating as free.

tooluniverse.remote_resident.admission.manifest_path(slug)

Where this checkout keeps slug’s manifest, whether or not it has one.

tooluniverse.remote_resident.admission.declared_budget_gb(slug)

[gpu] vram_budget_gb from this checkout’s manifest for slug.

None means UNDECLARED, which decide reports rather than summing as zero: a provider whose cost is unknown cannot be added up.

A CPU-ONLY PROVIDER IS 0.0, NOT None, and the distinction is the whole difference between a useful co-tenant and a blocker. A manifest with [gpu] policy = “none” is the one case where an absent vram_budget_gb is legal – the manifest layer enforces that – and it means the provider holds no device memory at all, not that nobody measured it. Mapping it to None made a held esm_if1 or proteinmpnn worker refuse every GPU provider on the box with “already held without a declared budget, so what the card is carrying cannot be summed”: exactly inverted, since a CPU provider is the one thing that can safely be resident while the card is full. Found by running tu remote room boltz with a real proteinmpnn worker up, which reported held: {“proteinmpnn”: null} and refused boltz on a card holding nothing.

Another checkout’s copy of the same provider could in principle declare a different figure; using the local one assumes the checkouts agree, which is worth stating because it is an assumption and not a reading.

tooluniverse.remote_resident.admission.declares_resident_worker(slug)

Whether slug holds its model in a worker process of its own.

WHICH IS WHAT DECIDES WHETHER A LOADED MODEL CAN BE SEEN FROM OUTSIDE, and that is the only reason anything outside this module asks. Neither kind of provider has its model in memory when its port starts answering – a resident one loads in a worker that the first request starts, and a provider without a manifest loads lazily in the server process, on the first request too. The difference is the evidence: a worker is a process, so machine_holdings can say whether it is up, where a model inside an MCP server leaves nothing a scheduler can read.

So a scheduler measuring what a provider costs uses this to know whether it can check that a model loaded at all, rather than attributing an empty process to a model. False for the 28 reviewed slugs with no manifest, and for a manifest that declares any other mode.

tooluniverse.remote_resident.admission.decide(slug, budget_gb, held=None, *, reserve_gb=4.0, mem=None, capacity=None, co_tenants=None)

Whether slug may be started now, given what is already held.

held is {slug: budget} as backend.running() returns it, and a None in it is an UNDECLARED budget, which is reported rather than treated as zero: a provider whose cost is unknown cannot be added up, and a sum that silently drops it would be an admission made on a number nobody measured.

reserve_gb is left for the host itself. It is not a safety fudge over the budgets – those are already upper bounds – but the room the operating system needs to not be the thing that fails: the wedge happened with the pool nominally sufficient and the machine unable to schedule anything.

The injectable arguments exist so the decision is testable without a card.

tooluniverse.remote_resident.admission.room_for(slug, budget_gb, **kw)

decide against what the MACHINE is holding, not what this process is.

backend.running() was the obvious source and is the wrong one: it sees the workers this interpreter started, and the ones that wedged this host came from another worktree. machine_holdings reads the process table instead, which covers both, so this function does not need to add them up.

A holder in this tree with the same slug is dropped, because restarting a worker releases its own budget. A holder of the same slug in ANOTHER tree is kept, because that one is still holding it.

tooluniverse.remote_resident.backend module

One resident worker per provider, built from its provider.toml.

The registry is process-wide and lazy: a provider’s worker object is created the first time something asks for it, and the model process behind it starts on the first request (and, with idle_timeout_s set, stops again when the requests do). Importing this module starts nothing.

[runtime] launch in the manifest names the callable that builds the worker’s command line, as package.module:function. That indirection is deliberate: the declarative facts – weights, budget, isolation, timeouts – belong in the manifest where anything can read them, while the one part that is genuinely provider-specific (how to invoke that model’s environment) is a named function rather than a growing set of TOML keys that only one provider ever sets.

tooluniverse.remote_resident.backend.warm_if_requested(provider_dir=None)

Load this provider’s model now, if whoever started this server asked for it.

WHAT THIS IS FOR. A provider’s MCP server is a CPU-only process that binds a port in seconds; the model behind it loads in a worker, on the first request, and that takes minutes. So starting the server is not warming the provider, and a scheduler that prewarms by starting servers hides nothing from the call that follows – it reports a provider ready and the caller still waits out the whole load. This is the other half: the server comes up AND the weights go in.

In a thread, because the port is what the thing that started this process is waiting for. ResidentWorker.ensure_started holds the worker’s lock for the whole load, so a request arriving mid-load queues on it and is served when the load finishes rather than starting a second one.

Returns the slug whose load was started, or None – no request in the environment, no manifest next to the entry script, or a manifest that keeps no worker. A warm must never be the reason a server fails to start, so every failure here is one of those Nones, and a load that fails says so in the log the worker was given.

tooluniverse.remote_resident.backend.log_path_for(slug)

tooluniverse.remote_resident.backend.worker_for(manifest_path)

The ResidentWorker of the provider manifest_path declares.

tooluniverse.remote_resident.backend.worker_for_provider_dir(directory)

The worker of the provider whose directory is directory.

tooluniverse.remote_resident.backend.running()

{slug: vram_budget_gb} for every worker currently holding the card.

What a scheduler above this reads to know what is resident right now; the budgets are the manifests’ measured peaks, not live device queries.

tooluniverse.remote_resident.backend.stop_all(*, grace_s=30.0)

Retire every worker this process started. Safe to call twice.

tooluniverse.remote_resident.exec_env module

How to run a command in the model’s environment: a venv, a container, or an image.

The optimization kits offer the same stack three ways — a Docker image, an Apptainer .sif converted from it, or a Python venv at the pinned versions — and note that steps 2 to 4 are identical whichever you pick. The same holds here: a resident worker is the same script, holding the same model, answering on the same socket. Only the argv that starts it differs, so only that is abstracted.

Paths are mapped identity. Every bind mounts a host path at the same path inside, -v /p:/p. The alternative — a private layout inside the container — would need translation for the socket, the weights, the worker script, the shared protocol file, and every per-request output directory the caller names, in both directions, forever. Identity mapping costs one convention and removes all of it: a path means the same thing on both sides of the boundary.

Containers run as the calling user. The kits’ own recipes say “Containers run as root. The Docker and Apptainer recipes do not create an unprivileged user”, which for a model writing predictions into a bind-mounted directory means root-owned output on the host. Docker is given –user; Apptainer runs as the caller already.

Apptainer is implemented here and has not been exercised on this project’s machine, which has no Apptainer installed. It is written from the kits’ own documented invocation. Treat it as untested until a host with Apptainer says otherwise.

exception tooluniverse.remote_resident.exec_env.ExecEnvironmentError

Bases: RuntimeError

The declared execution environment cannot be used as declared.

class tooluniverse.remote_resident.exec_env.ExecEnvironment

Bases: object

Turns a Python command into an argv that runs it where the model lives.

kind: str

python: str | None = None

image: str | None = None

sif: str | None = None

container_python: str = 'python'

binds: List[Path]

gpu: bool = True

network: str | None = None

cache_dir()

cache_env()

argv(args, env=None)

(argv, environment for Popen) running python <args> inside.

A venv inherits the environment; a container is given it explicitly, because the variables would otherwise reach only the client.

describe()

__init__(kind, python=None, image=None, sif=None, container_python='python', binds=<factory>, gpu=True, network=None)

tooluniverse.remote_resident.exec_env.from_manifest(manifest, *, extra_binds=())

The execution environment a provider declares, with what it must see.

The binds are the paths a resident worker touches across the boundary: the socket it serves on, the weights it reads, the ToolUniverse files it loads by path, its own directory, and the work root under which per-request output directories are created.

tooluniverse.remote_resident.kit_launch module

Start a resident worker under an optimization kit’s own launcher.

Every kit in anthropics/uplifting-biomolecular-modeling is built on one shared core, and the uniformity is usable: all 36 carry run.sh, environment/, opt/ and stock/PINS.json, and 35 name their package <kit>_opt (protenix_v2 is the one that does not, which is why [kit] package can be stated). There are two ways a kit engages, and the common one is the simpler:

Autoload (30 of 36). The kit installs a .pth that imports its _autoload at interpreter start. With its mode variable set — modes.ENV, e.g. BORZOI_OPT=exact — a meta-path finder waits for the model package’s own import and patches it in place. Our worker is then just a script run by the model’s interpreter with that variable in the environment. Nothing else.

worker_launch (1 of 36: boltz2). That kit adds a launcher that runs an arbitrary script under an explicit kernel route and attach list:

<model python> -m boltz2_opt.worker_launch --route … --attach … -- <script> …

Either way a resident worker is a script of ours in a position the kit already serves, and nothing in the kit is patched, copied or shadowed.

Which one a kit uses is detected, not declared: the probe reports whether <pkg>.worker_launch imports. That keeps the exception from becoming a manifest field every other provider has to get right.

One kit cannot be made resident with its levers at all. ChromBPNet’s own autoload says it: “THE ONE DEVIATION from an in-process hook: the kit’s form is a replacement entry script; it offers no in-process patch.” Its finder fires only when sys.argv[0] is the stock console script, and then runs a child. A provider like that declares mode = "subprocess" with the reason, rather than pretending.

The options and the environment row are deliberately not written down here. They belong to the kit, change with it, and cannot be read from this process at all — <kit>_opt lives in the model’s environment, whose pinned stack does not coexist with ToolUniverse’s. So the kit is asked, once per worker start, in its own interpreter. A hard-coded copy would drift silently the first time the kit was updated; this cannot.

One subtlety that looks like an oversight in a manifest and is not: a kit’s row may set a switch for an adapter that misbehaves on a given host. Leave it. An adapter named in --attach whose switch is missing from the row refuses the whole launch, whereas one whose switch is present and whose preconditions are not met declines by its own name and the launch proceeds. Removing the variable breaks the start; leaving it is what keeps the code out of the process.

exception tooluniverse.remote_resident.kit_launch.KitUnavailable

Bases: RuntimeError

The kit is not installed on the model environment’s interpreter.

tooluniverse.remote_resident.kit_launch.kit_package(manifest)

[kit] package, else <subdir>_opt — the convention 35 of 36 follow.

tooluniverse.remote_resident.kit_launch.ask_kit(python_or_env, package, mode, levers_off=())

The kit’s own launch options and environment row, read in its interpreter.

python_or_env is an interpreter path or an ExecEnvironment. The probe has to run where the kit is installed, which for a container provider is inside the container: asking this process’s interpreter would report a kit that is not the one the worker will load.

levers_off is set in the PROBE’s environment, not only in the worker’s. The kits read MODEL_OPT_LEVERS_OFF when they resolve a mode, so a probe run without it reports the unablated row – and a kit that then sees the ablated one refuses the whole activation over the mismatch. openfold3’s worker carries a _reconcile_kit_row to work around exactly that, which is a workaround every later provider with both a row and an ablation would have had to re-derive. AND THE SAME PRINCIPLE APPLIES TO THE OPERATOR’S STRAY VARIABLES, which is a gap this function had until opendde’s acceptance ran into it. Most kits’ .pth kills any interpreter that inherits an UNDECLARED <PACKAGE>_* name – opendde’s exits 3 on OPENDDE_OPT_X=1 python -c "print('x')". A provider’s launcher strips those from the WORKER by mapping them to None, but the supervisor applies that mapping to the command this function helps build, so the probe had already inherited them and died:

KitUnavailable: could not read opendde_opt’s mode row for mode=’off’ (rc 3). Its own words: [opendde-opt] NOT ACTIVE: reason=undeclared variable(s)

OPENDDE_OPT_BOGUS, OPENDDE_OPT_X

The probe has to run under the conditions the worker will, and the stray names are one of those conditions. The policed prefix is mechanical: every kit package is <slug>_opt, so package.upper() is the name the kit declares and anything extending it with an underscore is undeclared. The declared name itself is KEPT, because the probe exists to read the row that name selects.

tooluniverse.remote_resident.kit_launch.build_command(worker)

(argv, extra environment) for worker. The supervisor calls this.

tooluniverse.remote_resident.manifest module

provider.toml – what a GPU remote tool declares about itself.

One file per provider directory, beside its MCP server. It answers the questions a host has to answer before it runs somebody else’s model, and every field is a fact somebody measured or pinned rather than a preference:

[weights]

Every file, its size and its SHA-256. This exists because a Boltz-2 checkpoint in this project downloaded as 1.2 % of itself and served broken predictions for over a month before anyone noticed – a size check alone would have caught that one, a digest catches the rest. Nothing is fetched implicitly: a cache that is short a file is named and the provider refuses.

[runtime]

mode = "resident" or "subprocess". Resident is the default because a held model answers in a fraction of the time a fresh one does (Boltz-2 on this project’s GB10 box: 110 s per call as a subprocess, 47 s held), and a provider that cannot be held says so here with its reason.

[gpu]

vram_budget_gb is the measured peak reserved, and it is what lets a host decide how many providers fit on one card at once. isolation states what the deployment actually provides, never what one would like: on hardware without MIG the honest answer is process, and a manifest that claimed otherwise would be the dangerous kind of wrong.

[env]

The interpreter the model’s own environment lives in, and the lock file that rebuilds it. Providers do not share one environment – their pinned stacks conflict – so this is how the supervisor finds the right python.

Reading is strict on purpose: an unknown key is a typo that would otherwise apply silently as a default, so it is named and refused.

exception tooluniverse.remote_resident.manifest.ManifestError

Bases: ValueError

The manifest is missing, malformed, or says something impossible.

class tooluniverse.remote_resident.manifest.DigestMemo

Bases: object

Digests of files already hashed, keyed by what would change if they were edited.

The key is (size, modification time in nanoseconds). A file matching both is served from the memo; anything else is hashed again and recorded. It lives outside the weights directory so that a read-only or shared cache still benefits, and a memo that cannot be written is simply not written – the digests are then computed afresh, which is correct, only slower.

__init__(path)

get(name, st)

put(name, st, digest)

save()

class tooluniverse.remote_resident.manifest.WeightFile

Bases: object

WeightFile(name: ‘str’, size: ‘Optional[int]’ = None, sha256: ‘Optional[str]’ = None, unpacked_dir: ‘Optional[str]’ = None)

name: str

size: int | None = None

sha256: str | None = None

unpacked_dir: str | None = None

verify(root, memo=None)

None when the file is at its pin, else why it is not.

With a memo, a file whose size and modification time are unchanged since it was last hashed is taken at its recorded digest instead of being read again. Hashing this provider’s 6.1 GiB costs ~4 s warm and far more from cold disk, and tu remote check would otherwise pay it on every invocation. Any change to either attribute forces a re-hash, so the shortcut cannot hide an edit.

__init__(name, size=None, sha256=None, unpacked_dir=None)

class tooluniverse.remote_resident.manifest.ProviderManifest

Bases: object

ProviderManifest(path: ‘Path’, slug: ‘str’, description: ‘str’ = ‘’, operations: ‘tuple’ = (), status: ‘str’ = ‘untested’, verified_on: ‘str’ = ‘’, outputs: ‘tuple’ = (), output_keys: ‘tuple’ = (), mode: ‘str’ = ‘resident’, mode_reason: ‘str’ = ‘’, worker: ‘Optional[str]’ = None, launch: ‘Optional[str]’ = None, idle_timeout_s: ‘float’ = 0.0, start_timeout_s: ‘float’ = 900.0, request_timeout_s: ‘float’ = 3600.0, gpu_policy: ‘str’ = ‘none’, vram_budget_gb: ‘Optional[float]’ = None, driver_min: ‘Optional[float]’ = None, driver_min_reason: ‘str’ = ‘’, shareable: ‘bool’ = True, isolation: ‘str’ = ‘process’, env_python: ‘Optional[str]’ = None, env_lock: ‘Optional[str]’ = None, env_kind: ‘str’ = ‘venv’, env_verify: ‘Optional[str]’ = None, env_image: ‘Optional[str]’ = None, env_sif: ‘Optional[str]’ = None, env_container_python: ‘str’ = ‘python’, env_network: ‘Optional[str]’ = None, work_root: ‘Optional[str]’ = None, smoke: ‘Optional[str]’ = None, smoke_timeout_s: ‘float’ = 1800.0, worker_args: ‘tuple’ = (), weights_root_env: ‘Optional[str]’ = None, weights_root_default: ‘Optional[str]’ = None, weights_fetch: ‘Optional[str]’ = None, weights: ‘tuple’ = (), kit: ‘Dict[str, Any]’ = <factory>)

path: Path

slug: str

description: str = ''

operations: tuple = ()

status: str = 'untested'

verified_on: str = ''

outputs: tuple = ()

output_keys: tuple = ()

mode: str = 'resident'

mode_reason: str = ''

worker: str | None = None

launch: str | None = None

idle_timeout_s: float = 0.0

start_timeout_s: float = 900.0

request_timeout_s: float = 3600.0

gpu_policy: str = 'none'

vram_budget_gb: float | None = None

driver_min: float | None = None

driver_min_reason: str = ''

shareable: bool = True

isolation: str = 'process'

env_python: str | None = None

env_lock: str | None = None

env_kind: str = 'venv'

env_verify: str | None = None

env_image: str | None = None

env_sif: str | None = None

env_container_python: str = 'python'

env_network: str | None = None

work_root: str | None = None

smoke: str | None = None

smoke_timeout_s: float = 1800.0

worker_args: tuple = ()

weights_root_env: str | None = None

weights_root_default: str | None = None

weights_fetch: str | None = None

weights: tuple = ()

kit: Dict[str, Any]

property directory: Path

weights_root()

digest_memo_path()

verify_weights(*, deep=False)

Every problem, named. Empty means every declared file is at its pin.

deep re-reads every file whatever the memo says. Setup uses it, so that what it certifies it actually hashed; check does not.

resolve(relative)

A manifest-relative path as an absolute one.

python_executable()

The model environment’s interpreter.

[env] python may name an environment variable ($NAME) so a deployment can point at its own venv without editing the manifest.

__init__(path, slug, description='', operations=(), status='untested', verified_on='', outputs=(), output_keys=(), mode='resident', mode_reason='', worker=None, launch=None, idle_timeout_s=0.0, start_timeout_s=900.0, request_timeout_s=3600.0, gpu_policy='none', vram_budget_gb=None, driver_min=None, driver_min_reason='', shareable=True, isolation='process', env_python=None, env_lock=None, env_kind='venv', env_verify=None, env_image=None, env_sif=None, env_container_python='python', env_network=None, work_root=None, smoke=None, smoke_timeout_s=1800.0, worker_args=(), weights_root_env=None, weights_root_default=None, weights_fetch=None, weights=(), kit=<factory>)

tooluniverse.remote_resident.manifest.load(path)

Read and validate one provider.toml.

tooluniverse.remote_resident.protocol module

Line-delimited JSON over a Unix stream socket.

The transport between a remote tool’s MCP server (light, CPU-only, one per provider) and its resident model worker (heavy, holds the GPU). Both live on the same host: the worker is what keeps model weights on the device between requests, which is the whole point of the split. A Unix socket rather than a TCP port because the worker is never addressable from outside the machine – tu remote share publishes the MCP server, never this.

Framing is one JSON object per line, UTF-8, no embedded newlines (json.dumps escapes them). A request carries op; a reply carries ok plus either the operation’s own fields or error and reason. reason is one blank-free token so callers can branch on it without parsing prose.

exception tooluniverse.remote_resident.protocol.ProtocolError

Bases: RuntimeError

The peer closed mid-message, or sent something that is not one JSON line.

tooluniverse.remote_resident.protocol.socket_path(slug, runtime_dir=None)

The worker socket for slug.

Under XDG_RUNTIME_DIR when the session has one (tmpfs, 0700, cleaned on logout), else a per-uid directory under the temp dir – never a shared path another user could pre-create as a symlink.

tooluniverse.remote_resident.protocol.send_line(sock, payload)

tooluniverse.remote_resident.protocol.recv_line(sock, buf)

One JSON object from sock; buf carries bytes read past its newline.

tooluniverse.remote_resident.setup module

tu remote setup <slug> — bring one provider’s environment, kit and weights to what its manifest declares, then prove it by running the model once.

The design follows the optimization kits’ own install step, whose choices are worth stating because each answers a failure this project has actually had:

Idempotence comes from checking, not from a flag. There is no --force and no state file. Every step asks the filesystem whether it is already done, reports present, and skips. Re-running after a failure is the normal way to continue.

Nothing upstream owns is restated here. Weights are fetched by the provider’s own declared command, which calls upstream’s downloader with upstream’s URL lists and fallbacks. No URL appears in this file or in any manifest, so a mirror change upstream cannot break setup.

Nothing is deleted and nothing is retried automatically. A file whose digest is not the pin’s is named and left in place: upstream writes straight to the final name, so a truncated transfer and a corrupt file are indistinguishable, and deleting one to retry would destroy the evidence that says which it was. A 2,286 MB checkpoint that arrived as 28.5 MB of itself served predictions from this repository for over a month because nothing compared it to anything.

The environment is verified, not created. Setup installs into an interpreter the manifest names through a variable; it does not build venvs, because the outcome is what matters and uv, python -m venv, conda and pixi all reach it. An unset variable is refused with the recipe from the lock file’s own header.

It ends by running the model, under a deadline. Installing successfully is not evidence that anything works: of the four problems hit bringing up the first provider, three appeared only at run time, and one of those presented as a stall rather than an error. Without a deadline a setup waits forever and reports nothing.

class tooluniverse.remote_resident.setup.StepResult

Bases: object

StepResult(step: ‘str’, verdict: ‘str’, detail: ‘str’ = ‘’, reason: ‘str’ = ‘’, lines: ‘List[str]’ = <factory>, seconds: ‘float’ = 0.0)

step: str

verdict: str

detail: str = ''

reason: str = ''

lines: List[str]

seconds: float = 0.0

property ok: bool

__init__(step, verdict, detail='', reason='', lines=<factory>, seconds=0.0)

tooluniverse.remote_resident.setup.kit_root()

tooluniverse.remote_resident.setup.kit_checkout(manifest)

Where this provider’s kit lives: content-addressed by its pinned commit.

Two providers pinning the same commit share one checkout, and fetching a newer commit lands beside the working one instead of over it.

tooluniverse.remote_resident.setup.step_preflight(manifest)

Refuse for a host reason before anything large is downloaded.

Every check here is cheap and local. The expensive steps below assume it passed, so that a machine that cannot run this provider learns so in a second rather than after 16 GB.

tooluniverse.remote_resident.setup.step_environment(manifest)

Install the lock into the interpreter the manifest names. Never create one.

tooluniverse.remote_resident.setup.step_kit(manifest)

Fetch the pinned third-party kit, if this provider uses one, and install it.

tooluniverse.remote_resident.setup.step_weights(manifest)

Every declared file present and at its pin. Deep: what is certified is read.

tooluniverse.remote_resident.setup.measured_path(slug)

tooluniverse.remote_resident.setup.step_smoke(manifest)

Run the model once, under a deadline, and record what this machine did.

Installing is not evidence. Of the four problems hit bringing up the first provider, three appeared only here — a kit lever whose featurizer hangs, a Triton kernel that reports OutOfResources on this compute capability, and an nvidia-smi field that returns [N/A] on unified memory. The first of those presents as a stall, which is why the deadline is not optional.

What it writes is the point of the step. provider.toml’s numbers are the reference they were measured against; the file this writes is what this machine actually did, and a scheduler reads that one.

tooluniverse.remote_resident.setup.run_setup(slug_or_path, *, only=None)

Every step in order, stopping at the first that is not ok.

Stopping is deliberate: each step assumes the previous one holds, and a cascade of failures caused by one missing thing buries the one that mattered. Re-running after fixing it continues from where it stopped, because every step reports present for work already done.

tooluniverse.remote_resident.setup.exit_code(results)

tooluniverse.remote_resident.supervisor module

Start, reuse and retire one provider’s resident model worker.

A GPU remote tool costs most of its wall clock in work that is identical for every request: loading weights, building the CUDA context, capturing the kernels a first pass compiles. Measured for Boltz-2 on this project’s GB10 box, a fresh worker per request spends ~102 s where a held one spends ~47 s, and the structure pass alone falls from ~23 s to ~5.8 s because CUDA graphs captured on one request are replayed on the next. That is what this module exists to hold on to.

Holding a model resident spends device memory the whole time, which is the opposite of what a box running several models wants. Both are served by the same object: a worker is started on the first request, reused while requests keep arriving, and shut down once idle_timeout_s passes with none – so a provider nobody is calling is not occupying the card. vram_budget_gb from the provider’s manifest is what a scheduler above this reads to decide how many providers may be resident at once; this module reports it and never enforces it.

One request at a time per worker, by construction. Two requests sharing one CUDA context would interleave their allocations and make an out-of-memory attributable to neither, so callers queue on ResidentWorker.lock rather than racing.

exception tooluniverse.remote_resident.supervisor.WorkerStartError

Bases: RuntimeError

The worker could not be started, or did not bind its socket in time.

exception tooluniverse.remote_resident.supervisor.WorkerRequestError

Bases: RuntimeError

The worker answered, and the answer was a failure.

__init__(message, reason='request_failed')

tooluniverse.remote_resident.supervisor.exit_hint(rc)

The clause naming what rc means, or nothing when it has no name.

tooluniverse.remote_resident.supervisor.child_env(extra_env, environ)

The child’s environment: environ, with extra_env applied over it.

A launcher removes a name by mapping it to None. It cannot remove one by returning a pruned copy of environ, and this function exists as a separate, testable thing because that distinction was got wrong and the wrong version passed review. af3_jax’s launcher pruned a copy; the supervisor then rebuilt the child’s environment from os.environ and applied the pruned copy over it, so every name the launcher had removed came straight back. An operator with a stray AF3_JAX_BOGUS exported got a worker that exited 3 at import alphafold3 – exactly the failure the pruning existed to prevent.

The check that missed it read build_command’s RETURN VALUE, which is the one place a pruned copy looks correct. A launcher’s intent has to be expressed in something the supervisor acts on.

class tooluniverse.remote_resident.supervisor.ResidentWorker

Bases: object

One provider’s model, held in a process of its own.

slug: str

build_command: Callable[[ResidentWorker], Tuple[List[str], Dict[str, str | None]]]

start_timeout_s: float = 900.0

request_timeout_s: float = 3600.0

idle_timeout_s: float = 0.0

vram_budget_gb: float | None = None

admit: bool = True

log_path: str | None = None

manifest: Any = None

lock: RLock

property socket_path: str

is_running()

ping()

The worker’s own answer, or None when nothing is listening.

ensure_started()

Start the worker if it is not already serving. Holds lock.

STARTING IS WHERE THE BUDGET IS SPENT, so it is where the budget is checked. admission.room_for adds up what is already held and reads the machine, and a refusal raises AdmissionRefused rather than starting a worker the host cannot carry – this project wedged the host once by starting one anyway, with the numbers available.

admit is the escape hatch and it is an ATTRIBUTE, not an environment variable: a stray name in the environment is the class of mistake this module’s siblings are full of, and a host that genuinely wants an unchecked start says so in the object it is holding.

request(payload)

One request, serialized against every other on this worker.

A worker that dies answering is named rather than guessed at, in both of the shapes that takes. Usually it manages a last reply first – worker_base sends one for any SystemExit out of a prediction, which is how the kits’ KernelsRefused arrives – and then the only thing missing is what the worker could not know: where its transcript is, and what its exit code is called. Sometimes it cannot answer at all (os._exit, the OOM killer, a segfault), and then the socket just ends; a bare ProtocolError: peer closed the connection mid-message is true and useless, so the code it exited with goes in its place.

stop(*, grace_s=30.0)

Ask the worker to exit, then make sure it did. Frees the device.

__init__(slug, build_command, start_timeout_s=900.0, request_timeout_s=3600.0, idle_timeout_s=0.0, vram_budget_gb=None, admit=True, log_path=None, manifest=None, lock=<factory>, _proc=None, _log=None, _last_used=0.0, _reaper=None, _stopping=False)

tooluniverse.remote_resident.worker_base module

The half of a resident worker that is the same for every model.

This file runs inside the model’s environment, not ToolUniverse’s, and is loaded by path for the same reason protocol.py is: that environment holds a pinned scientific stack and nothing of this package, and importing tooluniverse there would drag in a dependency set it has no reason to carry. Its own imports are therefore the standard library only; the one function that needs the model’s stack (release_cublas_workspaces()) imports inside the call.

A provider’s worker script supplies what is genuinely its own — building the model, and turning one request into one prediction — and calls serve() with them. Everything below is what every model needs identically: the socket, the request loop, a failure that does not take the model down with it, and the two ways a held model gives the card back.

Giving the card back is the part worth being explicit about. A held model is fast because it keeps weights, a CUDA context and compiled kernels resident, and that is exactly why it must stop holding them when nobody is asking. The supervisor retires an idle worker; that covers the ordinary case. It does not cover its own death, and on this project SIGKILLing an MCP server left two worker processes holding a GPU that nothing could then reach. So the worker also watches for being orphaned — reparented to init — and exits once a grace period has passed with no request, long enough that a prediction in flight when the parent died still finishes and writes its answer.

tooluniverse.remote_resident.worker_base.protocol()

The wire protocol module, by path — never as a ToolUniverse import.

tooluniverse.remote_resident.worker_base.serve(*, slug, socket_path, on_predict, ready=None, on_reset=None, log=None, orphan_grace_s=None, release_cublas=True)

Answer requests on socket_path until told to stop.

on_predict receives the request dict and returns the body of a successful reply. Raising is fine and expected: the failure is named back to the caller and the model stays up, because a bad request must not cost the minutes it took to build one. on_reset runs just before each prediction (peak-memory counters and the like); ready adds fields to the ping.

Raising something that is not an Exception is the other case, and it is not a failed request: the caller is told and then the worker goes down. _fatal_reply() is where that is argued, because the argument is the whole of why this loop has two except clauses instead of one.

One request at a time, by construction. Two inside one CUDA context would interleave their allocations and make an out-of-memory attributable to neither, so this loop accepts and answers serially and callers queue.

After each prediction it calls release_cublas_workspaces(), which is where the reasoning for doing that for everybody is written down. A provider that wants it off passes release_cublas=False; so does TOOLUNIVERSE_RESIDENT_NO_CUBLAS_RELEASE=1, which exists to reproduce the leak rather than to live in a deployment.

tooluniverse.remote_resident.worker_base.bootstrap(worker_file)

Load this module from a provider’s worker script.

Every provider’s worker needs the same three lines to reach here, so they live here instead:

import importlib.util, pathlib
_wb = importlib.util.module_from_spec(...)   # see the providers

In practice a worker calls sys.path()-free path loading directly; this helper exists for the symmetric case where the base is found relative to the worker rather than through the environment.

tooluniverse.remote_resident.worker_base.release_cublas_workspaces()

Give back the cuBLAS workspaces torch caches per (handle, stream).

Returns the bytes freed, or 0 on a build with no such call and on CPU.

WHY A HELD MODEL NEEDS THIS. torch allocates a cuBLAS workspace the first time a GEMM runs on a (handle, stream) pair and keeps it in a C++ static map that nothing clears. A kit that captures a CUDA graph creates a fresh side stream per request to warm up on before capturing, so every request abandons a stream and leaves its workspace behind – measured at exactly torch.backends.cuda.blas_workspace_size(), 32 MiB on this GB10, per request on byte-identical input.

It is invisible to every ordinary check. The bytes are in memory_allocated but held by no Python object, so a census of every reachable CUDA tensor stays frozen while the number climbs, gc.collect() returns nothing, and empty_cache() returns nothing – it only drops blocks the allocator is caching, and these are live. What it looks like from outside is one new fully-allocated allocator segment per request, on a stream id that was not there before. chai1 and genie3 each grew ~0.04 GiB per request until this call, which is 140 requests to chai1’s whole declared budget; their manifests carry the measurements.

A HELD GRAPH, MEASURED RATHER THAN FEARED. This used to say the call was safe only where no captured graph outlives the request, on the reasoning that a graph which captured cuBLAS calls holds the workspace pointer, and on that reasoning it was a call each provider had to make for itself. rosettafold3 is why it finally got tested rather than repeated: its fast mode keeps pairformer-stack graphs across folds, so it could not use the argument. The reasoning is wrong twice over.

• A replay cannot read a freed workspace’s contents, because the workspace is scratch that each kernel writes before it reads. cudaMemset it to 0xAB and the replay returns the same bits, so “the output did not change” was never evidence of anything either.

• The failure that WOULD be real is the other direction – the replay still writes to that address, and writing over a tensor the allocator has since handed to someone else corrupts it. It does not happen: a workspace allocated during a capture belongs to that graph’s PRIVATE memory pool (segment_pool_id (1, 0), not (0, 0)), so clearing the map hands the block back to a pool only a capture into the same pool draws from.

Checked on every torch in this fleet – 2.7.0+cu128, 2.7.1+cu128, 2.8.0+cu129, 2.10.0+cu128, 2.11.0+cu128, 2.11.0+cu130, 2.12.0+cu130, 2.13.0+cu130 – and on each one with a POSITIVE CONTROL first, because without one the test is vacuous and two of the first three written here were. The control reads the workspace bytes back to show that this build’s replay does write to them; which GEMM does that is a function of the cuBLAS version and has to be searched for per build (fp32 mm works everywhere here; fp16 writes on 13.0 and not on 12.8, and a square 1024**3 writes on neither). With the control passing, 34 workspaces’ worth of live tensors on the default stream and on fresh ones never landed on the address, twenty cleared replays scribbled on nothing, and with five graphs captured and four dropped the survivor still wrote only into its own private pool. The workspace is 32 MiB on torch >= 2.12 and 8.12 MiB before it, and on the older builds it sits INSIDE a 20 MiB segment – a probe that filters segments by size rather than blocks finds nothing there and proves nothing, which is how the first pass missed it.

So this is not a call a provider has to prove its graphs dead to make, and it is no longer a call a provider makes at all: serve() does it after every request, for everybody. See NO_CUBLAS_RELEASE_ENV for the way back out.