Cloud errors: handle every failure the API returns
Provokes every failure the smol cloud API can return and shows what a client must branch on, including the failures that arrive as HTTP 200. Use when writing a client, harness or agent tool against the cloud API; when a call succeeded by status but the work did not happen; when a 422 needs to be told apart from a schema bug, a wrong type and a quota refusal; or when deciding which failures are safe to retry. Do not use it to drive a normal machine lifecycle, which is the cloud-machine packet, and do not expect it to cover billing suspension or rate limiting, which cannot be provoked on a healthy account.
Verified on smolfleet API 0.1.0 and smol v1.32.4, macOS 27.2 arm64 client, on
2026-11-09, twice, each time from a fresh HOME; first written on smol v1.14.3 on 2026-09-10. One
pass cost 911 micros (USD 0.00091) for 5 uptime-seconds; most of the packet costs nothing,
because most failures happen before a machine exists. What the re-run found changed is under Corrections; the steps and traps are as first written.
Three rules:
- A failing guest command returns HTTP 200. The exit code is in the body and nowhere else.
- Every error is
application/problem+json. Branch on itscode, read itsdetail, and never parse the sentence. retryablesays whether to send the request again, and a retryable error carriesRetry-After. A503with codemachine_interruptedis retryable, but the command may have partly run, so a retry runs it again from the start.
Updated 2026-10-12 against smolfleet a5580fd0: the problem bodies, retryable, Retry-After and the unauthenticated 404 were checked against the live API, and machine_interrupted against the control plane’s test suite. The rest of the procedure below
is the 2026-09-10 run.
Procedure
1. The failures that need no machine. Free.
scripts/provoke-http.sh unknown_key_401=ok (401)
nonexistent_path_404=ok (404)
missing_scope_403=ok (403)
not_json_400=ok (400)
missing_field_422=ok (422)
wrong_type_422=ok (422)
no_such_machine_404=ok (404)
result=all_provoked 2. The failures that arrive as 200. Creates one small machine.
scripts/provoke-exec.sh
scripts/cleanup.sh --reap duplicate_name_409=ok (409)
nonzero_exit_status=ok (200) nonzero_exit_code=ok (42)
timeout_status=ok (200) timeout_exit_code=ok (124)
missing_binary_status=ok (200) missing_binary_exit_code=ok (255)
result=all_provoked What to branch on
Every row but the last is a problem body: {"type","title","status","detail","code","retryable"}.
| Situation | Status | code | What a client must do |
|---|---|---|---|
| Bad or missing key | 401 | unknown_key, unauthorized | Stop. Do not retry with the same key |
| Path that does not exist | 404 | not_found | Fix the path. The detail links the OpenAPI document |
| Missing scope | 403 | forbidden | Stop. detail names the scope; the key needs minting again |
| Body is not JSON | 400 | bad_request | Fix the request |
| Body parses, schema unsatisfied | 422 | unprocessable | Fix the request. Not retryable |
| Semantic rule broken | 400 | bad_request | Fix the request. See the counterexample below |
| No such machine | 404 | not_found | Stop |
| Duplicate name | 409 | conflict | Choose another name |
| Rate limited | 429 | too_many_requests | Wait Retry-After, then resend. It never ran |
| Machine went away under an exec | 503 | machine_interrupted | Wait Retry-After. Resend only if the command is safe to run twice |
| Briefly unavailable | 502, 503, 504 | varies | Wait Retry-After, then resend |
| Route deliberately unimplemented | 501 | not_implemented | Stop. It will not start working |
| Guest command failed | 200 | none, JSON exitCode non-zero | Read exitCode |
Guest exit codes worth knowing: 42 is whatever the command returned, 124 is a timeout, and 255 is an executable that was not found.
The 400 versus 422 split does not hold
The documented split is that 400 means malformed and 422 means invalid. Neither direction is
reliable:
- A wrong field name, a wrong type, and being over quota are all
422with prose bodies. A client that treats422as a quota problem and retries will retry a schema bug forever. - A semantic rule on a well-formed body can be
400:
POST /v1/machines {"network":{"mode":"allowCidrs","cidrs":[]}}
allowCidrs network mode requires at least one CIDR or host
http=400 So a client cannot triage by status alone. The code narrows it, but every 422 shares unprocessable, so a quota refusal and a typo still differ only in detail.
Security defaults, and why they are the defaults
- Nothing here retries. Every failure this packet provokes is a client error
(
retryable: false), and the retryable ones cannot be provoked on a healthy account. - The 403 body names the missing scope, so a scope failure is diagnosable without widening the key. Read it rather than minting a broader key by reflex.
- The bad-key probe uses a deliberately invalid string, never a real key with a character changed, so nothing valid is ever sent to a log.
- Cleanup deletes only what this packet recorded, and the machine it makes is the smallest the plan allows.
Platform arms
- macOS arm64 client: verified. The provocations are
curland a URL and depend on nothing platform specific. - Linux and Windows clients: not run. Note that on Windows the API has been recorded returning empty bodies for 400 and 404, which would make a wrong field name undiagnosable; that was not re-run here.
What was not run
402, billing restriction. Needs a suspended or over-budget tenant. Not provokable on a healthy account, and not guessed.429, rate limiting. The budget is roughly 10000 requests per period and a burst did not approach it.x-ratelimit-remainingis returned and is undocumented; it is the cheapest way to see what a burst consumed.5xx. Nothing here can cause one on purpose.machine_interruptedis covered by the control plane’s tests instead: an exec whose machine is stopped under it.501was recorded from the export route by an earlier run and was not re-provoked here.
Corrections
2026-11-09, smol v1.32.4
Both scripts ran as written. provoke-exec.sh gave every line above. provoke-http.sh gave every
line but one:
missing_scope_403depends on the key. The key used on this run holdsvolume:read, soGET /v1/volumesanswered200with[]and the probe reportedFAIL. A route the key cannot reach still gives the 403:GET /v1/billing/metersanswered403withmissing scope: admin. Probe a route your own key’sAccessline does not cover.- The rate limit headers have standard names too. Besides
x-ratelimit-remaining, responses carryratelimit-limit(10000 on this account) withratelimit-remainingandratelimit-reset.
Related packets
cloud-authfor the credential, and for whysmol auth statuscannot gate anything.cloud-machinefor the lifecycle these failures interrupt, and the same 200-on-failure rule applied to a real run.
Scripts
The files the procedure runs, in the order it runs them. It calls each one by the path in its heading, relative to the directory the procedure is saved in.
scripts/lib.sh
# Shared by this packet's scripts. Sourced, not run.
# Never echoes the credential.
API="${SMOL_CLOUD_URL:-https://api.smolmachines.com}"
PREFIX="smolskill-"
STATE="${SMOLSKILL_STATE:-${XDG_STATE_HOME:-$HOME/.local/state}/smol-skills}"
mkdir -p "$STATE"
IDFILE="$STATE/cloud-machines.ids"
api() { # api METHOD PATH [JSON_BODY]
local method="$1" path="$2" body="${3:-}"
if [ -n "$body" ]; then
curl -sS -X "$method" --max-time 120 \
-H "Authorization: Bearer $SMOL_CLOUD_TOKEN" -H 'content-type: application/json' \
-d "$body" "$API$path"
else
curl -sS -X "$method" --max-time 120 \
-H "Authorization: Bearer $SMOL_CLOUD_TOKEN" "$API$path"
fi
}
jget() { python3 -c "import json,sys
try: d=json.load(sys.stdin)
except Exception: print(''); sys.exit(0)
for k in sys.argv[1].split('.'):
if d is None: break
d = d.get(k) if isinstance(d, dict) else None
print('' if d is None else d)" "$1"; }
require_token() {
[ -n "${SMOL_CLOUD_TOKEN:-}" ] && return 0
echo "credential=FAIL expected=SMOL_CLOUD_TOKEN set actual=unset"; echo "result=cannot_run"; exit 1
}
record() { echo "$1" >> "$IDFILE"; } # only ids this packet created scripts/provoke-http.sh
#!/usr/bin/env bash
# Provokes every documented failure that needs no machine, and asserts the status
# AND the body shape, because on this API the two do not agree about the format.
# Creates nothing and bills nothing.
set -uo pipefail
. "$(dirname "$0")/lib.sh"
require_token
fails=0
# probe NAME EXPECTED_STATUS EXPECTED_BODY_SUBSTRING curl-args...
probe() {
local name="$1" want="$2" needle="$3"; shift 3
local body status
body=$(curl -sS --max-time 45 -o /tmp/ce.$$ -w '%{http_code}' "$@"); status="$body"
body=$(head -c 300 /tmp/ce.$$ | tr '\n' ' ')
if [ "$status" = "$want" ] && case "$body" in *"$needle"*) true;; *) false;; esac; then
echo "$name=ok ($status)"
else
echo "$name=FAIL expected=$want+'$needle' actual=$status '$body'"; fails=$((fails+1))
fi
}
H="Authorization: Bearer $SMOL_CLOUD_TOKEN"
J='content-type: application/json'
# Every error is problem+json; 401 carries code unknown_key.
probe unknown_key_401 401 unknown_key -H 'Authorization: Bearer smk_wrong' "$API/v1/machines"
# A route that does not exist is 404 even without a key, and points at the OpenAPI document.
probe nonexistent_path_404 404 "openapi.json" "$API/v1/nope"
# 403 names the exact missing scope in its detail.
probe missing_scope_403 403 "missing scope" -H "$H" "$API/v1/volumes"
# 400 is reserved for a body that is not JSON at all.
probe not_json_400 400 "" -X POST -H "$H" -H "$J" -d 'not json' "$API/v1/machines"
# 422 is everything that parses but does not satisfy the schema: three causes, one status.
probe missing_field_422 422 source -X POST -H "$H" -H "$J" -d '{"name":"x"}' "$API/v1/machines"
probe wrong_type_422 422 "" -X POST -H "$H" -H "$J" \
-d '{"source":{"type":"image","reference":"library/alpine:3.20"},"resources":{"cpus":"lots"}}' "$API/v1/machines"
# 404 carries a message, unlike the unauthenticated 401 above.
probe no_such_machine_404 404 "not found" -H "$H" "$API/v1/machines/mach-does-not-exist"
echo "---"
echo "note=every error is problem+json: branch on code, read detail, retry only when retryable"
echo "note=a 422 cannot be triaged by status: a typo, a wrong type and a quota refusal are all 422"
rm -f /tmp/ce.$$
[ "$fails" -eq 0 ] && echo "result=all_provoked" || echo "result=unexpected_shapes"
[ "$fails" -eq 0 ] scripts/provoke-exec.sh
#!/usr/bin/env bash
# The failures that arrive as HTTP 200. These are the ones that matter: no status
# check finds them. Creates one small machine and deletes it via cleanup.sh.
set -uo pipefail
. "$(dirname "$0")/lib.sh"
require_token
fails=0
check() { if [ "$2" = "$3" ]; then echo "$1=ok ($3)"; else echo "$1=FAIL expected=$2 actual=$3"; fails=$((fails+1)); fi; }
NAME="${PREFIX}err1"
body=$(printf '{"name":"%s","source":{"type":"image","reference":"library/alpine:3.20"},"resources":{"cpus":1,"memoryMb":256},"network":{"mode":"open"}}' "$NAME")
resp=$(api POST /v1/machines "$body")
id=$(printf '%s' "$resp" | jget id)
[ -n "$id" ] || { echo "created=FAIL response=$(printf '%s' "$resp" | head -c 200)"; exit 1; }
record "$id"; echo "machine_id=$id"
api POST "/v1/machines/$id/start" '{}' >/dev/null
deadline=$(( $(date +%s) + 120 ))
while [ "$(date +%s)" -lt "$deadline" ]; do
[ -n "$(api POST "/v1/machines/$id/exec" '{"command":["sh","-c","echo up"]}' | jget stdout)" ] && break
sleep 3
done
# A duplicate name is one of the few real 4xx a running machine can produce.
dup=$(curl -sS --max-time 45 -o /tmp/ce2.$$ -w '%{http_code}' -X POST \
-H "Authorization: Bearer $SMOL_CLOUD_TOKEN" -H 'content-type: application/json' \
-d "$body" "$API/v1/machines")
check duplicate_name_409 409 "$dup"; rm -f /tmp/ce2.$$
ex() { curl -sS --max-time 60 -o /tmp/ce3.$$ -w '%{http_code}' -X POST \
-H "Authorization: Bearer $SMOL_CLOUD_TOKEN" -H 'content-type: application/json' \
-d "$1" "$API/v1/machines/$id/exec"; }
field() { python3 -c "import json,sys;print(json.load(open('/tmp/ce3.$$')).get(sys.argv[1],''))" "$1" 2>/dev/null; }
# 1. A command that exited non-zero.
check nonzero_exit_status 200 "$(ex '{"command":["sh","-c","echo out; echo err >&2; exit 42"]}')"
check nonzero_exit_code 42 "$(field exitCode)"
# 2. A command that timed out. 124 is the shell's timeout convention.
check timeout_status 200 "$(ex '{"command":["sh","-c","sleep 30"],"timeoutSeconds":3}')"
check timeout_exit_code 124 "$(field exitCode)"
# 3. An interpreter the image does not have. Not a transport failure either.
check missing_binary_status 200 "$(ex '{"command":["python3","-c","print(1)"]}')"
mb=$(field exitCode); [ "$mb" != 0 ] && echo "missing_binary_exit_code=ok ($mb)" || { echo "missing_binary_exit_code=FAIL expected=non-zero actual=0"; fails=$((fails+1)); }
rm -f /tmp/ce3.$$
echo "---"
echo "note=every failure above is HTTP 200; exitCode in the body is the only signal"
[ "$fails" -eq 0 ] && echo "result=all_provoked" || echo "result=unexpected_shapes"
[ "$fails" -eq 0 ] scripts/cleanup.sh
#!/usr/bin/env bash
# Deletes only the machines this packet recorded, takes the settled bill from the
# delete, and proves both surfaces are clean. --reap also removes any machine
# carrying this packet's name prefix, for a run that threw before recording.
set -uo pipefail
. "$(dirname "$0")/lib.sh"
require_token
CEILING_MICROS="${SMOLSKILL_CEILING_MICROS:-1000000}" # USD 1, asserted below
total=0
if [ -f "$IDFILE" ]; then
while read -r id; do
[ -n "$id" ] || continue
# ?includeUsage=true returns the settled bill. A mid-life /usage read is a
# documented lower bound, so this is the only figure worth recording.
resp=$(api DELETE "/v1/machines/$id?includeUsage=true")
micros=$(printf '%s' "$resp" | sed -n 's/.*"totalMicros":\([0-9]*\).*/\1/p')
secs=$(printf '%s' "$resp" | sed -n 's/.*"totalUptimeSeconds":\([0-9]*\).*/\1/p')
echo "deleted=$id uptime_s=${secs:-0} micros=${micros:-0}"
total=$(( total + ${micros:-0} ))
done < "$IDFILE"
rm -f "$IDFILE"
fi
echo "settled_micros=$total"
if [ "${1:-}" = "--reap" ]; then
api GET /v1/machines | python3 -c "
import json,sys
for m in json.load(sys.stdin):
if (m.get('name') or '').startswith('$PREFIX'): print(m['id'])" | while read -r id; do
[ -n "$id" ] || continue
echo "reaped=$id"; api DELETE "/v1/machines/$id" >/dev/null
done
fi
# Both surfaces, because the CLI list and the API list are different code paths.
left=$(api GET /v1/machines | python3 -c "
import json,sys
print(sum(1 for m in json.load(sys.stdin) if (m.get('name') or '').startswith('$PREFIX')))")
echo "api_prefix_machines_left=$left"
if command -v smol >/dev/null 2>&1; then
smol cloud ls 2>/dev/null | grep -q "$PREFIX" && echo "cli_prefix_machines_left=some" || echo "cli_prefix_machines_left=0"
else
echo "cli_prefix_machines_left=skipped"
fi
over=no; [ "$total" -gt "$CEILING_MICROS" ] && over=yes
echo "over_ceiling=$over (ceiling=${CEILING_MICROS} micros)"
[ "$left" -eq 0 ] && [ "$over" = no ] && echo "result=clean" || echo "result=NOT_CLEAN"
[ "$left" -eq 0 ] && [ "$over" = no ] Traps, with the observation behind each
Reproduced on smolfleet API 0.1.0 on 2026-09-10 unless an item says otherwise.
“Check the HTTP status before decoding a success response” is backwards for exec
The status is 200 for a command that succeeded, one that exited 42, one that timed out and one whose interpreter was missing:
{"command":["sh","-c","echo out; echo err >&2; exit 42"]} -> 200, exitCode 42
{"command":["sh","-c","sleep 30"],"timeoutSeconds":3} -> 200, exitCode 124
{"command":["python3","-c","print(1)"]} -> 200, exitCode 255 The last one’s stderr reads executable file 'python3' not found in $PATH: No such file or directory. All three are
indistinguishable from success by status.
The CLI does the right thing and propagates the guest exit code; raw HTTP does not. Both SDKs
propagate it too, but their exec promise resolves for a failed command, so a try/catch around it catches transport errors only.
Error bodies are one format now
Since 2026-10, every error is application/problem+json:
{"code":"unknown_key","detail":"unknown API key","retryable":false,"status":401,"title":"Unauthorized","type":"about:blank"} Older reports of plain-text 400, 403, 404, 409, 422 and 501 bodies predate this.
Branch on code and retryable; show detail to a person.
Retry what says it is retryable, and wait as asked
retryable is true for 408, 429, 502, 503 and 504, and those responses carry Retry-After in seconds. Two cases differ in what a retry costs:
429was refused before anything ran, so resending is always safe.503 machine_interruptedmeans the machine stopped or became unreachable while the command ran. The command may have partly run, and a resend runs it again from the start. Resend only a command that is safe to run twice.
The Node and Python SDKs do this for you on reads, deletes, start, stop and resume, up to three
times (maxRetries / max_retries). They never resend an exec or a create; the error carries retryable and retryAfterMs (retry_after_s in Python), so the caller decides.
A 422 cannot be triaged by status, and neither can a 400
Three distinct causes, one status:
| Cause | Status |
|---|---|
Missing required field (source) | 422 |
Wrong type ("cpus":"lots") | 422 |
| Over plan quota | 422 |
And the reverse, a well-formed body failing a semantic rule with a 400:
{"network":{"mode":"allowCidrs","cidrs":[]}}
allowCidrs network mode requires at least one CIDR or host
http=400 So the status separates a parse failure from a validation failure in neither direction. Read the body.
A path that does not exist is 404, with or without a key
GET /v1/nope with no credential is a 404 problem whose detail points at https://smolmachines.com/openapi.json. (Before 2026-10 it was an empty-bodied 401.)
The 403 body names the exact missing scope
{"code":"forbidden","detail":"missing scope: volume:read","retryable":false,"status":403,...} That is more useful than the status and more reliable than comparing against smol auth status,
because it names what this call needed rather than what the key has.
Do not use a machine count to detect a failed create
A create that fails after the record exists rolls the record back. smol cloud deploy with a bad
image prints Created: ... (id: mach-...) and then deletes it, and that id 404s afterwards. Do
not chase the id from a failed deploy, and do not infer a leak from the printed line.
x-ratelimit-remaining exists and is undocumented
It is returned on responses and appears in no reference page. It is the cheapest way to see how much of the request budget a burst consumed, which matters because the rate limit is the one documented failure that a healthy account cannot easily provoke.
A duplicate name is one of the few real 4xx a healthy account produces
Creating a second machine with a name already in use returns a 409 conflict problem. It is
worth having in a test suite for that reason: most of this API’s failure surface needs either a
malformed request or a broken account, and this one needs neither.