Failures and retries — failed jobs, submit errors and lost answers
This page covers a job that ends failed, the errors a submit can answer with, and how to find a job whose answer you did not receive; the envelope is on Jobs API.
Failed jobs
A failed job carries error, and no outputs:
{"id": "job_4be56281b681608b24640e8b82f56cf1",
"status": "failed",
"error": {"code": "provider_failed",
"type": "upstream_error",
"message": "The request failed because the output video may be related to copyright restrictions.",
"provider_message": "The request failed because the output video may be related to copyright restrictions.",
"provider_code": "OutputVideoSensitiveContentDetected.PolicyViolation"}}
Reading the error
Switch on error.type, log error.code, read error.message. type is one of the nine words on the
Errors page, and a failed job's code maps to it like this:
error.code |
error.type |
|---|---|
provider_rejected |
invalid_input |
provider_failed, timeout, expired, output_too_large, unsupported_media |
upstream_error |
quota_exceeded |
forbidden |
internal |
internal_error |
canceled |
none: a cancel you asked for is not an error, so a canceled job's error has code and message only |
provider_rejected has the same type as a refusal at submit, but on a job object it means the job existed and was
not charged; on a submit answer, job_created tells you whether a job exists. code is ours, from the fixed set
below. provider_message is the generator's own sentence with company names, links and e-mail addresses removed, at
most 400 characters, and it equals message where it is present. It is present only when the code is
provider_rejected or provider_failed and the generator gave a reason; in every other case (timeout, expired,
quota_exceeded, internal, canceled) the key is absent, not
null, so read it as optional. The same error object is in GET /v2/jobs/{id}, in a repeated submit with the same
key and in the lookup by key. When the generator gave
a reason, for a failure while rendering (provider_failed) or a refusal at submit (provider_rejected), message
is as a rule its own sentence, up to 400 characters; otherwise message is a fixed sentence such as "the provider
failed to complete this job". provider_code is as a rule there when the generator gave a code, with or without a
sentence of its own. Some refusals and failures carry only the fixed sentence and no provider_code even then, so
read both as optional. The names of the
services and hosts that run the job are substituted out of that text, the halves of the model id you called are left
alone, and a link or an e-mail address quoted back becomes [link] or [email]. The text can quote your own request back, so show it rather
than store it.
A busy generator
A generator that is still busy on the last retry we make, answering 429, ends the job provider_failed with the
generator's sentence and its provider_code when it gave them (else the generator is at capacity, retry shortly),
and it may be sent again later with a new Idempotency-Key. A job that runs out of retries for any other reason — a
408 on the last one, or an internal error after the generator's busy answer — ends internal.
What a failed job costs
A failed job is not charged, except an Ideogram job whose images were all withheld by the generator's safety
check: it ends failed with cost_usd set to the images produced, exactly as on /v1. Any other failed job's
cost_usd stays absent and the amount held for it is released.
The codes
code comes from a fixed set, and these two are the ones worth telling apart:
provider_rejected— the model refused the request before rendering. Usuallyinput: a parameter it does not take, or the wrong prompt shape for that family (see Model inputs).provider_failed— the model accepted the request and then did not finish it (or was still busy on the last retry we made at submit). On Seedance this is most often the generated soundtrack rather than the picture: when you ask for sound ("generate_audio": true) it checks that audio after the video has rendered. Sound is off unless you sendtrue, except with your own audio (reference_audio) or the fieldomni_reference_task_typeset toeditorextend(2.5 models only; a 2.0 edit or extension is asked for in the prompt and comes back silent unless you send"generate_audio": trueor send your own audio (reference_audio), with which sound is on by default), where the generator's default (on) applies; sendfalsethere if you do not need sound.
The rest of the set, in the order you are likely to meet them: timeout (did not finish in time),
expired (exceeded its maximum runtime; on Sociaro Leda the message reads "no GPU worker finished this job
in time; please retry", and the same request may be sent again with a new Idempotency-Key), canceled (you canceled it), quota_exceeded (your
account's storage is full), output_too_large (the generated file exceeded the allowed size),
unsupported_media (the output could not be stored) and internal (an internal error). A code outside this list is not
something the API produces; type is the coarser switch, and message is the sentence to show.
Errors at submit
Every error answer of a /v2 route has the same error object, {"type", "code", "message", "provider_message"},
and the errors of POST /v2/jobs also carry job_created at the top level; the Errors page has the full
list of types and codes.
{"error": {"type": "invalid_input", "code": "invalid_input",
"message": "input.duration must be an integer from 4 to 15", "provider_message": null},
"job_created": false}
typeis one of nine words:invalid_input,unauthorized,forbidden,not_found,insufficient_balance,rate_limited,concurrency_limited,upstream_error,internal_error. Switch ontype; logcode. Everycodethat existed before is unchanged, with one exception (a key's own rate limit onPOST /v2/jobsis nowrate_limit_exceeded; it wasinsufficient_budget), andtypewas added beside it.provider_messageis alwaysnullon a submit error: no generator is called before a job exists.429is split three ways.insufficient_balance(codeinsufficient_budget): your organisation's balance, top up.rate_limited(coderate_limit_exceeded): the API key's own rate limit; wait forRetry-After.concurrency_limited(codestoo_many_inflight_jobs,too_many_uploads): too many of your jobs or uploads in flight at once;Retry-Afteris set. A429whose cause we cannot classify is reported asinsufficient_balance.job_createdisfalsewhen this attempt created nothing: release the user's credits unless an earlier attempt with the same key ended ambiguously (see thenullcase below). On409 idempotency_conflictit isfalsefor this request, and the key already belongs to an earlier job with another body. It isnullwhen we cannot tell: a503after the job may have been stored or after a timeout inside the submit (a wait on an earlier attempt with the same key), a502or504from the gateway, or a500internal_error(an unexpected internal error). An answer that is not JSON, or no answer at all, means the same asnull. A success answer (202, or200for a repeated key) carries the job and nojob_created.- On
null, hold the credits and send the same request again with the sameIdempotency-Keyuntil the answer is the job (200or202), or look the key up: a404from the lookup settles it only under the lookup rule, and then you can release them. A laterfalsedoes not release them: it speaks for its own attempt only.
Look up a job by its key
GET /v2/jobs?idempotency_key=<key>, with the same API key that submitted. Use it after a submit whose answer you
did not receive. It never returns a list. It never creates a job and never starts a generation or a charge. Like reading the job by id, it reports the job's current state, and if the job has already finished at the generator it may record that result now: its final status, its outputs and the one charge that job gets anyway.
curl "https://api.sociaro.com/v2/jobs?idempotency_key=my-request-0001" \
-H "Authorization: Bearer $SOCIARO_API_KEY"
200: the same objectGET /v2/jobs/{id}returns, withretry_afterwhile the job is unfinished.404job_not_found: no job under this key. A key that belongs to another organisation answers exactly like a key that does not exist.400: the parameter is missing (invalid_input) or is not a valid key (invalid_idempotency_key: 1 to 255 characters ofA-Z a-z 0-9 . _ : -).401and503as elsewhere.503withRetry-Afterand the message "a request with this Idempotency-Key is still being processed": a submit with this key is being stored right now. It is not an answer about the job: ask again afterRetry-After. The lookup never answers404while that is going on.
After a lost answer
After a 502, 503 or 504 on submit, or job_created: null, send the same request again with the same
Idempotency-Key. It is safe: it returns the existing job or creates the one job, never two. The lookup is for finding
a job without sending the request again: a 200 means the job exists. A 404 from a lookup you sent at least 180 seconds after
your last attempt with that key ended (an answer arrived, the connection closed, or you stopped waiting and closed it) is final: no job
exists, none of those attempts will create one, and you can release the user's credits. Count from the end of the
attempt, not from its start, and from the attempt that ended last if you sent the same key more than once. What counts
is when you send the lookup, not when its answer arrives: a 404 from a lookup sent earlier means only that no job
existed at that moment, however late you receive it. An attempt may still be in progress, so it is not proof. Send
the lookup again once that time has passed. This holds for requests sent to the public API host, on the condition that a request reaches our server within 90 seconds after your attempt with that key ended; sending the same request again with the same Idempotency-Key is always safe and settles the outcome without relying on timing (it returns the existing job or creates exactly one).
Releasing the user's credits
job_created: false means this attempt created nothing. Release the user's credits on it when no earlier attempt with the same Idempotency-Key ended ambiguously (502, 503, 504, job_created: null, or no answer at all). After an ambiguous attempt, keep the credits held and either send the same request again with the same key until the answer is the job (200 or 202), or look the key up: the job, or a 404 that the lookup rule calls final, settles it. Until then the outcome of the earlier attempt is not known, because it may still create the job: an earlier 404 from the lookup is not proof, and a refusal of a later attempt (a 401, 403, 429 or 503, for example) or its job_created: false settles only that attempt. A job that does appear is charged once.
How long keys are remembered
Keys are remembered for at least 90 days from the first submit; today they are not removed at all, and if we ever
start removing them we will announce it and never go below 90 days. After a job's own data is deleted
(retention_until), the lookup still returns the job's final status, without outputs.