Sociaro

Failures and retries — failed jobs, submit errors and lost answers

This page covers a job that ends failed, the errors a submit can answer with, and how to find a job whose answer you did not receive; the envelope is on Jobs API.

Failed jobs

A failed job carries error, and no outputs:

{"id": "job_4be56281b681608b24640e8b82f56cf1",
 "status": "failed",
 "error": {"code": "provider_failed",
           "type": "upstream_error",
           "message": "The request failed because the output video may be related to copyright restrictions.",
           "provider_message": "The request failed because the output video may be related to copyright restrictions.",
           "provider_code": "OutputVideoSensitiveContentDetected.PolicyViolation"}}

Reading the error

Switch on error.type, log error.code, read error.message. type is one of the nine words on the Errors page, and a failed job's code maps to it like this:

error.code error.type
provider_rejected invalid_input
provider_failed, timeout, expired, output_too_large, unsupported_media upstream_error
quota_exceeded forbidden
internal internal_error
canceled none: a cancel you asked for is not an error, so a canceled job's error has code and message only

provider_rejected has the same type as a refusal at submit, but on a job object it means the job existed and was not charged; on a submit answer, job_created tells you whether a job exists. code is ours, from the fixed set below. provider_message is the generator's own sentence with company names, links and e-mail addresses removed, at most 400 characters, and it equals message where it is present. It is present only when the code is provider_rejected or provider_failed and the generator gave a reason; in every other case (timeout, expired, quota_exceeded, internal, canceled) the key is absent, not null, so read it as optional. The same error object is in GET /v2/jobs/{id}, in a repeated submit with the same key and in the lookup by key. When the generator gave a reason, for a failure while rendering (provider_failed) or a refusal at submit (provider_rejected), message is as a rule its own sentence, up to 400 characters; otherwise message is a fixed sentence such as "the provider failed to complete this job". provider_code is as a rule there when the generator gave a code, with or without a sentence of its own. Some refusals and failures carry only the fixed sentence and no provider_code even then, so read both as optional. The names of the services and hosts that run the job are substituted out of that text, the halves of the model id you called are left alone, and a link or an e-mail address quoted back becomes [link] or [email]. The text can quote your own request back, so show it rather than store it.

A busy generator

A generator that is still busy on the last retry we make, answering 429, ends the job provider_failed with the generator's sentence and its provider_code when it gave them (else the generator is at capacity, retry shortly), and it may be sent again later with a new Idempotency-Key. A job that runs out of retries for any other reason — a 408 on the last one, or an internal error after the generator's busy answer — ends internal.

What a failed job costs

A failed job is not charged, except an Ideogram job whose images were all withheld by the generator's safety check: it ends failed with cost_usd set to the images produced, exactly as on /v1. Any other failed job's cost_usd stays absent and the amount held for it is released.

The codes

code comes from a fixed set, and these two are the ones worth telling apart:

  • provider_rejected — the model refused the request before rendering. Usually input: a parameter it does not take, or the wrong prompt shape for that family (see Model inputs).
  • provider_failed — the model accepted the request and then did not finish it (or was still busy on the last retry we made at submit). On Seedance this is most often the generated soundtrack rather than the picture: when you ask for sound ("generate_audio": true) it checks that audio after the video has rendered. Sound is off unless you send true, except with your own audio (reference_audio) or the field omni_reference_task_type set to edit or extend (2.5 models only; a 2.0 edit or extension is asked for in the prompt and comes back silent unless you send "generate_audio": true or send your own audio (reference_audio), with which sound is on by default), where the generator's default (on) applies; send false there if you do not need sound.

The rest of the set, in the order you are likely to meet them: timeout (did not finish in time), expired (exceeded its maximum runtime; on Sociaro Leda the message reads "no GPU worker finished this job in time; please retry", and the same request may be sent again with a new Idempotency-Key), canceled (you canceled it), quota_exceeded (your account's storage is full), output_too_large (the generated file exceeded the allowed size), unsupported_media (the output could not be stored) and internal (an internal error). A code outside this list is not something the API produces; type is the coarser switch, and message is the sentence to show.

Errors at submit

Every error answer of a /v2 route has the same error object, {"type", "code", "message", "provider_message"}, and the errors of POST /v2/jobs also carry job_created at the top level; the Errors page has the full list of types and codes.

{"error": {"type": "invalid_input", "code": "invalid_input",
           "message": "input.duration must be an integer from 4 to 15", "provider_message": null},
 "job_created": false}
  • type is one of nine words: invalid_input, unauthorized, forbidden, not_found, insufficient_balance, rate_limited, concurrency_limited, upstream_error, internal_error. Switch on type; log code. Every code that existed before is unchanged, with one exception (a key's own rate limit on POST /v2/jobs is now rate_limit_exceeded; it was insufficient_budget), and type was added beside it.
  • provider_message is always null on a submit error: no generator is called before a job exists.
  • 429 is split three ways. insufficient_balance (code insufficient_budget): your organisation's balance, top up. rate_limited (code rate_limit_exceeded): the API key's own rate limit; wait for Retry-After. concurrency_limited (codes too_many_inflight_jobs, too_many_uploads): too many of your jobs or uploads in flight at once; Retry-After is set. A 429 whose cause we cannot classify is reported as insufficient_balance.
  • job_created is false when this attempt created nothing: release the user's credits unless an earlier attempt with the same key ended ambiguously (see the null case below). On 409 idempotency_conflict it is false for this request, and the key already belongs to an earlier job with another body. It is null when we cannot tell: a 503 after the job may have been stored or after a timeout inside the submit (a wait on an earlier attempt with the same key), a 502 or 504 from the gateway, or a 500 internal_error (an unexpected internal error). An answer that is not JSON, or no answer at all, means the same as null. A success answer (202, or 200 for a repeated key) carries the job and no job_created.
  • On null, hold the credits and send the same request again with the same Idempotency-Key until the answer is the job (200 or 202), or look the key up: a 404 from the lookup settles it only under the lookup rule, and then you can release them. A later false does not release them: it speaks for its own attempt only.

Look up a job by its key

GET /v2/jobs?idempotency_key=<key>, with the same API key that submitted. Use it after a submit whose answer you did not receive. It never returns a list. It never creates a job and never starts a generation or a charge. Like reading the job by id, it reports the job's current state, and if the job has already finished at the generator it may record that result now: its final status, its outputs and the one charge that job gets anyway.

curl "https://api.sociaro.com/v2/jobs?idempotency_key=my-request-0001" \
  -H "Authorization: Bearer $SOCIARO_API_KEY"
  • 200: the same object GET /v2/jobs/{id} returns, with retry_after while the job is unfinished.
  • 404 job_not_found: no job under this key. A key that belongs to another organisation answers exactly like a key that does not exist.
  • 400: the parameter is missing (invalid_input) or is not a valid key (invalid_idempotency_key: 1 to 255 characters of A-Z a-z 0-9 . _ : -). 401 and 503 as elsewhere.
  • 503 with Retry-After and the message "a request with this Idempotency-Key is still being processed": a submit with this key is being stored right now. It is not an answer about the job: ask again after Retry-After. The lookup never answers 404 while that is going on.

After a lost answer

After a 502, 503 or 504 on submit, or job_created: null, send the same request again with the same Idempotency-Key. It is safe: it returns the existing job or creates the one job, never two. The lookup is for finding a job without sending the request again: a 200 means the job exists. A 404 from a lookup you sent at least 180 seconds after your last attempt with that key ended (an answer arrived, the connection closed, or you stopped waiting and closed it) is final: no job exists, none of those attempts will create one, and you can release the user's credits. Count from the end of the attempt, not from its start, and from the attempt that ended last if you sent the same key more than once. What counts is when you send the lookup, not when its answer arrives: a 404 from a lookup sent earlier means only that no job existed at that moment, however late you receive it. An attempt may still be in progress, so it is not proof. Send the lookup again once that time has passed. This holds for requests sent to the public API host, on the condition that a request reaches our server within 90 seconds after your attempt with that key ended; sending the same request again with the same Idempotency-Key is always safe and settles the outcome without relying on timing (it returns the existing job or creates exactly one).

Releasing the user's credits

job_created: false means this attempt created nothing. Release the user's credits on it when no earlier attempt with the same Idempotency-Key ended ambiguously (502, 503, 504, job_created: null, or no answer at all). After an ambiguous attempt, keep the credits held and either send the same request again with the same key until the answer is the job (200 or 202), or look the key up: the job, or a 404 that the lookup rule calls final, settles it. Until then the outcome of the earlier attempt is not known, because it may still create the job: an earlier 404 from the lookup is not proof, and a refusal of a later attempt (a 401, 403, 429 or 503, for example) or its job_created: false settles only that attempt. A job that does appear is charged once.

How long keys are remembered

Keys are remembered for at least 90 days from the first submit; today they are not removed at all, and if we ever start removing them we will announce it and never go below 90 days. After a job's own data is deleted (retention_until), the lookup still returns the job's final status, without outputs.