← Blog 8 min read

Converting Thousands of Files Through an API

Converting one file through an API is a solved problem: post it, get it back, done. Converting forty thousand of them is a different exercise, and the naive version - a loop that fires one request per file as fast as it can - reliably produces the same three outcomes. You hit a limit you did not know existed, one bad file takes down a run that was 90% complete, and you cannot tell afterwards which files actually made it.

All three are avoidable. Here is how to structure the job.

Send batches, not loops

The single biggest change is to stop sending one file per request. POST /api/convert-batch takes up to 20 files in one request, runs one tool across all of them, and returns a ZIP:

curl -X POST https://privconvert.com/api/convert-batch \
  -H "Authorization: Bearer pk_live_..." \
  -F "tool=png-to-jpg" \
  -F "[email protected]" -F "[email protected]" -F "[email protected]" \
  -o converted.zip

The reason this matters is not convenience, it is arithmetic on your limits.

What batching actually saves - and what it does not

Be clear about this before you plan capacity, because it is easy to assume the wrong thing in either direction:

  • Rate limit: batching saves you 20x. A batch is one request against your requests-per-minute limit no matter how many files it carries. Twenty files sent one at a time consume twenty slots; sent as a batch they consume one.
  • Quota: batching saves you nothing. Each converted file counts as one conversion against your monthly quota either way. Twenty files cost twenty conversions, full stop.

So batching buys throughput, not cheaper conversions. That is the honest trade, and it is the one you want - the per-minute rate is what stalls a bulk job, while the monthly quota is a budgeting question you settle before you start.

Worth doing the sum for your own plan. At 60 requests per minute, one file at a time is 3,600 files an hour. The same 60 requests carrying 20 files each is 72,000 - and at that point your monthly quota, not your rate limit, is what you are actually spending.

The limit that will actually stop you is concurrency

Here is the one that catches people out. You compute your safe request rate, stay comfortably under it, and still get 429. The reason is that requests per minute and concurrent jobs are two different limits, and the second one is much smaller than people expect - a handful of simultaneous conversions, depending on plan.

Concurrency counts conversions running at the same instant. Fire 30 requests in parallel and you can be well inside your per-minute budget while badly over your concurrency ceiling, because all 30 are in flight together. They are reported differently precisely so you can tell them apart:

  • 429 rate_limited - too many requests per minute, or monthly quota exhausted. Wait longer.
  • 429 concurrency_limited - too many conversions in flight at once. Run fewer in parallel. Waiting longer between batches will not fix this if you still launch them all together.

The practical shape is a worker pool sized to your plan's concurrency limit rather than an unbounded Promise.all over your whole file list. A pool of four workers, each sending batches of 20, is both faster and better behaved than 200 parallel requests - and it never trips either limit. There is more on classifying and retrying these in the retry guide.

One bad file should not cost you the other nineteen

In a run of forty thousand real files, some will be broken. Truncated downloads, a .png that is actually HTML, a password-protected PDF someone dropped in the wrong folder. If a batch endpoint is all-or-nothing, every one of those poisons an entire batch, and you end up writing per-file retry logic to isolate the offender.

Batch conversion here degrades instead of failing. Everything that converted comes back, and the ZIP gains a PRIVCONVERT-ERRORS.txt naming what was skipped and why:

PrivConvert batch report
========================

Converted: 18
Skipped:   2

These files were skipped and are not in this archive:

  - scan_0041.png: could not be converted. It may be corrupted or in an unsupported format.
  - notes.png: could not be converted. It may be corrupted or in an unsupported format.

The important detail for your code: partial success is still HTTP 200. If you branch on the status code alone you will record a clean success and silently lose two files. Branch on the headers instead:

X-Converted-Count: 18
X-Failed-Count: 2
X-Partial-Success: 1

X-Partial-Success is present only when something was skipped, so it is the cheapest thing to test. You get the counts without unzipping anything, which means a bulk job can reconcile as it goes rather than at the end.

The one case that is a hard error: if nothing in the batch converted, there is no partial result worth shipping, so the response is 400 with the reason rather than an empty archive. Treat it as a signal about the batch as a whole - usually the wrong tool for those files.

Group by target format first

A batch runs one tool over all its files, so the shape of the job is: partition your input by the conversion you need, then batch within each partition in groups of 20. Mixed folders are the normal case, and a job that walks a directory tree converting whatever it finds needs that grouping step anyway. Discover what is available programmatically rather than hard-coding tool ids - GET /api/tools returns the full catalogue with input and output formats, which is also how you validate that a conversion exists before you queue ten thousand of them.

Size limits: two of them, and the smaller one wins

A batch has to fit in 100 MB total, and your plan's per-request size cap applies to the same total. Whichever is lower is your real ceiling, which means the practical batch size is often decided by file size rather than by the 20-file cap. Twenty 8 MB scans fit; twenty 40 MB ones do not.

So size batches by bytes, not by count - accumulate files until you approach the limit, then send. And note that batch is the wrong tool for a single large file: one file above your plan's cap belongs on the chunked upload endpoint, which exists for exactly that. Batch is for many ordinary files.

A job structure that finishes

Putting it together, in the order that matters:

  1. Partition the file list by target conversion.
  2. Pack each partition into batches, closing a batch when it nears 20 files or the size ceiling, whichever comes first.
  3. Run the batches through a worker pool sized to your plan's concurrency limit - not an unbounded parallel map.
  4. Reconcile each response by header, recording partial successes as partial rather than as success.
  5. Retry only what should be retried: 429 and 5xx with backoff, and never a 4xx that describes a bad file, because it will fail identically forever.
  6. Requeue the files named in the error manifests once, then set them aside for inspection rather than looping on them.

Nothing is retained, which changes the failure model

One structural difference worth knowing when you plan a large run: files are converted in memory and never written to disk, so there is no server-side job store, no file ids and nothing to clean up afterwards. A failed batch leaves nothing behind to reconcile or delete, and a crashed worker on your side does not leave uploads sitting on ours.

The flip side is that the response is the only copy - there is no "download it again later" endpoint, because there is nothing to download from. Write each response to your own storage as it arrives rather than treating the API as a staging area. Adding conversion to an app without storing user files covers the retention side of that in more detail, and the API overview has current plan limits.

Developer resources:
Batch endpoint docs Full API reference Plans & limits