Streaming large documents

Most of the time pdg builds the whole PDF in memory and writes it out in one go. That is simple and fast, and for an invoice, a report, or a book it is exactly what you want. But it has a ceiling: the entire document — every page, every font, every image, and the assembled PDF itself — has to fit in RAM at once.

For an ordinary document that ceiling is irrelevant. For a very large one it is the wall. Picture a 100,000-page statement run, or a catalogue whose output PDF is 10 GB. You cannot hold 10 GB of PDF in memory just to write it to a file — and you shouldn't have to.

Streaming mode removes that ceiling. Instead of assembling the whole document and then serializing it, pdg writes the PDF to disk as it goes, one object at a time, and never holds more than a small working set in memory. A document whose output is larger than your machine's RAM renders fine.

pdg build --stream big.xml big.pdf

That is the entire user-facing surface: add --stream. The input is the same XML, the output is the same kind of PDF, and the two modes produce structurally-equivalent results — the rendered pages are pixel-for-pixel identical. What changes is how the bytes get to disk, and how much memory that takes along the way.

Streaming is a CLI feature. In the browser playground there is nothing to stream — the page hands its bytes straight to the viewer — so this applies to pdg build on the command line.

Why this is unusual

Almost every PDF library, pdg's default mode included, follows the same recipe: build an in-memory model of the whole document, then serialize that model to bytes. It is the obvious design, and it means peak memory grows with the size of the output. A 10 GB PDF needs more than 10 GB of RAM to produce — often closer to double, because the assembled object graph and the output buffer both exist at once.

The PDF format itself, though, does not require this. A PDF file is a header, a flat run of numbered objects in (almost) any order, and a small cross-reference table at the end recording where each object landed. Nothing forces the objects to be built all-at-once or held all-at-once. You can write each object the moment it is ready, remember its byte offset, and emit the cross-reference table once the last object is out. No rewinding, no second pass over the bytes.

Streaming mode does exactly that. It is a genuinely different way to emit a PDF, and it is why pdg can write a document larger than the memory it runs in — a property very few PDF tools offer.

What stays in memory (and what doesn't)

Streaming removes three classes of held memory, in order of how much they cost.

1. The PDF itself. This is the big one. In the default mode the whole document is assembled as an in-memory object map and then copied into an output buffer — so near the end, two copies of a multi-gigabyte PDF coexist. Streaming writes each object to the file as soon as it is produced and immediately forgets it. The assembled document never exists; the output buffer never exists. Peak memory stops tracking the size of the output at all.

2. The page frames. Laying a page out produces a frame — the positioned text, rules, boxes, and image references that draw it. The default mode lays out every page up front and keeps all the frames. Streaming lays out and emits one page at a time: a page is rendered to the file and dropped before the next is laid out, so the per-page frames never pile up.

3. The image bytes. See Images, below — this is what lets a document that is large because of images stream too.

What streaming still holds is the flowed content — a lightweight, pre-template form of every page. It has to: a page template that prints "Page 3 of 4287" cannot know the 4287 until every page has been flowed and the total counted (see page templates and the {pages} placeholder). So streaming flows the whole document first to fix the count, then renders page-by-page. That held content is bounded by how much content the document has, not by how large its output is — which is the distinction that matters. The 10 GB of rendered PDF is never in memory; the comparatively small description of what to draw is.

Images: load once, or stream them

Images are the other thing that can blow past a memory budget — a document might be large precisely because it draws thousands of photographs. So image handling also has two modes, and pdg picks the right one for you based on how you're rendering.

Default mode keeps images in memory. Each <image> is decoded once when the document is parsed and the pixels are kept around, ready to embed. Drawing the same image on a hundred pages costs one decode and one copy in memory. For normal documents this is ideal.

Streaming mode keeps images out of memory. An <image> reads only its pixel dimensions when the document is parsed — enough to size and place the box — and leaves the actual bytes on disk. At the moment each distinct image is first drawn, pdg loads it, writes it into the PDF, and drops it. Only one image's bytes are ever live at a time, so a document with ten thousand distinct photographs streams without ever holding more than one of them.

In both modes, two <image> tags that name the same source share a single copy in the output — the image is embedded once and referenced wherever it is drawn. JPEGs are passed through with their original compression; other formats are re-encoded. A source that can't be loaded falls back to a 1×1 transparent placeholder so the rest of the page still renders, exactly as in the default mode.

You don't choose between these — pdg build loads eagerly, pdg build --stream defers. The only visible consequence is memory.

What is identical, and what differs

The output is equivalent. Same page count, same text, same fonts (streaming still subsets each embedded face down to the glyphs you actually use, with the same subset tags), same images, same links and bookmarks. Rendered side by side, the pages are pixel-identical. A reader cannot tell which mode produced a file.

The internal differences are deliberate and harmless:

• Object order. Streaming emits objects in the order it produces them, so the raw byte layout of the file differs from the default mode's. The cross-reference table accounts for it; every viewer reads either file the same way.
• A little more work. Streaming makes two passes over the pages (one to learn which glyphs each font needs before subsetting, one to render) and re-reads each image from its source at render time. That extra work is the price of bounded memory. For a document small enough to fit in RAM comfortably, it isn't worth paying — which is why streaming is opt-in rather than the default.

Strict mode and partial files

If the document declares strict diagnostics and an error would withhold output, streaming honors it before opening the output file. The document is flowed and checked first; if it is blocked, pdg reports the errors and writes nothing — you won't be left with a half-written or empty .pdf. The output file is created only once the document is known to be renderable.

Which mode should I use?

Default (pdg build) for essentially everything. It is simpler, slightly faster, and the memory ceiling it carries is one you'll never hit for an ordinary document.

Streaming (pdg build --stream) when the output is, or might be, larger than the memory you can give it — long statement or invoice runs, image-heavy catalogues, anything in the tens-of-thousands of pages. The rule of thumb: if you'd be nervous about the file size fitting in RAM, stream it.

# Everyday document — build it in memory:
pdg build invoice.xml invoice.pdf

# Hundred-thousand-page run — stream it to disk:
pdg build --stream statements.xml statements.pdf

Both commands take the same XML and produce the same kind of PDF. Streaming just promises to do it without ever holding the whole thing.