pxml

A parallel, StAX-style (pull) XML reader for Rust, built for one shape of document: a single root containing thousands of uniform, order-independent records — e.g. <trades><trade>…</trade>…</trades>.

Rust 1.88+ crates.io MIT rayon edition 2024

Two-phase, scan-then-parse

document
Vec / mmap — Phase A: sequential scan → records[]
byte ranges + Prelude — Phase B: rayon pool → workers
quick-xml per slice

Phase A — walks the buffer once with a memchr-driven state machine, finding depth-1 boundaries and capturing shared prolog context (encoding, namespaces, entities). Builds no tree — it is memory-bandwidth bound.

Phase B — hands each record's slice to a worker running a normal quick-xml reader over just that slice, seeded with the shared prelude so entity expansion is correct in isolation. Workers are fully independent.

API at a glance

The soundness assumption is that the root's direct children are independent and may be consumed in any order. Below a size threshold, both parallel entry points transparently fall back to a sequential pass.

ParallelXmlOwns the buffer (Vec / mmap) + Config; the entry point

Configparallel_threshold · min_records · record_path

ChunkIndex / PreludePhase A output: per-record byte ranges + shared context

StreamReaderBounded-memory streaming pipeline over a Read / zstd source

RecordOne top-level record; events() returns a pull cursor, index() its position

EventStart { name, attrs } · End { name } · Text(Cow<str>) · Cdata(&[u8])

XmlErrorMalformed(pos) · Encoding · Io · UnsupportedDtd · RecordError { index }

Install

Requires Rust 1.88+ (edition 2024). The default zstd feature transparently decompresses zstd input — build with default-features = false for a pure-Rust crate.

> cargo add pxml

> cargo run --release --example bench

Expect sub-linear scaling — roughly 3–6× wall-clock on large files with substantial per-record work. On a 2M-record file, the streaming path measured ~2.2× faster than resident.

Consumption modes

par_for_each — unordered

Workers fire as records complete. try_par_for_each takes a fallible closure and short-circuits on the first error.

map_collect — document order

Runs in parallel but slots typed results back into document order. Because that means visiting every record, it does not short-circuit on the parallel path.

StreamReader — bounded memory

A producer thread frames records incrementally while a rayon pool parses them, with a bounded channel for backpressure. Resident memory is independent of document size — and often faster on huge files.

index() & sequential()

index() runs Phase A only for record count and byte ranges; sequential() is a classic whole-document StAX cursor for records that are not order-independent.

← all projects github.com/glslang/pxml ↗