File-format routing¶

omniload reads each file format through the best available path rather than one generic reader. This page explains how that routing works and why, so the per-format pages (MessagePack, CBOR, BSON, …) can stay focused on how to use each format. It is background reading: you do not need any of it to load a file, but it explains the behaviour you will see around extended types, corrupt files, and which optional package a format needs.

Which path reads which format¶

Format	Read path	Why
CSV, JSONL, Parquet	Native Polars / pyarrow readers	Fast, columnar, already well tested.
BSON	Dedicated in-tree codec	Needs extended-type normalization the generic decoders do not do.
MessagePack	Streaming `iterabledata` bridge	No better native path; streamed record-by-record.
CBOR	Whole-file decode with `cbor2` directly	Whole-file format; a direct decode surfaces corruption instead of hiding it.

The routing policy is deliberately conservative: fill gaps, don’t replace working paths. A format is only routed to the generic bridge when there is no faster, better-tested native reader for it. New long-tail formats are added incrementally behind the optional iterable extra, without touching the CSV / JSONL / Parquet paths.

Two read mechanisms¶

Formats that go through the long-tail machinery are read one of two ways, chosen per format.

Streaming through the iterabledata bridge¶

MessagePack (and future streaming formats) are read through iterabledata, which exposes a uniform per-format class that yields record dicts from a file object. omniload feeds it the source’s own already-authenticated fsspec handle, so a file on S3, GCS, Azure or SFTP keeps flowing through the source’s existing credentials. iterabledata’s own cloud-storage / credential layer is never touched, so there is no second authentication surface to configure.

Records are pulled in batches until the reader signals end-of-file, and flushed per file so a multi-file glob never drops a partial final chunk. iterabledata rewinds the handle on construction, which fails on a non-seekable stream (a pipe, some compressed or SFTP handles); such a stream is spooled into memory first, while a seekable handle streams straight through.

Whole-file decode¶

Some formats are whole-file rather than streaming, and iterabledata’s wrapper for them wraps its decode in a broad except that yields nothing on a bad payload, meaning a truncated or corrupt file would load as zero rows with no error (silent data loss). For those, omniload decodes the bytes with the format’s own library directly (CBOR via cbor2) so decode errors propagate instead of vanishing. This is why CBOR needs only the cbor2 package and not iterabledata itself, even though both ship together in the iterable extra.

Extended-type normalization¶

Binary formats carry types JSON does not (raw bytes, timestamps, tagged values), and the decoders hand some of those back as native Python objects that a text or Parquet loader cannot serialize. omniload normalizes rows to portable values before handing data to the loader:

bytes becomes a base64-encoded string (portable across text loaders and Parquet alike).
A MessagePack Timestamp extension becomes a UTC datetime.
An unknown CBOR tag becomes a plain {"tag": ..., "value": ...} object rather than crashing the load.

Some values are made portable by the decoder itself rather than by omniload: cbor2 decodes the standard CBOR tags (datetime, big integers, decimals) into native Python types directly. Those load into Parquet and SQL destinations, but a native decimal cannot be serialized to a JSONL file destination, so use a Parquet or SQL destination for data that carries decimals. Nested maps and arrays are handled recursively. The exact per-format mapping is on each format’s own page under “Extended-type handling”.

Integrity and truncation¶

The read mechanism determines how a damaged file behaves, and it is worth knowing which guarantee you get:

Whole-file decode (CBOR) raises on a corrupt or truncated file rather than loading partial data. A CBOR source must also be a single top-level value; files that concatenate several top-level objects are read only up to the first, a decoder limitation that cannot be detected at read time.
Streaming formats (MessagePack) carry no length prefix, so a truncated tail reads as a clean end-of-file: the partial trailing record, and anything after a mid-stream corruption, are dropped silently. Validate file integrity upstream if partial loads would be a problem.

The `iterable` extra¶

Long-tail format support ships in the optional iterable extra (also part of full):

pip install 'omniload[iterable]'

It installs iterabledata plus the shipped tranche’s decoders (msgpack, cbor2). If a file in one of these formats is loaded without the extra installed, omniload fails with a clear error naming the exact pip install to run, rather than a bare ImportError.