One-sentence Summary

Technical University of Munich researchers introduce AnyBlox, a framework that bundles lightweight WebAssembly decoders directly with datasets to decouple decoding from both systems and format specifications, thereby enabling transparent format evolution, instance-optimized encodings, and mainstream adoption of novel storage formats while integrating seamlessly with systems like DuckDB, Spark, and Umbra.

Key Contributions

• AnyBlox is introduced as a self-decoding data framework that bundles lightweight WebAssembly decoders with datasets and exposes a portable scan interface, allowing systems such as DuckDB, Spark, and Umbra to read arbitrary encodings without implementing each new storage format.
• The design decouples decoders from host systems and file-format specifications, supports bundling decoders with data, secure remote references through cryptographic hashes, and transparent use of native decoders for popular encodings, enabling format evolution and instance-optimized encodings.
• The presented evaluation discusses integration with DuckDB, Spark, and Umbra and contrasts WebAssembly decoders with uBPF-based execution, showing that WebAssembly decoders support secure reading of bundled or referenced decoders while uBPF faces verifier bugs, pervasive bounds checks, limited loop support, and complex compilation requirements.

Introduction

The paper addresses the growing mismatch between modern data-processing systems and storage formats, as data increasingly moves through open data lakes, cross-platform analytics tools, and field-specific formats. This matters because new encodings often fail to be adopted when every system must separately implement support for every format, creating an unsustainable N x M maintenance problem and causing existing formats to ossify.

Prior approaches have significant limitations: native decoders are fast but hard to reuse and unsafe when integrated into system cores, extensions improve flexibility but remain system-specific and risky, isolated UDF containers offer security but suffer from high data-transfer overhead, and static verification approaches such as eBPF or uBPF provide stronger isolation but face portability and performance constraints.

The authors introduce AnyBlox, a framework for self-decoding datasets that packages lightweight WebAssembly decoders directly with the data or references them securely by cryptographic hash. Their bring-your-own-decoder architecture allows any consumer system to read arbitrary encodings without built-in format support, while zero-copy memory transfer helps preserve performance. The work aims to break the tight coupling between storage formats and systems, enabling future-proof, modular data ingestion.

Dataset

• The authors make the dataset and related artifacts available in the PVLDB artifact repository at https://github.com/AnyBlox/vldb-2025.
• The provided text only describes artifact availability and does not specify the dataset composition, sources, subset sizes, filtering rules, schema, statistics, training splits, mixture ratios, cropping strategy, metadata construction, or how the data is used in model training.

Method

The authors describe AnyBlox as a self-decoding data architecture built around the idea that the decoder itself should travel with the data. This addresses a central tension in future-proof data access: a system must hide format internals from query engines, while still allowing those engines to access files directly. Instead of placing the format decoder inside every host system, AnyBlox embeds or references a WebAssembly decoder alongside the encoded payload. The host program only needs to understand the AnyBlox runtime and the Arrow output format, not every possible data encoding.

The method is best understood as a set of small contracts between three components: the host, the AnyBlox runtime library, and the sandboxed decoder. As shown in the figure below:

The figure shows a bundle containing encoded data and a decoder module, an init step that feeds those resources into libanyblox, and a WebAssembly runtime executing the decoder. The host sends a request containing a start position and tuple count, and the runtime returns Apache Arrow Record Batches. In the AnyBlox design, the bundle may store the decoder and data together in a single .any file, or a data lake may store them separately and connect them through metadata. The host still uses the same read path because libanyblox resolves the bundle and exposes a uniform scanning interface.

The decoder layer is based on WebAssembly. The authors choose WebAssembly because it offers a portable binary format, broad language toolchains, sandboxed execution, and near-native performance through JIT compilation. The implementation uses the Rust-based runtime stack, with Wasmtime providing the sandbox and Cranelift lowering bytecode to native instructions. This gives strong isolation properties and allows the decoder to run without granting arbitrary code access to the host process. The majority of the AnyBlox codebase is safe Rust, while unsafe operations are concentrated around WebAssembly memory mapping and can be reviewed as a focused module.

The host-facing pipeline starts when a host operator creates a decoder job. The job receives the dataset file descriptor, metadata, and the decoder bytecode. libanyblox JIT-compiles the decoder if needed and caches the compiled module, so repeated scans using the same decoder do not repeatedly pay compilation overhead. After initialization, the host requests tuple ranges, usually in batches, and the runtime maps those requests into calls to the decoder’s single exposed function, decode_batch. The decoder returns a pointer to an Arrow batch, and libanyblox translates WebAssembly pointers into host-native addresses before passing the result back to the host.

The decoder contract is intentionally small. The decode_batch function receives pointers and counts that describe the encoded data, the tuple range to decode, a state page for optional caching, and a projection mask for column filtering. The state page is zero-initialized at the start of each job. This makes the decoder behavior deterministic and idempotent: it may use the state page to cache decoded positions or reuse allocations within a job, but it cannot rely on hidden external state, clocks, randomness, or system calls. This is important for query engines that may retry, split, or re-plan scans.

The runtime layer is where much of the AnyBlox design complexity resides. WebAssembly exposes a linear memory abstraction beginning at address zero, and the guest decoder can allocate additional pages through a host-provided memory.grow function. AnyBlox builds on Wasmtime’s software fault isolation model: it reserves a large protected virtual memory region, up to the maximum addressable range for 32-bit pointers, and then makes only the required number of pages readable and writable as the decoder grows. This removes the need for many runtime bounds checks while keeping the decoder confined to its virtual allocation.

The main runtime challenge is giving the decoder access to a large encoded dataset without copying it into WebAssembly memory. A direct copy would create expensive initialization costs, especially for large files. A lazy paging layer would require custom page-fault handling and would force the decoder to work on page-sized chunks rather than a transparent linear memory view. Instead, AnyBlox uses file descriptors and direct memory mapping. If the dataset can be represented as a file descriptor, shared memory object, or similar OS abstraction, libanyblox maps it into the decoder’s virtual linear memory. The decoder then accesses the encoded data transparently, using ordinary memory reads. As shown in the figure below:

The runtime diagram details the virtual memory layout used for each decoder instance. The low end of linear memory contains the initial code, stack, and static data region. Immediately after that, AnyBlox maps the encoded dataset into the decoder’s address space. A state page follows, then a working allocation area, and finally a guard region that is protected from reads and writes. The decoder is passed a pointer to the start of linear memory, a pointer to the encoded data, the data length, the state page pointer, the current end of accessible memory, and a reference to the memory.grow function. This layout allows the host and decoder to share the same data buffer with minimal indirection.

The runtime also introduces thread-local memory pooling to avoid repeatedly creating large virtual memory regions. A decoder instance is tied to a thread-local job because concurrent scanning is common, while two jobs sharing the same thread and dataset at the same time are unlikely. When a job finishes, libanyblox may reuse the existing linear memory map for another job if the initial region is large enough, the same bundle is being scanned, and all dynamically allocated pages are cleared. If no suitable map exists and the cache is below its limit, a new virtual allocation is created. If the cache is full, the least recently used map can be reused by unhooking the old dataset and reinitializing the region.

The actual decoding process can be understood through a simple columnar example. Suppose a host is scanning a tuple range such as $[400000, 500000)$[400000,500000) in a run-end encoded column. The first call asks for a subset of that range, for example starting at tuple $400000$400000 with tuple count $10000$10000. The decoder locates the run containing that row, often by binary searching the run index, then emits values until it reaches the requested range. It knows the output size ahead of time because it must produce exactly the requested number of values. If the decoder needs new output buffers, it calls memory.grow, the host expands the linear memory mapping, and the decoder writes into the newly allocated region. The returned Arrow batch contains pointers into the WebAssembly sandbox, which libanyblox converts to host-visible addresses.

Subsequent batch requests can be much faster if the decoder stores iteration state in the state page. For a run-end encoded column, the decoder can keep the current row ID, current run index, current run value, remaining run length, and a pointer to an allocated output buffer. When the next call starts exactly where the previous one ended, the decoder resumes the scan without repeating binary search or reallocating memory. This state usage is optional, and because the state is reset per job, the system still preserves correctness if the host changes the request or restarts a job.

The output format is another major architectural boundary. AnyBlox uses Apache Arrow because it is columnar, extensible, widely supported, and well suited for analytical database engines. The decoder produces Arrow Record Batches, which gives the host a standardized view of decoded data. Different hosts consume that output differently. A native Arrow system such as DataFusion can share data with minimal conversion. Systems with internal storage layouts can convert Arrow arrays into native representations. Primitive types often convert in place, while string representation can require more logic because different engines use different string layouts. This conversion happens after decoding and does not change the decoder itself.

Host integration is designed around operator or table-source boundaries. In C++ systems, libanyblox can be exposed through C-compatible bindings. In Rust-native systems, the integration is direct. In JVM-based systems, a native interface layer bridges the host to libanyblox and loads Arrow payloads into managed objects. This language interoperability layer is part of the system design because AnyBlox itself is implemented in Rust while the target hosts may be C++, Rust, Java, or Scala.

Concurrent scanning is also modularized. The AnyBlox component can know the total row count and apply projection masks, allowing hosts to prune columns before or during decoding. Hosts using static partitioning can assign fixed tuple ranges to workers during planning. Hosts using dynamic work stealing can request work units from a central executor and process them in thread-local decoder jobs. A thread may request a larger batch and then process it in smaller engine-sized chunks, reducing operator overhead while preserving the benefits of local decoder state.

AnyBlox can also be embedded inside conventional file formats. In the Parquet example, the authors describe a scheme where a column begins with a decoder page followed by opaque binary data pages. The Parquet reader acts as a host and passes those pages to AnyBlox, allowing arbitrary encoding schemes to be distributed within an existing file layout. This is useful because traditional file formats already provide metadata, statistics, and query pushdown support. When AnyBlox is embedded at the row-group or column level, engines can still use min/max statistics, row filtering hints, and optimizer estimates from the outer format. Standalone AnyBlox, by contrast, stores only minimal metadata such as schema and row count, leaving richer statistics to higher-level query engines or acceleration layers.

Experiment

Evaluations on a warmed-up 32-core machine integrate AnyBlox into multiple query engines and data formats to assess extensibility, portability, and decoder performance. The system-level experiments show that new decoders can be added with minimal host changes, and that sandboxed WebAssembly decoders can outperform native Parquet or approach native performance in TPC-H and ClickBench workloads. Microbenchmarks isolate WebAssembly overheads, finding that some decoders are affected by vectorization and arithmetic limitations, while scaling and initialization costs remain manageable in practical scans.

AnyBlox is rated strongest on portability and extensibility while also receiving positive ratings for performance and security. Other approaches show clearer tradeoffs, with native and extension-based designs favoring performance but weaker portability or security, and isolated designs improving security at the cost of performance. AnyBlox receives the strongest portability and extensibility ratings AnyBlox is rated positively on both performance and security Approaches with strong performance ratings tend to score weakly on portability or security

Integrating AnyBlox into compiled, vectorized, and hybrid database hosts required moderate engineering effort. Each system was integrated in under two weeks and with fewer than two thousand lines of code, indicating a straightforward port across different execution models. Spark required the most person-days among the hosts because it needed a JNI bridge. DataFusion and DuckDB had the smallest integration effort, while Umbra required more lines of code.

AnyBlox is evaluated through qualitative comparisons against existing approaches and through porting experiments across diverse database hosts. The comparisons validate its balanced strengths, showing highest portability and extensibility while remaining positively rated for performance and security, whereas other designs exhibit clearer tradeoffs. The integration experiments validate that AnyBlox can be ported across compiled, vectorized, and hybrid execution models with relatively modest engineering effort, though effort varies by host complexity and required bridges.