Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

Introduction

zarrs is a Rust library for the Zarr V2 and Zarr V3 array storage formats. If you don’t know what Zarr is, check out:

zarrs was originally designed exclusively as a Rust library for Zarr V3. However, it now supports a V3 compatible subset of Zarr V2, and has Python and C/C++ bindings. The community has since built zarrs-backed bindings for other languages, such as MATLAB, R, and Julia. This book details the Rust implementation.

🚀 zarrs is Fast 🚀

The zarr_benchmarks repository includes benchmarks of zarrs against other Zarr V3 implementations. Check out the benchmarks below that measure the time to round trip a \(1024x2048x2048\) uint16 array encoded in various ways. The zarr_benchmarks repository includes additional benchmarks.

Python Bindings: zarrs-python zarrs_python_ver zarrs_python_doc zarrs_python_repo

zarrs-python exposes a high-performance zarrs-backed codec pipeline to the reference zarr-python Python package. It is enabled as follows:

from zarr import config
import zarrs # noqa: F401

config.set({"codec_pipeline.path": "zarrs.ZarrsCodecPipeline"})

That’s it! There is no need to learn a new API and it is supported by downstream libraries like dask. However, zarrs-python has some limitations. Consult the zarrs-python README or PyPi docs for more details.

Rust Crates

The Zarr specification is inherently unstable. It is under active development and new extensions are regularly being introduced.

The zarrs crate has been split into multiple crates to:

  • allow external implementations of stores and extensions points to target a relatively stable API compatible with a range of zarrs versions,
  • enable automatic backporting of metadata compatibility fixes and changes due to standardisation,
  • stay up-to-date with unstable public dependencies (e.g. opendal, object_store, icechunk, etc) without impacting the release cycle of zarrs, and
  • improve compilation times.

Below is an overview of the crate structure. Crates highlighted in orange are community (third party) projects.

The core crate is:

  • zarrs zarrs_ver zarrs_doc zarrs_repo

For local filesystem stores (referred to as native Zarr), this is the only crate you need to depend on.

zarrs has quite a few supplementary crates:

  • zarrs_metadata zarrs_metadata_ver zarrs_metadata_doc zarrs_metadata_repo
  • zarrs_metadata_ext zarrs_metadata_ext_ver zarrs_metadata_ext_doc zarrs_metadata_ext_repo
  • zarrs_storage zarrs_storage_ver zarrs_storage_doc zarrs_storage_repo
  • zarrs_plugin zarrs_plugin_ver zarrs_plugin_doc zarrs_plugin_repo

Most Zarr extension points also have their own API crate:

  • zarrs_data_type zarrs_data_type_ver zarrs_data_type_doc zarrs_data_type_repo
  • zarrs_codec zarrs_codec_ver zarrs_codec_doc zarrs_codec_repo
  • zarrs_chunk_grid zarrs_chunk_grid_ver zarrs_chunk_grid_doc zarrs_chunk_grid_repo
  • zarrs_chunk_key_encoding zarrs_chunk_key_encoding_ver zarrs_chunk_key_encoding_doc zarrs_chunk_key_encoding_repo

Tip

The supplementary crates are transitive dependencies of zarrs, and are re-exported by it. You do not need to add them as direct dependencies. The extension point API crates are re-exported as zarrs::array::{data_type,codec,chunk_grid,chunk_key_encoding}::api, and their key types are re-exported directly in zarrs::array. The storage transformer API is the exception: it lives in zarrs itself as zarrs::array::storage_transformer.

Note

The supplementary crates are separated from zarrs to enable development of Zarr extensions and stores targeting a more stable API than zarrs itself.

Additional crates need to be added as dependencies in order to use:

  • remote stores (e.g. HTTP, S3, GCP, etc.),
  • zip stores, or
  • icechunk transactional storage.

The Stores chapter details the various types of stores and their associated crates.

C/C++ Bindings: zarrs_ffi zarrs_ffi_ver zarrs_ffi_doc zarrs_ffi_repo

A subset of zarrs exposed as a C/C++ API. zarrs_ffi is a single header library: zarrs.h. Consult the zarrs_ffi README and API docs for more information.

CLI Tools: zarrs_tools zarrs_tools_ver zarrs_tools_doc zarrs_tools_repo

Various tools for creating and manipulating Zarr v3 data with the zarrs rust crate. This crate is detailed in the zarrs_tools chapter.

Zarr Metadata Conventions

ome_zarr_metadata ome_zarr_metadata_ver ome_zarr_metadata_doc ome_zarr_metadata_repo

A Rust library for OME-Zarr (previously OME-NGFF) metadata.

OME-Zarr, formerly known as OME-NGFF (Open Microscopy Environment Next Generation File Format), is a specification designed to support modern scientific imaging needs. It is widely used in microscopy, bioimaging, and other scientific fields requiring high-dimensional data management, visualisation, and analysis.

Community (Third Party)

A growing ecosystem of projects builds on zarrs: bindings that expose it to other languages, and crates that extend it with additional stores, codecs, conventions, and format support.

Note

The projects in this section are developed and maintained by the community, independently of zarrs. Zarr feature coverage, maturity, and support vary considerably, so consult each project’s own documentation. Please open an issue to add a project to this list.

Language Bindings

  • zarrista (Python) zarrista_ver zarrista_doc zarrista_repo
  • zarr-matlab (MATLAB) zarr_matlab_repo
  • pizzarr (R) pizzarr_ver pizzarr_doc pizzarr_repo
  • Rzarrs (R) Rzarrs_repo
  • Zarrs.jl (Julia) Zarrs_jl_repo

zarrista (Python)

A low-level Zarr API for Python built directly on zarrs, in the style of zarrita. Unlike zarrs-python, which slots a zarrs-backed codec pipeline underneath zarr-python, zarrista exposes zarrs itself: it has its own API with synchronous and asynchronous variants, zero-copy numpy interoperability, and support for icechunk and object_store stores. The Python API is not yet stable.

zarr-matlab (MATLAB)

A Zarr V2 and V3 implementation for MATLAB based on zarrs, which is called through a Rust MEX layer. It supports reading and writing arrays and groups, chunking and sharding, a range of codecs, and filesystem and read-only HTTP stores.

pizzarr (R)

A Zarr implementation for R that ships in two builds of the same version. The CRAN build is pure R with no compiled dependencies, whereas the r-universe build links zarrs through extendr as a backend, adding parallel decompression, additional codec support, and cloud stores.

Rzarrs (R)

R bindings to zarrs through the savvy crate, distributed via r-universe. It is reader-first, supporting filesystem, HTTP/HTTPS, S3, and .zarr.zip stores.

Zarrs.jl (Julia)

Zarr V2 and V3 arrays for Julia backed by zarrs through a C FFI shim, including icechunk support.

Note

Zarrs.jl is experimental. Its README recommends Zarr.jl if a mature pure-Julia implementation is needed.

Extension Crates

These crates extend zarrs through its extension points and storage API.

  • zarrs_n5 zarrs_n5_ver zarrs_n5_doc zarrs_n5_repo
    • Read-only N5 support, including the gzip, bzip2, zstd, and blosc N5 compressors.
  • zarrs_conventions zarrs_conventions_ver zarrs_conventions_doc zarrs_conventions_repo
    • An implementation of the zarr-conventions specification, which standardises how conventions are declared in Zarr attributes.
    • Companion crates implement individual conventions:
      • zarrs_conventions_license zarrs_conventions_license_ver zarrs_conventions_license_doc implements the license convention, which communicates the licence(s) of the data in a group or array.
      • zarrs_conventions_thumbnails zarrs_conventions_thumbnails_ver zarrs_conventions_thumbnails_doc implements the thumbnails convention, which references images representing a node for use in galleries and previews.
      • zarrs_conventions_uom zarrs_conventions_uom_ver zarrs_conventions_uom_doc implements the units of measure convention, which communicates the units of a numeric array.
  • ndic-zarr ndic_zarr_ver ndic_zarr_doc ndic_zarr_repo
    • Zarr V3 codecs for multidimensional scientific images: nd_lift, htj2k, and nd_zfp.
  • zarrs_sqlite zarrs_sqlite_repo
  • zarrs_jpeg zarrs_jpeg_repo
    • An implementation of the proposed jpeg codec backed by libjpeg-turbo.

Warning

Some of these crates are works in progress that have not been published to crates.io.