Climate and ocean
Zarr v3 is the default for reanalysis and ensembles. A variable is stored as a directory of compressed chunks, so reading one location through time touches a few hundred megabytes rather than a whole year. Chunks in NMR-5 are 1 × 721 × 1440 for surface fields and 24 × 8 × 721 × 1440 for model levels, compressed with Zstandard level 3.
import xarray as xr
ds = xr.open_zarr("clm-1120/surface/t2m.zarr", consolidated=True)
series = ds.t2m.sel(latitude=60.17, longitude=24.94, method="nearest").sel(time="2019")
NetCDF-4 is used where depositors supply it (sea ice, ocean sections). Files follow CF-1.10 conventions and are chunked on the time axis.
Genomics
CRAM 3.1 alignments reference the assembly shipped in the same collection; the reference is named in the checksum list so that tools can find it. FASTQ is gzip-compressed and block-aligned (BGZF) to allow parallel decompression. VCF.gz files have tabix indices; BGEN 1.3 is used for large imputed panels. Gene catalogues and metadata are Parquet.
samtools view -T assemblies/acc-0417.fa alignments/acc-0417.cram chr3B:100000000-100050000 | head
Imagery
Tiles are cloud-optimised GeoTIFF with 512 × 512 internal blocks, overviews at 2, 4, 8 and 16, and Deflate compression with predictor 2. A tile on the Northmere grid is 110 km square; at 10 m that is 11,000 × 11,000 pixels and about 480 MB for twelve bands. Footprints are published as GeoParquet.
Compression and storage
| Format | Compression | Typical ratio | Reads with |
|---|---|---|---|
| Zarr v3 | Zstandard, level 3 | 2.1 : 1 | xarray, zarr-python, Zarr.jl |
| NetCDF-4 | Deflate, level 4, shuffle | 1.8 : 1 | netCDF4, xarray, CDO, NCO |
| CRAM 3.1 | reference-based, rANS | 1 : 4 vs. BAM | samtools, htslib, pysam |
| FASTQ.gz | BGZF | 3.4 : 1 | any gzip-aware tool |
| Parquet | Zstandard, level 6 | 4.0 : 1 | DuckDB, Arrow, pandas, R arrow |
| COG | Deflate + predictor 2 | 1.6 : 1 | GDAL, rasterio, QGIS |
Checksums
All files are listed in a SHA256SUMS file at the directory level. Zarr stores are served as one tar per top-level array to keep file counts manageable; extract them with tar -xf in the target location. Consolidated metadata is included.