Skip to content
All regions operational
NorthmereData Commons
Home/Documentation/File formats

File formats

Every format on the commons is documented and readable with free software. This page lists what to expect in each domain and a few settings that save hours.

Climate and ocean

Zarr v3 is the default for reanalysis and ensembles. A variable is stored as a directory of compressed chunks, so reading one location through time touches a few hundred megabytes rather than a whole year. Chunks in NMR-5 are 1 × 721 × 1440 for surface fields and 24 × 8 × 721 × 1440 for model levels, compressed with Zstandard level 3.

import xarray as xr
ds = xr.open_zarr("clm-1120/surface/t2m.zarr", consolidated=True)
series = ds.t2m.sel(latitude=60.17, longitude=24.94, method="nearest").sel(time="2019")

NetCDF-4 is used where depositors supply it (sea ice, ocean sections). Files follow CF-1.10 conventions and are chunked on the time axis.

Genomics

CRAM 3.1 alignments reference the assembly shipped in the same collection; the reference is named in the checksum list so that tools can find it. FASTQ is gzip-compressed and block-aligned (BGZF) to allow parallel decompression. VCF.gz files have tabix indices; BGEN 1.3 is used for large imputed panels. Gene catalogues and metadata are Parquet.

samtools view -T assemblies/acc-0417.fa alignments/acc-0417.cram chr3B:100000000-100050000 | head

Imagery

Tiles are cloud-optimised GeoTIFF with 512 × 512 internal blocks, overviews at 2, 4, 8 and 16, and Deflate compression with predictor 2. A tile on the Northmere grid is 110 km square; at 10 m that is 11,000 × 11,000 pixels and about 480 MB for twelve bands. Footprints are published as GeoParquet.

Compression and storage

FormatCompressionTypical ratioReads with
Zarr v3Zstandard, level 32.1 : 1xarray, zarr-python, Zarr.jl
NetCDF-4Deflate, level 4, shuffle1.8 : 1netCDF4, xarray, CDO, NCO
CRAM 3.1reference-based, rANS1 : 4 vs. BAMsamtools, htslib, pysam
FASTQ.gzBGZF3.4 : 1any gzip-aware tool
ParquetZstandard, level 64.0 : 1DuckDB, Arrow, pandas, R arrow
COGDeflate + predictor 21.6 : 1GDAL, rasterio, QGIS

Checksums

All files are listed in a SHA256SUMS file at the directory level. Zarr stores are served as one tar per top-level array to keep file counts manageable; extract them with tar -xf in the target location. Consolidated metadata is included.