Streaming large files

A 5-million-cell .h5ad with 60 markers is a few gigabytes on disk and rather more once AnnData has materialised X, raw.X, and every layer. Most spatial statistics do not need all of that in memory at once: tl.spatialDistance needs four obs columns, tl.phenotype needs the marker matrix a group at a time.

streamData=True runs those functions against the file on disk through CAP-AnnData, reading only the sections a function actually touches and writing back only the sections it changed.

import scimappro as sp

sp.tl.spatialDistance("big.h5ad", phenotype="phenotype", streamData=True)

The contract

• data must be a path. Streaming an in-memory AnnData makes no sense — it is already in memory. Pass the .h5ad path as a string.
• The file is modified in place and the function returns None. This is the one place where scimappro mutates your input; copy the file first if you want to keep the original.
• outputDir mostly does not apply. Streaming edits the input file, and most functions ignore outputDir in this mode. The exceptions are pp.log1p, pp.combat, tl.umap, and tl.spatialSimilarityLookup, which copy the (already-modified) file there as well. To keep an original, copy the file yourself before streaming into it.
• Results are identical to the in-memory path, up to floating-point noise in the stochastic methods (UMAP, LDA, NMF, Leiden), which is covered by the stream-vs-memory equality tests in tests/.

# in memory                      # streaming
adata = sp.tl.phenotype(         sp.tl.phenotype(
    adata,                           "big.h5ad",
    phenotype="workflow.csv",        phenotype="workflow.csv",
)                                    streamData=True,
                                 )
# -> returns AnnData             # -> returns None, big.h5ad updated on disk

Which functions stream

Module	Functions with streamData
pp	rescale, log1p, combat
tl	phenotype, cluster, umap, foldChange, spatialDistance, spatialCooccurrence, spatialProximityScore, spatialAggregate, spatialSimilarityLookup, neighCount, neighExp, neighLDA, neighNMF
pl	barplot (reads obs only)

Functions not in this table read little enough that streaming would not help, or need the whole matrix anyway.

A streaming pipeline

Each step edits the same file, so nothing round-trips through memory:

import scimappro as sp

path = "big.h5ad"

sp.pp.rescale(path, gate="manual_gates.csv", streamData=True)
sp.tl.phenotype(path, phenotype="phenotype_workflow.csv", streamData=True)
sp.tl.spatialDistance(path, phenotype="phenotype", streamData=True)
sp.tl.spatialCooccurrence(path, phenotype="phenotype", streamData=True)

Then read just what you need for plotting:

import anndata as ad

adata = ad.read_h5ad(path, backed="r")
sp.pl.spatialCooccurrence(adata)

What gets read and written

CAP-AnnData exposes the HDF5 sections individually, and scimappro asks for the narrowest slice each function can work with:

Function needs	It reads	It writes back
obs columns only (spatialDistance, spatialCooccurrence, spatialAggregate)	read_obs(columns=[…])	overwrite(["uns"]) or overwrite(["obs"])
marker names (phenotype, rescale)	read_var()	—
a matrix (log1p, combat, cluster)	X, raw.X, or layers[layer] sliced by the cells in play	overwrite(["X"]) / overwrite(["layers"])
a previous result (cluster(mode="spatial"))	read_uns([key])	overwrite(["uns"])

So a function that only labels cells never touches the expression matrix, and a function that only rewrites a layer never rewrites obs.

Memory and workers

maxWorkers bounds the parallelism over images and permutations. Streaming and maxWorkers interact: each worker holds its own slice, so lowering maxWorkers lowers peak memory.

sp.tl.spatialCooccurrence("big.h5ad", streamData=True, maxWorkers=4)

The default is max(1, cpu_count() - 1) for most functions and -1 (all cores) for tl.spatialCooccurrence's permutation loop.

Streaming and SpatialData

streamData=True applies to .h5ad paths only. Pass it alongside a SpatialData or a .zarr store and scimappro warns and continues in memory — a SpatialData's tables are already loaded as AnnData.

sp.tl.spatialDistance(sdata, streamData=True)
# UserWarning: streamData=True is not supported for SpatialData; its tables are
# already held in memory. Continuing without streaming.

Passing an in-memory AnnData with streamData=True warns similarly and runs in memory.

Caveats

Pandas 3 and Arrow-backed strings

Writing Pandas 3 objects back through AnnData's HDF5 registry is not reliable for Arrow string arrays. scimappro sanitises every DataFrame it writes with scimappro._io.h5ad_safe_dataframe. If you write to a streamed file yourself, do the same.

The file is open for writing

While a streaming call runs, the .h5ad is held open in r+ mode. Do not read it from another process at the same time, and do not interrupt a streaming call mid-write — the file may be left in a partially updated state.