anndata

AnnData — Annotated Data Matrix Processing for Single-Cell Genomics

Skills Overview

AnnData is the standard data structure in the Python ecosystem for processing annotated data matrices. Designed specifically for single-cell genomics, it supports a variety of bioinformatics data formats, including h5ad and zarr, and is a core component of the scverse ecosystem.

Use Cases

Single-cell RNA-seq analysis workflows

Read sequencing data from 10X Genomics or other platforms, perform quality control, normalization, and feature selection, and store analysis results. AnnData provides unified management of the gene expression matrix (X), cell metadata (obs), gene metadata (var), and multidimensional annotations (layers, obsm, and varm).

Memory optimization for large-scale datasets

When processing datasets that exceed available memory, use backed mode to load data on demand, or convert to sparse matrices (csr_matrix) to reduce memory usage. Zarr cloud storage is also supported, enabling distributed data access.

Integration of multiple batches

Concatenate multiple experimental batches or samples while automatically handling index alignment and metadata merging. Use the label parameter to track data sources, and support inner/outer join strategies to provide structured input for downstream batch correction.

Core Features

Data structure management

Create and manipulate AnnData objects containing core components such as X (the primary matrix), obs (observation metadata), var (variable metadata), layers (multiple matrices), obsm/varm (multidimensional annotations), obsp/varp (pairwise relationships), uns (unstructured annotations), and raw (a backup of the original data). Basic operations such as subsetting, transposition, copying, and renaming are supported.

Input/output and format conversion

Native support is provided for reading and writing h5ad and zarr formats, with compression options and backed mode. Through the anndata.io module, CSV, MTX, Loom, and other formats can be read, while Scanpy can be used to read 10X Genomics data. Format conversion and remote data access are also supported.

Concatenation and integration

Use ad.concat() to concatenate multiple AnnData objects along observations (axis=0) or variables (axis=1), with flexible configuration of join types (inner/outer) and merge strategies (same/unique/first/only). The experimental AnnCollection supports lazy concatenation and is suitable for ultra-large datasets. AnnData integrates seamlessly with downstream analysis tools such as Scanpy, scvi-tools, and Muon.

Frequently Asked Questions

What is the difference between AnnData and Scanpy?

AnnData is a data structure, while Scanpy is an analysis toolkit. AnnData is responsible for storing and manipulating annotated matrices, whereas Scanpy provides analysis workflows for quality control, normalization, dimensionality reduction, clustering, and more. In practice, they are typically used together: AnnData manages the data, and Scanpy performs the analysis. A simple analogy is that AnnData is the data container, while Scanpy is the analysis laboratory.

How do I read an h5ad file?

Use ad.read_h5ad('file.h5ad') to load the entire file into memory. For large files, add the backed='r' parameter to enable lazy-loading mode, so that data is read only when accessed. To write a file, use adata.write_h5ad('output.h5ad', compression='gzip'). Compression can reduce file size but slightly increase read time.

What should I do if I run out of memory when processing large-scale data?

First, use backed mode (ad.read_h5ad('large.h5ad', backed='r')) to avoid loading the entire dataset at once. Next, convert X to a sparse matrix (scipy.sparse.csr_matrix), as sparse data can significantly reduce memory usage. You can also consider converting the data to Zarr format and storing it in the cloud for on-demand, chunked access. If these options are still insufficient, it is recommended to first filter a subset of the data based on obs metadata, then load it into memory.