polars-bio

polars-bio - High-Performance Genomic Interval Operations and Bioinformatics File Handling

Skills Overview

polars-bio is a high-performance Python library for genomic interval operations and bioinformatics file handling based on Polars DataFrames. It supports common formats such as BED, VCF, BAM, CRAM, and GFF, and provides streaming processing and cloud storage access capabilities.

Use Cases

Large-Scale Genomic Interval Computations - When you need operations like overlap detection, nearest-neighbor search, merging, and coverage calculation on genomic intervals, polars-bio is 6–38× faster than traditional tools (e.g., bioframe). It is well-suited for human-genome-scale datasets.

Genomic Data Analysis with Very Tight Memory Limits - For oversized BAM/CRAM files beyond available RAM, or for massive collections of intervals, polars-bio’s streaming mode (scan_* functions) and the DataFusion engine can process data in batches without loading everything into memory at once.

Direct Analysis of Genomic Files in the Cloud - Supports direct read/write of bioinformatics files on S3, GCS, and Azure Blob Storage. Combined with predicate pushdown and columnar storage characteristics, it downloads only the needed data blocks, making it ideal for cloud-native analysis pipelines.

Core Features

Genomic Interval Operations Suite - Provides eight core interval operations: overlap (overlap detection), nearest (nearest-neighbor search), merge (interval merging), cluster (clustering), coverage (coverage statistics), complement (complement set), subtract (set difference), count_overlaps (overlap counting). It supports both a functional API and a chainable style.

Bioinformatics File Read/Write - A unified set of read_*, scan_*, and write_* interfaces supports 10+ formats including BED, VCF, BAM, CRAM, GFF/GTF, FASTA, FASTQ, and SAM. It automatically handles compression (GZIP, BGZF) and coordinate system conversion. Read operations return a LazyFrame to enable streaming.

SQL Data Processing Engine - By registering bioinformatics files as DataFusion SQL tables via register_* functions, you can query variant data and interval data using standard SQL. It supports complex queries such as joins across multiple tables, aggregations, and filtering, making it suitable for analysts familiar with SQL.

Common Questions

What bioinformatics file formats does polars-bio support?

It supports BED, VCF (including VCF Zarr), BAM, CRAM, GFF, GTF, FASTA, FASTQ, SAM, and Hi-C pairs files. For each format, it provides two functions: read_* (immediate loading) and scan_* (lazy scanning). Some formats also provide write_*/sink_* writing interfaces.

How do I process large genomic files that exceed memory size?

Use scan_* functions (e.g., scan_bam, scan_vcf) together with .collect(engine="streaming") for streaming processing. The DataFusion engine loads data in batches. For cloud storage files, you can pass S3/GCS/Azure URI paths directly and rely on predicate pushdown to read only the required regions.

What is the difference between polars-bio and bioframe?

polars-bio is built on Polars and Apache Arrow and is 6–38× faster than bioframe in real benchmarks. It supports streaming large datasets, direct cloud storage access, and SQL querying, whereas bioframe is based on pandas and is better suited for small in-memory computations. polars-bio uses a default 1-based coordinate system (genomics convention), while bioframe uses 0-based half-open intervals (BED convention), but both can be configured to switch.

Important Limitations: polars-bio requires Python 3.11–3.14, and genomic positions are stored using INT32 (upper limit about 2.1 billion, covering all known genomes). BAM files require a matching .bai index file, and CRAM files must provide a reference sequence path (reference_path parameter). The .pb accessor functionality differs between DataFrame and LazyFrame: interval operations are only available on LazyFrame, while on DataFrame only write methods are available.

Author

Category

Install