dnanexus-integration

DNAnexus Integration

Overview

DNAnexus is a cloud platform for biomedical data analysis and genomics. Build and deploy apps/applets, manage data objects, run workflows, and use the dxpy Python SDK for genomics pipeline development and execution.

When to Use This Skill

This skill should be used when:

Creating, building, or modifying DNAnexus apps/applets

Uploading, downloading, searching, or organizing files and records

Running analyses, monitoring jobs, creating workflows

Writing scripts using dxpy to interact with the platform

Setting up dxapp.json, managing dependencies, using Docker

Processing FASTQ, BAM, VCF, or other bioinformatics files

Managing projects, permissions, or platform resources

Core Capabilities

The skill is organized into five main areas, each with detailed reference documentation:

1. App Development

Purpose: Create executable programs (apps/applets) that run on the DNAnexus platform.

Key Operations:

Generate app skeleton with dx-app-wizard

Write Python or Bash apps with proper entry points

Handle input/output data objects

Deploy with dx build or dx build --app

Test apps on the platform

Common Use Cases:

Bioinformatics pipelines (alignment, variant calling)

Data processing workflows

Quality control and filtering

Format conversion tools

Reference: See references/app-development.md for:

Complete app structure and patterns

Python entry point decorators

Input/output handling with dxpy

Development best practices

Common issues and solutions

2. Data Operations

Purpose: Manage files, records, and other data objects on the platform.

Key Operations:

Upload/download files with dxpy.upload_local_file() and dxpy.download_dxfile()

Create and manage records with metadata

Search for data objects by name, properties, or type

Clone data between projects

Manage project folders and permissions

Common Use Cases:

Uploading sequencing data (FASTQ files)

Organizing analysis results

Searching for specific samples or experiments

Backing up data across projects

Managing reference genomes and annotations

Reference: See references/data-operations.md for:

Complete file and record operations

Data object lifecycle (open/closed states)

Search and discovery patterns

Project management

Batch operations

3. Job Execution

Purpose: Run analyses, monitor execution, and orchestrate workflows.

Key Operations:

Launch jobs with applet.run() or app.run()

Monitor job status and logs

Create subjobs for parallel processing

Build and run multi-step workflows

Chain jobs with output references

Common Use Cases:

Running genomics analyses on sequencing data

Parallel processing of multiple samples

Multi-step analysis pipelines

Monitoring long-running computations

Debugging failed jobs

Reference: See references/job-execution.md for:

Complete job lifecycle and states

Workflow creation and orchestration

Parallel execution patterns

Job monitoring and debugging

Resource management

4. Python SDK (dxpy)

Purpose: Programmatic access to DNAnexus platform through Python.

Key Operations:

Work with data object handlers (DXFile, DXRecord, DXApplet, etc.)

Use high-level functions for common tasks

Make direct API calls for advanced operations

Create links and references between objects

Search and discover platform resources

Common Use Cases:

Automation scripts for data management

Custom analysis pipelines

Batch processing workflows

Integration with external tools

Data migration and organization

Reference: See references/python-sdk.md for:

Complete dxpy class reference

High-level utility functions

API method documentation

Error handling patterns

Common code patterns

5. Configuration and Dependencies

Purpose: Configure app metadata and manage dependencies.

Key Operations:

Write dxapp.json with inputs, outputs, and run specs

Install system packages (execDepends)

Bundle custom tools and resources

Use assets for shared dependencies

Integrate Docker containers

Configure instance types and timeouts

Common Use Cases:

Defining app input/output specifications

Installing bioinformatics tools (samtools, bwa, etc.)

Managing Python package dependencies

Using Docker images for complex environments

Selecting computational resources

Reference: See references/configuration.md for:

Complete dxapp.json specification

Dependency management strategies

Docker integration patterns

Regional and resource configuration

Example configurations

Quick Start Examples

Upload and Analyze Data

import dxpy

# Upload input file
input_file = dxpy.upload_local_file("sample.fastq", project="project-xxxx")

# Run analysis
job = dxpy.DXApplet("applet-xxxx").run({
    "reads": dxpy.dxlink(input_file.get_id())
})

# Wait for completion
job.wait_on_done()

# Download results
output_id = job.describe()["output"]["aligned_reads"]["$dnanexus_link"]
dxpy.download_dxfile(output_id, "aligned.bam")

Search and Download Files

import dxpy

# Find BAM files from a specific experiment
files = dxpy.find_data_objects(
    classname="file",
    name="*.bam",
    properties={"experiment": "exp001"},
    project="project-xxxx"
)

# Download each file
for file_result in files:
    file_obj = dxpy.DXFile(file_result["id"])
    filename = file_obj.describe()["name"]
    dxpy.download_dxfile(file_result["id"], filename)

Create Simple App

# src/my-app.py
import dxpy
import subprocess

@dxpy.entry_point('main')
def main(input_file, quality_threshold=30):
    # Download input
    dxpy.download_dxfile(input_file["$dnanexus_link"], "input.fastq")

    # Process
    subprocess.check_call([
        "quality_filter",
        "--input", "input.fastq",
        "--output", "filtered.fastq",
        "--threshold", str(quality_threshold)
    ])

    # Upload output
    output_file = dxpy.upload_local_file("filtered.fastq")

    return {
        "filtered_reads": dxpy.dxlink(output_file)
    }

dxpy.run()

Workflow Decision Tree

When working with DNAnexus, follow this decision tree:

Need to create a new executable?

- Yes → Use App Development (references/app-development.md)
- No → Continue to step 2

Need to manage files or data?

- Yes → Use Data Operations (references/data-operations.md)
- No → Continue to step 3

Need to run an analysis or workflow?

- Yes → Use Job Execution (references/job-execution.md)
- No → Continue to step 4

Writing Python scripts for automation?

- Yes → Use Python SDK (references/python-sdk.md)
- No → Continue to step 5

Configuring app settings or dependencies?

- Yes → Use Configuration (references/configuration.md)

Often you'll need multiple capabilities together (e.g., app development + configuration, or data operations + job execution).

Installation and Authentication

Install dxpy

uv pip install dxpy

Login to DNAnexus

dx login

This authenticates your session and sets up access to projects and data.

Verify Installation

dx --version
dx whoami

Common Patterns

Pattern 1: Batch Processing

Process multiple files with the same analysis:

# Find all FASTQ files
files = dxpy.find_data_objects(
    classname="file",
    name="*.fastq",
    project="project-xxxx"
)

# Launch parallel jobs
jobs = []
for file_result in files:
    job = dxpy.DXApplet("applet-xxxx").run({
        "input": dxpy.dxlink(file_result["id"])
    })
    jobs.append(job)

# Wait for all completions
for job in jobs:
    job.wait_on_done()

Pattern 2: Multi-Step Pipeline

Chain multiple analyses together:

# Step 1: Quality control
qc_job = qc_applet.run({"reads": input_file})

# Step 2: Alignment (uses QC output)
align_job = align_applet.run({
    "reads": qc_job.get_output_ref("filtered_reads")
})

# Step 3: Variant calling (uses alignment output)
variant_job = variant_applet.run({
    "bam": align_job.get_output_ref("aligned_bam")
})

Pattern 3: Data Organization

Organize analysis results systematically:

# Create organized folder structure
dxpy.api.project_new_folder(
    "project-xxxx",
    {"folder": "/experiments/exp001/results", "parents": True}
)

# Upload with metadata
result_file = dxpy.upload_local_file(
    "results.txt",
    project="project-xxxx",
    folder="/experiments/exp001/results",
    properties={
        "experiment": "exp001",
        "sample": "sample1",
        "analysis_date": "2025-10-20"
    },
    tags=["validated", "published"]
)

Best Practices

Error Handling: Always wrap API calls in try-except blocks

Resource Management: Choose appropriate instance types for workloads

Data Organization: Use consistent folder structures and metadata

Cost Optimization: Archive old data, use appropriate storage classes

Documentation: Include clear descriptions in dxapp.json

Testing: Test apps with various input types before production use

Version Control: Use semantic versioning for apps

Security: Never hardcode credentials in source code

Logging: Include informative log messages for debugging

Cleanup: Remove temporary files and failed jobs

Resources

This skill includes detailed reference documentation:

references/

app-development.md - Complete guide to building and deploying apps/applets

data-operations.md - File management, records, search, and project operations

job-execution.md - Running jobs, workflows, monitoring, and parallel processing

python-sdk.md - Comprehensive dxpy library reference with all classes and functions

configuration.md - dxapp.json specification and dependency management

Load these references when you need detailed information about specific operations or when working on complex tasks.

Getting Help

Official documentation: https://documentation.dnanexus.com/

API reference: http://autodoc.dnanexus.com/

GitHub repository: https://github.com/dnanexus/dx-toolkit

Support: support@dnanexus.com