← Về thư mục
📄 / / profiles / r-and-d / raw_evidence / scientific-agent-skills / skills / imaging-data-commons / references / cli_guide.md

idc-index Command Line Interface Guide

The idc-index package provides command-line tools for downloading DICOM data from the NCI Imaging Data Commons without writing Python code.

Installation

Needs idc-index installed — run python scripts/check_version.py, which reports the installed version and prints the install command for the interpreter you are running.

After installation, the idc command is available in your terminal.

Available Commands

Command Purpose
idc download General-purpose download with auto-detection of input type
idc download-from-manifest Download from manifest file with validation and progress tracking
idc download-from-selection Filter-based download with multiple criteria

idc download

General-purpose download command that intelligently interprets input. It determines whether the input corresponds to a manifest file path or a list of identifiers (collection_id, PatientID, StudyInstanceUID, SeriesInstanceUID, crdc_series_uuid).

Usage

# Download entire collection
idc download rider_pilot --download-dir ./data

# Download specific series by UID
idc download "1.3.6.1.4.1.9328.50.1.69736" --download-dir ./data

# Download multiple items (comma-separated)
idc download "tcga_luad,tcga_lusc" --download-dir ./data

# Download from manifest file (auto-detected by file extension)
idc download manifest.txt --download-dir ./data

Options

Option Description
--download-dir Destination directory (default: current directory)
--dir-template Directory hierarchy template (default: %collection_id/%PatientID/%StudyInstanceUID/%Modality_%SeriesInstanceUID)
--log-level Verbosity: debug, info, warning, error, critical

Directory Template Variables

The same templates apply in Python, where the argument is dirTemplate= rather than the --dir-template flag. The default is %collection_id/%PatientID/%StudyInstanceUID/%Modality_%SeriesInstanceUID:

# Simplified hierarchy (omit StudyInstanceUID level)
client.download_from_selection(
    downloadDir="./data",
    collection_id="tcga_luad",
    dirTemplate="%collection_id/%PatientID/%Modality"
)
# Results in: ./data/tcga_luad/TCGA-05-4244/CT/

# dirTemplate="" disables the hierarchy, writing every file straight into downloadDir

Use these variables in --dir-template to organize downloads:

Examples:

# Flat structure (all files in one directory)
idc download rider_pilot --download-dir ./data --dir-template ""

# Simplified hierarchy
idc download rider_pilot --download-dir ./data --dir-template "%collection_id/%PatientID/%Modality"

idc download-from-manifest

Specialized for downloading from manifest files with built-in validation, progress tracking, and resume capability.

Usage

# Basic download from manifest
idc download-from-manifest --manifest-file cohort.txt --download-dir ./data

# With progress bar and validation
idc download-from-manifest --manifest-file cohort.txt --download-dir ./data --show-progress-bar

# Resume interrupted download with s5cmd sync
idc download-from-manifest --manifest-file cohort.txt --download-dir ./data --use-s5cmd-sync

Options

Option Description
--manifest-file Required. Path to manifest file containing S3 URLs
--download-dir Required. Destination directory
--validate-manifest Validate manifest before download (enabled by default)
--show-progress-bar Display download progress
--use-s5cmd-sync Enable resumable downloads - skips already-downloaded files
--quiet Suppress subprocess output
--dir-template Directory hierarchy template
--log-level Logging verbosity

Manifest File Format

Manifest files contain S3 URLs, one per line:

s3://idc-open-data/cb09464a-c5cc-4428-9339-d7fa87cfe837/*
s3://idc-open-data/88f3990d-bdef-49cd-9b2b-4787767240f2/*

How to get a manifest file:

  1. IDC Portal: Export cohort selection as manifest
  2. Python query: Generate from SQL results
from idc_index import IDCClient

client = IDCClient()
results = client.sql_query("""
    SELECT series_aws_url
    FROM index
    WHERE collection_id = 'rider_pilot' AND Modality = 'CT'
""")

with open('ct_manifest.txt', 'w') as f:
    for url in results['series_aws_url']:
        f.write(url + '\n')

idc download-from-selection

Download data using filter criteria. Filters are applied sequentially.

Usage

# Download by collection
idc download-from-selection --collection-id rider_pilot --download-dir ./data

# Download specific series
idc download-from-selection --series-instance-uid "1.3.6.1.4.1.9328.50.1.69736" --download-dir ./data

# Multiple filters
idc download-from-selection --collection-id nlst --patient-id "100004" --download-dir ./data

# Dry run - see what would be downloaded without actually downloading
idc download-from-selection --collection-id tcga_luad --dry-run --download-dir ./data

Options

Option Description
--download-dir Required. Destination directory
--collection-id Filter by collection identifier
--patient-id Filter by patient identifier
--study-instance-uid Filter by study UID
--series-instance-uid Filter by series UID
--crdc-series-uuid Filter by CRDC UUID
--dry-run Calculate cohort size without downloading
--show-progress-bar Display download progress
--use-s5cmd-sync Enable resumable downloads
--dir-template Directory hierarchy template

Dry Run for Size Estimation

Use --dry-run to estimate download size before committing:

idc download-from-selection --collection-id nlst --dry-run --download-dir ./data

This shows: - Number of series matching filters - Total download size - No files are downloaded

Common Workflows

1. Download Small Collection for Testing

# rider_pilot is ~1GB - good for testing
idc download rider_pilot --download-dir ./test_data

2. Large Dataset with Progress and Resume

# Use s5cmd sync for large downloads - can resume if interrupted
idc download-from-selection \
    --collection-id nlst \
    --download-dir ./nlst_data \
    --show-progress-bar \
    --use-s5cmd-sync

3. Estimate Size Before Download

# Check size first
idc download-from-selection --collection-id tcga_luad --dry-run --download-dir ./data

# Then download if size is acceptable
idc download-from-selection --collection-id tcga_luad --download-dir ./data

4. Download Specific Modality via Python + CLI

# First, query for series UIDs in Python
from idc_index import IDCClient

client = IDCClient()
results = client.sql_query("""
    SELECT SeriesInstanceUID
    FROM index
    WHERE collection_id = 'nlst'
      AND Modality = 'CT'
      AND BodyPartExamined = 'CHEST'
    LIMIT 50
""")

# Save to manifest
results['SeriesInstanceUID'].to_csv('my_series.csv', index=False, header=False)

# Then download via CLI
idc download my_series.csv --download-dir ./lung_ct

Built-in Safety Features

The CLI includes several safety features:

Troubleshooting

Download Interrupted

Use --use-s5cmd-sync to resume:

idc download-from-manifest --manifest-file cohort.txt --download-dir ./data --use-s5cmd-sync

Connection Timeout

For unstable networks, download in smaller batches using Python to generate multiple manifests, then download sequentially.

See Also