← Về thư mục
📄 / / profiles / r-and-d / raw_evidence / OpenAI4S / skills / bioskills / bio-alignment-files-duplicate-handling / references / usage-guide.md

Duplicate Handling - Usage Guide

Overview

Mark and remove PCR and optical duplicates from alignment files to prevent bias in downstream variant calling and peak detection.

Prerequisites

# samtools
conda install -c bioconda samtools

# pysam
pip install pysam

Quick Start

Tell your AI agent what you want to do: - "Mark duplicates in my BAM file" - "Remove PCR duplicates from my alignment" - "Calculate the duplicate rate for my sample" - "Run the full duplicate marking workflow"

Example Prompts

Marking Duplicates

"Mark duplicates in sample.bam and keep them flagged"

"Remove duplicates completely from my BAM file"

"Mark duplicates and generate statistics"

Workflow Operations

"Run the complete fixmate and markdup pipeline"

"Prepare my BAM for duplicate marking with fixmate"

"Mark duplicates with optical duplicate detection for NovaSeq data"

Quality Assessment

"Calculate the duplicate rate for my sample"

"Check if my duplicate rate is acceptable for WGS"

"Compare duplicate rates across my samples"

What the Agent Will Do

  1. Sort the BAM file by read name if not already sorted
  2. Add mate information using fixmate with the -m flag
  3. Re-sort by coordinate position
  4. Mark (or remove) duplicates using markdup
  5. Generate duplicate statistics
  6. Index the final BAM file
  7. Report the duplicate rate and assess quality

The Standard Workflow

Input BAM (any order)
    |
    v
samtools sort -n (name sort)
    |
    v
samtools fixmate -m (add mate info)
    |
    v
samtools sort (coordinate sort)
    |
    v
samtools markdup (mark/remove duplicates)
    |
    v
samtools index (create index)
    |
    v
Final BAM (coordinate sorted, duplicates marked)

Common Commands

Step-by-Step

samtools sort -n -@ 4 -o namesort.bam input.bam
samtools fixmate -m -@ 4 namesort.bam fixmate.bam
samtools sort -@ 4 -o coordsort.bam fixmate.bam
samtools markdup -@ 4 coordsort.bam marked.bam
samtools index marked.bam

Pipeline (No Intermediate Files)

samtools sort -n -@ 4 input.bam | \
    samtools fixmate -m -@ 4 - - | \
    samtools sort -@ 4 - | \
    samtools markdup -@ 4 - marked.bam
samtools index marked.bam

Mark vs Remove

samtools markdup input.bam marked.bam           # Mark only (FLAG 0x400)
samtools markdup -r input.bam deduped.bam       # Remove duplicates

With Statistics

samtools markdup -s input.bam marked.bam 2> stats.txt
samtools markdup -f stats.txt input.bam marked.bam

Optical Duplicates (Patterned Flowcells)

samtools markdup -d 100 input.bam marked.bam   # HiSeq (unpatterned)
samtools markdup -d 2500 input.bam marked.bam  # NovaSeq, NextSeq
# HiSeq: -d 100; NovaSeq/NextSeq patterned: -d 2500. See SKILL.md for full optical-distance matrix.

Check Duplicate Rate

samtools flagstat marked.bam | grep duplicates

# Calculate percentage
total=$(samtools view -c -F 256 marked.bam)
dups=$(samtools view -c -f 1024 -F 256 marked.bam)
echo "Duplicate rate: $(echo "scale=2; $dups * 100 / $total" | bc)%"

Python with pysam

Full Workflow

import pysam
import os

def mark_duplicates(input_bam, output_bam, threads=4):
    temp_ns = 'temp_namesort.bam'
    temp_fm = 'temp_fixmate.bam'
    temp_cs = 'temp_coordsort.bam'

    try:
        pysam.sort('-n', '-@', str(threads), '-o', temp_ns, input_bam)
        pysam.fixmate('-m', '-@', str(threads), temp_ns, temp_fm)
        pysam.sort('-@', str(threads), '-o', temp_cs, temp_fm)
        pysam.markdup('-@', str(threads), temp_cs, output_bam)
        pysam.index(output_bam)
    finally:
        for f in [temp_ns, temp_fm, temp_cs]:
            if os.path.exists(f):
                os.remove(f)

Check Duplicate Rate

import pysam

def duplicate_rate(bam_path):
    with pysam.AlignmentFile(bam_path, 'rb') as bam:
        total, duplicates = 0, 0
        for read in bam:
            if read.is_secondary or read.is_supplementary:
                continue
            total += 1
            if read.is_duplicate:
                duplicates += 1
    return {'total': total, 'duplicates': duplicates, 'rate': duplicates / total * 100 if total > 0 else 0}

Filter Duplicates

import pysam

def remove_duplicates(input_bam, output_bam):
    with pysam.AlignmentFile(input_bam, 'rb') as infile:
        with pysam.AlignmentFile(output_bam, 'wb', header=infile.header) as outfile:
            for read in infile:
                if not read.is_duplicate:
                    outfile.write(read)

Expected Duplicate Rates

See SKILL.md "When to Mark Duplicates" decision table for assay-specific expected duplicate rates.

High duplicate rates suggest: - Low library complexity - Over-amplification during PCR - Low input DNA quantity

Troubleshooting

"mate not found"

Input to fixmate is not name-sorted:

samtools sort -n -o namesorted.bam input.bam
samtools fixmate -m namesorted.bam fixmate.bam

"no MC tag found"

fixmate was not run with -m flag:

samtools fixmate -m namesorted.bam fixmate.bam  # Include -m

"not coordinate sorted"

Input to markdup is not coordinate-sorted:

samtools sort -o coordsorted.bam fixmate.bam
samtools markdup coordsorted.bam marked.bam

High Memory Usage

markdup loads read information into memory. Increase parallelization:

samtools markdup -@ 8 input.bam marked.bam

Tips