← Về thư mục
📄 / / profiles / r-and-d / raw_evidence / scientific-agent-skills / skills / pathml / references / data_management.md

Data management, h5path, manifests, datasets, and provenance

This reference targets PathML 3.0.5 stable and a local, de-identified research workflow.

Data boundaries

Separate four classes of data:

  1. Source slides — immutable, access-controlled originals.
  2. Linkage data — direct identifiers and the pseudonym mapping, held outside the analysis workspace by an authorized custodian.
  3. Analysis data — pseudonymous manifests, tiles, masks, counts, graphs, and features.
  4. Reports/models — potentially identifying derived artifacts that still require governance.

Do not assume derived images or embeddings are anonymous. Rare morphology, scanner metadata, dates, or cohort combinations can re-identify a participant. Apply the minimum-necessary principle and institutional retention policy.

Manifest first

Use one row per slide. Recommended columns:

slide_id,patient_id,specimen_id,path,split,stain,backend,site,scanner

Rules:

Validate before PathML:

python scripts/slide_manifest.py validate \
  --manifest metadata/manifest.csv \
  --root .

The validator checks strict CSV structure, duplicate IDs/paths, missing local files, unsafe paths, supported suffixes, and patient/slide leakage. It does not upload data or inspect arbitrary clinical fields.

h5path format

PathML processes slides into an HDF5-based .h5path file. Stable documentation describes:

root/
├── fields/
│   ├── labels/          # slide-level attributes
│   └── slide_type/      # stain/platform flags
├── masks/               # slide-level masks
├── counts/              # AnnData-like counts storage
└── tiles/
    ├── attributes       # tile_shape, tile_stride
    └── "(i, j)"/
        ├── array
        ├── masks/
        ├── labels/
        └── attributes   # coords, name

Write and reopen through public APIs:

from pathml.core import SlideData

slide.write("derived/slide-001.h5path")
reopened = SlideData("derived/slide-001.h5path")

There is no stable to_hdf5(), from_hdf5(), or load_tiles_from_hdf5() API. SlideDataset.write(directory, filenames=None) calls each slide's write().

Stable documentation states HDF5 datasets are stored as float16; confirm dtype for the exact arrays your workflow writes. Quantitative marker intensities can lose precision if silently cast. Record and test expected dtype, range, NaN/Inf, compression, and round-trip tolerance.

h5path trust boundary

Treat .h5path as a structured binary input, not harmless data:

Use a sidecar JSON manifest for provenance rather than relying on arbitrary HDF5 labels. Keep the JSON strict, bounded, pseudonymous, and versioned.

PyTorch tile dataset

The canonical stable import is:

from pathml.datasets import TileDataset
from torch.utils.data import DataLoader

tiles = TileDataset("derived/slide-001.h5path")
loader = DataLoader(
    tiles,
    batch_size=8,
    shuffle=False,
    num_workers=0,
)

Each item is:

(tile_image, tile_masks, tile_labels, slide_labels)

Shapes:

Do not assume mask dictionary order carries semantics. Persist ordered mask names in a separate schema and assert them when loading.

pathml.ml.TileDataset is also exported in 3.0.5, but pathml.datasets.TileDataset is the documented dataset API.

SlideDataset

SlideDataset(slides) accepts a list of already constructed SlideData objects:

from pathml.core import HESlide, SlideDataset

slides = [
    HESlide("data/slide-001.svs", backend="openslide"),
    HESlide("data/slide-002.svs", backend="openslide"),
]
cohort = SlideDataset(slides)
cohort.run(pipeline, distributed=False, tile_size=512, level=0)
cohort.write("derived")

It does not accept a glob/path list plus tiling arguments as a constructor. Preserve a deterministic manifest order and map output filenames explicitly.

Public data modules

Stable pathml.datasets exports:

from pathml.datasets import DeepFocusDataModule, PanNukeDataModule

PanNuke

pannuke = PanNukeDataModule(
    data_dir="approved_data/pannuke",
    download=False,
    shuffle=True,
    nucleus_type_labels=True,
    split=1,
    batch_size=8,
    hovernet_preprocess=True,
)

Published folds are not a substitute for verifying patient/source-slide independence for the intended claim.

DeepFocus

deepfocus = DeepFocusDataModule(
    data_dir="approved_data/deepfocus",
    download=False,
    shuffle=True,
    batch_size=8,
)

MD5 here is an upstream integrity check, not a modern provenance guarantee. Record a SHA-256 and dataset license/source separately.

PathML 3.0.5 does not export TCGADataModule. Use a separately governed data acquisition process for TCGA/GDC and document its API/version/consent terms.

Before changing any download flag to True, tell the user:

Require explicit opt-in. Never place downloaded archives inside the repository.

Graph datasets and unsafe .pt files

pathml.datasets.EntityDataset assembles cell graphs, tissue graphs, and assignment matrices. Stable source opens .pt files using PyTorch object deserialization with unrestricted object loading.

Consequences:

The bundled inference planner and graph validator never load .pt, .pth, .ckpt, pickle, ONNX, or other model/graph binaries.

Split design and leakage

Create the split column once, before tiling:

patient → specimen/block → slide/rescan/serial section → region → tile

Everything below a patient follows the patient's split unless the scientific design explicitly requires a stricter grouping.

Common leakage paths:

The manifest validator reports patient and slide leakage, but it cannot discover unknown biological relatedness. Document grouping assumptions.

Provenance sidecar

Recommended strict JSON fields:

{
  "schema_version": "1.0",
  "pathml_version": "3.0.5",
  "source_sha256": "hex-digest",
  "slide_id": "slide-001",
  "patient_id": "patient-001",
  "split": "train",
  "backend": "openslide",
  "level": 0,
  "downsample": 1.0,
  "mpp_x": null,
  "mpp_y": null,
  "tile_size_ij": [512, 512],
  "tile_stride_ij": [512, 512],
  "tile_pad": false,
  "pipeline_id": "he-v1",
  "code_revision": "project-commit",
  "created_utc": "RFC3339 timestamp"
}

Do not put a direct identifier in these fields. Add:

Use SHA-256 for provenance:

import hashlib
from pathlib import Path

def sha256_file(path: Path, chunk_bytes: int = 1024 * 1024) -> str:
    digest = hashlib.sha256()
    with path.open("rb") as handle:
        for chunk in iter(lambda: handle.read(chunk_bytes), b""):
            digest.update(chunk)
    return digest.hexdigest()

Hash only authorized local files and expect full-slide hashing to be I/O-heavy. Do not print paths containing identifiers.

Storage and lifecycle checklist

Sources, accessed 2026-07-23