Running the Pipeline¶

Every interaction goes through run_modules.sh at the repository root. It wraps sbatch, activates conda on the compute node, and launches Snakemake with the right executor.

The job hierarchy¶

flowchart TB
    A["./run_modules.sh --submit"] -->|sbatch| B["Wrapper job: snakemake_<module>"]
    B --> C["Snakemake process on a compute node"]
    C -->|submits| D["<module>_decomposition target=Gene1"]
    C -->|submits| E["<module>_decomposition target=Gene2"]
    C -->|submits| F["<module>_hotspot target=Gene1"]
    C -->|submits| G["<module>_hotspot target=Gene2"]

This two-tier structure explains most of the pipeline's operational behaviour:

• Killing the wrapper stalls everything. Snakemake is what submits and supervises jobs. If it dies, running jobs continue but nothing new is submitted and no outputs are finalized.
• squeue shows both tiers. Wrapper jobs are named snakemake_<module>; analysis jobs are named after the rule and target.
• Wrapper wall time must exceed total pipeline time, not per-job time.

Command reference¶

./run_modules.sh [ACTION] [MODULE SELECTION] [OPTIONS]

.snakemake_env.sh must exist in the repository root. If missing, the script prints the required contents and exits.

Actions¶

Choose exactly one.

--build-conda

Build the conda environments declared by each module's environment.yaml. Submitted as a SLURM job by default.

./run_modules.sh --build-conda --all
./run_modules.sh --build-conda --modules cnmf,cpca
./run_modules.sh --build-conda --all --interactive

--submit

Submit analysis jobs. The main action.

./run_modules.sh --submit --all
./run_modules.sh --submit --modules pca,cpca
./run_modules.sh --submit --all --dry-run
./run_modules.sh --submit --modules cpca --forcerun cpca_decomposition

--unlock

Release a stale Snakemake working-directory lock, left behind when a run is killed rather than exiting cleanly.

./run_modules.sh --unlock --all

--clear-logs

Delete the log directories for the selected modules and exit.

./run_modules.sh --clear-logs --modules cpca

This deletes your record of what happened

Logs are the only account of why a job failed. Clear them when starting a genuinely fresh run, not while debugging one.

Module selection¶

Required for every action.

Available modules: pca, cpca, contrapc, kcpca, kcontrapc, cnmf, contrastivevi, de-dgca.

Resource options¶

Apply to --build-conda and --submit, and size the wrapper job only. See Resources and SLURM.

Other options¶

--interactive

Build conda environments in the current shell rather than submitting a job. --build-conda only.

-d, --dry-run

Passes Snakemake's -n. Parses every Snakefile, reads the .h5ad, resolves targets, and prints the job plan without submitting.

Always dry-run before a large submission. It costs seconds and catches config errors, missing files, and unexpected target counts.

-f, --forcerun RULE

Force a specific rule to rerun even if its outputs exist.

Rule names follow <module>_decomposition and <module>_hotspot, with exceptions: cnmf uses cnmf_factorization, contrastivevi uses contrastivevi_model, and de-dgca uses de_analysis and dgca_analysis.

Forcing a rule reruns everything downstream

Forcing cpca_decomposition also reruns cpca_hotspot, since Hotspot consumes the projection. To rerun only Hotspot - after changing hotspot_n_neighbors, say - force cpca_hotspot instead.

Examples¶

./run_modules.sh --build-conda --all --mem 16
./run_modules.sh --submit --modules pca --dry-run
./run_modules.sh --submit --modules pca

./run_modules.sh --submit --all --mem 64 --time 24:00:00

./run_modules.sh --unlock --all
./run_modules.sh --submit --all

./run_modules.sh --submit --modules cpca --forcerun cpca_hotspot

./run_modules.sh --submit --modules kcpca,kcontrapc \
  --partition bigmem --time 24:00:00

Monitoring¶

Job status¶

# Everything you have queued or running
squeue -u $USER

# Just the Snakemake supervisors
squeue -u $USER | grep snakemake_

# Completed jobs with exit state and runtime
sacct -u $USER --format=JobID,JobName%40,State,Elapsed,MaxRSS,ReqMem

MaxRSS against ReqMem is the fastest way to right-size resources.mem_mb after a first run - including for jobs that succeeded but came close.

Log layout¶

<module>/logs/
├── wrapper/          Snakemake supervisor: pipeline-level progress
│   ├── snakemake_*.out
│   ├── snakemake_*.err
│   └── conda_build_*.err
├── <method>/         Per-target application logs
│   └── <target>.log
└── slurm/            Raw SLURM stdout/stderr per job
    └── *.out

The distinction matters: a Python traceback appears in the method log, but an OOM kill leaves no traceback at all - the process is terminated by the kernel, and only SLURM records it.

Reading the wrapper log¶

At startup each Snakefile prints its resolved configuration:

Skipping 12 perturbations with < 50 cells:
  - GENE_A: 31 cells
  - GENE_B: 44 cells
Singular mode: 488 perturbations to analyze (min_cells=50)

Analysis configuration:
  Mode: singular
  Background: non-targeting
  Targets: 488

Check this first on every run. It confirms your config took effect and tells you exactly which perturbations were excluded and why - information available nowhere else.

Progress by output¶

# Completed decompositions
ls cpca/results/*/cpca_projection.csv 2>/dev/null | wc -l

# Which targets have not finished
comm -13 \
  <(ls cpca/results/*/hotspot_cpca_modules.csv 2>/dev/null | cut -d/ -f3 | sort) \
  <(ls -d cpca/results/*/ | cut -d/ -f3 | sort)

Finding failures¶

sacct -u $USER --state=FAILED,OUT_OF_MEMORY,TIMEOUT \
  --format=JobID,JobName%40,State,Elapsed,MaxRSS

grep -l "Traceback\|Error" cpca/logs/cpca/*.log

OUT_OF_MEMORY and TIMEOUT are distinct from FAILED in SLURM and are frequently the real story - a job that "failed" with no traceback was almost certainly one of these.

Cancelling¶

scancel <job_id>        # one job
scancel -u $USER        # everything

Troubleshooting¶

min_cells: 50    # never below 25 in singular mode

./run_modules.sh --unlock --modules <module>
./run_modules.sh --submit --modules <module>

sacct -j <job_id> --format=JobID,State,Elapsed,MaxRSS,ReqMem

resources:
  mem_mb: 256000
hotspot_resources:
  mem_mb: 128000

cat <module>/logs/wrapper/conda_build_*.err
./run_modules.sh --build-conda --modules <module> --mem 16

temp_dir: "/path/to/scratch"

./run_modules.sh --submit --all --time 48:00:00

./run_modules.sh --submit --modules cpca

Getting help¶

When opening an issue on the pipeline repository, include the module and exact command, the Analysis configuration: block from the wrapper log, the relevant logs/<method>/<target>.log, sacct -j <job_id> --format=JobID,State,Elapsed,MaxRSS,ReqMem, and your master and module config.yaml.