Outputs¶

Every module writes to <output_dir>/<target>/, where output_dir defaults to results and <target> is the sanitized perturbation name, or the literal string pooled in pooled mode.

Because output_dir is relative, each module writes its own results/ tree inside its own directory:

perturbscape/
├── cpca/results/TP53/
│   ├── cpca_projection.csv
│   ├── cpca_eigenvalues.csv
│   └── hotspot_cpca_modules.csv
├── cnmf/results/TP53/
└── de-dgca/results/TP53/

Set output_dir to an absolute path to consolidate results elsewhere. All modules would then share one tree, but filenames are prefixed by method so they do not collide.

Two kinds of output¶

Embedding outputs are method-specific: projections, eigenvalues, eigenvectors, loadings. Names differ per module.

Hotspot outputs are uniform across the seven Hotspot-enabled modules, always named hotspot_<method>_*. These are the gene programs, and they are what disease enrichment consumes.

For most downstream work the two files that matter are:

Everything else is either input to those, or diagnostic.

modules = pd.read_csv("hotspot_cpca_modules.csv", index_col=0)
real = modules[modules["Module"] != -1]

Storage¶

The dominant cost is hotspot_<m>_local_correlations.csv, a dense gene by gene matrix over all genes passing the FDR cutoff. It scales quadratically and is usually larger than everything else combined. For one perturbation with 5,000 significant genes:

Multiply by perturbations times seven modules. On a genome-scale screen, plan for the local correlation files specifically, or delete them once modules have been created - the modules and scores are derived from them and do not need them retained.

File reference¶

Hotspot files¶

Produced by all modules except de-dgca, where <m> is the method tag.

Method tags match the module directory name, with one exception:

Module-specific files¶

Column conventions¶

Alpha-indexed columns¶

Contrastive modules run four alpha values and concatenate the results, so columns encode both:

alpha_0.0_pc_1     alpha_0.0_pc_2     ...
alpha_0.271_pc_1   alpha_0.271_pc_2   ...
alpha_1.0_pc_1     alpha_1.0_pc_2     ...

Alpha 0 is formatted with one decimal, other values with three. Total columns is 4 x n_components, not n_components. Splitting by alpha is usually what you want:

import pandas as pd

proj = pd.read_csv("cpca_projection.csv", index_col=0)
alphas = sorted({c.split("_pc_")[0] for c in proj.columns})

for a in alphas:
    block = proj[[c for c in proj.columns if c.startswith(a + "_pc_")]]
    print(a, block.shape)

Index conventions¶

cnmf gene spectra are transposed relative to the loadings files of other modules: programs are rows, genes are columns.

Metadata files¶

Every *_metadata.txt records the parameters actually used, which may differ from what you think you configured - module configs override the master. When a result is surprising, read the metadata before re-reading the config.

What feeds disease enrichment¶

Stage 2 consumes a single gene-by-program matrix per method, assembled from the Hotspot modules and, for de-dgca, from the DE and DGCA results. Program columns are named <method>_<perturbation>_<suffix>, where the suffix identifies the program type (cnmf, hotspot, alpha, DE, DGCA, PC). That naming is what lets the selection step filter to a single perturbation or exclude pooled runs.