Cluster and batch workflows

This chapter is for everyone who installs StochasticGene.jl (Pkg.add("StochasticGene")) and needs to:

run many fit jobs on a cluster (including NIH Biowulf) using command files (scheduler-agnostic) or swarm files (Biowulf naming) from the stage helpers [make_fitscript][makefitscript], [`makefitscriptsfromcsv][make_fitscripts_from_csv], [makecommandfile`][makecommandfile], [make_commandfile_from_csv][makecommandfilefromcsv], [`makefitscriptsandcommandfilefromcsv][make_fitscripts_and_commandfile_from_csv], and compatibility wrappers [makeswarmfilefromcsv`][makeswarmfilefromcsv], [make_fitscripts_and_swarm_from_csv][makefitscriptsandswarmfrom_csv];
follow the recommended coupled-model workflow: fit individual units first, then merge those fitted rates into one initial rate file for the coupled model.

The implementations live in stage.jl (stage-native scripts + command files), biowulf.jl (legacy and Biowulf-specific wrappers), and io.jl (merging rate tables). Function signatures and defaults are in the docstrings; this page is the narrative guide published with the GitHub-hosted documentation.

Source entry points:

Stage API (preferred)

Use these in new code:

Step	Functions
Write one key script	[`make_fitscript`][make_fitscript]
Write scripts from keys CSV	[`make_fitscripts_from_csv`][makefitscriptsfrom_csv]
Build one launch command string	[`build_julia_script_command`][buildjuliascript_command]
Write command file from script list	[`make_commandfile`][makecommandfile], [`writejuliacommandfile`][writejuliacommand_file]
Write command file from keys CSV	[`make_commandfile_from_csv`][makecommandfilefrom_csv]
Write scripts + command file from CSV	[`make_fitscripts_and_commandfile_from_csv`][makefitscriptsandcommandfilefrom_csv]
Biowulf naming wrappers	[`make_swarmfile_from_csv`][makeswarmfilefromcsv], [`makefitscriptsandswarmfromcsv`][makefitscriptsandswarmfrom_csv]

Gene panels: [write_fitfile_genes][writefitfilegenes] and [makeswarm_genes][makeswarmgenes] produce one shared script with gene=ARGS[1] and one command-file line per gene (still the recommended pattern for many genes, one model). Pass an explicit gene vector, or omit the vector and let `makeswarmgenes(; datapath=..., datacond=...)scan an RNA data folder for matchingGENECOND.txtfiles. By default, folder scanning usescheckgenesand restricts to genes with available halflife/allele metadata; passfiltermetadata=false` to scan filenames only.

For datatype="rna", each input file is a steady-state RNA histogram named GENE_COND.txt; the first column is the histogram over RNA copy number bins. For example:

data/HCT116_testdata/
├── CENPL_MOCK.txt
└── MYC_MOCK.txt

using StochasticGene

# v0.7-style RNA folder sweep, current API
makeswarm_genes(
    datapath="data/HCT116_testdata",
    datacond="MOCK",
    resultfolder="HCT116_test",
    filedir="run-HCT116-testdata-rna",
    cell="HCT116",
    datatype="rna",
    G=2,
    R=0,
    S=0,
    insertstep=1,
    transitions=([1, 2], [2, 1]),
    nchains=8,
    project="/home/carsonc/github/StochasticGene.jl/",
)

By default this writes a single Biowulf-style command file named fit.swarm, with one command per gene. For large gene panels on Biowulf, keep one command file and use swarm -b <bundle> when submitting. This reduces the Slurm array size without making many separate swarm submissions. The return value is a named tuple with genes, fitfile, and commandfiles.

out = makeswarm_genes(
    datapath="data/HCT116_testdata",
    datacond="MOCK",
    resultfolder="HCT116_test",
    filedir="run-HCT116-testdata-rna",
    nchains=2,
    nthreads=1,
)

out.commandfiles

Submit the command file on Biowulf. For thousands of genes, use -b to bundle multiple gene commands serially within each array task:

cd run-HCT116-testdata-rna
swarm -f fit.swarm -b 20 -g 4 -t 2 --time 02:00:00 --merge-output --module julia

For a larger scRNA folder on Biowulf, the same pattern applies. If the data are under /data/carsonc/scrna/data/RamosNELFA_NEG_IFNa_rep1_Sdata and files are named like AFF2_NIr1.txt, use root="/data/carsonc/scrna", datapath="RamosNELFA_NEG_IFNa_rep1_Sdata", and datacond="NIr1":

out = makeswarm_genes(
    root="/data/carsonc/scrna",
    datapath="RamosNELFA_NEG_IFNa_rep1_Sdata",
    datacond="NIr1",
    cell="U3A",
    resultfolder="RamosNELFA_NEG_IFNa_rep1",
    filedir="run-RamosNELFA_NEG_IFNa_rep1",
    nchains=2,
    nthreads=1,
    project="/home/carsonc/github/StochasticGene.jl/",
)

The function prints the number of genes and the number of generated command files. Submit the single command file with bundling:

cd run-RamosNELFA_NEG_IFNa_rep1
swarm -f fit.swarm -b 20 -g 4 -t 2 --time 02:00:00 --merge-output --module julia

Set batchsize=<N> only when you deliberately want several command files. The return value contains the gene list and paths:

length(out.genes)
out.commandfiles

The same writer can also add launch wrappers for non-Biowulf systems:

# Slurm array wrapper, submit with: sbatch fit_slurm.sh
makeswarm_genes(
    datapath="data/HCT116_testdata",
    datacond="MOCK",
    resultfolder="HCT116_test",
    filedir="run-HCT116-testdata-rna",
    project="/path/to/StochasticGene.jl",
    scheduler=:slurm,
    scheduler_jobs=100,       # max array jobs running at once
    slurm_mem="8G",
    slurm_time="02:00:00",
)

# GNU Parallel wrapper, run with: ./fit_parallel.sh
makeswarm_genes(
    datapath="data/HCT116_testdata",
    datacond="MOCK",
    resultfolder="HCT116_test",
    filedir="run-HCT116-testdata-rna",
    project="/path/to/StochasticGene.jl",
    scheduler=:parallel,
    scheduler_jobs=16,
)

In v0.7.8, makeswarm(; datafolder=..., conds=...) inferred genes from the data folder. Current makeswarm(["key1", ...]) is key-based; the folder-scanning RNA behavior now lives in makeswarm_genes(; datapath=..., datacond=...). By default it uses the older checkgenes metadata gate; pass filter_metadata=false only when you want to scan filenames directly.

RNA result tables after a batch run

After an RNA histogram/count swarm finishes, convert raw fit outputs into CSV tables with the dataframe helpers:

write_dataframes_only(
    "results/HCT116_test",
    "data/HCT116_testdata";
    datatype = "rna",
    rna_truncation = :legacy,
)

This calls make_dataframes, assembles raw rates_*.txt, measures_*.txt, and param-stats_*.txt files when needed, and writes summary CSVs such as Summary_rna-HCT116_2.csv. The summary includes fitted rates, parameter statistics, model/condition metadata, and observed RNA moments.

Match rna_truncation to the fit. If the jobs used burst=true, posterior burst-size files are assembled into burst_*.csv; if they used optimize=true, numerical optimizer files are assembled into optimized_*.csv. These are fit-time products and are not generated retroactively by write_dataframes_only.

Within rates_*.txt, row 1 is the largest-likelihood point encountered by the sampler, followed by posterior mean, posterior median, and last stored sample. The independent numerical optimizer result is stored only in optimized_*.txt/optimized_*.csv.

For key-based result folders, use the key-specific helper:

write_dataframes_only_key("results/HCT116_test"; datatype = "rna")

This writes Summary_key.csv, preserving the original fit key in the Key column and reading info_<key>.jld2 metadata when available.

Coupled models: recommended workflow (single units → merge → coupled fit)

For coupled transcriptional units (e.g. enhancer + gene, with or without a hidden unit), the suggested workflow is:

Fit each unit as its own single-unit model Run separate fits (e.g. enhancer-only and gene-only traces or histograms) so each produces a standard MCMC rates_*.txt (rows = posterior samples, columns = rate headers for that unit). Use normal [fit][fit] calls, or stage explicit run specs with [stage_write_run_specs][stagewriterun_specs] and emit jobs with [makeswarm][makeswarm].
Merge the fitted rates into one wide table Stack the columns from the two (or more) unit files and append coupling placeholder columns using [create_combined_file][createcombinedfile] for two-unit models, or [create_combined_file_mult][createcombinedfilemult] for models with more than two units. You choose Nenh / Ngene (or per-unit column counts) to match how each set of rates is laid out in your files (see docstrings). For many keys (e.g. from a CSV of model names), use [`createcombinedfiles`][createcombinedfiles] or [`createcombinedfilesdriver][create_combined_files_driver], which callcreatecombinedfileonce per key and name outputs with [combinedrateskey`][combinedrateskey].
Run the coupled fit using the combined file as the starting rates Point datapath / resultfolder / label (or your run spec) at that merged file so the coupled MCMC warm-starts from the stacked single-unit posteriors. The coupled [fit][fit] uses tuple G, R, coupling, joint datatype (e.g. tracejoint), etc. Coupling strengths are then estimated in the coupled run (the placeholder columns from step 2 get updated).
Optional: batch everything on the cluster Use stage-native [make_fitscripts_from_csv][makefitscriptsfromcsv] + [`makecommandfilefromcsv][make_commandfile_from_csv] (or [makefitscriptsandcommandfilefromcsv`][makefitscriptsandcommandfilefromcsv]) so each job runs fit(; key=..., ...) from prewritten info_<key> specs (see Run specification (info TOML)). For Biowulf-style *.swarm naming, use [make_swarmfile_from_csv][makeswarmfilefrom_csv].

This order—individual fits → merge → coupled fit—is the standard way to get a sensible initial combined rate file for coupled models without fitting all parameters cold.

Scheduler launch files and Biowulf swarm

Stage helpers do not submit jobs to a scheduler by themselves. They write files you submit with Biowulf’s swarm or another launcher (sbatch, GNU Parallel, custom wrappers):

<commandfile> (default fit.commands) — one command line per run key (julia ... fitscript_<key>.jl).
<swarmfile>.swarm — same content style, Biowulf-compatible extension via [make_swarmfile_from_csv][makeswarmfilefrom_csv].
fitscript_<key>.jl per key — typically calls fit(; key="<key>", ...) with shared options (resultfolder, maxtime, samplesteps, warmupsteps, inference_method, device, parallel, gradient, etc.).

The biowulf.jl convenience writers (makeswarm, makeswarm_folder, and makeswarm_genes) also accept:

scheduler=:swarm — default; write the .swarm command list and no extra wrapper.
scheduler=:slurm — write the .swarm command list plus a submit-ready Slurm array script, for example fit_slurm.sh.
scheduler=:parallel — write the .swarm command list plus a GNU Parallel wrapper, for example fit_parallel.sh.
scheduler=:command — write only the command list, useful for simple sequential shell runs.

For Slurm and GNU Parallel, scheduler_jobs means the maximum number of commands running at the same time.

Typical use on Biowulf

Install StochasticGene in your Julia environment (see Installation, including Biowulf Installation).
From Julia (interactive session or batch script), run something like:

using StochasticGene

# Stage-native pipeline: scripts + command file from a keys CSV.
# CSV requires a key column, default: Model_name
make_fitscripts_and_commandfile_from_csv(
    "keys.csv";
    filedir      = "my_jobs",
    commandfile  = "fit.commands",
    juliafile    = "fitscript",
    project      = "/path/to/your/StochasticGene.jl",
    nprocs       = 4,
    nthreads     = 1,
    resultfolder = "my_results",
    root         = ".",
    maxtime      = 72000.0,
    samplesteps  = 1_000_000,
)

# If you specifically want .swarm naming:
# make_fitscripts_and_swarm_from_csv("keys.csv"; filedir="my_jobs", swarmfile="fit", ...)

Submit the swarm from the shell (example):

cd my_jobs
swarm -g 4 -t 16 -b 1 --time 24:00:00 --module julialang -f fit.swarm

Adjust -g, -t, time, and module to match your allocation and Julia module name on Biowulf.

Typical use on Slurm

Generate the scripts with scheduler=:slurm:

makeswarm_genes(
    datapath="data/HCT116_testdata",
    datacond="MOCK",
    resultfolder="HCT116_test",
    filedir="run-HCT116-testdata-rna",
    project="/path/to/StochasticGene.jl",
    nchains=8,
    scheduler=:slurm,
    scheduler_jobs=100,
    slurm_mem="8G",
    slurm_time="02:00:00",
)

Then submit the generated wrapper:

cd run-HCT116-testdata-rna
sbatch fit_slurm.sh

The wrapper contains the Slurm array directive and reads one command line from fit.swarm for each array task.

Typical use with GNU Parallel

Generate the scripts with scheduler=:parallel:

makeswarm_genes(
    datapath="data/HCT116_testdata",
    datacond="MOCK",
    resultfolder="HCT116_test",
    filedir="run-HCT116-testdata-rna",
    project="/path/to/StochasticGene.jl",
    nchains=8,
    scheduler=:parallel,
    scheduler_jobs=16,
)

Then run the generated wrapper:

cd run-HCT116-testdata-rna
./fit_parallel.sh

Override the job count without regenerating files by setting JOBS:

JOBS=32 ./fit_parallel.sh

For a simple sequential run on any Unix shell, run the command file directly:

cd run-HCT116-testdata-rna
bash fit.swarm

Generating keys and info_<key> in bulk

[stage_write_run_specs][stagewriterunspecs] writes `info<key>.jld2` plus marker TOML files from explicit spec dictionaries.
[makeswarm][makeswarm] emits scheduler commands and fit scripts for staged keys.
[make_fitscripts_and_commandfile_from_csv][makefitscriptsandcommandfilefrom_csv] handles CSV-defined batches without a legacy model-sweep wrapper.

Swarm `julia -p`, `nchains`, merged `info_<key>`, and `root`

Parallel workers: The generated command should use -p N (or equivalent) consistent with how many chains run in parallel. Pass nchains=N to [makeswarm][makeswarm] and to the corresponding run specs. For NUTS/ADVI, nchains still controls how many independent chains are launched; within-chain parallelism follows each method’s options (parallel on [NUTSOptions][NUTSOptions] / [ADVIOptions][ADVIOptions], set via [load_options][load_options] from the run spec).
Biowulf gradient NUTS default: use gradient = :finite unless a per-model benchmark shows otherwise. On Biowulf, the usual production shape is julia -p N -t 1 with nchains = N, parallel = :distributed, and inference_method = :nuts.
Legacy trace metadata: Before staging migrated specs, run [normalize_trace_specs_legacy_t_end!][normalizetracespecslegacytend] so historical huge `tendsentinels become **-1.0**, matching current [defaulttracespecsforcoupled][default_trace_specs_for_coupled] and avoiding invalid frame indices in [readtracefiles`][readtracefiles].
root in generated fit scripts: Scripts list root exactly as in the run spec (no forced abspath). Use root="." if the job’s working directory is the project root (set cd in the swarm or submit from the right folder). Paths resolved in an interactive Biowulf session can differ from batch jobs; "." avoids baking in an interactive-only absolute path.

Key-based naming

Many batch helpers assume a string key per run:

results/<resultfolder>/info_<key>.toml and info_<key>.jld2
rates_<key>.txt

See Run specification (info TOML). Presets for cluster reruns are written with [stage_write_run_specs][stagewriterun_specs].

Combined rate files (`io.jl`) — reference

[read_rates_table][readratestable], [write_rates_table][writeratestable]
[merge_coupled_two_unit_rates][mergecoupledtwounitrates], [merge_coupled_stacked_units][mergecoupledstacked_units]
[create_combined_file][createcombinedfile], [create_combined_file_mult][createcombinedfile_mult]
[read_combined_file_specs_csv][readcombinedfilespecscsv], [create_combined_files_driver][createcombinedfilesdriver], [`createcombinedfiles`][createcombinedfiles], [`createcombinedfilesh3latent`][createcombinedfilesh3_latent]

After the coupled fit

Post-processing examples: [write_correlation_functions][writecorrelationfunctions], [write_traces][write_traces], and other analysis functions in the API Reference.

Maintainer note: where to document what

Topic	Canonical place
User workflows, stage command files, Biowulf submission, coupled merge order	This page (hosted docs)
`info_<key>` file format	runspectoml.md
README on GitHub	Short pointer + link to hosted documents
Exact function signatures	Docstrings in `stage.jl` / `biowulf.jl` / `io.jl`

[ADVIOptions]: https://github.com/nih-niddk-mbs/StochasticGene.jl/search?q=ADVIOptions&type=code [NUTSOptions]: https://github.com/nih-niddk-mbs/StochasticGene.jl/search?q=NUTSOptions&type=code [buildjuliascriptcommand]: https://github.com/nih-niddk-mbs/StochasticGene.jl/search?q=buildjuliascriptcommand&type=code [combinedrateskey]: https://github.com/nih-niddk-mbs/StochasticGene.jl/search?q=combinedrateskey&type=code [createcombinedfile]: https://github.com/nih-niddk-mbs/StochasticGene.jl/search?q=createcombinedfile&type=code [createcombinedfilemult]: https://github.com/nih-niddk-mbs/StochasticGene.jl/search?q=createcombinedfilemult&type=code [createcombinedfiles]: https://github.com/nih-niddk-mbs/StochasticGene.jl/search?q=createcombinedfiles&type=code [createcombinedfilesdriver]: https://github.com/nih-niddk-mbs/StochasticGene.jl/search?q=createcombinedfilesdriver&type=code [createcombinedfilesh3latent]: https://github.com/nih-niddk-mbs/StochasticGene.jl/search?q=createcombinedfilesh3latent&type=code [defaulttracespecsforcoupled]: https://github.com/nih-niddk-mbs/StochasticGene.jl/search?q=defaulttracespecsforcoupled&type=code [fit]: https://github.com/nih-niddk-mbs/StochasticGene.jl/search?q=fit&type=code [loadoptions]: https://github.com/nih-niddk-mbs/StochasticGene.jl/search?q=loadoptions&type=code [makecommandfile]: https://github.com/nih-niddk-mbs/StochasticGene.jl/search?q=makecommandfile&type=code [makecommandfilefromcsv]: https://github.com/nih-niddk-mbs/StochasticGene.jl/search?q=makecommandfilefromcsv&type=code [makefitscript]: https://github.com/nih-niddk-mbs/StochasticGene.jl/search?q=makefitscript&type=code [makefitscriptsandcommandfilefromcsv]: https://github.com/nih-niddk-mbs/StochasticGene.jl/search?q=makefitscriptsandcommandfilefromcsv&type=code [makefitscriptsandswarmfromcsv]: https://github.com/nih-niddk-mbs/StochasticGene.jl/search?q=makefitscriptsandswarmfromcsv&type=code [makefitscriptsfromcsv]: https://github.com/nih-niddk-mbs/StochasticGene.jl/search?q=makefitscriptsfromcsv&type=code [makeswarmfilefromcsv]: https://github.com/nih-niddk-mbs/StochasticGene.jl/search?q=makeswarmfilefromcsv&type=code [makeswarmgenes]: https://github.com/nih-niddk-mbs/StochasticGene.jl/search?q=makeswarmgenes&type=code [makeswarm]: https://github.com/nih-niddk-mbs/StochasticGene.jl/search?q=makeswarm&type=code [mergecoupledstackedunits]: https://github.com/nih-niddk-mbs/StochasticGene.jl/search?q=mergecoupledstackedunits&type=code [mergecoupledtwounitrates]: https://github.com/nih-niddk-mbs/StochasticGene.jl/search?q=mergecoupledtwounitrates&type=code [normalizetracespecslegacytend]: https://github.com/nih-niddk-mbs/StochasticGene.jl/search?q=normalizetracespecslegacytend%21&type=code [readcombinedfilespecscsv]: https://github.com/nih-niddk-mbs/StochasticGene.jl/search?q=readcombinedfilespecscsv&type=code [readratestable]: https://github.com/nih-niddk-mbs/StochasticGene.jl/search?q=readratestable&type=code [readtracefiles]: https://github.com/nih-niddk-mbs/StochasticGene.jl/search?q=readtracefiles&type=code [writecorrelationfunctions]: https://github.com/nih-niddk-mbs/StochasticGene.jl/search?q=writecorrelationfunctions&type=code [writefitfilegenes]: https://github.com/nih-niddk-mbs/StochasticGene.jl/search?q=writefitfilegenes&type=code [writejuliacommandfile]: https://github.com/nih-niddk-mbs/StochasticGene.jl/search?q=writejuliacommandfile&type=code [writeratestable]: https://github.com/nih-niddk-mbs/StochasticGene.jl/search?q=writeratestable&type=code [stagewriterunspecs]: https://github.com/nih-niddk-mbs/StochasticGene.jl/search?q=stagewriterunspecs&type=code [writetraces]: https://github.com/nih-niddk-mbs/StochasticGene.jl/search?q=writetraces&type=code