Per-task admission control for genomics pipelines
A gate that decides, for every task in a bioinformatics pipeline, whether it may run — and records why either way.
A Nextflow hook, a policy, and one decision record per task. End-to-end timing came to 119 µs at the median over 210 decisions, published as a corpus anyone can replay. The load-bearing outcome was not that number but establishing by experiment that AWS's managed service honours a directive its own linter called unsupported.
process.beforeScript hook before each task script: FASTQC admitted, SEQTK_TRIM refused because the offered minimum read length is 10 where the descriptor requires 20. A record is written either way. Illustration; the figures are the measured ones.Measured on a real nf-core pipeline rather than a synthetic one: 119 µs median for the whole gate process over 210 decisions, independently replayable from the published corpus.
The finding was not the latency. AWS's own Nextflow linter listed the hook the gate depends on among directives their managed service does not support, which would have put per-task governance out of reach. That file was two years stale and already contradicted by AWS's current documentation for a neighbouring directive. Treated as a hypothesis and probed against a local control: it runs. The gate now permits and refuses per task on the managed service.
Transferable principle: when an enforcement boundary matters, validate it experimentally rather than assume the documented boundary is the real one.
References
- Di Tommaso et al. (2017). Nextflow enables reproducible computational workflows. Nature Biotechnology 35(4), 316–319. doi:10.1038/nbt.3820
- Ewels et al. (2020). The nf-core framework for community-curated bioinformatics pipelines. Nature Biotechnology 38(3), 276–278. doi:10.1038/s41587-020-0439-x