Skip to content

Add nextflow module generate-meta command#6859

Open
jorgee wants to merge 109 commits intomasterfrom
260220-generate-meta-yml
Open

Add nextflow module generate-meta command#6859
jorgee wants to merge 109 commits intomasterfrom
260220-generate-meta-yml

Conversation

@jorgee
Copy link
Contributor

@jorgee jorgee commented Feb 24, 2026

Summary

  • Adds nextflow module generate-meta CLI subcommand that parses a module's main.nf and generates a meta.yml template pre-populated with all inferable values
  • Supports both V1 (classic qualifier syntax: val, path, tuple) and V2 (static type syntax: reads: Path, (meta, reads): Tuple<String, Path>)
  • Implements two-pass static analysis for V2 output type inference: builds a symbol map from input parameters and exec block assignments, then resolves output types via declared type → symbol map lookup → RHS expression inference chain
  • Adds -force flag to overwrite an existing meta.yml and -dry-run flag to print to stdout without writing

Changes

  • CmdModule.groovy — registers the new generate-meta subcommand
  • ModuleGenerateMeta.groovy — CLI entry point with path resolution, overwrite guard, dry-run routing
  • MetaYmlGenerator.groovy — AST extraction (V1 + V2) and YAML rendering via SnakeYAML
  • MetaYmlGeneratorTest.groovy — 14 Spock unit tests covering V1 qualifiers, V2 typed inputs, symbol map inference, tuple sub-entries, YAML rendering
  • ModuleGenerateMetaTest.groovy — 10 Spock CLI tests covering generation, overwrite guard, dry-run, error cases
  • VariableScopeVisitor.java — minor related fix

Test plan

  • Run ./gradlew :nextflow:test --tests "nextflow.module.MetaYmlGeneratorTest" — all 14 tests pass
  • Run ./gradlew :nextflow:test --tests "nextflow.cli.module.ModuleGenerateMetaTest" — CLI tests pass
  • Manual: nextflow module generate-meta <module-dir> generates a valid meta.yml
  • Manual: re-running without -force errors; with -force regenerates; -dry-run prints to stdout

🤖 Generated with Claude Code

pditommaso and others added 30 commits November 17, 2025 15:09
Introduce comprehensive development constitution documenting core principles
and practices for Nextflow development including modular architecture,
test-driven quality assurance, dataflow programming model, licensing
compliance, DCO requirements, semantic versioning, and Groovy code standards.

The constitution codifies existing best practices from CLAUDE.md and
CONTRIBUTING.md to provide clear governance and quality standards for
the project.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com>
Signed-off-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com>
Signed-off-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com>
Signed-off-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com>
Signed-off-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com>
- Add comprehensive documentation for all module CLI commands
- Add `nextflow module run` command for standalone module execution
- Remove `module update` command to simplify the design
- Use single-dash prefix for Nextflow options, double-dash for module inputs
- Remove @ prefix from scope in CLI commands (keep only in DSL syntax)

Signed-off-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com>
- Remove separate `dependencies` and `components` fields
- Expand `requires` to include:
  - `nextflow`: version constraint (unchanged)
  - `plugins`: array with name@constraint syntax
  - `modules`: array of module dependencies
  - `workflows`: array of workflow dependencies
- Unified version constraint syntax: `[scope/]name[@constraint]`
- Mark `components` as deprecated (use requires.modules)
- Update all examples in ADR and schema

Signed-off-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com>

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Signed-off-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com>
- Add `args` property to tools section for type-safe argument configuration
- Define toolArgSpec with flag, type, description, default, enum, required
- Support implicit variable `tools.<toolname>.args.<argname>` returning
  formatted flag+value (e.g., "-K 100000000")
- Support `tools.<toolname>.args` to return all args concatenated
- Document deprecation of ext.args/ext.args2/ext.args3 pattern
- Update ADR with Tool Arguments Configuration section and appendix

[ci skip]

Signed-off-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com>
Co-authored-by: Jorge Ejarque <jorgee@users.noreply.github.com>
Signed-off-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com>
Co-authored-by: Jorge Ejarque <jorgee@users.noreply.github.com>
Signed-off-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com>
Co-authored-by: Jorge Ejarque <jorgee@users.noreply.github.com>
Signed-off-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com>
Expand deprecation notice to cover all ext.* custom directives

Signed-off-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com>
- Use consistent module path format with version: modules/@scope/name@version/
- Fix directory structure example: samtools-view -> samtools/view
- Standardize on 'license' spelling (American English)
- Fix author -> authors (plural array format)

Signed-off-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com>
Remove @ prefix from scope in API path to match API definition

Signed-off-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com>
Signed-off-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com>
- Update API endpoints to match seqeralabs/plugin-registry#266
- Use /api/modules base path (no v1 prefix)
- Use single {name} parameter with namespace (e.g., "nf-core/fastqc")
- Add separate /releases endpoint
- Simplify publish to single POST endpoint

Signed-off-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com>
Signed-off-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com>
- Remove checksums from nextflow.config (registry is source of truth)
- Use single version per module locally (no version in directory path)
- Add .checksum file in module directory for integrity verification
- Simplify freeze command to only pin transitive dependency versions
- Checksum mismatch reports warning instead of automatic re-download
- Remove resolved open question about checksum verification

Signed-off-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com>
Co-authored-by: Jorge Ejarque <jorgee@users.noreply.github.com>
Signed-off-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com>
…tions [ci skip]

- Added Resolution Rules table clearly specifying behavior for each scenario:
  - Version change (local unmodified): automatically replace with declared version
  - Local modification (checksum mismatch): warn and protect local changes
  - Use -force flag to override locally modified modules
- Updated install command behavior to reflect checksum verification
- Updated Technical Details with expanded resolution flow

Signed-off-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com>
…skip]

- Fixed Resolution Rules table: locally modified modules will NOT be
  replaced unless -force is used (was incorrectly saying "will be replaced")
- Added Version 2.3 changelog entry documenting:
  - Resolution Rules table with clear behavior matrix
  - Local modification protection with -force flag
  - Simplified storage model (single version per module)
  - .checksum file for fast integrity verification

Signed-off-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com>
Signed-off-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com>
Signed-off-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com>
Specification for Nextflow module system client implementation based on
ADR 20251114-module-system.md. Covers:

P1 (Core):
- Install and use registry modules via @scope/name syntax
- Run modules directly from CLI without wrapper workflow
- Structured tool arguments replacing ext.args pattern

P2 (Important):
- Module version management and freeze command
- Module integrity protection with checksum validation

P3 (Nice to have):
- Remove module command
- Search and discover modules
- Publish module to registry

Registry backend is out of scope (assumed implemented).

Signed-off-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com>
- Remove `freeze` command from CLI
- Remove transitive dependency install behavior
- Remove orphaned transitive dependency removal from `remove` command
- Update rationale and consequences sections
- Simplify dependency resolution flow
- Update ADR to version 2.4

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Signed-off-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com>
Co-authored-by: Jorge Ejarque <jorgee@users.noreply.github.com>
Signed-off-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com>
Co-authored-by: Phil Ewels <phil.ewels@seqera.io>
Signed-off-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com>
Signed-off-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com>
bentsherman and others added 6 commits March 3, 2026 11:09
* Add NVMe disk allocation and diskMountPath support for Seqera executor

For task/node allocation the default is /tmp; for nvme allocation
the default is the server-configured NVMe mount path (typically /scratch).


Signed-off-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
… and add -scope

Signed-off-by: jorgee <jorge.ejarque@seqera.io>
@jorgee
Copy link
Contributor Author

jorgee commented Mar 4, 2026

Update:

  • Rename generate-meta subcommand by spec
  • Allow module reference as primary argument and add -scope when fullname is not possible
  • Remove TODOS in module fields. Still in input/output descriptions or when no type could be inferred.
  • Input/Output validation in ModuleSpec to avoid pushing arguments with TODOs
  • Some refactor: ModuleGenerataMeta renamed to CmdModuleSpec and MetaYmlGenerator integrated in ModuleSpec

jorgee and others added 3 commits March 4, 2026 16:20
…m [ci skip]

Signed-off-by: jorgee <jorge.ejarque@seqera.io>
Signed-off-by: jorgee <jorge.ejarque@seqera.io>
@pditommaso
Copy link
Member

Nit, I'd likely call this create-meta there's a more canonical term for file creation (I understand that now "generate" feels trendy ..)

@pditommaso
Copy link
Member

Code Review Findings

Critical (Must Fix)

1. Bug: authors guarded by license check in asMap()
ModuleSpec.groovy:293-294 — checks this.license instead of this.authors:

// CURRENT (wrong)
if( this.license )
    manifest['authors'] = this.authors

// FIX
if( this.authors )
    manifest['authors'] = this.authors

Authors will be silently dropped when license is null, or emit null when license is set but authors isn't.

2. Potential NPE in extractMethodCallType
ModuleSpec.groovy:776methodTarget can be null for unresolved method calls:

// CURRENT (missing null-safe on methodTarget)
return resolveTypeFromDeclaredType(simpleTypeName(methodExpr.methodTarget.returnType?.name))

// FIX
return resolveTypeFromDeclaredType(simpleTypeName(methodExpr.methodTarget?.returnType?.name))

Important (Should Fix)

# Issue Location
3 YAML header still says generate-meta instead of spec ModuleSpec.groovy:58
4 moduleName field lacks type declaration in @CompileStatic class CmdModuleSpec.groovy:71
5 validate() checks type.equals(DESCRIPTION_TODO) — should be description.equals(DESCRIPTION_TODO) ModuleSpec.groovy:87
6 resolveAsPath uses args[0] instead of module param CmdModuleSpec.groovy:129

Suggestions

  • buildParamSpec always emits DESCRIPTION_TODO even if entry.description is already populated (line 323)
  • load() doesn't close the InputStream from Files.newInputStream (line 141)
  • Log messages reference $args (full list) instead of $module (lines 117, 119, 123)
  • No test covering the authors field rendering in asMap() — would have caught bug A process that define only an input of type 'each' never stops #1

🤖 Generated with Claude Code

jorgee and others added 8 commits March 5, 2026 14:26
Signed-off-by: jorgee <jorge.ejarque@seqera.io>
Signed-off-by: jorgee <jorge.ejarque@seqera.io>
Signed-off-by: Ben Sherman <bentshermann@gmail.com>
…te-meta-yml

Signed-off-by: Jorge Ejarque <jorgee@users.noreply.github.com>
Signed-off-by: jorgee <jorge.ejarque@seqera.io>
Signed-off-by: jorgee <jorge.ejarque@seqera.io>
@jorgee
Copy link
Contributor Author

jorgee commented Mar 6, 2026

Nit, I'd likely call this create-meta there's a more canonical term for file creation (I understand that now "generate" feels trendy ..)

@ben suggested to call it 'spec'. This is the name in the current implementation. 'create-meta', 'create-spec' are also good alternatives to me.

I fixed other issues and comments. Apart from the name, it is ready for review from my side

@jorgee jorgee marked this pull request as ready for review March 6, 2026 19:48
@jorgee jorgee requested a review from a team as a code owner March 6, 2026 19:48
Base automatically changed from 251117-module-system-implementation to 251117-module-system March 10, 2026 13:22
Base automatically changed from 251117-module-system to master March 17, 2026 09:34
Signed-off-by: jorgee <jorge.ejarque@seqera.io>
@netlify
Copy link

netlify bot commented Mar 26, 2026

Deploy Preview for nextflow-docs-staging canceled.

Name Link
🔨 Latest commit 42bce58
🔍 Latest deploy log https://app.netlify.com/projects/nextflow-docs-staging/deploys/69c55f4c4bf26d00081ca59e

jorgee added 2 commits March 26, 2026 11:05
Signed-off-by: jorgee <jorge.ejarque@seqera.io>
Signed-off-by: jorgee <jorge.ejarque@seqera.io>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.