bug: git sync + wmill sync push, flow got deleted when I didn't want to

[luc-vocab]

Describe the bug

I have a git-synced workspace, I work with claude code to develop scripts and git commit / git push them. One flow f/stripe/mark_vocab_paid_customers_flow got accidentally deleted when I didn't want it to. It is likely user error, but I would like to understand how to avoid repeating this mistake.

To reproduce

The reproduction is not yet clear but I am adding all the evidence to help figure out what happened.

• the workspace is synced over git
• I use claude code to operate on the workspace
• I regularly git push / git pull to make sure my local copy is in sync with the windmill workspace

Expected behavior

When using git synced workspaces and working on claude code, flows shouldn't be deleted by accident.

Screenshots

No response

Browser information

Chromium Version 146.0.7680.80 (Official Build) Built from source for Fedora release 43 (Forty Three) (64-bit)

Application version

EE v1.657.0

Additional Context

git commit comment

Git commit comment which contains the deletion of mark_vocab_paid_customers_flow:

[WM]: Deployed 1 settings, 1 flow
[WM] Workspace Settings 'auto_invite' deployed
[WM] Flow 'f/stripe/mark_vocab_paid_customers_flow' deleted

windmill git sync job

job run/019d0d80-1262-0ca4-a34f-f77ded2d0359

arguments:

{
    "group_by_folder": false,
    "use_individual_branch": false,
    "workspace_id": "vocab",
    "repo_url_resource_path": "u/luc/windmill_vocab_repository",
    "items": [
        {
            "path": "settings.yaml",
            "path_type": "settings",
            "commit_msg": "[WM] Workspace Settings 'auto_invite' deployed",
            "parent_path": ""
        },
        {
            "path": "f/stripe/mark_vocab_paid_customers_flow",
            "path_type": "flow",
            "commit_msg": "[WM] Flow 'f/stripe/mark_vocab_paid_customers_flow' deleted",
            "parent_path": "f/stripe/mark_vocab_paid_customers_flow"
        }
    ],
    "parent_workspace_id": null
}

log

Debounce key: vocab:vocab/script/hub/28160/sync-script-to-git-repo-windmill#args:"u/luc/windmill_vocab_repository":"vocab":false:false:null
Accumulating debounced argument `items`:
  original: [{"path": "f/stripe/mark_vocab_paid_customers_flow", "path_type": "flow", "commit_msg": "[WM] Flow 'f/stripe/mark_vocab_paid_customers_flow' deleted", "parent_path": "f/stripe/mark_vocab_paid_customers_flow"}]
  accumulated: [{"path": "settings.yaml", "path_type": "settings", "commit_msg": "[WM] Workspace Settings 'auto_invite' deployed", "parent_path": ""},{"path": "f/stripe/mark_vocab_paid_customers_flow", "path_type": "flow", "commit_msg": "[WM] Flow 'f/stripe/mark_vocab_paid_customers_flow' deleted", "parent_path": "f/stripe/mark_vocab_paid_customers_flow"}]

job=019d0d80-1262-0ca4-a34f-f77ded2d0359 tag=dependency worker=wk-default-8a2789daa2a7-X6xSc hostname=8a2789daa2a7 isolation=unshare

loaded from local cache: /tmp/windmill/cache/bun/276b7f71e4b1cdcafbf977c822546e2ac82925adebe310a1620bd8607403a9c9


--- BUN BUNDLE SNAPSHOT EXECUTION ---

Syncing settings settings.yaml with parent 
Syncing flow f/stripe/mark_vocab_paid_customers_flow with parent f/stripe/mark_vocab_paid_customers_flow
Running 'git clone --quiet --depth 1 *** windmill-vocab ...'
Command successfully executed
Running 'git config --global --add safe.directory /tmp/windmill/wk-default-8a2789daa2a7-X6xSc/019d0d80-1262-0ca4-a34f-f77ded2d0359/windmill-vocab ...'
Command successfully executed
Running 'git rev-parse --abbrev-ref HEAD ...'
main
Command successfully executed
Pushing to repository windmill-vocab in subfolder  on branch 
Pulling workspace into git repo
Running 'wmill sync pull --token *** --workspace vocab --base-url http://127.0.0.1:42069/ --repository u/luc/windmill_vocab_repository --yes --skip-secrets --include-settings --extra-includes settings.yaml.*,f/stripe/mark_vocab_paid_customers_flow.flow/*,f/stripe/mark_vocab_paid_customers_flow__flow/*,f/stripe/mark_vocab_paid_customers_flow.flow/*,f/stripe/mark_vocab_paid_customers_flow__flow/* ...'
�[34mUsing non-dotted paths (__flow, __app, __raw_app)�[39m
�[33m⚠️  WARNING: In a Git repository, the 'gitBranches' section is recommended in wmill.yaml.
   Consider adding a gitBranches section with configuration for your Git branches.
   Run 'wmill init' to recreate the configuration file with proper branch setup.�[39m
�[34m�[90mComputing the files to update locally to match remote (taking wmill.yaml into account)�[39m�[39m
�[34mremote (vocab) -> local: 1 changes to apply�[39m
�[34m�[31m- flow f/stripe/mark_vocab_paid_customers_flow__flow/flow.yaml�[90m�[31m�[39m�[39m
�[34m�[90mApplying changes to files ...�[39m�[39m
�[34mDeleting flow f/stripe/mark_vocab_paid_customers_flow__flow/flow.yaml�[39m
�[34mAll local changes pulled, now updating wmill-lock.yaml�[39m
�[34m�[4m�[32m�[1m
Done! All 1 changes applied locally and wmill-lock.yaml updated.�[22m�[39m�[24m�[39m
Command successfully executed
Running 'git config user.email anonymous@email.com ...'
Command successfully executed
Running 'git config user.name luc ...'
Command successfully executed
Running 'git add wmill-lock.yaml 'settings.yaml**' ...'
Command successfully executed
Running 'git add wmill-lock.yaml 'f/stripe/mark_vocab_paid_customers_flow**' ...'
Command successfully executed
Running 'git add wmill-lock.yaml 'f/stripe/mark_vocab_paid_customers_flow**' ...'
Command successfully executed
Running 'git diff --cached --quiet ...'
Running 'git commit --author "luc <anonymous@email.com>" -m "[WM]: Deployed 1 settings, 1 flow" -m "[WM] Workspace Settings 'auto_invite' deployed
[WM] Flow 'f/stripe/mark_vocab_paid_customers_flow' deleted" ...'
[main 880e287] [WM]: Deployed 1 settings, 1 flow
 1 file changed, 43 deletions(-)
 delete mode 100644 f/stripe/mark_vocab_paid_customers_flow__flow/flow.yaml
Command successfully executed
Running 'git push --porcelain ...'
To https://github.com/Vocab-Apps/windmill-vocab
 	refs/heads/main:refs/heads/main	8d470b7..880e287
Done
Command successfully executed
deleting gpg keys
Running 'git config --global --unset safe.directory /tmp/windmill/wk-default-8a2789daa2a7-X6xSc/019d0d80-1262-0ca4-a34f-f77ded2d0359/windmill-vocab ...'
Command successfully executed
Finished syncing

workspace sync configuration

settings.yaml

name: Vocab
ai_config:
  default_model:
    model: claude-sonnet-4-6
    provider: anthropic
  providers:
    anthropic:
      models:
        - claude-sonnet-4-6
      resource_path: u/luc/anthropic_openrouter_windmill_ipv6n
auto_invite:
  enabled: false
  mode: invite
  operator: false
color: '#6975dd'
git_sync:
  repositories:
    - exclude_types_override: []
      git_repo_resource_path: $res:u/luc/windmill_vocab_repository
      group_by_folder: false
      settings:
        include_path:
          - '**'
        include_type:
          - script
          - flow
          - app
          - folder
          - variable
          - resource
          - resourcetype
          - secret
          - schedule
          - trigger
          - user
          - group
          - settings
      use_individual_branch: false
operator_settings:
  audit_logs: true
  folders: true
  groups: true
  resources: true
  runs: true
  schedules: true
  triggers: true
  variables: true
  workers: true

wmill.yaml

defaultTs: bun
includes:
  - "**"
excludes: []
codebases: []
skipVariables: false
skipResources: false
skipResourceTypes: false
skipSecrets: false
skipScripts: false
skipFlows: false
skipApps: false
skipFolders: false
includeSchedules: true
includeTriggers: true
includeUsers: true
includeGroups: true
includeSettings: true
includeKey: false
skipWorkspaceDependencies: false
nonDottedPaths: true
gitBranches: {}

.github/workflows/push-on-merge.yaml

name: "Push main to Windmill workspace"
on:
  workflow_dispatch:
  push:
    branches:
      - "main"
    # if the windmill workspace is persisted in a subfolder of this repos, you can add the following to avoid pushing to windmill when there's no change
    # paths:
    #   - wm/**

env:
  WMILL_URL: https://windmill.ipv6n.net/
  WMILL_WORKSPACE: vocab

jobs:
  sync:
    environment: windmill
    runs-on: "ubuntu-latest"
    steps:
      - name: Checkout
        uses: actions/checkout@v3

      - uses: actions/setup-node@v4
        with:
          node-version: 20

      # We check the commit to make sure it doesn't start with [WM] which commits coming from Windmill Git Sync do.\
      # If that's the case, then we stop the workflow as we want to avoid overwriting changes that are out-of-sync
      # (for instance if one were to deploy in quick succession)
      - name: Check commit message
        id: check_message
        run: |
          COMMIT_MESSAGE="${{ github.event.head_commit.message }}"
          if [[ "$COMMIT_MESSAGE" =~ ^\[WM\] ]]; then
            echo "Commit message starts with '[WM]', skipping push to Windmill to avoid overwriting deploy that immediately follows it"
            echo "skip=skip" >> $GITHUB_OUTPUT
          fi

      # (push will pull first to detect conflicts and only push actual changes)
      - name: Push changes
        if: steps.check_message.outputs.skip != 'skip'
        run: |
          npm install -g windmill-cli@1.393.3
          wmill sync push --yes --skip-variables --skip-secrets --skip-resources --workspace ${{ env.WMILL_WORKSPACE }} --token ${{ secrets.WMILL_TOKEN }} --base-url ${{ env.WMILL_URL }}

[luc-vocab]

Here's the analysis from Claude Code on what might have happened. Seems I really need to pay attention to the state of the local git repo before making any changes.

Flow Deletion Incident Analysis

Issue: #8468
Flow: f/stripe/mark_vocab_paid_customers_flow

Root Cause

The flow was deleted from Windmill by the GitHub Action's wmill sync push, and then Windmill's git sync propagated that deletion back to git.

How wmill sync push deletes flows

In cli/src/commands/sync/sync.ts, the push command (line 2575-2612):

1. Downloads the entire Windmill workspace as a zip (the "remote" side)
2. Reads the local filesystem (the checked-out git directory, the "local" side)
3. Compares them via compareDynFSElement() — local is els1/m1, remote is els2/m2

At line 1582-1588, the deletion detection:

for (const [k] of Object.entries(m2)) {  // m2 = remote (Windmill workspace)
  if (m1[k] === undefined) {              // m1 = local (git checkout)
    changes.push({ name: "deleted", path: k });
  }
}

Any item that exists on Windmill but NOT in the local directory is marked for deletion. For flows, this results in a call to wmill.deleteFlowByPath() at line 3162-3167.

Sequence of events

1. The flow existed in both Windmill and git (previously synced).
2. The user pushed a commit to main where the flow file (f/stripe/mark_vocab_paid_customers_flow__flow/flow.yaml) was not present in the checkout.
3. The GitHub Action (push-on-merge.yaml) triggered and ran:

   wmill sync push --yes --skip-variables --skip-secrets --skip-resources
   

4. wmill sync push downloaded the workspace zip (flow exists) and compared with the local checkout (flow file missing) → deleted the flow from Windmill via API.
5. Windmill detected the deletion and queued a git sync event (the "original" debounced item in the logs).
6. The auto_invite settings change was accumulated into the same debounced batch.
7. Windmill's git sync job ran wmill sync pull, saw the flow was deleted on Windmill, removed it from git, and committed:

   [WM] Flow 'f/stripe/mark_vocab_paid_customers_flow' deleted
   

Why the flow file was missing from the checkout

The most likely cause: the user's local git repo was missing the flow file when they pushed. This can happen with a bidirectional sync setup when:

• Race condition: The flow was created/modified in Windmill, and Windmill's git sync pushed it to git (a [WM] commit). But the user pushed their own commit before pulling that [WM] commit. If git allowed the push (e.g., through force push, or the [WM] commit hadn't landed yet), the Action checkout wouldn't have the flow file.
• Claude Code deleted the file: Since Claude Code operates on the local filesystem, it may have accidentally removed or moved the flow file as part of another operation, and the user committed without noticing.
• Git operation dropped the file: A rebase, merge conflict resolution, or git checkout could have accidentally removed it.

The fundamental issue

The push-on-merge.yaml Action runs wmill sync push with --yes (no confirmation) and no --skip-flows flag. This means wmill sync push performs a full reconciliation — anything on Windmill that's not in the local git directory gets deleted. With a bidirectional sync setup (Windmill→git AND git→Windmill), this creates a window where items can be accidentally deleted if the git repo and Windmill workspace are even briefly out of sync.

How to prevent this

1. Add --skip-flows or use --includes in the GitHub Action to limit what wmill sync push can delete, only pushing the types of items you manage through git.
2. Always git pull before git push to ensure your local copy includes all [WM] commits.
3. Consider using a one-directional sync (either Windmill→git OR git→Windmill, not both) to avoid these race conditions.