Allow workflow_run to wait for all workflows it depends on

[danielalves]

Hello!
I've been using Github Actions in my personal projects with great success, so kudos for the awesome tool 🙌

One feature that my colleagues and I have been missing, though, is to be able to make workflow_run wait for all the other workflows it depends on instead of resolving on the first completed one. Something like this would be awesome:

# Current / default behavior
on:
  workflow_run:
    workflows: [dependency-0, dependency-1, dependency-n]
    strategy: resolve-on-first
    types:
      - completed 

# Desired / additional behavior
on:
  workflow_run:
    workflows: [dependency-0, dependency-1, dependency-n]
    strategy: wait-all
    timeout: 5m
    types:
      - completed 

The reasoning behind this feature request is that we would like to be able to keep the nice UI you built for separate workflows while being able to create dependencies between them in a more versatile way (eg. making sure a given job is only executed once in the entire build pipeline). Currently, the only way to achieve the same thing (afaik) is to make the "parent" workflow call all the "child"/"dependency" workflows using workflow_call and put all of them in a needs directive. However that generates a worse UI since, at the end, workflow_call ends up embedding the called workflow's steps into the caller workflow. Thus we have to scan through a lot jobs to understand what failed, whereas in the separate workflows case the failure reports are spot-on.

Maybe an example would be better. Imagine a monorepo-like structure, where the repo's root contains many sub-projects. Also sitting on the root, there's a infra folder, containing code responsible for spinning up infrastructure somewhere (eg. terraform + AWS). Finally, the .github folder contains workflows for each separate project:

- root/
  - .github/
    - workflows/
      - infra.yml
      - project-1.yml
      - project-2.yml
      - project-n.yml
  - infra/
  - project-1 /
  - project-2 /
  - project-n /

The idea is to create this flow:

• Start all builds -> when they all pass, start infrastructure update -> start all deploys

Then a project-x.yaml could look like this:

name: project-x

on:
  workflow_run:
    workflows: [infra]
    types:
      - completed

  push:
    branches: [main]
    paths:
      - 'infra/**'
      - 'project-x/**'
      - '.github/workflows/infra.yml'
      - '.github/workflows/project-x.yml'

jobs:
  build:
    runs-on: ubuntu-latest
    if: ${{ github.event_name == 'push' && github.event.ref == 'refs/heads/main' }}
    steps:
      - uses: actions/checkout@v2
      - run: build

  deploy:
    runs-on: ubuntu-latest
    if: ${{ github.event_name == 'workflow_run' && github.event.workflow_run.conclusion == 'success' }}

    steps:
      - uses: actions/checkout@v2
      - run: deploy

And infra.yml could look like:

name: infra

on:
  workflow_run:
    workflows: [project-1, project-2, project-n]
    strategy: wait-all
    timeout: 5m
    branches: [main]
    types:
      - completed

jobs:
  infra-update:
    runs-on: ubuntu-latest

    steps:
      - uses: actions/checkout@v2
      - run: infra-update

Thanks in advance.

[danielalves]

Just to complete my example above, this is the way to achieve an equivalent flow today, which I must say results in much more code:

# A sub-project workflow
name: project-x

on:
  workflow_call:
    inputs:
      job:
        required: true
        type: string

jobs:
  build:
    runs-on: ubuntu-latest
    if: ${{ inputs.job == 'build' }}
    steps:
      - uses: actions/checkout@v2
      - run: build

  deploy:
    runs-on: ubuntu-latest
    if: ${{ inputs.job == 'deploy' }}
    concurrency: project-x-main
    steps:
      - uses: actions/checkout@v2
      - run: deploy


# Infrastructure workflow that orchestrates everything
name: infra

on:
  push:
    branches: [main]

jobs:
  run-project-1-build:
    name: Project 1
    uses: ./.github/workflows/project-1.yml
    with:
      job: build
    secrets: inherit

  run-project-2-build:
    name: Project 2
    uses: ./.github/workflows/project-2.yml
    with:
      job: build
    secrets: inherit

  run-project-n-build:
    name: Project N
    uses: ./.github/workflows/project-n.yml
    with:
      job: build
    secrets: inherit

  infra-check:
    runs-on: ubuntu-latest
    needs:
      - run-project-1-build
      - run-project-2-build
      - run-project-n-build
    steps:
      - uses: actions/checkout@v2
      - run: infra-check

  infra-update:
    runs-on: ubuntu-latest
    needs: infra-check
    concurrency: infra-main
    steps:
      - uses: actions/checkout@v2
      - run: infra-update

  run-project-1-deploy:
    needs: infra-update
    name: Project 1
    uses: ./.github/workflows/project-1.yml
    with:
      job: deploy
    secrets: inherit

  run-project-2-deploy:
    needs: infra-update
    name: Project 2
    uses: ./.github/workflows/project-2.yml
    with:
      job: deploy
    secrets: inherit

  run-project-n-deploy:
    needs: infra-update
    name: Project N
    uses: ./.github/workflows/project-n.yml
    with:
      job: deploy
    secrets: inherit

This means that, for each new sub-project in the mono repo, we have to add a call to build and a call to deploy, which is very error prone.

And the resulting UI would contain many skips:

[miparnisari]

This conceptual inconsistency is annoying.

If you specify multiple workflow dependencies, only one needs to run.

If you specify multiple job dependencies, all of them have to run.

[AnatolyBuga]

@miparnisari ,needs requires that there is at least one of the jobs not dependent on anything. Which is not always the case. Is there an option to wait for all jobs in workflows to be completed?

[divmgl]

Looking for this feature too. We really need a deployment workflow to happen only after both frontend AND server test workflows have finished.

[chudilka1]

Try out this publicly accessible "wait for workflows" action from Chainlink that waits either for all workflows or only for the specified ones and then run your job.

[dmwtf]

Unfortunately is the "wait for workflows" action only a workaround for us. We also need to deploy code when and only when all tests and build steps are done, but we still have some flakyness in the process and may need to restart some jobs/workflows. So we can not just wait a specific time, like in "wait for workflows" action. We need something, what will trigger the workflow, when all conditions will be met, independent when it will happen.

[richard-powers]

This is a major thorn in my side at the moment, it would be very helpful to be able to wait for many workflows.
Is this being considered, or looked at, by anyone?