Github Workflow Required check never resolving - "Expected - Waiting for status to be reported"

[AmyJoT]

Select Topic Area

Bug

Body

When opening PRs we are sometimes seeing this situation occur:

1. PR is raised, kicking off the workflow
2. A second workflow is kicked off, and initial workflow is immediately cancelled due to concurrency rules:
   Canceling since a higher priority waiting request for 'Flight Checks-refs/pull/142836/merge' exists
3. Second workflow completes, but status check stays like this:

Triggers and concurrency are set up as follows:

# Will run on the following PR events:
on:
  pull_request:
    types:
    - opened
    - edited
    - reopened
    - synchronize
    - labeled
    - unlabeled

# Cancel other in progress runs of this workflow 
concurrency:
  group: ${{ github.workflow }}-${{ github.ref }}
  cancel-in-progress: true

Any ideas why this is happening? It feels like some sort of race condition is being hit.

Note: this doesn't always happen, and this occured on an automated Renovate PR.

[sayhiben]

My organization is also encountering this issue. We made no recent relevant changes, and it seems to manifest across required checks without an obvious pattern. At a guess, there must be some issue with the check status webhooks not being received on GitHub's side.

[sayhiben]

We have at least one case reoccurring this evening. I'm unsure if this is resolved.

[AmyJoT]

Still occuring multiple times over here, so not resolved

[AmyJoT]

We're having more engineers reach out to us about seeing this issue appear on their PRs.

[jessebrennan]

Also seeing this a lot more recently. It seems to occur when pushing to the same branch multiple times in quick succession. I suspect setting cancel-in-progress: false may be a sufficient work around if you can afford to do so.

[lrotschy]

Hi folks, thanks for reporting this. Are you still seeing this issue? And could you clarify if it is happening with required status checks or with ruleset workflows, or both? Are you seeing it on other checks that are not required?

[AmyJoT]

This is still happening as far as I am aware yes, though I haven't had as many reports this week (though it is christmas, so maybe less folk to encounter it). It is happening in a workflow which runs on PRs which is also a required check (apologies if this is what a ruleset workflow is). I'm pretty sure there is something race-condition wise going on with the concurrency ruleset we need to use for this workflow.

The following is the main idea of the setup in which we are seeing this issue:

name: Flight Checks

# Will run on the following PR events:
on:
  pull_request:
    types:
      - opened
      - edited
      - reopened
      - synchronize
      - labeled
      - unlabeled

# Cancel other in progress runs of this workflow
concurrency:
  group: ${{ github.workflow }}-${{ github.ref }}
  cancel-in-progress: true

# Access scopes for the GITHUB_TOKEN created as part of the workflow
permissions:
  contents: read
  pull-requests: write
  repository-projects: read

jobs:
  flight-checks:
... etc

[lrotschy]

I think it would be worth opening a customer support issue for this, especially since others are seeing it as well. https://support.github.com/features/actions

[dcacenabes]

Hi @lrotschy we are also seeing the issue. I reported it already 3 weeks ago:
https://support.github.com/ticket/enterprise/36/2492908
But still no solution from Github.

I even created a sample project that demonstrates the issue very easily:
https://github.com/dcacenabes/demonstrate-event-trigger-issues/

It basically contains a script that creates N (defined via parameter) pull requests in a loop. Eventually some of those hit the issue (e.g.
1 and 2 )

It does not happen for every PR, but it is definitely easy to reproduce.

[lrotschy]

Not sure what's going on with the ticket. I'll check with support and see if we can track it down. My apologies for the delay.

[dcacenabes]

Thanks @lrotschy !
Please let me know if you need further info regarding steps to reproduce.
This is a very annoying bug because the current workaround is costing us money (triggering new workflow runs means more minutes for runners)

[lalten]

Hi, we're hitting this multiple times per day and it's becoming very annoying. Is there any solution/workaround?

[broksonic21]

We also have seen this a ton starting 2 days ago - anyone from Github able to fill us in if there's a solution? We use the exact same action setup as the Original Poster.

[dcacenabes]

The reply from Github was that this is expected behavior. This is from our ticket with support:

Our engineering team has shared the root cause of this issue. In essence, status checks will look up the status of the latest run and as cancel-in-progress is true, the latest runs will sometimes be cancelled and as a result they will not be reported. When using concurrency, we do not guarantee the order in which the runs are cancelled as outlined in the note in our documentation here. The current workaround would be to set cancel-in-progress to false.

[broksonic21]

That makes cancel-in-progress useless, to be honest and would cost us a large amount extra per month in $ since significant # of extra runs would run.

That's very unfortunate. @lrotschy please put us down as well as really wanting a fix for this.