Issues with GitHub REST API for Secret Scanning Alerts

[nmarkana-crest]

Select Topic Area

Bug

Body

REST APIs:

1. List Secret Scanning Alerts for an Enterprise
2. List Secret Scanning Alerts for an Organization
   

Bugs/Issues:

1. We have a use case to collect secret scanning alerts at particular intervals without duplication. We are interested in receiving the secret scanning alerts in ascending order of the updated field. To avoid data duplication, we are keeping track of the “after” or “page” parameter to fetch alerts. However, we have noticed that whenever we update any alert collected in previous page, in the subsequent pages, the updated alert is received but the some other alerts are lost. This results in data loss for already present alerts, which ideally should not occur. In the REST API for Code Scanning alerts (link), the API itself handles fetching all alerts, including the updated ones, in the upcoming pages.

Reproduction Steps:

Let us assume we have 5 alerts (1,2,3,4,5 - all in ascending order of update date) in our GitHub enterprise

Keeping the per_page to “2” and “sort” ascending on updated date

Expected outcome when no event is updated in between :
1st page - 1,2
2nd page - 3,4
3 page - 5

Expected outcome when any event is updated in between :
1st page - 1,2
Updating the alert 2 from GitHub
2nd page - 3,4
3 page - 5, 2

Actual outcome when any event is updated in between :
1st page - 1,2
Updating the alert 2 from GitHub
2nd page - 4,5
3 page - 2 

In actual outcome, we are not receiving the 3rd event and then that event is lost.

You can perform the same reproduction steps with the code scanning alerts API (link) and verify that we didn’t experience any loss of alerts while updating the previously fetched alerts.

2. In the REST API documentation (link), it is mentioned that there is support for “before” and “after” parameters. However, upon checking the actual response header, there are no URLs with “after” or “before” parameters. Only URLs with the “page” parameters are present.

Could someone please prioritise looking into this issue as we are currently blocked on it?

[nmarkana-crest]

Do we have any updates on this?

[HyHamza]

Yes

[nmarkana-crest]

@HyByte , if possible can you please share those updates/progress here?

[HyHamza]

Nope. sorry

[Jovierr]

Switch to a timestamp-based approach instead of using page parameters. Track the latest updated timestamp you processed and use that to fetch new alerts. This way, you only get new or updated alerts since your last check, avoiding duplicates and missed alerts.