replicate.npmjs.com package metadata is not synchronized with registry.npmjs.org

[ALyalin22]

Hello!

I am writing a follower for replicate.npmjs.com/_changes feed and I have noticed that not all of the changes of package metadata are present in this feed. For example, almost month ago version "7.0.0" of package patch-package was published, but replicate.npmjs.com is still responding with previous version of package meta where "6.5.1" is latest version.
https://registry.npmjs.org/patch-package
https://replicate.npmjs.com/patch-package

Moreover, there is no change sequence containing "7.0.0" version in changes feed. https://replicate.npmjs.com/registry/_changes?include_docs=true&filter=_doc_ids&doc_ids=[%22patch-package%22]

Is it supposed to work like this or its a bug?

[arcanis]

I suspect this might be why the Algolia indexes are off, thus making the information on yarnpkg.com outdated. For example, the last version of Vite we report on the Yarn website is 4.2.1 (from two months ago!), whereas the actual last one is 4.3.8 (note that this only affects the website - the registry itself is fine, since it's a cname to npm).

[sharadraju]

The yarn website stills show outdated information on the oracledb package. oracledb is currently at 6.0.3, whereas yarn shows the oracledb version as 5.5.
Please correct this.

[monishcm]

This should be resolved. Please let us know if you still face a lag with replciation.

[arcanis]

The 7.0.0 still doesn't appear in the patch-package stream:
https://replicate.npmjs.com/registry/_changes?include_docs=true&filter=_doc_ids&doc_ids=%5B%22patch-package%22%5D

[arcanis]

@monishcm This breaks a very important feature of the registry, as without it npm doesn't allow mirrors (only proxies), making it a single point of failure (no way to build a complete copy of the registry in case it goes down). Did you identify the change that caused the issue? It seems incident-worthy to me.

[nedl86]

I'm also still seeing old metadata for @newrelic/browser-agent

[ALyalin22]

New versions of patch-package finally appeared in replicate.npmjs.com/_changes feed, but it only happened after new version 8.0.0-canary.0 was released a day ago, which caused a new event to appear in the feed. This seems like it was not a replication lag, but a bug due to which the original event was lost in the feed. While i was working with this changes feed, i noticed a lot of packages with similar issues, so I'm sure the problem is still actual and I can't resolve this discussion.

[MartinKolarik]

@monishcm I'm also observing a problem with deletions not propagating. E.g.:

This package appears to have been removed from the registry:
https://registry.npmjs.org/@abrticketing/common

But it's still returned by:
https://replicate.npmjs.com/registry/%40abrticketing%2Fcommon
https://replicate.npmjs.com/registry/_changes?include_docs=true&filter=_doc_ids&doc_ids=%5B%22%40abrticketing%2Fcommon%22%5D

There are thousands of packages like this.

[SsomsakTH]

Hello!

I am writing a follower for replicate.npmjs.com/_changes feed and I have noticed that not all of the changes of package metadata are present in this feed. For example, almost month ago version "7.0.0" of package patch-package was published, but replicate.npmjs.com is still responding with previous version of package meta where "6.5.1" is latest version.
https://registry.npmjs.org/patch-package
https://replicate.npmjs.com/patch-package

Moreover, there is no change sequence containing "7.0.0" version in changes feed. https://replicate.npmjs.com/registry/_changes?include_docs=true&filter=_doc_ids&doc_ids=[%22patch-package%22]

Is it supposed to work like this or its a bug?

[SsomsakTH]

#Npm

[SsomsakTH]

Hello!

I am writing a follower for replicate.npmjs.com/_changes feed and I have noticed that not all of the changes of package metadata are present in this feed. For example, almost month ago version "7.0.0" of package patch-package was published, but replicate.npmjs.com is still responding with previous version of package meta where "6.5.1" is latest version.
https://registry.npmjs.org/patch-package
https://replicate.npmjs.com/patch-package

Moreover, there is no change sequence containing "7.0.0" version in changes feed. https://replicate.npmjs.com/registry/_changes?include_docs=true&filter=_doc_ids&doc_ids=[%22patch-package%22]

Is it supposed to work like this or its a bug?

[SsomsakTH]

Hello!

I am writing a follower for replicate.npmjs.com/_changes feed and I have noticed that not all of the changes of package metadata are present in this feed. For example, almost month ago version "7.0.0" of package patch-package was published, but replicate.npmjs.com is still responding with previous version of package meta where "6.5.1" is latest version.
https://registry.npmjs.org/patch-package
https://replicate.npmjs.com/patch-package

Moreover, there is no change sequence containing "7.0.0" version in changes feed. https://replicate.npmjs.com/registry/_changes?include_docs=true&filter=_doc_ids&doc_ids=[%22patch-package%22]

Is it supposed to work like this or its a bug?

[codergautam]

any update?

[quinton-ashley]

Can you guys do your jobs? @ npm

[dyizgngjzhkfhkdhkd]

Hello!

I am writing a follower for replicate.npmjs.com/_changes feed and I have noticed that not all of the changes of package metadata are present in this feed. For example, almost month ago version "7.0.0" of package patch-package was published, but replicate.npmjs.com is still responding with previous version of package meta where "6.5.1" is latest version. https://registry.npmjs.org/patch-package https://replicate.npmjs.com/patch-package

Moreover, there is no change sequence containing "7.0.0" version in changes feed. https://replicate.npmjs.com/registry/_changes?include_docs=true&filter=_doc_ids&doc_ids=[%22patch-package%22]

Is it supposed to work like this or its a bug?

[quinton-ashley]

@monishcm The issue was not resolved. Can you fix this problem? It's been 4 months.

[smulikHakipod]

Could this be related?
I am trying to use replicate to clone some repo, and getting for example "yallist-2.1.2.tgz' is not in this registry." while the regular registry works just fine.

When manually trying to see the 404 error, I get

Document is missing attachment which probably indicate some replication issue on NPM side.

 % npm i                                                        
npm WARN old lockfile 
npm WARN old lockfile The package-lock.json file was created with an old version of npm,
npm WARN old lockfile so supplemental metadata must be fetched from the registry.
npm WARN old lockfile 
npm WARN old lockfile This is a one-time fix-up, please be patient...
npm WARN old lockfile 
npm ERR! code E404
npm ERR! 404 Not Found - GET https://replicate.npmjs.com/registry/yallist/-/yallist-2.1.2.tgz - not_found
npm ERR! 404 
npm ERR! 404  'yallist@https://replicate.npmjs.com/registry/yallist/-/yallist-2.1.2.tgz' is not in this registry.
npm ERR! 404 
npm ERR! 404 Note that you can also install from a
npm ERR! 404 tarball, folder, http url, or git url.

[Giangpro89]

Thankyou

[dp0]

I've been seeing this issue intermittently for at least a few months now, and so I wanted to highlight that this isn't a one-off matter, but there seems to be a repeating issue occurring here. I'm really keen to try and help as much as I can, and to that end, I think it'd be helpful to show you exactly how the problem manifests as this might help with diagnosis:

A recent example was [email protected] being missing from the replicate feed days after publication. Looking at the replicate endpoint, two document revisions for zod-validation-error are relevant to this issue: 18-9fa86a3afc8f4779639a58e9f01d2015, and 19-05ead07d9cae87fabe55634692cc0acf.

In 18-9fa86a3afc8f4779639a58e9f01d2015, latest (and the newest version) was [email protected], published at 2023-08-17T06:48:42.759Z. [email protected] was not present/referenced at all in this revision of the document.

In 19-05ead07d9cae87fabe55634692cc0acf, latest was noted to be [email protected], published at 2023-10-20T07:17:46.780Z. But, [email protected] also (first) appeared in this revision, noting that it was published at 2023-08-18T10:02:02.559Z.

The surprise here is that two versions, despite being published on different days, first appeared in revision 19-05ead07d9cae87fabe55634692cc0acf. That is, a revision (for [email protected]'s publication) was seemingly missing.

As this is an ongoing issue, if there is any other information that you might find useful to diagnose/remediate this, I'd be happy to provide that, either in this specific case, or in any future cases.

[MylesBorins]

Thanks for sharing, I've surfaced to the eng team looking into this

[dp0]

I'm a little unsure if this is directly related to the above issue, but there's an incident right now where the update_seq value noted in the replicate endpoint appears to be stuck at 27723123, and has been for several hours. As a result, no updates are getting through to this endpoint. For completeness, we've opened ticket 2449580 with npm support to highlight this. If we can provide any further information or take any action to help with this issue or the general stability of this endpoint, please just let me know!