npm scores

[bnb]

historically, we've seen a lot of "package score" implementations. To name a few,

• npms.io, which is used by npm to surface the three metrics on npmjs.com right now.
• NodeSource Certified Modules (NCM), a product targeted at enterprises which included various quality signals giving a single number score to each module and allowing you to triage critical issues. Now deceased, from what I understand.
  • npmsearch, which is powered by the NCM data.
• packagephobia/bundlephobia, which are both focused on size.
• Skypack, which just launched something today.
• Various security tools (Snyk, audit, White Source, etc.) that all give a "score" specific to security.
• Lighthouse shipped [a tool] scoped to a few "old" packages that have solid alternatives.

Over the years, there's been a bunch of attempts to quantify the quality of an npm package. I can't imagine we're ever going to see folks stop building tools that do this, since the goalposts will always be moving.

Instead of building out a first-party competitor to these tools, I'd like to propose that npm aggregate the scores and surface them to end-users. There's a variety of ways such a mechanism could be accomplished, so I don't want to be prescriptive, but the collective input would both allow folks to have their own platform without having to deal with all the basic metrics (rewriting these over and over and over again), enabling more focused and niche scoring mechanisms to arise.

I've put this in General since I'm not sure what the ideal interface for this would be - registry, website, cli, independent API, or all of the above. Open to any implementation that could be easily accessed by users.

[ljharb]

Many of these scores are quite opinionated tho, and are not universally agreed upon, nor are the priorities of each criteria universally agreed upon. It seems like a very dangerous path to me for npm to "bless" existing scores, and thus implicitly bless their priorities and criteria.

[bnb]

@mcollina I mean, there are certain other things that scoring tools measure too. For example, Skypack includes ESM exports, export maps, license, keywords, README, repository URL, and types in their score. NodeSource listened to similar objective metrics, and surfaced additional information like "is there a patched direct dependency available".

Outside of subjective "good" and "bad" bundle sizes and opinions on whether or not TypeScript types matter, I've seen very few tools that don't exist in the context of objective facts about modules and their relationships to each other.

[mcollina]

None of the above is important from my evaluation. My first is the code understandable? Does it do what it is supposed to be doing? Could I patch it easily? What are its dependencies? Could I patch them if they need to be patched?

All the rest is just noise for me... putting those proxy values there will not encourage the users to look at the code they are adding to their projects.

[ljharb]

One example: npm search already (apparently) uses npms.io's algorithm which includes "time to issue close" - which is not, in fact, a good measure of how maintained a package is, but rather an incentive for maintainers to use bots to aggressively autoclose issues.

[bnb]

My first is the code understandable? Does it do what it is supposed to be doing? Could I patch it easily? What are its dependencies? Could I patch them if they need to be patched?

With the exception of "What are its dependencies?", how are any of these objective, as noted in your original comment of what you look for? 😅

[mcollina]

How I value those of those are not objective, and that's the full point.
The only one is "vulnerable dependencies" as I would not want to use a module with vulnerable dependencies.
Another very important criteria for me is: what is the bus factor? How many folks have publish rights?

I do not think there are objective criteria that can be a proxy measure for the quality of a package for a specific purpose. Telling people to trust those metrics does not provide any benefit to them. A significant part of the content available online teaches simple-for-beginners approaches that do not work whenever any complexity pops up.

---

I would rather use a different approach. Put some data crunching at work (maybe PageRank or analog) and for each module create a recommendation list:

1. what are the other modules that developers use this with?
2. which OSS author use this module, with what frequency?

Using a few ranking systems based on downloads, dependents and dependencies, etc can provide a good metric for a beginner.
I would also try to build a community of "module influencers" (it's there already, even if it is not endorsed in any way) to share list of what they use.

[ethomson]

One thing that's worth clarifying:

npms.io, which is used by npm to surface the three metrics on npmjs.com right now.

We actually don't use npms.io - at present, the scores in search results use a mechanism that was inspired by, but is not identical to, npms.io. The scores you see in npm's search results will differ from npms.io. We may want to move closer to npms.io's algorithm in the future, but I don't have any concrete plans to do that yet.

[YizYah]

Hi, this comes as a surprise, because https://docs.npmjs.com/searching-for-and-choosing-packages-to-download states that:

npm search uses npms and the npms analyzer; for more information on both, see https://npms.io/about.

Until your clarification just now, I was bewildered by your scoring. For two reasons:

• While our new package gets nearly 100% for maintenance in npms (https://api.npms.io/v2/package/ns-flip) you are scoring it as 33%! We are releasing almost daily, and only left one issue open for more than a few days.
• On Thursday I updated ns-flip to comply with all of the "Recommended Community Standards" at GitHub, and when I published a release my "quality" score on npm immediately tumbled from 92 to 62! This seems odd--does npm have something against providing PR templates, Codes of Conduct, Licences?? By contrast, quality on npms remains in the 90s.

Now at least I understand that, contrary to what I understand from your documentation, it's not npms. But may I ask, where do I find out about the criteria for your scores if they are different? One nice thing about npms is that they are transparent. I just want to know why you are scoring us so much lower.

[ethomson]

Hi, this comes as a surprise, because https://docs.npmjs.com/searching-for-and-choosing-packages-to-download states that...

Thanks for pointing that out. The documentation is outdated; at the time it was written, I suspect, we used the code from npms.io directly (though did not use their services). In time, both codebases have diverged.

I've filed a bug on our docs and we'll get this corrected.

But may I ask, where do I find out about the criteria for your scores if they are different? One nice thing about npms is that they are transparent. I just want to know why you are scoring us so much lower.

This is a trickier question to answer right now; parts of this process got moved around out of what was the npms.io code and into different parts of the registry service. My understanding is that there are several systems that work together to build this data. I've asked some engineers to investigate the mechanics of how this works today. I'm happy to publish this information once we pull this together - there's obviously no trade secrets here.

[jassmith]

It's been a bit, is this data available now? We're finding that in order to improve our npm package search score we need to do things that are not particularly good, like including our test files in the package itself even though it just bloats the package size.

[YizYah]

@ethomson Thank you for your sincere response! :)

If I understand the situation correctly:

1. the npm support themselves did not know that npms wasn't doing the scoring. [I ended up with two live support cases, and both representatives told me that scoring was by npms.] That may be understandable, because after all scoring is tangential to the mission of npm.
2. The documentation doesn't reflect the truth either. That is not so weird, since documentation sometimes lags behind development.
3. You personally are not certain whether the npms code was used directly in the past. (That's also understandable unless you were on the project.)
4. You don't know the current process and criteria. I appreciate your efforts to learn this, and much more so your honestly. I wonder--is there anyone at npm who does know?

I have to be careful what I say, because at the end of the day npm is a fine service that has had a profound impact on software development, and arguably humankind. 😃

All four issues above may soon be resolved. But I think we have a core problem here. Whatever scoring is being used, I think that it is failing for ns-flip, and by extension for everyone.

@bnb proposed using other rating systems in some way, given that their priorities have separate merits. I claim more fundamentally that your current scoring code simply has bugs. I claim that ns-flip reveals that. Please, Edward, do me a kindness and look for yourself at the activity on GitHub for ns-flip. Most simply, please just check out the maintenance. You are giving me a maintenance of 33, which is insane for such an active project. Nothing that I do changes it... it has actually declined. I claim something is broken in your collection or your analysis.

Then please look at the history of our quality score. I will be the first to say that ns-flip has its quality issues. Our testing coverage is currently awful. But how do you explain that until last Thursday you ranked us at 92, and then in one day our quality nosedived to 62? Especially when you understand what changed in the release. We had implemented the suggested community practices of GitHub!

I literally can run searches with our own keywords and I still cannot find us. I realize that npm is free, and you have a right to say to me "So sue me!" But for all the great strides for node here, npm is becoming a winner-takes-all platform. There are packages getting more than 50 million downloads per week, but surveying the rest of the 1.45 million packages suggests that most of npm is a graveyard for fizzled projects. A lot of them are junk to be sure, but some are passion projects. Those probably fizzled because nobody knew about them. Sadly, all of the documentation, tutorials, and maintenance in the world is wasted time if nobody sees a package.

If you want to empower promising new packages and tools to survive, the algorithm needs to be very fair and transparent for maintenance and quality. After all, those we can change. We certainly can't do anything about our "popularity" as newcomers.

[capricorn86]

I agree that the score system is confusing right now.

I increased the quality score by updating dependencies in my package (happy-dom) from about 50% to 93% at npms.io, but at npmjs.com it has remained the same. Maintanance is 100% at npms.io, but only 33% at npmjs.com, even though I do new releases on a weekly basis.

It would be great with a page explaining how it works and how I can improve my package. However, I suspect that there is some bug in the search at npmjs.com.

[vladmandic]

this thread is years old and has made no movement - devs stated the goal is to move closer to npms.io (which was apparently used sometime in the past, but diverged), but no actual plans. in the meantime, scores are stuck in limbo and no matter what i do i cannot make them to move.

for example, for one of my packages (scores are npmjs vs npms.io):

• overall: 32% vs 70%
• quality: 64% vs 95%
• popularity: 4% vs 19%
• maintenance: 33% vs 100%

this is 2x difference and nothing i do makes a difference

[erayhanoglu]

Still same, and no answers yet. I think that the problem is not in the algorithm that calculates the scores. I think that npmjs does not update the scores.

[michalusio]

I an assure you NPM does update the scores, but from what I've seen they can only go down.
My package's ratings after fixing a few tests and merging PRs:

• quality: 67% -> 40%
• maintenance: 45% -> 15% (!)

While at npms.io:

• quality: 94%
• maintenance: 100%

The difference is staggering and honestly I'm not sure whether creating packages has any sense, when they won't be used by anyone because they just don't show up on the list of results at this point.

[astoilkov]

I'm mentioning a related discussion I created that explores a way to make npm search better. The main idea is to give fewer points for popularity and more points for consistency of commits, releases, and issue responses. The longer the period of consistency, the better.

Discussion — https://github.com/npm/feedback/discussions/718
Proposal — https://astoilkov.com/better-npm-search-proposal

[kwhitley]

I actually disagree with this - fully understanding the popularity challenge it poses to incoming libraries (which I of course have faced myself many times).

1. When a typical developer is searching for a library, they typically do want the most widely-used "standard" library, even if it doesn't have great coverage, release frequency, etc.
2. Some of the most stable, battle-tested microlibs have been working flawlessly, without updates/attention, for years.
3. Issue responses should certainly be factored into the maintenance metric, but again - many libs never respond to the countless asks that pile up. Is that a flaw of the library? Not necessarily - the code may perform 100% as documented/expected, but the more popular, the more tangent issues pop up that may or may not be appropriate to address.

[vladmandic]

i'd love to have a conversation on this IF maintainers ever do anything about this.
unfortunately, at this point in time, i less care how score should ideally be calculated than "can we do anything about broken feature that had no update in years"

[kwhitley]

Totally agree... and if they're short-staffed, I'd be happy to take a look myself! :) We gotta get this fixed...

[astoilkov]

but the more popular, the more tangent issues pop up that may or may not be appropriate to address

that's my favorite argument. thanks for the feedback, i agree.

[kwhitley]

but the more popular, the more tangent issues pop up that may or may not be appropriate to address

that's my favorite argument. thanks for the feedback, i agree.

Certainly true, but the same can apply to the other metrics (maintenance and quality) as well. Optimizing search for any of the other two will lift other irrelevant (but perhaps well-maintained) libraries. The solution is to have better optimization/weight for relevance (keywords, description, topics, language, etc).

[michalusio]

Can confirm that NPM's metrics plain suck.
I have been fixing tests and updating dependencies. I had about 67% quality and 45% maintenance, and I was at about 6th, maybe 8th place on the list for the package's keywords.
After doing the upgrades and publishing the new version, the package's quality dropped to 40% while maintenance dropped to 15% ?!?!
Now I am at the 4th PAGE of the results, while at npms.io I am at the third position, with 94% quality and 100% maintenance.

What NPM does with their search metrics just sucks the fun out of creating and publishing packages - why should I strive for quality when populism is the only thing that matters, it seems?

[artginzburg]

I'm curious what happens if you publish the same package under a different name — does it get the same quality and maintenance scores, or are they recalculated from scratch?

Does this really mean that the more versions you publish — the less scores it's gonna get?

[ganeshkbhat]

I do not understand the searches metrics so much since I have ot done a deep dive in it. I wish not to comment on that. But here is a question I am raising on the basic download stats:

• I have two packages (both in development stage): cgijs, phpcgijs.
• The package [email protected] has a dependency [email protected]. [email protected] has bee published sometime back. However, [email protected] has been published a few days (a week) back with a dependency of cgijs.
• The issue: If I download [email protected], it has to download [email protected]. Considering last weeks download of 30 for [email protected], I yet have [email protected] at 12 downloads a week. (19th to 25th October 2022)
• I am expecting [email protected] downloads to be at downloads@[email protected] + downloads@[email protected]

This seems to be a gross actual download stats issue. All other stats of downloads and stats based on downloads depends on this calculation. I do not undertand why this was not noticed. I wonder why?

[monishcm]

Thanks for your feedback. We are already working on making the search much simpler to use.

[ganeshkbhat]

The dependency download stats do not match. The download/install stats of packages and related dep packages may be completely wrong all the way back dated to whatever historical dates.

[vladmandic]

Thanks for your feedback. We are already working on making the search much simpler to use.

Well, this issue is present for 2+ years and NPMjs hasnt made anything so far to fix it. Just saying "we're working to make search simpler" is really not saying anything.

[capricorn86]

If you are working on improving the search, please add support for explaining the reasoning behind the scores. For example making it possible to expand "quality" and see why it got a certain score (e.g. missing unit tests, readme.md etc.).

[kwhitley]

I too would ask that more clarity be provided for how to achieve a 100% in quality and maintenance metrics.

As others have stated, these are things we should be able to control, and if they actually represent indicators of quality, shouldn't we encourage everyone to specifically adhere to those?

I'm in a similar boat with my itty-router library. It's reasonably popular, and had > 99% on quality with npms.io (back when they were updating it). But on NPMJS? 63% quality, and nothing I do seems to shake it. I have far more quality controls in place than many libraries - yet somehow a brand new, virtually blank repo can achieve a > 80% quality score.

Quality Score on npms.io

Quality score on NPM

[ericmorand]

I'm also interested.

Twing is not even on the first page of the search result for the term "twig" when sorted by quality.

https://www.npmjs.com/search?q=twig&ranking=quality&page=0&perPage=20

What is the rational? From my external point of view the only explanation is that popularity is also taken into account when computing by quality.

I mean, here is the metrics computed by npm:

• Twing: 70% quality
• twig.js: 92% quality

Even though:

• Twing comes with a 100% code coverage unlike twig.js
• Twing exposes its build result unlike twig.js
• Twing exposed its coverage result unlike twig.js
• Twing comes with Typescript types and is fully typed unlike twig.js
• Twing comes with a dedicated website for its documentation unlike twig.js

I refuse to think that there are some marketing reason behind this low score even though it does everything quality-related better than twig.js, but it has been this way for years now and with no explanation at all.

So yeah, definitetely, an explanation and some clarification about the metric computation would be great.