Code search does not search critical large files

[rotu]

Select Topic Area

Product Feedback

Body

Some repos depend critically on large source files.
Two examples that have personally been a problem for me:

The core parsing code for the TypeScript compiler: https://github.com/microsoft/TypeScript/blob/main/src/compiler/parser.ts

The canonical ECMAScript language specification https://github.com/tc39/ecma262/blob/main/spec.html

These are omitted from code search omitted from code search with "Your search did not match any code".

https://github.com/search?q=repo%3Amicrosoft%2FTypeScript%20parseImportOrExportSpecifier&type=code
https://github.com/search?q=repo%3Atc39%2Fecma262%20ImageData&type=code

It would vastly increase the usefulness of code search for these repos if large files could be indexed for search, either by default or on an opt-in basis.

[LiteBrite82]

@look answered this post not too long ago: Very large repositories may not be indexed. What is considered a very large repository?

Alternatively, I have had luck using CodeSpaces to run the repo virtually and then searching the term in the VScode web browser (better than pulling it to your local machine):

It took about 2 minutes to build the code space.

Its not ideal, but it works. You can also run git log commands to search through the commit history.

Just be sure to stop the code space when you are done with it - to conserve your hours. Or, delete it if you are just using once.

P.S. This is a regularly requested enhancement; the dev team does monitor this discussion board, though they don't always have the bandwidth to respond.

[jaguilar]

A data point to consider. Some projects with lots of sources have lots of sources because a lot of people have contributed to them, indicating that they're very important to a lot of people. And so might be the search results. Please consider relaxing the search limits on large projects that have a high number of stars. For example, it was really a shame that I couldn't find PICO_CONFIG_RTOS_ADAPTER_HEADER in https://github.com/FreeRTOS/FreeRTOS-Kernel/blob/main/portable/ThirdParty/GCC/RP2040/library.cmake. These are the sorts of projects in which search is most desperately needed. Their size makes them hard to navigate, which is why we'd want search to work.

[look]

@jaguilar sorry for the inconvenience. To clarify, that match is excluded because it's in a directory named ThirdParty. We exclude some vendored code like that to keep the index size manageable.

[jaguilar]

@look Thanks kindly for the response. I'm not a FreeRTOS maintainer, but in my understanding, in this repository, ThirdParty is not "vendored" code. It's code that is part of the repository proper, but contributed by a third-party organization. (There is no external, canonical location of the code in this directory, for example.) Is there a way to mark a directory as non-vendored in some kind of .github config or something?

[look]

Unfortunately, there is not a way to configure this behavior at this time.

[jaguilar]

It seems a shame. To the extent you're able and feel willing to influence the team along these lines, I'd appreciate you raising the issue. It was a significant time sink for me, since I assumed GH search would Just Work(TM).