Thanks for mentioning this! Thoughts on the wishlist, BTW? Anything to add, remove, or move?

Sure, IMO one of the main drawbacks of tokei, apart from its lack of maintenance 😭 , is - as you mentioned - that it mostly relies on file extensions for language detection which can lead to inaccuracies in favor of performance.

Ideally, it would be great to have a Rust equivalent of GitHub Linguist since Linguist has emerged as the de facto standard for language detection. Users often refer to Linguist when pointing out discrepancies with onefetch.

As a sidenote, I wonder how/whether linguist handles nested languages in Markdown and Jupyter-notebooks 🤔

We've implemented this in onefetch, but I believe it would be better if this was implemented in the language analysis tool.

The chip color for each language could also be provided.

Currently we analyze the contents of the filesystem.

Yes, but filesystem + .gitignore ~= tracked files ?

github-linguist uses blob size, and returns 56% TypeScript, 43% JavaScript, which is a more accurate distribution in this example.

Very good point, I wasn't aware of that. We may need both in that case: blob size for the language distribution and LOC for the info line of the same name

Yes, but filesystem + .gitignore ~= tracked files ?

What I mean is that, if I did this:

git init
echo 'console.log("Hello, World!");' > foo.js
git add foo.js
git commit -m "Create foo.js"
mv foo.js foo.ts
github-linguist
onefetch

Then github-linguist detects JavaScript, and onefetch detects TypeScript, because linguist is analyzing HEAD, not the current state of the files.

My bad, I didn't know that Linguist only acknowledged committed changes.

However, I could see an argument where users would want to see their changes being taken into account live - without needing to commit.

Still (As you said), for onefetch - being a Git information tool, it does make sense to stick to HEAD when computing the language distribution.

Yeah, I think the biggest argument for analyzing HEAD is the confusion in #26 (comment). But I think the majority of onefetch's users expect the current files to be analyzed. Or at least, I don't remember anyone else raising an issue about it.

My first time executing github-linguist locally, I actually found it surprising that I needed to commit changes for them to be analyzed.

If you want to preview or contribute, let me know and I'll send an invite (I think I can send a few on the free plan).

Nevermind, I made it public.

Very happy to see this project coming to life 😊 . Do you have an estimated timeline for when we might be able to replace tokei with gengo? What are the key components still missing for the transition?

It would be great to see some benchmarks comparing gengo with other tools, especially on complex repositories. Also, how does gengo (will)handle nested languages, such as Markdown or Jupyter notebooks?

Will gengo support the exclusion of specific glob patterns as a parameter, similar to how tokei does?

I'll be keeping an eye on this project and hope to contribute. Please don't hesitate to create issues and tag them with "help wanted"

As far as replacing goes, we'll need a lot more language support 😆 Also, the API is pretty different from tokei, so that will take some work on this end.

I don't forsee nested languages being supported, since, like linguist, distribution is calculated by blob size, and for performance reasons it won't read the whole file (default is the first MB).

Will gengo support the exclusion of specific glob patterns as a parameter, similar to how tokei does?

Yes, via gitattributes. By default a file is exluded if it is detected as documentation, generated, or vendored code. But you can customize this behavior. For example

# include dist/
dist/* gengo-detectable
# exclude js files
*.js -gengo-detectable

🤔 Thinking about benchmarks, it's probably going to be pretty unfair to start comparing until there is a roughly equal amount of language entries as linguist and tokei. As it is right now, each language entry would add another iteration for attempting to identify a file. Although, now that I'm thinking about it, I could probably gain a lot of performance by mapping extensions, shebangs, etc. to lists of languages instead of mapping languages to extensions, shebangs, etc. 🤔

Speaking about benchmarks, when I ran tokei and gengo on the linux repo they were about equal. Tokei was about 20 seconds, gengo about 22. github-linguist took 5 minutes IIRC...