Search Feature #23

lhinderberger commented 2020年08月13日 12:52:38 +02:00

Once there is enough content in the Documentation, a search feature would probably become necessary.

Once there is enough content in the Documentation, a search feature would probably become necessary.

lhinderberger commented 2020年08月27日 22:12:33 +02:00

This article lays out nicely how a search feature can be realized using Eleventy and the Elasticlunr library: https://www.belter.io/eleventy-search/

This article lays out nicely how a search feature can be realized using Eleventy and the Elasticlunr library: https://www.belter.io/eleventy-search/

ojn commented 2020年10月23日 02:05:46 +02:00

How about DocSearch from Algolia

https://docsearch.algolia.com/docs/what-is-docsearch/
https://github.com/algolia/docsearch

It should be easy to integrate with Eleventy.

Codeberg docs might outgrow Elasticlunr fast, it is client side library right?

I wish there were better search tools.

Meilisearch works also, but it might need more work https://github.com/meilisearch/meilisearch

How about DocSearch from Algolia https://docsearch.algolia.com/docs/what-is-docsearch/ https://github.com/algolia/docsearch It should be easy to integrate with Eleventy. Codeberg docs might outgrow Elasticlunr fast, it is client side library right? I wish there were better search tools. Meilisearch works also, but it might need more work https://github.com/meilisearch/meilisearch

lhinderberger commented 2020年10月23日 03:15:01 +02:00

@ojn Welcome at Codeberg/Documentation! 🙂👋

I too have my concerns how long a client-side Lunr / Elasticlunr-based solution will be viable and we indeed might have to eventually switch to server-side search. I will implement a Lunr-based solution soon and then we can see how well it performs, especially with regards to index size and search speed.

Regarding DocSearch - is there a way to self-host this, including the search engine? Because at first glance, it looks like it is tied to a proprietary cloud service and would require us to transmit user data to a third-party company, which likely wouldn't be in line with Codeberg's goals.

Meilisearch looks like a great alternative though! I can very well imagine switching to that, once client-side search isn't feasible any more. Thank you for providing that link! 👍

@ojn Welcome at Codeberg/Documentation! :slightly_smiling_face::wave: I too have my concerns how long a client-side Lunr / Elasticlunr-based solution will be viable and we indeed might have to eventually switch to server-side search. I will implement a Lunr-based solution soon and then we can see how well it performs, especially with regards to index size and search speed. Regarding DocSearch - is there a way to self-host this, including the search engine? Because at first glance, it looks like it is tied to a proprietary cloud service and would require us to transmit user data to a third-party company, which likely wouldn't be in line with Codeberg's goals. Meilisearch looks like a great alternative though! I can very well imagine switching to that, once client-side search isn't feasible any more. Thank you for providing that link! :thumbsup:

ojn commented 2020年10月23日 10:33:22 +02:00

Regarding DocSearch

yeah that concerning for a project like Codeberg, it appears to be only partially open source and partially SaaS.

Another idea would be to selfhost meta search engine Searx and specify it to only show results for docs.codeberg.org and have some setting where one could search the whole codeberg.org website, it would still depend on external crawlers so that really not a standalone tool. This would require more people to chime in on how they feel about running a metasearch site for such purpose.

> Regarding DocSearch yeah that concerning for a project like Codeberg, it appears to be only partially open source and partially SaaS. Another idea would be to selfhost meta search engine Searx and specify it to only show results for docs.codeberg.org and have some setting where one could search the whole codeberg.org website, it would still depend on external crawlers so that really not a standalone tool. This would require more people to chime in on how they feel about running a metasearch site for such purpose.

hw commented 2020年10月23日 17:39:49 +02:00

Codeberg docs might outgrow Elasticlunr fast, it is client side library right?

how does it scale? If client-side, it should only depend on size of index (that is, size+complexity of indexed content, but not number of users?)?

> > Codeberg docs might outgrow Elasticlunr fast, it is client side library right? > how does it scale? If client-side, it should only depend on size of index (that is, size+complexity of indexed content, but not number of users?)?

lhinderberger commented 2020年10月24日 03:30:26 +02:00

Index size affects search performance and feasibility in two ways:

• The larger the site becomes, the larger the index needs to be, the more data needs to be transmitted up-front before search can be performed. If the index grows too large (think of megabytes), search may become unusable for many mobile users and traffic will rise significantly.
• Depending on the efficiency of the search algorithm and on the characteristics of JavaScript handling somewhat larger data structures, there may be a practical limit on the size of the search index, before client-side search performance (again, think mobile devices too) becomes too slow

Index size affects search performance and feasibility in two ways: - The larger the site becomes, the larger the index needs to be, the more data needs to be transmitted up-front before search can be performed. If the index grows too large (think of megabytes), search may become unusable for many mobile users and traffic will rise significantly. - Depending on the efficiency of the search algorithm and on the characteristics of JavaScript handling somewhat larger data structures, there may be a practical limit on the size of the search index, before client-side search performance (again, think mobile devices too) becomes too slow

hw commented 2020年10月24日 13:32:04 +02:00

What index size are we talking about for the current documentation? (and where would we arrive if we scale it via rule of thumb, by, say, a factor of 10x?)

What index size are we talking about for the current documentation? (and where would we arrive if we scale it via rule of thumb, by, say, a factor of 10x?)

lhinderberger commented 2020年10月24日 13:42:46 +02:00

Hard to estimate without having tried it in practice yet. But we currently have about 200K of Markdown sources. So it's not unthinkable as a worst-case estimate to have a search index of 2MB one day, that needs to be downloaded before users can search the site. But we'll only know for sure once we see how compact Lunr search indexes are and how well they can be compressed.

Hard to estimate without having tried it in practice yet. But we currently have about 200K of Markdown sources. So it's not unthinkable as a worst-case estimate to have a search index of 2MB one day, that needs to be downloaded before users can search the site. But we'll only know for sure once we see how compact Lunr search indexes are and how well they can be compressed.

HappyPony commented 2023年04月18日 07:17:16 +02:00

Is there a proect that deals with the realization of the search feature for https://codeberg.org/Codeberg/Documentation [1] ? I really miss a search feature and suspect that with a powerful search - and even better with a codebergChatbot trained with [1] content the codeberg space would become much more attractive for many users :-).

n hat das Label Status: Help wanted vor 2 Jahren hinzugefügt

I would like to participate in a "codebergChatbot as searchFeature for Codeberg/Documentation", if a fellow contributor with advanced Linux knowledge can be found. GPT4All looks promising IMHO. Dolly 2.0 could be interesting.

= bibliography =
[1] KONKURRENZ ZU CHATGPT: Dolly 2.0 ist ein komplett quelloffener Sprachgenerator, 17.04.2023 - https://www.golem.de/news/konkurrenz-zu-chatgpt-dolly-2-0-ist-ein-komplett-quelloffener-sprachgenerator-2304-173465.html
[2] gpt4all - https://github.com/nomic-ai/gpt4all
[2.1] "Das Projekt GPT4ALL ermöglicht es, sich einen ähnlich mächtigen Chatbot wie GPT auf lokalen Endgeräten zu installieren", Dennis-Kenji Kipker, 14.04.2023 - https://www.weser-kurier.de/landkreis-diepholz/datenkolumne-wie-kuenstliche-intelligenz-den-alltag-veraendert-doc7pr782s7zci15jhyoghp
[2.2] Prof. Dr. Dennis-Kenji Kipker - https://www.uni-bremen.de/jura/fachbereich-6-rechtswissenschaft/fachbereich/personen/prof-dr-dennis-kenji-kipker
[2.3] Dennis-Kenji Kipker - https://de.wikipedia.org/wiki/Dennis-Kenji_Kipker, retrieved on 17.04.2023

Is there a proect that deals with the realization of the search feature for https://codeberg.org/Codeberg/Documentation [1] **?** I really miss a search feature and suspect that with a powerful search - and even better with a *codebergChatbot* trained with [1] content the codeberg space would become much more attractive for many users :-). > n hat das Label Status: Help wanted vor 2 Jahren hinzugefügt I would like to participate in a "***codebergChatbot* as searchFeature for Codeberg/Documentation**", if a fellow contributor with advanced Linux knowledge can be found. [GPT4All](https://github.com/nomic-ai/gpt4all) looks promising IMHO. [Dolly 2.0](https://www.golem.de/news/konkurrenz-zu-chatgpt-dolly-2-0-ist-ein-komplett-quelloffener-sprachgenerator-2304-173465.html) could be interesting. = bibliography = [1] KONKURRENZ ZU CHATGPT: Dolly 2.0 ist ein komplett quelloffener Sprachgenerator, 17.04.2023 - https://www.golem.de/news/konkurrenz-zu-chatgpt-dolly-2-0-ist-ein-komplett-quelloffener-sprachgenerator-2304-173465.html [2] gpt4all - https://github.com/nomic-ai/gpt4all [2.1] "Das Projekt GPT4ALL ermöglicht es, sich einen ähnlich mächtigen Chatbot wie GPT auf lokalen Endgeräten zu installieren", Dennis-Kenji Kipker, 14.04.2023 - https://www.weser-kurier.de/landkreis-diepholz/datenkolumne-wie-kuenstliche-intelligenz-den-alltag-veraendert-doc7pr782s7zci15jhyoghp [2.2] Prof. Dr. Dennis-Kenji Kipker - https://www.uni-bremen.de/jura/fachbereich-6-rechtswissenschaft/fachbereich/personen/prof-dr-dennis-kenji-kipker [2.3] Dennis-Kenji Kipker - https://de.wikipedia.org/wiki/Dennis-Kenji_Kipker, retrieved on 17.04.2023

n0toose commented 2023年04月18日 11:02:30 +02:00

Hallucination in AI systems is a thing, an LLM backend is a lot of maintenance, work and computationally inefficient (or inefficient for the job it's intended to do), and there are a lot of legal gray areas surrounding these technologies right now.

Hallucination in AI systems is a thing, an LLM backend is a lot of maintenance, work and computationally inefficient (or inefficient for the job it's intended to do), and there are a lot of legal gray areas surrounding these technologies right now.

HappyPony commented 2023年04月18日 13:03:16 +02:00

Hi @n0toose,

so that we can support the exchange of opinions with factual arguments, I suggest that we handle valid verifiable terms and facts in the process. IMHO it is not purposeful when considering a language model to attribute the "hallucinate" ability to the language model. There is no scientific basis for the claim that ChatGPT can "hallucinate".

Hallucination (from the Latin alucinatio 'dreaming') is a perception for which there is no demonstrable external stimulus basis. Such perceptions can occur in any sensory area. This means, for example, that physically undetectable objects are seen or voices are heard without anyone speaking.
https://de.wikipedia.org/wiki/Halluzination

It seems that the "hallucination" attribution is simply adopted by many science journalists without sufficiently checking whether it is really correct. Detailed explanations on this -s. in [3].

LLM backend computationally inefficient

I am interested in computational examples. And as for efficiency, a recent example:

... at a start-up in Baden-Württemberg, where young entrepreneurs make more revenue thanks to AI without new employees"

[4] 16.04.2023 from the time mark 10:20 min

I think what leads to such efficiency increase in the free economy can also help codeberg.org users to work more productively.

Can we maybe make a poll what other codeberg.org users think about the idea of a self-trained codebergChatbot? Maybe there is a startup interested in sponsoring a language model instance for https://codeberg.org ;-)?

= bibliography =
[3] ChatGPT: Responsibility for faulty behavior of the machine is borne by the human being, 31.03.2023 - https://realtime.fyi/articles/nmoplus/chatgpt-responsibility-for-faulty-behavior-of-the-machine-is-borne-by-the-human-being
[4] Bericht aus Berlin, 16.04.2023, ab der Zeitmarke 10:00 Min - https://www.ardmediathek.de/video/bericht-aus-berlin/bericht-aus-berlin/das-erste/Y3JpZDovL2Rhc2Vyc3RlLmRlL2JlcmljaHQgYXVzIGJlcmxpbi8yMDIzLTA0LTE2XzE4LTAwLU1FU1o

Hi @n0toose, so that we can support the exchange of opinions with factual arguments, I suggest that we handle *valid verifiable terms and facts in the process*. IMHO it is not purposeful when considering a language model to attribute the "*hallucinate*" ability to the language model. **There is no scientific basis for the claim that ChatGPT can "hallucinate".** > Hallucination (from the Latin alucinatio 'dreaming') is a perception for which there is no demonstrable external stimulus basis. Such perceptions can occur in any sensory area. This means, for example, that physically undetectable objects are seen or voices are heard without anyone speaking. https://de.wikipedia.org/wiki/Halluzination It seems that the "hallucination" attribution is simply adopted by many science journalists *without sufficiently checking whether it is really correct.* Detailed explanations on this -s. in [[3]](https://realtime.fyi/articles/nmoplus/chatgpt-responsibility-for-faulty-behavior-of-the-machine-is-borne-by-the-human-being). > LLM backend computationally inefficient I am interested in computational examples. And as for efficiency, a recent example: > ... at a start-up in Baden-Württemberg, where young entrepreneurs make more revenue thanks to AI without new employees" [[4]](https://www.ardmediathek.de/video/bericht-aus-berlin/bericht-aus-berlin/das-erste/Y3JpZDovL2Rhc2Vyc3RlLmRlL2JlcmljaHQgYXVzIGJlcmxpbi8yMDIzLTA0LTE2XzE4LTAwLU1FU1o) 16.04.2023 from the time mark 10:20 min I think what leads to such efficiency increase in the free economy can also help *codeberg.org* users to work more productively. Can we maybe make a poll what other *codeberg.org* users think about the idea of a self-trained *codebergChatbot*? Maybe there is a startup interested in sponsoring a language model instance for https://codeberg.org ;-)? **= bibliography =** [3] ChatGPT: Responsibility for faulty behavior of the machine is borne by the human being, 31.03.2023 - https://realtime.fyi/articles/nmoplus/chatgpt-responsibility-for-faulty-behavior-of-the-machine-is-borne-by-the-human-being [4] Bericht aus Berlin, 16.04.2023, ab der Zeitmarke 10:00 Min - https://www.ardmediathek.de/video/bericht-aus-berlin/bericht-aus-berlin/das-erste/Y3JpZDovL2Rhc2Vyc3RlLmRlL2JlcmljaHQgYXVzIGJlcmxpbi8yMDIzLTA0LTE2XzE4LTAwLU1FU1o

n0toose commented 2023年04月18日 20:06:26 +02:00

There is no scientific basis for the claim that ChatGPT can "hallucinate".

Here's the first result I got off DuckDuckGo after I looked up "hallucination chatgpt ai" with the chief scientist being quoted as "having confirmed it" from IEEE Spectrum, not a peer-reviewed paper but I'd bet my money on being able to find something else: https://spectrum.ieee.org/ai-hallucination

Also see: https://arstechnica.com/tech-policy/2023/04/openai-may-be-sued-after-chatgpt-falsely-says-aussie-mayor-is-an-ex-con/

I find that the problems of AI are very easy to find, but I'll engage with your arguments in good faith under the overall assumption that you're excited and think it could help here.

I hold the personal belief that, right now, people are trying to apply a solution to as many problems as possible, instead of doing so the other way around, like w/ Blockchain.

I am not sure how some entrepreneurs making money in Baden-Württemberg prove in any capacity that an AI-based chatbot interface on top of our documentation would be less computationally expensive than more traditional solutions to the problem. I think it goes without saying that it would cost us more money and human resources to do something like this.

I admire your enthusiasm about the subject, and frankly, I can't say that I am not curious about it as well, but I really don't think that "people saying AI is going to change the world" is in any way making the case that this isn't a pure distraction. It's not like parts of it won't stay and define the future of how humanity interacts with technology, but the manufactured hype and "inverse" problem solving is really not what we need to focus right now. It's unpredictable (especially the self-hosted versions), there are big questions that have not been answered and absolutely not a panacea.

We don't have to pay attention to every "brand new shiny thing" from Silicon Valley. We don't have to make profit; we're a non-profit, and making profit isn't necessarily a sign of what we want to do: Host code and support the communities that work on code properly.

> **There is no scientific basis for the claim that ChatGPT can "hallucinate".** Here's the first result I got off DuckDuckGo after I looked up "hallucination chatgpt ai" with the chief scientist being quoted as "having confirmed it" from IEEE Spectrum, not a peer-reviewed paper but I'd bet my money on being able to find something else: https://spectrum.ieee.org/ai-hallucination Also see: https://arstechnica.com/tech-policy/2023/04/openai-may-be-sued-after-chatgpt-falsely-says-aussie-mayor-is-an-ex-con/ I find that the problems of AI are very easy to find, but I'll engage with your arguments in good faith under the overall assumption that you're excited and think it could help here. I hold the personal belief that, right now, people are trying to apply a solution to as many problems as possible, instead of doing so the other way around, like w/ Blockchain. I am not sure how some entrepreneurs making money in Baden-Württemberg prove in any capacity that an AI-based chatbot interface on top of our documentation would be less computationally expensive than more traditional solutions to the problem. I think it goes without saying that it would cost us *more* money and human resources to do something like this. I admire your enthusiasm about the subject, and frankly, I can't say that I am not curious about it as well, but I really don't think that "people saying AI is going to change the world" is in any way making the case that this isn't a pure distraction. It's not like parts of it won't stay and define the future of how humanity interacts with technology, but the manufactured hype and "inverse" problem solving is really not what we need to focus right now. It's unpredictable (especially the self-hosted versions), there are big questions that have not been answered and absolutely not a panacea. We don't have to pay attention to every "brand new shiny thing" from Silicon Valley. We don't have to make profit; we're a non-profit, and making profit isn't necessarily a sign of what we want to do: Host code and support the communities that work on code properly.

fnetX commented 2023年04月23日 14:36:15 +02:00

Sorry, but there is no need to deploy AI for a simple search feature. Other static site generators have search built in (for example "Material for MkDocs" which works fine with static sites, too.

There are libraries available to do the same. I think that's the way to go.

Sorry, but there is no need to deploy AI for a simple search feature. Other static site generators have search built in (for example ["Material for MkDocs" ](https://squidfunk.github.io/mkdocs-material/) which works fine with static sites, too. There are libraries available to do the same. I think that's the way to go.

n0toose commented 2023年12月08日 15:38:39 +01:00

Resolved in #370.

Resolved in #370.