[FEATURE] Search articles #41

BrightDV commented 2022年12月18日 11:30:14 +01:00 (Migrated from github.com)

Is your feature request related to a problem? Please describe.
/

Describe the solution you'd like
A search functionnality, at least for articles.

Describe alternatives you've considered
/

Additional context
#26

**Is your feature request related to a problem? Please describe.** / **Describe the solution you'd like** A search functionnality, at least for articles. **Describe alternatives you've considered** / **Additional context** #26

BrightDV commented 2022年12月18日 11:42:22 +01:00 (Migrated from github.com)

For now, the app search for articles using SearXNG instances. The instances selected allow showing the results in JSON format in order to avoid scraping. However, the requests are often blocked because of rate limits.
For the search, it filters the results using search parameters: it searches for "formula1.com/en/latest/article" $query so the url of the result must contain the string between the double quotes. Thus, it only returns articles.

One workaround is to use the RSS feed of Formula 1 and then search in it. For the moment, I didn't find any way to get more than 22 articles. Furthermore, I don't think that getting 1000 articles and then searching among them is a good solution, as it will use a lot of bandwidth and be very slow.

For now, the app search for articles using [SearXNG](https://searx.space) instances. The instances [selected](https://github.com/BrightDV/BoxBox/blob/3207500ac44456282ba6f2bcb6f73df35a194bcd/lib/api/searx.dart#L26) allow showing the [results in JSON format](https://docs.searxng.org/dev/search_api.html#parameters) in order to avoid scraping. However, the requests are often blocked because of rate limits. For the search, it filters the results using search parameters: it searches for `"formula1.com/en/latest/article" $query` so the url of the result must contain the string between the double quotes. Thus, it only returns articles. One workaround is to use the [RSS feed of Formula 1](https://www.formula1.com/content/fom-website/en/latest/all.xml) and then search in it. For the moment, I didn't find any way to get more than 22 articles. Furthermore, I don't think that getting 1000 articles and then searching among them is a good solution, as it will use a lot of bandwidth and be very slow.

sinfullad commented 2022年12月18日 12:58:41 +01:00 (Migrated from github.com)

For now, the app search for articles using SearXNG instances. The instances selected allow showing the results in JSON format in order to avoid scraping.

Sorry for the dumb question, but what do you mean by avoid scraping in this context?

Also in the worst case of scenario of all of the selected instances going down, are there any search engines you plan to use as the fallback option or will you use other instances? Currently I found Metager (metasearch similar to SearX), Mojeek (UK, uses its own crawler), Swisscows (data center in Switzerland, uses Bing Search and Bing Ads, though it uses its own indexes for Germany) to be viable options as well

> For now, the app search for articles using [SearXNG](https://searx.space) instances. The instances [selected](https://github.com/BrightDV/BoxBox/blob/3207500ac44456282ba6f2bcb6f73df35a194bcd/lib/api/searx.dart#L26) allow showing the [results in JSON format](https://docs.searxng.org/dev/search_api.html#parameters) in order to avoid scraping. Sorry for the dumb question, but what do you mean by **avoid scraping** in this context? Also in the worst case of scenario of all of the selected instances going down, are there any search engines you plan to use as the fallback option or will you use other instances? Currently I found [Metager](https://metager.org/datenschutz) ([metasearch](https://metager.org/search-engine) similar to SearX), [Mojeek](https://www.mojeek.com/about/privacy/) (UK, uses its own crawler), [Swisscows](https://swisscows.com/en/privacy) (data center in Switzerland, uses Bing Search and Bing Ads, though it uses its own indexes for Germany) to be viable options as well

BrightDV commented 2022年12月18日 14:13:28 +01:00 (Migrated from github.com)

Sorry for the dumb question, but what do you mean by avoid scraping in this context?

I don't like fetching a page and then extracting the content, but I will try to see if the rate limits still apply. If it doesn't, I will add the scraping if no results are found using the first method.

Also in the worst case of scenario of all of the selected instances going down, are there any search engines you plan to use as the fallback option or will you use other instances? Currently I found Metager (metasearch similar to SearX), Mojeek (UK, uses its own crawler), Swisscows (data center in Switzerland, uses Bing Search and Bing Ads, though it uses its own indexes for Germany) to be viable options as well

Thanks for these suggestions! However, I choose SearXNG because the backend is open-source, even if these propositions are made to be private.
With the scraping, there are up to 106 instances available, so I am going to try this way.

> Sorry for the dumb question, but what do you mean by avoid scraping in this context? I don't like fetching a page and then extracting the content, but I will try to see if the rate limits still apply. If it doesn't, I will add the scraping if no results are found using the first method. > Also in the worst case of scenario of all of the selected instances going down, are there any search engines you plan to use as the fallback option or will you use other instances? Currently I found [Metager](https://metager.org/datenschutz) ([metasearch](https://metager.org/search-engine) similar to SearX), [Mojeek](https://www.mojeek.com/about/privacy/) (UK, uses its own crawler), [Swisscows](https://swisscows.com/en/privacy) (data center in Switzerland, uses Bing Search and Bing Ads, though it uses its own indexes for Germany) to be viable options as well Thanks for these suggestions! However, I choose SearXNG because the backend is open-source, even if these propositions are made to be private. With the scraping, there are up to 106 instances available, so I am going to try this way.

BrightDV commented 2022年12月18日 14:40:10 +01:00 (Migrated from github.com)

The good news is that requesting the page in HTML format is not rate limited, so it will work better.
I have implemented a basic scraping when the five previous requests did not work, but I will improve it later.

The good news is that requesting the page in HTML format is not rate limited, so it will work better. I have implemented a basic scraping when the five previous requests did not work, but I will improve it later.