Consider the impact of LLMs #4

pyfisch commented 2024年03月11日 20:56:32 +01:00

Since the proposal was written by @s3lf in 2022 large language models have significantly improved and are now widely available. Of course spammers and criminals who upload malware to Codeberg use these tools. I fear that many questions humans will find simple will also be solved by LLMs. In addition the questions an LLM might struggle with are often specific to a subculture and therefore not suitable for a Captcha.

Since the proposal was written by @s3lf in 2022 large language models have significantly improved and are now widely available. Of course spammers and criminals who upload malware to Codeberg use these tools. I fear that many questions humans will find simple will also be solved by LLMs. In addition the questions an LLM might struggle with are often specific to a subculture and therefore not suitable for a Captcha.

fnetX commented 2024年03月11日 22:01:37 +01:00

Thank you for the comment. I think we completely gave up on Captchas. For reference, you might want to read https://arxiv.org/pdf/2307.12108.pdf on page 10 to get an idea of how bots already outperform humans with even the best proprietary systems, and whatever we come up will likely be cracked by LLMs, indeed.

Our focus shifted to Codeberg/Community#1297 and Codeberg/Community#1329 (tl;dr: encouraging more interaction with users and differentiating based on actual actions).

You can find concrete next steps in forgejo/discussions#107 (comment) (implementation is still pending, suggestions welcome).

I'll archive this repo.

Thank you for the comment. I think we completely gave up on Captchas. For reference, you might want to read https://arxiv.org/pdf/2307.12108.pdf on page 10 to get an idea of how bots already outperform humans with even the best proprietary systems, and whatever we come up will likely be cracked by LLMs, indeed. Our focus shifted to https://codeberg.org/Codeberg/Community/issues/1297 and https://codeberg.org/Codeberg/Community/issues/1329 (tl;dr: encouraging more interaction with users and differentiating based on actual actions). You can find concrete next steps in https://codeberg.org/forgejo/discussions/issues/107#issuecomment-1571379 (implementation is still pending, suggestions welcome). I'll archive this repo.