URL: https://linuxfr.org/users/killruana/journaux/selenium-anubis-et-cie
Title: Selenium>> Anubis et cie
Authors: jtremesay
Date: 2026年02月19日T19:03:09+01:00
License: CC By-SA
Tags: selenium, anubis, cyberpunk2077, python et hishel
Score: 13
Vous faites chier avec vos trucs genre [Anubis](https://github.com/TecharoHQ/anubis) qui cassent mes petits scrapers python :(
Là, je voulais gentiment scraper les missions sur le [Fandom de Cyberpunk 2077](https://cyberpunk.fandom.com/wiki/Cyberpunk_2077_Main_Jobs) pour pouvoir faire mon arbre de mission.
Et pouf, on se fait rembarrer parce qu'on utilise pas un vrai navigateur :
```shell
$ curl -v https://cyberpunk.fandom.com/wiki/Cyberpunk_2077_Main_Jobs> Host: cyberpunk.fandom.com> User-Agent: curl/8.18.0> Accept: */*> 
< HTTP/2 403 
```
Dans ces cas là, pas le choix, [Selenium WebDriver](https://www.selenium.dev/documentation/webdriver/).
WebDriver est lib multilangage pour contrôler un navigateur web tel que firefox ou chrome.
Minimal Viable Product®
```python3
#!/usr/bin/env python3
from pathlib import Path
from selenium import webdriver
from selenium.webdriver.common.by import By
def main():
 output_dir = Path("html")
 output_dir.mkdir(exist_ok=True)
 driver = webdriver.Firefox()
 driver.get("https://cyberpunk.fandom.com/wiki/Cyberpunk_2077_Main_Jobs")
 links = [
 (link.get_attribute("title"), link.get_attribute("href"))
 for link in driver.find_elements(By.CSS_SELECTOR, ".navbox li a")
 ]
 for link_title, link_href in links:
 page_file = output_dir / f"{link_title}.html"
 if page_file.exists():
 print(f"{link_title} already exists, skipping")
 continue
 print(f"Saving {link_title} from {link_href}")
 driver.get(link_href)
 content_node = driver.find_element(By.CSS_SELECTOR, ".mw-parser-output")
 page_file.write_text(content_node.get_attribute("outerHTML"))
 driver.quit()
if __name__ == "__main__":
 main()
```
j'trouve ça moins élégant que [Hishel](https://hishel.com/1.0/). Mais bon.
![Modern Problems Require Modern Solutions ](https://i.imgflip.com/akkbkd.jpg)
</div><div class="naked_ctrl">
<form action="/index.cgi/contrast" method="get" name="gate">
<p><a href="http://altstyle.alfasado.net">AltStyle</a> によって変換されたページ <a href="httpS://linuxfr.org/users/killruana/journaux/selenium-anubis-et-cie.md">(-&gt;オリジナル)</a>
/ <label>アドレス: <input type="text" name="naked_post_url" value="httpS://linuxfr.org/users/killruana/journaux/selenium-anubis-et-cie.md" size="22" /></label> <label>モード: <select name="naked_post_mode">
<option value="default">デフォルト</option>
<option value="speech">音声ブラウザ</option>
<option value="ruby">ルビ付き</option>
<option value="contrast" selected="selected">配色反転</option>
<option value="larger-text">文字拡大</option>
<option value="mobile">モバイル</option>
</select>
<input type="submit" value="表示" />
</p>
</form>
</div>