0

I'm trying to scrape the following two pages using beautifulSoap4

Both have the same HTML structure. When I load the first webpage, it's all fine and I get this:

<!DOCTYPE html>
<html dir="rtl" lang="fa-IR">
 <head>
 <style id="litespeed-optm-css-rules">
 ...

But the second webpage output is this:

Some characters could not be decoded, and were replaced with REPLACEMENT CHARACTER.
 �[s�Ƶ(��_���!��3�+ E:�|���lmI����.UИ��&amp; ���!���p���'ە�����~��?1��̩� f0�\ q�
<u*q�"�f��v�[�^}��~|�����e����4� 94�,4�pf�cӗ��̣[="%��[iv*#��0�T:P�kŃ��rӴ�" c��gm_vv۾l�gz���_���yˏ�����8�qw��ȳԕ�:h����="" �@��;��tr�="" �h�:a�="" ��@fy="">
 =���

Here is my python code:

from urllib.request import Request, urlopen
from bs4 import BeautifulSoup as soup
url = 'https://30nama.kim/top/30nama-movie.html'
req = Request(url , headers={'User-Agent': 'Mozilla/5.0'})
webpage = urlopen(req).read()
page_soup = soup(webpage, "html.parser")
print(page_soup.prettify())

I don't know what happens to the second page and what do these characters mean. I thought I should try to decode it using utf-8 but it didn't work. Any ideas?

asked Apr 8, 2020 at 18:34

1 Answer 1

1

BeautifulSoup uses Unicode, Dammit to detect the encoding. This is not always correct.

I sat the encoding manually and it worked:

page_soup = soup(webpage, "html.parser", from_encoding="ISO-8859-7")
answered Apr 8, 2020 at 19:04
Sign up to request clarification or add additional context in comments.

Comments

Your Answer

Draft saved
Draft discarded

Sign up or log in

Sign up using Google
Sign up using Email and Password

Post as a guest

Required, but never shown

Post as a guest

Required, but never shown

By clicking "Post Your Answer", you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.