Join the chat at https://gitter.im/lndj/1024Spider Only for researching, please pay attention to your health.
I mainly want to learn about web spider using Python. But, nothing to crawl. Anyway, the 1024 website is a good example.
Use image.py as a example, it is a spider to crawl images of 1024.
- Modify the value of
DIR_PATHin line 14. - Find a useful url of 1024 and modify the value of
URLin line 17.
Then, you can enjoy it.
1024Spider is depended on Pysipder. (https://github.com/binux/pyspider)
You can install Pyspider firstly and then create a project on WebUI. Copy my code into it.
If you need more infomation, please read the Pyspider Documentation(http://docs.pyspider.org/en/latest/Quickstart/).