webmagic: webmagic 是一个无须配置、便于二次开发的爬虫框架,它提供简单灵活的API,只需少量代码即可实现一个爬虫。

Fetch the repository succeeded.

richgiteeai

Watch

Unwatch Watching Releases Only Ignoring

1 Star 0 Fork 2K

study/webmagic

forked from 黄亿华/webmagic

Code Issues 0 Pull Requests 0 Wiki Insights Pipelines

Service

Tencent Cloud Serverless

Create your Gitee Account

Explore and code with more than 14 million developers,Free private repositories !:)

Already have an account? Sign in

文件

New file Create Diagram file

develop

Branches (36)

Tags (25)

Manage

develop

master

snyk-fix-4eb94e676367a3667c00737cdabdf3de

snyk-fix-8beb1ce977279ec88cdb90c43603581c

snyk-fix-a0dcc179d2c2f593e02424c579060a2d

snyk-fix-25c8898b02f8de0fc143c2eaa3e8c7cc

snyk-fix-3bde29486cddc7ed8a030e05f97999dd

snyk-fix-729e15e7770e9475a8504ae0fb1b5fe7

snyk-fix-cdf627b4ee226c4d9171b1960a044dc0

snyk-fix-3bb5020d5999f6b3ded6eb3378296acd

snyk-fix-5e9c00d3ecc1ab84f141bc680b00fe19

snyk-fix-f533124205b5e86f0bda976e921ffec2

snyk-fix-9e933ef3fbdfa89b1dad83f86cbbea2a

snyk-fix-9cd27ad57a7323a57b3b3f45ce02a123

revert-977-build

snyk-fix-ad3d8147b4d9640ed91bfc34fbfb9e38

snyk-fix-9aefd7a47e1f776c68e7fad3e85033a4

snyk-fix-5cfb952010fd88492e4e2d5516c6237d

snyk-fix-84ae966502c97386fb9c384e5874967b

snyk-fix-f96b01419b9ec0f23ce96beb068b24af

WebMagic-0.7.5

WebMagic-0.7.3

WebMagic-0.7.4

WebMagic-0.7.2

WebMagic-0.7.1

WebMagic-0.7.0

WebMagic-0.7.0.alpha

webmagic-parent-0.6.1

webmagic-parent-0.6.0

WebMagic-0.6.0

0.6.0

webmagic-0.5.3

WebMagic-0.5.2

WebMagic-0.5.1

WebMagic-0.5.0

webmaigc-0.4.3

webmagic-0.4.2

webmagic-0.4.1

webmagic-0.4.0

webmagic-0.3.2

Clone or Download

Clone/Download

HTTPS SSH SVN SVN+SSH Download ZIP

Prompt

To download the code, please copy the following command and execute it in the terminal

To ensure that your submitted code identity is correctly recognized by Gitee, please execute the following command.

git config --global user.name userName 
git config --global user.email userEmail

When using the SSH protocol for the first time to clone or push code, follow the prompts below to complete the SSH configuration.

1 Generate RSA keys.

2 Obtain the content of the RSA public key and configure it in SSH Public Keys

To use SVN on Gitee, please visit the usage guide

When using the HTTPS protocol, the command line will prompt for account and password verification as follows. For security reasons, Gitee recommends configure and use personal access tokens instead of login passwords for cloning, pushing, and other operations.

Username for 'https://gitee.com': userName

Password for 'https://userName@gitee.com': # Private Token

New file Create Diagram file

Branches 36

Tags 25

contribute

Sync branch

Create Pull Request

known more

See difference Through Pull Request Sync

Sync branch

Through Pull Request Sync

A Pull Request will be created to the current
branch and will be merged in to complete the sync

David Hsing change dependency versions into properties 54da7af

1136 commits

File empty ...

src

README.md

deploy.sh

pom.xml

README

webmagic-scripts
======
## 目标:
使得可以用简单脚本的方式编写爬虫,从而为一些常用场景提供可流通的脚本。如果已经有人写好了脚本,那么你直接使用就可以了!

## 实例:
例如:我需要抓github的仓库数据,可以这样写一个脚本(javascript):

```javascript
var name=xpath("//h1[@class='entry-title public']/strong/a/text()")
var readme=xpath("//div[@id='readme']/tidyText()")
var star=xpath("//ul[@class='pagehead-actions']/li[1]//a[@class='social-count js-social-count']/text()")
var fork=xpath("//ul[@class='pagehead-actions']/li[2]//a[@class='social-count']/text()")
var url=page.getUrl().toString()
if (name!=null){
 println(name)
 println(readme)
 println(star)
 println(url)
}

urls("(https://github\\.com/\\w+/\\w+)")
urls("(https://github\\.com/\\w+)")
```

然后使用webmagic加载并启动它,无需下载依赖、编写代码、执行的过程。目前已经有控制台版本,请下载[http://code4craft.qiniudn.com/webmagic-console.tar.gz](http://code4craft.qiniudn.com/webmagic-console.tar.gz)。

解压后,使用以下命令执行:
	
	java -jar -Dfile.encoding='utf-8' webmagic-console.jar -f 脚本文件名 [-l 语言,默认是javascript] [-t 线程数] [-s 抓取间隔,毫秒] url1 url2 ...

例如,对于github这个脚本,我可以这样执行:

java -jar -Dfile.encoding='utf-8' webmagic-console.jar -f github.js -t 2 -s 0 https://github.com/code4craft

目前这部分使用Java的ScriptEngine机制完成。

## 语言:

选用javascript是因为用户面比较广。目前还支持ruby语言,选用ruby是因为ruby的语法编写DSL更简洁:

```ruby
name= xpath "//h1[@class='entry-title public']/strong/a/text()"
readme = xpath "//div[@id='readme']/tidyText()"
star = xpath "//ul[@class='pagehead-actions']/li[1]//a[@class='social-count js-social-count']/text()"
fork = xpath "//ul[@class='pagehead-actions']/li[2]//a[@class='social-count']/text()"
url=$page.getUrl().toString()

puts name,readme,star,fork,url unless name==nil

urls "(https://github\\.com/\\w+/\\w+)"
urls "(https://github\\.com/\\w+)"
```

多语言通过参数-l区分,例如执行这个ruby脚本需要:

java -jar -Dfile.encoding='utf-8' webmagic-console.jar -f github.rb -t2 -s0 -l ruby https://github.com/code4craft

这个功能目前仍在实验阶段。欢迎大家积极参与并提出意见。