RestReader是基于DataX开发的rest请求的read插件,提供了读取rest请求数据的能力,并支持自动并发请求分页。 在底层实现上,通过首次rest请求获取到总记录数和页记录数算出总分页数。 总分页数即DataX的任务数,由DataX调度并发发起请求。
-
目前仅支持get和post请求
-
目前http请求响应仅支持json
-
post请求不支持form方式
-
数据格式:multiData 多条数据,返回为数组 oneData 单条数据,返回为Map。
{
"job": {
"setting": {
"speed": {
"channel":1
},
"errorLimit": {
"record": 0,
"percentage": 0.02
}
},
"content": [
{
"reader": {
"name": "restreader",
"parameter": {
"url": "http://10.68.6.40/usmp-backend/card/records?pageNo=1&pageIndex=0&pageSize=10&timeRangeType=0&doorId=",
"method": "get",
"timeout": 10,
"parameters": "",
"dataPath": "rows",
"pagination":true,
"totalParamPath":"total",
"startIndexParam":"pageIndex",
"pageSizeParam":"pageSize",
"pageNoParam":"pageNo",
"customHeader": {
"Jwt-Token": "eyJhbGciOiJSUzI1NiJ9.eyJhdXRoVXNlckRldGFpbHMiOiJ7XCJ1c2VyXCI6IHtcImNvZGVcIjogXCIwMDA4XCJ9fSIsInN1YiI6IlVzbXBBZG1pbiIsImRlZmF1bHRHcmFudFNjb3BlIjoiIiwiYW1yIjoibWZhIiwiaXNzIjoiaHR0cDovL2Rldi5jdHNwLmtlZGFjb20uY29tL2Nsb3VkLXJiYWMiLCJhdXRoVGltZSI6MTcwODIyMzUzNDk5NiwiYWNjZXNzVG9rZW4iOiJtakNBK2Q1QklsVmp6OXczTDgyRTB2aFVlVkRweVExNlV4SUJ0M0g3aTd0blNzZWNGY2s1QzBGQjlzSDFFam5LV21wZko4QlVvTFFIMDhDQkluaVVVQXhWaWRKdkNTNEU5b0pncjZwdmJORT0iLCJub25jZSI6bnVsbCwiYXVkIjoidHktdXNtcC1iYWNrZW5kIiwiYWNyIjoidXJuOm1hY2U6aW5jb21tb246aWFwOnNpbHZlciIsImF6cCI6IiIsInRlbmFudElkIjpudWxsLCJleHAiOjE3MDgyMjM1MzQsImlhdCI6MTcwNTYzMTUzNH0.pZvVozEi595HMt5A5Sj_WmCvNrubZOe0TFnycYF3MrmQbjcF_dDFLlYQh55Po_Zd2eJBwbm9oyO9HNyMsTOq0A"
},
"column" : [
{
"value": "id",
"type": "long"
},
{
"value": "gbid",
"type": "string"
}
]
}
},
"writer": {
"name": "mysqlwriter",
"parameter": {
"writeMode": "insert",
"username": "root",
"password": "root",
"column": [
"type",
"name"
],
"connection": [
{
"jdbcUrl": "jdbc:mysql://127.0.0.1:3306/datax?characterEncoding=utf-8",
"table": [
"test"
]
}
]
}
}
}
]
}
}{
"job": {
"setting": {
"speed": {
"channel":1
},
"errorLimit": {
"record": 0,
"percentage": 0.02
}
},
"content": [
{
"reader": {
"name": "restreader",
"parameter": {
"url": "http://transit.devdolphin.com/ga-pcs-bs-back/patrol/case_info_alarm/listAlarm",
"method": "post",
"dataPath": "result.content",
"pagination":true,
"totalParamPath":"result.totalElements",
"startIndexParam":"pageIndex",
"pageSizeParam":"pageSize",
"pageNoParam":"pageNo",
"body": "{\"caseNo\":\"\",\"caseName\":\"\",\"status\":\"\",\"deptCode\":\"\",\"acceptPeople\":\"\",\"closePoliceNo\":\"\",\"type\":\"0\",\"expireDateStart\":\"\",\"expireDateEnd\":\"\",\"pageNo\":0,\"pageSize\":20}",
"customHeader": {
"Jwt-Token": "eyJhbGciOiJSUzI1NiJ9.eyJhdXRoVXNlckRldGFpbHMiOiJ7XCJ1c2VyXCI6IHtcImNvZGVcIjogXCI4MDAwMDRcIn19Iiwic3ViIjoicWhwY3MiLCJkZWZhdWx0R3JhbnRTY29wZSI6IiIsImFtciI6Im1mYSIsImlzcyI6Imh0dHA6Ly9kZXYuY3RzcC5rZWRhY29tLmNvbS9jbG91ZC1yYmFjIiwiYXV0aFRpbWUiOjE3MDU3MjExODQ4OTksImFjY2Vzc1Rva2VuIjoibWpDQStkNUJJbFZqejl3M0w4MkUwa0JBY21Wb0x2SUJXTUxYbGVDU29tK25nMTVtUnlGY1NlVFRRSElpdHdrUnljc21XK0lFYzZOL1FSTmMveXZsS1JZWk5hYXJ4VEpyc2d0eWtMSUYvY1k9Iiwibm9uY2UiOm51bGwsImF1ZCI6ImdhLXBjcy1icy1iYWNrIiwiYWNyIjoidXJuOm1hY2U6aW5jb21tb246aWFwOnNpbHZlciIsImF6cCI6IiIsInRlbmFudElkIjpudWxsLCJleHAiOjE3MDU3MjExODQsImlhdCI6MTcwNTYzNDc4NH0.FdIPpL_OpeZ2I-wcS4KlJiIspRD2Kd7aTz-Akn-dth_X8c7h4zki0Mt0DAjpDIhtBdf-bWXR0hw0wa5KO8iRuDqpqKTfhV6QcGOPsWArT59wcowxkoSyidSGrC103hBmbjWnxlJONnP5uZF_zZtA503DS6Yaefn8awX4E1RdoRg"
},
"column" : [
{
"value": "id",
"type": "long"
},
{
"value": "caseName",
"type": "string"
}
]
}
},
"writer": {
"name": "mysqlwriter",
"parameter": {
"writeMode": "insert",
"username": "root",
"password": "root",
"column": [
"type",
"name"
],
"connection": [
{
"jdbcUrl": "jdbc:mysql://127.0.0.1:3306/datax?characterEncoding=utf-8",
"table": [
"test"
]
}
]
}
}
}
]
}
}
-
url
-
描述:rest请求,支持http,https协议
-
必选:是
-
默认值:无
-
-
method
-
描述:请求方法
-
必选:是 get或post
-
默认值:无
-
-
timeout
-
描述:超时时间,单位秒
-
必选:否
-
默认值:10
-
-
parameters
-
描述:如timeRangeType=0&doorId=,不需要带问号。
-
必选:否
-
默认值:无
-
-
body
-
描述:post请求时使用,字符串。见示例
-
必选:否
-
默认值:无
-
-
dataPath
-
描述:指向json响应数组的数据路径,如 result.content,就代表着result属性下级的content属性,content属性是一个数组
-
必选:否
-
默认值:""
-
-
pagination
-
描述:是否分页的标志。
-
必选:否。true使用分页,false不使用分页
-
默认值:false。
-
-
totalParamPath
-
描述:json响应指向总记录数的变量,和dataPath结合访问,如值为total,代表着result.content下的total变量是总记录数
-
必选:是
-
默认值:无
-
-
startIndexParam
-
描述:get请求中的url或者或者post请求的body中代表分页起始页的变量
-
必选:否
-
默认值:pageIndex
-
-
pageSizeParam
-
描述:get请求中的url或者或者post请求的body中代表分页页记录数的变量
-
必选:否
-
默认值:pageSize
-
-
pageNoParam
-
描述:get请求中的url或者或者post请求的body中代表分页页码的变量
-
必选:否
-
默认值:pageNo
-
-
customHeader
-
描述:自定义header值,是一个字典类型
-
必选:否
-
默认值:无
-
-
column
-
描述:读取字段列表,type指定源数据的类型,index指定当前列来自于文本第几列(以0开始),value指定当前类型为常量,不从源头文件读取数据,而是根据value值自动生成对应的列。
用户可以指定Column字段信息,配置如下:
{ "type": "long" }, { "type": "string", "value": "alibaba" }type目前只支持string,bool,long这些类型。
-
必选:是
-
默认值:全部按照string类型读取
-
略
略
微信:18106218216 QQ:38775033 欢迎批评指正