a golang micro-service framework compatible with alibaba dubbo. just using jsonrpc 2.0 protocol over http now.

dubbogo 目前版本(0.1.1)支持的codec 是jsonrpc 2.0,transport protocol是http。 只要你的java程序支持jsonrpc 2.0 over http,那么dubbogo程序就能调用它。 dubbogo自己的server端也已经实现,即dubbogo既能调用java service也能调用dubbogo实现的service。 由于dubbogo还没有上传到github,使用的时候就放到路径$/gopath}/github.com/AlexStocks/下面。

• 1 基于go1.7,修正package context的路径;

• 1 测试过程中发现,dubbogo/registry/zk/registry.go:(zookeeperRegistry)validateZookeeperClient()之后使用了RegistryZkClient,修改如下代码:

  a dubbogo/registry/zk/client.go:NewConsumerZookeeperRegistry()中调用newZookeeperRegistry后添加clause "reg.client.name = ConsumerRegistryZkClient"; b dubbogo/registry/zk/server.go:NewProviderZookeeperRegistry()中调用newZookeeperRegistry后添加clause "reg.client.name = ProviderRegistryZkClient";

• 2 修正dubbogo/registry/zk/client.go和dubbogo/registry/zk/server.go中的同样处理zk重启事件的函数handleZkRestart中的相关逻辑 此处须注意zk重启后会load之前的数据,导致启动之前注册的tmp节点数据又被load进来,大概10s之后才会被zk删除掉。
• 3 修正dubbogo/transport/http_transport.go中有关httpTransport成员相关的grammar error

• 1 测试过程中发现,如果一个zkPath下的node为空,而后添加一个node的时候,dubbo/registry/zk/watch.go:(zookeeperWatcher)watchDir()函数中for-select clause中time.After会导致watcher不能及时收到相关的event,如果剔除time.After又会导致疯狂for-select以致于影响程序性能,现做以下修改:

  a dubbo/registry/zk/zk_client.go中添加zookeeperClienit{entryRegistry}以及相关函数; b dubbo/registry/zk/watch.go:(zookeeperWatcher)watchDir() for-select添加一个case分支,以处理zkClient发来的event;

所以发生所有服务死掉的情况时,dubbogo的服务发现过程为:

a 第一个service的发现,要依靠dubbogo/registry/zk:(zookeeperClient) handleZkEvent()通知dubbogo/registry/zk:(zookeeperWatcher) watchDir(); b 第二个service启动后,dubbogo/registry/zk:(zookeeperWatcher) watchDir()自身能否及时发现值。

• 2 测试过程发现dubbogo/registry/zk:zookeeperClient) handleZkEvent()已经能够应对zk启停(死掉重启)的情况,所以注释掉这两个函数:

  a dubbogo/registry/zk:(consumerZookeeperRegistry)reconnectZkRegistr() b dubbogo/registry/zk:(providerZookeeperRegistry)reconnectZkRegistr()

• 1 dubbogo/selector/cache/cache.go:(cacheSelector)watch中添加一个内部exit chan,用于watch频繁创建的时候,让这个函数的内部goroutine退出;

• 2 dubbogo/registry/zk/watch.go:zookeeperWatcher添加once成员,以在(zookeeperWatcher)Stop函数中只执行一次client.Stop

• 3 dubbogo/registry/zk/client.go:(consumerZookeeperRegistry)Watch中,把for循环中的zkWatcher.watchService另起一个goroutine进行异步化操作,以防止阻塞for-loop。

• 4 dubbogo/registry/zk/watch.go:(zookeeperWatcher)watchService中,注释掉给this.events发送error的代码

  不要发送不必要的error给selector,以防止selector/cache/cache.go:(cacheSelector)watch,调用(zookeeperWatcher)Next获取error后,不断退出

• 5 为了dubbogo/registry/client与registry连接断开的情况下,dubbogo/selector能继续稳定地提供服务,修改:

  a dubbogo/registry/zk/watch.go:(zookeeperWatcher)watchDir()里面的watchServiceNode goroutine; b dubbogo/registry/zk/watch.go:(zookeeperWatcher)watchService()里面的watchServiceNode goroutine; c dubbogo/registry/zk/watch.go:(zookeeperWatcher)watchService()添加Valid method; d dubbogo/selector/cache.go:(cacheSelector) watch()中的for循环时收到delete event时检查watcher.Valid;

• 1 考虑到dubbogo/errs主要由dubbogo/client/rpc_client.go使用,把dubbogo/errs/errors.go挪到common之中;

• 2 修正dubbogo/selector/mode.go:roundRobin算法,给dubbogo/selector下的Next函数添加一个入参ID uint64

  因为selector仅仅返回了一个serviceURL array,如果不传入一个参数,每次roundRobin都从0开始,会导致只使用第一个地址

• 1 经过2016年08月06日#2的排查,查明了(rpcClient)call的poolConn.Close()&rpcStream.Close()两处close逻辑有冲突,做以下修改:

  a: 注释掉poolConn.Close处的代码; b: 给poolConn添加once成员,确保只释放一次; c: 注意将来添加长连接逻辑时候,此处相关逻辑可能还要修改; 相关的error为:

  [08/05/16 20:02:37] [DEBG] @i{0}, call(Id{3979924209}, ctx{context.Background.WithValue("dubbogo-ctx", map[string]string{"X-Proxy-Id":"dubbogo", "X-Services":"im.youni.iccs.status.api.StatusUpdateService", "X-Method":"login0"}).WithDeadline(2016年08月05日 20:02:37.478436456 +0800 CST [149.341097ms])}, address{116.211.15.189:30880}, path{/im.youni.iccs.status.api.StatusUpdateService2}, request{&{im.youni.iccs.status.api.StatusUpdateService login0 application/json [0xc82185be60] {false }}}, response{}) = err{{"id":"dubbogo.client","code":500,"detail":"Error sending request: dial tcp 116.211.15.189:30880: i/o timeout","status":"Internal Server Error"}}

• 2 把dubbogo/client/rpc_client.go:(rpcClient)call中conn.close和stream.close合并到一个defer clause中,将stream.Close单独交给一个rpcClient.close goroutine处理

• 3 修改dubbogo/registry/zk/client.go & dubbogo/registry/zk/server.go 两个文件中对zk的重连逻辑.

• 1 为了让dubbogo/client/rpc_pool.go只保存长连接对象:

  给dubbogo/client/rpc_client.go:call 中this.pool.getConn函数调用下面的defer语句段添加StreamingRequest判断条件

• 2 为了检查consumerZookeeperRegistry:Getservice()获取不到service的serviceURL的错误,先给GetService函数和Register函数添加debug log
• 3 把dubbogo/registry/zk/client.go中consumerZookeeperRegistry:services的key由ServiceConfig.String()修改为ServiceConfig.Service,以防止consumerZookeeperRegistry:GetService时检查service是否注册时被提示没有注册的error

• 1 dubbogo/selector/cache/cache.go:(cacheSelector)get 修改逻辑,以让超时发生时但无法获取新的serviceURL array的时候继续使用旧的serviceURL array

• 1 把cmg中连接zk的超时时间由5s修改为60s,以进行io timeout测试 func newZookeeperClient(name string, zkAddrs []string, timeout int)
• 2 给下面三个函数添加serviceURL filter检查 func (this *consumerZookeeperRegistry) ListServices() func (this *consumerZookeeperRegistry) ListService() func (this *zookeeperWatcher) watchDir

• 1 由于昨天第二项改进后引入了一个新的bug,使得dubbogo/codec/jsonrpc/client.go:(clientCodec)ReadHead中找不到response id对应的method

  详细的修改内容见函数dubbogo/codec/jsonrpc/client.go:(clientCodec)Write里面的注释

• 1 根据下面panic记录修改 registry/zk/zk_client.go, 确保zookeeperClient{conn}为nil的情况下程序不会panic // panic 1 2016年08月01日 11:10:26 Recv loop terminated: err=read tcp 116.211.15.192:59987->116.211.15.190:2181: i/o timeout 2016年08月01日 11:10:26 Send loop terminated: err= [08/01/16 11:10:26] [WARN] get a zookeeper event{zk.Event{Type:-1, State:0, Path:"", Err:error(nil), Server:"116.211.15.190:2181"}}, state{0}:zookeeper disconnected [08/01/16 11:10:26] [WARN] zk{addr:[116.211.15.190:2181]} state is StateDisconnected, so close the zk client{name:zk registry}. [08/01/16 11:10:26] [INFO] zk{path:[116.211.15.190:2181], name:zk registry} connection goroutine game over. 2016年08月01日 11:10:26 Recv loop terminated: err=read tcp 116.211.15.192:59988->116.211.15.190:2181: i/o timeout 2016年08月01日 11:10:26 Send loop terminated: err= [08/01/16 11:10:26] [WARN] get a zookeeper event{zk.Event{Type:-1, State:0, Path:"", Err:error(nil), Server:"116.211.15.190:2181"}}, state{0}:zookeeper disconnected panic: runtime error: invalid memory address or nil pointer dereference [signal 0xb code=0x1 addr=0x0 pc=0x888db8]

  goroutine 16 [running]: panic(0xa2e9e0, 0xc82000a0e0) C:/Program Files (x86)/Console2/go1.6.2/src/runtime/panic.go:481 +0x3e6 github.com/samuel/go-zookeeper/zk.(*Conn).nextXid(0x0, 0xc800000000) C:/Users/AlexStocks/share/test/golang/lib/src/github.com/samuel/go-zookeeper/zk/conn.go:671 +0x58 github.com/samuel/go-zookeeper/zk.(*Conn).queueRequest(0x0, 0xc, 0x9083c0, 0xc820132d80, 0x908420, 0xc820b3e060, 0xc820177590, 0x0) C:/Users/AlexStocks/share/test/golang/lib/src/github.com/samuel/go-zookeeper/zk/conn.go:686 +0x31 github.com/samuel/go-zookeeper/zk.(*Conn).request(0x0, 0xc80000000c, 0x9083c0, 0xc820132d80, 0x908420, 0xc820b3e060, 0xc820177590, 0x0, 0x0, 0x0) C:/Users/AlexStocks/share/test/golang/lib/src/github.com/samuel/go-zookeeper/zk/conn.go:698 +0x92 :209jjjjjj dubbogo/registry/zk.(*zookeeperClient).getChildrenW(0xc82014a000, 0xc8201b0600, 0x34, 0x0, 0x0, 0x0, 0xc820a11730, 0x0, 0x0) C:/Users/AlexStocks/share/test/golang/lib/src/dubbogo/registry/zk/zk_client.go:236 +0xc1 dubbogo/registry/zk.(*zookeeperWatcher).watchDir(0xc82000b710, 0xc8201b0600, 0x34) C:/Users/AlexStocks/share/test/golang/lib/src/dubbogo/registry/zk/watch.go:98 +0xe8 dubbogo/registry/zk.(*zookeeperWatcher).watchService.func2(0xc82000b710, 0xc8201b0600, 0x34) C:/Users/AlexStocks/share/test/golang/lib/src/dubbogo/registry/zk/watch.go:215 +0x77 created by dubbogo/registry/zk.(*zookeeperWatcher).watchService C:/Users/AlexStocks/share/test/golang/lib/src/dubbogo/registry/zk/watch.go:217 +0x317

  2016年08月02日 14:29:57 Recv loop terminated: err=read tcp 116.211.15.192:39716->116.211.15.190:2181: i/o timeout 2016年08月02日 14:29:57 Send loop terminated: err= panic: runtime error: invalid memory address or nil pointer dereference [signal 0xb code=0x1 addr=0x0 pc=0x888db8]

  // panic 2 goroutine 49 [running]: panic(0xa2e9e0, 0xc82000a0e0) C:/Program Files (x86)/Console2/go1.6.2/src/runtime/panic.go:481 +0x3e6 github.com/samuel/go-zookeeper/zk.(*Conn).nextXid(0x0, 0xc800000000) C:/Users/AlexStocks/share/test/golang/lib/src/github.com/samuel/go-zookeeper/zk/conn.go:671 +0x58 github.com/samuel/go-zookeeper/zk.(*Conn).queueRequest(0x0, 0x3, 0x908240, 0xc8201cd2e0, 0x9082a0, 0xc820510cd0, 0xc820ca7c50, 0x0) C:/Users/AlexStocks/share/test/golang/lib/src/github.com/samuel/go-zookeeper/zk/conn.go:686 +0x31 github.com/samuel/go-zookeeper/zk.(*Conn).request(0x0, 0xc800000003, 0x908240, 0xc8201cd2e0, 0x9082a0, 0xc820510cd0, 0xc820ca7c50, 0x0, 0x0, 0x0) C:/Users/AlexStocks/share/test/golang/lib/src/github.com/samuel/go-zookeeper/zk/conn.go:698 +0x92 github.com/samuel/go-zookeeper/zk.(*Conn).ExistsW(0x0, 0xc82006e1a0, 0x19e, 0xc82002d000, 0x0, 0x0, 0x0, 0x0) C:/Users/AlexStocks/share/test/golang/lib/src/github.com/samuel/go-zookeeper/zk/conn.go:833 +0x293 dubbogo/registry/zk.(*zookeeperClient).existW(0xc820016480, 0xc82006e1a0, 0x19e, 0x1, 0x0, 0x0) C:/Users/AlexStocks/share/test/golang/lib/src/dubbogo/registry/zk/zk_client.go:289 +0x8d dubbogo/registry/zk.(*zookeeperWatcher).watchServiceNode(0xc820504a80, 0xc82006e1a0, 0x19e) C:/Users/AlexStocks/share/test/golang/lib/src/dubbogo/registry/zk/watch.go:59 +0xb9 dubbogo/registry/zk.(*zookeeperWatcher).watchService.func1(0xc820504a80, 0xc820504ae0, 0xc82006e1a0, 0x19e, 0xc8200a41b0) C:/Users/AlexStocks/share/test/golang/lib/src/dubbogo/registry/zk/watch.go:205 +0x4b created by dubbogo/registry/zk.(*zookeeperWatcher).watchService C:/Users/AlexStocks/share/test/golang/lib/src/dubbogo/registry/zk/watch.go:209 +0x1016

• 2 jsonrpc java端程序不能解析jsonrpc 2.0中id超过uint32_max的数字, 在dubbogo/codec/jsonrpc/client.go对Id进行mask(MAX_JSONRPC_ID)运算,防止其超过int32_max return error{"code":-32603,"message":"json: cannot unmarshal number -1874143215 into Go value of type uint64"}

• 3 http抓包方法 tcpdump -XvvennSs 0 -i any tcp[20:2]=0x4745 or tcp[20:2]=0x4854 tcpdump -Xeennvvlps0 port 30880 -iany sar 1 0 -n DEV -e 15:00:00 > data.txt //每隔1秒记录网络使用情况,直到15点,数据将保存到data.txt文件中

• 1 为dubbogo/client/rpc_client.go:rpcClient添加了Id field
• 2 为serverCodec:ReadHeader添加请求方法是否是POST验证条件
• 3 根据下面两个命令追踪的http response流程,对httpTransportSocket:Send进行了优化 tcpdump -Xnlpvvs0 -S port 10000 -iany -w http.pcap strace -p 32732 -f

• 1 参照effecive go的leaky buffer的例子,修改dubbogo/server/rpc_server.go的server:

  type server struct { mu sync.Mutex // protects the serviceMap serviceMap map[string]*service // service name -> service // reqLock sync.Mutex // protects freeReq // freeReq *request freeReq chan *request // respLock sync.Mutex // protects freeRsp // freeRsp *response freeRsp chan *response listener transport.Listener }

  把lock和list修改为一个有size限制的req/rsp chan,通过select把锁也省略掉.

• 2 参考net/netutil/listen.go进行最大连接数限制 type httpTransportListener { sem chan struct{} }

• 3 给dubbogo/client & dubbogo/server添加once,以确保安全退出

• 1 dubbogo/selector/cache/cache.go var ( // selector每15分钟通过tick函数清空cache或者get函数去清空某个service的cache, // 以全量获取某个service的所有providers DefaultTTL = 5 * time.Minute )

• 2 dubbogo/server/http_transport.go DefaultMaxSleepTime = 1 * time.Second // accept中间最大sleep interval DefaultMAXConnNum = 50 * 1024 * 1024 // 默认最大连接数 50w

• 3 dubbogo/server/rpc_server.go const ( FREE_LIST_SIZE = 4 * 1024 // freeReq & freeRsp leaky buffer size )

• 4 dubbogo/registry/zk/watch.go const ( ZK_DELAY = 3 // watchDir中使用,防止不断地对zk重连 MAX_TIMES = 10 // 设置(wathcer)watchDir()等待时长 Wactch_Event_Channel_Size = 32 // 用于设置通知selector的event channel的size ZKCLIENT_EVENT_CHANNEL_SIZE = 4 // 设置用于zk client与watcher&consumer&provider之间沟通的channel的size )