AI专业者开发社区 1 号成员
  •  0 回帖  •  2 浏览  •  4 个月前
 

应用程序连接hbase报错：java.net.SocketTimeoutException: callTimeout=60000

背景说明：

　　今天对生产环境 hbase 增加了节点，下午的时候一个同事反馈，应用程序后台报错，如下：

Tue Feb 26 17:35:35 CST 2019, null, java.net.SocketTimeoutException: callTimeout=60000, callDuration=68451: row 'SYSTEM.CATALOG,TARGETCUST_DATA,99999999999999' on table 'hbase:meta' at region=hbase:meta,,1.1588230740, hostname=host-10-191-36-24,16020,1551146724629, seqNum=0

    at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.throwEnrichedException(RpcRetryingCallerWithReadReplicas.java:276)
    at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:210)
    at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:60)
    at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:210)
    at org.apache.hadoop.hbase.client.ClientSmallReversedScanner.loadCache(ClientSmallReversedScanner.java:212)
    at org.apache.hadoop.hbase.client.ClientSmallReversedScanner.next(ClientSmallReversedScanner.java:186)
    at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegionInMeta(ConnectionManager.java:1275)
    at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1181)
    at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1165)
    at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1122)
    at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.getRegionLocation(ConnectionManager.java:957)
    at org.apache.hadoop.hbase.client.HRegionLocator.getRegionLocation(HRegionLocator.java:83)
    at org.apache.hadoop.hbase.client.HTable.getRegionLocation(HTable.java:506)
    at org.apache.hadoop.hbase.client.HTable.getKeysAndRegionsInRange(HTable.java:722)
    at org.apache.hadoop.hbase.client.HTable.getKeysAndRegionsInRange(HTable.java:692)
    at org.apache.hadoop.hbase.client.HTable.getStartKeysInRange(HTable.java:1769)
    at org.apache.hadoop.hbase.client.HTable.coprocessorService(HTable.java:1724)
    at org.apache.hadoop.hbase.client.HTable.coprocessorService(HTable.java:1704)
    at org.apache.phoenix.query.ConnectionQueryServicesImpl.metaDataCoprocessorExec(ConnectionQueryServicesImpl.java:1301)
    ... 47 more



Caused by: java.net.SocketTimeoutException: callTimeout=60000, callDuration=68451: row 'SYSTEM.CATALOG,TARGETCUST_DATA,99999999999999' on table 'hbase:meta' at region=hbase:meta,,1.1588230740, hostname=host-10-191-36-24,16020,1551146724629, seqNum=0
at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:169)
at org.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture.run(ResultBoundedCompletionService.java:65)
... 3 more
Caused by: java.net.UnknownHostException: host-10-191-36-24
at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.<init>(AbstractRpcClient.java:315)
at org.apache.hadoop.hbase.ipc.AbstractRpcClient.createBlockingRpcChannel(AbstractRpcClient.java:267)
at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.getClient(ConnectionManager.java:1639)
at org.apache.hadoop.hbase.client.ScannerCallable.prepare(ScannerCallable.java:162)
at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.prepare(ScannerCallableWithReplicas.java:376)
at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:134)
... 4 more
2019-02-26 17:35:35 [com.asiainfo.cb2.consumer.ReceiveSMSID]-[ERROR]:33 - sms consumer begin
2019-02-26 17:35:35 [com.asiainfo.cb2.consumer.ReceiveSMSID]-[ERROR]:38 - sms consumer pushType3

 

分析：

　　开始的时候，思路还是纠结在，超时的错误 java.net.SocketTimeoutException: callTimeout=60000, 想着有没有办法能够增加客户端的超时时间，然后还看了 datanode 的日志，认为是 datanode 在写数据导磁盘的时候性能问题，新增加 hadoop 节点，节点间在进行平衡，导致影响了性能，但是，对于以上的分析都没有更好的方法来解决问题。

结果：

　　等到后续，在仔细看错误，发现，Caused by: java.net.UnknownHostException: host-10-191-36-24，找不到主机的异常，才突然明白，应用程序首先连接到 zk，然后 zk 告知 region 在哪个 regionserver 上，然后，应用程序再连接到 hbase 的 regionserver 上读写数据。

解决：

　　所以，在应用程序的 /etc/hosts 上配置新增的 hbase regionserver 节点的 hosts 解析，再次观察应用程序，该问题解决

 

文档创建时间：2019 年 2 月 27 日 10:34:38