Bootstrap

生产排查org.apache.http.NoHttpResponseException: 127.0.0.1:9000 failed to respond

生产环境,请求方调用我方地址,发生异常NoHttpResponseException,错误详情:

org.apache.http.NoHttpResponseException: 127.0.0.1:9000 failed to respond
	at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:141)
	at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56)
	at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:259)
	at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:163)
	at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:157)
	at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:273)
	at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125)
	at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272)
	at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186)
	at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)
	at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
	at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
	at com.http.CommonHttpClient.sendPost(CommonHttpClient.java:96)
	at com.http.CommonHttpClient.lambda$main$0(CommonHttpClient.java:121)
	at java.util.stream.ForEachOps$ForEachOp$OfInt.accept(ForEachOps.java:205)
	at java.util.Spliterators$IntArraySpliterator.forEachRemaining(Spliterators.java:1032)
	at java.util.Spliterator$OfInt.forEachRemaining(Spliterator.java:693)
	at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)
	at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:291)
	at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731)
	at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289)
	at java.util.concurrent.ForkJoinPool$WorkQueue.execLocalTasks(ForkJoinPool.java:1040)
	at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1058)
	at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692)
	at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:157)

看一下报错的源代码,相关注释提示服务端关闭了连接导致的:
在这里插入图片描述
想要解决问题,首先需要复现一下,于是本地启动服务,设置tomcat保活时间1s,

server:
  tomcat:
    keep-alive-timeout: 1000

使用common-httpclient4.5.10连接池,模拟一下请求,这里设置:

connManager.setMaxTotal(5);
connManager.setDefaultMaxPerRoute(5);

因为连接就是一组 ip:port 抽象,所以在模拟代码里获取一下本地的连接端口,在main测试中模拟20个请求,发送请求后休眠一会,推进连接过期,服务端关闭连接:

BasicHttpContext httpContext = new BasicHttpContext();
final long l = System.currentTimeMillis();
try {
    response = httpClient.execute(httpPost, httpContext);
    local = JSONObject.parseObject(JSON.toJSONString(httpContext.getAttribute("http.connection"))).getString("localPort");
    HttpEntity httpEntity = response.getEntity();
    result = EntityUtils.toString(httpEntity, "utf-8");
    System.out.println("normal" + " : " + l + " : " + result + " : " + local);
} catch (Exception e) {
    System.out.println("error " + " : " + l + " : " + local);
    e.printStackTrace();
}
-----------------------------------------------------------------------------------
public static void main(String[] args) {
        try {
            IntStream.range(1, 20).parallel().sorted().forEach(e -> {
                try {
                    sendPost("http://127.0.0.1:9000/receive", "{}");
                    TimeUnit.MILLISECONDS.sleep(995L);
                } catch (Exception exception) {
                    exception.printStackTrace();
                }
            });
        } catch (Exception e) {
            e.printStackTrace();
        }
    }

在这里插入图片描述
可以看到本地和服务器共有5个连接,51573,51574,51575,51576,51577,其中有两笔异常数据报错:NoHttpResponseException,其中75,76两个端口连接后续没有被继续使用,怀疑已经被关闭了看一下抓包数据:
在这里插入图片描述
这两个端口处确实已经四次挥手关闭连接了,看样子应该是连接已不可用,发起请求时返回的错误。如果连接不可用,连接池会清除掉不可用连接,重新创建新的连接。再看下httpclient默认的保活策略:
在这里插入图片描述
问题也复现了,相关原因也基本确定了,解决办法就是保证conn的可用性:
1.关闭keep-alive,设置header Connection close
2.客户端实现一下保活策略,时间小于服务端的keep-alive-timeout,连接提前过期释放;当然应该配置合理的连接池大小,不必过大
3.客户端发生此类错误时,应该启用重试机制

;