聊聊HttpClient的重试机制

序

本文主要研究一下HttpClient的重试机制

HttpRequestRetryHandler

org/apache/http/client/HttpRequestRetryHandler.java

public interface HttpRequestRetryHandler {

    /**
     * Determines if a method should be retried after an IOException
     * occurs during execution.
     *
     * @param exception the exception that occurred
     * @param executionCount the number of times this method has been
     * unsuccessfully executed
     * @param context the context for the request execution
     *
     * @return {@code true} if the method should be retried, {@code false}
     * otherwise
     */
    boolean retryRequest(IOException exception, int executionCount, HttpContext context);

}
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17

HttpRequestRetryHandler接口定义了retryRequest方法，它接收IOException、executionCount及context，然后判断是否可以重试

DefaultHttpRequestRetryHandler

org/apache/http/impl/client/DefaultHttpRequestRetryHandler.java

@Contract(threading = ThreadingBehavior.IMMUTABLE)
public class DefaultHttpRequestRetryHandler implements HttpRequestRetryHandler {

    public static final DefaultHttpRequestRetryHandler INSTANCE = new DefaultHttpRequestRetryHandler();

    /** the number of times a method will be retried */
    private final int retryCount;

    /** Whether or not methods that have successfully sent their request will be retried */
    private final boolean requestSentRetryEnabled;

    private final Set> nonRetriableClasses;

    /**
     * Create the request retry handler using the specified IOException classes
     *
     * @param retryCount how many times to retry; 0 means no retries
     * @param requestSentRetryEnabled true if it's OK to retry requests that have been sent
     * @param clazzes the IOException types that should not be retried
     * @since 4.3
     */
    protected DefaultHttpRequestRetryHandler(
            final int retryCount,
            final boolean requestSentRetryEnabled,
            final Collection> clazzes) {
        super();
        this.retryCount = retryCount;
        this.requestSentRetryEnabled = requestSentRetryEnabled;
        this.nonRetriableClasses = new HashSet>();
        for (final Class clazz: clazzes) {
            this.nonRetriableClasses.add(clazz);
        }
    }

    /**
     * Create the request retry handler using the following list of
     * non-retriable IOException classes: 

     * 

     * 
InterruptedIOException
     * 
UnknownHostException
     * 
ConnectException
     * 
SSLException
     * 
     * @param retryCount how many times to retry; 0 means no retries
     * @param requestSentRetryEnabled true if it's OK to retry non-idempotent requests that have been sent
     */
    @SuppressWarnings("unchecked")
    public DefaultHttpRequestRetryHandler(final int retryCount, final boolean requestSentRetryEnabled) {
        this(retryCount, requestSentRetryEnabled, Arrays.asList(
                InterruptedIOException.class,
                UnknownHostException.class,
                ConnectException.class,
                SSLException.class));
    }

    /**
     * Create the request retry handler with a retry count of 3, requestSentRetryEnabled false
     * and using the following list of non-retriable IOException classes: 

     * 

     * 
InterruptedIOException
     * 
UnknownHostException
     * 
ConnectException
     * 
SSLException
     * 
     */
    public DefaultHttpRequestRetryHandler() {
        this(3, false);
    }
    /**
     * Used {@code retryCount} and {@code requestSentRetryEnabled} to determine
     * if the given method should be retried.
     */
    @Override
    public boolean retryRequest(
            final IOException exception,
            final int executionCount,
            final HttpContext context) {
        Args.notNull(exception, "Exception parameter");
        Args.notNull(context, "HTTP context");
        if (executionCount > this.retryCount) {
            // Do not retry if over max retry count
            return false;
        }
        if (this.nonRetriableClasses.contains(exception.getClass())) {
            return false;
        }
        for (final Class rejectException : this.nonRetriableClasses) {
            if (rejectException.isInstance(exception)) {
                return false;
            }
        }
        final HttpClientContext clientContext = HttpClientContext.adapt(context);
        final HttpRequest request = clientContext.getRequest();

        if(requestIsAborted(request)){
            return false;
        }

        if (handleAsIdempotent(request)) {
            // Retry if the request is considered idempotent
            return true;
        }

        if (!clientContext.isRequestSent() || this.requestSentRetryEnabled) {
            // Retry if the request has not been sent fully or
            // if it's OK to retry methods that have been sent
            return true;
        }
        // otherwise do not retry
        return false;
    }

    /**
     * @return {@code true} if this handler will retry methods that have
     * successfully sent their request, {@code false} otherwise
     */
    public boolean isRequestSentRetryEnabled() {
        return requestSentRetryEnabled;
    }

    /**
     * @return the maximum number of times a method will be retried
     */
    public int getRetryCount() {
        return retryCount;
    }

    /**
     * @since 4.2
     */
    protected boolean handleAsIdempotent(final HttpRequest request) {
        return !(request instanceof HttpEntityEnclosingRequest);
    }

    /**
     * @since 4.2
     *
     * @deprecated (4.3)
     */
    @Deprecated
    protected boolean requestIsAborted(final HttpRequest request) {
        HttpRequest req = request;
        if (request instanceof RequestWrapper) { // does not forward request to original
            req = ((RequestWrapper) request).getOriginal();
        }
        return (req instanceof HttpUriRequest && ((HttpUriRequest)req).isAborted());
    }

}
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149

DefaultHttpRequestRetryHandler实现了HttpRequestRetryHandler接口，其无参构造器默认将InterruptedIOException、UnknownHostException、ConnectException、SSLException设定为不重试的异常，默认retryCount为3，requestSentRetryEnabled为false；其retryRequest方法先判断executionCount是否超出retryCount，接着判断异常类型是否是不重试的异常类型，若request为aborted则返回false，若request非HttpEntityEnclosingRequest则表示幂等请求，返回true，若请求未完全发送则返回true，其余的默认返回false。

RetryExec

org/apache/http/impl/execchain/RetryExec.java

@Contract(threading = ThreadingBehavior.IMMUTABLE_CONDITIONAL)
public class RetryExec implements ClientExecChain {

    private final Log log = LogFactory.getLog(getClass());

    private final ClientExecChain requestExecutor;
    private final HttpRequestRetryHandler retryHandler;

    public RetryExec(
            final ClientExecChain requestExecutor,
            final HttpRequestRetryHandler retryHandler) {
        Args.notNull(requestExecutor, "HTTP request executor");
        Args.notNull(retryHandler, "HTTP request retry handler");
        this.requestExecutor = requestExecutor;
        this.retryHandler = retryHandler;
    }

    @Override
    public CloseableHttpResponse execute(
            final HttpRoute route,
            final HttpRequestWrapper request,
            final HttpClientContext context,
            final HttpExecutionAware execAware) throws IOException, HttpException {
        Args.notNull(route, "HTTP route");
        Args.notNull(request, "HTTP request");
        Args.notNull(context, "HTTP context");
        final Header[] origheaders = request.getAllHeaders();
        for (int execCount = 1;; execCount++) {
            try {
                return this.requestExecutor.execute(route, request, context, execAware);
            } catch (final IOException ex) {
                if (execAware != null && execAware.isAborted()) {
                    this.log.debug("Request has been aborted");
                    throw ex;
                }
                if (retryHandler.retryRequest(ex, execCount, context)) {
                    if (this.log.isInfoEnabled()) {
                        this.log.info("I/O exception ("+ ex.getClass().getName() +
                                ") caught when processing request to "
                                + route +
                                ": "
                                + ex.getMessage());
                    }
                    if (this.log.isDebugEnabled()) {
                        this.log.debug(ex.getMessage(), ex);
                    }
                    if (!RequestEntityProxy.isRepeatable(request)) {
                        this.log.debug("Cannot retry non-repeatable request");
                        throw new NonRepeatableRequestException("Cannot retry request " +
                                "with a non-repeatable request entity", ex);
                    }
                    request.setHeaders(origheaders);
                    if (this.log.isInfoEnabled()) {
                        this.log.info("Retrying request to " + route);
                    }
                } else {
                    if (ex instanceof NoHttpResponseException) {
                        final NoHttpResponseException updatedex = new NoHttpResponseException(
                                route.getTargetHost().toHostString() + " failed to respond");
                        updatedex.setStackTrace(ex.getStackTrace());
                        throw updatedex;
                    }
                    throw ex;
                }
            }
        }
    }

}
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69

RetryExec实现了ClientExecChain接口，其execute方法会循环执行requestExecutor.execute，它catch了IOException，对于retryHandler.retryRequest(ex, execCount, context)返回true的，会在通过RequestEntityProxy.isRepeatable(request)判断一下是否是可重复读取的request，不是则抛出NonRepeatableRequestException；对于retryHandler.retryRequest返回false的则针对NoHttpResponseException重新包装一下，将targetHost体现在message里头然后重新抛出

HttpEntityEnclosingRequest

org/apache/http/HttpEntityEnclosingRequest.java

public interface HttpEntityEnclosingRequest extends HttpRequest {

    /**
     * Tells if this request should use the expect-continue handshake.
     * The expect continue handshake gives the server a chance to decide
     * whether to accept the entity enclosing request before the possibly
     * lengthy entity is sent across the wire.
     * @return true if the expect continue handshake should be used, false if
     * not.
     */
    boolean expectContinue();

    /**
     * Associates the entity with this request.
     *
     * @param entity the entity to send.
     */
    void setEntity(HttpEntity entity);

    /**
     * Returns the entity associated with this request.
     *
     * @return entity
     */
    HttpEntity getEntity();

}
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27

HttpEntityEnclosingRequest定义了getEntity、setEntity、expectContinue方法，它的子类有HttpPut、HttpPost、HttpPatch、HttpDelete等

RequestEntityProxy.isRepeatable

org/apache/http/impl/execchain/RequestEntityProxy.java

class RequestEntityProxy implements HttpEntity  {

    private final HttpEntity original;
    private boolean consumed = false;

    public boolean isConsumed() {
        return consumed;
    }

    static boolean isRepeatable(final HttpRequest request) {
        if (request instanceof HttpEntityEnclosingRequest) {
            final HttpEntity entity = ((HttpEntityEnclosingRequest) request).getEntity();
            if (entity != null) {
                if (isEnhanced(entity)) {
                    final RequestEntityProxy proxy = (RequestEntityProxy) entity;
                    if (!proxy.isConsumed()) {
                        return true;
                    }
                }
                return entity.isRepeatable();
            }
        }
        return true;
    }
}
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25

RequestEntityProxy提供了静态方法isRepeatable用于判断该request的entity是否可以重复读取，对于非HttpEntityEnclosingRequest的返回true，是HttpEntityEnclosingRequest类型的话则判断entity.isRepeatable()，若entity是RequestEntityProxy类型的，则通过RequestEntityProxy.isConsumed来判断

entity.isRepeatable()

public interface HttpEntity {

    /**
     * Tells if the entity is capable of producing its data more than once.
     * A repeatable entity's getContent() and writeTo(OutputStream) methods
     * can be called more than once whereas a non-repeatable entity's can not.
     * @return true if the entity is repeatable, false otherwise.
     */
    boolean isRepeatable();

    //......
}    
1
2
3
4
5
6
7
8
9
10
11
12

HttpEntity接口定义了isRepeatable方法，用于表示entity的content及OutputStream是否可以读写多次。其实现类里头，BufferedHttpEntity、ByteArrayEntity、EntityTemplate、FileEntity、SerializableEntity、StringEntity为true，BasicHttpEntity、InputStreamEntity、StreamingHttpEntity为false

小结

HttpRequestRetryHandler接口定义了retryRequest方法，它接收IOException、executionCount及context，然后判断是否可以重试

DefaultHttpRequestRetryHandler实现了HttpRequestRetryHandler接口，其无参构造器默认将InterruptedIOException、UnknownHostException、ConnectException、SSLException设定为不重试的异常，默认retryCount为3，requestSentRetryEnabled为false；其retryRequest方法先判断executionCount是否超出retryCount，接着判断异常类型是否是不重试的异常类型，若request为aborted则返回false，若request非HttpEntityEnclosingRequest则表示幂等请求，返回true，若请求未完全发送则返回true，其余的默认返回false。

RetryExec实现了ClientExecChain接口，其execute方法会循环执行requestExecutor.execute，它catch了IOException，对于retryHandler.retryRequest(ex, execCount, context)返回true的，会在通过RequestEntityProxy.isRepeatable(request)判断一下是否是可重复读取的request，不是则抛出NonRepeatableRequestException

DefaultHttpRequestRetryHandler针对不是幂等请求的HttpEntityEnclosingRequest类型(HttpPut、HttpPost、HttpPatch、HttpDelete)，不会重试；若retryHandler.retryRequest返回可以重试，RetryExec还有一个repeatable的判断，BufferedHttpEntity、ByteArrayEntity、EntityTemplate、FileEntity、SerializableEntity、StringEntity为true，BasicHttpEntity、InputStreamEntity、StreamingHttpEntity为false