编写不易，转载请注明（http://shihlei.iteye.com/blog/2067688）！
一 概述
HttpClient是最常见的Http请求工具，很好的封装 Http请求，响应，Cookies操作，适合网络爬虫处理静态页面及其他资源。
HttpCient能自动处理302客户端重定向，这个很喜欢！
HttpClient4.3.3 API较之前的版本有不少改变，本文描述请求响应，设置cookies，设置代理等方法。
二 版本
<dependency> <groupId>org.apache.httpcomponents</groupId> <artifactId>httpclient</artifactId> <version>4.3.3</version> </dependency>
三 样例
（1）请求响应
/**
* 发送get请求
*
* @param url
* @throws Exception
*/
public byte[] sendGetRequest(String url) throws Exception {
HttpClient httpClient = HttpClients.createDefault();
HttpGet httpGet = new HttpGet(url);
System.out.println("Send Get Request : " + url);
return sendRequest(httpClient,httpGet);
}
/**
* 发送post 请求
*
* @param request
* @throws Exception
*/
public byte[] sendPostRequest(String url, Map<String, String> params) throws Exception {
HttpClient httpClient = HttpClients.createDefault();
HttpPost httpPost = new HttpPost(url);
// 设置参数
if (params != null && params.size() > 0) {
List<NameValuePair> nvps = new ArrayList<NameValuePair>();
for (Entry<String, String> param : params.entrySet()) {
nvps.add(new BasicNameValuePair(param.getKey(), param.getValue()));
}
httpPost.setEntity(new UrlEncodedFormEntity(nvps));
}
System.out.println("Send Post Request : " + url + " , " + params);
return sendRequest(httpClient,httpPost);
}
// 底层请求获得响应数据
private byte[] sendRequest(HttpClient httpClient ,HttpRequestBase request) throws Exception {
CloseableHttpResponse response = httpclient.execute(request);
// 获取返回的状态列表
StatusLine statusLine = response.getStatusLine();
System.out.println("StatusLine : " + statusLine);
if (statusLine.getStatusCode() == HttpStatus.SC_OK) {
try {
// 获取response entity
HttpEntity entity = response.getEntity();
System.out.println("getContentType ： " + entity.getContentType().getValue());
System.out.println("getContentEncoding ： " + entity.getContentEncoding().getValue());
// 读取响应内容
byte[] responseBody = EntityUtils.toByteArray(entity);
// 关闭响应流
EntityUtils.consume(entity);
return responseBody;
} finally {
response.close();
}
}
return null;
}
（2）超时和重定向
在execute前针对Request设置超时的Config
//设置超时
private void setTimeOut(HttpRequestBase request, int timeout) {
RequestConfig requestConfig = RequestConfig.custom().setConnectionRequestTimeout(timeout)
.setConnectTimeout(timeout).setSocketTimeout(timeout).setRedirectsEnabled(true).build();
request.setConfig(requestConfig);
}
（3）代理
在创建HttpClient实例的时候，设置代理
//获得设置代理的httpClent实例
private CloseableHttpClient getHttpClientWithProxy(HttpProxy proxy){
HttpHost proxyHost = new HttpHost(proxy.getHost(), proxy.getPort());
CredentialsProvider credentialsProvider = new BasicCredentialsProvider();
credentialsProvider.setCredentials(new AuthScope(proxy.getHost(), proxy.getPort()),
new UsernamePasswordCredentials(proxy.getUser(), proxy.getPassword()));
return HttpClients.custom().setProxy(proxyHost).setDefaultCredentialsProvider(credentialsProvider)
.build();
}
这里有个辅助类：
package x.http.simple;
/**
* Http代理
*
* @author shilei
*
*/
public class HttpProxy {
private String proxy = "http";
private String host;
private int port;
private String user;
private String password;
public String getProxy() {
return proxy;
}
public void setProxy(String proxy) {
this.proxy = proxy;
}
public String getHost() {
return host;
}
public void setHost(String host) {
this.host = host;
}
public int getPort() {
return port;
}
public void setPort(int port) {
this.port = port;
}
public String getUser() {
return user;
}
public void setUser(String user) {
this.user = user;
}
public String getPassword() {
return password;
}
public void setPassword(String password) {
this.password = password;
}
}
（4）Cookies：可以用于认证数据设置
1）全局Cookies设置：在创建HttpClient时
/**
* 设置全局Cookies
*
* @param cookies
* @return
*/
public HttpClient setGlobeCookieHttpClient(Map<String, String> cookies) {
HttpClient client = null;
if (cookies != null && cookies.size() > 0) {
BasicCookieStore cookieStore = new BasicCookieStore();
for (Entry<String, String> c : cookies.entrySet()) {
BasicClientCookie cookie = new BasicClientCookie(c.getKey(), c.getValue());
cookie.setDomain("sina.com");
cookie.setPath("/");
cookieStore.addCookie(cookie);
}
client = HttpClientBuilder.create().setDefaultCookieStore(cookieStore).build();
} else {
client = HttpClients.createDefault();
}
return client;
}
2）请求Cookies设置
法一：直接在请求头添加
/**
* 设置Cookie方法1
*
* @param request
* @param cookies
*/
public void setRequestCookiesStr(HttpUriRequest request, Map<String, String> cookies) {
StringBuilder cookieStr = new StringBuilder();
if (cookies != null && cookies.size() > 0) {
for (Entry<String, String> cookie : cookies.entrySet()) {
cookieStr.append(cookie.getKey()).append('=').append(cookie.getValue()).append(';');
}
cookieStr.deleteCharAt(cookieStr.length() - 1);
request.addHeader("Cookie", cookieStr.toString());
}
}
法二：请求是通过context设置
/**
* 设置Cookie 方法2
*
* httpclient.execute(request, context) 方法进行添加
*
* @param cookies
* @return
*/
public HttpContext getetRequestCookieContext(Map<String, String> cookies) {
HttpContext localContext = null;
if (cookies != null && cookies.size() > 0) {
BasicCookieStore cookieStore = new BasicCookieStore();
for (Entry<String, String> c : cookies.entrySet()) {
BasicClientCookie cookie = new BasicClientCookie(c.getKey(), c.getValue());
cookie.setDomain("sina.com");
cookie.setPath("/");
cookieStore.addCookie(cookie);
}
localContext = new BasicHttpContext();
localContext.setAttribute(HttpClientContext.COOKIE_STORE, cookieStore);
}
return localContext;
}
3）读取响应的Cookies
/**
* 响应中获取cookie
*
* @param response
*/
public void getResponseCookies(HttpResponse response) {
//法1
HeaderIterator headerIter = response.headerIterator("Set-Cookie");
while (headerIter.hasNext()) {
Object obj = headerIter.next();
if (obj != null) {
String cookie = ((Header) headerIter.next()).getValue();
String cookieRecord = cookie.substring(0, cookie.indexOf(';'));
System.out.println(cookieRecord);
}
}
//法2
HeaderElementIterator headerElementIter = new BasicHeaderElementIterator(response.headerIterator("Set-Cookie"));
while (headerElementIter.hasNext()) {
HeaderElement elem = headerElementIter.nextElement();
System.out.println("Element : " + elem.getName() + " = " + elem.getValue());
NameValuePair[] params = elem.getParameters();
for (int i = 0; i < params.length; i++) {
System.out.println(" Params : " + params[i].getName() + " | " + params[i].getValue());
}
}
}
附录：完整代码
package x.http.simple.httpclient;
import java.util.ArrayList;
import java.util.List;
import java.util.Map;
import java.util.Map.Entry;
import org.apache.http.Header;
import org.apache.http.HeaderElement;
import org.apache.http.HeaderElementIterator;
import org.apache.http.HeaderIterator;
import org.apache.http.HttpEntity;
import org.apache.http.HttpHost;
import org.apache.http.HttpResponse;
import org.apache.http.HttpStatus;
import org.apache.http.NameValuePair;
import org.apache.http.StatusLine;
import org.apache.http.auth.AuthScope;
import org.apache.http.auth.UsernamePasswordCredentials;
import org.apache.http.client.CredentialsProvider;
import org.apache.http.client.HttpClient;
import org.apache.http.client.config.RequestConfig;
import org.apache.http.client.entity.UrlEncodedFormEntity;
import org.apache.http.client.methods.CloseableHttpResponse;
import org.apache.http.client.methods.HttpGet;
import org.apache.http.client.methods.HttpPost;
import org.apache.http.client.methods.HttpRequestBase;
import org.apache.http.client.methods.HttpUriRequest;
import org.apache.http.client.protocol.HttpClientContext;
import org.apache.http.impl.client.BasicCookieStore;
import org.apache.http.impl.client.BasicCredentialsProvider;
import org.apache.http.impl.client.CloseableHttpClient;
import org.apache.http.impl.client.HttpClientBuilder;
import org.apache.http.impl.client.HttpClients;
import org.apache.http.impl.cookie.BasicClientCookie;
import org.apache.http.message.BasicHeaderElementIterator;
import org.apache.http.message.BasicNameValuePair;
import org.apache.http.protocol.BasicHttpContext;
import org.apache.http.protocol.HttpContext;
import org.apache.http.util.EntityUtils;
import x.http.simple.HttpProxy;
public class HttpClientDemo {
private CloseableHttpClient httpclient;
public HttpClientDemo() {
this(null);
}
/**
* 代理方式
*
* @param proxy
*/
public HttpClientDemo(HttpProxy proxy) {
if (proxy != null) {
httpclient = getHttpClientWithProxy(proxy);
} else {
httpclient = HttpClients.createDefault();
}
}
// 获得设置代理的httpClent实例
private CloseableHttpClient getHttpClientWithProxy(HttpProxy proxy) {
HttpHost proxyHost = new HttpHost(proxy.getHost(), proxy.getPort());
CredentialsProvider credentialsProvider = new BasicCredentialsProvider();
credentialsProvider.setCredentials(new AuthScope(proxy.getHost(), proxy.getPort()),
new UsernamePasswordCredentials(proxy.getUser(), proxy.getPassword()));
return HttpClients.custom().setProxy(proxyHost).setDefaultCredentialsProvider(credentialsProvider).build();
}
/**
* 发送get请求
*
* @param url
* @throws Exception
*/
public byte[] sendGetRequest(String url) throws Exception {
HttpGet httpGet = new HttpGet(url);
System.out.println("Send Get Request : " + url);
return sendRequest(httpGet);
}
/**
* 发送post 请求
*
* @param request
* @throws Exception
*/
public byte[] sendPostRequest(String url, Map<String, String> params) throws Exception {
HttpPost httpPost = new HttpPost(url);
// 设置参数
if (params != null && params.size() > 0) {
List<NameValuePair> nvps = new ArrayList<NameValuePair>();
for (Entry<String, String> param : params.entrySet()) {
nvps.add(new BasicNameValuePair(param.getKey(), param.getValue()));
}
httpPost.setEntity(new UrlEncodedFormEntity(nvps));
}
System.out.println("Send Post Request : " + url + " , " + params);
return sendRequest(httpPost);
}
// 底层请求打印页面
private byte[] sendRequest(HttpRequestBase request) throws Exception {
CloseableHttpResponse response = httpclient.execute(request);
// 设置超时
setTimeOut(request, 5000);
// 获取返回的状态列表
StatusLine statusLine = response.getStatusLine();
System.out.println("StatusLine : " + statusLine);
if (statusLine.getStatusCode() == HttpStatus.SC_OK) {
try {
// 获取response entity
HttpEntity entity = response.getEntity();
System.out.println("getContentType ： " + entity.getContentType().getValue());
System.out.println("getContentEncoding ： " + entity.getContentEncoding().getValue());
// 读取响应内容
byte[] responseBody = EntityUtils.toByteArray(entity);
// 关闭响应流
EntityUtils.consume(entity);
return responseBody;
} finally {
response.close();
}
}
return null;
}
// 设置超时重定向
private void setTimeOut(HttpRequestBase request, int timeout) {
RequestConfig requestConfig = RequestConfig.custom().setConnectionRequestTimeout(timeout)
.setConnectTimeout(timeout).setSocketTimeout(timeout).setRedirectsEnabled(true).build();
request.setConfig(requestConfig);
}
/**
* 设置全局Cookies
*
* @param cookies
* @return
*/
public HttpClient setGlobeCookieHttpClient(Map<String, String> cookies) {
HttpClient client = null;
if (cookies != null && cookies.size() > 0) {
BasicCookieStore cookieStore = new BasicCookieStore();
for (Entry<String, String> c : cookies.entrySet()) {
BasicClientCookie cookie = new BasicClientCookie(c.getKey(), c.getValue());
cookie.setDomain("sina.com");
cookie.setPath("/");
cookieStore.addCookie(cookie);
}
client = HttpClientBuilder.create().setDefaultCookieStore(cookieStore).build();
} else {
client = HttpClients.createDefault();
}
return client;
}
/**
* 设置Cookie方法1
*
* @param request
* @param cookies
*/
public void setRequestCookiesStr(HttpUriRequest request, Map<String, String> cookies) {
StringBuilder cookieStr = new StringBuilder();
if (cookies != null && cookies.size() > 0) {
for (Entry<String, String> cookie : cookies.entrySet()) {
cookieStr.append(cookie.getKey()).append('=').append(cookie.getValue()).append(';');
}
cookieStr.deleteCharAt(cookieStr.length() - 1);
request.addHeader("Cookie", cookieStr.toString());
}
}
/**
* 设置Cookie 方法2
*
* httpclient.execute(request, context) 方法进行添加
*
* @param cookies
* @return
*/
public HttpContext getetRequestCookieContext(Map<String, String> cookies) {
HttpContext localContext = null;
if (cookies != null && cookies.size() > 0) {
BasicCookieStore cookieStore = new BasicCookieStore();
for (Entry<String, String> c : cookies.entrySet()) {
BasicClientCookie cookie = new BasicClientCookie(c.getKey(), c.getValue());
cookie.setDomain("sina.com");
cookie.setPath("/");
cookieStore.addCookie(cookie);
}
localContext = new BasicHttpContext();
localContext.setAttribute(HttpClientContext.COOKIE_STORE, cookieStore);
}
return localContext;
}
/**
* 响应中获取cookie
*
* @param response
*/
public void getResponseCookies(HttpResponse response) {
// 法1
HeaderIterator headerIter = response.headerIterator("Set-Cookie");
while (headerIter.hasNext()) {
Object obj = headerIter.next();
if (obj != null) {
String cookie = ((Header) headerIter.next()).getValue();
String cookieRecord = cookie.substring(0, cookie.indexOf(';'));
System.out.println(cookieRecord);
}
}
// 法2
HeaderElementIterator headerElementIter = new BasicHeaderElementIterator(response.headerIterator("Set-Cookie"));
while (headerElementIter.hasNext()) {
HeaderElement elem = headerElementIter.nextElement();
System.out.println("Element : " + elem.getName() + " = " + elem.getValue());
NameValuePair[] params = elem.getParameters();
for (int i = 0; i < params.length; i++) {
System.out.println(" Params : " + params[i].getName() + " | " + params[i].getValue());
}
}
}
public void shutdown() throws Exception {
httpclient.close();
}
public static void main(String[] args) throws Exception {
String url = "http://www.google.com.hk";
HttpClientDemo httpclient = new HttpClientDemo();
byte[] getResponse = httpclient.sendGetRequest(url);
System.out.println("Get Body : " + new String(getResponse, "utf-8"));
byte[] postResponse = httpclient.sendPostRequest(url, null);
System.out.println("Get Body : " + new String(postResponse, "utf-8"));
httpclient.shutdown();
}
}
相关推荐
1. 使用连接池：HttpClient支持连接池，可提高并发性能并减少资源消耗。 2. 正确处理异常：对可能出现的网络异常进行捕获和处理，避免程序意外终止。 3. 使用异步模式：HttpClient 4.3.3支持异步操作，对于大量并发...
《HttpClient 4.3.3：构建高效网络通信的核心库》 HttpClient 4.3.3 是 Apache 组织提供的一款强大的 HTTP 客户端库，它为开发者提供了丰富的功能，便于实现各种复杂的HTTP请求和处理。HttpClient 4.x 系列在功能、...
在使用HttpClient 4.3.3时，开发者需要注意以下几点： 1. **版本兼容性**：确保项目中的其他依赖与HttpClient 4.3.3兼容，避免出现版本冲突。 2. **正确配置连接池**：合理设置连接池的最大连接数和超时时间，防止...
10. **性能优化**：HttpClient 4.3.3 版本针对内存管理、线程安全和资源释放等方面进行了优化，确保在高并发场景下的稳定运行。 在实际开发中，HttpClient 4.3.3 可以与各种 Java 框架（如 Spring、Struts 等）无缝...
在实际项目中，HTTPClient 4.3.3和HTTPCore 4.3.2的组合使用，可以构建出高效、灵活的网络通信模块。开发者可以根据需要定制请求处理逻辑，比如自定义认证机制、错误处理策略等。同时，这两个库都遵循Apache软件许可...
在使用HTTPClient和HTTPCore时，需要注意的是，这两个库都需要正确配置连接管理，以确保有效利用系统资源，避免过多的并发连接导致的问题。此外，对于安全性要求较高的应用，应考虑使用SSL/TLS加密，设置信任的证书...
httpclient-4.3.3.jar 三、使用方法 使用HttpClient发送请求、接收响应很简单，一般需要如下几步即可。 1. 创建HttpClient对象。 2. 创建请求方法的实例，并指定请求URL。如果需要发送GET请求，创建HttpGet对象；...
10. **拦截器**：HttpClient 允许使用`HttpRequestInterceptor`和`HttpResponseInterceptor`来在请求发送前或响应接收后添加自定义逻辑，这对于日志记录、请求修改或特殊处理非常有用。 HttpClient 4.5.3的文档...
本项目涉及的是使用Apache HttpClient 4.3.3和Jsoup 1.7.3这两个开源库来实现对CSDN（China Software Developer Network）网站的模拟登录，并在成功登录后抓取用户的个人主页内容。下面将详细阐述这两个库的功能以及...
本篇文章将深入探讨如何使用`HttpClient`来获取网络资源，以及处理返回的XML数据。 首先，我们需要了解`HttpClient`的基本用法。在Java中，创建一个`HttpClient`实例是获取网络资源的第一步。`HttpClient`提供了...
在这个场景中，我们使用HttpClient来调用远程接口，从电信公司的网站获取可用的新手机号码信息，并将这些数据存储到本地文件中。以下是对这个过程的详细解释： 1. **HttpClient的基本使用**： HttpClient提供了一...
GET用于获取资源，POST用于提交数据。比如： ```java HttpGet httpGet = new HttpGet("http://example.com"); HttpPost httpPost = new HttpPost("http://example.com"); ``` 3. **设置请求头**：你可以通过`...
标题中的“httpclient-4.3.1.jar”，“httpcore-4.3.2.jar”和“httpmime-4.3.5.jar”分别指的是Apache HttpClient库的不同组件和版本。这些是Java开发中用于HTTP通信的重要库，由Apache软件基金会提供。下面将详细...
【标题】"httpclient 静态化网站 project"是一个基于Apache HttpClient库的项目，旨在实现网站内容的抓取和静态化。HttpClient是一个强大的HTTP客户端编程工具包，它允许开发者在Java环境中执行HTTP请求，获取网页...
在4.3.3版本中，HttpClient对SSL/TLS支持进行了增强，改进了连接池管理，并修复了一些已知问题，提高了整体的稳定性和兼容性。 对于POST请求来说，HttpClient是理想的工具。POST请求常用于向服务器提交数据，例如...
Android 中使用 HttpClient 获取网络数据的实例 Android 平台中，HttpClient 是一个常用的网络请求组件，通过它可以与远程服务器进行交互，获取所需的数据。在 Android 应用程序中，我们可以使用 HttpClient 来获取...
在这个例子中，我们使用HttpClient来调用远程接口，从电信公司的网站获取可用手机号码的数据，并将其存储到本地文件系统。以下是对这个过程的详细解释： 1. **HttpClient的引入与配置**： 首先，我们需要在项目中...
ATL自带 HTTP Client访问 HTTP Server的Server侧代码。
通过httpclient post去获取,response返回码是302，返回的code放在header的Location中。 请求的时候client_id，response_type，redirect_uri，state拼接在url后面，account和password放在body表单（x-...
ETL KETTLE 中利用Httpclient 调用webservice接口获取XML数据，并解析XML 数据。 完整的KTR实例