java网络编程之Http多线程下载应用实例

本demo 通过RandomAccessFile, URLConnection和多线程机制实现了Http下载功能. 从 这里 可以下载到完整的java代码工程:  http://download.csdn.net/detail/hejiangtao /4029935.  相对于别的网上的例子来看, 本demo 是可运行的, 可以判断网络资源是否支持分段下载. 你是否遇到了java下载的图片 显示不出来或者RAR解压不了的情况, 可以参考本demo的解决方案

设计思路:

1. 首先读取文件的长度, 并判断网站是否支持分段下载

2. 如果支持分段下载则创建多个线程同时下载该文件,否则使用单线程下载

3. 在各个线程中,分别使用 RandomAccessFile对象写入对应的文件位置

具体实现解析:

      1. 为了方便理解,先把util类贴出来.

SysValue.java:

在实际应用中,这些参数应该可以在界面配置或者配置文件配置.

package
 com.ross.httpdownload.util;
/**
 * Author: Jiangtao He; Email: [e-mail]
 * Date: 2012-1-18
 * Since: MyJavaExpert v1.0
 * Description:
 */
public
 
class
 SysValue
{
    //constant values
    //maximum number of the download thread
    public
 
static
 
int
 MAX_Num_Of_Thread = 
3
;
    //download file store path
    public
 
static
 String Save_As_Path = 
"E:/myspace/download/"
;
    //buffer size
    public
 
static
 
int
 Buffer_Size = 
1024
;
}

HttpProcess.java:

将Http相关的公共操作放在这个类里面,避免写重复代码.

package
 com.ross.httpdownload.util;
import
 java.io.IOException;
import
 java.net.*;
/**
 * Author: Jiangtao He; Email: [e-mail]
 * Date: 2012-1-18
 * Since: MyJavaExpert v1.0
 * Description: This class will provide the common process of http
 */
public
 
class
 HttpProcess
{
    /**
     * Author: Jiangtao He; Email: [e-mail]
     * Description: get an object of opened URL connection
     * @param sURL: completed URL string
     * @return oHttpURLCon: an object of opened URL connection
     * @throws IOException
     */
    public
 URLConnection  getURLConnection(String sURL) 
throws
 IOException
    {
        URLConnection  oURLCon = null
;
        // replace space with '+' in the URL string
        if
 (
null
 != sURL)
        {
            sURL = sURL.replaceAll("\\s"
, 
"+"
);
            // generate URL
            URL oURL = new
 URL(sURL);
            oURLCon = (URLConnection) oURL.openConnection();
        }
        return
 oURLCon;
    }

}

    2. 看我们的多线程下载实现

a. 首先看下如何判断网站资源是否支持分段下载

很 多网络资源,即使在java里面设置了http的InputStream流的范围后, 取的的流的长度还是整个文件的长度. 针对这种情况我就认为该资源不支持分段下载,也就是无法多线程下载了(使用原来的方式下载你就发现下载的图片只能显示一半,或者不能显示,RAR也是损坏 的, 字节不对吗 :)). 我也试过通过使用InputStream 流的skip方法跳过前面内容的方式实现多线程,但是事实证明,skip太不靠谱,跳的步长并不是一定的,哎什么玩意,只好放弃.

/**
 * Author: Jiangtao He; Email: [e-mail]
 * Description: Check whether the server support multiple thread download or not
 * @param sURL: the download file URL
 * @return bRet: true - support, false - not support
 * @throws IOException
 */
public
 
boolean
 isSupportMultipleThreadDown(String sURL) 
throws
 IOException
{
    boolean
 bRet = 
true
;
   // get URL connection
    URLConnection oURLCon = oHttpProcess.getURLConnection(sURL);
   // set the resource range
    oURLCon.setRequestProperty("Range"
, 
"bytes="
 + 
0
 + 
"-"
 + 
1023
);

    if
 (
1024
 < oURLCon.getContentLength())
    {
        bRet = false
;
    }
    return
 bRet;
}

b. 然后我们看下如何,将文件分段下载

1) 首先是预处理

先是做了个简单的判断,防止线程过多或者小于1; 然后判断网络资源是否支持分段下载,如果不支持则将线程数重置为1; 在获取整个文件长度后, 使用总长度除以线程数得出每个线程负责下载的字节范围.

2) 使用线程池管理并启动多线程下载

DownloadTread就是我们的线程下载实现类, 后面会单独说明. 众所周知java下标是从0开始的,所以各个线程分配字节范围的时候,要考虑进去. 将对应分配的字节范围,URL,文件名信息设置给DownloadTread对象后,使用ExecutorService启动线程.

3)当然不要忘记在finally里面调用线程池的关闭方法.

/**
 * Author: Jiangtao He; Email: [e-mail]
 * Description: download tool entry method
 * @param sURL: the completed URL of the file, which is wanted to download.
 * @param Num: Number of concurrent download thread
 * @param sSaveAsName: the the file name saved in disk, like: myphoto.jpg
 */
public
 
void
 download(String sURL, 
int
 iNumOfThread, String sSaveAsName)
{
    long
 lFileLen = 
0
;
    long
 lFileSegementLen = 
0
;
    long
 iLeft = 
0
;
    URLConnection oURLCon = null
;
    String sFullFileName = SysValue.Save_As_Path + sSaveAsName;

    if
 (iNumOfThread > SysValue.MAX_Num_Of_Thread)
    {
        iNumOfThread = SysValue.MAX_Num_Of_Thread;
    }
    if
 (iNumOfThread < 
1
)
    {
        iNumOfThread = 1
;
    }

    // build the thread pool to manage the thread
    ExecutorService oExecService = Executors
            .newFixedThreadPool(iNumOfThread);
    try
    {
        if
 (!isSupportMultipleThreadDown(sURL))
        {
            System.out
                    .println("This site do not support multiple thread download, reset the thread to 1"
);
            iNumOfThread = 1
;
        }
        // get URL connection
        oURLCon = oHttpProcess.getURLConnection(sURL);
        // get the length of the file
        lFileLen = oURLCon.getContentLength();
        // split the length of the file according to the number of thread.
        lFileSegementLen = lFileLen / iNumOfThread;
        iLeft = lFileLen % iNumOfThread;

        for
 (
int
 i = 
0
; i < iNumOfThread; i++)
        {
            // initial the thread data
            DownloadThread oDownloadThread = new
 DownloadThread();
            oDownloadThread.setIThreadId(i);
            // Start index of the current thread
            oDownloadThread.setLStartIndex(i * lFileSegementLen);

            // last thread will take all the rest content
            if
 (i == (iNumOfThread - 
1
))
            {
                oDownloadThread.setLEndIndex((i + 1
) * lFileSegementLen
                        + iLeft - 1
);
            }
            else
            {
                oDownloadThread
                        .setLEndIndex((i + 1
) * lFileSegementLen - 
1
);
            }
            // set save file name with path
            oDownloadThread.setSFullFileName(sFullFileName);
            // set URL
            oDownloadThread.setSFileURL(sURL);
            // execute the download thread
            oExecService.execute(oDownloadThread);
        }
        System.out.println("the file is stored as \""
 + sFullFileName
                + "\"."
);
        System.out.println("the file length is "
 + lFileLen + 
"."
);
    }
    catch
 (IOException e)
    {
        System.out.println("download the file failed, IO Exception"
                + e.getMessage());
        e.printStackTrace();
    }
    finally
    {
        oExecService.shutdown();
    }
}

c.最后就是我们的核心下载线程实现类了

1) DownloadTread类实现了Runnalbe抽象接口, 实现了其run()方法.

2) 两个关键的地方一是通过使用URLConnection 的setRequestProperty方法设置本线程读取的流的范围, 另外一个是通过RandomAccessFile 的seek方法或者当前线程写入文件的位置, 一个字节都不能错哦,错了你的文件就有差错了, 还是要记住java下标是从0开始的.

3)然后通过循环从URLConnection的InputStream里面通过 buffer一组一组的读出来, 通过RandomAccessFile 对象一组一组的写入到目标文件里面(代码里面判断实际读取长度部分是不需要的, 可以去掉, 是我探索不支持分段下载网络资源的多线程下载方式时留下的).

4)需要提一下的是run()方法是没有输入参数的,所以我们通过增加类字段的方式将参数传入该方法.

package
 com.ross.httpdownload;
import
 java.io.*;
import
 java.net.*;
import
 java.text.*;
import
 java.util.Date;
import
 com.ross.httpdownload.util.*;
/**
 * Author: Jiangtao He; Email: [e-mail]
 * Date: 2012-1-18
 * Since: MyJavaExpert v1.0
 * Description: download thread implementation.
 */
public
 
class
 DownloadThread 
implements
 Runnable
{
    private
 String sFileURL;
    private
 
long
 lStartIndex;
    private
 
long
 lEndIndex;
    private
 
int
 iThreadId;
    private
 String sFullFileName;
    private
 HttpProcess oHttpProcess;
    /**
     * Description: default constructor, it is used for initializing the fields.
     */
    public
 DownloadThread()
    {
        oHttpProcess = new
 HttpProcess();
    }
    /**
     * Author: Jiangtao He; Email: [e-mail]
     * Description: it will implement the download logic
     */
    public
 
void
 run()
    {
        System.out.println("The download thread "
                + this
.iThreadId
                + " is started at "
                + (new
 SimpleDateFormat(
"yyyy-MM-dd HH:mm:ss"
))
                        .format(new
 Date()));
        System.out.println("The download index of "
 + 
this
.iThreadId + 
" is "
                + this
.lStartIndex + 
" to "
 + 
this
.lEndIndex);

        // content length
        long
 lContentlen = 
0
;
        // used to store the new url connection
        URLConnection oURLCon = null
;
        // used to store the input stream of the response
        BufferedInputStream oBIn = null
;
        // used to store the output writer
        RandomAccessFile oRAFile = null
;
        // create the buffer
        byte
[] bBuffer = 
new
 
byte
[SysValue.Buffer_Size];

        // get URL connection
        try
        {
            // create a link for each thread
            oURLCon = oHttpProcess.getURLConnection(this
.sFileURL);
            // allow the user interaction, for example: a verify pop window
            oURLCon.setAllowUserInteraction(true
);
            // set the resource range
            oURLCon.setRequestProperty("Range"
, 
"bytes="
 + 
this
.lStartIndex
                    + "-"
 + 
this
.lEndIndex);

            // get the url connection input stream
            oBIn = new
 BufferedInputStream(oURLCon.getInputStream());

            // initialize the random access file object
            oRAFile = new
 RandomAccessFile(
this
.sFullFileName, 
"rw"
);
            oRAFile.seek(this
.lStartIndex);

            // read the stream from http connection
            int
 iLen = 
0
;
            int
 iActualLen = 
0
;
            while
 (iActualLen < (
this
.lEndIndex - 
this
.lStartIndex + 
1
))
            {
                iLen = oBIn.read(bBuffer, 0
, SysValue.Buffer_Size);
                if
 (-
1
 == iLen)
                {
                    break
;
                }

                // write the read data to file
                oRAFile.write(bBuffer, 0
, iLen);

                // move the position mark
                iActualLen = iActualLen + iLen;
            }
            System.out.println("Thread "
 + 
this
.iThreadId
                    + " download is finished, totaly: "
 + iActualLen);
        }
        catch
 (IOException e)
        {
            System.out.println("download the file failed, IO Exception"
                    + e.getMessage());
            e.printStackTrace();
        }
        finally
        {
            if
 (
null
 != oRAFile)
            {
                try
                {
                    oRAFile.close();
                }
                catch
 (IOException e)
                {
                    System.out
                            .println("close object of RandomAccessFile failed, IO Exception"
                                    + e.getStackTrace());
                }
            }

            if
 (
null
 != oBIn)
            {
                try
                {
                    oBIn.close();
                }
                catch
 (IOException e)
                {
                    System.out
                            .println("close input stream of http connection failed, IO Exception"
                                    + e.getStackTrace());
                }
            }
        }
        System.out.println("The download thread "
                + this
.iThreadId
                + " is ended at "
                + (new
 SimpleDateFormat(
"yyyy-MM-dd HH:mm:ss"
))
                        .format(new
 Date()));
    }
   //省略了set/get方法
}

来测试下我们的实现:

看下我们的main函数, 我们就通过3个例子来测试我们的实现: 一个可以多线程下载的zip文件,一个可以多线程下载的图片,一个不能多线程下载图片.

MyMain.java:

package
 com.ross.httpdownload;
import
 com.ross.httpdownload.util.*;
/**
 * Author: Jiangtao He; Email: [e-mail]
 * Date: 2012-1-19
 * Since: MyJavaExpert v1.0
 * Description: Test the download tool class
 */
public
 
class
 MyMain
{
    public
 
static
 
void
 main(String[] args)
    {
        // set system configuration, in real software, it can be configured
        // through GUI, file,etc.
        SysValue.MAX_Num_Of_Thread = 3
;
        SysValue.Save_As_Path = "E:/myspace/download/"
;
        SysValue.Buffer_Size = 1024
;
        //file web address
        //String sFileURL = "http://xinsheng-image.huawei.com/cn/forumimage/showimage-991441-20990f56c5b11261fa63e969d8d5580c-self.jpg";
        //String sFileURL = "http://lh5.googleusercontent.com/-F7a0loCDyoQ/AAAAAAAAAAI/AAAAAAAAAFY/TrDxSDdQuhQ/s512-c/photo.jpg";
        //String sFileURL = "http://xinsheng-image.huawei.com/cn/forumimage/showimage-816607-f61c8fe14fc359d49b044376a0956acd-self.jpg";
        //String sFileURL = "http://3.bp.blogspot.com/-oVO24VW8F6M/TxJ9O0opP3I/AAAAAAAAANY/anpu8S4FAC4/s640/100_9223.JPG";
        String sFileURL = "http://hi.csdn.net/attachment/201201/1/0_1325433530Bg5e.gif"
;
        sFileURL = "http://west263.newhua.com:82/down/jia-audio-converter.zip"
;
        //file save as name
        String sFileName = sFileURL.substring(sFileURL.lastIndexOf("/"
) + 
1
);

        // create a object of download tool
        HttpDownloadTool oHttpDownloadTool = new
 HttpDownloadTool();
        // download the file
        oHttpDownloadTool.download(sFileURL, 3
, sFileName);
    }
}

a. 可以多线程下载的zip文件测试

URL: http://west263.newhua.com:82/down/jia-audio-converter.zip

线程数: 3

控制台打印信息:

the file is stored as "E:/myspace/download/jia-audio-converter.zip".
the file length is 5720120.
The download thread 1 is started at 2012-01-21 00:49:23
The download index of 1 is 1906706 to 3813411
The download thread 0 is started at 2012-01-21 00:49:23
The download index of 0 is 0 to 1906705
The download thread 2 is started at 2012-01-21 00:49:23
The download index of 2 is 3813412 to 5720119
Thread 0 download is finished, totaly: 1906706
The download thread 0 is ended at 2012-01-21 00:58:09
Thread 1 download is finished, totaly: 1906706
The download thread 1 is ended at 2012-01-21 01:06:17
Thread 2 download is finished, totaly: 1906708
The download thread 2 is ended at 2012-01-21 01:07:05

下载的文件抓图:

b.可以多线程下载的图片测试

URL: http://hi.csdn.net/attachment/201201/1/0_1325433530Bg5e.gif

线程数: 3

控制台打印信息:

the file is stored as "E:/myspace/download/0_1325433530Bg5e.gif".
the file length is 23304.
The download thread 2 is started at 2012-01-21 02:03:00
The download thread 0 is started at 2012-01-21 02:03:00
The download index of 2 is 15536 to 23303
The download index of 0 is 0 to 7767
The download thread 1 is started at 2012-01-21 02:03:00
The download index of 1 is 7768 to 15535
Thread 2 download is finished, totaly: 7768
The download thread 2 is ended at 2012-01-21 02:03:01
Thread 1 download is finished, totaly: 7768
The download thread 1 is ended at 2012-01-21 02:03:01
Thread 0 download is finished, totaly: 7768
The download thread 0 is ended at 2012-01-21 02:03:02

下载的文件抓图:

c. 不能多线程下载图片

URL: http://3.bp.blogspot.com/-oVO24VW8F6M/TxJ9O0opP3I/AAAAAAAAANY/anpu8S4FAC4/s640/100_9223.JPG

线程数: 3

控制台打印信息:

This site do not support multiple thread download, reset the thread to 1
the file is stored as "E:/myspace/download/100_9223.JPG".
the file length is 170398.
The download thread 0 is started at 2012-01-21 02:06:54
The download index of 0 is 0 to 170397
Thread 0 download is finished, totaly: 170398
The download thread 0 is ended at 2012-01-21 02:06:55

下载的文件抓图:

到这里整个demo完成了.

注: 转载请注明出处: http://hejiangtao.iteye.com ,  用于商业得给我分成

评论

1 楼 砚台观月 2016-07-15  

你好，例子还有吗，我想要份学习看下。提供的链接找不到了。