ruby多线程下载

多线程批量下载http://blog.ntjy.net上的文章。此方法也可以用于批量下载图片......

 

  require "open-uri"

  def download
    if $*[0] == nil or $*[1] == nil or $*[2] == nil
      abort "用法示例：ruby #$0 开始数　结束数　存放的目录 EX:如ruby #$0 200 300 d:\\ "
    end
    time = Time.now
    threads = []
    for i in $*[0]..$*[1]
      exports = "http://blog.ntjy.net/my_blogs/" + i.to_s
      threads << Thread.new(i) do |thread|
        begin
          data = open(exports){|f|f.read}
          open("#{$*[2]}\\#{thread}.html","wb"){|f|f.write(data)}
          print thread, "-"
        rescue
        end
      end
    end
    threads.each {|thr| thr.join}
    puts " 下载完成，共耗时：#{Time.now - time}秒"
  end

 

输入如下命令即可：

 

ruby script/server 40000 40010 d:\

 

示例输出：

 

40004-40000-40009-40001-40003-40005-40006-40008-40010-40007-下载完成，共耗时：0.329秒 

 

评论

2 楼 gigix 2009-12-14  

(40000..40010).each{|i| system "wget http://blog.ntjy.net/my_blogs/#{i} -O /my_blogs/#{i}.html"}

一分钟的事…

1 楼 juliancg65 2009-12-14  

 
  #####################################################
  # Download Picture Tool by jenas
  #
  #####################################################
  require "open-uri"
  
  i=0
  time = Time.now
  threads = []
  imgurls = []
  exports = "http://www.iteye.com" #任何URL地址
  filepath = "C:\\TMP\\" #保存本地图片地址
  data = open(exports){|f|f.read}
  data.scan(/<img src="(.*?)"/) { |x| imgurls << x if !imgurls.include?(x)  } #检查是否有重复内容
  
  for tmp in imgurls  
    i = i+1
    threads << Thread.new(i) do |thread|
      begin
        img = (open(tmp.to_s){|f| f.read }).to_s
        open(filepath+"#{thread}.jpg","wb"){|f|f.write(img)}
        print thread, "-completed"
      rescue
        puts "Error! with", thread
      end
    end
  end
  
  threads.each {|thr| thr.join}
  puts " 下载完成，共耗时：#{Time.now - time}秒"

补充下楼主对于批量下载图片的内容，呵呵。支持下楼主。有点粗糙，大家不要P我！