网络爬虫工具

http://hi.baidu.com/whhzthfnayhntwe/item/a4f9ae056f08b012cc34eadc

Ruby Web Spidering and Data extraction

Anemone:

http://anemone.rubyforge.org

Example:

Anemone.crawl("http://www.example.com/") do |anemone|   

      anemone.on_every_page do |page|

           puts page.url  

     end

end 

Anemone提供了5个动词

after_crawl - 在crawl完了之后在所有抓取到的页面上运行一个block

focus_crawl - 用一个block去选择每个页面跟随哪些链接

on_every_page - 对每个页面运行block

on_pages_like - 给定一个模式，URL匹配的页面才运行block

skip_links_like - 跳过URL模式匹配的页面

每个page对象包含下面属性

url - page的URL

aliases - 重定位到这个page的URI，或这个page重定位到的页面

headers - HTTP响应头部信息

code - HTTP响应码

doc - 页面的Nokogiri::HTML::Document

links - 页面上的指向同样域名的所有URL数组



---------------------------------------------------------------------------

Mechanize:

http://mechanize.rubyforge.org



examples:

require 'rubygems'

require 'mechanize'

#创建实例

agent = Mechanize.new

#加载网页

page = agent.get("http://www.inruby.com")

#使用Mechanize::Page方法

page.title

page.content_type

page.encoding

page.images

page.links

page.forms

page.frames

page.iframes

page.labels

signup_page = page.link_with(:href =>/signup/).click

#使用Mechanize::Form

u_form = signup_page.form_with(:action =>/users/)

u_form['user[login]'] = 'maiaimi'

u_form['user[password]'] = 'maiami'

u_form['user[password_confirmation]'] = 'maiami'

u_form.submit

---------------------------------------------------------------------------------------------

#example2

This is an example of how to access a login protected site with WWW ::Mechanize. In this example, the login form has two fields named user and password. In other words, the HTML contains the following code:



   1  <input name="user" .../>

   2  <input name="password" .../>



Note that this example also shows how to enable WWW ::Mechanize logging and how to capture the HTML response:



   1  require 'rubygems'

   2  require 'logger'

   3  require 'mechanize'

   4 

   5 agent = WWW::Mechanize.new{|a| a.log = Logger.new(STDERR) }

       agent = Mechanize.new({|a| a.log = Logger.new(STDERR)}

   6  #agent.set_proxy('a-proxy', '8080')

   7  page = agent.get 'http://bobthebuilder.com'

   8 

   9  form = page.forms.first

  10  form.user = 'bob'

  11  form.password = 'password'

  12 

  13  page = agent.submit form

  14 

  15  output = File.open("output.html", "w") { |file|  file << page.body }



Use the search method to scrape the page content. In this example I extract all text contained by span elements, which in turn are contained by a table element having a class attribute equal to ‘list-of-links’:



   1  puts page.search("//table[@class='list-of-links']//span/text()") # do |row|



Mechanize Tips

1. agent alias

irb(main):071:0> Mechanize::AGENT_ALIASES.keys

=> ["Mechanize", "Linux Firefox", "Mac Mozilla", "Linux Mozilla", "Windows IE 6", "iPhone", "Linux Konqueror", "Windows IE 7", "Mac FireFox", "Mac Safari", "Windows Mozilla"]

2. reassign Mechanize's html parser

Mechanize.html_parser = Hpricot 

agent = Mechanize.new   

agent.user_agent_alias = 'Windows IE 7'