闲来无聊,用python抓取天气信息,简单就是美啊

/**
* 作者：张荣华
* 日期：2009-02-26
**/

晚上闲来无聊,于是用python写了一个抓取天气信息的脚本,您要是不想去网站上看天气,每天开机的时候运行一下脚本就完事了

同学们请看

#encoding:UTF-8
'''
@author: ahuaxuan (张荣华)
@date: 2009-02-06
'''
import re
from httplib import HTTPConnection
from datetime import datetime
import urllib
import urllib2
import sys

header = {}
header['User-Agent'] = 'Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.0.3) Gecko/2008092510 Ubuntu/8.04 (hardy) Firefox/3.0.3'
header['Accept-Language'] = 'en-us,en;q=0.7,zh-cn;q=0.3'
header['Accept-Encoding'] = 'UTF-8'
header['Accept-Charset'] = 'ISO-8859-1,utf-8;q=0.7,*;q=0.7'
header['Keep-Alive'] = '300'
header['Connection'] = 'keep-alive'
header['Referer'] = 'http://www.weather.com.cn/'

def postData(url, body):  
    
    opener = urllib2.build_opener()
    urllib2.install_opener(opener)
    req = urllib2.Request(url = url, data = urllib.urlencode(body), headers=header)
    u = urllib2.urlopen(req)
    
    htmlSource = u.read()
   
    return htmlSource

def getData(url):
    
    req = urllib2.Request(url,headers = header)
    res = urllib2.urlopen(req)
    html = res.read()
    
    res.close()
    
    return html

todayPattern = '<div class="box_contenttodayinwea" id="c_1_1">[\s]*?'+\
                '<p>[\s]*?<span>[\s\S]*?</span>[\s]*?' + \
                '<em><strong>(?P<wea>[\s\S]*?)</strong></em>[\s]*?' + \
                '<em class="no_today">(?P<temp>[\s\S]*?)</em>[\s]*?' + \
                '<em>(?P<wind>[\s\S]*?)</em><br/>[\s]*?</p>'
                
futurePattern = '<div class="fut_weatherbox7">[\s]*?' + \
                    '<h3>(?P<date>[\s\S]*?)</h3>[\s]*?'+ \
                    '<p>(?P<aa>[\s\S]*?)</p>[\s]*?'+ \
                    '<h4 class="temp00_dn">(?P<wea>[\s\S]*?)</h4>[\s]*?'+ \
                    '<h4 class="temp01_dn">(?P<tempH>[\s\S]*?)</h4>[\s]*?'+ \
                    '<h4 class="temp02_dn">(?P<tempL>[\s\S]*?)</h4>[\s]*?'+ \
                    '<h4 class="temp03_dn"><a name="sk">(?P<wind>[\s\S]*?)</a></h4>[\s]*?'+ \
                '</div>'

'''
return the json format data of weather
http://search.weather.com.cn/static/url.php
'''
def getWeather(cityName, url):
    body = {}
    body['cityinfo'] = cityName
    text = postData(url, body)
    
    pattern = re.compile("<meta http-equiv=\"refresh\" content=\"0;URL=([\\s\\S]*?)\">")
    rlst = pattern.findall(text)
    
    text = getData(rlst[0])
    fieldList = ['wind', 'wea', 'temp']
    lst = parserWeaPattern(fieldList, text, todayPattern)
    
    fieldList2 = ['wind', 'wea', 'date', 'tempH', 'tempL']
    lst2 = parserWeaPattern(fieldList2, text, futurePattern)
    
    print u"--------------今日天气:-------------".encode("GBK")
    for aa in lst:
        for key,value in aa.items():
            print value
        
    print u"\r\n--------------未来几天:--------------".encode("GBK")
    for aa in lst2:
        print '%s,%s,%s,%s,%s'%(aa['date'],aa['wind'],aa['wea'],aa['tempH'],aa['tempL'])
    return lst

def parserWeaPattern(fieldList, text, pattern):  
    
    list = []
    p = re.compile(pattern)
    iterator = p.finditer(text)
    for matcher in iterator:
        data = {}
        for field in fieldList:
            data[field] = matcher.group(field).encode("GBK")
        list.append(data)
    return list
       
if __name__ == '__main__':
    
    #也可以用区号
    city = u'上海'.encode("GBK")
    reload(sys)
    sys.setdefaultencoding('UTF-8')
    
    print city
    getWeather(city, 'http://search.weather.com.cn/static/url.php')
    

运行后的结果为:

上海
--------------今日天气:-------------
多云
14℃/ 6℃
东南风3-4级

--------------未来几天:--------------
7日星期六,东南风3-4级,多云,高温：15℃,低温：7℃
8日星期日,东南风4-5级,多云转阴,高温：14℃,低温：9℃
9日星期一,微风,晴,高温：10℃,低温：4℃
10日星期二,微风,晴,高温：16℃,低温：9℃
11日星期三,微风转北风3-4级,晴,高温：21℃,低温：14℃
12日星期四,北风3-4级转微风,晴转中雨,高温：21℃,低温：16℃

评论

5 楼 ^=^ 2012-03-25  

http://search.weather.com.cn/static/url.php
这个URL已经失效了，请问楼主如何找到新的URL？

4 楼 ahuaxuan 2009-04-09  

stylecn 写道

谢谢，不过我运行时出现了错误：
unexpected character after line continuation character

看来它页面的格式变化了，所以只能修改一下正则表达式了，呵呵，这种通过分析页面得到数据的方法最大的问题就是一旦页面的组织方式变化，就无法正常得到页面上的数据了。呵呵，只能根据它的新页面来修改正则表达式了

3 楼 stylecn 2009-04-09  

谢谢，不过我运行时出现了错误：
unexpected character after line continuation character

2 楼 ahuaxuan 2009-02-20  

好的,谢谢,header部分是我随便写的,其实只要一个user-agent和是否gzip的就可以了

1 楼 bit_kevin 2009-02-20  

urllib2的 头部参数keep-alive是不支持的