自动检测文件编码

对于抓取的网页处理时，我们往往要判断它的字符编码，今天发现了python的一个自动判断编码的lib，特与众分享.

名称：chardet

介绍：Character encoding auto-detection in Python 2 and 3. As smart as your browser. Open source.

 

下载地址：

http://pypi.python.org/pypi/chardet/1.0.1

文档地址：

http://chardet.feedparser.org/docs/index.html

 

文档目录：

 

 

• Frequently asked questions
  • What is character encoding?
  • What is character encoding auto-detection?
  • Isn’t that impossible?
  • Who wrote this detection algorithm?
  • Yippie! Screw the standards, I’ll just auto-detect everything!
  • Why bother with auto-detection if it’s slow, inaccurate, and non-standard?
• Supported encodings
• Usage
  • Basic usage
  • Advanced usage
• How it works
  • UTF-n with a BOM
  • Escaped encodings
  • Multi-byte encodings
  • Single-byte encodings
  • windows-1252
• Revision history
• Terms of use

basic usage：

>>> import urllib
>>> rawdata = urllib.urlopen('http://yahoo.co.jp/').read()

>>> import chardet
>>> chardet.detect(rawdata)

{'encoding': 'EUC-JP', 'confidence': 0.99}