Here's the list of the User-Agent fields that we know are used by robots (such as web crawlers). There are currently three entries ending with an asterisk, which indicates that the name is a prefix. This prefix is used by many User-Agent's, and all of them are robots. For example, there are many different versions of the "FAST-WebCrawler" agent.
| "DIIbot/1.2 http://www.findsame.com/robot.html" |
| AcoiRobot/1.0 libwww/5.3.2 | |
| Acoon Robot v1.01 (www.acoon.de) | |
| AgentName/0.1 libwww-perl/5.50 | |
| AlkalineBOT/1.4 (1.4.0326.0 RTM) | |
| AltaVista Intranet* | |
| AnzwersCrawl/2.0 (anzwerscrawl@anzwers.com.au; http://faq.anzwers.com.au/anzwerscrawl.html) | |
| appie/1.1 | |
| Arachnoidea (arachnoidea@euroseek.com) | |
| ArchitextSpider | |
| asterias/2.0 | |
| Autonomy Spider |
| BaiDuSpider | |
| beholder (www.vigiltech.com/esensedisclaim.html) | |
| BlogBot/1.1 | |
| bumblebee@relevare.com |
| CherryPickerElite/1.0 |  |
| CherryPickerSE/1.0 | |
| cosmos/0.7_(mihai.preda@xyleme.com) | |
| cosmos/0.8_(robot@xyleme.com) | |
| Crawl_Application | |
| Crescent Internet ToolPak HTTP OLE Control v.1.0 | |
| daypopbot/0.2 | |
| DIIbot/1.2 http://www.findsame.com/robot.html |
| EmailCollector/1.0 | |
| EmailSiphon | |
| EmailWolf 1.00 | |
| Explorer www@openxxx.net | |
| ExtractorPro | |
| FAST-WebCrawler/* | |
| fastlwspider/1.0 | |
| fido/1.0 Harvest/1.4.pl2 |
| GAIS Robot/1.0B2 | |
| gazz/2.1 (gazz@nttrd.com) | |
| gigabaz/3.14 (baz@gigabaz.com; http://gigabaz.com/gigabaz/) | |
| Googlebot-for-IDG/2.1 (+http://www.googlebot.com/bot.html) | |
| Googlebot/2.1 (+http://googlebot.com/bot.html) | |
| Googlebot/2.1 (+http://www.googlebot.com/bot.html) | |
| Gulliver/1.3 | |
| Gulper Web Bot 0.2.4 (www.ecsl.cs.sunysb.edu/~maxim/cgi-bin/Link/GulperBot) |
| hcat/1.0 | |
| HLoader | |
| HomePageSearch(hpsearch.uni-trier.de) | |
| htdig/3.1.2 (webmaster@box.sk) | |
| htdig/3.1.5 (unconfigured@htdig.searchengine.maintainer) | |
| http://pawel.dyndns.org/ pol76@wanadoo.fr | |
| httpGlooton/1.0 |
| ia_archiver | |
| IncyWincy data gatherer(webmaster@loopimprovements.com,http://www.loopimprovements.com/robot.html) | |
| InfoNaviRobot(F107) | |
| Infoseek Sidewinder/0.9 | |
| Inktomi Search | |
| InternetAmi IOR/0.2 (http://internetami.se/ior.html) | |
| InternetLinkAgent/3.1 | |
| InternetSeer.com |
| JennyBot/0.1 | |
| jokescan/0.08 |
| Kenjin Spider | |
| KIT-Fireball/2.0 | |
| KIT_Fireball/2.0 | |
| kobot/5.2.8 libwww/5.2.8 |
| larbin_1.2.2 larbin1.2.2@somewhere.com | |
| larbin_1.2.2 nobody@nowhere | |
| larbin_2.1.0 larbin2.1.0@somewhere.com | |
| larbin_2.1.1 larbin2.1.1@somewhere.com | |
| larbin_devel sebastien.ailleret@inria.fr | |
| larbinDeViennot Laurent.Viennot@inria.fr | |
| LEIA/3.01pr (LEIAcrawler; leia@gseek.com; http://www.gseek.com) | |
| LexiBot/1.00 | |
| libWeb/clsHTTP -- hiongun@kt.co.kr | |
| Linkbot | |
| LinkLint-checkonly/2.1 | |
| Links (0.95; NetBSD 1.5U i386) | |
| LinkWalker | |
| LNSpiderguy | |
| lwp-trivial/1.27 | |
| Lycos_Spider_(modspider) | |
| Lycos_Spider_(T-Rex) | |
| Lycos_Spider_(T-Rex)/3.0 |
| MARS SV | |
| Mata Hari/2.00 | |
| MedicalMatrix | |
| Mercator-1.0 | |
| Mercator-1.1 | |
| Microsoft URL Control - 6.00.8169 | |
| MIIxpc/4.2 | |
| moget/2.1 (moget@goo.ne.jp) | |
| Mozilla/2.0 (compatible; Ask Jeeves) | |
| Mozilla/2.0 (compatible; EZResult -- Internet Search Engine) | |
| Mozilla/2.0 (compatible; NEWT ActiveX; Win32) | |
| Mozilla/2.0 (compatible; T-H-U-N-D-E-R-S-T-O-N-E) | |
| Mozilla/3.0 (compatible; Indy Library) | |
| Mozilla/3.0 (compatible; MuscatFerret/1.4.1; olly@muscat.co.uk) | |
| Mozilla/3.0 (compatible; MuscatFerret/1.5.2; olly@muscat.co.uk) | |
| Mozilla/3.0 (compatible; MuscatFerret/1.5.3; olly@muscat.co.uk) | |
| Mozilla/3.0 (Slurp/cat; slurp@inktomi.com; http://www.inktomi.com/slurp.html) | |
| Mozilla/3.0 (Slurp/si; slurp@inktomi.com; http://www.inktomi.com/slurp.html) | |
| Mozilla/4.0 (compatible; FastCrawler3, support-fastcrawler3@fast.no) | |
| Mozilla/4.0 (compatible; MSIE 4.01; Windows NT; MS Search 4.0 Robot) Microsoft | |
| Mozilla/4.0 (compatible; MuscatFerret/2.0; http://www.webtop.com/) | |
| Mozilla/4.0 compatible ZyBorg/1.0 (ZyBorg@WISEnutbot.com; http://www.WISEnutbot.com) | |
| Mozilla/4.04 [de] (Win95; I ;Kolibri gncwebbot) | |
| Mozilla/4.04 [de] (Win95; I ;Nav; Kolibri gncwebbot) | |
| Mozilla/4.1 | |
| MuscatFerret |
| NationalDirectory-WebSpider/1.3 |
| Openfind data gatherer, Openbot/3.0+(robot-response@openfind.com.tw;+http://www.openfind.com.tw/robot.html) | |
| Openfind piranha,Shark/0.95h+(tjsheu@gais.cs.ccu.edu.tw;+http://www.openfind.com/) |
| PlantyNet_WebRobot_V1.9 dhkang@plantynet.com | |
| PM WebSpyder V1.0 | |
| polybot 1.0 | |
| psbot/0.1 (+http://www.picsearch.com/bot.html) | |
| psbot/0.1 (+http://www.picsearch.org/bot.html) |
| QUOSA |
| RaBot/1.0 Agent-admin/webmaster@krivet2.or.kr | |
| RealNamesBot/2.7 | |
| Robozilla/1.0 |
| Scooter-3.0.d1 | |
| scooter-3.0.DY | |
| Scooter-3.0.EU | |
| Scooter-3.0.FS | |
| Scooter-3.0.g36-47 | |
| Scooter-3.0.g60-71 | |
| Scooter-3.0.g72-83 | |
| Scooter-3.0.g84-95 | |
| Scooter-3.0.s1 | |
| Scooter-W3-1.0 | |
| Scooter/1.0 | |
| Scooter/1.0 scooter@pa.dec.com | |
| Scooter/1.1 (custom) | |
| Scooter/2.0 G.R.A.B. V1.1.0 | |
| Scooter/2.0 G.R.A.B. X2.0 | |
| search.at V1.2 | |
| sexsearcher | |
| Slurp.so/1.0 (slurp@inktomi.com; http://www.inktomi.com/slurp.html) | |
| Slurp/2.0-condor_daily_west (slurp@inktomi.com; http://www.inktomi.com/slurp.html) | |
| Slurp/2.0-condor_hourly (slurp@inktomi.com; http://www.inktomi.com/slurp.html) | |
| Slurp/2.0-Redtail (slurp@inktomi.com; http://www.inktomi.com/slurp.html) | |
| Slurp/cat (slurp@inktomi.com; http://www.inktomi.com/slurp.html) | |
| Slurp/si (slurp@inktomi.com; http://www.inktomi.com/slurp.html) | |
| Slurp/si-emb (slurp@inktomi.com; http://www.inktomi.com/slurp.html) | |
| SlySearch (fitzboy@bmrc.berkeley.edu) | |
| SlySearch (slysearch@slysearch.com) | |
| SlySearch fitzboy@bmrc.berkeley.edu | |
| SlySearch slysearch@slysearch.com | |
| SOFTWING_TEAR_AGENT_1_0 | |
| SpiderMan | |
| Sqworm/2.9.43-BETA | |
| Surfnomore Spider v1.1 | |
| SurfWatchSpider/2.0 | |
| SwishSpider | |
| SwissSearch V1.2 |
| The Informant | |
| tivraSpider/1.0 | |
| tivraSpider/1.0 (crawler@tivra.com) | |
| TridentSpider3 | |
| True_Robot/1.0 libwww/5.2.8 | |
| TV33_Mercator_1-1.0 |
| Ultraseek | |
| URL Checker |
| VbTcpQuery sample (http://www.hexillion.com) | |
| vischeck_spiderBot/0.1libwww-perl/5.48 | |
| vspider | |
| vspider/3.6 |
| WebBandit/2.1 | |
| WebBandit/3.50 | |
| Webbandit/4.00.0 | |
| WebCrawler-AddURL/2.0 | |
| WebCrawler/3.0 Robot libwww/5.0a | |
| WebTrends Link Analyzer | |
| WiseDoc-Robot (Google WAP Proxy/1.0) | |
| WiseWire | |
| WiseWire* |
| x-spyonit/1.1 | |
| Xenu's Link Sleuth 1.1a | |
| xyro_(xcrawler@cosmos.inria.fr) |
| ZyBorg/1.0 (ZyBorg@WISEnutbot.com; http://www.WISEnutbot.com) |
分享到：
发表评论
相关推荐
还在为 user agent swicher 只有IPHONE、IE、search robot这三种烦恼吗？ 导入这个列表吧，包括了目前主流的浏览器标识 使用方法 先安装好,重启火狐浏览器之后，打开菜单栏“工具”“Dafault User Agent”“Edit ...
标题 "robotframework-excellibrary-0.0.2.zip" 提供的是一个名为 ExcelLibrary 的 Robot Framework 模块的特定版本（0.0.2）。Robot Framework 是一个通用的自动化测试框架，它允许用户使用关键词驱动的方法进行...
robotframework-ride-1.5a1.win-amd64 exe
《Python库Robotframework-SSHLibrary 3.5.0rc1详解》 在Python的世界里，库是开发者的重要工具，它们极大地丰富了Python的功能并提高了开发效率。本篇文章将深入探讨一个名为`robotframework-sshlibrary`的Python...
要让robotframework-ide能正常运行，需要wxpython安装到python2.7目录下(注：我的装是python2.7版本) $cd /usr/tools/wxwidgets2.8.7/wxPython $python setup.py install WX_CONFIG=/opt/wx/2.8/bin/wx-config ...
robotframework，官方用户指南完整版，很全面的入门教程
Robot Framework User Guide 2.7.3 是一个详细的文档资源，专门针对 Robot Framework 的一个特定版本——2.7.3。Robot Framework 是一个通用的自动化框架，广泛应用于软件测试和其他重复性任务。这个用户指南提供了...
robotframework-2.9.py64bit.exe
robotframework-excellibrary适配python 3版本，在python3.7.9上安装正常。 下载好压缩包 ->解压 -> 在解压目录的当前文件夹下，打开终端，输入 ： python setup.py install 即可，安装成功后，pip list查看
robotframework-selenium2library-1.7.3.win-amd64.exe
ROBOTIS-OP2是一款人形机器人，它是ROBOTIS-OP系列的升级产品，原名被称为“DARWIN2”或“DARWIN-OP2”。ROBOTIS-OP2与它的前身ROBOTIS-OP（原名“DARwIn-OP”）在外观上保持了较高的一致性，但在计算机性能上进行了...
**FANUC Robot R-30iA 控制装置维修要点详析** FANUC Robot R-30iA 控制装置是工业自动化领域中的一款高性能机器人控制系统，它集成了先进的运动控制技术，广泛应用于制造业的装配、搬运、焊接、喷涂等多个环节。...
Robotframework-AppiumLibrary是基于Robot Framework的自动化测试库，专为移动应用自动化测试而设计。Robot Framework是一个通用的自动化框架，支持关键字驱动测试方法，而AppiumLibrary则是其扩展，用于对接Appium...
Robot Framework Selenium2Library 1.5.0 是一个专门针对Web自动化测试的强大的库，它构建于Robot Framework之上，提供了一套简洁而易用的接口，使得测试人员能够高效地编写自动化测试用例。Robot Framework 是一个...
**Robot Framework RIDE 1.5.2.1：强大的自动化测试编辑器** Robot Framework（RF）是一款开源的、基于关键字驱动的通用自动化框架，它适用于各种自动化测试场景，包括功能测试、验收测试、UI 测试等。RIDE（Robot ...
robort framework RIDE win64,用于64位机。安装64bit的robot framework 需要wxPython2.8的，不然打不开RIDE的快捷方式
标题中的"robotframework-selenium2library-3.0.0.tar.gz"是一个开源自动化测试框架Robot Framework的一个扩展库——Selenium2Library的3.0.0版本的压缩包。Robot Framework是一个通用的、基于关键字驱动的自动化...
robotframework-selenium2library-1.7.4.win-amd64.exe
**PyPI 官网下载 | robotframework-zoomba-2.8.5.tar.gz** `robotframework-zoomba` 是一个基于 Python 的自动化测试框架，它建立在 Robot Framework 之上，专为 Web 应用程序的端到端测试提供了一套全面的解决方案...
标题中的“已经修复安装报错：robotframework-excellibrary-0.0.2.zip”表明这是一个关于Robot Framework的扩展库——ExcelLibrary的修复版本，版本号为0.0.2，且该库在之前的安装过程中遇到了错误，但现在已经被...