论坛首页 综合技术论坛

nginx强行屏蔽——微软(BING),无语。。。

浏览 2808 次
精华帖 (0) :: 良好帖 (0) :: 新手帖 (0) :: 隐藏帖 (0)
作者 正文
   发表时间:2009-08-26   最后修改:2009-08-26
微软(BING)完全不遵守robots规则
以下是我的robots文件
User-agent: * 
Disallow: /


结果在我的日志里却发现有大量的:
[26/Aug/2009:15:23:02 +0800] "GET /xxxxxx HTTP/1.0" 302 165 "-" "msnbot/2.0b (+http://search.msn.com/msnbot.htm)" 65.55.106.115
[26/Aug/2009:15:23:08 +0800] "GET /xxxxxx HTTP/1.0" 302 165 "-" "msnbot/2.0b (+http://search.msn.com/msnbot.htm)" 65.55.106.155
[26/Aug/2009:15:23:29 +0800] "GET /xxxxxx HTTP/1.0" 302 165 "-" "msnbot/2.0b (+http://search.msn.com/msnbot.htm)" 65.55.106.137
[26/Aug/2009:15:23:30 +0800] "GET /xxxxxx HTTP/1.0" 302 165 "-" "msnbot/2.0b (+http://search.msn.com/msnbot.htm)" 65.55.207.95
[26/Aug/2009:15:23:31 +0800] "GET /xxxxxx HTTP/1.0" 302 165 "-" "msnbot/2.0b (+http://search.msn.com/msnbot.htm)" 65.55.106.159
[26/Aug/2009:15:23:34 +0800] "GET /xxxxxx HTTP/1.0" 302 165 "-" "msnbot/2.0b (+http://search.msn.com/msnbot.htm)" 65.55.106.211
[26/Aug/2009:15:23:59 +0800] "GET /xxxxxx HTTP/1.0" 302 165 "-" "msnbot/2.0b (+http://search.msn.com/msnbot.htm)" 65.55.106.227
[26/Aug/2009:15:23:59 +0800] "GET /xxxxxx HTTP/1.0" 302 165 "-" "msnbot/2.0b (+http://search.msn.com/msnbot.htm)" 65.55.106.227
[26/Aug/2009:15:23:59 +0800] "GET /xxxxxx HTTP/1.0" 302 165 "-" "msnbot/2.0b (+http://search.msn.com/msnbot.htm)" 65.55.106.232
[26/Aug/2009:15:23:59 +0800] "GET /xxxxxx HTTP/1.0" 302 165 "-" "msnbot/2.0b (+http://search.msn.com/msnbot.htm)" 65.55.106.182



从日志来看,bing算法相当差,爬行的频率相当高。

这对于我这种动态的应用简直就是一个噩梦,无奈只能强行屏蔽

服务器使用的是nginx。
在配置文件中,添加如下代码:
                if ($http_user_agent ~ (msnbot) )
                {
                        return 404;
                }



没想到大名鼎鼎的微软,居然也如此无赖


再次来到bing.com
输入
site:我的服务器的域名

可以看到已经没有快照了,虽然有大量的地址。。。。
论坛首页 综合技术版

跳转论坛:
Global site tag (gtag.js) - Google Analytics