BabyDuncan
- 浏览: 584151 次
- 性别:
- 来自: 北京
-
文章分类
最新评论
-
liuzeyuss：
好文章，顶顶
java 线程死锁的检测 -
ysite：
hugh.wang 写道请问你这个能还原吗？短网址用的是HAS ...
Java版短网址(ShortUrl)的算法 -
lv双：
没有注释，看的费劲
java 线程死锁的检测 -
dgj：
好帖子竟然没人顶...
java 线程死锁的检测 -
天空趋虚：
不好,你这只是记录式的博客,不是分享式的博客,对浏览着不友好. ...
一些技术牛人的博客
使用hive分析nginx 日志文件
数据格式
分析共有几种数据格式：
more access.log | awk -F "[\"]" '{print $2}' | awk -F "[?]" '{print $1}'| sort | uniq | sort
共有以下种请求：
****************************************************
stat
api/databus/stat/click
GET /click.html
api/databus/stat/display
GET /display.html
api/databus/stat/game
GET /game.html
ane
api/databus/ane/end
GET /end_ane.html √
api/databus/ane/run
GET /run_ane.html √
api/databus/ane/login
GET /login_ane.html √
api/databus/ane/loader
GET /loader_run.html
***************************************************
GET /click.html
举例：
124.88.232.193 - - [13/Jan/2013:22:59:59 +0800] "GET /click.html?9*925008604*1*nlink*https://itunes.apple.com/cn/app/xue-di-zi-sheng-dan-ban/id588116262?ls=1&mt=8
HTTP/1.1" 301 0 "-"
1.205.1.200 - - [13/Jan/2013:22:59:59 +0800] "GET /click.html?9*838001018*1*nlink*http://itunes.apple.com/cn/app/shi-hun-zhi-ren/id588118382?mt=8 HTTP/1.1" 301 0 "-"
125.85.93.141 - - [13/Jan/2013:22:59:59 +0800] "GET /click.html?9*838000737*1*nlink*https://itunes.apple.com/cn/app/shi-hun-zhi-ren/id588118382?mt=8 HTTP/1.1" 301 0 "-"
——————————————————————————————————————————————————————————————————————————————
GET /click.html HTTP/1.0
62.24.222.131 - - [13/Jan/2013:23:26:18 +0800] "GET /click.html HTTP/1.0" 200 3 "http://tj.pet.mop.com/click.html"
——————————————————————————————————————————————————————————————————————————————
GET /display.html
115.45.201.131 - - [13/Jan/2013:23:00:01 +0800] "GET /display.html?80134|952000101|0|receiver|0|mop HTTP/1.1" 304 0
36.40.191.155 - - [13/Jan/2013:23:00:02 +0800] "GET /display.html?500293|1|0|receiver|0|mop HTTP/1.1" 200 23
116.22.217.54 - - [13/Jan/2013:23:00:05 +0800] "GET /display.html?9|9040010018|0|receiver|0|mop HTTP/1.1" 200 23
——————————————————————————————————————————————————————————————————————————————
GET /end_ane.html
221.194.30.138 - - [13/Jan/2013:22:59:59 +0800] "GET /end_ane.html?
connecttype=wwan&uuid1=d51c772dd73e48c29dd76269e021d418&mac=28E02C12BE8D&detaildevicetype=iPhone3,1&apptype=ios&versionid=1.2.1&deviceversion=6.0.1&i
mei=null&uuid2=bdd0d548eb1edbcd459dbf7f32cc4751&devicetype=iPhone&sdkver=1.0.7&gameid=rrnhzf&jailbroken=false&appid=com.imop.game.pirate2iphone&f=885
001201&domain=x39.nhzf.renren.com&userid=221106049&roleid=295271648616025261&starttime=1358088689&endtime=1358089200&onlinetime=511&launchtime=13580
88689&startcount=1&activetime=511 HTTP/1.1" 200 53 "-"
114.221.147.156 - - [13/Jan/2013:22:59:59 +0800] "GET /end_ane.html?
roleid=295553123592712996&uuid2=null&starttime=1358089089&apptype=null&domain=&endtime=1358089095&uuid1=null&onlinetime=6&devicetype=null&launch
time=1358080763&deviceversion=null&startcount=12&f=838000707&activetime=436&detaildevicetype=null&gameid=ms&jailbroken=null&versionid=1.0.8&connec
ttype=null&userid=497886004&appid=com.msqy.weiduan HTTP/1.1" 200 43 "app:/Loading.swf/[[DYNAMIC]]/1"
175.142.180.206 - - [13/Jan/2013:23:00:00 +0800] "GET /end_ane.html?
connecttype=wifi&uuid1=0d40e54d523a48428ad10ecc5253e153&mac=A467069329AE&detaildevicetype=iPad2,2&apptype=ios&versionid=1.1.2&deviceversion=4.3.1&ime
i=null&uuid2=ee5bdbc81fb02ee3bc0b362a5281a935&devicetype=iPad&sdkver=1.0.8&gameid=csj&jailbroken=false&appid=com.renren.game.csj&f=null&domain=x21.cs
j.renren.com&userid=null&roleid=null&starttime=1358089192&endtime=1358089200&onlinetime=8&launchtime=1358089192&startcount=1&activetime=8 HTTP/1.1"
200 46 "-"
——————————————————————————————————————————————————————————————————————————————
GET /favicon.ico HTTP/1.1
120.33.159.158 - - [13/Jan/2013:23:00:45 +0800] "GET /favicon.ico HTTP/1.1" 404 564 "-"
14.118.120.122 - - [13/Jan/2013:23:01:25 +0800] "GET /favicon.ico HTTP/1.1" 404 564 "-"
116.230.225.94 - - [13/Jan/2013:23:05:25 +0800] "GET /favicon.ico HTTP/1.1" 404 564 "-"
——————————————————————————————————————————————————————————————————————————————
GET /game.html
222.209.150.169 - - [13/Jan/2013:22:59:59 +0800] "GET /game.html?0|932000101|3400|renren|0|http://x20.war.renren.com/ HTTP/1.1" 304 0
222.209.150.169 - - [13/Jan/2013:22:59:59 +0800] "GET /game.html?0|932000101|2400|renren|1|http://x20.war.renren.com/ HTTP/1.1" 304 0
182.242.227.214 - - [13/Jan/2013:22:59:59 +0800] "GET /game.html?-1|855004201|3400|0|0|0 HTTP/1.1" 200 20
——————————————————————————————————————————————————————————————————————————————
GET / HTTP/1.1
113.108.11.50 - - [13/Jan/2013:23:09:21 +0800] "GET / HTTP/1.1" 200 20
112.90.84.224 - - [13/Jan/2013:23:09:21 +0800] "GET / HTTP/1.1" 200 20
123.190.215.78 - - [13/Jan/2013:23:13:21 +0800] "GET / HTTP/1.1" 200 0 "-"
——————————————————————————————————————————————————————————————————————————————
GET http://tj.imop.com/click.html
112.97.192.31 - - [13/Jan/2013:23:11:07 +0800] "GET /click.html?9*855000865*1*nlink*http://itunes.apple.com/cn/app/lu-bu-yu-diao-chan/id555912419?mt=8 HTTP/1.1" 301 0
"http://c.w.inmobi.com/ios35x.html?http://tj.imop.com/click.html?9*855000865*1*nlink*http://itunes.apple.com/cn/app/lu-bu-yu-diao-chan/id555912419?mt=8"
118.116.81.28 - - [13/Jan/2013:23:11:39 +0800] "GET /click.html?9*838001010*1*nlink*http://itunes.apple.com/cn/app/shi-hun-zhi-ren/id588118382?mt=8 HTTP/1.1" 301 0
"http://c.w.inmobi.com/ios35x.html?http://tj.imop.com/click.html?9*838001010*1*nlink*http://itunes.apple.com/cn/app/shi-hun-zhi-ren/id588118382?mt=8"
125.126.32.95 - - [13/Jan/2013:23:12:00 +0800] "GET /click.html?9*838001017*1*nlink*http://itunes.apple.com/cn/app/shi-hun-zhi-ren/id588118382?mt=8 HTTP/1.1" 301 0
"http://c.w.inmobi.com/ios35x.html?http://tj.imop.com/click.html?9*838001017*1*nlink*http://itunes.apple.com/cn/app/shi-hun-zhi-ren/id588118382?mt=8"
——————————————————————————————————————————————————————————————————————————————
GET http://tj.pet.mop.com/click.html
119.97.151.160 - - [13/Jan/2013:23:54:42 +0800] "GET http://tj.pet.mop.com/click.html?60101*150101*1*nlink*http://itunes.apple.com/cn/app/id588116262?mt=8 HTTP/1.1" 301 0
"-"
——————————————————————————————————————————————————————————————————————————————
GET http://tj.pet.mop.com/end_ane.html
216.36.142.199 - - [13/Jan/2013:23:03:52 +0800] "GET http://tj.pet.mop.com/end_ane.html?
appid=elite.gen.ms.shzr&gameid=ms&uuid1=2ced6028e80a433886db6db82e9bfe8e&devicetype=iPhone&onlinetime=4&endtime=1357999139&deviceversion=6.0&version
id=8.1.1&starttime=1357999135&launchtime=1357998799&f=838000707&detaildevicetype=iPhone3,1&userid=467415015&activetime=220&apptype=ios&jailbroken=f
alse&startcount=5&uuid2=f80462b8c55ac2a64c8812779b40afbe&domain=&connecttype=wifi&roleid=295834598569411941 HTTP/1.1" 200 43 "app:/com.renren.gameskit"
216.36.142.199 - - [13/Jan/2013:23:07:07 +0800] "GET http://tj.pet.mop.com/end_ane.html?
appid=elite.gen.ms.shzr&gameid=ms&uuid1=2ced6028e80a433886db6db82e9bfe8e&devicetype=iPhone&onlinetime=157&endtime=1358089586&deviceversion=6.0&versi
onid=8.1.1&starttime=1358089429&launchtime=1358089429&f=838000707&detaildevicetype=iPhone3,1&userid=null&activetime=157&apptype=ios&jailbroken=fals
e&startcount=1&uuid2=f80462b8c55ac2a64c8812779b40afbe&domain=&connecttype=wifi&roleid=null HTTP/1.1" 200 43 "app:/com.renren.gameskit"
174.93.120.82 - - [13/Jan/2013:23:08:11 +0800] "GET http://tj.pet.mop.com/end_ane.html?
appid=elite.gen.ms.shzr&gameid=ms&uuid1=2ced6028e80a433886db6db82e9bfe8e&devicetype=iPhone&onlinetime=51&endtime=1358089676&deviceversion=6.0&versio
nid=8.1.1&starttime=1358089625&launchtime=1358089429&f=838000707&detaildevicetype=iPhone3,1&userid=454219948&activetime=208&apptype=ios&jailbroken=
false&startcount=2&uuid2=f80462b8c55ac2a64c8812779b40afbe&domain=&connecttype=wifi&roleid=292456898848886914 HTTP/1.1" 200 43 "app:/com.renren.gameskit
——————————————————————————————————————————————————————————————————————————————
GET http://tj.pet.mop.com/login_ane.html
216.36.142.199 - - [13/Jan/2013:23:07:22 +0800] "GET http://tj.pet.mop.com/login_ane.html?
appid=elite.gen.ms.shzr&gameid=ms&uuid1=2ced6028e80a433886db6db82e9bfe8e&devicetype=iPhone&deviceversion=6.0&versionid=8.1.1&f=838000707&detaildevicet
ype=iPhone3,1&userid=454219948&apptype=ios&jailbroken=false&uuid2=f80462b8c55ac2a64c8812779b40afbe&domain=&connecttype=wifi&roleid=292456898848886
914 HTTP/1.1" 200 43 "app:/com.renren.gameskit"
174.93.120.82 - - [13/Jan/2013:23:08:25 +0800] "GET http://tj.pet.mop.com/login_ane.html?
appid=elite.gen.ms.shzr&gameid=ms&uuid1=2ced6028e80a433886db6db82e9bfe8e&devicetype=iPhone&deviceversion=6.0&versionid=8.1.1&f=838000707&detaildevicet
ype=iPhone3,1&userid=454219948&apptype=ios&jailbroken=false&uuid2=f80462b8c55ac2a64c8812779b40afbe&domain=&connecttype=wifi&roleid=292456898848886
914 HTTP/1.1" 200 43 "app:/com.renren.gameskit"
119.97.151.160 - - [13/Jan/2013:23:08:55 +0800] "GET http://tj.pet.mop.com/login_ane.html?
roleid=42a040000008200&f=926004501&userid=71461626@qq.com&jailbroken=false&appid=522657364&devicetype=iPhone&uuid2=3c648eb51633153a9a3cba3c0476004
e&deviceversion=6.0.1&detaildevicetype=iPhone4,1&versionid=4.2.0&uuid1=4f94d34e68724d8488782b7632a3935f&domain=x42%20-%20%E7%AA%81%E8%A2%AD
%E8%8B%8F%E8%81%94-60.29.251.165&gameid=com.imop.game.IronMarshal&connecttype=wwan&apptype=ios HTTP/1.1" 200 43 "app:/com.renren.gameskit"
——————————————————————————————————————————————————————————————————————————————
GET http://tj.pet.mop.com/run_ane.html
216.36.142.199 - - [13/Jan/2013:23:03:52 +0800] "GET http://tj.pet.mop.com/run_ane.html?
appid=elite.gen.ms.shzr&gameid=ms&uuid1=2ced6028e80a433886db6db82e9bfe8e&devicetype=iPhone&deviceversion=6.0&versionid=8.1.1&f=838000707&detaildevicet
ype=iPhone3,1&apptype=ios&jailbroken=false&uuid2=f80462b8c55ac2a64c8812779b40afbe&domain=&connecttype=wifi HTTP/1.1" 200 43 "app:/com.renren.gameskit"
216.36.142.199 - - [13/Jan/2013:23:07:08 +0800] "GET http://tj.pet.mop.com/run_ane.html?
appid=elite.gen.ms.shzr&gameid=ms&uuid1=2ced6028e80a433886db6db82e9bfe8e&devicetype=iPhone&deviceversion=6.0&versionid=8.1.1&f=838000707&detaildevicet
ype=iPhone3,1&apptype=ios&jailbroken=false&uuid2=f80462b8c55ac2a64c8812779b40afbe&domain=&connecttype=wifi HTTP/1.1" 200 43 "app:/com.renren.gameskit"
174.93.120.82 - - [13/Jan/2013:23:08:11 +0800] "GET http://tj.pet.mop.com/run_ane.html?
appid=elite.gen.ms.shzr&gameid=ms&uuid1=2ced6028e80a433886db6db82e9bfe8e&devicetype=iPhone&deviceversion=6.0&versionid=8.1.1&f=838000707&detaildevicet
ype=iPhone3,1&apptype=ios&jailbroken=false&uuid2=f80462b8c55ac2a64c8812779b40afbe&domain=&connecttype=wifi HTTP/1.1" 200 43 "app:/com.renren.gameskit"
——————————————————————————————————————————————————————————————————————————————
GET /loader_run.html
180.172.254.182 - - [13/Jan/2013:23:00:01 +0800] "GET /loader_run.html?
appid=air.nceo.as.android8560021510&sdkver=0.0.16&gameid=null&f=8560021510&apptype=android&versionid=0.0.16&uuid2=354356050087566&detaildevicetype=HTC
+T329t&imei=354356050087566&uuid1=810816ca06a04f2a3b468adbb3616aff&mac=a0f450ff66e7&jailbroken=false&devicetype=HTC
+T329t&domain=&deviceversion=&connecttype=4.0.4 HTTP/1.1" 200 3 "-"
1.192.245.84 - - [13/Jan/2013:23:00:02 +0800] "GET /loader_run.html?
appid=air.nceo.as.android8560021519&sdkver=0.0.14&gameid=null&f=8560021519&apptype=android&versionid=0.0.14&uuid2=866508010354368&detaildevicetype=ONN
+K7&imei=866508010354368&uuid1=44d6b021cc7d04d1fb78d2bfeb1adfee&mac=5ce2f4f4ab17&jailbroken=false&devicetype=ONN
+K7&domain=&deviceversion=&connecttype=4.0.4 HTTP/1.1" 200 3 "-"
112.93.35.68 - - [13/Jan/2013:23:00:03 +0800] "GET /loader_run.html?
appid=air.nceo.as.android8560021518&sdkver=0.0.15&gameid=null&f=8560021518&apptype=android&versionid=0.0.15&uuid2=359548049282598&detaildevicetype=GT-
N7000&imei=359548049282598&uuid1=946161396c64523669bc52f7cf39a6cb&mac=50CCF8AABE67&jailbroken=false&devicetype=GT-
N7000&domain=&deviceversion=&connecttype=2.3.6 HTTP/1.1" 200 3 "-"
——————————————————————————————————————————————————————————————————————————————
GET /login_ane.html
101.85.147.146 - - [13/Jan/2013:23:00:00 +0800] "GET /login_ane.html?
uuid2=70f784738fe0d2f645e15df6de3ffade&devicetype=iPad&mac=40300426A829&detaildevicetype=iPad2,2&connecttype=wifi&sdkver=1.0.7&uuid1=0e7431baeb7846c
0b1caa73a5647684b&apptype=ios&jailbroken=false&imei=null&deviceversion=6.0.1&userid=447393827&roleid=294567961174220907 HTTP/1.1" 200 46 "-"
218.83.227.131 - - [13/Jan/2013:23:00:00 +0800] "GET /login_ane.html?
f=925004401&gameid=lstx&domain=http://x66.lstx.renren.com&appid=588116262&versionid=1.5.0.1&devicetype=iPad&deviceversion=6.0&jailbroken=false&connectt
ype=wifi&uuid1=93f3dc7b9e2547149a980e94cbb02cbe&udid2=6137ccd66654bc518207c33e460c8986&mac=4c:b1:99:08:bc:d8 HTTP/1.1" 200 43 "-"
210.242.214.26 - - [13/Jan/2013:23:00:00 +0800] "GET /login_ane.html?
uuid2=09b67afe070498e4c2601f1be7d33f30&devicetype=iPad&mac=A4D1D25BF26B&detaildevicetype=iPad2,1&connecttype=wifi&sdkver=1.0.7&uuid1=65fe6e70e77c457
89fe988c1ace8bb38&apptype=ios&jailbroken=false&imei=null&deviceversion=6.0.1&userid=8089268&roleid=290768048988618933 HTTP/1.1" 200 46 "-"
——————————————————————————————————————————————————————————————————————————————
GET /login.html
115.197.239.203 - - [13/Jan/2013:23:00:01 +0800] "GET /login.html?
lhsg*ios*870001315*null*efc073686f92362f2663fcc7cc6e64c97eae0c8d*iPad*5.1.1*com.renren.game.nqj2hd*1.0.1 HTTP/1.1" 200 3 "app:/lhsg.swf"
115.197.239.203 - - [13/Jan/2013:23:00:01 +0800] "GET /login.html?
lhsg*ios*870001315*x10.lhsg.renren.com*efc073686f92362f2663fcc7cc6e64c97eae0c8d*iPad*5.1.1*com.renren.game.nqj2hd*1.0.1 HTTP/1.1" 200 3 "app:/lhsg.swf"
111.110.226.84 - - [13/Jan/2013:23:00:03 +0800] "GET /login.html?
lstx*android*8820017*http://x14.tx.rrgame.jp*SSHGE005483_a1000016c6b272_78:1C:5A:3C:34:74*ANDROID_IS17SH*4.0.4*and10301*1.3.317 HTTP/1.1" 200 3 "-"
——————————————————————————————————————————————————————————————————————————————
GET /robots.txt HTTP/1.0
62.24.222.131 - - [13/Jan/2013:23:26:17 +0800] "GET /robots.txt HTTP/1.0" 404 564 "http://tj.pet.mop.com/robots.txt"
GET /robots.txt HTTP/1.1
180.76.5.91 - - [13/Jan/2013:23:51:58 +0800] "GET /robots.txt HTTP/1.1" 404 162 "-"
180.76.5.49 - - [13/Jan/2013:23:52:01 +0800] "GET /robots.txt HTTP/1.1" 404 345
——————————————————————————————————————————————————————————————————————————————
GET /run_ane.html
112.96.131.124 - - [13/Jan/2013:22:59:59 +0800] "GET /run_ane.html?
appid=com.imop.game.IronMarshalX&gameid=ms&uuid1=abea9df0f87e49b7984f4ca373421a33&devicetype=iPhone&deviceversion=5.1.1&versionid=3.1.1&f=838000729&de
taildevicetype=iPhone4,1&apptype=ios&jailbroken=true&uuid2=a98488fc51257fc2ac574ab1983d054d&domain=&connecttype=wwan HTTP/1.1" 200 43
"app:/com.renren.gameskit"
114.221.147.156 - - [13/Jan/2013:22:59:59 +0800] "GET /run_ane.html?
uuid2=null&apptype=null&domain=&uuid1=null&devicetype=null&deviceversion=null&f=838000707&detaildevicetype=null&gameid=ms&jailbroken=null&versio
nid=1.0.8&connecttype=null&appid=com.msqy.weiduan HTTP/1.1" 200 43 "app:/Loading.swf/[[DYNAMIC]]/1"
221.194.31.155 - - [13/Jan/2013:23:00:00 +0800] "GET /run_ane.html?f=925004443&gameid=jzsg&domain={domain}
&appid=564762410&versionid=1.0.8&devicetype=iPhone&deviceversion=4.3.3&jailbroken=false&connecttype=wwan&uuid1=c95a9e14a1e34702999b57fb1e9ddb53&udid
2=041357b8146f516b3ca9a96b9e88cc90&mac=14:5a:05:e7:6c:25 HTTP/1.1" 200 43 "app:/musouClient.swf"
——————————————————————————————————————————————————————————————————————————————
GET /run.html
175.141.66.138 - - [13/Jan/2013:23:00:01 +0800] "GET /run.html?
lstx*android*925010017*http://x82.lstx.renren.com*cff19b4b01dc53f_358401049472325_98:0C:82:F0:BA:B2*ANDROID_GT-I9100G*4.0.3*and10101*1.3.318 HTTP/1.1" 200 3 "-"
210.242.214.5 - - [13/Jan/2013:23:00:03 +0800] "GET /run.html?lstx*android*9250082140**HT243W104686_359188048172357_18:87:96:8b:dc:6d*ANDROID_HTC%20One
%20X*4.1.1*and10101*1.3.299 HTTP/1.1" 200 3 "-"
89.24.14.32 - - [13/Jan/2013:23:00:05 +0800] "GET /run.html?lstx*android*862004601**4df1eea16a789f3f_354833058446879_20:02:AF:B9:F7:F8*ANDROID_GT-
N7100*4.1.1*and02501*1.3.263 HTTP/1.1" 200 3 "-"
&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&
统计的url仅限：
次数 url
50924 GET /login_ane.html
72678 GET /end_ane.html
78684 GET /run_ane.html
13 GET http://tj.pet.mop.com/login_ane.html
16 GET http://tj.pet.mop.com/end_ane.html
16 GET http://tj.pet.mop.com/run_ane.html
……………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………
GET http://tj.pet.mop.com/run_ane.html √
216.36.142.199 - - [13/Jan/2013:23:03:52 +0800] "GET http://tj.pet.mop.com/run_ane.html?
appid=elite.gen.ms.shzr&gameid=ms&uuid1=2ced6028e80a433886db6db82e9bfe8e&devicetype=iPhone&deviceversion=6.0&versionid=8.1.1&f=838000707&detaildevicet
ype=iPhone3,1&apptype=ios&jailbroken=false&uuid2=f80462b8c55ac2a64c8812779b40afbe&domain=&connecttype=wifi HTTP/1.1" 200 43 "app:/com.renren.gameskit"
……………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………
GET http://tj.pet.mop.com/end_ane.html √
216.36.142.199 - - [13/Jan/2013:23:03:52 +0800] "GET http://tj.pet.mop.com/end_ane.html?
appid=elite.gen.ms.shzr&gameid=ms&uuid1=2ced6028e80a433886db6db82e9bfe8e&devicetype=iPhone&onlinetime=4&endtime=1357999139&deviceversion=6.0&version
id=8.1.1&starttime=1357999135&launchtime=1357998799&f=838000707&detaildevicetype=iPhone3,1&userid=467415015&activetime=220&apptype=ios&jailbroken=f
alse&startcount=5&uuid2=f80462b8c55ac2a64c8812779b40afbe&domain=&connecttype=wifi&roleid=295834598569411941 HTTP/1.1" 200 43 "app:/com.renren.gameskit"
……………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………
GET http://tj.pet.mop.com/login_ane.html √
216.36.142.199 - - [13/Jan/2013:23:07:22 +0800] "GET http://tj.pet.mop.com/login_ane.html?
appid=elite.gen.ms.shzr&gameid=ms&uuid1=2ced6028e80a433886db6db82e9bfe8e&devicetype=iPhone&deviceversion=6.0&versionid=8.1.1&f=838000707&detaildevicet
ype=iPhone3,1&userid=454219948&apptype=ios&jailbroken=false&uuid2=f80462b8c55ac2a64c8812779b40afbe&domain=&connecttype=wifi&roleid=292456898848886
914 HTTP/1.1" 200 43 "app:/com.renren.gameskit"
……………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………
GET /run_ane.html √
112.96.131.124 - - [13/Jan/2013:22:59:59 +0800] "GET /run_ane.html?
appid=com.imop.game.IronMarshalX&gameid=ms&uuid1=abea9df0f87e49b7984f4ca373421a33&devicetype=iPhone&deviceversion=5.1.1&versionid=3.1.1&f=838000729&de
taildevicetype=iPhone4,1&apptype=ios&jailbroken=true&uuid2=a98488fc51257fc2ac574ab1983d054d&domain=&connecttype=wwan HTTP/1.1" 200 43
"app:/com.renren.gameskit"
……………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………
GET /end_ane.html √
221.194.30.138 - - [13/Jan/2013:22:59:59 +0800] "GET /end_ane.html?
connecttype=wwan&uuid1=d51c772dd73e48c29dd76269e021d418&mac=28E02C12BE8D&detaildevicetype=iPhone3,1&apptype=ios&versionid=1.2.1&deviceversion=6.0.1&i
mei=null&uuid2=bdd0d548eb1edbcd459dbf7f32cc4751&devicetype=iPhone&sdkver=1.0.7&gameid=rrnhzf&jailbroken=false&appid=com.imop.game.pirate2iphone&f=885
001201&domain=x39.nhzf.renren.com&userid=221106049&roleid=295271648616025261&starttime=1358088689&endtime=1358089200&onlinetime=511&launchtime=13580
88689&startcount=1&activetime=511 HTTP/1.1" 200 53 "-
……………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………
GET /login_ane.html √
101.85.147.146 - - [13/Jan/2013:23:00:00 +0800] "GET /login_ane.html?
uuid2=70f784738fe0d2f645e15df6de3ffade&devicetype=iPad&mac=40300426A829&detaildevicetype=iPad2,2&connecttype=wifi&sdkver=1.0.7&uuid1=0e7431baeb7846c
0b1caa73a5647684b&apptype=ios&jailbroken=false&imei=null&deviceversion=6.0.1&userid=447393827&roleid=294567961174220907 HTTP/1.1" 200 46 "-"
……………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………
hadoop fs -ls /data/raw/adwall/test
hadoop fs -rm /data/raw/adwall/test/adwall-1.0.jar
hadoop fs -put ./adwall-1.0.jar /data/raw/adwall/test
hadoop fs -put /data/var/zghtest/adwallSerde/target/adwall-1.0.jar /data/raw/adwall/test
hive> add jar /data/var/zghtest/adwall-1.0.jar;
Added /data/var/zghtest/adwall-1.0.jar to class path
Added resource: /data/var/zghtest/adwall-1.0.jar
cp /data/var/zghtest/adwallSerde/target/adwall-1.0.jar /data/var/zghtest
删除的话就是delete jar
delete jar /data/var/zghtest/adwall-1.0.jar;
建表语句相关
CREATE EXTERNAL TABLE raw.t_adwall_test
(
ip STRING ,
datee STRING ,
interfaze STRING
)
ROW FORMAT SERDE 'com.renren.aneSerDe.AneSerDe';
CREATE EXTERNAL TABLE raw.t_adwall_test2
(
ip STRING ,
datee STRING ,
interfaze STRING,
ipanddate STRING
)
ROW FORMAT SERDE 'com.renren.aneSerDe.AneSerDe';
CREATE EXTERNAL TABLE raw.t_adwall_test3
(
ip STRING ,
datee STRING ,
interfaze STRING,
ipanddate STRING
)
ROW FORMAT SERDE 'com.renren.aneSerDe.AneSerDe';
CREATE EXTERNAL TABLE raw.t_adwall
(
f_ip STRING ,
f_datee STRING ,
f_interfaze STRING,
f_appid STRING,
f_gameid STRING,
f_uuid1 STRING,
f_devicetype STRING,
f_deviceversion STRING,
f_versionid STRING,
f_f STRING,
f_detaildevicetype STRING,
f_apptype STRING,
f_jailbroken STRING,
f_uuid2 STRING,
f_domain STRING,
f_connecttype STRING,
f_roleid STRING,
f_mac STRING,
f_sdkver STRING,
f_userid STRING,
f_starttime BIGINT,
f_endtime BIGINT,
f_onlinetime BIGINT,
f_launchtime BIGINT,
f_startcount INT,
f_activetime BIGINT,
f_formatright SMALLINT,
f_formaterrorreason STRING,
f_raw STRING
)
ROW FORMAT SERDE 'com.renren.aneSerDe.AneSerDe';
LOAD DATA LOCAL INPATH '/data/var/zghtest/access.log' OVERWRITE INTO TABLE raw.t_adwall;
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
hive> select * from t_adwall where f_formatright=0 limit 10;
Total MapReduce jobs = 1
Launching Job 1 out of 1
Number of reduce tasks is set to 0 since there's no reduce operator
Starting Job = job_201206261006_286591, Tracking URL = http://h1-1.io8.org:50030/jobdetails.jsp?jobid=job_201206261006_286591
Kill Command = /data/opt/hadoop/bin/hadoop job -Dmapred.job.tracker=h1-1.io8.org:9001 -kill job_201206261006_286591
Hadoop job information for Stage-1: number of mappers: 2; number of reducers: 0
2013-01-16 22:30:26,367 Stage-1 map = 0%, reduce = 0%
2013-01-16 22:30:31,394 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 11.23 sec
2013-01-16 22:30:32,401 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 11.23 sec
2013-01-16 22:30:33,408 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 11.23 sec
MapReduce Total cumulative CPU time: 11 seconds 230 msec
Ended Job = job_201206261006_286591
MapReduce Jobs Launched:
Job 0: Map: 2 Accumulative CPU: 11.23 sec HDFS Read: 33726867 HDFS Write: 2843 SUCESS
Total MapReduce CPU Time Spent: 11 seconds 230 msec
OK
1024null null null com.msqy.weiduan ms null null null 1.0.8 838000707 null null null null null null null null NULL 1358089234
NULL 0 1358089234 1 0 0 {"checkRule3_5":"no_mac"}
1024null null null 520025006 jzsg 39bfe5c7178c4882a06a19202f0aee6b iPad 5.1.1 1.0.6 iPad3,1 ios false ec147b2821c8030b4c6812538ef5d9ed
http://x41.s.jzsg.renren.com wifi null null null NULL 1358089273 1358089273 0 1358089273 1 0 0 {"checkRule3_5":"no_mac"}
1024null null null elite.gen.ms.shzr ms 27329507945c40afbf588a5330a2246b iPhone 6.0.1 8.1.1 838000707 iPhone4,1 ios false
d38a920af438c405b08ee7d150d35bd4 wifi null HTTP/1.1" 200 43 "app:/com.renren.gameskit" null null NULL 1357915585 1357915585 0 1357915585
1 0 0 {"checkRule3_5":"no_mac"}
1024null null null com.imop.game.wosiphone ms 407c7060d95946fdae79054f307bb1e3 iPad 6.0.1 1.0.7 838000707 iPad2,2 ios false
338955105bad19c4f57641b05eb0a77e wifi null HTTP/1.1" 200 43 "app:/com.renren.gameskit" null null NULL 1358077428 1358077428 0 1358077428
1 0 0 {"checkRule3_5":"no_mac"}
1024null null null com.imop.game.wosiphone ms 8c81ef15e8ea4c5fbcf8856153bef980 iPhone 4.3.1 1.0.6 838000707 iPhone3,1 ios true
f4db9de6d3d08e956e3378f03a3ceb70 wifi null HTTP/1.1" 200 43 "app:/com.renren.gameskit" null null NULL 1357871821 1357871821 0 1357871821
1 0 0 {"checkRule3_5":"no_mac"}
1024null null null com.msqy.weiduan ms null null null 1.0.8 HTTP/1.1" 200 43 "app:/Loading.swf/[[DYNAMIC]]/1" 838000707 null null null null
null null null null NULL 1358089493 1358089493 0 1358089493 1 0 0 {"checkRule3_5":"no_mac"}
1024null null null elite.gen.ms.shzr ms 01bd91e55d564dde83d665851328f4b7 iPhone 6.0.1 8.1.1 838000707 iPhone4,1 ios false
f427006b61c116e6ea3c53d18e5e2f9e wifi null HTTP/1.1" 200 43 "app:/com.renren.gameskit" null null NULL 1358089478 1358089478 0 1358089478
1 0 0 {"checkRule3_5":"no_mac"}
1024null null null com.imop.game.wos1iphone ms 970b9f38cd464251a53bf39c2f3ebddc iPhone 6.0.1 1.1.6 838000707 iPhone4,1 ios false
5133ffcde7cf5c69d7fd141e436c265e wifi null HTTP/1.1" 200 43 "app:/com.renren.gameskit" null null NULL 1358088195 1358088195 0 1358088195
1 0 0 {"checkRule3_5":"no_mac"}
1024null null null wostest ms HTTP/1.1" 200 43 "app:/Loading.swf/[[DYNAMIC]]/1" null null null 00.00.04 838000707 null null null null
nullnull null null NULL 1358089515 1358089515 0 1358089515 1 0 0 {"checkRule3_5":"no_mac"}
1024null null null and10101 lstx 2842f80102034bc1b59f674ae2649f27 MB525 2.3.7 HTTP/1.1" 200 43 "app:/com.renren.gameskit" 1.6.075 925010018
MB525 android false 2842f80102034bc1b59f674ae2649f27 lstx.renren.com 2G null 984b4a1561ab 1.0.7 NULL 1358088814 1358088814 0 1358088814
1 0 1 {}
Time taken: 14.202 seconds
hive>
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
hive -S -e "use raw;add jar /data/var/zghtest/adwall-1.0.jar; select * from t_adwall where f_formatright=1 limit 10;" >> /data/var/zghtest.cvs
set hive.cli.print.header=true;
set hive.cli.print.row.to.vertical=true;
set hive.cli.print.row.to.vertical.num=1;
开启显示列名的功能。
CREATE EXTERNAL TABLE dw0.t_adwall
(
f_ip STRING ,
f_datee STRING ,
f_interfaze STRING,
f_appid STRING,
f_gameid STRING,
f_uuid1 STRING,
f_devicetype STRING,
f_deviceversion STRING,
f_versionid STRING,
f_f STRING,
f_detaildevicetype STRING,
f_apptype STRING,
f_jailbroken STRING,
f_uuid2 STRING,
f_domain STRING,
f_connecttype STRING,
f_roleid STRING,
f_mac STRING,
f_sdkver STRING,
f_userid STRING,
f_starttime BIGINT,
f_endtime BIGINT,
f_onlinetime BIGINT,
f_launchtime BIGINT,
f_startcount INT,
f_activetime BIGINT,
f_formatright SMALLINT,
f_formaterrorreason STRING,
f_raw STRING
)
建立洗过一次的数据表：
insert overwrite table dw0.t_adwall
select f_ip, f_datee,f_interfaze, f_appid, f_gameid , f_uuid1, f_devicetype , f_deviceversion
, f_versionid,f_f , f_detaildevicetype , f_apptype , f_jailbroken, f_uuid2, f_domain, f_connecttype,
f_roleid , f_mac , f_sdkver , f_userid , f_starttime , f_endtime , f_onlinetime , f_launchtime , f_startcount ,
f_activetime , f_formatright , f_formaterrorreason , f_raw
from raw. t_adwall
新的查询结果：
hive> select * from dw0.t_adwall where f_formatright=1 limit 10;
Total MapReduce jobs = 1
Launching Job 1 out of 1
Number of reduce tasks is set to 0 since there's no reduce operator
Starting Job = job_201206261006_287114, Tracking URL = http://h1-1.io8.org:50030/jobdetails.jsp?jobid=job_201206261006_287114
Kill Command = /data/opt/hadoop/bin/hadoop job -Dmapred.job.tracker=h1-1.io8.org:9001 -kill job_201206261006_287114
Hadoop job information for Stage-1: number of mappers: 2; number of reducers: 0
2013-01-17 17:01:10,566 Stage-1 map = 0%, reduce = 0%
2013-01-17 17:01:13,580 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 2.55 sec
2013-01-17 17:01:14,585 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 2.55 sec
MapReduce Total cumulative CPU time: 2 seconds 550 msec
Ended Job = job_201206261006_287114
MapReduce Jobs Launched:
Job 0: Map: 2 Accumulative CPU: 2.55 sec HDFS Read: 140255 HDFS Write: 6541 SUCESS
Total MapReduce CPU Time Spent: 2 seconds 550 msec
OK
f_ip f_datee f_interfaze f_appid f_gameid f_uuid1 f_devicetype f_deviceversion f_versionid f_f f_detaildevicetype f_apptype f_jailbroken f_uuid2 f_domain f_connecttype f_roleid f_mac f_sdkver f_userid f_starttime f_endtime f_onlinetime f_launchtime f_startcount f_activetime f_formatright f_formaterrorreason f_raw
221.194.30.138 - null "GET /end_ane com.imop.game.pirate2iphone rrnhzf d51c772dd73e48c29dd76269e021d418 iPhone 6.0.1 1.2.1 885001201 iPhone3,1 ios falsbdd0d548eb1edbcd459dbf7f32cc4751 x39.nhzf.renren.com null 295271648616025261 28E02C12BE8D 1.0.7 221106049 1358088689 1358089200 511 1358088689 1 NULL{} 221.194.30.138 - - [13/Jan/2013:22:59:59 +0800] "GET /end_ane.html?connecttype=wwan&uuid1=d51c772dd73e48c29dd76269e021d418&mac=28E02C12BE8D&detaildevicetype=iPhone3,1&apptype=ios&versionid=1.2.1&deviceversion=6.0.1&imei=null&uuid2=bdd0d548eb1edbcd459dbf7f32cc4751&devicetype=iPhone&sdkver=1.0.7&gameid=rrnhzf&jailbroken=false&appid=com.imop.game.pirate2iphone&f=885001201&domain=x39.nhzf.renren.com&userid=221106049&roleid=295271648616025261&starttime=1358088689&endtime=1358089200&onlinetime=511&launchtime=1358088689&startcount=1&activetime=511 HTTP/1.1" 200 53 "-"
175.142.180.206 null "GET /end_ane com.renren.game.csj csj 0d40e54d523a48428ad10ecc5253e153 iPad 4.3.1 1.1.2 null iPad2,2 ios false ee5bdbc81fb02ee3bc0b362a5281a935 x21.csj.renren.com null null A467069329AE 1.0.8 null 1358089192 1358089200 8 1358089192 1 NULL 1 {} 175.142.180.206 - - [13/Jan/2013:23:00:00 +0800] "GET /end_ane.html?connecttype=wifi&uuid1=0d40e54d523a48428ad10ecc5253e153&mac=A467069329AE&detaildevicetype=iPad2,2&apptype=ios&versionid=1.1.2&deviceversion=4.3.1&imei=null&uuid2=ee5bdbc81fb02ee3bc0b362a5281a935&devicetype=iPad&sdkver=1.0.8&gameid=csj&jailbroken=false&appid=com.renren.game.csj&f=null&domain=x21.csj.renren.com&userid=null&roleid=null&starttime=1358089192&endtime=1358089200&onlinetime=8&launchtime=1358089192&startcount=1&activetime=8 HTTP/1.1" 200 46 "-"
221.5.86.37 - - null "GET /end_ane com.renren.games.csj4 com.renren.games.csj4 e368ba2a6ec14a02bb32cb5cef284027 iPhone 5.0.1 1.1.4 8560011158 iPhone4,1 nulltrue eaacae952f7b421abf062978cb8898d1 x32.csj.renren.com wifi 297241973452978776 60FACD22FC83 1.0.9 505529877 1358088339 1358089202 863 1358088339 NULL 1 {} 221.5.86.37 - - [13/Jan/2013:23:00:00 +0800] "GET /end_ane.html?apptype=ios&uuid1=e368ba2a6ec14a02bb32cb5cef284027&mac=60FACD22FC83&detaildevicetype=iPhone4,1&versionid=1.1.4&platformid=null&roleid=297241973452978776&deviceversion=5.0.1&imei=null&uuid2=eaacae952f7b421abf062978cb8898d1&devicetype=iPhone&sdkver=1.0.9&gameid=com.renren.games.csj4&jailbroken=true&appid=com.renren.games.csj4&f=8560011158&domain=x32.csj.renren.com&userid=505529877&connecttype=wifi&starttime=1358088339&launchtime=1358088339&endtime=1358089202&onlinetime=863&startcount=1&activetime=863 HTTP/1.1" 200 53 "-"
112.96.30.111 - null "GET /end_ane air.nceo.as.android8560021732 air.nceo.as.android8560021732 f440d14885466879b11dc79e1b7b2346 GT-I9100 2.3.3 1.1.4 8560021732 GT-I9100 android false f440d14885466879b11dc79e1b7b2346 x30.csj.renren.com 3G 296679023499553278 null 1.0.9 503021509 1358089185 1358089253 68 1358088880 3 NULL 1 {} 112.96.30.111 - - [13/Jan/2013:23:00:00 +0800] "GET /end_ane.html?activetime=230&appid=air.nceo.as.android8560021732&detaildevicetype=GT-I9100&endtime=1358089253&startcount=3&sdkver=1.0.9&gameid=air.nceo.as.android8560021732&jailbroken=false&connecttype=3G&apptype=android&f=8560021732&domain=x30.csj.renren.com&starttime=1358089185&uuid2=f440d14885466879b11dc79e1b7b2346&versionid=1.1.4&launchtime=1358088880&imei=359090046272710&platformid=null&uuid1=f440d14885466879b11dc79e1b7b2346&mac=null&devicetype=GT-I9100&userid=503021509&onlinetime=68&deviceversion=2.3.3&roleid=296679023499553278 HTTP/1.0" 200 43 "app:/com.renren.gameskit"
58.243.250.37 - null "GET /end_ane com.renren.game.csj csj 60124ca558884db38eb5e50c024f72bc iPhone 6.0.1 1.1.2 null iPhone4,1 ios false 5ef2936b2d7fe62766648bd07394e59b x10.csj.renren.com null 291049523965365363 5C95AE8A5860 1.0.8 487981608 1358086504 1358089197 2693 1358086504 1 NULL 1 {} 58.243.250.37 - - [13/Jan/2013:23:00:00 +0800] "GET /end_ane.html?connecttype=wwan&uuid1=60124ca558884db38eb5e50c024f72bc&mac=5C95AE8A5860&detaildevicetype=iPhone4,1&apptype=ios&versionid=1.1.2&deviceversion=6.0.1&imei=null&uuid2=5ef2936b2d7fe62766648bd07394e59b&devicetype=iPhone&sdkver=1.0.8&gameid=csj&jailbroken=false&appid=com.renren.game.csj&f=null&domain=x10.csj.renren.com&userid=487981608&roleid=291049523965365363&starttime=1358086504&endtime=1358089197&onlinetime=2693&launchtime=1358086504&startcount=1&activetime=2693 HTTP/1.1" 200 46 "-"
58.243.250.37 - null "GET /end_ane com.renren.game.csj csj 60124ca558884db38eb5e50c024f72bc iPhone 6.0.1 1.1.2 null iPhone4,1 ios false 5ef2936b2d7fe62766648bd07394e59b x10.csj.renren.com null 291049523965365363 5C95AE8A5860 1.0.8 487981608 1358086504 1358089197 2693 1358086504 1 NULL 1 {} 58.243.250.37 - - [13/Jan/2013:23:00:00 +0800] "GET /end_ane.html?connecttype=wwan&uuid1=60124ca558884db38eb5e50c024f72bc&mac=5C95AE8A5860&detaildevicetype=iPhone4,1&apptype=ios&versionid=1.1.2&deviceversion=6.0.1&imei=null&uuid2=5ef2936b2d7fe62766648bd07394e59b&devicetype=iPhone&sdkver=1.0.8&gameid=csj&jailbroken=false&appid=com.renren.game.csj&f=null&domain=x10.csj.renren.com&userid=487981608&roleid=291049523965365363&starttime=1358086504&endtime=1358089197&onlinetime=2693&launchtime=1358086504&startcount=1&activetime=2693 HTTP/1.1" 200 46 "-"
112.224.67.29 - null "GET /end_ane com.renren.game.csjhd com.renren.game.csjhd d23c7659b1d944e190a89c22ea1b4997 iPhone 4.3.5 1.1.3 856001130 iPhone3,1 nullfalse 4305f44c04c402abcf58b36e005fcf68 x23.csj.renren.com wwan 294708698662585180 148FC67E6E0A 1.0.9 496599565 1358089157 1358089199 42 1358066165 NULL 1 {} 112.224.67.29 - - [13/Jan/2013:23:00:00 +0800] "GET /end_ane.html?apptype=ios&uuid1=d23c7659b1d944e190a89c22ea1b4997&mac=148FC67E6E0A&detaildevicetype=iPhone3,1&versionid=1.1.3&platformid=null&roleid=294708698662585180&deviceversion=4.3.5&imei=null&uuid2=4305f44c04c402abcf58b36e005fcf68&devicetype=iPhone&sdkver=1.0.9&gameid=com.renren.game.csjhd&jailbroken=false&appid=com.renren.game.csjhd&f=856001130&domain=x23.csj.renren.com&userid=496599565&connecttype=wwan&starttime=1358089157&launchtime=1358066165&endtime=1358089199&onlinetime=42&startcount=2&activetime=721 HTTP/1.1" 200 53 "-"
58.22.114.175 - null "GET /end_ane 494778270 cq dc06015fd37842eea06b2143eacd1c2a iPhone 5.1.1 1.5.3 855000877 iPhone4,1 null false 1cd5a157ec972135bba67b722b50ba99 http://s51.cq.renren.com/ wwan 043304000000453c 0C771A63EFF8 1.0.9 499040717 1358083127 1358084580 1453 1358083127 1 NULL{} 58.22.114.175 - - [13/Jan/2013:23:00:00 +0800] "GET /end_ane.html?apptype=ios&uuid1=dc06015fd37842eea06b2143eacd1c2a&mac=0C771A63EFF8&detaildevicetype=iPhone4,1&versionid=1.5.3&platformid=null&roleid=043304000000453c&deviceversion=5.1.1&imei=null&uuid2=1cd5a157ec972135bba67b722b50ba99&devicetype=iPhone&sdkver=1.0.9&gameid=cq&jailbroken=false&appid=494778270&f=855000877&domain=http://s51.cq.renren.com/&userid=499040717&connecttype=wwan&starttime=1358083127&launchtime=1358083127&endtime=1358084580&onlinetime=1453&startcount=1&activetime=1453 HTTP/1.1" 200 53 "-"
211.140.5.110 - null "GET /run_ane 511848500 cq null GT-I9018 2.2.2 1.5.3 855002850 GT-I9018 android false 537dfb91f38b2964d6fe60ad4c0c8558 http://s59.cq.renren.com/ 2G null 1234569D4E60 1.1.0 null 1358089277 NULL NULL 1358089277 NULL NULL 1 {} 211.140.5.110 - - [13/Jan/2013:23:00:00 +0800] "GET /run_ane.html?uuid1=537dfb91f38b2964d6fe60ad4c0c8558&uuid2=537dfb91f38b2964d6fe60ad4c0c8558&versionid=1.5.3&imei=867715001384978&gameid=cq&mac=1234569D4E60&detaildevicetype=GT-I9018&f=855002850&userid=null&connecttype=2G&platformid=null&roleid=null&starttime=1358089277&jailbroken=false&devicetype=GT-I9018&sdkver=1.1.0&deviceversion=2.2.2&appid=511848500&domain=http://s59.cq.renren.com/&launchtime=1358089277&apptype=android HTTP/1.1" 200 53 "app:/com.renren.gameskit"
182.249.240.38 - null "GET /run_ane 563138738 cq aa78cb3cfb8347bfddebe4f4b686defc URBANO%20PROGRESSO 4.0.3 1.2.1 821000401 URBANO%20PROGRESSO android false null http://x2.jcq.rrgame.jp/ 2G null 245fdf41e931 1.0.7 null NULL NULL NULL NULL NULL NULL 1 {} 182.249.240.38 - - [13/Jan/2013:23:00:00 +0800] "GET /run_ane.html?uuid2=aa78cb3cfb8347bfddebe4f4b686defc&devicetype=URBANO%20PROGRESSO&deviceversion=4.0.3&versionid=1.2.1&f=821000401&appid=563138738&detaildevicetype=URBANO%20PROGRESSO&apptype=android&gameid=cq&uuid1=aa78cb3cfb8347bfddebe4f4b686defc&sdkver=1.0.7&domain=http://x2.jcq.rrgame.jp/&jailbroken=false&imei=35228005085670&connecttype=2G&mac=245fdf41e931 HTTP/1.1" 200 43 "app:/com.renren.gameskit"
Time taken: 8.832 seconds
hive>
分析共有几种数据格式：
more access.log | awk -F "[\"]" '{print $2}' | awk -F "[?]" '{print $1}'| sort | uniq | sort
共有以下种请求：
****************************************************
stat
api/databus/stat/click
GET /click.html
api/databus/stat/display
GET /display.html
api/databus/stat/game
GET /game.html
ane
api/databus/ane/end
GET /end_ane.html √
api/databus/ane/run
GET /run_ane.html √
api/databus/ane/login
GET /login_ane.html √
api/databus/ane/loader
GET /loader_run.html
***************************************************
GET /click.html
举例：
124.88.232.193 - - [13/Jan/2013:22:59:59 +0800] "GET /click.html?9*925008604*1*nlink*https://itunes.apple.com/cn/app/xue-di-zi-sheng-dan-ban/id588116262?ls=1&mt=8
HTTP/1.1" 301 0 "-"
1.205.1.200 - - [13/Jan/2013:22:59:59 +0800] "GET /click.html?9*838001018*1*nlink*http://itunes.apple.com/cn/app/shi-hun-zhi-ren/id588118382?mt=8 HTTP/1.1" 301 0 "-"
125.85.93.141 - - [13/Jan/2013:22:59:59 +0800] "GET /click.html?9*838000737*1*nlink*https://itunes.apple.com/cn/app/shi-hun-zhi-ren/id588118382?mt=8 HTTP/1.1" 301 0 "-"
——————————————————————————————————————————————————————————————————————————————
GET /click.html HTTP/1.0
62.24.222.131 - - [13/Jan/2013:23:26:18 +0800] "GET /click.html HTTP/1.0" 200 3 "http://tj.pet.mop.com/click.html"
——————————————————————————————————————————————————————————————————————————————
GET /display.html
115.45.201.131 - - [13/Jan/2013:23:00:01 +0800] "GET /display.html?80134|952000101|0|receiver|0|mop HTTP/1.1" 304 0
36.40.191.155 - - [13/Jan/2013:23:00:02 +0800] "GET /display.html?500293|1|0|receiver|0|mop HTTP/1.1" 200 23
116.22.217.54 - - [13/Jan/2013:23:00:05 +0800] "GET /display.html?9|9040010018|0|receiver|0|mop HTTP/1.1" 200 23
——————————————————————————————————————————————————————————————————————————————
GET /end_ane.html
221.194.30.138 - - [13/Jan/2013:22:59:59 +0800] "GET /end_ane.html?
connecttype=wwan&uuid1=d51c772dd73e48c29dd76269e021d418&mac=28E02C12BE8D&detaildevicetype=iPhone3,1&apptype=ios&versionid=1.2.1&deviceversion=6.0.1&i
mei=null&uuid2=bdd0d548eb1edbcd459dbf7f32cc4751&devicetype=iPhone&sdkver=1.0.7&gameid=rrnhzf&jailbroken=false&appid=com.imop.game.pirate2iphone&f=885
001201&domain=x39.nhzf.renren.com&userid=221106049&roleid=295271648616025261&starttime=1358088689&endtime=1358089200&onlinetime=511&launchtime=13580
88689&startcount=1&activetime=511 HTTP/1.1" 200 53 "-"
114.221.147.156 - - [13/Jan/2013:22:59:59 +0800] "GET /end_ane.html?
roleid=295553123592712996&uuid2=null&starttime=1358089089&apptype=null&domain=&endtime=1358089095&uuid1=null&onlinetime=6&devicetype=null&launch
time=1358080763&deviceversion=null&startcount=12&f=838000707&activetime=436&detaildevicetype=null&gameid=ms&jailbroken=null&versionid=1.0.8&connec
ttype=null&userid=497886004&appid=com.msqy.weiduan HTTP/1.1" 200 43 "app:/Loading.swf/[[DYNAMIC]]/1"
175.142.180.206 - - [13/Jan/2013:23:00:00 +0800] "GET /end_ane.html?
connecttype=wifi&uuid1=0d40e54d523a48428ad10ecc5253e153&mac=A467069329AE&detaildevicetype=iPad2,2&apptype=ios&versionid=1.1.2&deviceversion=4.3.1&ime
i=null&uuid2=ee5bdbc81fb02ee3bc0b362a5281a935&devicetype=iPad&sdkver=1.0.8&gameid=csj&jailbroken=false&appid=com.renren.game.csj&f=null&domain=x21.cs
j.renren.com&userid=null&roleid=null&starttime=1358089192&endtime=1358089200&onlinetime=8&launchtime=1358089192&startcount=1&activetime=8 HTTP/1.1"
200 46 "-"
——————————————————————————————————————————————————————————————————————————————
GET /favicon.ico HTTP/1.1
120.33.159.158 - - [13/Jan/2013:23:00:45 +0800] "GET /favicon.ico HTTP/1.1" 404 564 "-"
14.118.120.122 - - [13/Jan/2013:23:01:25 +0800] "GET /favicon.ico HTTP/1.1" 404 564 "-"
116.230.225.94 - - [13/Jan/2013:23:05:25 +0800] "GET /favicon.ico HTTP/1.1" 404 564 "-"
——————————————————————————————————————————————————————————————————————————————
GET /game.html
222.209.150.169 - - [13/Jan/2013:22:59:59 +0800] "GET /game.html?0|932000101|3400|renren|0|http://x20.war.renren.com/ HTTP/1.1" 304 0
222.209.150.169 - - [13/Jan/2013:22:59:59 +0800] "GET /game.html?0|932000101|2400|renren|1|http://x20.war.renren.com/ HTTP/1.1" 304 0
182.242.227.214 - - [13/Jan/2013:22:59:59 +0800] "GET /game.html?-1|855004201|3400|0|0|0 HTTP/1.1" 200 20
——————————————————————————————————————————————————————————————————————————————
GET / HTTP/1.1
113.108.11.50 - - [13/Jan/2013:23:09:21 +0800] "GET / HTTP/1.1" 200 20
112.90.84.224 - - [13/Jan/2013:23:09:21 +0800] "GET / HTTP/1.1" 200 20
123.190.215.78 - - [13/Jan/2013:23:13:21 +0800] "GET / HTTP/1.1" 200 0 "-"
——————————————————————————————————————————————————————————————————————————————
GET http://tj.imop.com/click.html
112.97.192.31 - - [13/Jan/2013:23:11:07 +0800] "GET /click.html?9*855000865*1*nlink*http://itunes.apple.com/cn/app/lu-bu-yu-diao-chan/id555912419?mt=8 HTTP/1.1" 301 0
"http://c.w.inmobi.com/ios35x.html?http://tj.imop.com/click.html?9*855000865*1*nlink*http://itunes.apple.com/cn/app/lu-bu-yu-diao-chan/id555912419?mt=8"
118.116.81.28 - - [13/Jan/2013:23:11:39 +0800] "GET /click.html?9*838001010*1*nlink*http://itunes.apple.com/cn/app/shi-hun-zhi-ren/id588118382?mt=8 HTTP/1.1" 301 0
"http://c.w.inmobi.com/ios35x.html?http://tj.imop.com/click.html?9*838001010*1*nlink*http://itunes.apple.com/cn/app/shi-hun-zhi-ren/id588118382?mt=8"
125.126.32.95 - - [13/Jan/2013:23:12:00 +0800] "GET /click.html?9*838001017*1*nlink*http://itunes.apple.com/cn/app/shi-hun-zhi-ren/id588118382?mt=8 HTTP/1.1" 301 0
"http://c.w.inmobi.com/ios35x.html?http://tj.imop.com/click.html?9*838001017*1*nlink*http://itunes.apple.com/cn/app/shi-hun-zhi-ren/id588118382?mt=8"
——————————————————————————————————————————————————————————————————————————————
GET http://tj.pet.mop.com/click.html
119.97.151.160 - - [13/Jan/2013:23:54:42 +0800] "GET http://tj.pet.mop.com/click.html?60101*150101*1*nlink*http://itunes.apple.com/cn/app/id588116262?mt=8 HTTP/1.1" 301 0
"-"
——————————————————————————————————————————————————————————————————————————————
GET http://tj.pet.mop.com/end_ane.html
216.36.142.199 - - [13/Jan/2013:23:03:52 +0800] "GET http://tj.pet.mop.com/end_ane.html?
appid=elite.gen.ms.shzr&gameid=ms&uuid1=2ced6028e80a433886db6db82e9bfe8e&devicetype=iPhone&onlinetime=4&endtime=1357999139&deviceversion=6.0&version
id=8.1.1&starttime=1357999135&launchtime=1357998799&f=838000707&detaildevicetype=iPhone3,1&userid=467415015&activetime=220&apptype=ios&jailbroken=f
alse&startcount=5&uuid2=f80462b8c55ac2a64c8812779b40afbe&domain=&connecttype=wifi&roleid=295834598569411941 HTTP/1.1" 200 43 "app:/com.renren.gameskit"
216.36.142.199 - - [13/Jan/2013:23:07:07 +0800] "GET http://tj.pet.mop.com/end_ane.html?
appid=elite.gen.ms.shzr&gameid=ms&uuid1=2ced6028e80a433886db6db82e9bfe8e&devicetype=iPhone&onlinetime=157&endtime=1358089586&deviceversion=6.0&versi
onid=8.1.1&starttime=1358089429&launchtime=1358089429&f=838000707&detaildevicetype=iPhone3,1&userid=null&activetime=157&apptype=ios&jailbroken=fals
e&startcount=1&uuid2=f80462b8c55ac2a64c8812779b40afbe&domain=&connecttype=wifi&roleid=null HTTP/1.1" 200 43 "app:/com.renren.gameskit"
174.93.120.82 - - [13/Jan/2013:23:08:11 +0800] "GET http://tj.pet.mop.com/end_ane.html?
appid=elite.gen.ms.shzr&gameid=ms&uuid1=2ced6028e80a433886db6db82e9bfe8e&devicetype=iPhone&onlinetime=51&endtime=1358089676&deviceversion=6.0&versio
nid=8.1.1&starttime=1358089625&launchtime=1358089429&f=838000707&detaildevicetype=iPhone3,1&userid=454219948&activetime=208&apptype=ios&jailbroken=
false&startcount=2&uuid2=f80462b8c55ac2a64c8812779b40afbe&domain=&connecttype=wifi&roleid=292456898848886914 HTTP/1.1" 200 43 "app:/com.renren.gameskit
——————————————————————————————————————————————————————————————————————————————
GET http://tj.pet.mop.com/login_ane.html
216.36.142.199 - - [13/Jan/2013:23:07:22 +0800] "GET http://tj.pet.mop.com/login_ane.html?
appid=elite.gen.ms.shzr&gameid=ms&uuid1=2ced6028e80a433886db6db82e9bfe8e&devicetype=iPhone&deviceversion=6.0&versionid=8.1.1&f=838000707&detaildevicet
ype=iPhone3,1&userid=454219948&apptype=ios&jailbroken=false&uuid2=f80462b8c55ac2a64c8812779b40afbe&domain=&connecttype=wifi&roleid=292456898848886
914 HTTP/1.1" 200 43 "app:/com.renren.gameskit"
174.93.120.82 - - [13/Jan/2013:23:08:25 +0800] "GET http://tj.pet.mop.com/login_ane.html?
appid=elite.gen.ms.shzr&gameid=ms&uuid1=2ced6028e80a433886db6db82e9bfe8e&devicetype=iPhone&deviceversion=6.0&versionid=8.1.1&f=838000707&detaildevicet
ype=iPhone3,1&userid=454219948&apptype=ios&jailbroken=false&uuid2=f80462b8c55ac2a64c8812779b40afbe&domain=&connecttype=wifi&roleid=292456898848886
914 HTTP/1.1" 200 43 "app:/com.renren.gameskit"
119.97.151.160 - - [13/Jan/2013:23:08:55 +0800] "GET http://tj.pet.mop.com/login_ane.html?
roleid=42a040000008200&f=926004501&userid=71461626@qq.com&jailbroken=false&appid=522657364&devicetype=iPhone&uuid2=3c648eb51633153a9a3cba3c0476004
e&deviceversion=6.0.1&detaildevicetype=iPhone4,1&versionid=4.2.0&uuid1=4f94d34e68724d8488782b7632a3935f&domain=x42%20-%20%E7%AA%81%E8%A2%AD
%E8%8B%8F%E8%81%94-60.29.251.165&gameid=com.imop.game.IronMarshal&connecttype=wwan&apptype=ios HTTP/1.1" 200 43 "app:/com.renren.gameskit"
——————————————————————————————————————————————————————————————————————————————
GET http://tj.pet.mop.com/run_ane.html
216.36.142.199 - - [13/Jan/2013:23:03:52 +0800] "GET http://tj.pet.mop.com/run_ane.html?
appid=elite.gen.ms.shzr&gameid=ms&uuid1=2ced6028e80a433886db6db82e9bfe8e&devicetype=iPhone&deviceversion=6.0&versionid=8.1.1&f=838000707&detaildevicet
ype=iPhone3,1&apptype=ios&jailbroken=false&uuid2=f80462b8c55ac2a64c8812779b40afbe&domain=&connecttype=wifi HTTP/1.1" 200 43 "app:/com.renren.gameskit"
216.36.142.199 - - [13/Jan/2013:23:07:08 +0800] "GET http://tj.pet.mop.com/run_ane.html?
appid=elite.gen.ms.shzr&gameid=ms&uuid1=2ced6028e80a433886db6db82e9bfe8e&devicetype=iPhone&deviceversion=6.0&versionid=8.1.1&f=838000707&detaildevicet
ype=iPhone3,1&apptype=ios&jailbroken=false&uuid2=f80462b8c55ac2a64c8812779b40afbe&domain=&connecttype=wifi HTTP/1.1" 200 43 "app:/com.renren.gameskit"
174.93.120.82 - - [13/Jan/2013:23:08:11 +0800] "GET http://tj.pet.mop.com/run_ane.html?
appid=elite.gen.ms.shzr&gameid=ms&uuid1=2ced6028e80a433886db6db82e9bfe8e&devicetype=iPhone&deviceversion=6.0&versionid=8.1.1&f=838000707&detaildevicet
ype=iPhone3,1&apptype=ios&jailbroken=false&uuid2=f80462b8c55ac2a64c8812779b40afbe&domain=&connecttype=wifi HTTP/1.1" 200 43 "app:/com.renren.gameskit"
——————————————————————————————————————————————————————————————————————————————
GET /loader_run.html
180.172.254.182 - - [13/Jan/2013:23:00:01 +0800] "GET /loader_run.html?
appid=air.nceo.as.android8560021510&sdkver=0.0.16&gameid=null&f=8560021510&apptype=android&versionid=0.0.16&uuid2=354356050087566&detaildevicetype=HTC
+T329t&imei=354356050087566&uuid1=810816ca06a04f2a3b468adbb3616aff&mac=a0f450ff66e7&jailbroken=false&devicetype=HTC
+T329t&domain=&deviceversion=&connecttype=4.0.4 HTTP/1.1" 200 3 "-"
1.192.245.84 - - [13/Jan/2013:23:00:02 +0800] "GET /loader_run.html?
appid=air.nceo.as.android8560021519&sdkver=0.0.14&gameid=null&f=8560021519&apptype=android&versionid=0.0.14&uuid2=866508010354368&detaildevicetype=ONN
+K7&imei=866508010354368&uuid1=44d6b021cc7d04d1fb78d2bfeb1adfee&mac=5ce2f4f4ab17&jailbroken=false&devicetype=ONN
+K7&domain=&deviceversion=&connecttype=4.0.4 HTTP/1.1" 200 3 "-"
112.93.35.68 - - [13/Jan/2013:23:00:03 +0800] "GET /loader_run.html?
appid=air.nceo.as.android8560021518&sdkver=0.0.15&gameid=null&f=8560021518&apptype=android&versionid=0.0.15&uuid2=359548049282598&detaildevicetype=GT-
N7000&imei=359548049282598&uuid1=946161396c64523669bc52f7cf39a6cb&mac=50CCF8AABE67&jailbroken=false&devicetype=GT-
N7000&domain=&deviceversion=&connecttype=2.3.6 HTTP/1.1" 200 3 "-"
——————————————————————————————————————————————————————————————————————————————
GET /login_ane.html
101.85.147.146 - - [13/Jan/2013:23:00:00 +0800] "GET /login_ane.html?
uuid2=70f784738fe0d2f645e15df6de3ffade&devicetype=iPad&mac=40300426A829&detaildevicetype=iPad2,2&connecttype=wifi&sdkver=1.0.7&uuid1=0e7431baeb7846c
0b1caa73a5647684b&apptype=ios&jailbroken=false&imei=null&deviceversion=6.0.1&userid=447393827&roleid=294567961174220907 HTTP/1.1" 200 46 "-"
218.83.227.131 - - [13/Jan/2013:23:00:00 +0800] "GET /login_ane.html?
f=925004401&gameid=lstx&domain=http://x66.lstx.renren.com&appid=588116262&versionid=1.5.0.1&devicetype=iPad&deviceversion=6.0&jailbroken=false&connectt
ype=wifi&uuid1=93f3dc7b9e2547149a980e94cbb02cbe&udid2=6137ccd66654bc518207c33e460c8986&mac=4c:b1:99:08:bc:d8 HTTP/1.1" 200 43 "-"
210.242.214.26 - - [13/Jan/2013:23:00:00 +0800] "GET /login_ane.html?
uuid2=09b67afe070498e4c2601f1be7d33f30&devicetype=iPad&mac=A4D1D25BF26B&detaildevicetype=iPad2,1&connecttype=wifi&sdkver=1.0.7&uuid1=65fe6e70e77c457
89fe988c1ace8bb38&apptype=ios&jailbroken=false&imei=null&deviceversion=6.0.1&userid=8089268&roleid=290768048988618933 HTTP/1.1" 200 46 "-"
——————————————————————————————————————————————————————————————————————————————
GET /login.html
115.197.239.203 - - [13/Jan/2013:23:00:01 +0800] "GET /login.html?
lhsg*ios*870001315*null*efc073686f92362f2663fcc7cc6e64c97eae0c8d*iPad*5.1.1*com.renren.game.nqj2hd*1.0.1 HTTP/1.1" 200 3 "app:/lhsg.swf"
115.197.239.203 - - [13/Jan/2013:23:00:01 +0800] "GET /login.html?
lhsg*ios*870001315*x10.lhsg.renren.com*efc073686f92362f2663fcc7cc6e64c97eae0c8d*iPad*5.1.1*com.renren.game.nqj2hd*1.0.1 HTTP/1.1" 200 3 "app:/lhsg.swf"
111.110.226.84 - - [13/Jan/2013:23:00:03 +0800] "GET /login.html?
lstx*android*8820017*http://x14.tx.rrgame.jp*SSHGE005483_a1000016c6b272_78:1C:5A:3C:34:74*ANDROID_IS17SH*4.0.4*and10301*1.3.317 HTTP/1.1" 200 3 "-"
——————————————————————————————————————————————————————————————————————————————
GET /robots.txt HTTP/1.0
62.24.222.131 - - [13/Jan/2013:23:26:17 +0800] "GET /robots.txt HTTP/1.0" 404 564 "http://tj.pet.mop.com/robots.txt"
GET /robots.txt HTTP/1.1
180.76.5.91 - - [13/Jan/2013:23:51:58 +0800] "GET /robots.txt HTTP/1.1" 404 162 "-"
180.76.5.49 - - [13/Jan/2013:23:52:01 +0800] "GET /robots.txt HTTP/1.1" 404 345
——————————————————————————————————————————————————————————————————————————————
GET /run_ane.html
112.96.131.124 - - [13/Jan/2013:22:59:59 +0800] "GET /run_ane.html?
appid=com.imop.game.IronMarshalX&gameid=ms&uuid1=abea9df0f87e49b7984f4ca373421a33&devicetype=iPhone&deviceversion=5.1.1&versionid=3.1.1&f=838000729&de
taildevicetype=iPhone4,1&apptype=ios&jailbroken=true&uuid2=a98488fc51257fc2ac574ab1983d054d&domain=&connecttype=wwan HTTP/1.1" 200 43
"app:/com.renren.gameskit"
114.221.147.156 - - [13/Jan/2013:22:59:59 +0800] "GET /run_ane.html?
uuid2=null&apptype=null&domain=&uuid1=null&devicetype=null&deviceversion=null&f=838000707&detaildevicetype=null&gameid=ms&jailbroken=null&versio
nid=1.0.8&connecttype=null&appid=com.msqy.weiduan HTTP/1.1" 200 43 "app:/Loading.swf/[[DYNAMIC]]/1"
221.194.31.155 - - [13/Jan/2013:23:00:00 +0800] "GET /run_ane.html?f=925004443&gameid=jzsg&domain={domain}
&appid=564762410&versionid=1.0.8&devicetype=iPhone&deviceversion=4.3.3&jailbroken=false&connecttype=wwan&uuid1=c95a9e14a1e34702999b57fb1e9ddb53&udid
2=041357b8146f516b3ca9a96b9e88cc90&mac=14:5a:05:e7:6c:25 HTTP/1.1" 200 43 "app:/musouClient.swf"
——————————————————————————————————————————————————————————————————————————————
GET /run.html
175.141.66.138 - - [13/Jan/2013:23:00:01 +0800] "GET /run.html?
lstx*android*925010017*http://x82.lstx.renren.com*cff19b4b01dc53f_358401049472325_98:0C:82:F0:BA:B2*ANDROID_GT-I9100G*4.0.3*and10101*1.3.318 HTTP/1.1" 200 3 "-"
210.242.214.5 - - [13/Jan/2013:23:00:03 +0800] "GET /run.html?lstx*android*9250082140**HT243W104686_359188048172357_18:87:96:8b:dc:6d*ANDROID_HTC%20One
%20X*4.1.1*and10101*1.3.299 HTTP/1.1" 200 3 "-"
89.24.14.32 - - [13/Jan/2013:23:00:05 +0800] "GET /run.html?lstx*android*862004601**4df1eea16a789f3f_354833058446879_20:02:AF:B9:F7:F8*ANDROID_GT-
N7100*4.1.1*and02501*1.3.263 HTTP/1.1" 200 3 "-"
&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&
统计的url仅限：
次数 url
50924 GET /login_ane.html
72678 GET /end_ane.html
78684 GET /run_ane.html
13 GET http://tj.pet.mop.com/login_ane.html
16 GET http://tj.pet.mop.com/end_ane.html
16 GET http://tj.pet.mop.com/run_ane.html
……………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………
GET http://tj.pet.mop.com/run_ane.html √
216.36.142.199 - - [13/Jan/2013:23:03:52 +0800] "GET http://tj.pet.mop.com/run_ane.html?
appid=elite.gen.ms.shzr&gameid=ms&uuid1=2ced6028e80a433886db6db82e9bfe8e&devicetype=iPhone&deviceversion=6.0&versionid=8.1.1&f=838000707&detaildevicet
ype=iPhone3,1&apptype=ios&jailbroken=false&uuid2=f80462b8c55ac2a64c8812779b40afbe&domain=&connecttype=wifi HTTP/1.1" 200 43 "app:/com.renren.gameskit"
……………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………
GET http://tj.pet.mop.com/end_ane.html √
216.36.142.199 - - [13/Jan/2013:23:03:52 +0800] "GET http://tj.pet.mop.com/end_ane.html?
appid=elite.gen.ms.shzr&gameid=ms&uuid1=2ced6028e80a433886db6db82e9bfe8e&devicetype=iPhone&onlinetime=4&endtime=1357999139&deviceversion=6.0&version
id=8.1.1&starttime=1357999135&launchtime=1357998799&f=838000707&detaildevicetype=iPhone3,1&userid=467415015&activetime=220&apptype=ios&jailbroken=f
alse&startcount=5&uuid2=f80462b8c55ac2a64c8812779b40afbe&domain=&connecttype=wifi&roleid=295834598569411941 HTTP/1.1" 200 43 "app:/com.renren.gameskit"
……………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………
GET http://tj.pet.mop.com/login_ane.html √
216.36.142.199 - - [13/Jan/2013:23:07:22 +0800] "GET http://tj.pet.mop.com/login_ane.html?
appid=elite.gen.ms.shzr&gameid=ms&uuid1=2ced6028e80a433886db6db82e9bfe8e&devicetype=iPhone&deviceversion=6.0&versionid=8.1.1&f=838000707&detaildevicet
ype=iPhone3,1&userid=454219948&apptype=ios&jailbroken=false&uuid2=f80462b8c55ac2a64c8812779b40afbe&domain=&connecttype=wifi&roleid=292456898848886
914 HTTP/1.1" 200 43 "app:/com.renren.gameskit"
……………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………
GET /run_ane.html √
112.96.131.124 - - [13/Jan/2013:22:59:59 +0800] "GET /run_ane.html?
appid=com.imop.game.IronMarshalX&gameid=ms&uuid1=abea9df0f87e49b7984f4ca373421a33&devicetype=iPhone&deviceversion=5.1.1&versionid=3.1.1&f=838000729&de
taildevicetype=iPhone4,1&apptype=ios&jailbroken=true&uuid2=a98488fc51257fc2ac574ab1983d054d&domain=&connecttype=wwan HTTP/1.1" 200 43
"app:/com.renren.gameskit"
……………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………
GET /end_ane.html √
221.194.30.138 - - [13/Jan/2013:22:59:59 +0800] "GET /end_ane.html?
connecttype=wwan&uuid1=d51c772dd73e48c29dd76269e021d418&mac=28E02C12BE8D&detaildevicetype=iPhone3,1&apptype=ios&versionid=1.2.1&deviceversion=6.0.1&i
mei=null&uuid2=bdd0d548eb1edbcd459dbf7f32cc4751&devicetype=iPhone&sdkver=1.0.7&gameid=rrnhzf&jailbroken=false&appid=com.imop.game.pirate2iphone&f=885
001201&domain=x39.nhzf.renren.com&userid=221106049&roleid=295271648616025261&starttime=1358088689&endtime=1358089200&onlinetime=511&launchtime=13580
88689&startcount=1&activetime=511 HTTP/1.1" 200 53 "-
……………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………
GET /login_ane.html √
101.85.147.146 - - [13/Jan/2013:23:00:00 +0800] "GET /login_ane.html?
uuid2=70f784738fe0d2f645e15df6de3ffade&devicetype=iPad&mac=40300426A829&detaildevicetype=iPad2,2&connecttype=wifi&sdkver=1.0.7&uuid1=0e7431baeb7846c
0b1caa73a5647684b&apptype=ios&jailbroken=false&imei=null&deviceversion=6.0.1&userid=447393827&roleid=294567961174220907 HTTP/1.1" 200 46 "-"
……………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………
hadoop fs -ls /data/raw/adwall/test
hadoop fs -rm /data/raw/adwall/test/adwall-1.0.jar
hadoop fs -put ./adwall-1.0.jar /data/raw/adwall/test
hadoop fs -put /data/var/zghtest/adwallSerde/target/adwall-1.0.jar /data/raw/adwall/test
hive> add jar /data/var/zghtest/adwall-1.0.jar;
Added /data/var/zghtest/adwall-1.0.jar to class path
Added resource: /data/var/zghtest/adwall-1.0.jar
cp /data/var/zghtest/adwallSerde/target/adwall-1.0.jar /data/var/zghtest
删除的话就是delete jar
delete jar /data/var/zghtest/adwall-1.0.jar;
建表语句相关
CREATE EXTERNAL TABLE raw.t_adwall_test
(
ip STRING ,
datee STRING ,
interfaze STRING
)
ROW FORMAT SERDE 'com.renren.aneSerDe.AneSerDe';
CREATE EXTERNAL TABLE raw.t_adwall_test2
(
ip STRING ,
datee STRING ,
interfaze STRING,
ipanddate STRING
)
ROW FORMAT SERDE 'com.renren.aneSerDe.AneSerDe';
CREATE EXTERNAL TABLE raw.t_adwall_test3
(
ip STRING ,
datee STRING ,
interfaze STRING,
ipanddate STRING
)
ROW FORMAT SERDE 'com.renren.aneSerDe.AneSerDe';
CREATE EXTERNAL TABLE raw.t_adwall
(
f_ip STRING ,
f_datee STRING ,
f_interfaze STRING,
f_appid STRING,
f_gameid STRING,
f_uuid1 STRING,
f_devicetype STRING,
f_deviceversion STRING,
f_versionid STRING,
f_f STRING,
f_detaildevicetype STRING,
f_apptype STRING,
f_jailbroken STRING,
f_uuid2 STRING,
f_domain STRING,
f_connecttype STRING,
f_roleid STRING,
f_mac STRING,
f_sdkver STRING,
f_userid STRING,
f_starttime BIGINT,
f_endtime BIGINT,
f_onlinetime BIGINT,
f_launchtime BIGINT,
f_startcount INT,
f_activetime BIGINT,
f_formatright SMALLINT,
f_formaterrorreason STRING,
f_raw STRING
)
ROW FORMAT SERDE 'com.renren.aneSerDe.AneSerDe';
LOAD DATA LOCAL INPATH '/data/var/zghtest/access.log' OVERWRITE INTO TABLE raw.t_adwall;
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
hive> select * from t_adwall where f_formatright=0 limit 10;
Total MapReduce jobs = 1
Launching Job 1 out of 1
Number of reduce tasks is set to 0 since there's no reduce operator
Starting Job = job_201206261006_286591, Tracking URL = http://h1-1.io8.org:50030/jobdetails.jsp?jobid=job_201206261006_286591
Kill Command = /data/opt/hadoop/bin/hadoop job -Dmapred.job.tracker=h1-1.io8.org:9001 -kill job_201206261006_286591
Hadoop job information for Stage-1: number of mappers: 2; number of reducers: 0
2013-01-16 22:30:26,367 Stage-1 map = 0%, reduce = 0%
2013-01-16 22:30:31,394 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 11.23 sec
2013-01-16 22:30:32,401 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 11.23 sec
2013-01-16 22:30:33,408 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 11.23 sec
MapReduce Total cumulative CPU time: 11 seconds 230 msec
Ended Job = job_201206261006_286591
MapReduce Jobs Launched:
Job 0: Map: 2 Accumulative CPU: 11.23 sec HDFS Read: 33726867 HDFS Write: 2843 SUCESS
Total MapReduce CPU Time Spent: 11 seconds 230 msec
OK
1024null null null com.msqy.weiduan ms null null null 1.0.8 838000707 null null null null null null null null NULL 1358089234
NULL 0 1358089234 1 0 0 {"checkRule3_5":"no_mac"}
1024null null null 520025006 jzsg 39bfe5c7178c4882a06a19202f0aee6b iPad 5.1.1 1.0.6 iPad3,1 ios false ec147b2821c8030b4c6812538ef5d9ed
http://x41.s.jzsg.renren.com wifi null null null NULL 1358089273 1358089273 0 1358089273 1 0 0 {"checkRule3_5":"no_mac"}
1024null null null elite.gen.ms.shzr ms 27329507945c40afbf588a5330a2246b iPhone 6.0.1 8.1.1 838000707 iPhone4,1 ios false
d38a920af438c405b08ee7d150d35bd4 wifi null HTTP/1.1" 200 43 "app:/com.renren.gameskit" null null NULL 1357915585 1357915585 0 1357915585
1 0 0 {"checkRule3_5":"no_mac"}
1024null null null com.imop.game.wosiphone ms 407c7060d95946fdae79054f307bb1e3 iPad 6.0.1 1.0.7 838000707 iPad2,2 ios false
338955105bad19c4f57641b05eb0a77e wifi null HTTP/1.1" 200 43 "app:/com.renren.gameskit" null null NULL 1358077428 1358077428 0 1358077428
1 0 0 {"checkRule3_5":"no_mac"}
1024null null null com.imop.game.wosiphone ms 8c81ef15e8ea4c5fbcf8856153bef980 iPhone 4.3.1 1.0.6 838000707 iPhone3,1 ios true
f4db9de6d3d08e956e3378f03a3ceb70 wifi null HTTP/1.1" 200 43 "app:/com.renren.gameskit" null null NULL 1357871821 1357871821 0 1357871821
1 0 0 {"checkRule3_5":"no_mac"}
1024null null null com.msqy.weiduan ms null null null 1.0.8 HTTP/1.1" 200 43 "app:/Loading.swf/[[DYNAMIC]]/1" 838000707 null null null null
null null null null NULL 1358089493 1358089493 0 1358089493 1 0 0 {"checkRule3_5":"no_mac"}
1024null null null elite.gen.ms.shzr ms 01bd91e55d564dde83d665851328f4b7 iPhone 6.0.1 8.1.1 838000707 iPhone4,1 ios false
f427006b61c116e6ea3c53d18e5e2f9e wifi null HTTP/1.1" 200 43 "app:/com.renren.gameskit" null null NULL 1358089478 1358089478 0 1358089478
1 0 0 {"checkRule3_5":"no_mac"}
1024null null null com.imop.game.wos1iphone ms 970b9f38cd464251a53bf39c2f3ebddc iPhone 6.0.1 1.1.6 838000707 iPhone4,1 ios false
5133ffcde7cf5c69d7fd141e436c265e wifi null HTTP/1.1" 200 43 "app:/com.renren.gameskit" null null NULL 1358088195 1358088195 0 1358088195
1 0 0 {"checkRule3_5":"no_mac"}
1024null null null wostest ms HTTP/1.1" 200 43 "app:/Loading.swf/[[DYNAMIC]]/1" null null null 00.00.04 838000707 null null null null
nullnull null null NULL 1358089515 1358089515 0 1358089515 1 0 0 {"checkRule3_5":"no_mac"}
1024null null null and10101 lstx 2842f80102034bc1b59f674ae2649f27 MB525 2.3.7 HTTP/1.1" 200 43 "app:/com.renren.gameskit" 1.6.075 925010018
MB525 android false 2842f80102034bc1b59f674ae2649f27 lstx.renren.com 2G null 984b4a1561ab 1.0.7 NULL 1358088814 1358088814 0 1358088814
1 0 1 {}
Time taken: 14.202 seconds
hive>
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
hive -S -e "use raw;add jar /data/var/zghtest/adwall-1.0.jar; select * from t_adwall where f_formatright=1 limit 10;" >> /data/var/zghtest.cvs
set hive.cli.print.header=true;
set hive.cli.print.row.to.vertical=true;
set hive.cli.print.row.to.vertical.num=1;
开启显示列名的功能。
CREATE EXTERNAL TABLE dw0.t_adwall
(
f_ip STRING ,
f_datee STRING ,
f_interfaze STRING,
f_appid STRING,
f_gameid STRING,
f_uuid1 STRING,
f_devicetype STRING,
f_deviceversion STRING,
f_versionid STRING,
f_f STRING,
f_detaildevicetype STRING,
f_apptype STRING,
f_jailbroken STRING,
f_uuid2 STRING,
f_domain STRING,
f_connecttype STRING,
f_roleid STRING,
f_mac STRING,
f_sdkver STRING,
f_userid STRING,
f_starttime BIGINT,
f_endtime BIGINT,
f_onlinetime BIGINT,
f_launchtime BIGINT,
f_startcount INT,
f_activetime BIGINT,
f_formatright SMALLINT,
f_formaterrorreason STRING,
f_raw STRING
)
建立洗过一次的数据表：
insert overwrite table dw0.t_adwall
select f_ip, f_datee,f_interfaze, f_appid, f_gameid , f_uuid1, f_devicetype , f_deviceversion
, f_versionid,f_f , f_detaildevicetype , f_apptype , f_jailbroken, f_uuid2, f_domain, f_connecttype,
f_roleid , f_mac , f_sdkver , f_userid , f_starttime , f_endtime , f_onlinetime , f_launchtime , f_startcount ,
f_activetime , f_formatright , f_formaterrorreason , f_raw
from raw. t_adwall
新的查询结果：
hive> select * from dw0.t_adwall where f_formatright=1 limit 10;
Total MapReduce jobs = 1
Launching Job 1 out of 1
Number of reduce tasks is set to 0 since there's no reduce operator
Starting Job = job_201206261006_287114, Tracking URL = http://h1-1.io8.org:50030/jobdetails.jsp?jobid=job_201206261006_287114
Kill Command = /data/opt/hadoop/bin/hadoop job -Dmapred.job.tracker=h1-1.io8.org:9001 -kill job_201206261006_287114
Hadoop job information for Stage-1: number of mappers: 2; number of reducers: 0
2013-01-17 17:01:10,566 Stage-1 map = 0%, reduce = 0%
2013-01-17 17:01:13,580 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 2.55 sec
2013-01-17 17:01:14,585 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 2.55 sec
MapReduce Total cumulative CPU time: 2 seconds 550 msec
Ended Job = job_201206261006_287114
MapReduce Jobs Launched:
Job 0: Map: 2 Accumulative CPU: 2.55 sec HDFS Read: 140255 HDFS Write: 6541 SUCESS
Total MapReduce CPU Time Spent: 2 seconds 550 msec
OK
f_ip f_datee f_interfaze f_appid f_gameid f_uuid1 f_devicetype f_deviceversion f_versionid f_f f_detaildevicetype f_apptype f_jailbroken f_uuid2 f_domain f_connecttype f_roleid f_mac f_sdkver f_userid f_starttime f_endtime f_onlinetime f_launchtime f_startcount f_activetime f_formatright f_formaterrorreason f_raw
221.194.30.138 - null "GET /end_ane com.imop.game.pirate2iphone rrnhzf d51c772dd73e48c29dd76269e021d418 iPhone 6.0.1 1.2.1 885001201 iPhone3,1 ios falsbdd0d548eb1edbcd459dbf7f32cc4751 x39.nhzf.renren.com null 295271648616025261 28E02C12BE8D 1.0.7 221106049 1358088689 1358089200 511 1358088689 1 NULL{} 221.194.30.138 - - [13/Jan/2013:22:59:59 +0800] "GET /end_ane.html?connecttype=wwan&uuid1=d51c772dd73e48c29dd76269e021d418&mac=28E02C12BE8D&detaildevicetype=iPhone3,1&apptype=ios&versionid=1.2.1&deviceversion=6.0.1&imei=null&uuid2=bdd0d548eb1edbcd459dbf7f32cc4751&devicetype=iPhone&sdkver=1.0.7&gameid=rrnhzf&jailbroken=false&appid=com.imop.game.pirate2iphone&f=885001201&domain=x39.nhzf.renren.com&userid=221106049&roleid=295271648616025261&starttime=1358088689&endtime=1358089200&onlinetime=511&launchtime=1358088689&startcount=1&activetime=511 HTTP/1.1" 200 53 "-"
175.142.180.206 null "GET /end_ane com.renren.game.csj csj 0d40e54d523a48428ad10ecc5253e153 iPad 4.3.1 1.1.2 null iPad2,2 ios false ee5bdbc81fb02ee3bc0b362a5281a935 x21.csj.renren.com null null A467069329AE 1.0.8 null 1358089192 1358089200 8 1358089192 1 NULL 1 {} 175.142.180.206 - - [13/Jan/2013:23:00:00 +0800] "GET /end_ane.html?connecttype=wifi&uuid1=0d40e54d523a48428ad10ecc5253e153&mac=A467069329AE&detaildevicetype=iPad2,2&apptype=ios&versionid=1.1.2&deviceversion=4.3.1&imei=null&uuid2=ee5bdbc81fb02ee3bc0b362a5281a935&devicetype=iPad&sdkver=1.0.8&gameid=csj&jailbroken=false&appid=com.renren.game.csj&f=null&domain=x21.csj.renren.com&userid=null&roleid=null&starttime=1358089192&endtime=1358089200&onlinetime=8&launchtime=1358089192&startcount=1&activetime=8 HTTP/1.1" 200 46 "-"
221.5.86.37 - - null "GET /end_ane com.renren.games.csj4 com.renren.games.csj4 e368ba2a6ec14a02bb32cb5cef284027 iPhone 5.0.1 1.1.4 8560011158 iPhone4,1 nulltrue eaacae952f7b421abf062978cb8898d1 x32.csj.renren.com wifi 297241973452978776 60FACD22FC83 1.0.9 505529877 1358088339 1358089202 863 1358088339 NULL 1 {} 221.5.86.37 - - [13/Jan/2013:23:00:00 +0800] "GET /end_ane.html?apptype=ios&uuid1=e368ba2a6ec14a02bb32cb5cef284027&mac=60FACD22FC83&detaildevicetype=iPhone4,1&versionid=1.1.4&platformid=null&roleid=297241973452978776&deviceversion=5.0.1&imei=null&uuid2=eaacae952f7b421abf062978cb8898d1&devicetype=iPhone&sdkver=1.0.9&gameid=com.renren.games.csj4&jailbroken=true&appid=com.renren.games.csj4&f=8560011158&domain=x32.csj.renren.com&userid=505529877&connecttype=wifi&starttime=1358088339&launchtime=1358088339&endtime=1358089202&onlinetime=863&startcount=1&activetime=863 HTTP/1.1" 200 53 "-"
112.96.30.111 - null "GET /end_ane air.nceo.as.android8560021732 air.nceo.as.android8560021732 f440d14885466879b11dc79e1b7b2346 GT-I9100 2.3.3 1.1.4 8560021732 GT-I9100 android false f440d14885466879b11dc79e1b7b2346 x30.csj.renren.com 3G 296679023499553278 null 1.0.9 503021509 1358089185 1358089253 68 1358088880 3 NULL 1 {} 112.96.30.111 - - [13/Jan/2013:23:00:00 +0800] "GET /end_ane.html?activetime=230&appid=air.nceo.as.android8560021732&detaildevicetype=GT-I9100&endtime=1358089253&startcount=3&sdkver=1.0.9&gameid=air.nceo.as.android8560021732&jailbroken=false&connecttype=3G&apptype=android&f=8560021732&domain=x30.csj.renren.com&starttime=1358089185&uuid2=f440d14885466879b11dc79e1b7b2346&versionid=1.1.4&launchtime=1358088880&imei=359090046272710&platformid=null&uuid1=f440d14885466879b11dc79e1b7b2346&mac=null&devicetype=GT-I9100&userid=503021509&onlinetime=68&deviceversion=2.3.3&roleid=296679023499553278 HTTP/1.0" 200 43 "app:/com.renren.gameskit"
58.243.250.37 - null "GET /end_ane com.renren.game.csj csj 60124ca558884db38eb5e50c024f72bc iPhone 6.0.1 1.1.2 null iPhone4,1 ios false 5ef2936b2d7fe62766648bd07394e59b x10.csj.renren.com null 291049523965365363 5C95AE8A5860 1.0.8 487981608 1358086504 1358089197 2693 1358086504 1 NULL 1 {} 58.243.250.37 - - [13/Jan/2013:23:00:00 +0800] "GET /end_ane.html?connecttype=wwan&uuid1=60124ca558884db38eb5e50c024f72bc&mac=5C95AE8A5860&detaildevicetype=iPhone4,1&apptype=ios&versionid=1.1.2&deviceversion=6.0.1&imei=null&uuid2=5ef2936b2d7fe62766648bd07394e59b&devicetype=iPhone&sdkver=1.0.8&gameid=csj&jailbroken=false&appid=com.renren.game.csj&f=null&domain=x10.csj.renren.com&userid=487981608&roleid=291049523965365363&starttime=1358086504&endtime=1358089197&onlinetime=2693&launchtime=1358086504&startcount=1&activetime=2693 HTTP/1.1" 200 46 "-"
58.243.250.37 - null "GET /end_ane com.renren.game.csj csj 60124ca558884db38eb5e50c024f72bc iPhone 6.0.1 1.1.2 null iPhone4,1 ios false 5ef2936b2d7fe62766648bd07394e59b x10.csj.renren.com null 291049523965365363 5C95AE8A5860 1.0.8 487981608 1358086504 1358089197 2693 1358086504 1 NULL 1 {} 58.243.250.37 - - [13/Jan/2013:23:00:00 +0800] "GET /end_ane.html?connecttype=wwan&uuid1=60124ca558884db38eb5e50c024f72bc&mac=5C95AE8A5860&detaildevicetype=iPhone4,1&apptype=ios&versionid=1.1.2&deviceversion=6.0.1&imei=null&uuid2=5ef2936b2d7fe62766648bd07394e59b&devicetype=iPhone&sdkver=1.0.8&gameid=csj&jailbroken=false&appid=com.renren.game.csj&f=null&domain=x10.csj.renren.com&userid=487981608&roleid=291049523965365363&starttime=1358086504&endtime=1358089197&onlinetime=2693&launchtime=1358086504&startcount=1&activetime=2693 HTTP/1.1" 200 46 "-"
112.224.67.29 - null "GET /end_ane com.renren.game.csjhd com.renren.game.csjhd d23c7659b1d944e190a89c22ea1b4997 iPhone 4.3.5 1.1.3 856001130 iPhone3,1 nullfalse 4305f44c04c402abcf58b36e005fcf68 x23.csj.renren.com wwan 294708698662585180 148FC67E6E0A 1.0.9 496599565 1358089157 1358089199 42 1358066165 NULL 1 {} 112.224.67.29 - - [13/Jan/2013:23:00:00 +0800] "GET /end_ane.html?apptype=ios&uuid1=d23c7659b1d944e190a89c22ea1b4997&mac=148FC67E6E0A&detaildevicetype=iPhone3,1&versionid=1.1.3&platformid=null&roleid=294708698662585180&deviceversion=4.3.5&imei=null&uuid2=4305f44c04c402abcf58b36e005fcf68&devicetype=iPhone&sdkver=1.0.9&gameid=com.renren.game.csjhd&jailbroken=false&appid=com.renren.game.csjhd&f=856001130&domain=x23.csj.renren.com&userid=496599565&connecttype=wwan&starttime=1358089157&launchtime=1358066165&endtime=1358089199&onlinetime=42&startcount=2&activetime=721 HTTP/1.1" 200 53 "-"
58.22.114.175 - null "GET /end_ane 494778270 cq dc06015fd37842eea06b2143eacd1c2a iPhone 5.1.1 1.5.3 855000877 iPhone4,1 null false 1cd5a157ec972135bba67b722b50ba99 http://s51.cq.renren.com/ wwan 043304000000453c 0C771A63EFF8 1.0.9 499040717 1358083127 1358084580 1453 1358083127 1 NULL{} 58.22.114.175 - - [13/Jan/2013:23:00:00 +0800] "GET /end_ane.html?apptype=ios&uuid1=dc06015fd37842eea06b2143eacd1c2a&mac=0C771A63EFF8&detaildevicetype=iPhone4,1&versionid=1.5.3&platformid=null&roleid=043304000000453c&deviceversion=5.1.1&imei=null&uuid2=1cd5a157ec972135bba67b722b50ba99&devicetype=iPhone&sdkver=1.0.9&gameid=cq&jailbroken=false&appid=494778270&f=855000877&domain=http://s51.cq.renren.com/&userid=499040717&connecttype=wwan&starttime=1358083127&launchtime=1358083127&endtime=1358084580&onlinetime=1453&startcount=1&activetime=1453 HTTP/1.1" 200 53 "-"
211.140.5.110 - null "GET /run_ane 511848500 cq null GT-I9018 2.2.2 1.5.3 855002850 GT-I9018 android false 537dfb91f38b2964d6fe60ad4c0c8558 http://s59.cq.renren.com/ 2G null 1234569D4E60 1.1.0 null 1358089277 NULL NULL 1358089277 NULL NULL 1 {} 211.140.5.110 - - [13/Jan/2013:23:00:00 +0800] "GET /run_ane.html?uuid1=537dfb91f38b2964d6fe60ad4c0c8558&uuid2=537dfb91f38b2964d6fe60ad4c0c8558&versionid=1.5.3&imei=867715001384978&gameid=cq&mac=1234569D4E60&detaildevicetype=GT-I9018&f=855002850&userid=null&connecttype=2G&platformid=null&roleid=null&starttime=1358089277&jailbroken=false&devicetype=GT-I9018&sdkver=1.1.0&deviceversion=2.2.2&appid=511848500&domain=http://s59.cq.renren.com/&launchtime=1358089277&apptype=android HTTP/1.1" 200 53 "app:/com.renren.gameskit"
182.249.240.38 - null "GET /run_ane 563138738 cq aa78cb3cfb8347bfddebe4f4b686defc URBANO%20PROGRESSO 4.0.3 1.2.1 821000401 URBANO%20PROGRESSO android false null http://x2.jcq.rrgame.jp/ 2G null 245fdf41e931 1.0.7 null NULL NULL NULL NULL NULL NULL 1 {} 182.249.240.38 - - [13/Jan/2013:23:00:00 +0800] "GET /run_ane.html?uuid2=aa78cb3cfb8347bfddebe4f4b686defc&devicetype=URBANO%20PROGRESSO&deviceversion=4.0.3&versionid=1.2.1&f=821000401&appid=563138738&detaildevicetype=URBANO%20PROGRESSO&apptype=android&gameid=cq&uuid1=aa78cb3cfb8347bfddebe4f4b686defc&sdkver=1.0.7&domain=http://x2.jcq.rrgame.jp/&jailbroken=false&imei=35228005085670&connecttype=2G&mac=245fdf41e931 HTTP/1.1" 200 43 "app:/com.renren.gameskit"
Time taken: 8.832 seconds
hive>
分享到：
发表评论
相关推荐
这里，我们指定了Nginx日志文件的路径，设置了一个内存型Channel用于短暂存储数据，并配置了HiveSink，将数据写入名为`nginx_logs`的Hive表。 然后，Flume需要与Hive、Hadoop集群进行交互，因此在运行Flume之前，...
Flume采集Nginx日志到Hive的事务表时需要导入到Flume下的Jar文件，具体使用方式可参见博文：https://blog.csdn.net/l1028386804/article/details/97975539
nginx + lua 日志文件埋点的 基于Hadoop网站流量日志数据分析系统 1、典型的离线流数据分析系统 2、技术分析 - Hadoop - nginx - flume - hive - mysql - springboot + mybatisplus+vcharts nginx + lua 日志文件...
【标题】"taotao-weblog-analysis基于openresty kafka hadoop hive 日志点击流数据分析"涉及的关键技术点包括OpenResty、Kafka、Hadoop和Hive，这些都是大数据处理和分析领域的重要组件。 OpenResty是基于Nginx与...
数据分析层使用Hive、MapReduce和Spark对日志文件进行清洗、分析和提取出需要的价值信息。数据展示层使用SpringMVC和Highcharts对统计分析后的结果进行展示。 项目技术选型和特点： 1. JsSDK：捕获前端页面数据，...
JsSDK 用于捕获前端页面数据，JavaSDK 用于后台服务日志生成与发送工具，Nginx 用于生成日志文件，Flume 用于上传日志文件到文件存储系统中，HBase 用于存储解析后的数据，Hive 用于对日志文件进行清洗、分析和提取...
这是一个Logparsing框架，旨在简化和访问日志文件的解析。 基本思想是，您应该能够拥有一个解析器，您可以通过简单地告诉该行写入了哪些配置选项来构造该解析器。 这些配置选项是访问日志行的架构。 因此，我们...
这个系统允许用户通过Web界面执行HQL（Hive Query Language）语句，进行大数据的分析和操作。 【描述】: 这个系统的设计目的是为非技术背景的用户提供一个易用的接口，以便他们可以方便地与Hadoop集群上的Hive...
- 通过Nginx服务器生成日志文件，收集前端发送的行为数据。 - 利用Shell脚本和Flume将日志文件上传到文件存储系统（通常是HDFS）。 - 使用MapReduce作业对日志文件进行清洗，去除脏数据和不必要的字段。 - 清洗...
- **Shell脚本**：对于较小规模的数据，使用Shell脚本分割和上传日志文件到HDFS。 - **Flume**：更大量的数据可以通过Flume进行收集和传输，它是一个可靠的、分布式的、可用于聚合和移动大量数据的工具。 3. **...
预处理后的数据通常会被存入适合大数据处理的系统，如Hadoop HDFS、NoSQL数据库（如MongoDB）、数据仓库（如Hive）或分布式文件系统（如Spark的DataFrame）。 8. 数据预处理代码实现： 实现预处理代码可能涉及...