黎明lm
- 浏览: 307790 次
- 性别:
- 来自: 北京
-
文章分类
- 全部博客 (165)
- hadoop (47)
- linux (11)
- nutch (7)
- hbase (7)
- solr (4)
- zookeeper (4)
- J2EE (1)
- jquery (3)
- java (17)
- mysql (14)
- perl (2)
- compass (4)
- suse (2)
- memcache (1)
- as (1)
- roller (1)
- web (7)
- MongoDB (8)
- struts2 (3)
- lucene (2)
- 算法 (4)
- 中文分词 (3)
- hive (17)
- noIT (1)
- 中间件 (2)
- maven (2)
- sd (0)
- php (2)
- asdf (0)
- kerberos 安装 (1)
- git (1)
- osgi (1)
- impala (1)
- book (1)
- python 安装 科学计算包 (1)
最新评论
-
dandongsoft：
你写的不好用啊
solr 同义词搜索 -
黎明lm：
meifangzi 写道楼主真厉害 都分析源码了 用了很久. ...
hadoop 源码分析(二) jobClient 通过RPC 代理提交作业到JobTracker -
meifangzi：
楼主真厉害 都分析源码了
hadoop 源码分析(二) jobClient 通过RPC 代理提交作业到JobTracker -
zhdkn：
顶一个，最近也在学习设计模式，发现一个问题，如果老是看别人的博 ...
Java观察者模式（Observer）详解及应用 -
lvwenwen：
木南飘香 写道
高并发网站的架构
nutch1.3 +hadoop 分布式部署(亲测)
1.确保hadoop正常启动
2.下载nutch1.3 安装包 解压到指定路径
3.抓取
nutch1.3 有两个conf 一个在NUTCH_HOME/conf ,另一个在rumtime/local/conf
runtime/local/conf 为 local(本地抓取的配置文件所用)
NUTCH_HOME/conf 为分布式抓取所用
下面我们着重讲解 分布式抓取
4.分布式抓取：
rutime/deply/bin/nutch下执行分布式抓取命令(分布式抓取一定是在这个下面，local为本地抓取所用)
chmod +x bin/nutch 赋予执行权限
5.拷贝hadoop环境
将HADOOP_HOME/conf下的 6个文件：
core-site.xml
hadoop-env.sh
hdfs-site.xml
mapred-site.xml
masters
slaves
拷贝到NUTCH_HOME/conf下
6.配置nutch-site.xml
简单配置一个http.agent.name 即可
<property>
<name>http.agent.name</name>
<value>MyCrawl001</value>
</property>
7.配置regex-urlfilter.txt抓取动态网页
# skip file: ftp: and mailto: urls
-^(file|ftp|mailto):
# skip image and other suffixes we can't yet parse
-\.(gif|GIF|jpg|JPG|png|PNG|ico|ICO|css|sit|eps|wmf|zip|ppt|mpg|xls|gz|rpm|tgz|mov|MOV|exe|jpeg|JPEG|bmp|BMP)$
# skip URLs containing certain characters as probable queries, etc.
+[?*!@=]
# skip URLs with slash-delimited segment that repeats 3+ times, to break loops
-.*(/[^/]+)/[^/]+\1/[^/]+\1/
# accept anything else
+.
8.native/lib
这是配置nutch1.3 hadoop集群最重要一点
下面是NUTCH_HOME/lib/native下的 文档README.txt
These libraries are purely optional, and if they are missing Hadoop will
use corresponding pure Java components. The impact of native compression
becomes noticeable with larger datasets and weaker CPU-s - if you notice
that the CPU is routinely saturated when a job is sorting or reducing,
then using these libs may help.
Installation instructions
=========================
You can obtain the necessary files from a distribution package of Hadoop,
e.g. hadoop-0.20.2.tar.gz. Unpack this archive, and copy the content of
lib/native here, so that the layout looks like this:
<Nutch home>/lib/native/Linux-amd64-64/...
<Nutch home>/lib/native/Linux-i386-32/...
Local runtime
-------------
The build process will include these native libraries when preparing
the /runtime/local environment for running in local mode.
/runtime/local/bin/nutch knows how to use these libs - if they are
found and correctly used you should see lines like this in your logs:
Distributed runtime
-------------------
If you want to use this component in an existing Hadoop cluster (when using
/runtime/deploy artifacts) you need to make sure these files are placed in
Hadoop/lib/native directory on each node, and then restart the cluster. If
you installed the cluster from a distribution package of Hadoop then these
libraries should already be in the right place and you shouldn't need to do
anything else.
~
大体意思就是说 可以将 HADOOP_HOME下的 lib/native中的文件
Linux-amd64-64
Linux-i386-32
拷贝到NUTCH_HOME/lib/native下(按英文原意要确保you need to make sure these files are placed in
Hadoop/lib/native directory on each node, and then restart the cluster 确保这个文件在每一个节点上 并且重启集群,我拷贝了,)
9.执行
runtime/deply/bin/nutch hdfs://server0:9000/user/suse/urls -dir crawl -depth 200 -threads 200 -topN 1000
10.成功
看到map-reduce 任务成功执行 则配置成功
11/08/22 16:33:26 INFO mapred.JobClient: Reduce input records=48148
11/08/22 16:33:26 INFO crawl.CrawlDb: CrawlDb update: finished at 2011-08-22 16:33:26, elapsed: 00:00:39
11/08/22 16:33:26 INFO crawl.Generator: Generator: starting at 2011-08-22 16:33:26
11/08/22 16:33:26 INFO crawl.Generator: Generator: Selecting best-scoring urls due for fetch.
11/08/22 16:33:26 INFO crawl.Generator: Generator: filtering: true
11/08/22 16:33:26 INFO crawl.Generator: Generator: normalizing: true
11/08/22 16:33:26 INFO crawl.Generator: Generator: topN: 1000
11/08/22 16:33:27 INFO mapred.FileInputFormat: Total input paths to process : 8
11/08/22 16:33:28 INFO mapred.JobClient: Running job: job_201108221601_0022
11/08/22 16:33:29 INFO mapred.JobClient: map 0% reduce 0%
11/08/22 16:33:36 INFO mapred.JobClient: map 25% reduce 0%
11/08/22 16:33:39 INFO mapred.JobClient: map 50% reduce 0%
11/08/22 16:33:42 INFO mapred.JobClient: map 75% reduce 0%
11/08/22 16:33:45 INFO mapred.JobClient: map 75% reduce 4%
11/08/22 16:33:46 INFO mapred.JobClient: map 100% reduce 4%
11/08/22 16:33:48 INFO mapred.JobClient: map 100% reduce 13%
11/08/22 16:33:49 INFO mapred.JobClient: map 100% reduce 19%
11/08/22 16:33:51 INFO mapred.JobClient: map 100% reduce 33%
11/08/22 16:33:54 INFO mapred.JobClient: map 100% reduce 53%
11/08/22 16:33:57 INFO mapred.JobClient: map 100% reduce 71%
11/08/22 16:33:58 INFO mapred.JobClient: map 100% reduce 90%
11/08/22 16:34:00 INFO mapred.JobClient: map 100% reduce 100%
11/08/22 16:34:02 INFO mapred.JobClient: Job complete: job_201108221601_0022
11.solr 索引
nutch1.3下只有三个文件夹
crawldb
linkdb
segments
必须要用solr去建立索引
12.遇到错误
question：
Exception in thread "main" java.lang.IllegalArgumentException: Fetcher: No agents listed in 'http.agent.name' property.
at org.apache.nutch.fetcher.Fetcher.checkConfiguration(Fetcher.java:1166)
at org.apache.nutch.fetcher.Fetcher.fetch(Fetcher.java:1068)
at org.apache.nutch.crawl.Crawl.run(Crawl.java:135)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
at org.apache.nutch.crawl.Crawl.main(Crawl.java:54)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:616)
at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
answer:
using 1.3? If so make sure you changed nutch-site.xml (and not default)
in runtime/local/conf Changing the conf in NUTCH_HOME/conf won't be copied
to the runtime dirs unless you rebuild with ant.
BTW why don't you ask on the mailing list instead? You are more likely to get some help there
这个是英文元版回答。主要意思就是 nutch1.3 修改完nutch-site.xml 要重新ant一下
question：
Caused by: java.lang.NullPointerException
at java.io.Reader.<init>(Reader.java:61)
at java.io.BufferedReader.<init>(BufferedReader.java:76)
at java.io.BufferedReader.<init>(BufferedReader.java:91)
at org.apache.nutch.urlfilter.api.RegexURLFilterBase.readRules(RegexURLFilterBase.java:180)
at org.apache.nutch.urlfilter.api.RegexURLFilterBase.setConf(RegexURLFilterBase.java:156)
at org.apache.nutch.plugin.Extension.getExtensionInstance(Extension.java:162)
at org.apache.nutch.net.URLFilters.<init>(URLFilters.java:57)
at org.apache.nutch.crawl.Injector$InjectMapper.configure(Injector.java:72)
... 18 more
answer：
在runtime/local/下执行分布式抓取则报这个错误
注意：分布式抓取一定是在runtime/deply下执行
bin/nutch 命令
urls是HDFS上url地址的文件夹
hdfs://202.193.58.99:8888/urls 这个应该是文件而不是文件夹,你直接用
bin/hadoop fs -ls 查看下你的urls文件在哪里 目录是什么 然后直接写上全路径或是相对路径 (根据实际情况)
完整路径：hdfs://202.193.58.99:8888/urls/url
错误还是一模一样。我好想在哪儿看过，这里用文件夹和集体的文件 都可以！
hdfs://202.193.58.99:8888/urls 这个应该是文件而不是文件夹,你直接用
bin/hadoop fs -ls 查看下你的urls文件在哪里 目录是什么 然后直接写上全路径或是相对路径 (根据实际情况)
可以多贴点错误日志？
Exception in thread "main" java.io.IOException: Job failed!
at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1252)
at org.apache.nutch.crawl.Injector.inject(Injector.java:217)
at org.apache.nutch.crawl.Crawl.run(Crawl.java:127)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
at org.apache.nutch.crawl.Crawl.main(Crawl.java:55)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
可以多贴点错误日志？
1.确保hadoop正常启动
2.下载nutch1.3 安装包 解压到指定路径
3.抓取
nutch1.3 有两个conf 一个在NUTCH_HOME/conf ,另一个在rumtime/local/conf
runtime/local/conf 为 local(本地抓取的配置文件所用)
NUTCH_HOME/conf 为分布式抓取所用
下面我们着重讲解 分布式抓取
4.分布式抓取：
rutime/deply/bin/nutch下执行分布式抓取命令(分布式抓取一定是在这个下面，local为本地抓取所用)
chmod +x bin/nutch 赋予执行权限
5.拷贝hadoop环境
将HADOOP_HOME/conf下的 6个文件：
core-site.xml
hadoop-env.sh
hdfs-site.xml
mapred-site.xml
masters
slaves
拷贝到NUTCH_HOME/conf下
6.配置nutch-site.xml
简单配置一个http.agent.name 即可
<property>
<name>http.agent.name</name>
<value>MyCrawl001</value>
</property>
7.配置regex-urlfilter.txt抓取动态网页
# skip file: ftp: and mailto: urls
-^(file|ftp|mailto):
# skip image and other suffixes we can't yet parse
-\.(gif|GIF|jpg|JPG|png|PNG|ico|ICO|css|sit|eps|wmf|zip|ppt|mpg|xls|gz|rpm|tgz|mov|MOV|exe|jpeg|JPEG|bmp|BMP)$
# skip URLs containing certain characters as probable queries, etc.
+[?*!@=]
# skip URLs with slash-delimited segment that repeats 3+ times, to break loops
-.*(/[^/]+)/[^/]+\1/[^/]+\1/
# accept anything else
+.
8.native/lib
这是配置nutch1.3 hadoop集群最重要一点
下面是NUTCH_HOME/lib/native下的 文档README.txt
These libraries are purely optional, and if they are missing Hadoop will
use corresponding pure Java components. The impact of native compression
becomes noticeable with larger datasets and weaker CPU-s - if you notice
that the CPU is routinely saturated when a job is sorting or reducing,
then using these libs may help.
Installation instructions
=========================
You can obtain the necessary files from a distribution package of Hadoop,
e.g. hadoop-0.20.2.tar.gz. Unpack this archive, and copy the content of
lib/native here, so that the layout looks like this:
<Nutch home>/lib/native/Linux-amd64-64/...
<Nutch home>/lib/native/Linux-i386-32/...
Local runtime
-------------
The build process will include these native libraries when preparing
the /runtime/local environment for running in local mode.
/runtime/local/bin/nutch knows how to use these libs - if they are
found and correctly used you should see lines like this in your logs:
Distributed runtime
-------------------
If you want to use this component in an existing Hadoop cluster (when using
/runtime/deploy artifacts) you need to make sure these files are placed in
Hadoop/lib/native directory on each node, and then restart the cluster. If
you installed the cluster from a distribution package of Hadoop then these
libraries should already be in the right place and you shouldn't need to do
anything else.
~
大体意思就是说 可以将 HADOOP_HOME下的 lib/native中的文件
Linux-amd64-64
Linux-i386-32
拷贝到NUTCH_HOME/lib/native下(按英文原意要确保you need to make sure these files are placed in
Hadoop/lib/native directory on each node, and then restart the cluster 确保这个文件在每一个节点上 并且重启集群,我拷贝了,)
9.执行
runtime/deply/bin/nutch hdfs://server0:9000/user/suse/urls -dir crawl -depth 200 -threads 200 -topN 1000
10.成功
看到map-reduce 任务成功执行 则配置成功
11/08/22 16:33:26 INFO mapred.JobClient: Reduce input records=48148
11/08/22 16:33:26 INFO crawl.CrawlDb: CrawlDb update: finished at 2011-08-22 16:33:26, elapsed: 00:00:39
11/08/22 16:33:26 INFO crawl.Generator: Generator: starting at 2011-08-22 16:33:26
11/08/22 16:33:26 INFO crawl.Generator: Generator: Selecting best-scoring urls due for fetch.
11/08/22 16:33:26 INFO crawl.Generator: Generator: filtering: true
11/08/22 16:33:26 INFO crawl.Generator: Generator: normalizing: true
11/08/22 16:33:26 INFO crawl.Generator: Generator: topN: 1000
11/08/22 16:33:27 INFO mapred.FileInputFormat: Total input paths to process : 8
11/08/22 16:33:28 INFO mapred.JobClient: Running job: job_201108221601_0022
11/08/22 16:33:29 INFO mapred.JobClient: map 0% reduce 0%
11/08/22 16:33:36 INFO mapred.JobClient: map 25% reduce 0%
11/08/22 16:33:39 INFO mapred.JobClient: map 50% reduce 0%
11/08/22 16:33:42 INFO mapred.JobClient: map 75% reduce 0%
11/08/22 16:33:45 INFO mapred.JobClient: map 75% reduce 4%
11/08/22 16:33:46 INFO mapred.JobClient: map 100% reduce 4%
11/08/22 16:33:48 INFO mapred.JobClient: map 100% reduce 13%
11/08/22 16:33:49 INFO mapred.JobClient: map 100% reduce 19%
11/08/22 16:33:51 INFO mapred.JobClient: map 100% reduce 33%
11/08/22 16:33:54 INFO mapred.JobClient: map 100% reduce 53%
11/08/22 16:33:57 INFO mapred.JobClient: map 100% reduce 71%
11/08/22 16:33:58 INFO mapred.JobClient: map 100% reduce 90%
11/08/22 16:34:00 INFO mapred.JobClient: map 100% reduce 100%
11/08/22 16:34:02 INFO mapred.JobClient: Job complete: job_201108221601_0022
11.solr 索引
nutch1.3下只有三个文件夹
crawldb
linkdb
segments
必须要用solr去建立索引
12.遇到错误
question：
Exception in thread "main" java.lang.IllegalArgumentException: Fetcher: No agents listed in 'http.agent.name' property.
at org.apache.nutch.fetcher.Fetcher.checkConfiguration(Fetcher.java:1166)
at org.apache.nutch.fetcher.Fetcher.fetch(Fetcher.java:1068)
at org.apache.nutch.crawl.Crawl.run(Crawl.java:135)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
at org.apache.nutch.crawl.Crawl.main(Crawl.java:54)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:616)
at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
answer:
using 1.3? If so make sure you changed nutch-site.xml (and not default)
in runtime/local/conf Changing the conf in NUTCH_HOME/conf won't be copied
to the runtime dirs unless you rebuild with ant.
BTW why don't you ask on the mailing list instead? You are more likely to get some help there
这个是英文元版回答。主要意思就是 nutch1.3 修改完nutch-site.xml 要重新ant一下
question：
Caused by: java.lang.NullPointerException
at java.io.Reader.<init>(Reader.java:61)
at java.io.BufferedReader.<init>(BufferedReader.java:76)
at java.io.BufferedReader.<init>(BufferedReader.java:91)
at org.apache.nutch.urlfilter.api.RegexURLFilterBase.readRules(RegexURLFilterBase.java:180)
at org.apache.nutch.urlfilter.api.RegexURLFilterBase.setConf(RegexURLFilterBase.java:156)
at org.apache.nutch.plugin.Extension.getExtensionInstance(Extension.java:162)
at org.apache.nutch.net.URLFilters.<init>(URLFilters.java:57)
at org.apache.nutch.crawl.Injector$InjectMapper.configure(Injector.java:72)
... 18 more
answer：
在runtime/local/下执行分布式抓取则报这个错误
注意：分布式抓取一定是在runtime/deply下执行
bin/nutch 命令
1
顶
顶
3
踩
踩
分享到：
评论
8 楼
schaha123
2012-02-24
urls是HDFS上url地址的文件夹
hdfs://202.193.58.99:8888/urls 这个应该是文件而不是文件夹,你直接用
bin/hadoop fs -ls 查看下你的urls文件在哪里 目录是什么 然后直接写上全路径或是相对路径 (根据实际情况)
完整路径：hdfs://202.193.58.99:8888/urls/url
错误还是一模一样。我好想在哪儿看过，这里用文件夹和集体的文件 都可以！
7 楼
黎明lm
2012-02-24
schaha123 写道
我的执行命令是
runtime/deploy/bin/nutch crawl hdfs://202.193.58.99:8888/urls -dir crawltest -depth 3 -topN 8
urls是HDFS上url地址的文件夹
runtime/deploy/bin/nutch crawl hdfs://202.193.58.99:8888/urls -dir crawltest -depth 3 -topN 8
urls是HDFS上url地址的文件夹
hdfs://202.193.58.99:8888/urls 这个应该是文件而不是文件夹,你直接用
bin/hadoop fs -ls 查看下你的urls文件在哪里 目录是什么 然后直接写上全路径或是相对路径 (根据实际情况)
6 楼
schaha123
2012-02-24
我的执行命令是
runtime/deploy/bin/nutch crawl hdfs://202.193.58.99:8888/urls -dir crawltest -depth 3 -topN 8
urls是HDFS上url地址的文件夹
runtime/deploy/bin/nutch crawl hdfs://202.193.58.99:8888/urls -dir crawltest -depth 3 -topN 8
urls是HDFS上url地址的文件夹
5 楼
schaha123
2012-02-24
黎明lm 写道
schaha123 写道
楼主，那个Nutch1.3是不是还有配置细节忘记写出来了啊，我的hadoop平台确定是正确的。按照配置还是运行部起来。出现错误：
Exception in thread "main" java.io.IOException: Job failed!
at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1252)
at org.apache.nutch.crawl.Injector.inject(Injector.java:217)
.............
Exception in thread "main" java.io.IOException: Job failed!
at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1252)
at org.apache.nutch.crawl.Injector.inject(Injector.java:217)
.............
可以多贴点错误日志？
Exception in thread "main" java.io.IOException: Job failed!
at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1252)
at org.apache.nutch.crawl.Injector.inject(Injector.java:217)
at org.apache.nutch.crawl.Crawl.run(Crawl.java:127)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
at org.apache.nutch.crawl.Crawl.main(Crawl.java:55)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
4 楼
黎明lm
2012-02-24
schaha123 写道
楼主，那个Nutch1.3是不是还有配置细节忘记写出来了啊，我的hadoop平台确定是正确的。按照配置还是运行部起来。出现错误：
Exception in thread "main" java.io.IOException: Job failed!
at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1252)
at org.apache.nutch.crawl.Injector.inject(Injector.java:217)
.............
Exception in thread "main" java.io.IOException: Job failed!
at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1252)
at org.apache.nutch.crawl.Injector.inject(Injector.java:217)
.............
可以多贴点错误日志？
3 楼
schaha123
2012-02-24
楼主，那个Nutch1.3是不是还有配置细节忘记写出来了啊，我的hadoop平台确定是正确的。按照配置还是运行部起来。出现错误：
Exception in thread "main" java.io.IOException: Job failed!
at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1252)
at org.apache.nutch.crawl.Injector.inject(Injector.java:217)
.............
Exception in thread "main" java.io.IOException: Job failed!
at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1252)
at org.apache.nutch.crawl.Injector.inject(Injector.java:217)
.............
2 楼
黎明lm
2011-12-05
爬虫也是分布式的，crawl方法里里面 的编程是采用mapreduce 的方式
1 楼
chenyuxxgl
2011-12-05
请问，爬行也是分布式的吗？每台nutch都会去公网上抓取页面，还是仅仅利用了hadoop的存储和计算。
发表评论
-
博客地址变更
2013-08-16 10:29 1247all the guys of visiting the bl ... -
hadoop 源码分析(六)hadoop taskTracker 生成map 和reduce任务流程
2013-04-09 17:36 2760taskTracker 生成map reduce ... -
hadoop 源码分析(六)hadoop taskTracker 生成map 和reduce任务流程
2013-04-09 17:33 0taskTracker 生成map reduce ... -
hadoop 源码分析(五)hadoop 任务调度TaskScheduler
2013-04-01 11:07 3957hadoop mapreduce 之所有能够实现job的运行 ... -
hadoop 源码分析(四)JobTracker 添加job 到schduler 队列中
2013-03-29 18:37 2891启动 JobTracker 1. 进入main方法： ... -
hadoop 源码分析(三) hadoop RPC 机制
2013-03-28 15:13 2422Hadoop 通信机制采用自己编写的RPC. 相比于 ... -
hadoop 源码分析(二) jobClient 通过RPC 代理提交作业到JobTracker
2013-03-27 12:57 38151.JobClient 客户端类 通过 ... -
hadoop 源码分析(一) jobClient 提交到JobTracker
2013-03-26 13:41 3630Hadoop 用了2年多了.从最初一起创业的 ... -
RHadoop 安装教程
2013-02-01 17:18 1643RHadoop 环境安装 硬件： centos6 ... -
pig
2012-11-16 19:28 1227转自：http://www.hadoopor.c ... -
hadoop与hive的映射
2012-11-15 10:21 2384hadoop与hive的映射 ... -
hadoop distcp
2012-07-31 10:00 2839hadoop distcp 使用:distcp ... -
MapReduce中Mapper类和Reducer类4函数解析
2012-07-20 18:05 2139MapReduce中Mapper类和Reducer类4函数解析 ... -
hadoop metrics 各参数解释
2012-07-17 18:59 1528hadoop metrics 各参数解释 研究使用hadoo ... -
Hbase几种数据入库（load）方式比较
2012-07-17 14:52 13931. 预先生成HFile入库 这个地址有详细的说明http:/ ... -
Hadoop客户端环境配置
2012-05-11 14:59 1787Hadoop客户端环境配置 1. 安装客户端（通过端用户可以 ... -
hadoop 通过distcp进行并行复制
2012-05-02 15:25 2493通过distcp进行并行复制 前面的HDFS访问模型都集中于 ... -
linux crontab 执行hadoop脚本 关于hadoop环境变量引入
2012-04-10 12:11 0crontab问题 crontab的特点：PATH不全和无终 ... -
hadoop fs 命令封装
2012-04-09 09:39 0hadoop fs 命令封装 #!/usr/bin/env ... -
map-reduce编程核心问题
2012-02-22 13:38 12891-How do we break up a large p ...
相关推荐
3. **Hadoop**：Nutch可以与Hadoop集成，用于分布式处理和存储数据。安装Hadoop 1.x或2.x版本，并配置好`HADOOP_HOME`环境变量。 **获取Nutch源代码** 1. 使用Git克隆Nutch 1.2的源代码库： ``` git clone ...
- **定义**: Hadoop是由Apache软件基金会维护的一个开源分布式计算平台，它以Hadoop分布式文件系统（HDFS）和MapReduce为核心，提供了一个简单易用的分布式计算框架。 - **特点**: 高容错性、高伸缩性、高效率以及...
- **数据接收方式**：采用数据网关代理模式，数据直接写入到Hadoop分布式文件系统 (HDFS) 中。 - **硬件配置**：4台接收机，每台支持10个进程用于数据接收和写入，数据的复制因子设定为3。 - **性能指标**：经过测试...
**1.3 Hadoop在大数据、云计算中的位置和关系** - **云计算的定义**：云计算是一种模型，它可以按需通过网络提供共享的计算资源和服务。它包含了IaaS、PaaS和SaaS三种服务模式。 - **Hadoop的角色**：Hadoop作为...
- **起源**：Hadoop最初是为了提升Apache Nutch（一个开源的Web搜索引擎项目）的可扩展性而诞生的。其发展受到两篇谷歌论文的影响：《The Google File System》和《MapReduce: Simplified Data Processing on Large ...
3.2 HDFS（Hadoop分布式文件系统） HDFS是一个高容错性的文件系统，设计目标是能在廉价硬件上运行。它将大文件分割成块，并在多台节点上冗余存储，确保数据的可靠性和可用性。 3.2.1 HDFS体系结构 HDFS有两个主要...
Nutch 使用 Hadoop 的分布式文件系统（HDFS）来存储大规模的数据集。 ##### 7.2 MapReduce - **概述**：Nutch 利用 MapReduce 实现了数据处理的分布式计算。 - **文件系统语法**：定义了用于操作文件系统的命令。 ...
#### 三、Hadoop分布式文件系统(HDFS) **3.1 HDFS的设计** - **设计原则**: HDFS的设计着重于提供高吞吐量的数据访问，适合那些需要存储和处理大量数据的应用场景。 **3.2 HDFS的概念** - **节点角色**: HDFS...
- **分布式部署**：采用分布式部署模式，提高系统的可扩展性和可用性。 - **多租户支持**：系统支持多租户管理，每个租户可以根据自身需求定制爬虫规则和数据处理方式。 - **自动化运维**：实现自动化监控、报警和...
- **2.2.4 Nutch网络搜索软件**：基于Hadoop的分布式爬虫工具。 - **2.2.5 用户界面**：提供友好的交互方式，方便用户使用。 ##### 2.3 商业搜索引擎技术介绍 - **2.3.1 通用搜索**：涵盖所有类型的信息搜索。 - *...
- Nutch是一个开源框架，基于Hadoop和Lucene构建。 - 用于大规模网络爬取和全文搜索。 - **2.2.5 用户界面** - 设计友好的用户界面，让用户可以方便地进行搜索操作。 - 支持多种展示形式，如列表、卡片式等。 ...