name |
value |
description |
dfs.namenode.logging.level |
info |
The logging level for dfs namenode. Other values are "dir"(trac e namespace mutations), "block"(trace block under/over replications and block creations/deletions), or "all". |
dfs.secondary.http.address |
0.0.0.0:50090 |
The secondary namenode http server address and port. If the port is 0 then the server will start on a free port. |
dfs.datanode.address |
0.0.0.0:50010 |
The address where the datanode server will listen to. If the port is 0 then the server will start on a free port. |
dfs.datanode.http.address |
0.0.0.0:50075 |
The datanode http server address and port. If the port is 0 then the server will start on a free port. |
dfs.datanode.ipc.address |
0.0.0.0:50020 |
The datanode ipc server address and port. If the port is 0 then the server will start on a free port. |
dfs.datanode.handler.count |
3 |
The number of server threads for the datanode. |
dfs.http.address |
0.0.0.0:50070 |
The address and the base port where the dfs namenode web ui will listen on. If the port is 0 then the server will start on a free port. |
dfs.https.enable |
false |
Decide if HTTPS(SSL) is supported on HDFS |
dfs.https.need.client.auth |
false |
Whether SSL client certificate authentication is required |
dfs.https.server.keystore.resource |
ssl-server.xml |
Resource file from which ssl server keystore information will be extracted |
dfs.https.client.keystore.resource |
ssl-client.xml |
Resource file from which ssl client keystore information will be extracted |
dfs.datanode.https.address |
0.0.0.0:50475 |
|
dfs.https.address |
0.0.0.0:50470 |
|
dfs.datanode.dns.interface |
default |
The name of the Network Interface from which a data node should report its IP address. |
dfs.datanode.dns.nameserver |
default |
The host name or IP address of the name server (DNS) which a DataNode should use to determine the host name used by the NameNode for communication and display purposes. |
dfs.replication.considerLoad |
true |
Decide if chooseTarget considers the target's load or not |
dfs.default.chunk.view.size |
32768 |
The number of bytes to view for a file on the browser. |
dfs.datanode.du.reserved |
0 |
Reserved space in bytes per volume. Always leave this much space free for non dfs use. |
dfs.name.dir |
${hadoop.tmp.dir}/dfs/name |
Determines where on the local filesystem the DFS name node should store the name table(fsimage). If this is a comma-delimited list of directories then the name table is replicated in all of the directories, for redundancy. |
dfs.name.edits.dir |
${dfs.name.dir} |
Determines where on the local filesystem the DFS name node should store the transaction (edits) file. If this is a comma-delimited list of directories then the transaction file is replicated in all of the directories, for redundancy. Default value is same as dfs.name.dir |
dfs.web.ugi |
webuser,webgroup |
The user account used by the web interface. Syntax: USERNAME,GROUP1,GROUP2, ... |
dfs.permissions |
true |
If "true", enable permission checking in HDFS. If "false", permission checking is turned off, but all other behavior is unchanged. Switching from one parameter value to the other does not change the mode, owner or group of files or directories. |
dfs.permissions.supergroup |
supergroup |
The name of the group of super-users. |
dfs.block.access.token.enable |
false |
If "true", access tokens are used as capabilities for accessing datanodes. If "false", no access tokens are checked on accessing datanodes. |
dfs.block.access.key.update.interval |
600 |
Interval in minutes at which namenode updates its access keys. |
dfs.block.access.token.lifetime |
600 |
The lifetime of access tokens in minutes. |
dfs.data.dir |
${hadoop.tmp.dir}/dfs/data |
Determines where on the local filesystem an DFS data node should store its blocks. If this is a comma-delimited list of directories, then data will be stored in all named directories, typically on different devices. Directories that do not exist are ignored. |
dfs.datanode.data.dir.perm |
755 |
Permissions for the directories on on the local filesystem where the DFS data node store its blocks. The permissions can either be octal or symbolic. |
dfs.replication |
3 |
Default block replication. The actual number of replications can be specified when the file is created. The default is used if replication is not specified in create time. |
dfs.replication.max |
512 |
Maximal block replication. |
dfs.replication.min |
1 |
Minimal block replication. |
dfs.block.size |
67108864 |
The default block size for new files. |
dfs.df.interval |
60000 |
Disk usage statistics refresh interval in msec. |
dfs.client.block.write.retries |
3 |
The number of retries for writing blocks to the data nodes, before we signal failure to the application. |
dfs.blockreport.intervalMsec |
3600000 |
Determines block reporting interval in milliseconds. |
dfs.blockreport.initialDelay |
0 |
Delay for first block report in seconds. |
dfs.heartbeat.interval |
3 |
Determines datanode heartbeat interval in seconds. |
dfs.namenode.handler.count |
10 |
The number of server threads for the namenode. |
dfs.safemode.threshold.pct |
0.999f |
Specifies the percentage of blocks that should satisfy the minimal replication requirement defined by dfs.replication.min. Values less than or equal to 0 mean not to start in safe mode. Values greater than 1 will make safe mode permanent. |
dfs.safemode.extension |
30000 |
Determines extension of safe mode in milliseconds after the threshold level is reached. |
dfs.balance.bandwidthPerSec |
1048576 |
Specifies the maximum amount of bandwidth that each datanode can utilize for the balancing purpose in term of the number of bytes per second. |
dfs.hosts |
|
Names a file that contains a list of hosts that are permitted to connect to the namenode. The full pathname of the file must be specified. If the value is empty, all hosts are permitted. |
dfs.hosts.exclude |
|
Names a file that contains a list of hosts that are not permitted to connect to the namenode. The full pathname of the file must be specified. If the value is empty, no hosts are excluded. |
dfs.max.objects |
0 |
The maximum number of files, directories and blocks dfs supports. A value of zero indicates no limit to the number of objects that dfs supports. |
dfs.namenode.decommission.interval |
30 |
Namenode periodicity in seconds to check if decommission is complete. |
dfs.namenode.decommission.nodes.per.interval |
5 |
The number of nodes namenode checks if decommission is complete in each dfs.namenode.decommission.interval. |
dfs.replication.interval |
3 |
The periodicity in seconds with which the namenode computes repliaction work for datanodes. |
dfs.access.time.precision |
3600000 |
The access time for HDFS file is precise upto this value. The default value is 1 hour. Setting a value of 0 disables access times for HDFS. |
dfs.support.append |
false |
Does HDFS allow appends to files? This is currently set to false because there are bugs in the "append code" and is not supported in any prodction cluster. |
dfs.namenode.delegation.key.update-interval |
86400000 |
The update interval for master key for delegation tokens in the namenode in milliseconds. |
dfs.namenode.delegation.token.max-lifetime |
604800000 |
The maximum lifetime in milliseconds for which a delegation token is valid. |
dfs.namenode.delegation.token.renew-interval |
86400000 |
The renewal interval for delegation token in milliseconds. |
dfs.datanode.failed.volumes.tolerated |
0 |
The number of volumes that are allowed to fail before a datanode stops offering service. By default any volume failure will cause a datanode to shutdown. |
相关推荐
Hadoop 2.9.0版本的HDFS配置文件hdfs-site.xml定义了分布式文件系统的主要配置参数,下面详细说明这些属性的关键知识点。 1. hadoop.hdfs.configuration.version 这是一个配置属性,用于记录当前使用的Hadoop HDFS...
本篇文章将深入探讨Hadoop的默认配置文件,包括`core-default.xml`、`hdfs-default.xml`、`mapred-default.xml`和`yarn-default.xml`,以及这些文件中的关键配置选项。 首先,`core-default.xml`是Hadoop的核心配置...
有时候需要查看修改或者回复hdfs的默认配置,在这个jar包里面,可以把hadoop-default.xml拿出来
export CLASSPATH="$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar:${HADOOP_HOME}/lib/commons-logging-1.0.4.jar:${HADOOP_HOME}/lib/guava-r09-jarjar.jar:${HADOOP_HOME}/hadoop-core-0.20.2-cdh3u3.jar:/usr/...
Hadoop 2.9.0版本中的mapred-default.xml文件包含了MapReduce作业的配置属性,这些属性定义了MapReduce作业执行过程中的各种行为和参数。下面我们来详细介绍mapred-site.xml文件中的一些关键属性。 1. mapreduce....
在Hadoop的配置目录($HADOOP_HOME/etc/hadoop)下,有两个主要的配置文件:core-site.xml和hdfs-site.xml。核心配置core-site.xml主要设置临时目录和HDFS的访问方式,例如: ```xml <name>fs.defaultFS ...
2. 修改`etc/hadoop/hdfs-site.xml`,配置HDFS的副本数量和其他参数: ```xml <name>dfs.replication <value>1 ``` 3. 修改`etc/hadoop/yarn-site.xml`,配置YARN的相关参数: ```xml <!-- ... --> ...
Hadoop的配置主要通过修改conf目录下的两个核心文件:hadoop-env.cmd和core-site.xml。在hadoop-env.cmd中,设置HADOOP_OPTS以包含Java内存参数,例如"-Xmx1024m",根据你的系统资源进行适当调整。在core-site.xml中...
在HBase中,`hbase-default.xml`是一个非常重要的配置文件,它包含了HBase运行时的默认配置参数。这个文件为系统提供了基础设置,比如RegionServer的内存大小、ZooKeeper的地址等,它是所有HBase实例的基础配置模板...
4. `hdfs-default.xml`、`mapred-default.xml`、`yarn-default.xml`、`core-default.xml`:这些都是 Hadoop 的默认配置文件,定义了 HDFS(Hadoop 分布式文件系统)、MapReduce(Hadoop 的并行计算模型)和 YARN 的...
在搭建Hadoop高可用(High Availability, HA)集群时,`core-site.xml`是其中一个至关重要的配置文件。它主要用于定义Hadoop集群的基本属性,如默认的文件系统路径、临时目录以及与ZooKeeper集成的相关设置等。本文...
对于Windows用户,可能还需要配置Hadoop与本地文件系统的交互,例如通过修改`hdfs-site.xml`中的`fs.defaultFS`属性,使其指向本地的Hadoop分布式文件系统(HDFS)模拟器,如Apache Hadoop的附加工具——WinUtils。...
3. **配置Hadoop**:在Hadoop的conf目录下,编辑`core-site.xml`,设置HDFS的默认FS(如`fs.defaultFS`为`hdfs://localhost:9000`),以及临时目录(如`hadoop.tmp.dir`为`C:\Hadoop\tmp`)。然后编辑`hdfs-site.xml...
1. **hdfs-default.xml**:这是Hadoop分布式文件系统的默认配置文件。它包含了HDFS的各种配置参数,如命名节点(NameNode)和数据节点(DataNode)的设置、副本数量、块大小、安全模式、文件权限等。例如,`dfs....
`conf`目录则存放配置文件,如`core-site.xml`, `hdfs-site.xml`, `mapred-site.xml`和`yarn-site.xml`。 配置Hadoop集群时,需要在这些配置文件中指定诸如NameNode、DataNode、ResourceManager、NodeManager等节点...
这些配置包括了Java环境的安装、SSH免密码登录的配置、Hadoop和HBase的核心文件配置以及防火墙规则的设置等关键环节。这些步骤确保了Hadoop和HBase可以在单机或集群环境下稳定运行,为后续的大数据分析提供了坚实的...
1. **修改配置文件**: 打开"Hadoop-2.7.5-master"目录下的`conf`文件夹,主要需要编辑两个核心配置文件:`core-site.xml`和`hdfs-site.xml`。 2. **core-site.xml**: 这个文件主要设置Hadoop的基本属性,例如数据...
2. 在配置文件夹中创建两个XML文件:`core-site.xml`和`hdfs-site.xml`。 3. `core-site.xml`配置: ```xml <name>fs.defaultFS <value>hdfs://localhost:9000 ``` 4. `hdfs-site.xml`配置: ```xml ...
3. **配置core-site.xml**:这是Hadoop的核心配置文件,主要设定HDFS的数据存储方式。在本地模式下,你可以将`fs.defaultFS`设置为`file:///`,表示使用本地文件系统。 4. **配置hdfs-site.xml**:用于设定HDFS的...
HBase 的配置文件 hbase-site.xml 是其核心配置文件,本文将详细解释 hbase-site.xml 的配置参数 DEFAULT 值和描述。 一、HBase 临时目录配置 * hbase.tmp.dir:${java.io.tmpdir}/hbase-${user.name},这个参数...