Hadoop学习七：Hadoop-Hdfs源码 conf包

一.conf包下四个类

 

二.详细描述

1.  Configurable接口：Something that may be configured with a {@link Configuration}.，很绕口，简单理解为Hdfs系统配置文件的接口。
2. Configured：Base class for things that may be configured with a {@link Configuration}，Hdfs系统配置文件的抽象类。持有Configuration对象的应用。
3. Configured：配置文件资源管理类。用DOM解析xml配置文件；默认加载资源core-default.xml，core-site.xml；只有在get(key)时才去加载资源文件。
4. ConfServlet：A servlet to print out the running configuration data。

     @Override
     public void doGet(HttpServletRequest request, HttpServletResponse response)
         throws ServletException, IOException {
       String format = request.getParameter("format");
       if (null == format) {
         format = "xml";
       }
   
       if ("xml".equals(format)) {
         response.setContentType("text/xml; charset=utf-8");
       } else if ("json".equals(format)) {
         response.setContentType("application/json; charset=utf-8");
       }
   
       Writer out = response.getWriter();
       try {
         out.write(getServletContext().getAttribute("hadoop.conf");)
       } catch (BadFormatException bfe) {
         response.sendError(HttpServletResponse.SC_BAD_REQUEST, bfe.getMessage());
       }
       out.close();
     }

 

三.Configuration代码 

//1.静态代码块，把默认资源放到CopyOnWriteArrayList<String> defaultResources里。
 static{
    //print deprecation warning if hadoop-site.xml is found in classpath
    ClassLoader cL = Thread.currentThread().getContextClassLoader();
    if (cL == null) {
      cL = Configuration.class.getClassLoader();
    }
    if(cL.getResource("hadoop-site.xml")!=null) {
      LOG.warn("DEPRECATED: hadoop-site.xml found in the classpath. " +
          "Usage of hadoop-site.xml is deprecated. Instead use core-site.xml, "
          + "mapred-site.xml and hdfs-site.xml to override properties of " +
          "core-default.xml, mapred-default.xml and hdfs-default.xml " +
          "respectively");
    }
    addDefaultResource("core-default.xml");
    addDefaultResource("core-site.xml");
  }

public static synchronized void addDefaultResource(String name) {
    if(!defaultResources.contains(name)) {
      defaultResources.add(name);
      for(Configuration conf : REGISTRY.keySet()) {
        if(conf.loadDefaults) {
          conf.reloadConfiguration();
        }
      }
    }
  }

//2.初始化代码块，把该实例放入WeakHashMap<Configuration,Object>里。
public Configuration(boolean loadDefaults) {
    this.loadDefaults = loadDefaults;
    updatingResource = new HashMap<String, String>();
    synchronized(Configuration.class) {
      REGISTRY.put(this, null);
    }
  }

//3.第一次get(key)时，才开始解析资源，保存到Properties properties里。
public String get(String name) {
    return substituteVars(getProps().getProperty(name));
  }

private synchronized Properties getProps() {
    if (properties == null) {
      properties = new Properties(); //这个时候才初始化properties
      loadResources(properties, resources, quietmode);
      if (overlay!= null) {
        properties.putAll(overlay);
        for (Map.Entry<Object,Object> item: overlay.entrySet()) {
          updatingResource.put((String) item.getKey(), UNKNOWN_RESOURCE);
        }
      }
    }
    return properties;
  }

private void loadResource(Properties properties, Object name, boolean quiet) {
  //DOM解析资源，保存到properties里。
  //需要注意的时，解析的是代码运行环境下的core-default.xml，core-site.xml。
}

 

四.Configuration使用例子 

 

Configuration conf = new Configuration();
FileSystem fs = FileSystem.get(URI, conf);

1.  第一段代码：按照三里的12步先静态代码块，再初始化。
2. 第二段代码，创建FileSystem时，会调用conf.getBoolean(key)方法获取所需参数，此时第一次加载资源，后续还会调用conf.get(key)，直接从properties取就行了。
3. 若上述两段代码在eclipse运行，必须在eclipse classpath下配置core-default.xml，core-site.xml。

评论

1 楼 chen030112 2014-04-13  

很喜欢博主的文章，刚刚用豆约翰博客备份专家备份了您的全部博文。