Netcdf （二）

附件文档：

4 NetCDF Java
4.1 概述（Overview）
参考网址：http://www.unidata.ucar.edu/software/netcdf-java/documentation.htm
The NetCDF-Java library implements a Common Data Model (CDM), a generalization of the NetCDF, OpenDAP and HDF5 data models. The library is a prototype for the NetCDF-4 project, which provides a C language API for the "data access layer" of the CDM, on top of the HDF5 file format. The NetCDF-Java library is a 100% Java framework for reading netCDF and other file formats into the CDM, as well as writing to the netCDF-3 file format. The NetCDF-Java library also implements NcML, which allows you to add metadata to CDM datasets, as well as to create virtual datasets through aggregation.

NetCDF-Java库实现了CDM(通用数据模型)，CDM包括NetCDF,OpenDAP,HDF5数据模型，它是NetCDF-4项目的一个原型，NetCDF-4项目是紧跟HDF5文件格式后采用C语言作为CDM的的数据访问层API。在CDM中NetCDF-Java包是完全用Java架构来读取NetCDF和其他格式文件，和写netCDF-3格式文件一样。它还实现了NCML，允许你为CDM数据集添加元数据，和通过运算生成实际数据一样。
4.2 CDM（通用数据模型）
参考网址：http://www.unidata.ucar.edu/software/netcdf-java/CDM/index.html
Unidata’s Common Data Model (CDM) is an abstract data model for scientific datasets. It merges the netCDF, OPeNDAP, and HDF5 data models to create a common API for many types of scientific data. The NetCDF Java library is an implementation of the CDM which can read many file formats besides netCDF. We call these CDM files, a shorthand for files that can be read by the NetCDF Java library and accessed through the CDM data model. 

Unidata社区的CDM(通用数据模型)是一个科学数据的抽象数据模型，它包括了NetCDF, OPeNDAP, HDF5数据模型并为不同类型的科学数据创建了一个通用的API。NetCDF Java库实现了CDM，CDM除了能读取NetCDF格式外还有其他类型文件，我们把这些CDM文件作为那些能被NetCDF Java库读取和访问的的文件的简称。

4.3 NetCDF-Java/CDM  Architecture
 
4.4 CDM-FILES
General: NetCDF, OPeNDAP, HDF5, NetCDF4, HDF4, HDF-EOS
Gridded: GRIB-1, GRIB-2, GEMPAK
Radar: NEXRAD 2&3, DORADE, CINRAD, Universal Format, TDWR
Point: BUFR, ASCII
Satellite: DMSP, GINI, McIDAS AREA
Misc: GTOPO, Lightning, etc
Others in development (partial): 
AVHRR, GPCP, GACP, SRB, SSMI, HIRS (NCDC)
4.5 Data Access Layer Object Model
 
4.6 NetCDF举例
4.6.1 下载
下载地址: http://www.unidata.ucar.edu/software/netcdf-java/documentation.htm
在线API: http://www.unidata.ucar.edu/software/netcdf-java/v4.2/javadoc/index.html
目前最新版本为4.2.20
最新版本需要JDK6
4.6.2  生成NC文件
package my.demo;

import java.io.IOException;
import java.util.ArrayList;

import ucar.ma2.Array;
import ucar.ma2.DataType;
import ucar.nc2.Dimension;
import ucar.nc2.NetcdfFileWriteable;

public class CreateNetcdf {
	@SuppressWarnings("unchecked")
	public static void main(String[] args) throws Exception {
		String filename = "testWrite.nc";
		NetcdfFileWriteable ncfile = NetcdfFileWriteable.createNew(filename,true); // add
		// dimensions
		Dimension latDim = ncfile.addDimension("lat", 3);
		Dimension lonDim = ncfile.addDimension("lon", 3); // define
		// Variable
		ArrayList dims = new ArrayList();
		dims.add(latDim);
		dims.add(lonDim);
		ncfile.addVariable("temperature", DataType.DOUBLE, dims);
		ncfile.addVariableAttribute("temperature", "units", "K"); // add a
		// 1D
		// attribute
		// of
		// length
		// 3
		Array data = Array.factory(int.class, new int[] { 3 }, new int[] { 1,2,3 });
		ncfile.addVariableAttribute("temperature", "scale", data);
		// add a string-valued variable: char svar(80)
		Dimension svar_len = ncfile.addDimension("svar_len", 80);
		dims = new ArrayList();
		dims.add(svar_len);
		ncfile.addVariable("svar", DataType.CHAR, dims);
		// string array: char names(3, 80)
		Dimension names = ncfile.addDimension("names", 3);
		ArrayList dima = new ArrayList();
		dima.add(names);
		dima.add(svar_len);
		ncfile.addVariable("names", DataType.CHAR, dima);
		// how about a scalar variable?
		ncfile.addVariable("scalar", DataType.DOUBLE, new ArrayList()); // add
		// global
		// attributes
		ncfile.addGlobalAttribute("yo", "face");
		ncfile.addGlobalAttribute("versionD", new Double(1.2));
		ncfile.addGlobalAttribute("versionF", new Float(1.2));
		ncfile.addGlobalAttribute("versionI", new Integer(1));
		ncfile.addGlobalAttribute("versionS", new Short((short) 2));
		ncfile.addGlobalAttribute("versionB", new Byte((byte) 3)); // create
		// the
		// file
		try {
			ncfile.create();
		} catch (IOException e) {
			System.err.println("ERROR creating file " + ncfile.getLocation()+ "\n" + e);
		}
	}
}
会生成一个testWrite.nc文件，该文件不能直接打开，可以通过下载的包中netcdfUI-4.2.jar打开：
 
4.6.3 读取NC文件
package my.demo;

import java.io.IOException;
import java.util.List;

import ucar.nc2.Dimension;
import ucar.nc2.NetcdfFile;
import ucar.nc2.Variable;

public class ReadNetcdf {
	public static void main(String[] args) {
		String filename = "D:\\work\\netcdf\\testWrite.nc";
		NetcdfFile ncfile = null;
		try {
			ncfile = NetcdfFile.open(filename);
			
			//read dimensions
			List<Dimension> list =  ncfile.getDimensions();
			for(Dimension d : list){
				System.out.println("name="+d.getName()+" length="+d.getLength());
			}
			//read variables
			List<Variable> variables = ncfile.getVariables();
			System.out.println();
			for(Variable v : variables){
				System.out.println("name="+v.getName()+" NameAndDimension="+v.getNameAndDimensions()+" ElementSize="+v.getElementSize());
			}
			
		} catch (IOException ioe) {
		} finally {
			if (null != ncfile)
				try {
					ncfile.close();
				} catch (IOException ioe) {
				}
		}
	}
}
运行打印如下：
name=lat length=3
name=lon length=3
name=svar_len length=80
name=names length=3

name=temperature NameAndDimension=temperature(lat=3, lon=3) ElementSize=8
name=svar NameAndDimension=svar(svar_len=80) ElementSize=1
name=names NameAndDimension=names(names=3, svar_len=80) ElementSize=1
name=scalar NameAndDimension=scalar ElementSize=8
4.6.4 读写文件
package my.demo;

import java.io.IOException;

import ucar.ma2.ArrayDouble;
import ucar.ma2.Index;
import ucar.ma2.InvalidRangeException;
import ucar.nc2.Dimension;
import ucar.nc2.NetcdfFileWriteable;

public class WriteDataToNetcdf {

	/**
	 * @param args
	 * @throws IOException
	 */
	public static void main(String[] args) throws IOException {
		NetcdfFileWriteable ncfile = NetcdfFileWriteable.openExisting("D:\\work\\netcdf\\testWrite.nc", true);
		Dimension latDim = ncfile.getDimensions().get(0);
		Dimension lonDim = ncfile.getDimensions().get(1);
		ArrayDouble A = new ArrayDouble.D2(latDim.getLength(), lonDim.getLength());
		int i, j;
		Index ima = A.getIndex();
		for (i = 0; i < latDim.getLength(); i++) {
			for (j = 0; j < lonDim.getLength(); j++) {
				A.setDouble(ima.set(i, j), (double) (2));
			}
		}
		int[] origin = new int[2];
		try {
			ncfile.write("temperature", origin, A);
			ncfile.close();
		} catch (IOException e) {
			System.err.println("ERROR writing file");
		} catch (InvalidRangeException e) {
			e.printStackTrace();
		}
	}
}
该方法为Variable temperature进行赋值，可以将修改后的testWrite.nc在netcdfUI-4.2.jar中查看：
   double temperature(lat=3, lon=3);
     :units = "K";
     :scale = 1, 2, 3; // int
 data:
  {
    {2.0, 2.0, 2.0},
    {2.0, 2.0, 2.0},
    {2.0, 2.0, 2.0}
  }
4.6.5 读取二维数据
通过v.read()可以读取数据：
 Variable v = ncfile.findVariable(varName);
Variable v = ncfile.findVariable(varName);
 Array data = v.read("0:2:1, 0:19:1");
package my.demo;

import java.io.IOException;

import ucar.ma2.Array;
import ucar.nc2.NCdumpW;
import ucar.nc2.NetcdfFile;
import ucar.nc2.Variable;

public class ReadData {
	public static void main(String[] args) {
		String filename = "D:\\work\\netcdf\\testWrite.nc";
		NetcdfFile ncfile = null;
		try {
			ncfile = NetcdfFile.open(filename);
			//find variable
			String variable = "temperature";
			Variable varBean = ncfile.findVariable(variable);
			//Reading data from a Variable 
			if(null != varBean) {
				Array all = varBean.read();
				Array data = varBean.read("0:2:1, 0:2:1");
				Array data1 = varBean.read("0:2:2, 0:2:2");
				System.out.println("读取所有：\n"+NCdumpW.printArray(all, variable, null));
				System.out.println("x轴从0到2 跨度为1 y轴从0到2 跨度为1：\n"+NCdumpW.printArray(data, variable, null));
				System.out.println("x轴从0到2 跨度为2 y轴从0到2 跨度为2：\n"+NCdumpW.printArray(data1, variable, null));
			}
			
			if(null != varBean) {
				int[] origin = new int[] { 0 , 0};
				int[] size = new int[] { 3,3};
				Array data2D = varBean.read(origin, size);
				System.out.println("读取所有：\n"+NCdumpW.printArray(data2D, variable, null));
			}
			
			if(null != varBean) {
				int[] origin = new int[] { 1 , 1};
				int[] size = new int[] { 2,1};
				Array data2D = varBean.read(origin, size);
				System.out.println("读取从第二行第二列开始为起点x数量为1，y数量为2：\n"+NCdumpW.printArray(data2D, variable, null));
			}
			System.out.println("由此可得结论：维上的起点都以数组0开始，且阵列顺序在坐标中是从右至左\n如：int[] size = new int[] { 2,1}，1代表x轴，2代表的是y轴....");
		} catch (Exception ioe) {
			ioe.printStackTrace();
		} finally {
			if (null != ncfile)
				try {
					ncfile.close();
				} catch (IOException ioe) {
				}
		}
	}
}

打印结果如下：根据结果可以知道read("0:2:2, 0:2:2")和read(origin, size)的差别
读取所有：
temperature =
  {
    {0.0, 1.0, 2.0},
    {1.0, 2.0, 3.0},
    {2.0, 3.0, 4.0}
  }

x轴从0到2 跨度为1 y轴从0到2 跨度为1：
temperature =
  {
    {0.0, 1.0, 2.0},
    {1.0, 2.0, 3.0},
    {2.0, 3.0, 4.0}
  }

x轴从0到2 跨度为2 y轴从0到2 跨度为2：
temperature =
  {
    {0.0, 2.0},
    {2.0, 4.0}
  }

读取所有：
temperature =
  {
    {0.0, 1.0, 2.0},
    {1.0, 2.0, 3.0},
    {2.0, 3.0, 4.0}
  }

读取从第二行第二列开始为起点x数量为1，y数量为2：
temperature =
  {
    {2.0},
    {3.0}
  }

由此可得结论：维上的起点都以数组0开始，且阵列顺序在坐标中是从右至左
如：int[] size = new int[] { 2,1}，1代表x轴，2代表的是y轴....
4.6.6 多维NetCDF(三维)
4.6.6.1 创建
创建三维NetCDF文件：
package my.demo;
import java.io.IOException;
import java.util.ArrayList;
import ucar.ma2.Array;
import ucar.ma2.DataType;
import ucar.nc2.Dimension;
import ucar.nc2.NetcdfFileWriteable;
public class Create3DNetCDF {
	@SuppressWarnings("unchecked")
	public static void main(String[] args) throws Exception {
		String filename = "test3D.nc";
		NetcdfFileWriteable ncfile = NetcdfFileWriteable.createNew(filename,true); // add
		Dimension timeDim = ncfile.addDimension("time",2);
		Dimension latDim = ncfile.addDimension("lat", 3);
		Dimension lonDim = ncfile.addDimension("lon", 3); // define
		ArrayList dims = new ArrayList();
		dims.add(timeDim);
		dims.add(latDim);
		dims.add(lonDim);
		ncfile.addVariable("temperature", DataType.DOUBLE, dims);
		ncfile.addVariableAttribute("temperature", "units", "K"); // add a
		Array data = Array.factory(int.class, new int[] { 3 }, new int[] { 1,2,3 });
		ncfile.addVariableAttribute("temperature", "scale", data);
		try {
			ncfile.create();
		} catch (IOException e) {
			System.err.println("ERROR creating file " + ncfile.getLocation()+ "\n" + e);
		}
	}
}
4.6.6.2 写数据
package my.demo;
import java.io.IOException;
import ucar.ma2.ArrayDouble;
import ucar.ma2.Index;
import ucar.ma2.InvalidRangeException;
import ucar.nc2.Dimension;
import ucar.nc2.NetcdfFileWriteable;
public class Write3DNetCDF {
	public static void main(String[] args) throws IOException {
		NetcdfFileWriteable ncfile = NetcdfFileWriteable.openExisting("D:\\work\\netcdf\\test3D.nc", true);
		Dimension timeDim = ncfile.getDimensions().get(0);
		Dimension latDim = ncfile.getDimensions().get(1);
		Dimension lonDim = ncfile.getDimensions().get(2);
		ArrayDouble A = new ArrayDouble.D3(timeDim.getLength(),latDim.getLength(), lonDim.getLength());
		int k,i, j;
		Index ima = A.getIndex();
		for(k = 0; k < timeDim.getLength(); k++){
			for (i = 0; i < latDim.getLength(); i++) {
				for (j = 0; j < lonDim.getLength(); j++) {
					A.setDouble(ima.set(k,i,j), (double) (k+i+j));
				}
			}
		}
		int[] origin = new int[3];
		try {
			ncfile.write("temperature", origin, A);
			ncfile.close();
		} catch (IOException e) {
			System.err.println("ERROR writing file");
		} catch (InvalidRangeException e) {
			e.printStackTrace();
		}
	}
}

对应的CDL格式如下：
   double temperature(time=2, lat=3, lon=3);
     :units = "K";
     :scale = 1, 2, 3; // int
 data:
  {
    {
      {0.0, 1.0, 2.0},
      {1.0, 2.0, 3.0},
      {2.0, 3.0, 4.0}
    },
    {
      {1.0, 2.0, 3.0},
      {2.0, 3.0, 4.0},
      {3.0, 4.0, 5.0}
    }
  }
4.6.6.3 读数据
package my.demo;
import java.io.IOException;
import ucar.ma2.Array;
import ucar.nc2.NCdumpW;
import ucar.nc2.NetcdfFile;
import ucar.nc2.Variable;
public class Read3DNetCDF {
	public static void main(String[] args) {
		String filename = "D:\\work\\netcdf\\test3D.nc";
		NetcdfFile ncfile = null;
		try {
			ncfile = NetcdfFile.open(filename);
			String variable = "temperature";
			Variable varBean = ncfile.findVariable(variable);
			//read all data
			if(null != varBean) {
				Array all = varBean.read();
				System.out.println("读取所有：\n"+NCdumpW.printArray(all, variable, null));
			}
			if(null != varBean) {
				int[] origin = new int[] { 0,1,1};
				int[] size = new int[] { 2,2,2};
				Array data2D = varBean.read(origin, size);
				System.out.println("读取从第一维的0开始,第二维从1开始,第三维从1开始,数量分别为2,2,2：\n"+NCdumpW.printArray(data2D, variable, null));
			}			
			// invoke reduce trans 3D to 2D
			if(null != varBean) {
				int[] origin = new int[] { 0,1,1};
				int[] size = new int[] { 1,2,2};
				Array data2D = varBean.read(origin, size).reduce().reduce();
				System.out.println("读取从第一维的0开始,第二维从1开始,第三维从1开始,数量分别为1,2,2并转为二维：\n"+NCdumpW.printArray(data2D, variable, null));
			}
		} catch (Exception ioe) {
			ioe.printStackTrace();
		} finally {
			if (null != ncfile)
				try {
					ncfile.close();
				} catch (IOException ioe) {
				}
		}
	}
}
打印：
读取所有：
temperature =
  {
    {
      {0.0, 1.0, 2.0},
      {1.0, 2.0, 3.0},
      {2.0, 3.0, 4.0}
    },
    {
      {1.0, 2.0, 3.0},
      {2.0, 3.0, 4.0},
      {3.0, 4.0, 5.0}
    }
  }
读取从第一维的0开始,第二维从1开始,第三维从1开始,数量分别为2,2,2：
temperature =
  {
    {
      {2.0, 3.0},
      {3.0, 4.0}
    },
    {
      {3.0, 4.0},
      {4.0, 5.0}
    }
  }

读取从第一维的0开始,第二维从1开始,第三维从1开始,数量分别为1,2,2并转为二维：
temperature =
  {
    {2.0, 3.0},
    {3.0, 4.0}
  }
4.7 NetCDF-NCML(Modifying existing files)
通过NCML标记语言可以对NetCDF文件修改
参考网址：http://www.unidata.ucar.edu/software/netcdf/ncml/v2.2/Tutorial.html
4.8 NCML- Aggregation
通过NCML合并存在的多个NetCDF文件
参考网站：http://www.unidata.ucar.edu/software/netcdf/ncml/v2.2/Aggregation.html
4.9 NetCDF-IOSP(I/O Service Provide)
参考网址：http://www.unidata.ucar.edu/software/netcdf-java/tutorial/IOSPoverview.html
4.9.1 Overview
A client uses the NetcdfFile, NetcdfDataset, or one of the Scientific Feature Type APIs to read data from a CDM file. These provide a rich and sometimes complicated API to the client. Behind the scenes, when any of these APIs actually read from a dataset, however, they use a very much simpler interface, the I/O Service Provider or IOSP for short. The Netcdf Java library has many implementations of this interface, one for each different file format that it knows how to read. This design pattern is called a Service Provider.

IOSPs are managed by the NetcdfFile class. When a client requests a dataset (by calling NetcdfFile.open), the file is opened as a ucar.unidata.io.RandomAccessFile (an improved version of java.io.RandomAccessFile). Each registered IOSP is then asked "is this your file?" by calling isValidFile( ucar.unidata.io.RandomAccessFile). The first one that returns true claims it. When you implement isValidFile() in your IOSP, it must be very fast and accurate.
4.9.2 IOServiceProvider
package ucar.nc2.iosp;
import ucar.ma2.Section;
import ucar.ma2.InvalidRangeException;
import ucar.ma2.StructureDataIterator;
import ucar.nc2.ParsedSectionSpec;
import ucar.nc2.Structure;
import java.io.IOException;
import java.nio.channels.WritableByteChannel;
/**
 * This is the service provider interface for the low-level I/O access classes (read only).
 * This is only used by service implementors.
 *
 * The NetcdfFile class manages all registered IOServiceProvider classes.
 * When NetcdfFile.open() is called:
 * <ol>
 * <li> the file is opened as a ucar.unidata.io.RandomAccessFile;</li>
 * <li> the file is handed to the isValidFile() method of each registered
 * IOServiceProvider class (until one returns true, which means it can read the file).</li>
 * <li> the open() method on the resulting IOServiceProvider class is handed the file.</li>
 *
 * @see ucar.nc2.NetcdfFile#registerIOProvider(Class) ;
 *
 * @author caron
 */
public interface IOServiceProvider {

   /**
    * Check if this is a valid file for this IOServiceProvider.
    * You must make this method thread safe, ie dont keep any state.
    * 
    * @param raf RandomAccessFile
    * @return true if valid.
    * @throws java.io.IOException if read error
    */
  public boolean isValidFile( ucar.unidata.io.RandomAccessFile raf) throws IOException;
}
其他方法见官网介绍或API文档。
4.9.3 AbstractIOServiceProvider
Your implementataion class should extend ucar.nc2.iosp.AbstractIOServiceProvider. This provides default implementation of some of the methods, so minimally, you only have to implement 4 methods:
public class MyIosp extends ucar.nc2.iosp.AbstractIOServiceProvider {
 1)  public boolean isValidFile(RandomAccessFile raf) throws IOException {}
 2)  public void open(RandomAccessFile raf, NetcdfFile ncfile, CancelTask cancelTask) throws IOException {}
 3)  public Array readData(Variable v2, Section wantSection) throws IOException, InvalidRangeException {}
 4)  public void close() throws IOException {}

 5)  public String getFileTypeId() {}
 5)  public String getFileTypeVersion() {}
 5)  public String getFileTypeDescription();
}
4.9.4 IOSP-Example
通过IOSP对数据处理生成NetCDF文件已经读取NetCDF数据例子：
参考网址：
http://www.unidata.ucar.edu/software/netcdf-java/tutorial/index.html
 
图中的例子为雷电数据，卫星数据，雷达数据相关

评论

2 楼 lilixu 2017-02-23  

  

1 楼 chemingliang 2012-01-05  

不错！