`
bnmnba
  • 浏览: 293556 次
  • 性别: Icon_minigender_1
社区版块
存档分类
最新评论

在Hbase表上运行mapreduce，Maven打包

 
阅读更多

 

运行了一个《HBase权威指南》的示例ch07。

代码：https://github.com/larsgeorge/hbase-book

 

不需要更改hadoop和hbase以及zookeeper的任何配置。只要它们正常运行即可。我用的是CDH5.8.0版本。

大概方法如下：

 

1.先命令行建表

bin/hbase shell
create 't2','c2'
put 't2','row1','c2:col1','value1'

 

2.用maven插件shade打包并拷贝jar到服务器。

具体方法是首先建立maven项目，拷贝代码，增加必要依赖，增加打包插件，编译。

 

3.配置好服务的环境变量HADOOP_HOME、HBASE_HOME、ZK_HOME。

例如命令，一行一行执行：

echo 'export HADOOP_PREFIX=/usr/local/app/cdh5.8.0/hadoop-2.6.0-cdh5.8.0/' >> /etc/profile
echo 'export HADOOP_CONF_DIR=$HADOOP_PREFIX/etc/hadoop/' >> /etc/profile
echo 'export HADOOP_HOME=$HADOOP_PREFIX' >> /etc/profile
echo 'export HBASE_HOME=/usr/local/app/cdh5.8.0/hbase-1.2.0-cdh5.8.0/' >> /etc/profile
echo 'export ZK_HOME=/usr/local/app/cdh5.8.0/zookeeper-3.4.5-cdh5.8.0/' >> /etc/profile
source /etc/profile

 

4.对hbase表进行mapreduce计算测试，运行命令：(注意：这个很长的是一个linux命令，其中重音符号(`)并不是单引号(')，写错成单引号将出现错误！)

HADOOP_CLASSPATH=`${HBASE_HOME}/bin/hbase classpath` ${HADOOP_HOME}/bin/hadoop jar ${HBASE_HOME}/lib/hbase-server-1.2.0-cdh5.8.0.jar rowcounter t2

 

5.运行自己打包的jar：

HADOOP_CLASSPATH=`${HBASE_HOME}/bin/hbase classpath` ${HADOOP_HOME}/bin/hadoop jar test.Maven-0.0.1-SNAPSHOT.jar AnalyzeData -t t2 -c c2:col1 -o analyze1

 

 

 

附录：

pom.xml：比较大，大部分是拷贝的。关键是依赖和shade打包插件。那个fatjar貌似不行。

 

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
  xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
  <modelVersion>4.0.0</modelVersion>

  <groupId>com.xxx</groupId>
  <artifactId>test.Maven</artifactId>
  <version>0.0.1-SNAPSHOT</version>
  <packaging>jar</packaging>

  <name>test.Maven</name>
  <url>http://maven.apache.org</url>

  <properties>
    <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
    <jdkLevel>1.7</jdkLevel>
    <requiredMavenVersion>[2.1,)</requiredMavenVersion>
    <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
    <project.build.outputEncoding>UTF-8</project.build.outputEncoding>

    <hadoop.version>2.7.2</hadoop.version>
    <zookeeper.version>3.4.8</zookeeper.version>
    <hbase.version>1.2.1</hbase.version>
  </properties>

  <build>
  
    <plugins>
      <plugin>
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-jar-plugin</artifactId>
        <version>2.3.1</version>
        <configuration>
          <archive>
            <index>true</index>
            <manifest>
              <mainClass>mapreduce/Driver</mainClass>
            </manifest>
          </archive>
        </configuration>
      </plugin>
      
  		<plugin>
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-shade-plugin</artifactId>
        <version>2.4.3</version>
        <executions>
          <execution>
            <phase>package</phase>
            <goals>
              <goal>shade</goal>
            </goals>
	        <configuration>
	          <!-- put your configurations here -->
	          <transformers>
	          	<transformer implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">
	          		<mainClass>mapreduce.Driver</mainClass>
	          	</transformer>
				<transformer  
					implementation="org.apache.maven.plugins.shade.resource.AppendingTransformer">  
					<resource>META-INF/spring.handlers</resource>  
				</transformer>  
				<transformer  
					implementation="org.apache.maven.plugins.shade.resource.AppendingTransformer">  
					<resource>META-INF/spring.schemas</resource>  
				</transformer>
	          </transformers>
	        </configuration>
          </execution>
        </executions>
      </plugin>
    </plugins>
  </build>
  <dependencies>
  
		<dependency>
		  <groupId>log4j</groupId>
		  <artifactId>log4j</artifactId>
		  <version>1.2.17</version>
		</dependency>
		<dependency>
  <groupId>commons-logging</groupId>
  <artifactId>commons-logging</artifactId>
  <version>1.1.3</version>
</dependency>
  	<dependency>
  		<groupId>junit</groupId>
  		<artifactId>junit</artifactId>
  		<version>4.9</version>
  		<scope>test</scope>
  	</dependency>
  	<dependency>
	  <groupId>org.apache.hbase</groupId>
	  <artifactId>hbase-client</artifactId>
	  <version>1.2.0</version>
	</dependency>
	<dependency>
	    <groupId>jdk.tools</groupId>
	    <artifactId>jdk.tools</artifactId>
	    <version>1.6</version>
	    <scope>system</scope>
	    <systemPath>${JAVA_HOME}/lib/tools.jar</systemPath>
	</dependency>
	
    <dependency>
      <groupId>org.apache.hadoop</groupId>
      <artifactId>hadoop-mapreduce-client-core</artifactId>
      <version>2.6.0</version>
      <scope>provided</scope>
      <exclusions>
        <exclusion>
          <artifactId>hadoop-yarn-common</artifactId>
          <groupId>org.apache.hadoop</groupId>
        </exclusion>
        <exclusion>
          <artifactId>avro</artifactId>
          <groupId>org.apache.avro</groupId>
        </exclusion>
        <exclusion>
          <artifactId>slf4j-log4j12</artifactId>
          <groupId>org.slf4j</groupId>
        </exclusion>
        <exclusion>
          <artifactId>guice-servlet</artifactId>
          <groupId>com.google.inject.extensions</groupId>
        </exclusion>
        <exclusion>
          <artifactId>netty</artifactId>
          <groupId>io.netty</groupId>
        </exclusion>
      </exclusions>
    </dependency>
    <dependency>
      <groupId>commons-cli</groupId>
      <artifactId>commons-cli</artifactId>
      <version>1.2</version>
      <scope>provided</scope>
    </dependency>
    <dependency>
      <groupId>com.googlecode.json-simple</groupId>
      <artifactId>json-simple</artifactId>
      <version>1.1.1</version>
      <exclusions>
        <exclusion>
          <artifactId>hamcrest-core</artifactId>
          <groupId>org.hamcrest</groupId>
        </exclusion>
        <exclusion>
          <artifactId>junit</artifactId>
          <groupId>junit</groupId>
        </exclusion>
      </exclusions>
    </dependency>
    <dependency>
      <groupId>commons-codec</groupId>
      <artifactId>commons-codec</artifactId>
      <version>1.10</version>
    </dependency>
    <dependency>
      <groupId>org.slf4j</groupId>
      <artifactId>log4j-over-slf4j</artifactId>
      <version>1.7.10</version>
    </dependency>
   
    <!-- HBase and Hadoop dependencies -->
    <dependency>
      <groupId>org.apache.hadoop</groupId>
      <artifactId>hadoop-client</artifactId>
      <version>${hadoop.version}</version>
      <exclusions>
        <exclusion>
          <groupId>commons-cli</groupId>
          <artifactId>commons-cli</artifactId>
        </exclusion>
        <exclusion>
          <groupId>xmlenc</groupId>
          <artifactId>xmlenc</artifactId>
        </exclusion>
        <exclusion>
          <groupId>commons-httpclient</groupId>
          <artifactId>commons-httpclient</artifactId>
        </exclusion>
        <exclusion>
          <groupId>commons-codec</groupId>
          <artifactId>commons-codec</artifactId>
        </exclusion>
        <exclusion>
          <groupId>commons-net</groupId>
          <artifactId>commons-net</artifactId>
        </exclusion>
        <exclusion>
          <groupId>org.mortbay.jetty</groupId>
          <artifactId>jetty</artifactId>
        </exclusion>
        <exclusion>
          <groupId>org.mortbay.jetty</groupId>
          <artifactId>jetty-util</artifactId>
        </exclusion>
        <exclusion>
          <groupId>tomcat</groupId>
          <artifactId>jasper-runtime</artifactId>
        </exclusion>
        <exclusion>
          <groupId>tomcat</groupId>
          <artifactId>jasper-compiler</artifactId>
        </exclusion>
        <exclusion>
          <groupId>org.mortbay.jetty</groupId>
          <artifactId>jsp-api-2.1</artifactId>
        </exclusion>
        <exclusion>
          <groupId>org.mortbay.jetty</groupId>
          <artifactId>jsp-2.1</artifactId>
        </exclusion>
        <exclusion>
          <groupId>commons-el</groupId>
          <artifactId>commons-el</artifactId>
        </exclusion>
        <exclusion>
          <groupId>net.java.dev.jets3t</groupId>
          <artifactId>jets3t</artifactId>
        </exclusion>
        <exclusion>
          <groupId>org.mortbay.jetty</groupId>
          <artifactId>servlet-api-2.5</artifactId>
        </exclusion>
        <exclusion>
          <groupId>net.sf.kosmosfs</groupId>
          <artifactId>kfs</artifactId>
        </exclusion>
        <exclusion>
          <groupId>hsqldb</groupId>
          <artifactId>hsqldb</artifactId>
        </exclusion>
        <exclusion>
          <groupId>oro</groupId>
          <artifactId>oro</artifactId>
        </exclusion>
        <exclusion>
          <groupId>org.eclipse.jdt</groupId>
          <artifactId>core</artifactId>
        </exclusion>
        <exclusion>
          <artifactId>commons-math3</artifactId>
          <groupId>org.apache.commons</groupId>
        </exclusion>
        <exclusion>
          <artifactId>commons-collections</artifactId>
          <groupId>commons-collections</groupId>
        </exclusion>
        <exclusion>
          <artifactId>commons-lang</artifactId>
          <groupId>commons-lang</groupId>
        </exclusion>
        <exclusion>
          <artifactId>commons-configuration</artifactId>
          <groupId>commons-configuration</groupId>
        </exclusion>
        <exclusion>
          <artifactId>jackson-core-asl</artifactId>
          <groupId>org.codehaus.jackson</groupId>
        </exclusion>
        <exclusion>
          <artifactId>avro</artifactId>
          <groupId>org.apache.avro</groupId>
        </exclusion>
        <exclusion>
          <artifactId>protobuf-java</artifactId>
          <groupId>com.google.protobuf</groupId>
        </exclusion>
        <exclusion>
          <artifactId>hadoop-auth</artifactId>
          <groupId>org.apache.hadoop</groupId>
        </exclusion>
        <exclusion>
          <artifactId>jsr305</artifactId>
          <groupId>com.google.code.findbugs</groupId>
        </exclusion>
        <exclusion>
          <artifactId>commons-compress</artifactId>
          <groupId>org.apache.commons</groupId>
        </exclusion>
        <exclusion>
          <artifactId>hadoop-mapreduce-client-common</artifactId>
          <groupId>org.apache.hadoop</groupId>
        </exclusion>
        <exclusion>
          <artifactId>hadoop-mapreduce-client-shuffle</artifactId>
          <groupId>org.apache.hadoop</groupId>
        </exclusion>
        <exclusion>
          <artifactId>hadoop-yarn-common</artifactId>
          <groupId>org.apache.hadoop</groupId>
        </exclusion>
        <exclusion>
          <artifactId>hadoop-common</artifactId>
          <groupId>org.apache.hadoop</groupId>
        </exclusion>
        <exclusion>
          <artifactId>hadoop-hdfs</artifactId>
          <groupId>org.apache.hadoop</groupId>
        </exclusion>
        <exclusion>
          <artifactId>hadoop-mapreduce-client-app</artifactId>
          <groupId>org.apache.hadoop</groupId>
        </exclusion>
        <exclusion>
          <artifactId>hadoop-mapreduce-client-core</artifactId>
          <groupId>org.apache.hadoop</groupId>
        </exclusion>
        <exclusion>
          <artifactId>hadoop-mapreduce-client-jobclient</artifactId>
          <groupId>org.apache.hadoop</groupId>
        </exclusion>
        <exclusion>
          <artifactId>hadoop-annotations</artifactId>
          <groupId>org.apache.hadoop</groupId>
        </exclusion>
        <exclusion>
          <artifactId>hadoop-yarn-api</artifactId>
          <groupId>org.apache.hadoop</groupId>
        </exclusion>
      </exclusions>
    </dependency>
    <dependency>
      <groupId>org.apache.hadoop</groupId>
      <artifactId>hadoop-common</artifactId>
      <version>${hadoop.version}</version>
      <exclusions>
        <exclusion>
          <artifactId>hadoop-annotations</artifactId>
          <groupId>org.apache.hadoop</groupId>
        </exclusion>
        <exclusion>
          <artifactId>commons-cli</artifactId>
          <groupId>commons-cli</groupId>
        </exclusion>
        <exclusion>
          <artifactId>commons-math3</artifactId>
          <groupId>org.apache.commons</groupId>
        </exclusion>
        <exclusion>
          <artifactId>xmlenc</artifactId>
          <groupId>xmlenc</groupId>
        </exclusion>
        <exclusion>
          <artifactId>commons-codec</artifactId>
          <groupId>commons-codec</groupId>
        </exclusion>
        <exclusion>
          <artifactId>commons-math3</artifactId>
          <groupId>org.apache.commons</groupId>
        </exclusion>
        <exclusion>
          <artifactId>commons-io</artifactId>
          <groupId>commons-io</groupId>
        </exclusion>
        <exclusion>
          <artifactId>commons-httpclient</artifactId>
          <groupId>commons-httpclient</groupId>
        </exclusion>
        <exclusion>
          <artifactId>commons-httpclient</artifactId>
          <groupId>commons-httpclient</groupId>
        </exclusion>
        <exclusion>
          <artifactId>commons-net</artifactId>
          <groupId>commons-net</groupId>
        </exclusion>
        <exclusion>
          <artifactId>commons-collections</artifactId>
          <groupId>commons-collections</groupId>
        </exclusion>
        <exclusion>
          <artifactId>servlet-api</artifactId>
          <groupId>javax.servlet</groupId>
        </exclusion>
        <exclusion>
          <artifactId>jersey-core</artifactId>
          <groupId>com.sun.jersey</groupId>
        </exclusion>
        <exclusion>
          <artifactId>jersey-json</artifactId>
          <groupId>com.sun.jersey</groupId>
        </exclusion>
        <exclusion>
          <artifactId>jetty</artifactId>
          <groupId>org.mortbay.jetty</groupId>
        </exclusion>
        <exclusion>
          <artifactId>jetty-util</artifactId>
          <groupId>org.mortbay.jetty</groupId>
        </exclusion>
        <exclusion>
          <artifactId>jersey-server</artifactId>
          <groupId>com.sun.jersey</groupId>
        </exclusion>
        <exclusion>
          <artifactId>jsp-api</artifactId>
          <groupId>javax.servlet.jsp</groupId>
        </exclusion>
        <exclusion>
          <artifactId>commons-el</artifactId>
          <groupId>commons-el</groupId>
        </exclusion>
        <exclusion>
          <artifactId>jets3t</artifactId>
          <groupId>net.java.dev.jets3t</groupId>
        </exclusion>
        <exclusion>
          <artifactId>hadoop-auth</artifactId>
          <groupId>org.apache.hadoop</groupId>
        </exclusion>
        <exclusion>
          <artifactId>jasper-compiler</artifactId>
          <groupId>tomcat</groupId>
        </exclusion>
        <exclusion>
          <artifactId>jasper-runtime</artifactId>
          <groupId>tomcat</groupId>
        </exclusion>
        <exclusion>
          <artifactId>commons-lang</artifactId>
          <groupId>commons-lang</groupId>
        </exclusion>
        <exclusion>
          <artifactId>commons-lang</artifactId>
          <groupId>commons-lang</groupId>
        </exclusion>
        <exclusion>
          <artifactId>commons-configuration</artifactId>
          <groupId>commons-configuration</groupId>
        </exclusion>
        <exclusion>
          <artifactId>slf4j-log4j12</artifactId>
          <groupId>org.slf4j</groupId>
        </exclusion>
        <exclusion>
          <artifactId>jackson-mapper-asl</artifactId>
          <groupId>org.codehaus.jackson</groupId>
        </exclusion>
        <exclusion>
          <artifactId>protobuf-java</artifactId>
          <groupId>com.google.protobuf</groupId>
        </exclusion>
        <exclusion>
          <artifactId>slf4j-api</artifactId>
          <groupId>org.slf4j</groupId>
        </exclusion>
        <exclusion>
          <artifactId>jackson-core-asl</artifactId>
          <groupId>org.codehaus.jackson</groupId>
        </exclusion>
        <exclusion>
          <artifactId>avro</artifactId>
          <groupId>org.apache.avro</groupId>
        </exclusion>
        <exclusion>
          <artifactId>jsch</artifactId>
          <groupId>com.jcraft</groupId>
        </exclusion>
        <exclusion>
          <artifactId>jsr305</artifactId>
          <groupId>com.google.code.findbugs</groupId>
        </exclusion>
        <exclusion>
          <artifactId>commons-compress</artifactId>
          <groupId>org.apache.commons</groupId>
        </exclusion>
        <exclusion>
          <artifactId>commons-logging</artifactId>
          <groupId>commons-logging</groupId>
        </exclusion>
        <exclusion>
          <artifactId>log4j</artifactId>
          <groupId>log4j</groupId>
        </exclusion>
        <exclusion>
          <artifactId>log4j</artifactId>
          <groupId>log4j</groupId>
        </exclusion>
        <exclusion>
          <artifactId>htrace-core</artifactId>
          <groupId>org.htrace</groupId>
        </exclusion>
        <exclusion>
          <artifactId>gson</artifactId>
          <groupId>com.google.code.gson</groupId>
        </exclusion>
        <exclusion>
          <artifactId>curator-client</artifactId>
          <groupId>org.apache.curator</groupId>
        </exclusion>
        <exclusion>
          <artifactId>curator-recipes</artifactId>
          <groupId>org.apache.curator</groupId>
        </exclusion>
      </exclusions>
    </dependency>
    <dependency>
      <groupId>org.apache.hadoop</groupId>
      <artifactId>hadoop-auth</artifactId>
      <version>${hadoop.version}</version>
      <exclusions>
        <exclusion>
          <artifactId>commons-codec</artifactId>
          <groupId>commons-codec</groupId>
        </exclusion>
        <exclusion>
          <artifactId>log4j</artifactId>
          <groupId>log4j</groupId>
        </exclusion>
        <exclusion>
          <artifactId>slf4j-log4j12</artifactId>
          <groupId>org.slf4j</groupId>
        </exclusion>
        <exclusion>
          <artifactId>httpclient</artifactId>
          <groupId>org.apache.httpcomponents</groupId>
        </exclusion>
        <exclusion>
          <artifactId>apacheds-kerberos-codec</artifactId>
          <groupId>org.apache.directory.server</groupId>
        </exclusion>
        <exclusion>
          <artifactId>curator-framework</artifactId>
          <groupId>org.apache.curator</groupId>
        </exclusion>
      </exclusions>
    </dependency>
    <dependency>
      <groupId>org.apache.hadoop</groupId>
      <artifactId>hadoop-annotations</artifactId>
      <version>${hadoop.version}</version>
    </dependency>
    <dependency>
      <groupId>org.apache.zookeeper</groupId>
      <artifactId>zookeeper</artifactId>
      <version>${zookeeper.version}</version>
      <exclusions>
        <exclusion>
          <groupId>log4j</groupId>
          <artifactId>log4j</artifactId>
        </exclusion>
        <exclusion>
          <groupId>jline</groupId>
          <artifactId>jline</artifactId>
        </exclusion>
        <exclusion>
          <artifactId>netty</artifactId>
          <groupId>io.netty</groupId>
        </exclusion>
        <exclusion>
          <artifactId>slf4j-api</artifactId>
          <groupId>org.slf4j</groupId>
        </exclusion>
        <exclusion>
          <artifactId>slf4j-log4j12</artifactId>
          <groupId>org.slf4j</groupId>
        </exclusion>
      </exclusions>
    </dependency>
    <dependency>
      <groupId>org.apache.hbase</groupId>
      <artifactId>hbase-client</artifactId>
      <version>${hbase.version}</version>
      <exclusions>
        <exclusion>
          <groupId>org.apache.avro</groupId>
          <artifactId>avro</artifactId>
        </exclusion>
        <exclusion>
          <groupId>commons-lang</groupId>
          <artifactId>commons-lang</artifactId>
        </exclusion>
        <exclusion>
          <groupId>com.google.guava</groupId>
          <artifactId>guava</artifactId>
        </exclusion>
        <exclusion>
          <groupId>commons-logging</groupId>
          <artifactId>commons-logging</artifactId>
        </exclusion>
        <exclusion>
          <groupId>log4j</groupId>
          <artifactId>log4j</artifactId>
        </exclusion>
        <exclusion>
          <groupId>org.apache.hadoop</groupId>
          <artifactId>avro</artifactId>
        </exclusion>
        <exclusion>
          <groupId>org.apache.zookeeper</groupId>
          <artifactId>zookeeper</artifactId>
        </exclusion>
        <exclusion>
          <groupId>org.apache.thrift</groupId>
          <artifactId>thrift</artifactId>
        </exclusion>
        <exclusion>
          <groupId>org.jruby</groupId>
          <artifactId>jruby-complete</artifactId>
        </exclusion>
        <exclusion>
          <groupId>org.slf4j</groupId>
          <artifactId>slf4j-api</artifactId>
        </exclusion>
        <exclusion>
          <groupId>org.slf4j</groupId>
          <artifactId>slf4j-log4j12</artifactId>
        </exclusion>
        <exclusion>
          <groupId>com.google.protobuf</groupId>
          <artifactId>protobuf-java</artifactId>
        </exclusion>
        <exclusion>
          <groupId>com.sun.jersey</groupId>
          <artifactId>jersey-core</artifactId>
        </exclusion>
        <exclusion>
          <groupId>com.sun.jersey</groupId>
          <artifactId>jersey-json</artifactId>
        </exclusion>
        <exclusion>
          <groupId>com.sun.jersey</groupId>
          <artifactId>jersey-server</artifactId>
        </exclusion>
        <exclusion>
          <groupId>javax.ws.rs</groupId>
          <artifactId>jsr311-api</artifactId>
        </exclusion>
        <exclusion>
          <groupId>javax.xml.bind</groupId>
          <artifactId>jaxb-api</artifactId>
        </exclusion>
        <exclusion>
          <groupId>stax</groupId>
          <artifactId>stax-api</artifactId>
        </exclusion>
        <exclusion>
          <groupId>commons-cli</groupId>
          <artifactId>commons-cli</artifactId>
        </exclusion>
        <exclusion>
          <groupId>commons-codec</groupId>
          <artifactId>commons-codec</artifactId>
        </exclusion>
        <exclusion>
          <groupId>commons-httpclient</groupId>
          <artifactId>commons-httpclient</artifactId>
        </exclusion>
        <exclusion>
          <groupId>org.mortbay.jetty</groupId>
          <artifactId>jetty</artifactId>
        </exclusion>
        <exclusion>
          <groupId>org.mortbay.jetty</groupId>
          <artifactId>jetty-util</artifactId>
        </exclusion>
        <exclusion>
          <groupId>org.mortbay.jetty</groupId>
          <artifactId>jsp-2.1</artifactId>
        </exclusion>
        <exclusion>
          <groupId>org.mortbay.jetty</groupId>
          <artifactId>jsp-api-2.1</artifactId>
        </exclusion>
        <exclusion>
          <groupId>org.mortbay.jetty</groupId>
          <artifactId>servlet-api-2.5</artifactId>
        </exclusion>
        <exclusion>
          <groupId>tomcat</groupId>
          <artifactId>jasper-compiler</artifactId>
        </exclusion>
        <exclusion>
          <groupId>tomcat</groupId>
          <artifactId>jasper-runtime</artifactId>
        </exclusion>
        <exclusion>
          <groupId>com.github.stephenc.high-scale-lib</groupId>
          <artifactId>high-scale-lib</artifactId>
        </exclusion>
        <exclusion>
          <groupId>org.apache.thrift</groupId>
          <artifactId>libthrift</artifactId>
        </exclusion>
        <exclusion>
          <groupId>org.jamon</groupId>
          <artifactId>jamon-runtime</artifactId>
        </exclusion>
        <exclusion>
          <artifactId>jdk.tools</artifactId>
          <groupId>jdk.tools</groupId>
        </exclusion>
        <exclusion>
          <artifactId>commons-collections</artifactId>
          <groupId>commons-collections</groupId>
        </exclusion>
        <exclusion>
          <artifactId>netty-all</artifactId>
          <groupId>io.netty</groupId>
        </exclusion>
        <exclusion>
          <artifactId>commons-io</artifactId>
          <groupId>commons-io</groupId>
        </exclusion>
        <exclusion>
          <artifactId>htrace-core</artifactId>
          <groupId>org.htrace</groupId>
        </exclusion>
        <exclusion>
          <artifactId>hadoop-common</artifactId>
          <groupId>org.apache.hadoop</groupId>
        </exclusion>
        <exclusion>
          <artifactId>hbase-annotations</artifactId>
          <groupId>org.apache.hbase</groupId>
        </exclusion>
        <exclusion>
          <artifactId>hbase-common</artifactId>
          <groupId>org.apache.hbase</groupId>
        </exclusion>
        <exclusion>
          <artifactId>hbase-protocol</artifactId>
          <groupId>org.apache.hbase</groupId>
        </exclusion>
        <exclusion>
          <artifactId>jackson-mapper-asl</artifactId>
          <groupId>org.codehaus.jackson</groupId>
        </exclusion>
        <exclusion>
          <artifactId>joni</artifactId>
          <groupId>org.jruby.joni</groupId>
        </exclusion>
        <exclusion>
          <artifactId>hadoop-mapreduce-client-core</artifactId>
          <groupId>org.apache.hadoop</groupId>
        </exclusion>
        <exclusion>
          <artifactId>findbugs-annotations</artifactId>
          <groupId>com.github.stephenc.findbugs</groupId>
        </exclusion>
        <exclusion>
          <artifactId>junit</artifactId>
          <groupId>junit</groupId>
        </exclusion>
        <exclusion>
          <artifactId>hadoop-auth</artifactId>
          <groupId>org.apache.hadoop</groupId>
        </exclusion>
        <exclusion>
          <artifactId>htrace-core</artifactId>
          <groupId>org.apache.htrace</groupId>
        </exclusion>
        <exclusion>
          <artifactId>jcodings</artifactId>
          <groupId>org.jruby.jcodings</groupId>
        </exclusion>
        <exclusion>
          <artifactId>hadoop-auth</artifactId>
          <groupId>org.apache.hadoop</groupId>
        </exclusion>
      </exclusions>
    </dependency>
    <dependency>
      <groupId>org.apache.hbase</groupId>
      <artifactId>hbase-common</artifactId>
      <version>${hbase.version}</version>
      <exclusions>
        <exclusion>
          <artifactId>hbase-annotations</artifactId>
          <groupId>org.apache.hbase</groupId>
        </exclusion>
        <exclusion>
          <artifactId>hadoop-mapreduce-client-core</artifactId>
          <groupId>org.apache.hadoop</groupId>
        </exclusion>
        <exclusion>
          <artifactId>commons-codec</artifactId>
          <groupId>commons-codec</groupId>
        </exclusion>
        <exclusion>
          <artifactId>commons-lang</artifactId>
          <groupId>commons-lang</groupId>
        </exclusion>
        <exclusion>
          <artifactId>commons-collections</artifactId>
          <groupId>commons-collections</groupId>
        </exclusion>
        <exclusion>
          <artifactId>protobuf-java</artifactId>
          <groupId>com.google.protobuf</groupId>
        </exclusion>
        <exclusion>
          <artifactId>htrace-core</artifactId>
          <groupId>org.htrace</groupId>
        </exclusion>
        <exclusion>
          <artifactId>findbugs-annotations</artifactId>
          <groupId>com.github.stephenc.findbugs</groupId>
        </exclusion>
        <exclusion>
          <artifactId>junit</artifactId>
          <groupId>junit</groupId>
        </exclusion>
        <exclusion>
          <artifactId>commons-io</artifactId>
          <groupId>commons-io</groupId>
        </exclusion>
        <exclusion>
          <artifactId>jetty-util</artifactId>
          <groupId>org.mortbay.jetty</groupId>
        </exclusion>
        <exclusion>
          <artifactId>commons-logging</artifactId>
          <groupId>commons-logging</groupId>
        </exclusion>
        <exclusion>
          <artifactId>htrace-core</artifactId>
          <groupId>org.apache.htrace</groupId>
        </exclusion>
        <exclusion>
          <artifactId>log4j</artifactId>
          <groupId>log4j</groupId>
        </exclusion>
      </exclusions>
    </dependency>
    <dependency>
      <groupId>org.apache.hbase</groupId>
      <artifactId>hbase-protocol</artifactId>
      <version>${hbase.version}</version>
      <exclusions>
        <exclusion>
          <artifactId>findbugs-annotations</artifactId>
          <groupId>com.github.stephenc.findbugs</groupId>
        </exclusion>
        <exclusion>
          <artifactId>log4j</artifactId>
          <groupId>log4j</groupId>
        </exclusion>
        <exclusion>
          <artifactId>hbase-annotations</artifactId>
          <groupId>org.apache.hbase</groupId>
        </exclusion>
        <exclusion>
          <artifactId>junit</artifactId>
          <groupId>junit</groupId>
        </exclusion>
        <exclusion>
          <artifactId>commons-logging</artifactId>
          <groupId>commons-logging</groupId>
        </exclusion>
      </exclusions>
    </dependency>
    <dependency>
      <groupId>org.apache.hbase</groupId>
      <artifactId>hbase-server</artifactId>
      <version>${hbase.version}</version>
      <exclusions>
        <exclusion>
          <artifactId>hbase-protocol</artifactId>
          <groupId>org.apache.hbase</groupId>
        </exclusion>
        <exclusion>
          <artifactId>hbase-prefix-tree</artifactId>
          <groupId>org.apache.hbase</groupId>
        </exclusion>
        <exclusion>
          <artifactId>commons-collections</artifactId>
          <groupId>commons-collections</groupId>
        </exclusion>
        <exclusion>
          <artifactId>hbase-hadoop-compat</artifactId>
          <groupId>org.apache.hbase</groupId>
        </exclusion>
        <exclusion>
          <artifactId>hadoop-auth</artifactId>
          <groupId>org.apache.hadoop</groupId>
        </exclusion>
        <exclusion>
          <artifactId>hadoop-mapreduce-client-core</artifactId>
          <groupId>org.apache.hadoop</groupId>
        </exclusion>
        <exclusion>
          <artifactId>commons-httpclient</artifactId>
          <groupId>commons-httpclient</groupId>
        </exclusion>
        <exclusion>
          <artifactId>hbase-hadoop2-compat</artifactId>
          <groupId>org.apache.hbase</groupId>
        </exclusion>
        <exclusion>
          <artifactId>metrics-core</artifactId>
          <groupId>com.yammer.metrics</groupId>
        </exclusion>
        <exclusion>
          <artifactId>commons-io</artifactId>
          <groupId>commons-io</groupId>
        </exclusion>
        <exclusion>
          <artifactId>jetty</artifactId>
          <groupId>org.mortbay.jetty</groupId>
        </exclusion>
        <exclusion>
          <artifactId>hadoop-hdfs</artifactId>
          <groupId>org.apache.hadoop</groupId>
        </exclusion>
        <exclusion>
          <artifactId>findbugs-annotations</artifactId>
          <groupId>com.github.stephenc.findbugs</groupId>
        </exclusion>
        <exclusion>
          <artifactId>junit</artifactId>
          <groupId>junit</groupId>
        </exclusion>
        <exclusion>
          <artifactId>commons-cli</artifactId>
          <groupId>commons-cli</groupId>
        </exclusion>
        <exclusion>
          <artifactId>commons-lang</artifactId>
          <groupId>commons-lang</groupId>
        </exclusion>
        <exclusion>
          <artifactId>commons-math</artifactId>
          <groupId>org.apache.commons</groupId>
        </exclusion>
        <exclusion>
          <artifactId>jetty-sslengine</artifactId>
          <groupId>org.mortbay.jetty</groupId>
        </exclusion>
        <exclusion>
          <artifactId>jsp-2.1</artifactId>
          <groupId>org.mortbay.jetty</groupId>
        </exclusion>
        <exclusion>
          <artifactId>servlet-api-2.5</artifactId>
          <groupId>org.mortbay.jetty</groupId>
        </exclusion>
        <exclusion>
          <artifactId>jackson-mapper-asl</artifactId>
          <groupId>org.codehaus.jackson</groupId>
        </exclusion>
        <exclusion>
          <artifactId>jackson-jaxrs</artifactId>
          <groupId>org.codehaus.jackson</groupId>
        </exclusion>
        <exclusion>
          <artifactId>jasper-runtime</artifactId>
          <groupId>tomcat</groupId>
        </exclusion>
        <exclusion>
          <artifactId>netty-all</artifactId>
          <groupId>io.netty</groupId>
        </exclusion>
        <exclusion>
          <artifactId>htrace-core</artifactId>
          <groupId>org.htrace</groupId>
        </exclusion>
        <exclusion>
          <artifactId>disruptor</artifactId>
          <groupId>com.lmax</groupId>
        </exclusion>
        <exclusion>
          <artifactId>jsp-api-2.1</artifactId>
          <groupId>org.mortbay.jetty</groupId>
        </exclusion>
        <exclusion>
          <artifactId>jackson-core-asl</artifactId>
          <groupId>org.codehaus.jackson</groupId>
        </exclusion>
        <exclusion>
          <artifactId>jasper-compiler</artifactId>
          <groupId>tomcat</groupId>
        </exclusion>
        <exclusion>
          <artifactId>jamon-runtime</artifactId>
          <groupId>org.jamon</groupId>
        </exclusion>
        <exclusion>
          <artifactId>jetty-util</artifactId>
          <groupId>org.mortbay.jetty</groupId>
        </exclusion>
        <exclusion>
          <artifactId>commons-codec</artifactId>
          <groupId>commons-codec</groupId>
        </exclusion>
        <exclusion>
          <artifactId>jersey-core</artifactId>
          <groupId>com.sun.jersey</groupId>
        </exclusion>
        <exclusion>
          <artifactId>jersey-server</artifactId>
          <groupId>com.sun.jersey</groupId>
        </exclusion>
        <exclusion>
          <artifactId>log4j</artifactId>
          <groupId>log4j</groupId>
        </exclusion>
        <exclusion>
          <artifactId>htrace-core</artifactId>
          <groupId>org.apache.htrace</groupId>
        </exclusion>
        <exclusion>
          <artifactId>commons-logging</artifactId>
          <groupId>commons-logging</groupId>
        </exclusion>
        <exclusion>
          <artifactId>commons-logging</artifactId>
          <groupId>commons-logging</groupId>
        </exclusion>
      </exclusions>
    </dependency>
    <dependency>
      <groupId>org.apache.hbase</groupId>
      <artifactId>hbase-annotations</artifactId>
      <version>${hbase.version}</version>
      <exclusions>
        <exclusion>
          <artifactId>findbugs-annotations</artifactId>
          <groupId>com.github.stephenc.findbugs</groupId>
        </exclusion>
        <exclusion>
          <artifactId>log4j</artifactId>
          <groupId>log4j</groupId>
        </exclusion>
        <exclusion>
          <artifactId>junit</artifactId>
          <groupId>junit</groupId>
        </exclusion>
      </exclusions>
    </dependency>
   
    
		<dependency>
		  <groupId>commons-logging</groupId>
		  <artifactId>commons-logging</artifactId>
		  <version>1.1.3</version>
		</dependency>
		<dependency>
		  <groupId>commons-collections</groupId>
		  <artifactId>commons-collections</artifactId>
		  <version>3.0</version>
		</dependency>
		<dependency>
		  <groupId>commons-lang</groupId>
		  <artifactId>commons-lang</artifactId>
		  <version>2.6</version>
		</dependency>
   
	   <dependency>
		  <groupId>io.netty</groupId>
		  <artifactId>netty-all</artifactId>
		  <version>4.1.5.Final</version>
		</dependency>
  </dependencies>

  <profiles>
    <profile>
      <id>fatjar</id>
      <activation>
        <activeByDefault>false</activeByDefault>
        <property>
          <name>fatjar</name>
        </property>
      </activation>
      <!--dependencies>
        <dependency>
          <groupId>org.apache.hadoop</groupId>
          <artifactId>hadoop-snappy</artifactId>
          <version>${hadoop-snappy.version}</version>
        </dependency>
      </dependencies-->
      <build>
        <plugins>
          <plugin>
            <groupId>org.apache.maven.plugins</groupId>
            <artifactId>maven-assembly-plugin</artifactId>
            <version>2.5.3</version>
            <configuration>
              <tarLongFileMode>gnu</tarLongFileMode>
              <descriptors>
                <descriptor>src/main/assembly/job.xml</descriptor>
              </descriptors>
              <!--descriptorRefs>
                <descriptorRef>jar-with-dependencies</descriptorRef>
              </descriptorRefs-->
              <archive>
                <index>true</index>
                <manifest>
                  <mainClass>mapreduce/Driver</mainClass>
                </manifest>
              </archive>
            </configuration>
            <executions>
              <execution>
                <id>tarball</id>
                <phase>package</phase>
                <goals>
                  <goal>single</goal>
                </goals>
              </execution>
            </executions>
          </plugin>
        </plugins>
      </build>
    </profile>
  </profiles>
</project>

 

 

 

 

 

 

分享到：
| java客户端连接操作Hbase示例代码
评论
发表评论

您还没有登录,请您登录后再发表评论

相关推荐

    HBase与MapReduce处理操作（基于JavaAPI）

    该案例中主要使用MapReduce作为处理组件进行数据处理，实现的案例有如通过javaapi实现hbase数据写入hdfs、hbase表数据复制到另一个表中等操作 对应（《HBase分布式存储系统应用》胡鑫喆 张志刚著）教材中案例

    HBase使用的jar包

    在实际开发中，我们通常会将所有这些依赖的jar包打包到一个fat jar或者使用Maven或Gradle的依赖管理工具，这样可以确保所有必要的库都在运行环境中可用。对于明细查询，HBase提供了多种查询方式，如Get、Scan等，...

    基于SpringMVC+Spring+HBase+Maven搭建的Hadoop分布式云盘系统.zip

    这是一个基于Java技术栈，利用SpringMVC、Spring、HBase和Maven构建的Hadoop分布式云盘系统的项目。该项目旨在实现一个高效的、可扩展的云存储解决方案，利用Hadoop的分布式特性来处理大规模数据存储需求。 首先，...

    云应用系统开发第二次项目（mapreduce）

    8. 打包：将项目打包成 JAR 文件，以便在 Hadoop 集群上运行。 9. 上传 JAR 和 TXT：上传 JAR 文件和 TXT 文件到 Hadoop 集群上，以便运行 MapReduce 任务。 10. 运行截图：截取 MapReduce 任务的运行结果，以便查看...

    Hbase中文文档

    2.4. HBase运行模式:单机和分布式 2.5. ZooKeeper 2.6. 配置文件 2.7. 配置示例 2.8. 重要配置 2.9. Bloom Filter 3. 升级 3.1. 从HBase 0.20.x or 0.89.x 升级到 HBase 0.90.x 3.2. 从 0.90.x 到 0.92.x 4. The ...

    hadoop hbase 全jar包

    在实际使用中，开发者可能需要将这些jar包添加到类路径（classpath）中，以便编译和运行HBase相关的Java程序。例如，如果要编写一个连接HBase的Java应用程序，需要包含HBase的client jar包，这样程序才能调用HBase的...

    java操作Hbase之从Hbase中读取数据写入hdfs中源码

    在Java编程环境中，操作HBase并将其数据写入HDFS（Hadoop Distributed File System）是一项常见的任务，特别是在大数据处理和分析的场景下。本篇将详细介绍如何使用Java API实现这一功能，以及涉及到的关键技术和...

    java从本地读文件并上传Hbase

    在Java编程环境中，将本地文件读取并上传到HBase是一项常见的任务，特别是在大数据处理和存储的场景下。HBase是一个分布式、版本化的NoSQL数据库，基于Apache Hadoop，适用于大规模数据存储。以下是一个详细的过程，...

    Hadoop之HBase学习笔记

    HBase是一个基于Google Bigtable理念设计的开源NoSQL数据库，它运行在Hadoop之上，提供高性能、高可靠性以及可水平扩展的数据存储能力。这篇学习笔记可能是作者在深入研究HBase的过程中对关键概念、架构、操作和最佳...

    HBase_官方文档-中文翻译

    此外，可以在 MapReduce 作业中访问其他 HBase 表，实现复杂的分析任务。 **Schema 设计** HBase 的 Schema 设计直接影响其性能和可扩展性。表的创建、Column Family 数量、单调递增 Row Key（适用于时序数据）...

    hadoop1.0.2 hbase0.94安装

    本教程主要涵盖的是在较旧版本的Hadoop 1.0.2上安装并配置HBase 0.94，以及相关的MapReduce开发和Hadoop-Eclipse插件的编译。这些内容对于理解大数据处理的基本流程和工具使用具有重要意义。 首先，我们来详细讨论...

    spring与hbase集成

    Spring 与 HBase 集成是大数据领域中常见的技术结合，主要应用于处理大规模结构化数据存储和检索。...此外，Spring Data HBase 还提供了对 MapReduce 和 Spark 作业的支持，便于在 HBase 上进行大数据分析。

    java开发hbase1.2.6需要的jar包整合

    在Java开发HBase 1.2.6的过程中，理解依赖的jar包以及它们的作用至关重要，因为这直接影响到程序的正确运行。HBase是基于Apache Hadoop构建的一个分布式、版本化的非关系型数据库，用于存储大规模结构化数据。而...

    java集成hadoop-hbase用到的jar包

    Hadoop是一个分布式存储和计算框架，而HBase是一个构建在Hadoop之上的非关系型数据库（NoSQL），特别适合处理大规模数据。这里我们将详细探讨Java如何与这两个组件进行交互，并重点关注所需的jar包。 首先，Java...

    Hadoop-2.4.0+Hbase-0.94.18+Nutch-2.3集群爬虫配置攻略

    在搭建Hadoop-2.4.0集群时，首先需要确保系统已安装必要的依赖包和工具，包括Maven 3.0或更高版本，Findbugs 1.3.9（如果要运行findbugs），ProtocolBuffer 2.5.0以及CMake 2.6或更新版本（如果要编译本地代码）。...

    华为MapReduce服务应用开发指南.pdf

    华为MapReduce服务应用开发指南是一份详细的技术文档，指导开发者如何在华为提供的MapReduce服务（MRS）上进行应用开发。文档详细说明了如何准备开发环境，包括如何通过开源镜像站点获取样例工程，如何处理开源jar包...

    eclipe 连接hbase

    此外，还需要确保你的系统中已经安装并运行了Hadoop，因为HBase是构建在Hadoop之上的。 接下来，你需要在Eclipse中创建一个新的Java项目。在项目中，添加HBase相关的依赖库。如果你使用的是Maven，可以在pom.xml...

    Mapreduce的基础搭建以及Java API编程

    Eclipse作为开发工具，可以配合Maven或Gradle构建MapReduce项目，将编写的Java代码打包成JAR文件，然后通过Hadoop的`hadoop jar`命令提交到集群执行。 七、优化与实践 在实际应用中，为了提高MapReduce性能，需要...

    hadoop的各种jar包

    7. **打包与部署**：最后，将项目打包成可执行的JAR文件，部署到Hadoop集群上运行。 在实际开发中，你可能还需要关注Hadoop的版本兼容性，因为不同的Hadoop版本之间可能存在API的不兼容性。此外，了解Hadoop的其他...

    零基础学习hadoop编程篇 (3).docx

    6. **Hadoop集群连接**：使用Hadoop-Eclipse-Plugin可以方便地在Eclipse中远程调试和运行MapReduce程序。 7. **源码编译**：如果你需要修改或扩展Hadoop源码，你需要了解如何使用Maven来编译Hadoop。这包括下载源码...

Magicbox
Global site tag (gtag.js) - Google Analytics