datanucleus

hiveserver
./hive --service start-hive
./hive --service stop-hive

hive的metastore使用了datanucleus

下载datanucleus-samples-jdo-tutorial-2.0-src.zip
解压datanucleus-samples-jdo-tutorial-2.0，把里面的lib目录，src目录，datanucleus.properties  log4j.properties orm.xml都拷到另外一个目录下面

创建 target/classes文件。

修改datanucleus.properties文件为：
javax.jdo.PersistenceManagerFactoryClass=org.datanucleus.jdo.JDOPersistenceManagerFactory

javax.jdo.option.ConnectionDriverName=com.mysql.jdbc.Driver
javax.jdo.option.ConnectionURL=jdbc:mysql://localhost:3306/hive?createDatabaseIfNotExist=true
javax.jdo.option.ConnectionUserName=hive
javax.jdo.option.ConnectionPassword=123456
javax.jdo.option.Mapping=hsql

datanucleus.metadata.validate=false
datanucleus.autoCreateSchema=true
datanucleus.validateTables=false
datanucleus.validateConstraints=false


加上下面这个选项会出现问题：Unable to fetch table ***
<property>
  <name>datanucleus.cache.collections</name>
  <value>false</value>
</property>


datanucleus.cache.level2.type       从 soft  改为  none


datanucleus.fixedDatastore = true; 出现问题。
https://issues.apache.org/jira/browse/HIVE-1841

hive-site.xml中加入下面的配置能够解决下面的问题：Failures with DELETEME tables
<property>
<name>datanucleus.fixedDatastore</name>
<value>true</value>
</property>

并发造成的错误：
Caused by: com.mysql.jdbc.exceptions.MySQLSyntaxErrorException: Table 'dataoven_prod_hadoop.DELETEME1309959999747' doesn't exist

http://search-hadoop.com/m/bi3w92UVF3L/Failures+with+DELETEME+tables&subj=Failures+with+DELETEME+tables

The datastore exception was thrown because com.mysql.jdbc.DatabaseMetaData.getColumns has a race condition. It was called with tableNamePattern == null and because of that it went and fetched a list of all tables, and for each table, it executed a query to fetch table info. Before all table-info fetches were done, if a table were dropped by some other process, the fetch would fail and getColumns would throw an exception. This is a race condition bug.

For now, I have enabled datanucleus.fixedDatastore in hive-site.xml to eliminate the use of DELTEME tables, virtually eliminating the problem. But the problem can still happen with non-DELETEME causes due to the 2 bugs described above or when datanucleus.fixedDatastore needs to be disabled occasionally.


hiveserver take too much memory：
http://search-hadoop.com/m/PIsyn1GZxfh1&subj=java+lang+OutOfMemoryError+PermGen+space+when+running+as+a+service+

move all properties from jpox.properties to hive-site.xml
https://issues.apache.org/jira/browse/HIVE-610





操作如下：
编译生成的class文件到target/classes
tianzhao@tianzhao-VirtualBox:~/software/datanucleus/example$ javac  -cp lib/jdo2-api-2.3-ec.jar:lib/datanucleus-core-2.0.3.jar   -sourcepath src/java/ src/java/org/datanucleus/samples/jdo/tutorial/*.java -d target/classes

tianzhao@tianzhao-VirtualBox:~/software/datanucleus/example$ cp src/java/org/datanucleus/samples/jdo/tutorial/package-hsql.orm target/classes/org/datanucleus/samples/jdo/tutorial/

tianzhao@tianzhao-VirtualBox:~/software/datanucleus/example$ java -cp target/classes/:lib/datanucleus-enhancer-2.0.3.jar:lib/datanucleus-core-2.0.3.jar:lib/jdo2-api-2.3-ec.jar:lib/asm-3.1.jar org.datanucleus.enhancer.DataNucleusEnhancer target/classes/org/datanucleus/samples/jdo/tutorial/package-hsql.orm

在数据库中创建表
tianzhao@tianzhao-VirtualBox:~/software/datanucleus/example$ java -cp target/classes/:lib/datanucleus-rdbms-2.0.3.jar:lib/datanucleus-core-2.0.3.jar:lib/jdo2-api-2.3-ec.jar:lib/mysql-connector-java-5.1.6.jar org.datanucleus.store.rdbms.SchemaTool -props datanucleus.properties -create target/classes/org/datanucleus/samples/jdo/tutorial/package-hsql.orm

执行相关操作
tianzhao@tianzhao-VirtualBox:~/software/datanucleus/example$ java -cp lib/jdo2-api-2.3-ec.jar:lib/datanucleus-core-2.0.3.jar:lib/datanucleus-rdbms-2.0.3.jar:lib/mysql-connector-java-5.1.6.jar:target/classes/:.  org.datanucleus.samples.jdo.tutorial.Main


datanucleus-samples-jdo-tutorial-2.0 下面有build.xml   pom.xml两个文件
pom.xml
Files used by Maven2 for building and running the tutorial.

build.xml
File used by Ant for building the tutorial.
这里只是转成直接使用java 和 javac的的命令










hive用的datanucleus版本。

build.properties :
#
# Data nucleus repository - needed for jdo2-api-2.3-ec.jar download
#
datanucleus.repo=http://www.datanucleus.org/downloads/maven2


ivy/libraries.properties :
datanucleus-connectionpool.version=2.0.3
datanucleus-core.version=2.1.1
datanucleus-enhancer.version=2.1.0-release
datanucleus-rdbms.version=2.1.1


metastore/ivy.xml :
        <dependency org="org.datanucleus" name="datanucleus-connectionpool" rev="${datanucleus-connectionpool.version}">
          <exclude module="proxool" />
          <exclude module="c3p0" />
          <exclude module="datanucleus-core" />
          <exclude module="datanucleus-rdbms" />
          <exclude module="commons-collections" />
          <exclude module="commons-pool" />
          <exclude module="commons-dbcp" />
          <exclude org="com.jolbox" module="bonecp"/>
        </dependency>
        <dependency org="org.datanucleus" name="datanucleus-core" rev="${datanucleus-core.version}">
          <exclude org="javax.jdo" module="jdo2-api"/>
          <exclude org="org.apache.geronimo.specs" module="geronimo-jta_1.1_spec"/>
          <exclude org="org.eclipse.equinox" module="org.eclipse.equinox.registry"/>
          <exclude org="org.eclipse.equinox" module="org.eclipse.equinox.common"/>
          <exclude org="org.eclipse.core" module="org.eclipse.core.runtime"/>
          <exclude org="org.eclipse.osgi" module="org.eclipse.osgi"/>
          <exclude org="log4j" module="log4j"/>
        </dependency>
        <dependency org="org.datanucleus" name="datanucleus-enhancer" rev="${datanucleus-enhancer.version}">
          <exclude org="org.datanucleus" module="datanucleus-core"/>
          <exclude org="javax.jdo" module="jdo2-api"/>
          <exclude org="asm" module="asm"/>
          <exclude org="org.apache.ant" module="ant"/>
        </dependency>
        <dependency org="org.datanucleus" name="datanucleus-rdbms" rev="${datanucleus-rdbms.version}">
          <exclude org="org.datanucleus" module="datanucleus-core"/>
          <exclude org="javax.jdo" module="jdo2-api"/>
          <exclude org="org.apache.geronimo.specs" module="geronimo-jta_1.1_spec"/>
          <exclude org="org.datanucleus" module="datanucleus-thirdparty"/>
          <exclude org="org.apache.ant" module="ant"/>
          <exclude org="oracle" module="ojdbc14_g"/>
        </dependency>

http://www.jpox.org/servlet/jira/browse/NUCCORE-559?page=com.atlassian.jira.plugin.system.issuetabpanels%3Aall-tabpanel