- 浏览: 346064 次
- 性别:
- 来自: 上海
文章分类
最新评论
-
tpxcer:
不开启时可以的,而且开启以后各种坑。。。。
hue beeswax权限管理 -
yangze:
博主请教一个问题,hue 控制hive表的权限怎么弄? 怎么联 ...
cloudera新增用户权限配置 -
linux91:
楼主你好,我用CM配置LDAP用户组映射,进入impala时, ...
sentry配置 -
linux91:
版主:按你的步骤配置了,可是,执行 impala-shell ...
impala集成LDAP -
lookqlp:
super_a 写道你好!找不到表这个问题是如何解决的,可以描 ...
hcatalog读取hive数据并写入hive
升级需求
1.为支持spark kerberos安全机制
2.为满足impala trunc函数
3.为解决impala import时同时query导致impala hang问题
升级步骤
参考http://www.cloudera.com/content/cloudera/en/documentation/core/latest/topics/installation_upgrade.html
优先升级cloudera manager,再升级cdh
1.准备工作:
统一集群root密码,需要运维帮忙操作下
agent自动重启关闭
事先下载好parcals包
2.CM升级
登录cmserver安装的主机,执行命令:
cat /etc/cloudera-scm-server/db.properties
备份CM数据:
pg_dump -U scm -p 7432 > scm_server_db_backup.bak
检查/tmp下是否有文件生成,期间保证tmp下文件不要被删除。
停止CM server :
sudo service cloudera-scm-server stop
停止CM server依赖的数据库:
sudo service cloudera-scm-server-db stop
如果这台CM server上有agent在运行也停止:
sudo service cloudera-scm-agent stop
修改yum的 cloudera-manager.repo文件:
sudo vim /etc/yum.repos.d/cloudera-manager.repo
[cloudera-manager]
# Packages for Cloudera Manager, Version 5, on RedHat or CentOS 6 x86_64
name=Cloudera Manager
baseurl=http://archive.cloudera.com/cm5/redhat/6/x86_64/cm/5/
gpgkey = http://archive.cloudera.com/cm5/redhat/6/x86_64/cm/RPM-GPG-KEY-cloudera
gpgcheck = 1
安装:
sudo
yum clean all
sudo yum upgrade 'cloudera-*'
检查:
rpm -qa 'cloudera-manager-*'
启动CM server 数据库:
sudo service cloudera-scm-server-db start
启动CM server:
sudo service cloudera-scm-server start
登录http://172.20.0.83:7180/
安装agent(步骤略)
升级如果升级jdk,会改变java_home路径,导致java相关服务不可用,需要重新配置java_home
升级CM后需要重启CDH。
3.CDH升级
停止集群所有服务
备份namenode元数据:
进入namenode dir,执行:
tar -cvf /root/nn_backup_data.tar ./*
下载parcels
分发包->激活包->关闭(非重启)
开启zk服务
进入HDFS服务->升级hdfs metadata
namenode上启动元数据
启动剩余HDFS角色
namenode响应RPC
HDFS退出安全模式
备份hive metastore数据库
mysqldump -h172.20.0.67 -ucdhhive -p111111 cdhhive > /tmp/database-backup.sql
进入hive服务->更新hive metastore
database scheme
更新oozie sharelib:oozie->install
oozie share lib
创建 oozie user
sharelib
创建 oozie user
Dir
更新sqoop:进入sqoop服务->update
sqoop
更新sqoop2 server
更新spark(略,可先卸载原来版本,升级后直接安装新版本)
启动集群所有服务:zk->hdfs->spark->flume->hbase->hive->impala->oozie->sqoop2->hue
分发客户端文件:deploy client
configuration
deploy hdfs client configuration
deploy spark client configuration
deploy hbase client configuration
deploy yarn client configuration
deploy hive client configuration
删除老版本包:
sudo yum remove bigtop-utils bigtop-jsvc bigtop-tomcat hue-common sqoop2-client
启动agent:
sudo service cloudera-scm-agent restart
HDFS
metadata update
hdfs server->instance->namenode=>action->Finalize
Metadata Upgrade
升级过程遇主要问题:
com.cloudera.server.cmf.FeatureUnavailableException: The feature Navigator Audit Server is not available.
at com.cloudera.server.cmf.components.LicensedFeatureManager.check(LicensedFeatureManager.java:49)
at com.cloudera.server.cmf.components.OperationsManagerImpl.setConfig(OperationsManagerImpl.java:1312)
at com.cloudera.server.cmf.components.OperationsManagerImpl.setConfigUnsafe(OperationsManagerImpl.java:1352)
at com.cloudera.api.dao.impl.ManagerDaoBase.updateConfigs(ManagerDaoBase.java:264)
at com.cloudera.api.dao.impl.RoleConfigGroupManagerDaoImpl.updateConfigsHelper(RoleConfigGroupManagerDaoImpl.java:214)
at com.cloudera.api.dao.impl.RoleConfigGroupManagerDaoImpl.updateRoleConfigGroup(RoleConfigGroupManagerDaoImpl.java:97)
at com.cloudera.api.dao.impl.RoleConfigGroupManagerDaoImpl.updateRoleConfigGroup(RoleConfigGroupManagerDaoImpl.java:79)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at com.cloudera.api.dao.impl.ManagerDaoBase.invoke(ManagerDaoBase.java:208)
at com.sun.proxy.$Proxy82.updateRoleConfigGroup(Unknown Source)
at com.cloudera.api.v3.impl.RoleConfigGroupsResourceImpl.updateRoleConfigGroup(RoleConfigGroupsResourceImpl.java:69)
at com.cloudera.api.v3.impl.MgmtServiceResourceV3Impl$RoleConfigGroupsResourceWrapper.updateRoleConfigGroup(MgmtServiceResourceV3Impl.java:54)
at com.cloudera.cmf.service.upgrade.RemoveBetaFromRCG.upgrade(RemoveBetaFromRCG.java:80)
at com.cloudera.cmf.service.upgrade.AbstractApiAutoUpgradeHandler.upgrade(AbstractApiAutoUpgradeHandler.java:36)
at com.cloudera.cmf.service.upgrade.AutoUpgradeHandlerRegistry.performAutoUpgradesForOneVersion(AutoUpgradeHandlerRegistry.java:233)
at com.cloudera.cmf.service.upgrade.AutoUpgradeHandlerRegistry.performAutoUpgrades(AutoUpgradeHandlerRegistry.java:167)
at com.cloudera.cmf.service.upgrade.AutoUpgradeHandlerRegistry.performAutoUpgrades(AutoUpgradeHandlerRegistry.java:138)
at com.cloudera.server.cmf.Main.run(Main.java:587)
at com.cloudera.server.cmf.Main.main(Main.java:198)
2014-11-26 03:17:42,891 INFO ParcelUpdateService:com.cloudera.parcel.components.ParcelDownloade
原先版本使用了60天试用企业版本,该期限已经过期,升级时Navigator服务启动不了,导致整个cloduera manager server启动失败
升级后问题
a.升级后flume原先提供的第三方jar丢失,需要将包重新放在/opt....下
b.sqoop导入mysql的驱动包找不到,需要将包重新放在/opt....下
c.hbase服务异常
Unhandled exception. Starting shutdown.
org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.authorize.AuthorizationException): User hbase/ip-10-1-33-20.ec2.internal@YEAHMOBI.COM (auth:KERBEROS) is not authorized for protocol interface org.apache.hadoop.hdfs.protocol.ClientProtocol, expected client Kerberos principal is null
at org.apache.hadoop.ipc.Client.call(Client.java:1409)
at org.apache.hadoop.ipc.Client.call(Client.java:1362)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206)
at com.sun.proxy.$Proxy15.setSafeMode(Unknown Source)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:186)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
at com.sun.proxy.$Proxy15.setSafeMode(Unknown Source)
at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.setSafeMode(ClientNamenodeProtocolTranslatorPB.java:594)
at org.apache.hadoop.hdfs.DFSClient.setSafeMode(DFSClient.java:2224)
at org.apache.hadoop.hdfs.DistributedFileSystem.setSafeMode(DistributedFileSystem.java:993)
at org.apache.hadoop.hdfs.DistributedFileSystem.setSafeMode(DistributedFileSystem.java:977)
at org.apache.hadoop.hbase.util.FSUtils.isInSafeMode(FSUtils.java:432)
at org.apache.hadoop.hbase.util.FSUtils.waitOnSafeMode(FSUtils.java:851)
at org.apache.hadoop.hbase.master.MasterFileSystem.checkRootDir(MasterFileSystem.java:435)
at org.apache.hadoop.hbase.master.MasterFileSystem.createInitialFileSystemLayout(MasterFileSystem.java:146)
at org.apache.hadoop.hbase.master.MasterFileSystem.<init>(MasterFileSystem.java:127)
at org.apache.hadoop.hbase.master.HMaster.finishInitialization(HMaster.java:789)
at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:606)
at java.lang.Thread.run(Thread.java:744)
通过cm将safe配置文件里的hbase.rpc.engine org.apache.hadoop.hbase.ipc.SecureRpcEngine去掉后重启成功。
后来发现是cm server的问题,之前修改了一个hostname,cloudera manager server未重启,重启后,加入该配置重启hbase不会有问题。
d.service monitor,zookeeper也有警告,其他服务都有部分红色异常
Exception in scheduled runnable.
java.lang.IllegalStateException
at com.google.common.base.Preconditions.checkState(Preconditions.java:133)
at com.cloudera.cmon.firehose.polling.CdhTask.checkClientConfigs(CdhTask.java:712)
at com.cloudera.cmon.firehose.polling.CdhTask.updateCacheIfNeeded(CdhTask.java:675)
at com.cloudera.cmon.firehose.polling.FirehoseServicesPoller.getDescriptorAndHandleChanges(FirehoseServicesPoller.java:615)
at com.cloudera.cmon.firehose.polling.FirehoseServicesPoller.run(FirehoseServicesPoller.java:179)
at com.cloudera.enterprise.PeriodicEnterpriseService$UnexceptionablePeriodicRunnable.run(PeriodicEnterpriseService.java:67)
at java.lang.Thread.run(Thread.java:745)
后来发现是cm server的问题,之前修改了一个hostname,cloudera manager server未重启,重启后,加入该配置重启hbase不会有问题。
e.mapreduce访问安全机制下的hbase失败
去除client hbase-site safe配置文件内容:hbase.rpc.protection privacy,旧版本中必须加此配置,而新版本文档中也提到需要加此配置,但经过测试加此配置后报如上异常。
14/11/27 12:38:26 INFO zookeeper.ClientCnxn: Socket connection established to ip-10-1-33-24.ec2.internal/10.1.33.24:2181, initiating session
14/11/27 12:38:26 INFO zookeeper.ClientCnxn: Session establishment complete on server ip-10-1-33-24.ec2.internal/10.1.33.24:2181, sessionid = 0x549ef6088f20309, negotiated timeout = 60000
14/11/27 12:38:41 WARN ipc.RpcClient: Couldn't setup connection for hbase/ip-10-1-10-15.ec2.internal@YEAHMOBI.COM to hbase/ip-10-1-34-31.ec2.internal@YEAHMOBI.COM
14/11/27 12:38:55 WARN ipc.RpcClient: Couldn't setup connection for hbase/ip-10-1-10-15.ec2.internal@YEAHMOBI.COM to hbase/ip-10-1-34-31.ec2.internal@YEAHMOBI.COM
14/11/27 12:39:15 WARN ipc.RpcClient: Couldn't setup connection for hbase/ip-10-1-10-15.ec2.internal@YEAHMOBI.COM to hbase/ip-10-1-34-31.ec2.internal@YEAHMOBI.COM
14/11/27 12:39:34 WARN ipc.RpcClient: Couldn't setup connection for hbase/ip-10-1-10-15.ec2.internal@YEAHMOBI.COM to hbase/ip-10-1-34-31.ec2.internal@YEAHMOBI.COM
14/11/27 12:39:55 WARN ipc.RpcClient: Couldn't setup connection for hbase/ip-10-1-10-15.ec2.internal@YEAHMOBI.COM to hbase/ip-10-1-34-31.ec2.internal@YEAHMOBI.COM
14/11/27 12:40:19 WARN ipc.RpcClient: Couldn't setup connection for hbase/ip-10-1-10-15.ec2.internal@YEAHMOBI.COM to hbase/ip-10-1-34-31.ec2.internal@YEAHMOBI.COM
14/11/27 12:40:36 WARN ipc.RpcClient: Couldn't setup connection for hbase/ip-10-1-10-15.ec2.internal@YEAHMOBI.COM to hbase/ip-10-1-34-31.ec2.internal@YEAHMOBI.COM
Caused by: java.io.IOException: Couldn't setup connection for hbase/ip-10-1-33-20.ec2.internal@YEAHMOBI.COM to hbase/ip-10-1-34-32.ec2.internal@YEAHMOBI.COM
at org.apache.hadoop.hbase.ipc.RpcClient$Connection$1.run(RpcClient.java:821)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1614)
at org.apache.hadoop.hbase.ipc.RpcClient$Connection.handleSaslConnectionFailure(RpcClient.java:796)
at org.apache.hadoop.hbase.ipc.RpcClient$Connection.setupIOstreams(RpcClient.java:898)
at org.apache.hadoop.hbase.ipc.RpcClient.getConnection(RpcClient.java:1543)
at org.apache.hadoop.hbase.ipc.RpcClient.call(RpcClient.java:1442)
at org.apache.hadoop.hbase.ipc.RpcClient.callBlockingMethod(RpcClient.java:1661)
at org.apache.hadoop.hbase.ipc.RpcClient$BlockingRpcChannelImplementation.callBlockingMethod(RpcClient.java:1719)
at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$BlockingStub.execService(ClientProtos.java:30014)
at org.apache.hadoop.hbase.protobuf.ProtobufUtil.execService(ProtobufUtil.java:1623)
at org.apache.hadoop.hbase.ipc.RegionCoprocessorRpcChannel$1.call(RegionCoprocessorRpcChannel.java:93)
at org.apache.hadoop.hbase.ipc.RegionCoprocessorRpcChannel$1.call(RegionCoprocessorRpcChannel.java:90)
at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:114)
... 31 more
Caused by: javax.security.sasl.SaslException: No common protection layer between client and server
at com.sun.security.sasl.gsskerb.GssKrb5Client.doFinalHandshake(GssKrb5Client.java:252)
at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:187)
at org.apache.hadoop.hbase.security.HBaseSaslRpcClient.saslConnect(HBaseSaslRpcClient.java:210)
at org.apache.hadoop.hbase.ipc.RpcClient$Connection.setupSaslConnection(RpcClient.java:770)
at org.apache.hadoop.hbase.ipc.RpcClient$Connection.access$600(RpcClient.java:357)
at org.apache.hadoop.hbase.ipc.RpcClient$Connection$2.run(RpcClient.java:891)
at org.apache.hadoop.hbase.ipc.RpcClient$Connection$2.run(RpcClient.java:888)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1614)
at org.apache.hadoop.hbase.ipc.RpcClient$Connection.setupIOstreams(RpcClient.java:888)
... 40 more
<property>
<name>hbase.rpc.engine</name>
<value>org.apache.hadoop.hbase.ipc.SecureRpcEngine</value>
</property>
mr中使用http://www.cloudera.com/content/cloudera/en/documentation/cdh5/v5-0-0/CDH5-Installation-Guide/cdh5ig_mapreduce_hbase.html TableMapReduceUtil.addDependencyJars(job);方式加载。
并且使用user api加入例如:
hbase.master.kerberos.principal=hbase/ip-10-1-10-15.ec2.internal@YEAHMOBI.COM
hbase.keytab.path=/home/dev/1015q.keytab
f.升级后impala jdbc安全机制下不可用
java.sql.SQLException: Could not open connection to jdbc:hive2://ip-10-1-33-22.ec2.internal:21050/ym_system;principal=impala/ip-10-1-33-22.ec2.internal@YEAHMOBI.COM: GSS initiate failed
at org.apache.hive.jdbc.HiveConnection.openTransport(HiveConnection.java:187)
at org.apache.hive.jdbc.HiveConnection.<init>(HiveConnection.java:164)
at org.apache.hive.jdbc.HiveDriver.connect(HiveDriver.java:105)
at java.sql.DriverManager.getConnection(DriverManager.java:571)
at java.sql.DriverManager.getConnection(DriverManager.java:233)
at com.cloudera.example.ClouderaImpalaJdbcExample.main(ClouderaImpalaJdbcExample.java:37)
Caused by: org.apache.thrift.transport.TTransportException: GSS initiate failed
at org.apache.thrift.transport.TSaslTransport.sendAndThrowMessage(TSaslTransport.java:221)
at org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:297)
at org.apache.thrift.transport.TSaslClientTransport.open(TSaslClientTransport.java:37)
at org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport$1.run(TUGIAssumingTransport.java:52)
at org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport$1.run(TUGIAssumingTransport.java:49)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
at org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport.open(TUGIAssumingTransport.java:49)
at org.apache.hive.jdbc.HiveConnection.openTransport(HiveConnection.java:185)
... 5 more
解决:
hadoop-auth-2.5.0-cdh5.2.0.jar
hive-shims-common-secure-0.13.1-cdh5.2.0.jar
两个包回退版本即可
1.为支持spark kerberos安全机制
2.为满足impala trunc函数
3.为解决impala import时同时query导致impala hang问题
升级步骤
参考http://www.cloudera.com/content/cloudera/en/documentation/core/latest/topics/installation_upgrade.html
优先升级cloudera manager,再升级cdh
1.准备工作:
统一集群root密码,需要运维帮忙操作下
agent自动重启关闭
事先下载好parcals包
2.CM升级
登录cmserver安装的主机,执行命令:
cat /etc/cloudera-scm-server/db.properties
备份CM数据:
pg_dump -U scm -p 7432 > scm_server_db_backup.bak
检查/tmp下是否有文件生成,期间保证tmp下文件不要被删除。
停止CM server :
sudo service cloudera-scm-server stop
停止CM server依赖的数据库:
sudo service cloudera-scm-server-db stop
如果这台CM server上有agent在运行也停止:
sudo service cloudera-scm-agent stop
修改yum的 cloudera-manager.repo文件:
sudo vim /etc/yum.repos.d/cloudera-manager.repo
[cloudera-manager]
# Packages for Cloudera Manager, Version 5, on RedHat or CentOS 6 x86_64
name=Cloudera Manager
baseurl=http://archive.cloudera.com/cm5/redhat/6/x86_64/cm/5/
gpgkey = http://archive.cloudera.com/cm5/redhat/6/x86_64/cm/RPM-GPG-KEY-cloudera
gpgcheck = 1
安装:
sudo
yum clean all
sudo yum upgrade 'cloudera-*'
检查:
rpm -qa 'cloudera-manager-*'
启动CM server 数据库:
sudo service cloudera-scm-server-db start
启动CM server:
sudo service cloudera-scm-server start
登录http://172.20.0.83:7180/
安装agent(步骤略)
升级如果升级jdk,会改变java_home路径,导致java相关服务不可用,需要重新配置java_home
升级CM后需要重启CDH。
3.CDH升级
停止集群所有服务
备份namenode元数据:
进入namenode dir,执行:
tar -cvf /root/nn_backup_data.tar ./*
下载parcels
分发包->激活包->关闭(非重启)
开启zk服务
进入HDFS服务->升级hdfs metadata
namenode上启动元数据
启动剩余HDFS角色
namenode响应RPC
HDFS退出安全模式
备份hive metastore数据库
mysqldump -h172.20.0.67 -ucdhhive -p111111 cdhhive > /tmp/database-backup.sql
进入hive服务->更新hive metastore
database scheme
更新oozie sharelib:oozie->install
oozie share lib
创建 oozie user
sharelib
创建 oozie user
Dir
更新sqoop:进入sqoop服务->update
sqoop
更新sqoop2 server
更新spark(略,可先卸载原来版本,升级后直接安装新版本)
启动集群所有服务:zk->hdfs->spark->flume->hbase->hive->impala->oozie->sqoop2->hue
分发客户端文件:deploy client
configuration
deploy hdfs client configuration
deploy spark client configuration
deploy hbase client configuration
deploy yarn client configuration
deploy hive client configuration
删除老版本包:
sudo yum remove bigtop-utils bigtop-jsvc bigtop-tomcat hue-common sqoop2-client
启动agent:
sudo service cloudera-scm-agent restart
HDFS
metadata update
hdfs server->instance->namenode=>action->Finalize
Metadata Upgrade
升级过程遇主要问题:
com.cloudera.server.cmf.FeatureUnavailableException: The feature Navigator Audit Server is not available.
at com.cloudera.server.cmf.components.LicensedFeatureManager.check(LicensedFeatureManager.java:49)
at com.cloudera.server.cmf.components.OperationsManagerImpl.setConfig(OperationsManagerImpl.java:1312)
at com.cloudera.server.cmf.components.OperationsManagerImpl.setConfigUnsafe(OperationsManagerImpl.java:1352)
at com.cloudera.api.dao.impl.ManagerDaoBase.updateConfigs(ManagerDaoBase.java:264)
at com.cloudera.api.dao.impl.RoleConfigGroupManagerDaoImpl.updateConfigsHelper(RoleConfigGroupManagerDaoImpl.java:214)
at com.cloudera.api.dao.impl.RoleConfigGroupManagerDaoImpl.updateRoleConfigGroup(RoleConfigGroupManagerDaoImpl.java:97)
at com.cloudera.api.dao.impl.RoleConfigGroupManagerDaoImpl.updateRoleConfigGroup(RoleConfigGroupManagerDaoImpl.java:79)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at com.cloudera.api.dao.impl.ManagerDaoBase.invoke(ManagerDaoBase.java:208)
at com.sun.proxy.$Proxy82.updateRoleConfigGroup(Unknown Source)
at com.cloudera.api.v3.impl.RoleConfigGroupsResourceImpl.updateRoleConfigGroup(RoleConfigGroupsResourceImpl.java:69)
at com.cloudera.api.v3.impl.MgmtServiceResourceV3Impl$RoleConfigGroupsResourceWrapper.updateRoleConfigGroup(MgmtServiceResourceV3Impl.java:54)
at com.cloudera.cmf.service.upgrade.RemoveBetaFromRCG.upgrade(RemoveBetaFromRCG.java:80)
at com.cloudera.cmf.service.upgrade.AbstractApiAutoUpgradeHandler.upgrade(AbstractApiAutoUpgradeHandler.java:36)
at com.cloudera.cmf.service.upgrade.AutoUpgradeHandlerRegistry.performAutoUpgradesForOneVersion(AutoUpgradeHandlerRegistry.java:233)
at com.cloudera.cmf.service.upgrade.AutoUpgradeHandlerRegistry.performAutoUpgrades(AutoUpgradeHandlerRegistry.java:167)
at com.cloudera.cmf.service.upgrade.AutoUpgradeHandlerRegistry.performAutoUpgrades(AutoUpgradeHandlerRegistry.java:138)
at com.cloudera.server.cmf.Main.run(Main.java:587)
at com.cloudera.server.cmf.Main.main(Main.java:198)
2014-11-26 03:17:42,891 INFO ParcelUpdateService:com.cloudera.parcel.components.ParcelDownloade
原先版本使用了60天试用企业版本,该期限已经过期,升级时Navigator服务启动不了,导致整个cloduera manager server启动失败
升级后问题
a.升级后flume原先提供的第三方jar丢失,需要将包重新放在/opt....下
b.sqoop导入mysql的驱动包找不到,需要将包重新放在/opt....下
c.hbase服务异常
Unhandled exception. Starting shutdown.
org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.authorize.AuthorizationException): User hbase/ip-10-1-33-20.ec2.internal@YEAHMOBI.COM (auth:KERBEROS) is not authorized for protocol interface org.apache.hadoop.hdfs.protocol.ClientProtocol, expected client Kerberos principal is null
at org.apache.hadoop.ipc.Client.call(Client.java:1409)
at org.apache.hadoop.ipc.Client.call(Client.java:1362)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206)
at com.sun.proxy.$Proxy15.setSafeMode(Unknown Source)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:186)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
at com.sun.proxy.$Proxy15.setSafeMode(Unknown Source)
at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.setSafeMode(ClientNamenodeProtocolTranslatorPB.java:594)
at org.apache.hadoop.hdfs.DFSClient.setSafeMode(DFSClient.java:2224)
at org.apache.hadoop.hdfs.DistributedFileSystem.setSafeMode(DistributedFileSystem.java:993)
at org.apache.hadoop.hdfs.DistributedFileSystem.setSafeMode(DistributedFileSystem.java:977)
at org.apache.hadoop.hbase.util.FSUtils.isInSafeMode(FSUtils.java:432)
at org.apache.hadoop.hbase.util.FSUtils.waitOnSafeMode(FSUtils.java:851)
at org.apache.hadoop.hbase.master.MasterFileSystem.checkRootDir(MasterFileSystem.java:435)
at org.apache.hadoop.hbase.master.MasterFileSystem.createInitialFileSystemLayout(MasterFileSystem.java:146)
at org.apache.hadoop.hbase.master.MasterFileSystem.<init>(MasterFileSystem.java:127)
at org.apache.hadoop.hbase.master.HMaster.finishInitialization(HMaster.java:789)
at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:606)
at java.lang.Thread.run(Thread.java:744)
通过cm将safe配置文件里的hbase.rpc.engine org.apache.hadoop.hbase.ipc.SecureRpcEngine去掉后重启成功。
后来发现是cm server的问题,之前修改了一个hostname,cloudera manager server未重启,重启后,加入该配置重启hbase不会有问题。
d.service monitor,zookeeper也有警告,其他服务都有部分红色异常
Exception in scheduled runnable.
java.lang.IllegalStateException
at com.google.common.base.Preconditions.checkState(Preconditions.java:133)
at com.cloudera.cmon.firehose.polling.CdhTask.checkClientConfigs(CdhTask.java:712)
at com.cloudera.cmon.firehose.polling.CdhTask.updateCacheIfNeeded(CdhTask.java:675)
at com.cloudera.cmon.firehose.polling.FirehoseServicesPoller.getDescriptorAndHandleChanges(FirehoseServicesPoller.java:615)
at com.cloudera.cmon.firehose.polling.FirehoseServicesPoller.run(FirehoseServicesPoller.java:179)
at com.cloudera.enterprise.PeriodicEnterpriseService$UnexceptionablePeriodicRunnable.run(PeriodicEnterpriseService.java:67)
at java.lang.Thread.run(Thread.java:745)
后来发现是cm server的问题,之前修改了一个hostname,cloudera manager server未重启,重启后,加入该配置重启hbase不会有问题。
e.mapreduce访问安全机制下的hbase失败
去除client hbase-site safe配置文件内容:hbase.rpc.protection privacy,旧版本中必须加此配置,而新版本文档中也提到需要加此配置,但经过测试加此配置后报如上异常。
14/11/27 12:38:26 INFO zookeeper.ClientCnxn: Socket connection established to ip-10-1-33-24.ec2.internal/10.1.33.24:2181, initiating session
14/11/27 12:38:26 INFO zookeeper.ClientCnxn: Session establishment complete on server ip-10-1-33-24.ec2.internal/10.1.33.24:2181, sessionid = 0x549ef6088f20309, negotiated timeout = 60000
14/11/27 12:38:41 WARN ipc.RpcClient: Couldn't setup connection for hbase/ip-10-1-10-15.ec2.internal@YEAHMOBI.COM to hbase/ip-10-1-34-31.ec2.internal@YEAHMOBI.COM
14/11/27 12:38:55 WARN ipc.RpcClient: Couldn't setup connection for hbase/ip-10-1-10-15.ec2.internal@YEAHMOBI.COM to hbase/ip-10-1-34-31.ec2.internal@YEAHMOBI.COM
14/11/27 12:39:15 WARN ipc.RpcClient: Couldn't setup connection for hbase/ip-10-1-10-15.ec2.internal@YEAHMOBI.COM to hbase/ip-10-1-34-31.ec2.internal@YEAHMOBI.COM
14/11/27 12:39:34 WARN ipc.RpcClient: Couldn't setup connection for hbase/ip-10-1-10-15.ec2.internal@YEAHMOBI.COM to hbase/ip-10-1-34-31.ec2.internal@YEAHMOBI.COM
14/11/27 12:39:55 WARN ipc.RpcClient: Couldn't setup connection for hbase/ip-10-1-10-15.ec2.internal@YEAHMOBI.COM to hbase/ip-10-1-34-31.ec2.internal@YEAHMOBI.COM
14/11/27 12:40:19 WARN ipc.RpcClient: Couldn't setup connection for hbase/ip-10-1-10-15.ec2.internal@YEAHMOBI.COM to hbase/ip-10-1-34-31.ec2.internal@YEAHMOBI.COM
14/11/27 12:40:36 WARN ipc.RpcClient: Couldn't setup connection for hbase/ip-10-1-10-15.ec2.internal@YEAHMOBI.COM to hbase/ip-10-1-34-31.ec2.internal@YEAHMOBI.COM
Caused by: java.io.IOException: Couldn't setup connection for hbase/ip-10-1-33-20.ec2.internal@YEAHMOBI.COM to hbase/ip-10-1-34-32.ec2.internal@YEAHMOBI.COM
at org.apache.hadoop.hbase.ipc.RpcClient$Connection$1.run(RpcClient.java:821)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1614)
at org.apache.hadoop.hbase.ipc.RpcClient$Connection.handleSaslConnectionFailure(RpcClient.java:796)
at org.apache.hadoop.hbase.ipc.RpcClient$Connection.setupIOstreams(RpcClient.java:898)
at org.apache.hadoop.hbase.ipc.RpcClient.getConnection(RpcClient.java:1543)
at org.apache.hadoop.hbase.ipc.RpcClient.call(RpcClient.java:1442)
at org.apache.hadoop.hbase.ipc.RpcClient.callBlockingMethod(RpcClient.java:1661)
at org.apache.hadoop.hbase.ipc.RpcClient$BlockingRpcChannelImplementation.callBlockingMethod(RpcClient.java:1719)
at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$BlockingStub.execService(ClientProtos.java:30014)
at org.apache.hadoop.hbase.protobuf.ProtobufUtil.execService(ProtobufUtil.java:1623)
at org.apache.hadoop.hbase.ipc.RegionCoprocessorRpcChannel$1.call(RegionCoprocessorRpcChannel.java:93)
at org.apache.hadoop.hbase.ipc.RegionCoprocessorRpcChannel$1.call(RegionCoprocessorRpcChannel.java:90)
at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:114)
... 31 more
Caused by: javax.security.sasl.SaslException: No common protection layer between client and server
at com.sun.security.sasl.gsskerb.GssKrb5Client.doFinalHandshake(GssKrb5Client.java:252)
at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:187)
at org.apache.hadoop.hbase.security.HBaseSaslRpcClient.saslConnect(HBaseSaslRpcClient.java:210)
at org.apache.hadoop.hbase.ipc.RpcClient$Connection.setupSaslConnection(RpcClient.java:770)
at org.apache.hadoop.hbase.ipc.RpcClient$Connection.access$600(RpcClient.java:357)
at org.apache.hadoop.hbase.ipc.RpcClient$Connection$2.run(RpcClient.java:891)
at org.apache.hadoop.hbase.ipc.RpcClient$Connection$2.run(RpcClient.java:888)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1614)
at org.apache.hadoop.hbase.ipc.RpcClient$Connection.setupIOstreams(RpcClient.java:888)
... 40 more
<property>
<name>hbase.rpc.engine</name>
<value>org.apache.hadoop.hbase.ipc.SecureRpcEngine</value>
</property>
mr中使用http://www.cloudera.com/content/cloudera/en/documentation/cdh5/v5-0-0/CDH5-Installation-Guide/cdh5ig_mapreduce_hbase.html TableMapReduceUtil.addDependencyJars(job);方式加载。
并且使用user api加入例如:
hbase.master.kerberos.principal=hbase/ip-10-1-10-15.ec2.internal@YEAHMOBI.COM
hbase.keytab.path=/home/dev/1015q.keytab
f.升级后impala jdbc安全机制下不可用
java.sql.SQLException: Could not open connection to jdbc:hive2://ip-10-1-33-22.ec2.internal:21050/ym_system;principal=impala/ip-10-1-33-22.ec2.internal@YEAHMOBI.COM: GSS initiate failed
at org.apache.hive.jdbc.HiveConnection.openTransport(HiveConnection.java:187)
at org.apache.hive.jdbc.HiveConnection.<init>(HiveConnection.java:164)
at org.apache.hive.jdbc.HiveDriver.connect(HiveDriver.java:105)
at java.sql.DriverManager.getConnection(DriverManager.java:571)
at java.sql.DriverManager.getConnection(DriverManager.java:233)
at com.cloudera.example.ClouderaImpalaJdbcExample.main(ClouderaImpalaJdbcExample.java:37)
Caused by: org.apache.thrift.transport.TTransportException: GSS initiate failed
at org.apache.thrift.transport.TSaslTransport.sendAndThrowMessage(TSaslTransport.java:221)
at org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:297)
at org.apache.thrift.transport.TSaslClientTransport.open(TSaslClientTransport.java:37)
at org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport$1.run(TUGIAssumingTransport.java:52)
at org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport$1.run(TUGIAssumingTransport.java:49)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
at org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport.open(TUGIAssumingTransport.java:49)
at org.apache.hive.jdbc.HiveConnection.openTransport(HiveConnection.java:185)
... 5 more
解决:
hadoop-auth-2.5.0-cdh5.2.0.jar
hive-shims-common-secure-0.13.1-cdh5.2.0.jar
两个包回退版本即可
发表评论
-
hive相关元数据迁移(mysql)
2015-11-18 18:27 2569mysqldump -hhost -uroot -ppassw ... -
hive dynamic partitions insert java.lang.OutOfMemoryError: Java heap space
2015-10-26 18:03 3095动态分区问题,如果数据量大或者当动态分区大甚至只有十几个时 ... -
yarn NullPointerException
2015-03-26 17:03 1435yarn重启后,部分nm启动不了,报空指针问题 20 ... -
mapreduce mapper access security hbase
2015-03-17 14:42 1220环境: security cdh 5.2.0 secu ... -
sentry服务后,几个权限问题
2015-03-10 16:08 8974以账户bi为例 问题一:账户bi beeline ldap后 ... -
cloudera新增用户权限配置
2015-03-05 16:13 3958目标: 给各个业务组提供不同用户及用户组,并有限制的访问h ... -
hive集成LDAP
2015-02-13 10:09 9747cloudera manager hive- sevice ... -
impala HA
2014-12-11 17:36 1880目的: 为impala jdbc提供统一的接口,作用参照htt ... -
impala集成LDAP
2014-12-11 12:55 7112目的: 为解决kerberos安 ... -
security cdh mapreduce access hbase
2014-12-02 15:09 1086执行mapreduce的用户必须是可以访问hdfs相应目录和执 ... -
hive gateway(client) configuration
2014-12-02 14:32 8091配置hive gateway机器 Caused by: Met ... -
hcatalog读取hive数据并写入hive
2014-12-01 17:49 18121参考http://www.cloudera.com/cont ...
相关推荐
CDH 5.0.2 是 Cloudera 的一个关键版本,它是一个全面的数据平台,专为大数据处理和分析设计。这个版本包含了多个组件,如 Hadoop、HBase、Spark、Impala 和 Hive 等,这些组件都是企业级大数据环境中的核心工具。...
本文将详细介绍如何在CDH集群环境下将ZooKeeper从3.4.5版本升级至3.5.5版本的过程及注意事项。 #### 二、准备工作 1. **下载ZooKeeper-3.5.5** - 使用命令`cd /opt/software/zookeeper`切换到软件安装目录。 - ...
为了满足更高的性能需求及功能扩展,本文将详细介绍如何将Spark升级至3.3.1版本的过程。此次升级的主要原因是团队对Hive进行了版本升级,从较低版本升级到了3.1.3。然而,在升级过程中发现并没有与之匹配的spark-...
以上知识点详细解读了CDH集群从5.8.0升级至5.10.0版本的整个过程,包括升级前的准备工作、CMServer和CMAgent的升级方法,以及CDH软件升级的详细步骤和注意事项。这份文档对于从事Hadoop集群维护和管理的专业人士具有...
- **目标版本**:Cloudera Manager (CM) 由5.7.1升级至5.14.2;CDH版本相应升级。 - **操作系统**:Linux CentOS 6.5 - **升级方式**:主要采用Package升级方式。 #### 准备工作 在正式开始升级之前,确保以下准备...
### CDH跨版本升级知识点详解 #### 一、CDH跨版本升级概述 Cloudera Distribution Including Apache Hadoop (CDH) 是一个基于Apache Hadoop的企业级大数据平台,由Cloudera公司提供支持与维护。在企业环境中,随着...
#### 五、CM和CDH 5.0.2环境准备工作 1. **配置主机信息** - 查看目标服务器的IP配置。 - 更新 `/etc/hosts` 文件以确保所有节点间的正确解析。 2. **关闭防火墙** - 使用 `service iptables stop` 和 `...
标题 "native(hadoop-2.5.0-cdh5.2.0).tar" 提供的是一个针对Hadoop 2.5.0 CDH5.2.0版本的本地库文件集合,它已经过编译,可以用于在特定操作系统上支持Hadoop运行。CDH(Cloudera Distribution Including Apache ...
在升级至 CDH 5.11.2 版本的过程中,主要目标是利用最新版本的功能改进和性能提升来提高现有 Hadoop 集群的稳定性和效率。CDH 5.11.2 相对于早期版本,在安全性、性能优化、易用性等方面都有显著增强。 #### 三、...
Spark3.2.2是Apache Spark的一个重要版本,它提供了许多新特性和性能优化,而适配CDH6.3.2则意味着该版本的Spark已经经过了与Cloudera Data Hub (CDH) 6.3.2的兼容性验证。CDH是一个流行的Hadoop发行版,包含了...
4. 若是由 CDH 5.3 升级至 CDH 5.4,则确保环境使用的 JDK 版本是 1.7 5. 如果需要安装的 CDH 版本高于 CM 版本,则建议先升级 Cloudera Manager,然后在升级 CDH 6. 用较老版本的 Cloudera Manager 安装新版的 CDH ...
内容概要:由于cdh6.3.2的spark版本为2.4.0,并且spark-sql被阉割,现基于cdh6.3.2,scala2.12.0,java1.8,maven3.6.3,,对spark-3.2.2源码进行编译 应用:该资源可用于cdh6.3.2集群配置spark客户端,用于spark-sql
2. **JDK 7 Update 80**:Java是运行Hadoop和CDH的基础,因此确保安装正确的JDK版本至关重要。可以从[Oracle官网](http://download.oracle.com/otn/java/jdk/7u80-b15/jdk-7u80-linux-x64.rpm)下载对应的RPM包进行...
### CDH 6.3.2 升级 Hive 至 3.x 后登录 HiveServer2 连接失败问题分析及解决方法 #### 一、问题背景与现象 在将 Cloudera Data Hub (CDH) 6.3.2 版本中的 Hive 升级到 3.x 版本后,用户在尝试通过 JDBC 连接到 ...
### CDH-5.XX离线安装与升级详解 #### 一、概述 随着大数据技术的发展,Apache Hadoop已经成为处理大规模数据集的核心工具之一。基于Hadoop基础框架,Cloudera提供了自己的商业发行版——Cloudera Distribution ...
CDH6.3.2完整安装包网盘下载,包含 CDH-6.3.2-1.cdh6.3.2.p0.1605554-bionic.parcel、CDH-6.3.2-1.cdh6.3.2.p0.1605554-bionic.parcel.sha1、CDH-6.3.2-1.cdh6.3.2.p0.1605554-bionic.parcel.sha256、CDH-6.3.2-1....
随着CDH社区的持续活跃,CDH版本不断更新,为了使用版本新特性,本次将进行CM和CDH的版本升级,本次升级主要采用package升级,linux系统centos6.5,升级前CLOUDERA ... CDH升级主要分为两部分1.CM的升级、2.CDH的升级
cdh7.1.7包括: CDH-7.1.7-1.cdh7.1.7.p0.15945976-el7.parcel CDH-7.1.7-1.cdh7.1.7.p0.15945976-el7.parcel.sha1 CDH-7.1.7-1.cdh7.1.7.p0.15945976-el7.parcel.sha256 manifest.json cm7.4.7包括: cloudera-...
CDP是最新的一代大数据平台架构,提供了许多优势,例如: cloud-native架构、自动化的安装和升级、实时的安全监控和安全问题解决、手动解决问题、24/7支持等。 大数据平台架构的演进对企业级解决方案的影响是深远的...
在CDH 5.0.2版本中,提供了HBase、Solr和HBase Indexer的集成包,包括Hbase-0.96.1.1-cdh5.0.2.tar.gz、Hbase-solr-1.3-cdh5.0.2.tar.gz和Solr-4.4.0-cdh5.0.2.tar.gz。 HBase Indexer的工作原理基于HBase的...