canal使用配置详解

liyonghui160com

浏览: 777046 次
性别:
来自: 北京

最近访客更多访客>>

lingxiajiudu

jerry830518

zhutiehan

liang3yibo

博主相关

博客

微博

相册

留言

关于我

文章分类

社区版块

存档分类

博客分类：

canal

canal使用配置详解 canal配置详解

a. canal的原理是基于mysql binlog技术，所以这里一定需要开启mysql的binlog写入功能，并且配置binlog模式为row.

[mysqld]  
log-bin=mysql-bin #添加这一行就ok  
binlog-format=ROW #选择row模式  
server_id=1 #配置mysql replaction需要定义，不能和canal的slaveId重复

canal的原理是模拟自己为mysql slave，所以这里一定需要做为mysql slave的相关权限

CREATE USER canal IDENTIFIED BY 'canal';    
GRANT SELECT, REPLICATION SLAVE, REPLICATION CLIENT ON *.* TO 'canal'@'%'  IDENTIFIED BY 'canal' WITH GRANT OPTION;
FLUSH PRIVILEGES;

针对已有的账户可通过grants查询权限：

show grants for 'canal'

启动步骤：

1. 下载canal

下载部署包

wget http://canal4mysql.googlecode.com/files/canal.deployer-1.0.0.tar.gz

自己编译

git clone git@github.com:otter-projects/canal.git

cd canal;

mvn clean install -Dmaven.test.skip -Denv=release

编译完成后，会在根目录下产生target/canal.deployer-$version.tar.gz

2. 解压缩

mkdir /tmp/canal

tar zxvf canal.deployer-1.0.0.tar.gz -C /tmp/canal

解压完成后，进入/tmp/canal目录，可以看到如下结构：

drwxr-xr-x 2 jianghang jianghang 136 2013-02-05 21:51 bin

drwxr-xr-x 4 jianghang jianghang 160 2013-02-05 21:51 conf

drwxr-xr-x 2 jianghang jianghang 1.3K 2013-02-05 21:51 lib

drwxr-xr-x 2 jianghang jianghang 48 2013-02-05 21:29 logs

3. 配置修改

公用参数：

vi conf/canal.properties

#################################################

######### common argument #############

#################################################

canal.id= 1

canal.address=

canal.port= 11111

canal.zkServers=

# flush data to zk

canal.zookeeper.flush.period = 1000

## memory store RingBuffer size, should be Math.pow(2,n)

canal.instance.memory.buffer.size = 32768

## detecing config

canal.instance.detecting.enable = false

canal.instance.detecting.sql = insert into retl.xdual values(1,now()) on duplicate key update x=now()

canal.instance.detecting.interval.time = 3

canal.instance.detecting.retry.threshold = 3

canal.instance.detecting.heartbeatHaEnable = false

# support maximum transaction size, more than the size of the transaction will be cut into multiple transactions delivery

canal.instance.transactionn.size = 1024

# network config

canal.instance.network.receiveBufferSize = 16384

canal.instance.network.sendBufferSize = 16384

canal.instance.network.soTimeout = 30

#################################################

######### destinations #############

#################################################

canal.destinations= example

canal.instance.global.mode = spring

canal.instance.global.lazy = true ##修改为false，代表立马启动

#canal.instance.global.manager.address = 127.0.0.1:1099

canal.instance.global.spring.xml = classpath:spring/memory-instance.xml

#canal.instance.global.spring.xml = classpath:spring/default-instance.xml

应用参数：

vi conf/example/instance.properties

Instance.properties

#################################################

## mysql serverId

canal.instance.mysql.slaveId = 1234

# position info

canal.instance.master.address = 127.0.0.1:3306 #改成自己的数据库地址

canal.instance.master.journal.name =

canal.instance.master.position =

canal.instance.master.timestamp =

#canal.instance.standby.address =

#canal.instance.standby.journal.name =

#canal.instance.standby.position =

#canal.instance.standby.timestamp =

# username/password

canal.instance.dbUsername = retl #改成自己的数据库信息

canal.instance.dbPassword = retl #改成自己的数据库信息

canal.instance.defaultDatabaseName = #改成自己的数据库信息

canal.instance.connectionCharsetNumber = 33 #改成自己的数据库信息

canal.instance.connectionCharset = UTF-8 #改成自己的数据库信息

# table regex

canal.instance.filter.regex = .*\\..*

#################################################

说明：

canal.instance.connectionCharset 代表数据库的编码方式对应到java中的编码类型，比如UTF-8，GBK , ISO-8859-1

canal.instance.connectionCharsetNumber 代表数据库的编码方式对应mysql中的唯一id，详细的映射关系可查看：com.mysql.jdbc.CharsetMapping.INDEX_TO_CHARSET

针对常见的编码：

utf-8 <=> 33

gb2312 <=> 24

gbk <=> 28

4. 准备启动

sh bin/startup.sh

5. 查看日志

vi logs/canal/canal.log

2013-02-05 22:45:27.967 [main] INFO com.alibaba.otter.canal.deployer.CanalLauncher - ## start the canal server.

2013-02-05 22:45:28.113 [main] INFO com.alibaba.otter.canal.deployer.CanalController - ## start the canal server[10.1.29.120:11111]

2013-02-05 22:45:28.210 [main] INFO com.alibaba.otter.canal.deployer.CanalLauncher - ## the canal server is running now ......

具体instance的日志：

vi logs/example/example.log

2013-02-05 22:50:45.636 [main] INFO c.a.o.c.i.spring.support.PropertyPlaceholderConfigurer - Loading properties file from class path resource [canal.properties]

2013-02-05 22:50:45.641 [main] INFO c.a.o.c.i.spring.support.PropertyPlaceholderConfigurer - Loading properties file from class path resource [example/instance.properties]

2013-02-05 22:50:45.803 [main] INFO c.a.otter.canal.instance.spring.CanalInstanceWithSpring - start CannalInstance for 1-example

2013-02-05 22:50:45.810 [main] INFO c.a.otter.canal.instance.spring.CanalInstanceWithSpring - start successful....

6. 关闭

sh bin/stop.sh

使用中出现以下错误：Could not find first log file name in binary log index file

2015-01-15 10:40:01.756 [destination = example , address = /127.0.0.1:3306 , EventParser] ERROR com.alibaba.otter.canal.common.alarm.LogAlarmHandler - destination:example[java.io.IOException: Received error packet: errno = 1236, sqlstate = HY000 errmsg = Could not find first log file name in binary log index file
        at com.alibaba.otter.canal.parse.inbound.mysql.dbsync.DirectLogFetcher.fetch(DirectLogFetcher.java:95)
        at com.alibaba.otter.canal.parse.inbound.mysql.MysqlConnection.dump(MysqlConnection.java:116)
        at com.alibaba.otter.canal.parse.inbound.AbstractEventParser$3.run(AbstractEventParser.java:209)
        at java.lang.Thread.run(Thread.java:662)

最终解决因为没有权限，使用root用户运行成功！

分享到：

Hbase通过 Mapreduce 写入数据到Mysql | Linux系统上安装MySQL 5.5prm，rpm依赖性 ...

2015-01-15 10:42
浏览 27983
评论(2)
分类:互联网
查看更多

2 楼 liyonghui160com 2017-11-29

RAYFUXK 写道

您好，我最近也在使用Canal，但是遇到了一个问题。
ERROR com.alibaba.otter.canal.common.alarm.LogAlarmHandler - destination:example[com.alibaba.otter.canal.parse.exception.CanalParseException: java.io.IOException: connect /127.0.0.1:3306 failure:java.io.IOException: Error When doing Client Authentication:ErrorPacket [errorNumber=1045, fieldCount=-1, message=Access denied for user 'canal'@'localhost' (using password: YES), sqlState=28000, sqlStateMarker=#]
具体细节我发了issue，请问可以帮我看一下么？？？
https://github.com/alibaba/canal/issues/146

用户名密码不对吧要不就是没有权限

1 楼 RAYFUXK 2016-03-05

发表评论

您还没有登录,请您登录后再发表评论

最近访客更多访客>>

博主相关

文章分类

社区版块

存档分类

最新评论