canal admin管理端配置-多个实例的创建（二）

下载安装

下载地址：
在这里插入图片描述
下载解压即可

配置

修改canal.admin-1.1.5\conf\application.yml

server:
  port: 8089 #端口根据是否冲突修改
spring:
  jackson:
    date-format: yyyy-MM-dd HH:mm:ss
    time-zone: GMT+8

spring.datasource:
  address: 192.0.16.12:3306#数据库ip和端口
  database: canal_manager #默认会创建的数据库
  username: canal
  password: canal
  driver-class-name: com.mysql.cj.jdbc.Driver #原配置是com.mysql.jdbc.Driver，但是Mysql 8的驱动类名是com.mysql.cj.jdbc.Driver，因此需要替换lib包下的msyql驱动包
  url: jdbc:mysql://${spring.datasource.address}/${spring.datasource.database}?useUnicode=true&characterEncoding=UTF-8&useSSL=false&allowPublicKeyRetrieval=true&serverTimezone=Asia/Shanghai #增加Mysql8配置serverTimezonee=Asia/Shanghai
  hikari:
    maximum-pool-size: 30
    minimum-idle: 1

canal:
  adminUser: admin
  adminPasswd: admin

替换lib包下的msyql驱动包

本地maven仓库找到mysql驱动jar，并替换掉lib目录下的mysql-connector-java-5.1.48.jar
在这里插入图片描述

执行conf/canal_manager.sql

会生成用于管理Canal的相关表
在这里插入图片描述

创建canal用户

#如果已有用户则不需要执行下面第一条语句
CREATE USER 'canal'@'%' IDENTIFIED BY 'canal';
# 赋予canal用户对于canal_manager库的所有操作权限
GRANT ALL PRIVILEGES ON canal_manager.* TO 'canal'@'%';
#刷新权限
FLUSH PRIVILEGES;

启动admin

进入bin目录，执行启动命令

./startup.sh

进入log目录查看日志信息，如下就是启动成功

2023-03-05 11:11:14.167 [main] INFO  o.s.w.s.m.m.annotation.ExceptionHandlerExceptionResolver - Detected @ExceptionHandler methods in customExceptionHandler
2023-03-05 11:11:14.207 [main] INFO  o.s.b.a.web.servlet.WelcomePageHandlerMapping - Adding welcome page: class path resource [public/index.html]
2023-03-05 11:11:14.489 [main] INFO  o.s.jmx.export.annotation.AnnotationMBeanExporter - Registering beans for JMX exposure on startup
2023-03-05 11:11:14.493 [main] INFO  o.s.jmx.export.annotation.AnnotationMBeanExporter - Bean with name 'dataSource' has been autodetected for JMX exposure
2023-03-05 11:11:14.498 [main] INFO  o.s.jmx.export.annotation.AnnotationMBeanExporter - Located MBean 'dataSource': registering with JMX server as MBean [com.zaxxer.hikari:name=dataSource,type=HikariDataSource]
2023-03-05 11:11:14.510 [main] INFO  org.apache.coyote.http11.Http11NioProtocol - Starting ProtocolHandler ["http-nio-8090"]
2023-03-05 11:11:14.516 [main] INFO  org.apache.tomcat.util.net.NioSelectorPool - Using a shared selector for servlet write/read
2023-03-05 11:11:14.530 [main] INFO  o.s.boot.web.embedded.tomcat.TomcatWebServer - Tomcat started on port(s): 8090 (http) with context path ''
2023-03-05 11:11:14.535 [main] INFO  com.alibaba.otter.canal.admin.CanalAdminApplication - Started CanalAdminApplication in 3.872 seconds (JVM running for 4.538)

配置canal-deployer

删除 canal.properties 将 canal_local.properties 重命名为 canal.properties，添加配置：

# canal admin config
canal.admin.manager = 127.0.0.1:8090
canal.admin.port = 11110
canal.admin.user = admin
canal.admin.passwd = 4ACFE3202A5FF5CF467898FC58AAB1D615029441
# admin auto register
canal.admin.register.auto = true
canal.admin.register.cluster =
canal.admin.register.name =

然后进入canal.deployer-1.1.5/bin，使用./startup.sh命令启动canal服务端
此时canal服务端会自动注册到管理端
在这里插入图片描述

配置canal-adapter

参照之前博客中配置客户端canal-adapter
canal实时同步mysql数据到elasticsearch(部署，配置，测试)(一)

canal-deployer启动可能遇到的问题

example config is not found

问题原因：注册到管理端后，会以管理端的instance配置为准，原canal.deployer-1.1.5/conf/example 会自动失效，这是因为管理端为了保证在集群状态下，多个节点使用的配置文件保持一致

2023-03-05 13:25:23.588 [main] ERROR c.a.o.c.common.zookeeper.running.ServerRunningMonitor - start failed
com.google.common.util.concurrent.UncheckedExecutionException: com.alibaba.otter.canal.common.CanalException: com.alibaba.otter.canal.common.CanalException: instance : example config is not found

需要在 Canal Admin 手动添加实例
在这里插入图片描述
然后重启canal-deployer,可以看到auto notify start 新建instance successful.

2023-03-08 14:41:22.220 [canal-instance-scan-0] INFO  com.alibaba.otter.canal.deployer.CanalController - auto notify start 新建instance successful.
2023-03-08 14:41:41.865 [Thread-3] INFO  com.alibaba.otter.canal.deployer.CanalStarter - ## stop the canal server
2023-03-08 14:41:42.515 [Thread-3] INFO  com.alibaba.otter.canal.deployer.CanalController - ## stop the canal server[192.168.122.1(192.168.122.1):11111]
2023-03-08 14:41:42.521 [Thread-3] INFO  com.alibaba.otter.canal.deployer.CanalStarter - ## canal server is down.
2023-03-08 14:41:43.428 [main] INFO  com.alibaba.otter.canal.deployer.CanalLauncher - ## set default uncaught exception handler
2023-03-08 14:41:43.463 [main] INFO  com.alibaba.otter.canal.deployer.CanalLauncher - ## load canal configurations
2023-03-08 14:41:43.879 [main] INFO  com.alibaba.otter.canal.deployer.CanalStarter - ## start the canal server.
2023-03-08 14:41:43.921 [main] INFO  com.alibaba.otter.canal.deployer.CanalController - ## start the canal server[192.168.122.1(192.168.122.1):11111]
2023-03-08 14:41:44.002 [main] INFO  com.alibaba.otter.canal.deployer.CanalStarter - ## the canal server is running now ......
2023-03-08 14:41:45.319 [canal-instance-scan-0] INFO  com.alibaba.otter.canal.deployer.CanalController - auto notify start 新建instance successful.

服务端的conf下也会出现新建的instance配置，并且其他目录下的配置会失效！
在这里插入图片描述

Error[INSERT command denied to user ‘canal’@‘192.168.0.119’ for table ‘canal_node_server’]

2023-03-05 12:21:50.590 [main] ERROR com.alibaba.otter.canal.deployer.CanalLauncher - ## Something goes wrong when starting up the canal Server:
com.alibaba.otter.canal.common.CanalException: load manager config failed.
Caused by: com.alibaba.otter.canal.common.CanalException: requestGet for canal config error: Error[INSERT command denied to user 'canal'@'192.168.0.119' for table 'canal_node_server']

原因：配置的 MySQL 账户没有具有 canal_manager 库的插入权限
解决：直接全部权限给过去

GRANT ALL PRIVILEGES ON *.* TO 'canal'@'%';

java.sql.SQLNonTransientConnectionException: Public Key Retrieval is not allowed

org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'ebeanServer' defined in class path resource [com/alibaba/otter/canal/admin/config/EbeanConfig.class]: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [io.ebean.EbeanServer]: Factory method 'ebeanServer' threw exception; nested exception is javax.persistence.PersistenceException: java.sql.SQLNonTransientConnectionException: Public Key Retrieval is not allowed
	at org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:587)
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.instantiateUsingFactoryMethod(AbstractAutowireCapableBeanFactory.java:1250)
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1099)
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:541)
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:501)
	at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:317)
	at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:228)
	at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:315)
	at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:199)
	at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:760)
	at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:869)
	at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:550)
	at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.refresh(ServletWebServerApplicationContext.java:140)
	at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:759)
	at org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:395)
	at org.springframework.boot.SpringApplication.run(SpringApplication.java:327)
	at com.alibaba.otter.canal.admin.CanalAdminApplication.main(CanalAdminApplication.java:19)
Caused by: org.springframework.beans.BeanInstantiationException: Failed to instantiate [io.ebean.EbeanServer]: Factory method 'ebeanServer' threw exception; nested exception is javax.persistence.PersistenceException: java.sql.SQLNonTransientConnectionException: Public Key Retrieval is not allowed
	at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:185)
	at org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:579)
	... 16 common frames omitted
Caused by: javax.persistence.PersistenceException: java.sql.SQLNonTransientConnectionException: Public Key Retrieval is not allowed

解决方案：
修改/canal.admin-1.1.5/conf/application.yml的url，增加配置

&allowPublicKeyRetrieval=true

server中的状态为’断开’

应该是密码验证问题：

canal server 与 canal admin 的认证机制
canal admin 的 conf/application.yml 里面定义了账号密码明文
canal.adminUser: admin
canal.adminPasswd：admin

canal server 的 conf/canal_local.properties 里面定义了账号密码密文
canal.admin.user: admin
canal.admin.passwd:4ACFE3202A5FF5CF467898FC58AAB1D615029441

密文的生成方式: select password(‘xxxxxx’);

双向认证: canal server 向 canal admin 注册的时候会以密码密文做认证, canal admin 对 canal
server 做连通性测试的时候也会将密码明文加密之后做认证 (连通性测试失败的时候，canal admin web 会显示对应的
canal server 处于 “断开” 状态)

admin中的单机效果

在这里插入图片描述

创建多个instance实例并同步

新增instance，修改canal.mq.topic

需求是将当前服务的数据同步到另一个es数据库，因此需要创建一个新的实例
在这里插入图片描述
启动该实例，/conf目录下会自动创建该实例目录

修改canal adapter的conf/application.yml

适配器列表新增cxstar_oa_release实例，名称与文章上面服务端instance的配置相同

server:
  port: 8189
spring:
  jackson:
    date-format: yyyy-MM-dd HH:mm:ss
    time-zone: GMT+8
    default-property-inclusion: non_null
    
canal.conf:
  mode: tcp # 客户端的模式，可选tcp kafka rocketMQ
  flatMessage: true # 扁平message开关, 是否以json字符串形式投递数据, 仅在kafka/rocketMQ模式下有效
  zookeeperHosts:    # 对应集群模式下的zk地址
  syncBatchSize: 1000 # 每次同步的批数量
  retries: 0 # 重试次数, -1为无限重试
  timeout: # 同步超时时间, 单位毫秒
  accessKey:
  secretKey:
  consumerProperties:
    # canal tcp consumer
    canal.tcp.server.host: 127.0.0.1:11111 #设置canal-server的地址
    canal.tcp.zookeeper.hosts:
    canal.tcp.batch.size: 500
    canal.tcp.username:
    canal.tcp.password:
 
  srcDataSources: # 源数据库配置
    #测试数据库连接
    defaultDS:
      url: jdbc:mysql://192.168.0.224:4706/cxstar_oa?useUnicode=true 
      username: *** #数据库账号
      password: *** #数据库密码
      
  canalAdapters: # 适配器列表
  - instance: cxstar_oa # canal实例名或者MQ topic名
    groups: # 分组列表
    - groupId: g1 # 分组id, 如果是MQ模式将用到该值
      outerAdapters:
      - name: logger # 日志打印适配器
      - name: es7 # ES同步适配器
        hosts: http://192.168.0.***:11700 # ES连接地址
        properties:
          mode: rest # 模式可选transport(9300) 或者 rest(9200)
          # security.auth: elastic:123456 #  连接es的用户和密码，仅rest模式使用
          cluster.name: es-test # ES集群名称, 与es目录下 elasticsearch.yml文件cluster.name对应
          
  - instance: cxstar_oa_release #线上发布示例
    groups: # 分组列表
    - groupId: g1 # 分组id, 如果是MQ模式将用到该值
      outerAdapters:
      - name: logger # 日志打印适配器
      - name: es7 # ES同步适配器
        hosts: http://192.168.0.***:8300 # ES连接地址
        properties:
          mode: rest # 模式可选transport(9300) 或者 rest(9200)
          # security.auth: elastic:123456 #  连接es的用户和密码，仅rest模式使用
          cluster.name: es-en # ES集群名称, 与es目录下 elasticsearch.yml文件cluster.name对应

canal.adapter-1.1.5/conf/es7新增映射文件

注意修改destination，保持一致

dataSourceKey: defaultDS # 源数据源的key, 对应上面配置的srcDataSources中的值
destination: cxstar_oa_release  # canal的instance或者MQ的topic
groupId: g1 # 对应MQ模式下的groupId, 只会同步对应groupId的数据
esVersion: es7
esMapping:
  _index: cnoa_data_1987051001 # es 的索引名称
  _id: _id  # es 的_id, 如果不配置该项必须配置下面的pk项_id则会由es自动分配
  sql: "SELECT
			CONCAT('cxstaroa',d.id) as _id,
			d.id as `101`,
			d.title as `10100001`,
			d.isbn as `281450001`,
			d.publish_date as `10500011`,
			YEAR ( d.publish_date ) as `10500021`,
			d.author as `888000002`,
			d.author as `15900001`,
			d.subjects as `331440002`,
			d.price as `10310004`,
			d.revision as `51022`,
			d.impression as `51020`,
			d.pages as `524100001`,
			d.words as `64013`,
			d.catalog as `64017`,
			d.content as `10210001`,
		  (select file_name from archive where sid=d.cover_id) as `294750001`,
			d.create_by as `2`,
			d.create_time as `3`,
			d.update_by as `4`,
			d.update_time as `5`,
			d.remark as `99003`,
			da1.archiveSids as `350050001`,
			'subject_default' as `331440001`,
			'1987052001' as `100`,
			'1987052001' as dmcode,
			'A' as `7`,
			'A' as `wfstatus`,
			'2' as `50001`,
			'1' as `103`
        FROM
			data d
			LEFT JOIN
					(select da.data_id,GROUP_CONCAT(da.archive_sid SEPARATOR ',') as archiveSids 
					 from data_archive da 
					 GROUP BY da.data_id) as da1
			on d.id=da1.data_id"        # sql映射
  #etlCondition: "where p.id>={}"   #etl的条件参数
  objFields:
    350050001: array:,
  commitBatch: 3000   # 提交批大小

执行全量同步语句

数据已经成功同步

[root@test-server-001 bin]# curl http://127.0.0.1:8189/etl/es7/data_release.yml -X POST
{"succeeded":true,"resultMessage":"导入ES 数据：190 条"}