数据采集工具之maxwell

1、安装maxwell

网站

https://maxwells-daemon.io/changelog/

选择v1.29.2

（v130.0以后的全是jdk11 ，就不是jdk8了）

选择maxwell-1.29.2.tar.gz点一下就下载了

安装就是把安装包扔到虚拟机的指定位置

tar -zxvf maxwell-1.29.2.tar.gz -C /usr/local/

我没配置环境变量，其实也可去配置环境变量

2、mysql的环境配置

1)修改mysql的配置

 vim /etc/my.cnf
 
[mysqld]
server_id=1
log-bin=mysql-bin
binlog_format=row
binlog-do-db=test_maxwell

binlog-do-db就是监听的数据库

systemctl restart mysqld

mysql -uroot -p'@Mmforu45'

 show variables like '%binlog%';
 #查看
 binlog_format | ROW

查看真实的binlog

 cd /var/lib/mysql
 ll
  mysqlbin.000001
  mysqlbin.index

2)初始化元数据库

mysql -uroot -p'@Mmforu45'

mysql> CREATE DATABASE maxwell;
mysql> set global validate_password_length=4;
mysql> set global validate_password_policy=0;
mysql> GRANT ALL ON maxwell.* TO 'maxwell'@'%' IDENTIFIED BY '@Mmforu45';
mysql> GRANT SELECT ,REPLICATION SLAVE , REPLICATION CLIENT ON *.* TO maxwell@'%';
mysql> flush privileges;

3、Maxwell的启动方式

3.1第一种启动方式

启动一下maxwell，但是maxwell和mysql一定是在同一个电脑里面！

cd /usr/local/maxwell-1.29.2

bin/maxwell --user='maxwell' --password='@Mmforu45' --host='qianfeng03' --producer=stdout

因为maxwell是前台打印，所以就关闭就直接ctrl+c

3.2 第二种启动方式

就是修改配置，然后启动配置

cd /usr/local/maxwell-1.29.2
cp  config.properties.example config.properties

修改

# tl;dr config
log_level=info

producer=stdout
kafka.bootstrap.servers=localhost:9092

# mysql login info
host=qianfeng03
user=maxwell
password=@Mmforu45

启动

bin/maxwell --config ./config.properties

打印在控制台的操作

bin/maxwell --config ./config.properties

在mysql中

4、案例实操

4.1 案例1

mysql -uroot -p'@Mmforu45'

use test_maxwell;
create table test01(
id tinyint,
name varchar(24)
);
insert into test01 value(1,'aaaa'),(2,'bbbbb'),(3,'cccc');

回看一下前台打印日志

[root@qianfeng03 maxwell-1.29.2]# bin/maxwell --config ./config.properties
Using kafka version: 1.0.0
23:25:22,490 INFO  Maxwell - Starting Maxwell. maxMemory: 652738560 bufferMemory                                                                               Usage: 0.25
23:25:22,538 INFO  Maxwell - Maxwell v1.29.2 is booting (StdoutProducer), starti                                                                               ng at Position[BinlogPosition[mysql-bin.000001:77135], lastHeartbeat=16573794609                                                                               62]
23:25:22,632 INFO  MysqlSavedSchema - Restoring schema id 1 (last modified at Po                                                                               sition[BinlogPosition[mysql-bin.000001:5598], lastHeartbeat=0])
23:25:22,730 INFO  BinlogConnectorReplicator - Setting initial binlog pos to: my                                                                               sql-bin.000001:77135
23:25:22,740 INFO  BinaryLogClient - Connected to qianfeng03:3306 at mysql-bin.0                                                                               00001/77135 (sid:6379, cid:68)
23:25:22,740 INFO  BinlogConnectorReplicator - Binlog connected.
23:32:59,024 INFO  AbstractSchemaStore - storing schema @Position[BinlogPosition                                                                               [mysql-bin.000001:108271], lastHeartbeat=1657380774724] after applying "create t                                                                               able test01( id tinyint, name varchar(24) )" to test_maxwell, new schema id is 2
{
   "database":"test_maxwell","table":"test01","type":"insert","ts":1657380895,"xid                                                                               ":2227,"xoffset":0,"data":{
   "id":1,"name":"aaaa"}}
{
   "database":"test_maxwell","table":"test01","type":"insert","ts":1657380895,