文章
问答
冒泡
基于 Flink CDC 构建 MySQL 和 Postgres 的 Streaming ETL

官方文档:

Local dev:M2 Max
虚拟机:Ubuntu 20.04
Docker:23.0.4
Docker-compose:v2.17.2

问题:

按照官方docker-compose.yml进行配置,执行docker-compose up -d 后,容器没有正常启动,有报警日志
The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested
0

原因:

因为所拉取镜像为amd64架构,但虚拟机系统为arm64(此处本机为m2 arm64v8架构),故导致容器无法正常运行
0
 
查看虚拟机内核

解决:

从docker hub查找arm64对应镜像,并修改docker-compose.yml配置
docker pull debezium/example-postgres:2.2
0
docker pull debezium/example-mysql:2.2
0
docker pull arm64v8/elasticsearch:7.17.9
0
docker pull arm64v8/elasticsearch:7.17.9
0
完整配置:
version: '2.17.2'
services:
  postgres:
    image: debezium/example-postgres:2.2
    ports:
      - "5432:5432"
    environment:
      - POSTGRES_DB=postgres
      - POSTGRES_USER=postgres
      - POSTGRES_PASSWORD=postgres
  mysql:
    image: debezium/example-mysql:2.2
    ports:
      - "3306:3306"
    environment:
      - MYSQL_ROOT_PASSWORD=123456
      - MYSQL_USER=mysqluser
      - MYSQL_PASSWORD=mysqlpw
  elasticsearch:
    image: arm64v8/elasticsearch:7.17.9
    environment:
      - cluster.name=docker-cluster
      - bootstrap.memory_lock=true
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
      - discovery.type=single-node
    ports:
      - "9200:9200"
      - "9300:9300"
    ulimits:
      memlock:
        soft: -1
        hard: -1
      nofile:
        soft: 65536
        hard: 65536
  kibana:
    image: arm64v8/kibana:7.17.9
    ports:
      - "5601:5601"
 
注意:
elasticsearch最新版本为8.7.0,但配置却用的7.x
这是应为 flink-sql-connector-elasticsearch7-1.16.0.jar 依赖包当前只支持到es7 (2023-04-24)
0
 
查看flink-connector插件
 
0
 
0
 
参考:
  1. https://github.com/ververica/flink-cdc-connectors
  2. https://ververica.github.io/flink-cdc-connectors/release-2.3/index.html#
flink cdc

关于作者

Miraclewcg
上善若水
获得点赞
文章被阅读