Debezium Connector Vitess

Overview

Debezium Connector for Vitess is an innovative component designed for low-latency data streaming, primarily for change data capture (CDC). Though still in its early stages of development, this open-source project offers a promising solution for environments needing reliable and efficient data synchronization. It serves as an excellent exploration tool for those looking to get ahead in the data streaming space, despite not being recommended for production usage just yet.

This connector simplifies the complexities of data management in a Vitess environment, making it easier to monitor and manage changes across multiple shards. With specific features aimed at flexibility in subscription options and the handling of events and transactions, the Debezium Connector for Vitess is worth the attention of developers and data engineers alike.

Features

• Multi-Shard Subscription: One connector instance can subscribe to all shards of a given keyspace or a single shard, providing flexibility based on user needs.
• Reshard Operations: Supports vitess Reshard operations without data loss or duplication when subscribing to all shards, enhancing data integrity.
• Event Handling: Each transaction can contain multiple events, enabling comprehensive tracking of changes in the database.
• Data Format Support: Compatible with AVRO and JSON connect converters, allowing for varied data formats according to user preference.
• Extraction Capabilities: Features the ability to extract only the new record state through the use of the ExtractNewRecordState single message transformation (SMT).
• Graceful Coordination: Utilizes async grpc stub for vstream service, ensuring that change event source coordination can be managed smoothly.
• Event Generation from Sequence Tables: Automatically generates events from Vitess Sequence tables with optional filtering capabilities for the users.
• Integration Testing: Offers robust testing options during the development phase, ensuring that users can build the connector efficiently and verify its functionality.