Scaling CDC ClickPipes via OpenAPI
Default configuration of CDC ClickPipes is designed to handle the majority of workloads out of the box. If you think your workload requires scaling, open a support case and we'll guide you through the optimal settings for the use case.
Scaling API may be useful for:
- Large initial loads (over 4 TB)
- Migrating a moderate amount of data at a very high speed (sub-1 hour)
- Supporting over 8 CDC ClickPipes under the same service
Other signs that scaling may be necessary:
- Initial load is taking longer than 24 hours while the load on the source DB is low
- First consider adjusting the initial load parallelism and partitioning when creating the ClickPipe
- The new rows taking more than 2× the sync interval to appear on the destination table
- As long as there are no long-running transactions on the source
Increasing the CDC scale will proportionally increase your ClickPipes compute costs. If you're scaling up just for the initial loads, it's important to scale down after the snapshot is finished to avoid unexpected charges. For more details on pricing, see Postgres CDC Pricing.
Prerequisites for this process
Before you get started you will need:
- ClickHouse API key with Admin permissions on the target ClickHouse Cloud service.
- A CDC ClickPipe (Postgres, MySQL or MongoDB) provisioned in the service at some point in time. CDC infrastructure gets created along with the first ClickPipe, and the scaling endpoints become available from that point onwards.
Steps to scale CDC ClickPipes
Set the following environment variables before running any commands:
Fetch the current scaling configuration (optional):
Set the desired scaling. Supported configurations include 1..16 CPU cores and memory GB that is 4× the core count:
Wait for the command to propagate - this usually takes 3-5 minutes. After the scaling is finished, the GET endpoint will reflect the new values:
