We identified the cause of the issue to live on Clickhouse Cloud. Its actually a combination of several factors.
In summary:
System logs tables cleanup is a single transaction with a lot of operation in it.
While this transaction is running, other writes to the logs on this given replica are locked, and sometimes it might take time.
Because of the lock, Idle controller was not able to get the service usage stats and considered this "no activity" as the idle trigger.
The existing pod was not terminated immediately due to ongoing background activity (there is an up to 1h grace period to let the replica complete ongoing tasks before pod termination), and this caused delays in launching the new pod when the service was un-idled a few mins later.
Clickhouse team has a confirmed fix for this in the newer version and their engineering team is working on the back-port. As soon as the back-port is ready, our cluster will be upgraded to the version with the fix.
Resolved
We identified the cause of the issue to live on Clickhouse Cloud. Its actually a combination of several factors.
In summary:
System logs tables cleanup is a single transaction with a lot of operation in it.
While this transaction is running, other writes to the logs on this given replica are locked, and sometimes it might take time.
Because of the lock, Idle controller was not able to get the service usage stats and considered this "no activity" as the idle trigger.
The existing pod was not terminated immediately due to ongoing background activity (there is an up to 1h grace period to let the replica complete ongoing tasks before pod termination), and this caused delays in launching the new pod when the service was un-idled a few mins later.
Clickhouse team has a confirmed fix for this in the newer version and their engineering team is working on the back-port. As soon as the back-port is ready, our cluster will be upgraded to the version with the fix.
Monitoring
Clickhouse is restoring service. OneView API is back online.
Monitoring
We've identified a slowdown on Clickhouse Cloud europe-west4 Cluster, we're monitoring the situation with Clickhouse team.
