Continuent Community

Hi

We have some tables with a few million rows. Every now and again, an update is done to the tables which results in a few thousand rows updating or being inserted.

Since I am using ROW replication, the Tungsten master fails on these with either a packet too large exception or a heap error.

I have tried 72M max_allowed_packet, and 1024M java heap size, with the same results.

I am loathe to continually push these variables higher and higher, but am hoping that tungsten has a way of breaking this up into smaller chunks that can be managed more effectively.

Are there any settings i can look at to help with this?

Hi Chris,

Which version of Tungsten are you using? We put support for transaction fragmentation into Tungsten 1.3--you should be able to use the latest RC build and solve this problem, because it will break up large transactions into smaller chunks that will not blow out the memory on the VM. Here's the URL for download of the open source version:

https://sourceforge.net/projects/tungsten/files/

Please try it out if you have not done so, and post your results. If you are already using Tungsten 1.3, I would like to dig in bit more on the problem so ensure we fix it.

Cheers, Robert

Hi Robert

I am using version 1.2.3 - I am a bit put off by the RC in 1.3RC2

Do you recommend 1.3 over 1.2 from a stability point of view?

Is the upgrade path seemless (uninstall 1.2, install 1.3, configure, start replicator)?

Many thanks

Chris

PS: I have built RPMs of 1.2.3 for my purposes - not sure if you would find the spec files useful at all?

Hi Chris,

The current RC-2 build is quite stable. However, there's one final round of improvements going in this week, culminating in RC-4. After this we expect the next build to remove the RC label. You may want to upgrade at that time.

Upgrade is pretty straight-forward and works as you describe. One issue in the upgrade, though: there are log format changes, which means that you will need to clear logs when you upgrade. This requires (unfortunately) a maintenance window. We hope to avoid these in future.

Cheers, Robert

Thanks Robert

I will wait for 1.3 to be released, and give you feedback.

Im the meantime, I will try convince my developers to modify some of their insert/update queries to minimise data transfer

Hi Chris,

I'm sure you are headed this way already, but something like the following should eliminate the problem with large updates:

SET SESSION BINLOG_FORMAT='STATEMENT';
UPDATE mytable SET ... WHERE ... ;

You'll also see a jump in replication performance as you'll be replicating a single statement only.

Cheers, Robert