Continuent Community

Hi all

I am trying to work out how to replicate data efficiently between two of my mysql servers. One is a standalone mysql server, the other is a mysql cluster.

Different apps use each of these systems, and the schema definitions are different on the server vs the cluster. So, the standalone server could have

Code:

CREATE TABLE `t1` ( `grpid` int(11) DEFAULT NULL, `priority` enum('1','2','3','4','5') DEFAULT NULL, `val` varchar(64) DEFAULT NULL, UNIQUE KEY `grpid` (`grpid`,`priority`) ) engine=innodb;

While the cluster has

Code:

create table cluster_t1 ( grpid int primary key, priority1val varchar(64), priority2val varchar(64), priority3val varchar(64), priority4val varchar(64), priority5val varchar(64) ) engine=ndb;

Is there any way I can effectively use tungsten replicator for this. Is the filter plugin flexible enough to be able to handle it correctly? As a further, more complicated question, if I want a view of the standalone server which comprises a subset of multiple tables to be replicated to a single table on the cluster, is this possible?

Hi Chris!

Have a look at the Java script filters provided as samples in the tungsten-replicator directory. (I assume you are using the full Tungsten clustering.) It's pretty flexible.

You can also write filters in Java as well if you know that language, but it requires a little more effort to compile and add them in.

Cheers, Robert

I have tungsten replicator set up on a master and a slave, but I cannot get the javascript filter to register.

My master only has the logging prefilter (and this works)

I am trying to register the javascript filter, my config file has:

Code:

replicator.prefilter=javascript replicator.postfilter=logger replicator.filter.javascript=com.continuent.tungsten.replicator.filter.JavaScrip tFilter replicator.filter.javascript.script=/opt/continuent/replicator/samples/ extensions/javascript/filter.js replicator.filter.javascript.sample_custom_property="Sample value *+*+*+*+*+*+*+*+*+*+*+*+*+*+*+*"

The startup logs on the slave indicate that the plugin is loading:

Code:

2010-03-31 16:08:49,300 INFO replicator.conf.ReplicatorRuntime Loading plugin: key=replicator.filter.javascript class name=com.continuent.tungsten.replicator.filter.JavaScriptFilter 2010-03-31 16:08:49,304 INFO replicator.conf.ReplicatorRuntime Plug-in configured successfully: key=replicator.filter.javascript class name=com.continuent.tungsten.replicator.filter.JavaScriptFilter

However, I see no logs coming from the prepare() function of the javascript code. I do see logs from the logger filter, but the javascript filter appears to not run. I have confirmed that the path to the js file is correct, but even if I point it to a non-existant file, the logs show exactly the same thing. Any clues for me?

Hi Chris!

I think you are configuring a pre-filter you actually want a post-filter. Pre-filters only run on masters. If you want this to run on the slave it needs to be declared as a post-filter. We load all filters regardless of whether they are used to ensure you don't have dud information in the replicator.properties file.

The scheme for filters is slightly confusing in the current 1.2.x implementation. In 1.3 we have something called replication pipelines that do away with the asymmetry between master and slave replicators. Stay posted--code and docs are on the way.

Let me know if this seems to solve your problem.

Cheers, Robert

Thank - the filters are now running

Are there any tools to help with statement decoding?

It seems like ROW format binlogs are easier to handle if you need to manipulate the data.

Hi Chris,

Glad to hear things are working.

On the statement decoding--do you mean parsing? Within Tungsten we do a lot of "parsing" using regular expressions. They are really fast and handle most cases pretty effectively.

On row replication, yes in general it's easier for data manipulation. There is one unfortunate problem with the MySQL binlog in that it does not put in column names, only column numbers.

Cheers, Robert