Monday, December 13, 2010

One way transactional replication from MSSQL 2005 to MYSQL 5.x

At my place of work, we ran into what should be a normal problem. We use mssql 2005 for our internal operations, however we use mysql for our database backend. We wanted to update the changes that happened on the mssql server up to the mysql server so that the website would reflect the changes.

**This how-to only applies to Windows XP, MSSQL Server 2005, MYSQL 5.1.x and MYSQL ODBC connecter 5.1**
You must first allow the account that you will be using for the connection access to the mysql database. Since I was only testing, this is the command I issued on the Linux box that is hosting the MYSQL server
mysql> GRANT ALL PRIVILEGES ON *.* to 'username@windowsmachinename.domain' IDENTIFIED BY 'password';
So if I was using "root" as the username, "lemon" as the machine name and "orange" as the domain name, with the password of "home1234" it would look like this: 
mysql> GRANT ALL PRIVILEGES ON *.* to root@lemon.orange.local IDENTIFIED BY 'home1234';
Once that is done, then you must visit the mysql ODBC driver website and download the correct driver for the MSSQL server that  you are using.
After installing the driver, you must open up the ODBC connections on your machine. Once there, you must add the MYSQL ODBC driver as a system dsn.
My configuration is shown below for the MYSQL ODBC connector.
After you have successfully tested that the connection is valid, open up SSMS, connect to the database engine. Once you are connected, expand the "replication", right-click on "Local Publications", select "Publisher Properties". Once that window is up click on "Publication Databases", then select the requested database as transactional.

Expand "linked servers", then expand providers, right click on "MSDASQL", in the pop-up, select the choices: Nested queries, level zero only, allow inprocess, supports 'like' operator

After that, fire up a new query window and copy/paste the below code.

The steps below
--step 1
-- Adding the transactional publication
use [repl_test]
exec sp_addpublication @publication = N'Repl_test'
, @description = N'Transactional publication of database'
, @sync_method = N'concurrent_c'
, @retention = 0
, @allow_push = N'true'
, @allow_pull = N'false'
, @allow_anonymous = N'true'
, @enabled_for_internet = N'false'
, @snapshot_in_defaultfolder = N'true'
, @compress_snapshot = N'false'
, @ftp_port = 21
, @allow_subscription_copy = N'false'
, @add_to_active_directory = N'false'
, @repl_freq = N'continuous'
, @status = N'active'
, @independent_agent = N'true'
, @immediate_sync = N'true'
, @allow_sync_tran = N'false'
, @allow_queued_tran = N'false'
, @allow_dts = N'false'
, @replicate_ddl = 0
, @allow_initialize_from_backup = N'false'
, @enabled_for_p2p = N'false'
, @enabled_for_het_sub = N'true'
, @autogen_sync_procs = 'false'
GO
--add the article to the publication
exec sp_addarticle @publication = N'Repl_test'
, @article = N'TestTable'
, @source_owner = N'dbo'
, @source_object = N'TestTable'
, @type = N'logbased'
, @pre_creation_cmd = N'none'
, @ins_cmd = N'SQL'
, @del_cmd = N'SQL'
, @upd_cmd = N'SQL'
, @schema_option = 0x20025081
, @status = 24
GO

--add all of the columns to the article
exec sp_articlecolumn @publication = N'Repl_test'
, @article = N'TestTable'
, @refresh_synctran_procs = 1
GO

--end step1

--step2
--add the publication snaphot
exec sp_addpublication_snapshot @publication = N'Repl_test'
, @frequency_type = 4
, @frequency_interval = 4
, @frequency_relative_interval = 1
, @frequency_recurrence_factor = 0
, @frequency_subday = 4
, @frequency_subday_interval = 1
, @active_start_time_of_day = 0
, @active_end_time_of_day = 235959
, @active_start_date = 0
, @active_end_date = 0
, @job_login = null
, @job_password = null
, @publisher_security_mode = 1
GO
--end step2

--step3
--add the subscriber(s)
use [repl_test]
exec sp_addsubscription @publication = N'Repl_test'
, @subscriber = N'mysqltest'
, @destination_db = N'repl_test'
, @subscription_type = N'Push'
, @sync_type = N'automatic'
, @article = N'all'
, @update_mode = N'read only'
, @subscriber_type = 3
GO

--add the pushing subscription agent
exec sp_addpushsubscription_agent @publication = N'Repl_test'
, @subscriber = N'mysqltest'
, @subscriber_db = N'repl_test'
, @job_login = null
, @job_password = null
, @subscriber_security_mode = 0
, @subscriber_login = N'root'
, @subscriber_password = 'PASSWORD'
, @subscriber_provider = N'MSDASQL'
, @subscriber_datasrc = N'mysqltest'
, @frequency_type = 64
, @frequency_interval = 1
, @frequency_relative_interval = 0
, @frequency_recurrence_factor = 0
, @frequency_subday = 0
, @frequency_subday_interval = 0
, @active_start_time_of_day = 0
, @active_end_time_of_day = 235959
, @active_start_date = 20101202
, @active_end_date = 99991231
, @enabled_for_syncmgr = N'False'
, @dts_package_location = N'Distributor'
GO
--end step3

A brief breakdown of the arguments is as follows, I'll only be touching on the ones that I think are big.

@publication is the name that you want to use for the publication, must be the same throughout the stored procs.

In sp_addpublication
@sync_method should be concurrent_c, that will get  transactional replication
@allow_push must be true in order for the push subscription to be true
@enable_for_het_sub must be set to true, enable for heterogeneous subscriber

In sp_addarticle
@type must be logbased if you want transactional repl (which we do as we set the @sync_method to concurrent_c)
@pre_creation_cmd must be set to none. If this is not set to none, replication will drop the mysql table, then error out stating "no such table"
@schema_option You will have to look at the BoL for the various options, use google to add them together to get the final product.
The 3 command variables we just want to mirror the SQL commands

In sp_articlecolumn
Not sure if this is actually needed or not :)

In sp_addpublication_snaphot
These should be set to your requirements.

In sp_addsubscription
@subscriber should be the system dsn that you set up, at the start of this page
@update_mode should be set to read only, that ensures one-way replication from MSSQL to MYSQL
@subscriber_type should be set to 3(OLE DB) I have not tried it set to 1(ODBC)

In sp_addpushsubscription
@subscriber should be the system dsn again
@subscriber_login should be set to the login from the "grant all privileges" step up above
@subscriber_provider should be MSDASQL
@subscriber_datasrc should be set to the system dsn  again
@enable_for_syncmgr set to false
@dts_package_location set to Distributor since this is a push replication, everything is run from the publisher/distributor


*** Please this is what worked on my box, if it doesn't work, you can try posting here and we may be able to work it out. ***
Please check out BoL for the full list of options for all of the stored procs
If you are on twitter, try the hashtag #sqlhelp
Try posting to ServerFault

18 comments:

  1. Thanks! It took me a wek or so to get everything wrapped up. With all the hard work done, it only takes minutes to setup a new replication set.

    ReplyDelete
  2. initial transfer works, but replicating any data by deleting, inserting oder updating on the master breaks the replication.
    Any help?

    ReplyDelete
  3. What error are you getting when you open up the replication monitor?

    ReplyDelete
  4. Try changing @schema_option to "0x8000000" instead of "0x20025081"

    ReplyDelete
  5. what's your MySQL character set and collation ? I tried following your method but I keep on getting multi-step OLEDB errors.

    I tried setting the subscriber_type to ODBC instead of OLEDB and that didn't help either because it fails after initial snapshot delivery with some weird update statement for MSREPL7 table.

    ReplyDelete
  6. I did run into the multi-step OLEDB error, let me go through my notes and get back to you.
    Can you paste the error you are getting?

    ReplyDelete
  7. character_set_database = latin1
    collation_database = latin1_swedish_ci

    Note, I had nothing to do with the initial setup/configuration of this mysql server. I had to work with what was pre-existing

    I was getting the multi-step error because of data types. For example, if on the SQL side, the data type was "decimal(18, 2)" it would throw an error, however if I created the MYSQL tables before hand, with the data type of "float(18, 2)" I would not get the error.

    I really didn't find a whole lot of useful sites, so it was mostly trial and error.

    I am still trying to find a better fit than using "float" as we have to deal with rounding issues.

    Let me know how it works out for you.

    ReplyDelete
  8. Thanks for the quick response, I think I will recreate the databases and modify the schema_option to not execute create table or create index scripts. hopefully that would solve my issue. i have been trying so many different things but always failing.

    regarding the specific error, there is no more details besides the multi-step OLEDB errors and i can't seem to figure out which table is failing.

    ReplyDelete
  9. That is a good idea. When I set it up, I went for the basic setup. I add my indexes on the MYSQL side after I create the tables.

    Good luck.
    Reply back with what you find out.

    ReplyDelete
  10. i thought i will provide a quick update, i have been able to setup the initial snapshot to go without any errors now. the data shows up fine. The problem is now continuous replication, it is failing at tables which have decimal columns (i am assuming since i finally nailed down the command which is failing after the snapshot has been delivered)

    ReplyDelete
  11. Yup, SQL's decimal data type stopped me cold also. I am currently using float for the MYSQL side.
    Let me know which data type you choose for the MYSQL server

    ReplyDelete
  12. yup it was the decimal datatype which was causing me issues. I have gone down the route of float too since my columns are not really used for financial calculations and hence can live with rounding. But eventually i would like to figure out a better approach. I upgraded to the latest ODBC too which had some fixes for decimal datatypes but that didn't seem to help either.

    Thanks for all your help !!!

    ReplyDelete
  13. I am happy that everything has worked out!

    ReplyDelete
  14. I am trying to setup replication between SQL Server 2008 R2 and mysql 5.5. I can setup the publication and subscription and run the snapshot agent. But when I look at the Replication monitor, it says "The initial snapshot for the article 'Replicatedtable' is not yet available." and then a couple of minutes later, it errors out saing "The job failed. The job was invoked by sa. The last step to run was step 3(Detect nonlogged agent shutdown.)"

    I am not sure where I am going wrong. I would appreciate any suggestions.

    Thanks!

    ReplyDelete
  15. The account that the SQL agent runs under, does that have the correct permissions on the server, and where ever the bcp files are going on the file system?

    ReplyDelete
  16. i finished step 1, 2, 3 and then i look at the Replication Monitor, it says " Retrying failed command ".

    I am not sure where I am going wrong. I would appreciate any suggestions.

    Thanks!

    ReplyDelete
  17. Check the SQL Server agent error logs for the errors related to the replication job.

    Is the error "retrying failed command" the error that is present when you view the details for the subscription from the replication monitor?

    I ran into something like this, when the commands for inserting/updating/deleting weren't working correctly.

    ReplyDelete