Thursday, December 10, 2009

Data migration : Part 1

Having written and looked after ImpEx for vBulletin I've dealt with a few matters regarding users and their identification, along with their data, and moving it about. I thought I'd share some thoughts that I've had on the matter as most people seem to balk at the idea of data migration due it it's "messiness", it dosen't have to be like that.

First an overview

There are just over 120 source systems for ImpEx now and growing.

They are split into three tiers, sorting themselves into these three groups on a criteria of most commonly used, most requested (by customers) and frequency of change.

For example the most common and used importer is top of Tier 1, it gets a lot of use & attention from moi. Tier 2 ones are on tick over and Tier 3 are in the grave yard, the most common reason being that the version supported no longer is in active development, there for the importer will never change.

The moving target has stopped moving, much like playing Unreal Tournament and getting a head shot every time on a target ........ oh the days.

Whats in a name

ImpEx's primary requirement is to protect the data model of vBulletin, i.e. the integrity of the database. All manner of things could be imported and done during an import, passwords being the most common hot topic of conversation. And yes I know how to do it and how it can be done, it's not a case that I don't know, it's that something else takes precedence !!

Which is integrity, there should be no change to the schema or functionality of a standard install after an import, just a bunch more data that could of been created by vBulletin itself (with the exception of logs and the import ids a.k.a. foreign keys).

Breaking the name down ImpEx is Import Export. It's not strictly a part of vBulletin, though it can run in the admincp and take the config details from vBulletin's config.php I typically run it standalone just against a default vBulletin database schema. The ImpEx Target refers that that schema, i.e. "what version of vBulletin are we Exporting this data from ImpEx into". The Source is the other half of the equation.

The ImpEx washing machine

"Stick it in one side, clean it up, pump it out the other. ImpEx, gets your posts whiter than white"

"Cleanup imports", an import done to upgrade from a board with hacks or from one version of vBulletin to another (or even the same version) in an attempt to 'start again' aren't really the primary purpose of ImpEx, but can be a last ditch option. Though frowned upon and not officially supported.

All the bits

So we have the system, of which there is a core (ImpEx itself) and all the supported source systems, these are the source modules. Now to the Ex bit, the target modules.

ImpEx only officially supports the latest stable release (3.8.4 at time of writing) though each time a new major release is available, there is a new target module (or upgraded existing one) released (so yes I have one in the wings being tuned up for vB 4).

A simple way of maintaining an offering to the ongoing customers who upgrade at their own pace.

So what does it do

How data is sourced, linked, referred, captured, verified and written will be the topic of Part 2.


  1. hello jerry

    i need your help plz

    can u transfer my forum from phphbb3 to vb4 plz

  2. If I had a dollar for every time ........