Database management

[funwork]

Hello,
We've been doing VDP printing for some years now and things are going pretty.
So far w've been dealing with Excel files (most fo the times) provided by our customer.
Some times, we had to merge/deduplicate/reformat the data as they would come in like in 5 xls files with 5 sheets each !!! (If you laught you know what I'm talking about...If you don't just pray this never occurs !)

Anyway, now i'm thinking of developping/acquiring a tool that would let us import the data 'as they come' and that would assist us in all the formating/deduplication/etc tasks.
I don't need a tool that will compare the adresse with postal rules as we are operating in the Caribean and Postal adresses rules are not realy a problem.

What we need is some kind of tool that will handle incoming data as they are and help us input these data into some big database, cleaned and ready to be used.

On top of this we would like to start a new business in providing data to our customers.
Aside of where we will find these data, we would like to have a tool that will let us handle these datas.

I was thinking the same database+soft would be able to do both of these.
What do you thing about this and have you got any feedback and soft, experience ?

Thanks for your time and once again congratulations to the people behind this site which is a great source of info !

Frank

 

[JasonFF]

Well funwork, I happen to know exactly what you mean. Annoying isnt it? Every customer always has a completely different database style, which makes the kind of product you're looking for hard to use.

I think you already have what you need in Excel. You would need a handful of scripts to analyze the type of data in a given column - names, addresses, phone numbers, prices, emails, websites, product ID, and well the list could go on. Once you were able to quickly (automatically) determine what type of data was in each row, you could call other scripts that would sort out names into Title First Middle Last, and otherwise organize and trim your data.

The problem becomes that the more automation you introduce, the higher the chance for a mistake.

If you're looking to purchase some software to do this, it might be expensive, and it would probably be based around SQL style databases. If you're going to manage and hold data for customers you would almost have to do it from a larger database system such as SQL because it would be a lot easier to manage or make changes to multiple databases, as opposed to making changes to 10 different excel sheets.

Maybe someone with a history of database engineering and administration can recommend some management software. I've never actually used a third party program to help admin databases, but that's only because I've never had a large enough collection of databases to warrant it.
Hope this helps.

 

[funwork]

Hi Jason,

You are right.
After some research, we ended up with the confirmation that we would have to move to a SQL database architecture.

Currently, we are in the process of writing our needs so we can have some IT person develop an application that will rely on a SQL database server.
On top of it, a few fonctions will be coupled with the ability to edit the grid manualy (a la Excel style)

The most of the work will be done as the importing, remaping and analysing phases would be semi automated.

Anyway, thanks for your feedback and yeah, it's always a PITA to deal with the differents files...

 

[DanB]

Maybe this would be helpful. dQuery

 

[funwork]

Wow...It looks amazing ! I'll give it a try.
Thanks for the link

 

[funwork]

Just for the record, we endend up with Datamartist, a tool that looks like dquery but that we found to be more userfriendly !

Thanks for all.

Frank