Upload, convert, and store multiple different product data feeds into a database

Question

I'm not sure if this right location for this question, but here it is.

I currently have about 30 different CSV or TXT product data feeds. I am trying to build a tool to

1) upload each feed

2) break apart each feed line by line

3) extract every title, category, price, etc. into a product object, and...

4) either create or update each product in the database.

Here is my problem. Each feed has a different name for header attributes like title, category, price, etc. Also each feed's categories are unique as well.

So far the approach I have been leaning toward is: (Note this is a rough overview)

1. Admin fills out UI form. The Admin selects a file type & attaches the product data feed
2. Upon submit, the product data feed is sent chunk by chunk and saved to disk via socket.io
3. Once the file is saved a job is added to our job queue to process the data feed
4. Once the job is called, we first check for new header data attributes, if a new attribute is discovered, we put the job on hold and set a UI prompt for the Admin to define what the header attribute maps to, Fe. the Admin would map Classification -> category, Name -> title, or Product Name -> title.
5. Once all header attributes are recognized, we break the file down line by line using streams.
6. With each broken down line we grab the title, category, price, etc. We then check to see if the product exists, if so, update the product, if not, create the product.
7. Once each line is processed we delete the file and mark the job as complete.

What would be a good way to map incoming feed categories to my db categories? I was thinking again to set a UI prompt, Fe. Mens Footwear -> Mens Shoes.

Am I going in the right direction? Should I consolidate all the feeds and then import? Or am I just going about this all wrong?

• Vladislav Rastrusny: I guess this question contains a lot of irrelevant details. What you ask is how you map one categorization system into another, right? I guess... with a map. Asking user each time you encounter a new relation.

Answer 1

A little late go the game, maybe.. Stumbled upon this old unanswered question and thought I'd add an idea.

The attributes you want to map between seem to be quite low in quantity? I mean, there probably aren't very many synonyms to "category", etc.

How about taking a guess-and-learn approach. Each time you encounter a new attribute name you add it to one of your maps or ask the user to add it.

After a while you should be capable of decoding most common structures. That is if your input material is much like you describe in your question.