For a project that I working on, I need to manage data that is updated on a regular basis. In addition to the bulk loaded data, we store additional metadata. Thus, a challenge arises when some of the data remains valid, but other data requires an update. I thought it'd be best to break this somewhat complex problem into smaller pieces. By breaking the process up like this, I hope to be able to be able to scale the sourcing of information in a pretty straight forward fashion. Grab the source down to local. Then from there, try to upload it to the database, and resolve any discrepancies that may be reported on the loading of the data.

Source

• pull data from the source to 1 or more local .json files

Local Store

• grab the local files and try to load them against 1 or more tables. write validation and exception logic.

Database

• show the results in a report format and allow for: Create, Update, Delete, Resolve