Designing storage service data structure for decoupled models sharing same data

Question

Surely most of you remember the Norton Commander application where similar (sometimes the same) data is displayed in separate decoupled views.

I'm building a web application that follows the same principle. The amount of data is too big to be fetched/displayed entirely on the page, and potentially overlapping subsets of data need to be displayed in separate views.

My architecture looks like this:

 ┌──────── EventBus ────────┐
 │ │ │
 │ StorageService │
 │ ╱ ╲ │
 PresenterLeft┤ ├PresenterRight
 ╱ ╲ ╲ ╱ ╱ ╲ 
 ViewLeft ModelLeft ModelRight ViewRight

The application creates both presenters and passes them the instance of EventBus and StorageService. The presenters, in their turn, create their respective models and views. The StorageService is passed down to models. The communication between presenters is done via the EventBus.

In this scenario the left side has no awareness of the right side, yet models should possess the same information without a need to do a double round trip to the server.

To achieve this I use the StorageService. Models implement the model.fetch() method which asks the storage for data. If the same data was previously requested, the storage will simply return it acting like an in-memory cache.

Here comes the actual question. In the above context, how to design a data structure for a network-wise efficient storage service for the following use case:

1. Left model requests GET /files?limit=100
2. Right model requests GET /files?limit=110 (but actually I only need the last 10)

My attempted approach was:

• store the requests made by models
• for every additional request, do my best to figure out the request diff
• do the diff request (e.g. GET /files?offset=100&limit=10)
• store the fetched items under {"files": {"id1": item1, ..., "id110": item110}}

Unfortunately the above steps don't help, because there's no relation between the url and the ids. When a model requests the next set of values, I never know if they are actually in the in-memory cache.

Of course, I could use the per-url in-memory caching strategy, but then I would have duplicate entries in the cache, which is something I'd like to avoid.

What are your thoughts? I feel like I'm reinventing the wheel here. Is there a better/simpler/commonly-accepted method? Perhaps my architecture is flawed...

• gnat: Unclear what help you need. Please clarify your specific problem or provide additional details to highlight exactly what you need. As it's currently written, it’s hard to tell what problem you are trying to solve or what aspect of your approach needs to be corrected or explained. See the How to Ask page for help clarifying this question.
• Oleg: Made some adjustments to the formulation of the question. Not sure if it's sufficient...
• theinsaneone: From the problem I understood from your question, I feel your architecture needs a framework to store and manage data locally and temporarily. I believe Document-oriented database (en.wikipedia.org/wiki/Document-oriented_database) could be the solution for this. One such framework I know is mongo DB. Try googling it...Ignore if this solution is not relevant.
• albert: how are you structuring your urls? can you show an example or two please?

Answer 1

The urls should just map internally to some call to a repository that implements the caching behavior you want.

LeftViewModel should call IFileRepository getRecentFiles(100) through /files?limit=100
RightViewModel should call IFileRepository getRecentFiles(110) through /files?limit=110

Then create something like:

CachingFileRepository implements IFileRepository
 depends on another IFileRepository for permanent storage
 function getRecentFiles (limit)
 if cacheSize < limit
 get the rest of the files from the permanent repository
 add the files to the cache
 return the files

If you want to have 1:1 class to url mapping, you should have more urls (one more for the permanent storage, use the existing one for the cache).

If you want to have one url for the service, you should have more classes behind that one url.

---

EDIT: In addition, if you want to do this without two roundtrips to storage, you could keep track of the views you are spawning, and request model data once after you know what the models need. Something like

BaseFileModel
 limit = 0
 require()
 storageService->requireLimit(limit)
 populate()
 storageService->getFiles(limit)
ModelLeft extends BaseFileModel
 limit = 100
ModelRight extends BaseFileModel
 limit = 110
Application
 createModels() //creates ModelLeft and ModelRight, sending storageService and calling populate
 storageService->getRemoteData() - one call to get data
 populateModels()
 renderViews()