How to handle pagination logic in a layered architecture without making premature decisions?

Question

I'm working with a layered architecture and I'm unsure where to place the pagination logic. This answer suggests that it's not a domain concern, which I think makes sense. In most cases, pagination seems to be motivated by the presentation layer (e.g., displaying a limited number of items on a screen) and then accommodated by the data access layer (e.g., retrieving data in chunks) or vice versa. It makes sense for the domain layer to stop its concern at "show a list of X to the user."

However, the author points towards the application layer as the solution. I have some concerns with this approach:

1. Leakage of constraints: In a world with unlimited resources, why would an application choose to limit the number of items shown to the user? User experience is a possible reason, but that seems more like a presentation concern than a business rule. The constraint originates from external factors.
2. Premature decision: One goal of layered architecture is to defer decisions by treating change as a first-class citizen. Implementing pagination at the application level solidifies this choice, forcing outer layers to conform, regardless of their needs.

To address these concerns, I'm exploring handling pagination where it becomes relevant. My current approach involves Iterable entities and "paging" presenters.

Sequence diagram of Iterable entities, the interactor, and the "paging" presenter.

However, this introduces a potential issue: the domain may lose control over the flow of events, particularly when errors occur during subsequent page loading. In such cases, the domain remains unaware of the error since the "Load More" action originates in the presentation layer, while the error occurs in the data access layer.

This leads me back to the question of whether pagination should be handled entirely outside the domain. However, I believe my concerns are valid. Are there alternative approaches to pagination in a layered architecture that address these concerns? Or are these trade-offs unavoidable?

• Rik D: 9/10 times (made up statistic) the need for pagination is a sign of bad UI/UX design. Users don’t want to browse several pages to get to the thing they need. Before diving into technical solutions, talk to your users about how you can present the data they need, without displaying it in a paginated list.
• Hadi Satrio: Regardless of whether it's a good UI/UX design, I think it's a common enough problem in user-facing applications to be discussed.
• Doc Brown: @RikD: the typical approach to avoid pagination at the UI level is to provide a filter mechanism to the user which lets them reduce a large list of items (>1000) to a number which is small enough to be handled by scanning visually (<20). Unfortunately, sometimes the data/filter combination returns a number of items like 100~200, from where the user may decide not to fiddle out a better search criteria, but still want to scan visually. Or in other words: pagination can be a valid concern, and this question is definitely worth to be answered.
• Doc Brown: @HadiSatrio: I still I have some trouble to understand the question, especially the "domain may lose control over the flow of events, particularly when errors occur ....". You implement things using iterables and exceptions. In case an error occurs in the data access layer during some iteration, it generates an exception which will bubble up through the intermediate layers, which can ignore it, catch and process it, or catch & process it partially and rethrow it. Where in this scenario does the domain may lose control over the flow of events?
• Hadi Satrio: @DocBrown, in subsequent loads, the presenter would indirectly talk to the entity through Iterator#next hence bypassing the interactor. Perhaps I should have wrote "application" rather than domain, apologies for the confusion.

Answer 1

From what you wrote, I guess your "current approach with iterables" has simply a suboptimal implementation. I am most fluent with C#, where "Iterables" are called "IEnumerable", but I guess you will understand my examples and can transfer it to a different language. You said in the comments

in subsequent loads, the presenter would indirectly talk to the entity through Iterator#next hence bypassing the interactor.

and the sequence diagram in the question illustrates this.

This would be the case when your implementation looks like this:

// Entity
IEnumerable<MyData> GetManyRecords()
{
 // ...
 // uses chunks under the hood, 
 // may throw SomeException when a new chunk is loaded
}
// Interactor 
IEnumerable<MyData> GetManyRecords()
{
 try
 {
 // lazy evaluation will prevent an exception to be thrown here
 return entity.GetManyRecords();
 }
 catch(SomeException ex)
 {
 // does not catch what it should 
 // ...
 }
}
// Presenter:
var records = interactor.GetManyRecords();
// ...
// inside some loop over pages
try
{
 var pageRecords = records.Take(pageSize);
 // ... display pageRecords ...
 records = records.Skip(pageSize);
}
catch(...)
// ...

(more examples: SO question "Paging with LINQ for objects").

This implementation indeed lets exceptions from the entity directly bubble up to the presenter, and the exception handling inside the interactor is useless. However, this can easily be fixed by implementing the interactor differently:

// Interactor - implementation with working exception handling
IEnumerable<MyData> GetManyRecords()
{ 
 foreach(var record in entity.GetManyRecords())
 {
 try
 {
 yield return record;
 }
 catch(SomeException ex)
 {
 // ...
 }
 
 }
}

This way, the interactor keeps full control about what happens in case of an error, though it has actually no knowledge about both kinds of pagination, neither the chunk loading at the lower layers, not the UI pagination at the Presenter layer.

Of course, in case you want the presenter layer to directly deal with exceptions from the lower layers, then one can use the first implementation (and strip the useless try-catch out of it).

TLDR; Iterables with lazy evaluation allow you to keep pagination mechanics at the layer where it belongs to - you may just have to add another layer of indirection to make this work correctly.

• Hadi Satrio: This approach solves the problem beautifully, thank you so much! In Java/Kotlin, one can apply Doc's suggestion here in the form of ExceptionCatchingIterable which takes in a delegate and a callback/lambda to execute should the delegate's hasNext() throws.

Answer 2

Pagination logic comes in two forms.

• Data access
• Presentation

If a DB is only capable of loading 1000 rows at a time you must not exceed that. But if your display can only handle 12 at a time...

What you need to do is express the paging requirements from where they each originate and accept that they don't all originate in one place.

Don’t assume the 12 row limitation of your presentation will protect the DB. Presentations change.

• Greg Burghardt: And performance. If the database limits results to 1000 rows, but a particular query makes the database pass out, then pagination is not just for presentation purposes.
• Hadi Satrio: I agree to both points. But they are outer layer constraints, still. Presentations change indeed, so does DBs. And when their substitutes somehow resolve the need of pagination, ideally it should also go away. This won't happen if we implement it as part of the application layer.
• gnasher729: I’ve seen cases where pagination was caused by someone gaming the system. Looking for a hard drive with reasonable criteria, some manufacturer had the same hard drive in 300 colors, and sold 300 "hard drives for computer x", 300 identical "hard drives for computer y" and so on. That problem is not solved with pagination.