Handle Paging with RxJava

Question

I'm using Retrofit + RxJava on an Android app and am asking myself about how to handle the API pagination to chain calls until all data is being retrieved. Is something like this:

Observable<ApiResponse> getResults(@Query("page") int page);

The ApiResponse object has a simple structure:

class ApiResponse {
 int current;
 Integer next;
 List<ResponseObject> results;
}

The API will return a next value until is last page.

There's some good way to achieve this? Tried to combine some flatMaps(), but had no success.

• zsxwing: Could you clarify your input and your output? A method signature with its description would be better.

Answer 1

You could model it recursively:

Observable<ApiResponse> getPageAndNext(int page) {
 return getResults(page)
 .concatMap(new Func1<ApiResponse, Observable<ApiResponse>>() {
 @Override
 public Observable<ApiResponse> call(ApiResponse response) {
 // Terminal case.
 if (response.next == null) {
 return Observable.just(response);
 }
 return Observable.just(response)
 .concatWith(getPageAndNext(response.next));
 }
 });
}

Then, to consume it,

getPageAndNext(0)
 .concatMap(new Func1<ApiResponse, Observable<ResponseObject>>() {
 @Override
 public Observable<ResponseObject> call(ApiResponse response) {
 return Observable.from(response.results);
 }
 })
 .subscribe(new Action1<ResponseObject>() { /** Do something with it */ });

That should get you a stream of ResponseObject that will arrive in order, and most likely arrive in page-size chunks.

• Rafael Toledo: Every day I discover a new RxJava operator (concatMap is new for me). Tested and approved :)
• Iñaqui: In this example, is there a way to wait for all the results to come in and then combine the results into a single response that you can subscribe to?
• lopar: toList is probably the easiest, which will emit a single list of all elements once the Observable completes. there are a few other operators for grouping into collections as well, such as reduce.
• lopar: Note that I refined this a bit in another answer: stackoverflow.com/a/29594194/1424355 for handling cases where recursion may blow the stack. Also, it may just be a simpler alternative in general.
• Krzysztof Skrzynecki: "where recursion may blow the stack" and it blows. @lopar I really appreciate your solution, but could you add info that this solution causes serious memory leak? It would be great if you add a link to you another answer, because I didn't notice your last comment in the first place.

Answer 2

Iopar gave a great example.

Just a small addition.
If you want to get all pages in one onNext() call.
It can be helpful when you want to zip this result with one more Observable.
You should write:

private List<String> list = new LinkedList() {
 {
 add("a");
 add("b");
 add("c");
 }
};
int count = 1;
public Observable<List<String>> getAllStrings(int c) {
 return Observable.just(list)
 .concatMap(
 strings -> {
 if (c == 3) {
 return Observable.just(list);
 } else {
 count += 1;
 return Observable.zip(
 Observable.just(list),
 getAllStrings(count),
 (strings1, strings2) -> {
 strings1.addAll(strings2);
 return strings1;
 }
 );
 }
 }
 );
}

Usages:

getAllStrings(0)
 .subscribe(strings -> {
 Log.w(TAG, "call: " + strings);
 });

and you will get:

call: [a, b, c, a, b, c, a, b, c, a, b, c]

Answer 3

I've answered my solution in a similar post: https://stackoverflow.com/a/34378263/143733

The trick or amendment to the solution provided by @Iopar is the inclusion of a 'trigger' Observable that can be emitted by a variety of ways.

In the code I posted, it is emitted once a full page of elements have been processed however it could also occur based on a user clicking a button/scrolling.