Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly

Sign up
Appearance settings

Add batchSize option to RedisItemReader for MGET optimization #4941 #4948

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Conversation

Copy link

@khj68 khj68 commented Aug 16, 2025

Summary

This PR addresses issue #4941 by adding a batchSize option to RedisItemReader that enables batch fetching using Redis MGET operations, significantly improving performance by reducing network round-trips.

Changes

  • Added parameter to RedisItemReader with default value of 1 (backward compatible)
  • Implemented batch fetching logic using Redis MGET when batchSize > 1
  • Added internal buffering mechanism using Queue for efficient data handling
  • Updated RedisItemReaderBuilder to support batchSize configuration
  • Added comprehensive test cases for batch reading functionality

Performance Impact

  • Before: N keys require N network round-trips (individual GET operations)
  • After: N keys require N/batchSize network round-trips (MGET operations)
  • Example: 1000 keys with batchSize=100 reduces network calls by 90% (10 vs 1000)

Backward Compatibility

  • Default batchSize = 1 preserves existing behavior (100% backward compatible)
  • Existing code continues to work without any changes
  • batchSize > 1 enables optimized batch processing

Testing

  • Added test for batch reading with multiple batches
  • Added test for handling null values in batch results
  • Verified backward compatibility with existing tests
  • All existing tests pass without modification

Usage Example

// Backward compatible (existing behavior)
RedisItemReader<String, Object> reader = new RedisItemReaderBuilder<String, Object>()
 .redisTemplate(template)
 .scanOptions(scanOptions)
 .build(); // Uses batchSize = 1
// Optimized batch processing
RedisItemReader<String, Object> optimizedReader = new RedisItemReaderBuilder<String, Object>()
 .redisTemplate(template)
 .scanOptions(scanOptions)
 .batchSize(100) // Process 100 keys per MGET
 .build();

Fixes #4941

- Add batchSize parameter to RedisItemReader with default value of 1
- Implement batch fetching using Redis MGET for better performance
- Maintain backward compatibility with existing code
- Add comprehensive test cases for batch reading functionality
This enhancement addresses the N+1 problem by reducing network round-trips
when reading multiple keys from Redis. With batchSize > 1, the reader uses
MGET operations instead of individual GET calls.
Resolves spring-projects#4941
Signed-off-by: khj68 <junthewise@gmail.com>
@khj68 khj68 force-pushed the feature/4941-redis-item-reader-batch-size branch from b5bbf0e to 5a1c765 Compare August 16, 2025 14:08
Signed-off-by: khj68 <junthewise@gmail.com>
Copy link
Author

khj68 commented Aug 16, 2025

Thank you for the valuable feedback from @noojung regarding the potential risks of MGET in shared Redis environments.

Addressing the Concerns

1. Redis Single-Thread Blocking

You're absolutely right that MGET with large batch sizes can block Redis. This is why the implementation includes:

  • Default batchSize = 1: No change from current behavior unless explicitly opted in
  • Maximum limit of 1000: Prevents excessive blocking even when configured
  • User control: Administrators can tune based on their specific environment

2. Shared Environment Considerations

For environments where batch and online applications share Redis:

  • Recommended batch sizes:
    • Dedicated Redis: 100-500
    • Shared Redis: 10-50
    • High-latency networks: May benefit from larger sizes despite blocking risk

3. Documentation Enhancement

I suggest adding comprehensive documentation warning about these trade-offs:

/**
 * WARNING: Large batch sizes can block Redis for extended periods.
 * Consider your environment carefully:
 * - Monitor Redis SLOWLOG to detect blocking issues
 * - Start with small batch sizes and increase gradually
 * - In shared environments, prefer smaller batches (10-50)
 */

Alternative Approaches for Future

  1. Pipeline mode: Could be less blocking than MGET
  2. Async processing: Using Lettuce's async API
  3. Dynamic batch sizing: Adjust based on Redis response times

Conclusion

The current implementation provides a safe, opt-in performance optimization that:

  • Maintains 100% backward compatibility
  • Gives users control based on their environment
  • Can provide significant performance gains when used appropriately

The concerns are valid and should be clearly documented, but the feature can still provide value when used with proper understanding of the trade-offs.

DevSeongmin reacted with thumbs up emoji

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Enhance RedisItemReader with batchSize option to optimize N+1 problem via MGET operations
1 participant

AltStyle によって変換されたページ (->オリジナル) /