-
Notifications
You must be signed in to change notification settings - Fork 2.4k
Add batchSize option to RedisItemReader for MGET optimization #4941 #4948
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add batchSize option to RedisItemReader for MGET optimization #4941 #4948
Conversation
- Add batchSize parameter to RedisItemReader with default value of 1 - Implement batch fetching using Redis MGET for better performance - Maintain backward compatibility with existing code - Add comprehensive test cases for batch reading functionality This enhancement addresses the N+1 problem by reducing network round-trips when reading multiple keys from Redis. With batchSize > 1, the reader uses MGET operations instead of individual GET calls. Resolves spring-projects#4941 Signed-off-by: khj68 <junthewise@gmail.com>
b5bbf0e
to
5a1c765
Compare
Signed-off-by: khj68 <junthewise@gmail.com>
Thank you for the valuable feedback from @noojung regarding the potential risks of MGET in shared Redis environments.
Addressing the Concerns
1. Redis Single-Thread Blocking
You're absolutely right that MGET with large batch sizes can block Redis. This is why the implementation includes:
- Default batchSize = 1: No change from current behavior unless explicitly opted in
- Maximum limit of 1000: Prevents excessive blocking even when configured
- User control: Administrators can tune based on their specific environment
2. Shared Environment Considerations
For environments where batch and online applications share Redis:
- Recommended batch sizes:
- Dedicated Redis: 100-500
- Shared Redis: 10-50
- High-latency networks: May benefit from larger sizes despite blocking risk
3. Documentation Enhancement
I suggest adding comprehensive documentation warning about these trade-offs:
/** * WARNING: Large batch sizes can block Redis for extended periods. * Consider your environment carefully: * - Monitor Redis SLOWLOG to detect blocking issues * - Start with small batch sizes and increase gradually * - In shared environments, prefer smaller batches (10-50) */
Alternative Approaches for Future
- Pipeline mode: Could be less blocking than MGET
- Async processing: Using Lettuce's async API
- Dynamic batch sizing: Adjust based on Redis response times
Conclusion
The current implementation provides a safe, opt-in performance optimization that:
- Maintains 100% backward compatibility
- Gives users control based on their environment
- Can provide significant performance gains when used appropriately
The concerns are valid and should be clearly documented, but the feature can still provide value when used with proper understanding of the trade-offs.
Summary
This PR addresses issue #4941 by adding a batchSize option to RedisItemReader that enables batch fetching using Redis MGET operations, significantly improving performance by reducing network round-trips.
Changes
Performance Impact
Backward Compatibility
Testing
Usage Example
Fixes #4941