-
Notifications
You must be signed in to change notification settings - Fork 1.2k
Add multiple reactive keys exist checker #2918
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add multiple reactive keys exist checker #2918
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I just changed the command to make better readability according to sonarlint rules.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I created a validation method which is for checking if parameterized all keys exist in redis cache. If one of them does not exist, it returns false.
@kutlueren , please check if this function is what you meant to use.
@kutlueren
kutlueren
May 27, 2024
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Hey @AnneMayor
Awesome to see that you added the function which carries out what I suggested. It is actually more straight forward than I initially thought, maybe I got confused in the codebase but nice! I shall try it out as soon as it gets merged.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This implementation calls EXISTS
for each key issuing a command for each element in the list.
Redis' EXISTS
command accepts multiple arguments and returns how many of the keys exist. Our interface should use the existing functionality instead of putting another layer on top of commands with a worse performance behavior.
The issue we have here is however that exists(Publisher<KeyCommand> keys)
and exists(Publisher<List<KeyCommand>> keys)
or exists(Publisher<MultiKeyCommand>> keys)
erase to the same type and we cannot simply introduce such an override.
I suggest to implement Mono<Long> exists(List<ByteBuffer> keys)
directly instead of using default
methods. Also, ReactiveRedisOperations
should be extended with countExistingKeys(...)
similar to RedisTemplate.countExistingKeys(...)
.
78c6c7f
to
cf160e7
Compare
kutlueren
commented
Aug 23, 2024
Hey @AnneMayor, I just wanted to follow up on the topic :) Is this being planned to be merged soon?
Hey @kutlueren ,
Unfortunately I have no feedback from our maintainer🥲
I am still waiting for their feedback.
kutlueren
commented
Aug 23, 2024
Hey @kutlueren , Unfortunately I have no feedback from our maintainer🥲 I am still waiting for their feedback.
@mp911de would you mind taking a look at this one when you have time?
@mp911de ,
Please let me know if this PR has some issues🙏
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I did a review pass and left a comment regarding the design and implementation. Care to have a look?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This implementation calls EXISTS
for each key issuing a command for each element in the list.
Redis' EXISTS
command accepts multiple arguments and returns how many of the keys exist. Our interface should use the existing functionality instead of putting another layer on top of commands with a worse performance behavior.
The issue we have here is however that exists(Publisher<KeyCommand> keys)
and exists(Publisher<List<KeyCommand>> keys)
or exists(Publisher<MultiKeyCommand>> keys)
erase to the same type and we cannot simply introduce such an override.
I suggest to implement Mono<Long> exists(List<ByteBuffer> keys)
directly instead of using default
methods. Also, ReactiveRedisOperations
should be extended with countExistingKeys(...)
similar to RedisTemplate.countExistingKeys(...)
.
I did a review pass and left a comment regarding the design and implementation. Care to have a look?
Thanks a lot. I am going to apply your review within this weekend 👍
Any luck solving this?
I have the same issue. Wanting to check the existence of multiple keys using exists in a single network call, without resorting to Lua, but using Springs own implementation
My current workaround, since the spring API does not support exists with multiple keys yet.
/**
* Executes expired session cleanup in Redis.
*
* Spring Session Redis uses multiple keys per session:
* 1. Hash key (namespace:sessions:{id}) - Stores session data with 35 min TTL
* 2. ZSet entry - Tracks session expiration times for batch cleanup
*
* Process flow:
* 1. Query ZSet for sessions that should be expired
* 2. Access their hash keys to trigger Redis expiration events
* - Redis only guarantees expiration events when keys are accessed
* - Spring Session listens for these events to cleanup related keys
* 3. Remove processed entries from the ZSet
*
* Implementation details:
* - Uses Lua script for efficient multi-key EXISTS checks
* - Handles quoted session IDs in ZSet vs unquoted in hash keys
* - Relies on Redis keyspace events for actual session deletion
* - Bounded elastic scheduling for background processing
*
* @param context Operation boundaries and limits for ZSet query
* @return Mono<Void> completing when cleanup process finishes
* @throws RedisConnectionException if Redis operations fail
*
* @see "https://docs.spring.io/spring-session/reference/configuration/reactive-redis-indexed.html#how-spring-session-cleans-up-expired-sessions"
*/
private fun cleanupExpiredSessions(context: CleanupContext): Mono<Void> =
redisOperations.opsForZSet()
.reverseRangeByScore(redisKeyExpirations, context.range, context.limit)
.collectList()
.flatMap { sessionIds ->
if (sessionIds.isEmpty()) {
logger.debug("No expired sessions found")
Mono.empty()
} else {
logger.debug("Found {} expired sessions to remove", sessionIds.size)
// Prepare session keys (remove quotes around session id in zset)
val sessionKeys = sessionIds.map { sessionId ->
"${sessionProperties.redis?.sessionNamespace}:sessions:${sessionId.trim('"')}"
}.also { keys ->
logger.debug("Checking Redis keys: {}", keys)
}
// First read the hash keys to trigger expiration event
val script = """
local keys = ARGV
local count = 0
for i, key in ipairs(keys) do
count = count + redis.call('EXISTS', key)
end
return count
"""
redisOperations.execute(
RedisScript.of(script, Long::class.java),
listOf(), // KEYS list is empty
sessionKeys
)
.doOnNext { count ->
logger.debug("Accessed {} session hash keys", count)
}
.flatMap { _ ->
redisOperations.opsForZSet()
.remove(redisKeyExpirations, *sessionIds.toTypedArray())
.doOnSuccess { logger.debug("Successfully removed {} expired sessions", sessionIds.size) }
.doOnError { e -> logger.error("Error during removal: ${e.message}") }
}
.then()
}
}
.subscribeOn(Schedulers.boundedElastic())
@dreamstar-enterprises Oh, for real? I am gonna check. Sorry for the late response. Since I have some business, I ended up keeping going this PR again. Will review your issue and give some feedback.
Sorry for being late. I have lots of personal business and I am going to resolve this PR comments until this weekend. Please, keep that in mind.
5b033a1
to
79ee7fd
Compare
Signed-off-by: Anne Lee <anne.lee@buzzvil.com>
79ee7fd
to
38637d8
Compare
@mp911de could you review this PR again for me?
Also, I have a question. Do you think that it would be better if I add this exist method into class ReactiveRedisOperations
?
No worries, some pull requests remain open for quite some time because we have to prioritize sometimes other things and so sometimes, pull requests don't get much attention.
In any case, thank you for your contribution. That's merged and polished now.
Resolved #2883