-
Notifications
You must be signed in to change notification settings - Fork 666
Comments
FEAT analytics: analyze attack success rate with generic dimensions#1362
FEAT analytics: analyze attack success rate with generic dimensions #1362romanlutz wants to merge 7 commits intoAzure:main from
Conversation
Co-authored-by: romanlutz <10245648+romanlutz@users.noreply.github.com>
Co-authored-by: romanlutz <10245648+romanlutz@users.noreply.github.com>
Co-authored-by: romanlutz <10245648+romanlutz@users.noreply.github.com>
...d-asr-by-converter-type
...analytics_asr_by_generic_dimensions
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
should we check whether there are repeats in group_by? Fringe case but counts could be off in that specific case.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
should we warn if group_by = [] but custom_dimensions is non-empty?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
maybe not this PR, but should we have a way to pretty print the result?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I might be going down a rabbithole but I'm imagining that if there is an attack (attack_type_x) with multiple converters (let's say c1 and c2), then this cross product will include (c1, attack_type_x) and (c2, attack_type_y) (and also get more complicated if we have, say, custom dimensions that are one to many). I wonder if that's a bit misleading to a user because it wasn't really c1 attack_type_x that led to a specific result, it was the combination of both converters with attack_type_x. Maybe combinations of converters or other dimensions used in one attack should be treated as one entity?
Description
Generalizing
analyze_resultsto work with attack type, converter type, labels, or any dimension we can define from attack results.Tests and Documentation
unit tests and new notebook