Approach for implementing access permission tests for a large number of actions and multiple affiliations

Question

I'm obviously having trouble creating a question that fits StackExchange guidelines in regard of opinions vs metrics. Any help to improve this question is highly appreciated.

I'm searching for a suitable approach to implement tests for access permission checks.

I'm writing an API backend (should be irrelevant). The system has 150 actions (= API endpoints) (= a lot) - mostly CRUD for various resource types (example: groups, users, documents, ...).

The system has concepts like tenants, groups, users, services (groups are affiliated with services) and permissions (e.g. permission.documents.list, permission.documents.read).

A naive approach for implementing tests would be to implement each use case for each action (= API endpoint). That would leave me with (roughly) 16*150 = 2400 tests to manually write and maintain. For each new API 16 new tests have to written.

That leaves plenty of room for forgotten or misimplemented test cases.

Question

What is the common approach for implementing tests for access permission checks for a system with many actions and multiple affiliation combinations (tenant, group), while keeping it maintainable**.

**maintainable :=

1. don't implement every use case for every action by hand.
2. When permission management policy changes, all 2400 cases should be updatable by one person within 50 hours.
3. Bugs in the test code should be easily findable (i.e. test itself fails or obvious by reading the code).

+α constraint: The "2400 tests" should be resilient to changes in the underlying affiliation structure. I.e. if the concept of groups disappear from the system, I want to rewrite only a small part of the tests.

If you don't have an answer, but a good reference on the topic, I'd appreciate mentioning it in the comments.

My own thoughts

My current testing pattern looks like this:

• create user/group/.. data
• create resource for user2
• create resource for user1
• result = callAPI(action, as=user1)
• expect(result).toNotContain(resoures of user2)

I understand that I need to centralize the part of the test that does all the affiliation stuff (tenant/group/user/service). I also think I should centralize the "16" testcases. I think each test case would then only provide action-specific test data to be created. Perhaps something like testAll16PermissionTests(api="/documents.list", resources=...). However, I'm not sure how that would look like specifically and if that's even the common approach for this kind of situation.

• Greg Burghardt: We have no objective criteria or metric to measure "common" so I'm afraid that still means it is likely opinion-based.
• DarkTrick: @GregBurghardt Thank you for the feedback! Highly appreciated. Can you help me on how I can state my question without making it opinion-based?
• Greg Burghardt: This can be tricky, because the question is so open-ended. I think you have clearly stated the situation, we just need to focus the question so we can answer it using facts. Even "maintainable" is largely left up to interpretation. Maybe start by refining what you mean by "maintainable" in this situation?
• DarkTrick: Tricky to pin down. Perhaps maintainable := (1) don't implement every use case for every action by hand. (2) When permission management policy changes, all 2400 cases should be updatable by one person within 50 hours. (3) Bugs in the *test* code should be easily findable (i.e. test itself fails or obvious by reading the code). . (I guess obvious is a non-metric. I don't see how I can get rid of it atm)
• Greg Burghardt: That is good information. Can you edit your question to include this?

Answer 1

The problem with access permission tests is that you are testing static configuration. It's like testing an enum has the right set of values, your test lacks value because it's just writing the same thing twice.

If you have to test these things I would use reflection to pull back the permissions on each endpoint, sort, serialise, save to file, compare against known good (human checked) output.

• DarkTrick: (1) Clarification: You would not write automated tests for any permission checks? (2) Clarification: Instead of testing you would analyze the source code? (3) Going from your logic, arent all tests just testing static configuration ?
• Ewan: 1: not in the way you describe, maybe run a security scan, 2: the kind of test I describe analyses the source code i suppose? 3: you really want to test your logic, not the configuration of third party libs. obvs there is some crossover

Answer 2

In essence, you have two things you want to verify:

1. Does the permission system work as it should?
2. Does endpoint E use the permission system (correctly)?

The extent to which you can answer those questions independently depends mostly on to what extent the business logic of the endpoint can be separated from the permission logic.

If they are really interwoven, then you are stuck with testing each combination and you may find that you cannot even write a generic testAll16PermissionTests helper.

On the other hand, if they are clearly separated (for example, the permission system uses the user, group, etc. to determine a permission level and the endpoint only makes decisions based on that permission level), then you can create independent sets of tests:

• One set of tests per endpoint to verify the correct behavior for each permission level (one test per level)
• One set of tests to exercise the logic of the permission module, to verify the correct permission level is calculated for each combination of inputs and configuration. Preferably this is done as a component test, but if your test setup doesn't allow for that, you can choose one endpoint to run those tests against.

• DarkTrick: Question: Can you clarify the meaning of "permission level"?
• Bart van Ingen Schenau: @DarkTrick, 'permission level' could be something like "regular user" (can see only their own items); "manager" (can see items of subordinates); "administrator" (can see everything).