User Research - How to move on #156

fnetX commented 2024年04月17日 18:16:32 +02:00

Hi all, as announced in the chat, I published the notes I took during recent interviews here: https://codeberg.org/forgejo/user-research/src/branch/main/interviews/2024-04

I recommend everyone to read some minutes, especially the accessibility posts. Filenames try to categorize subjectively to indicate points of interests during the session.

How can we move on? How can we ensure the learnings find their way and actually improve Forgejo?

My proposal:

• more people in the User Research team are welcome, it might be renamed to "UX" to widen the scope a little bit
• as described here forgejo/governance#104 (comment), I propose to update the "User Research" labels to classify areas of interest, e.g. code review, exploring for external users, etc.
• a label "User Research - Needs Feedback" is introduced to indicate that an issue or PR is waiting for feedback
• the UX "team" is trying to provide insights or work on a plan on how to test the proposal
• existing issues are sorted according to their expected gain w.r.t usability. I recommend the following scores:
  • very high gain: Forgejo core team is encouraged to work on this with higher priority
  • high gain: Really good to have this improvement, interested contributors are encouraged to pick this.
  • low gain: questionable gain. Patches are still accepted if someone wants to work on this, but contributors are discouraged to pick this issue.
  • very low gain: The issue is closed and sending a patch for this feature should be discussed first, because maintenance burden might not be worth the effort; or because adding "one more button" could be "one more step to mess" ¹
• new feature requests should go through some research, and are categorized
• if a feature has a wide scope (e.g. "dashboard redesign"), the designs are discussed somewhere else and actionable issues are then returned to the forgejo repository

This workflow will of course be hard to establish, require more people doing the UX/UI/User Research work. Nonetheless, I am motivated. The most important steps, in my opinion, are to have a distinction between issues that are "confirmed / backed" by user research and give them a higher priority in Forgejo.

And I do believe that user testing can help spot problems in the design phase of a feature, either by testing simple mockups (e.g. static html pages) or testing a pull request that is still pending review on the code level, tests etc. A new workflow for contributions with user testing could be:

• a PR is opened, no tests yet, not reviewed yet
• it is decided to do user testing
• technical code review is postponed
• the patch is deployed to a Codeberg or Forgejo test instance, prefilled with relevant data to test the change
• users are observed while interacting with the new feature, and the feedback is implemented in the PR
• once some iterations have been made, the normal PR workflow (technical review, improvements, tests etc) is continued

All of this also needs some better tooling or automation. For example, instead of having researches do live sessions, people could

• open a website ("ux.forgejo.org")
• start a screenshare (which is recorded in the backend)
• perform their steps (with a short and precise instruction what should be done)
• they review the recording and submit it if they agree
• Forgejo User Researches review the recordings, take notes, and destroy them afterwards

Hi all, as announced in the chat, I published the notes I took during recent interviews here: https://codeberg.org/forgejo/user-research/src/branch/main/interviews/2024-04 I recommend everyone to read some minutes, especially the accessibility posts. Filenames try to categorize subjectively to indicate points of interests during the session. How can we move on? How can we ensure the learnings find their way and actually improve Forgejo? My proposal: - more people in the User Research team are welcome, it might be renamed to "UX" to widen the scope a little bit - as described here https://codeberg.org/forgejo/governance/issues/104#issuecomment-1741929, I propose to update the "User Research" labels to classify areas of interest, e.g. code review, exploring for external users, etc. - a label "User Research - Needs Feedback" is introduced to indicate that an issue or PR is waiting for feedback - the UX "team" is trying to provide insights or work on a plan on how to test the proposal - existing issues are sorted according to their expected gain w.r.t usability. I recommend the following scores: - very high gain: Forgejo core team is encouraged to work on this with higher priority - high gain: Really good to have this improvement, interested contributors are encouraged to pick this. - low gain: questionable gain. Patches are still accepted if someone wants to work on this, but contributors are discouraged to pick this issue. - very low gain: The issue is closed and sending a patch for this feature should be discussed first, because maintenance burden might not be worth the effort; or because adding "one more button" could be "one more step to mess" [^1] - new feature requests should go through some research, and are categorized - if a feature has a wide scope (e.g. "dashboard redesign"), the designs are discussed somewhere else and actionable issues are then returned to the forgejo repository [^1]: This could be automated with a nice message explaining the reasoning for this behaviour. As soon as the label is added, a bot explains what this means and closes the issue. This workflow will of course be hard to establish, require more people doing the UX/UI/User Research work. Nonetheless, I am motivated. The most important steps, in my opinion, are to have a distinction between issues that are "confirmed / backed" by user research and give them a higher priority in Forgejo. And I do believe that user testing can help spot problems in the design phase of a feature, either by testing simple mockups (e.g. static html pages) or testing a pull request that is still pending review on the code level, tests etc. A new workflow for contributions with user testing could be: - a PR is opened, no tests yet, not reviewed yet - it is decided to do user testing - technical code review is postponed - the patch is deployed to a Codeberg or Forgejo test instance, prefilled with relevant data to test the change - users are observed while interacting with the new feature, and the feedback is implemented in the PR - once some iterations have been made, the normal PR workflow (technical review, improvements, tests etc) is continued All of this also needs some better tooling or automation. For example, instead of having researches do live sessions, people could - open a website ("ux.forgejo.org") - start a screenshare (which is recorded in the backend) - perform their steps (with a short and precise instruction what should be done) - they review the recording and submit it if they agree - Forgejo User Researches review the recordings, take notes, and destroy them afterwards

earl-warren commented 2024年04月17日 18:29:40 +02:00

This workflow will of course be hard to establish, require more people doing the UX/UI/User Research work. Nonetheless, I am motivated. The most important steps, in my opinion, are to have a distinction between issues that are "confirmed / backed" by user research and give them a higher priority in Forgejo.

I can't stress enough how crucial your work is to the future of Forgejo. Your approach feels right to me and it will have a heavy influence on how I decide to spend my volunteer time.

Being a volunteer, I would respond better to something written a little differently although I think it conveys the same meaning. Of course the wording can be changed to a more managerial tone when paid staff is concerned. But that does not need to be the case in the context of Forgejo:

• existing issues are sorted according to their expected gain w.r.t usability. I recommend the following scores:
  • very high gain: Forgejo contributors are made aware that there is evidence it is an important topic for users.
  • high gain: Really good to have this improvement, interested contributors are encouraged to pick this.
  • low gain: questionable gain. It is a warning to anyone working or discussing about that topic that it will likely not gain much attention.
  • very low gain: The issue is closed and sending a patch for this feature should be discussed first, because maintenance burden might not be worth the effort; or because adding "one more button" could be "one more step to mess" [^1]

> This workflow will of course be hard to establish, require more people doing the UX/UI/User Research work. Nonetheless, I am motivated. The most important steps, in my opinion, are to have a distinction between issues that are "confirmed / backed" by user research and give them a higher priority in Forgejo. I can't stress enough how crucial your work is to the future of Forgejo. Your approach feels right to me and it will have a heavy influence on how I decide to spend my volunteer time. Being a volunteer, I would respond better to something written a little differently although I think it conveys the same meaning. Of course the wording can be changed to a more managerial tone when paid staff is concerned. But that does not need to be the case in the context of Forgejo: - existing issues are sorted according to their expected gain w.r.t usability. I recommend the following scores: - very high gain: Forgejo contributors are made aware that there is evidence it is an important topic for users. - high gain: Really good to have this improvement, interested contributors are encouraged to pick this. - low gain: questionable gain. It is a warning to anyone working or discussing about that topic that it will likely not gain much attention. - very low gain: The issue is closed and sending a patch for this feature should be discussed first, because maintenance burden might not be worth the effort; or because adding "one more button" could be "one more step to mess" [^1]

fnetX commented 2024年04月17日 18:36:30 +02:00

Thank you!

I might want to add: I'd like to experiment changing the workflow for Forgejo. Currently, it works like most other Free Software projects. When someone has a problem or needs something, they suggest how it is done. Feature requests often include detailed expectations on where to add the button, how to name it in the database etc.

I'd love to turn this around: Feature requests explain what someone wants to do. The User Research team tries to learn more about the needs. Afterwards, they create actionable issues that are ready for implementation.

I hope to keep the backlog of open issues low, because time is only spent on issues that got researched, or are really small fixes. If this workflow manifests, I hope that more people would join the user research efforts, because it is the only way to get features into Forgejo. And I believe that understanding how users interact with the software they build will make them better developers in the end.

Thank you! I might want to add: I'd like to experiment changing the workflow for Forgejo. Currently, it works like most other Free Software projects. When someone has a problem or needs something, they suggest **how it is done**. Feature requests often include detailed expectations on where to add the button, how to name it in the database etc. I'd love to turn this around: Feature requests explain what someone wants to do. The User Research team tries to learn more about the needs. Afterwards, they create actionable issues that are ready for implementation. I hope to keep the backlog of open issues low, because time is only spent on issues that got researched, or are really small fixes. If this workflow manifests, I hope that more people would join the user research efforts, because it is the only way to get features into Forgejo. And I believe that understanding how users interact with the software they build will make them better developers in the end.

fnetX commented 2024年04月17日 18:39:13 +02:00

... And: The User Research team should be bound to priorities assigned by the wider Forgejo community. They should not be the only team in control over the general direction of the project. For example: The Forgejo community decides what should be improved first: Dashboard? Notifications? Moderation features?

The User Research team then picks their choice and defines priorities inside this scope, e.g. x is needed the most to improve the dashboard, y is not that important etc etc.

... And: The User Research team should be bound to priorities assigned by the wider Forgejo community. They should not be the only team in control over the general direction of the project. For example: The Forgejo community decides what should be improved first: Dashboard? Notifications? Moderation features? The User Research team then picks their choice and defines priorities inside this scope, e.g. x is needed the most to improve the dashboard, y is not that important etc etc.

earl-warren commented 2024年04月17日 18:44:18 +02:00

In my view the User Research team makes recommendation, grounded in evidence. They carry a lot of weight by nature, they are an invaluable tool to stop endless debates based on opinions only. But they are in no way binding and will never be, this is a different kind of decision that requires the governance decision process.

I'd love to turn this around: Feature requests explain what someone wants to do. The User Research team tries to learn more about the needs. Afterwards, they create actionable issues that are ready for implementation.

Here is an actionable idea: pick one of the 200+ features request and attempt to apply that idea to this particular feature request. The person who proposed the feature may welcome the attention and the method. I know I would.

In my view the User Research team makes recommendation, grounded in evidence. They carry a lot of weight by nature, they are an invaluable tool to stop endless debates based on opinions only. But they are in no way binding and will never be, this is a different kind of decision that requires the governance decision process. > I'd love to turn this around: Feature requests explain what someone wants to do. The User Research team tries to learn more about the needs. Afterwards, they create actionable issues that are ready for implementation. Here is an actionable idea: pick one of the 200+ features request and attempt to apply that idea to this particular feature request. The person who proposed the feature may welcome the attention and the method. I know I would.

fnetX commented 2024年06月19日 11:53:09 +02:00

The following was done:

• the User Research labels were cleaned up. Some generic ones were archived, some new ones were introduced to sort issues according to new areas of interest for user research
  • there is now a label "User Research - Needs Input" to ask for feedback. An additional ping would be good, because I only check the issues there from time to time.
  • there are four new classifier labels: "User Research: Classified Gain/x"
• a new issue was created to conclude the open aspect and create a proposal for general future design work and feature requests: #178

The remaining open aspects of this issue are:

• if and how to ease the user research via screen recordings or similar
• how to integrate User Research during open PRs

But I'd agree to closing the issue in case no further discussion is about to happen here for the time being.

The following was done: - the User Research labels were cleaned up. Some generic ones were archived, some new ones were introduced to sort issues according to new areas of interest for user research - there is now a label "User Research - Needs Input" to ask for feedback. An additional ping would be good, because I only check the issues there from time to time. - there are four new classifier labels: "User Research: Classified Gain/x" - a new issue was created to conclude the open aspect and create a proposal for general future design work and feature requests: https://codeberg.org/forgejo/discussions/issues/178 The remaining open aspects of this issue are: - if and how to ease the user research via screen recordings or similar - how to integrate User Research during open PRs But I'd agree to closing the issue in case no further discussion is about to happen here for the time being.