Managing large-scale change: user research session 1 proposal #65

mfenniak commented 2025年07月08日 23:26:19 +02:00

This PR adds the first section of a user research project, which includes a project description and a plan for a first research session, in the area of managing large-scale software change with Forgejo.

As I understand (from #49) there isn't a clearly documented user research process right now. My thought process is that I'll get review on this document and research session plan, and this PR can be merged when ready to execute the first research session. A follow-up PR will be introduced that includes data from the first research session, and likely additional PRs that could include follow-up research sessions and their data. If there's a different approach preferred, please let me know and I'm happy to adapt.

For clarity; I intend to undertake the work related to this user research. I know that with an Open Source project it isn't always clear where a contributor is planning to start and end, but this is an area that I'm interested in taking end-to-end to implementation (assuming, of course, that the results of diligent and bias-free user research justify a complete work implementation). However, support in reviewing the research proposal and later publicizing the survey through social media would be welcomed greatly.

Related links:

• Design PR for a "Pull Request Stack" feature -- a design inspired only by personal experience and desk research, which I'm hoping will find alignment with this user research proposal.

This PR adds the first section of a user research project, which includes a project description and a plan for a first research session, in the area of managing large-scale software change with Forgejo. As I understand (from #49) there isn't a clearly documented user research process right now. My thought process is that I'll get review on this document and research session plan, and this PR can be merged when ready to execute the first research session. A follow-up PR will be introduced that includes data from the first research session, and likely additional PRs that could include follow-up research sessions and their data. If there's a different approach preferred, please let me know and I'm happy to adapt. For clarity; I intend to undertake the work related to this user research. I know that with an Open Source project it isn't always clear where a contributor is planning to start and end, but this is an area that I'm interested in taking end-to-end to implementation (assuming, of course, that the results of diligent and bias-free user research justify a complete work implementation). However, support in reviewing the research proposal and later publicizing the survey through social media would be welcomed greatly. Related links: - [Design PR for a "Pull Request Stack" feature](https://codeberg.org/forgejo/design/pulls/48) -- a design inspired only by personal experience and desk research, which I'm hoping will find alignment with this user research proposal.

earl-warren commented 2025年07月09日 00:47:06 +02:00

@mfenniak wrote in #65 (comment):

As I understand (from #49) there isn't a clearly documented user research process right now.

Indeed. This is still very new despite being around for years. I still struggle to find a balance between too heavy handed (weeks of work) and nothing (which is currently how things are done most of the time, including by me).

I actually thought about it quite a lot today in the context of re-organizing the Forgejo runner issue tracker and I'm more and more convinced that at the very minimum what should be done is collecting first hand experience that relate to the problem. User research, in all its forms, is a set of tools and techniques to derive sensible conclusions from those first hand experiences. And currently there are very few.

@mfenniak wrote in https://codeberg.org/forgejo/user-research/pulls/65#issue-1921457: > As I understand (from #49) there isn't a clearly documented user research process right now. Indeed. This is still very new despite being around for years. I still struggle to find a balance between too heavy handed (weeks of work) and nothing (which is currently how things are done most of the time, including by me). I actually thought about it quite a lot today in the context of re-organizing the Forgejo runner issue tracker and I'm more and more convinced that at the very minimum what should be done is collecting first hand experience that relate to the problem. User research, in all its forms, is a set of tools and techniques to derive sensible conclusions from those first hand experiences. And currently there are very few.

earl-warren commented 2025年07月09日 00:48:50 +02:00

However, support in reviewing the research proposal and later publicizing the survey through social media would be welcomed greatly.

I'll happily review and support advertising the effort. I suspect @fnetX and a few other people will be similarly interested 😁

> However, support in reviewing the research proposal and later publicizing the survey through social media would be welcomed greatly. I'll happily review and support advertising the effort. I suspect @fnetX and a few other people will be similarly interested 😁

mfenniak commented 2025年07月09日 03:42:55 +02:00

Indeed. This is still very new despite being around for years. I still struggle to find a balance between too heavy handed (weeks of work) and nothing (which is currently how things are done most of the time, including by me).

I can see that this would be a tricky balance. I'm thinking of it like this, right now: 1 user interview is ∞% more information. Incrementally each piece of data after that is 100% more information, then 50% more, etc. it's easy to have a mental model that there is "low hanging fruit" easily collected in the research, and very quickly "diminishing returns" where you mostly repeat the same responses.

So there's probably the most value in doing something, and then iterating to either conclusions or new research questions quickly.

Another way of putting it is that there's a lot of value in being confident about the problems faced by 100% of users, or 50% of users, which would be easily surfaced. But there's very little value in collecting enough information so that we can find out problems that affect 10% or 1% of users.

> Indeed. This is still very new despite being around for years. I still struggle to find a balance between too heavy handed (weeks of work) and nothing (which is currently how things are done most of the time, including by me). I can see that this would be a tricky balance. I'm thinking of it like this, right now: 1 user interview is ∞% more information. Incrementally each piece of data after that is 100% more information, then 50% more, etc. it's easy to have a mental model that there is "low hanging fruit" easily collected in the research, and very quickly "diminishing returns" where you mostly repeat the same responses. So there's probably the most value in doing something, and then iterating to either conclusions or new research questions quickly. Another way of putting it is that there's a lot of value in being confident about the problems faced by 100% of users, or 50% of users, which would be easily surfaced. But there's very little value in collecting enough information so that we can find out problems that affect 10% or 1% of users.

earl-warren commented 2025年07月09日 07:52:23 +02:00

I think it says in https://jdittrich.github.io/userNeedResearchBook/ that it is ok to not have many participants. Not everybody can do quantitative user research.

What I've seen done successfully in the past is to be the one collecting the data (tedious) and asking other people who know the field but are not invested in the solution you have in mind to draw the conclusions. They will point to unpleasant facts that contradict your assumptions where you would be more inclined to either lower their impact or dismiss them as not relevant.

And there only needs a few actual user interviews to do that, provided you choose the participants well.

In the case of the feature you have in mind, I would look one or two "power user of pull requests" and one or two "basic user of pull requests" if you see what I mean.

I'm willing to be one of the "power user of pull requests" and set aside time to answer your questions, preferably in writing so I get the time I need to think.

I think it says in https://jdittrich.github.io/userNeedResearchBook/ that it is ok to not have many participants. Not everybody can do quantitative user research. What I've seen done successfully in the past is to be the one collecting the data (tedious) and asking other people who know the field but are not invested in the solution you have in mind to draw the conclusions. They will point to unpleasant facts that contradict your assumptions where you would be more inclined to either lower their impact or dismiss them as not relevant. And there only needs a few actual user interviews to do that, provided you choose the participants well. In the case of the feature you have in mind, I would look one or two "power user of pull requests" and one or two "basic user of pull requests" if you see what I mean. I'm willing to be one of the "power user of pull requests" and set aside time to answer your questions, preferably in writing so I get the time I need to think.

fnetX commented 2025年07月10日 17:21:43 +02:00

there isn't a clearly documented user research process right now

There is a documented process about the work that I have done in the past. It worked very well to learn about user needs in Forgejo and I plan to continue doing so.

I find your approach interesting. However, I'm not a big fan of checking in AI-generated noise into this repository. My personal aim for this repository is to deliver raw data and conclusions that can be consumed by Forgejo contributors. Having the repo containing LLM noise contradicts this idea, especially because the proportion seems wrong. I acknowledge that you evaluate your survey question with the help of LLM, but I'm not sure if it really needs to stick here.

My primary concern with regards to your proposal is that you won't find a lot of data. I have rarely seen people answer long surveys. LLMs can do this, because they are good at generating walls of texts. Real humans will rarely do so, especially when a set of questions is not actionable for them.

From my (admittedly also limited) experience, user research needs to be a good mix of quantitative and qualitative research. If you do a survey, do a short one. Allow people to submit a lot of text, but try not to force them to do so. Additionally, do qualitative reserach, sit down with people and observe their behaviour. You can do interviews as live sessions, using conversations or observing the publicly visible artifacts of their work (e.g. finding large projects using Forgejo and working through their pull reviews).

If you are interested to proceed using your route, I suggest you to try and find participants before fine-tuning the survey. You'll get a better idea about the potential quantity (I fear there will be a low single-digit number of potential candidates, and doing manual conversation might be easier than tuning a survey to be perfect).
If you are interested in doing it "my way", I propose we find participants that are interested in a live research session. We would ask them some questions, but also observe them doing code review using screen sharing.

> there isn't a clearly documented user research process right now There is a documented process about the work that I have done in the past. It worked very well to learn about user needs in Forgejo and I plan to continue doing so. I find your approach interesting. However, I'm not a big fan of checking in AI-generated noise into this repository. My personal aim for this repository is to deliver raw data and conclusions that can be consumed by Forgejo contributors. Having the repo containing LLM noise contradicts this idea, especially because the proportion seems wrong. I acknowledge that you evaluate your survey question with the help of LLM, but I'm not sure if it really needs to stick here. My primary concern with regards to your proposal is that you won't find a lot of data. I have rarely seen people answer long surveys. LLMs can do this, because they are good at generating walls of texts. Real humans will rarely do so, especially when a set of questions is not actionable for them. From my (admittedly also limited) experience, user research needs to be a good mix of quantitative and qualitative research. If you do a survey, do a short one. Allow people to submit a lot of text, but try not to force them to do so. Additionally, do qualitative reserach, sit down with people and observe their behaviour. You can do interviews as live sessions, using conversations or observing the publicly visible artifacts of their work (e.g. finding large projects using Forgejo and working through their pull reviews). If you are interested to proceed using your route, I suggest you to try and find participants before fine-tuning the survey. You'll get a better idea about the potential quantity (I fear there will be a low single-digit number of potential candidates, and doing manual conversation might be easier than tuning a survey to be perfect). If you are interested in doing it "my way", I propose we find participants that are interested in a live research session. We would ask them some questions, but also observe them doing code review using screen sharing.

earl-warren commented 2025年07月10日 17:45:44 +02:00

FWIW I'm fine with how @mfenniak is approaching this, although it is different from what I've done in the past. Or from what you've done in the past. At this stage I think what matters most is that we each walk our own path. And once we've done that, after we've completed our own personal first user research, sharing and cooperating will be more meaningful.

It would be different if one of us could claim any real expertise beyond what we've learned in the past year or so.

FWIW I'm fine with how @mfenniak is approaching this, although it is different from what I've done in the past. Or from what you've done in the past. At this stage I think what matters most is that we each walk our own path. And once we've done that, after we've completed our own personal first user research, sharing and cooperating will be more meaningful. It would be different if one of us could claim any real expertise beyond what we've learned in the past year or so.

mfenniak commented 2025年07月10日 19:38:07 +02:00

I'm not a big fan of checking in AI-generated noise into this repository.

Understandable; I've removed that content. With respect to documenting the process of this research I've kept a small note about their use during survey evaluation.

My primary concern with regards to your proposal is that you won't find a lot of data.

I'm aligned with this concern. On the other hand, if we compare the barriers-to-entry between "schedule and perform a synchronous interview" versus "receive a survey and write a response", I'm not sure that it's a foregone conclusion that interviews would provide more data than surveys. As a data-point-of-one, note @earl-warren's comment "I'm willing ..., preferably in writing so I get the time I need to think."

... user research needs to be a good mix of quantitative and qualitative research.

I'm afraid there may be a (I promise, good-faith) terminology disconnect here that prevents me from fully understanding this comment. In my understanding, a survey with free-form responses is qualitative research, not quantitative research. I'm not intending to turn the responses into numbers, statistics, etc. where they are analyzed at scale, which are the factors that I would associate with the term "quantitative".

I do understand from your comment that you believe observation would be a valuable factor in user research, rather than descriptive responses, as people's actual behavior and described behavior are often different. Assuming I'm understanding this correctly, I think that's a great insight for me to incorporate... but it sure seems like it makes finding participants 10x more difficult, as they'd also have to have a "large scale change" that they were ready to review, and the content of that review would have to be non-confidential such that it can be shared to an outsider of that project.

Perhaps a practical step towards observational research would be to follow-up with survey participants about their willingness to take another step?

I propose we find participants that are interested in a live research session.

I'd certainly be open to this; what is your method for finding participants?

> I'm not a big fan of checking in AI-generated noise into this repository. Understandable; I've removed that content. With respect to documenting the process of this research I've kept a small note about their use during survey evaluation. > My primary concern with regards to your proposal is that you won't find a lot of data. I'm aligned with this concern. On the other hand, if we compare the barriers-to-entry between "schedule and perform a synchronous interview" versus "receive a survey and write a response", I'm not sure that it's a foregone conclusion that interviews would provide more data than surveys. As a data-point-of-one, note @earl-warren's comment "I'm willing ..., preferably in writing so I get the time I need to think." > ... user research needs to be a good mix of quantitative and qualitative research. I'm afraid there may be a (I promise, good-faith) terminology disconnect here that prevents me from fully understanding this comment. In my understanding, a survey with free-form responses is qualitative research, not quantitative research. I'm not intending to turn the responses into numbers, statistics, etc. where they are analyzed at scale, which are the factors that I would associate with the term "quantitative". I do understand from your comment that you believe *observation* would be a valuable factor in user research, rather than descriptive responses, as people's actual behavior and described behavior are often different. Assuming I'm understanding this correctly, I think that's a great insight for me to incorporate... but it sure seems like it makes finding participants 10x more difficult, as they'd *also* have to have a "large scale change" that they were ready to review, and the content of that review would have to be non-confidential such that it can be shared to an outsider of that project. Perhaps a practical step towards observational research would be to follow-up with survey participants about their willingness to take another step? > I propose we find participants that are interested in a live research session. I'd certainly be open to this; what is your method for finding participants?

earl-warren commented 2025年07月15日 11:37:28 +02:00

---

Before answering the survey questions, I had the following random thoughts.

Federation is a large change. F3 is a large change. Moderation is a large change. Forgejo Actions notifications is a large change.
I'm involved to various degrees in those and they all have different strategies. I will focus on my experience with Forgejo Actions notifications because it is likely closer to what Mathieu Fenniak is interested in. It started with a nicely organized, chronologically ordered set of pull requests by Chris

Another similar example of series of PR is when oliverpool reworked the webhooks early 2024. But that's a more distant memory and I was just a reviewer amazed by how well it all fit together. I could not even begin to imagine how he made it work. It could not fit in my brain, that would be too much.

Another example is how gusted is gradually making security related changes where one can find commonality if you look at those PRs over time. For instance how SSH based alternatives to OpenPGP eventually make their way in the codebase. gusted: approach feels more like an impressionist approach to series of pull requests. Touches here and there that are not necessarily related but if you take a step back, they paint a picture.

F3 is not in scope I think because it is a good example of a feature that has a very simple interface (cp a b) and is best moved to an independent package.

---

Motivations

1. When you're planning a large development effort, what influences whether you tackle it as one change or multiple?

The primary reason for having one large change is because I could not figure out a way to break it down into smaller independent changes. Examples:

• Improving Forgejo migrations (1, 2) requires a complete re-architecture of the package. Instead of making it a (very) large change in Forgejo, it is developed as an independent package F3. It takes a very long time for this package to mature but when it is ready, because the API it provides is simple, the ancient code can be removed in Forgejo and replaced with calls to this API. The scale of the change was shifted out of Forgejo and when the change lands, it will be a single change, not multiple changes.
• Forgejo Actions notifications is an example of a single change that was broken down into multiple changes. I was on the reviewer side of this series but I would have had almost exactly the same approach as the author. It started with code walks to understand where things are and get a mental model of what needs changing. It discovered that bits of infrastructure were missing to implement that change and that made it large instead of small. To not resort to a hack, the author then proceeded to plan for a series of pull request to create the foundations on which the change could stand.

2. Can you tell me about a time when you decided that a software change was 'too big'? What made you realize that?

• Some migrations improvements and bug fixes over a long period of time demonstrated multiple times that the architecture was not good enough to support a number of bug fixes and features. At the time I got involved, it was already clear to everyone that this part of the codebase was stuck and required a large change. An architectural change.
• Forgejo Actions notifications realization came out of exploring the codebase. That is how I discover that a change is not as simple as it should, most of the time.
• A recent example is when fixing a bug. Exploring how this should be properly implemented, I could not help but notice that there were multiple inconsistencies in how environment variables are set. I tried to clean that up a bit and ended up with a large change, changing behaviors that were for a large part not covered by tests but known to work in practice. I found myself stumbling into a large change and came to my senses before being too committed. I changed my approach to fixing the bug in a way that is consistent with the current codebase while not refactoring it. Trying to strike an (unsatisfactory) balance between a hack and a large refactor.

3. What's the most frustrating thing about reviewing large software changes?

• Loosing context when days or weeks pass between two reviews. The larger the change the longer context switching takes when time passes.
• When the author does not explain in plain English the mental model they have in mind and that it needs to be reconstructed by reading the code.
• When the author makes changes that are not explained and the reason for the change is not obvious when reading them.
• When the author resolves a review comment without linking it to the change that resolves it.

Although all those frustrations also exist in small changes, their inconvenience is in proportion of the size of the change.

Activities

4. Referencing your answer from question 2, what did you do when the software change was too big?

• Migration => move the development to a dependency.
• Forgejo Actions notifications => (I'm not the author but I would have done almost exactly that) plan for a logical and chronologically organized series of pull requests. It can be difficult because the first PR is mostly useful in the context of the larger change so it is important that the reviewer understands how it fits in the series.

5. How do you typically organize commits in your changes?

• for small changes, most of the time, I make it a single commit and I rebase the pull request and force push to keep it a single commit.
• for small changes that contain unrelated but interdependent changes, I separate them in different commits so they can be reviewed independently and merged as is instead of squash for forensic analysis in the future.
  Example: https://code.forgejo.org/forgejo/act/pulls/177/commits
• for larger changes that fit in a single pull request I keep a well organized series of commit with titles that tell a story that can be read by the reviewer.
  Examples:
  • https://code.forgejo.org/forgejo/act/pulls/170/commits (I'm the author)
  • forgejo/forgejo#8451/commits (I'm the reviewer)

6. If you had to split a change up into two or more separate changes, what steps would you take?

As suggested by the examples above, it highly depends on the magnitude of the change and the strategies to break it down into manageable / reviewable changes. It also depends on who is involved (author / reviewer). I'll answer in the particular context of a change that requires a series of pull requests that depend on each other.

After exploring the codebase to figure out what needs to be done, I would look for ways to not make them all into a single pull request. If I see that part of the change has value in isolation, I would start with a pull request for that change alone. Ideally it stands on its own and no context needs to be explained. But in the case of Forgejo Actions notifications, the first pull requests (1 and 2) only had meaning in the larger context and explaining it helped.

The most difficult part in engaging in such a series of pull request is that it is taking a risk that something is overlooked and that it is only revealed after the last pull request is merged. In this case it happened as I (as a reviewer) failed to see that a pull request implementing the relationship between two workflow runs was missing. Part of the change had to be reverted and remains incomplete until that is done.

I do not think there is a way to prevent that from happening. It is a bug of sorts, although more of an architectural bug than a simpler behavior bug. Forgejo and other software have a lot of those and they are the most difficult to fix. But I don't think they can be prevented and the key is to not be paralyzed by fear when engaging in a large change. Only be prepared to revisit the change later and not let the problem linger forever.

7. When you're reviewing a complex change, what's your approach? Do you review it all at once or break it up somehow?

• I will first take a very high level look to see if the general direction makes sense, so as to not let the author waste a lot of time polishing a change that will be turned down because it is orthogonal to how it should be done
• I will then wait until tests are implemented: it is by reading them that I will know what and how the change is done. I will ask the author to verify that coverage is good (because there is a lack of tooling for that in most cases but it should be automated).
• After the tested behavior makes logical sense to me and appears to address the change that is proposed, I will read the code looking for significant problems (I'm not nitpicking on style, lint does it well enough).

If the change is broken up in individual commits to help the reviewer, I will look at those commits one by one. If this is not the case, I will not ask the author to do that. If someone prefers to submit their work as a single change, I'm fine with that.

The short answer is that I review in conceptual layers and not all at once. There also are variations depending on the author. If I'm used to how someone writes code, I will have a very different review process than if I review a change from someone I've never worked with before.

Problems

8. Tell me about a time when managing a large change went wrong. What happened?

As a reviewer forgejo/forgejo#5564 was a frustrating experience (for the author as well I suspect). Primarily because it went on for a very long time with weeks in between reviews. I struggled to figure out what changed in between reviews. I also struggled to rebuild the mental model that I created in my head but that was no longer in short term memory.

An an author I enthusiastically engaged in the implementation of a large feature and spent days working on it. Only to be kindly told by reviewers that it was not in scope for a Forgejo feature. This is a mistake I did a lot years ago. Nowadays I'm more careful to first discuss about the idea of a large change and be ready to change my point of view entirely based on other people inputs. This failed attempt at a large change was a reminder of the sanity of this approach. I also think that by doing so the magnitude of the change gets reduced to something more manageable. If only because a discussion about a very large change is likely to stall and lead nowhere. I don't know really, it is just a gut feeling.

9. What makes you avoid breaking down a large change, even if you know you should?

I don't remember that happening. I sure it did happen, only I don't remember that and I don't think it ever was deliberate.

A large change should be broken down if it breaks the promises of the codebase.

• If a codebase is only ever used by you or a few people you know (e.g. https://code.forgejo.org/f3/gof3), a large change that breaks everything is not really a problem. If it is difficult to break down the change, that's ok to have it as is.
• However, in a codebase such as Forgejo the promise is that the codebase is improved upon in ways that is manageable for reviewers and other authors. In that case the change should be broken down to manageable pull requests. And that is what happens, if only because very large changes struggle to get the attention of reviewers because they are too time consuming.

10. What's missing from your current tools that would make large changes easier to manage?

• As an author I frequently have to copy/paste the URL to the change I made in response to a review comment
• I sometime struggle to find the pull requests that relate to particular topic or codebase. For instance there are a number of pull requests related to the Forgejo Actions notifications, even after the initial ones were merged. I remember they are related but there is nothing that groups them.

And then there is the big question of the roadmap that does not exist as such but exists in various shapes in the mind of every Forgejo contributor for instance. The items in those roadmaps are big changes (Federation, moderation, migrations, etc.). But they are at a higher level than what you mean by big change I suppose.

In short I don't miss much but I'm also very aware that this is because I'm so used to this workflow. Not because it is perfect. Just that I'm now well adjusted to it and do not feel there are roadblocks creating recurring frustrations.

--- Before answering the survey questions, I had the following random thoughts. Federation is a large change. F3 is a large change. Moderation is a large change. Forgejo Actions notifications is a large change. I'm involved to various degrees in those and they all have different strategies. I will focus on my experience with Forgejo Actions notifications because it is likely closer to what Mathieu Fenniak is interested in. It started with a nicely organized, chronologically ordered set of pull requests by Chris Another similar example of series of PR is when oliverpool reworked the webhooks early 2024. But that's a more distant memory and I was just a reviewer amazed by how well it all fit together. I could not even begin to imagine how he made it work. It could not fit in my brain, that would be too much. Another example is how gusted is gradually making security related changes where one can find commonality if you look at those PRs over time. For instance how SSH based alternatives to OpenPGP eventually make their way in the codebase. gusted: approach feels more like an impressionist approach to series of pull requests. Touches here and there that are not necessarily related but if you take a step back, they paint a picture. F3 is not in scope I think because it is a good example of a feature that has a very simple interface (cp a b) and is best moved to an independent package. --- > #### Motivations > > 1. When you're planning a large development effort, what influences whether you tackle it as one change or multiple? The primary reason for having one large change is because I could not figure out a way to break it down into smaller independent changes. Examples: - Improving Forgejo migrations ([1](https://codeberg.org/forgejo/forgejo/src/branch/forgejo/modules/migration), [2](https://codeberg.org/forgejo/forgejo/src/branch/forgejo/services/migrations/)) requires a complete re-architecture of the package. Instead of making it a (very) large change in Forgejo, it is developed [as an independent package F3](https://codeberg.org/forgejo/forgejo/src/branch/forgejo/services/migrations/). It takes a very long time for this package to mature but when it is ready, because the API it provides is simple, the ancient code can be removed in Forgejo and replaced with calls to this API. The scale of the change was shifted out of Forgejo and when the change lands, it will be a single change, not multiple changes. - [Forgejo Actions notifications](https://codeberg.org/forgejo/forgejo/issues/3719) is an example of a single change that was broken down into [multiple changes](). I was on the reviewer side of this series but I would have had almost exactly the same approach as the author. It [started with code walks](https://codeberg.org/forgejo/forgejo/issues/3719#issuecomment-3494810) to understand where things are and get a mental model of what needs changing. It discovered that bits of infrastructure were missing to implement that change and that made it large instead of small. To not resort to a hack, the author then proceeded to plan for a series of pull request to create the foundations on which the change could stand. > 2. Can you tell me about a time when you decided that a software change was 'too big'? What made you realize that? - Some migrations improvements and bug fixes over a long period of time demonstrated multiple times that the architecture was not good enough to support a number of bug fixes and features. At the time I got involved, it was already clear to everyone that this part of the codebase was stuck and required a large change. An architectural change. - Forgejo Actions notifications realization came out of exploring the codebase. That is how I discover that a change is not as simple as it should, most of the time. - A recent example is when [fixing a bug](https://code.forgejo.org/forgejo/act/pulls/178/files). Exploring how this should be properly implemented, I could not help but notice that there were multiple inconsistencies in how environment variables are set. I tried to clean that up a bit and ended up with a large change, changing behaviors that were for a large part not covered by tests but known to work in practice. I found myself stumbling into a large change and came to my senses before being too committed. I changed my approach to fixing the bug in a way that is consistent with the current codebase while not refactoring it. Trying to strike an (unsatisfactory) balance between a hack and a large refactor. > 3. What's the most frustrating thing about reviewing large software changes? - Loosing context when days or weeks pass between two reviews. The larger the change the longer context switching takes when time passes. - When the author does not explain in plain English the mental model they have in mind and that it needs to be reconstructed by reading the code. - When the author makes changes that are not explained and the reason for the change is not obvious when reading them. - When the author resolves a review comment without linking it to the change that resolves it. Although all those frustrations also exist in small changes, their inconvenience is in proportion of the size of the change. > #### Activities > > 4. Referencing your answer from question 2, what did you do when the software change was too big? - Migration => move the development to a dependency. - Forgejo Actions notifications => (I'm not the author but I would have done almost exactly that) plan for a logical and chronologically organized series of pull requests. It can be difficult because the first PR is mostly useful in the context of the larger change so it is important that the reviewer understands how it fits in the series. > 5. How do you typically organize commits in your changes? - for small changes, most of the time, I make it a single commit and I rebase the pull request and force push to keep it a single commit. - for small changes that contain unrelated but interdependent changes, I separate them in different commits so they can be reviewed independently and merged as is instead of squash for forensic analysis in the future. Example: https://code.forgejo.org/forgejo/act/pulls/177/commits - for larger changes that fit in a single pull request I keep a well organized series of commit with titles that tell a story that can be read by the reviewer. Examples: - https://code.forgejo.org/forgejo/act/pulls/170/commits (I'm the author) - https://codeberg.org/forgejo/forgejo/pulls/8451/commits (I'm the reviewer) > 6. If you had to split a change up into two or more separate changes, what steps would you take? As suggested by the examples above, it highly depends on the magnitude of the change and the strategies to break it down into manageable / reviewable changes. It also depends on who is involved (author / reviewer). I'll answer in the particular context of a change that requires a series of pull requests that depend on each other. After [exploring the codebase](https://codeberg.org/forgejo/forgejo/issues/3719#issuecomment-3494810) to figure out what needs to be done, I would look for ways to not make them all into a single pull request. If I see that part of the change has value in isolation, I would start with a pull request for that change alone. Ideally it stands on its own and no context needs to be explained. But in the case of Forgejo Actions notifications, the first pull requests ([1](https://codeberg.org/forgejo/forgejo/pulls/7510) and [2](https://codeberg.org/forgejo/forgejo/pulls/7491)) only had meaning in the larger context and explaining it helped. The most difficult part in engaging in such a series of pull request is that it is taking a risk that something is overlooked and that it is only revealed after the last pull request is merged. In this case it happened as I (as a reviewer) failed to see that a pull request implementing the relationship between two workflow runs was missing. Part of the change had to be reverted and remains incomplete until that is done. I do not think there is a way to prevent that from happening. It is a bug of sorts, although more of an architectural bug than a simpler behavior bug. Forgejo and other software have a lot of those and they are the most difficult to fix. But I don't think they can be prevented and the key is to not be paralyzed by fear when engaging in a large change. Only be prepared to revisit the change later and not let the problem linger forever. > 7. When you're reviewing a complex change, what's your approach? Do you review it all at once or break it up somehow? - I will first take a very high level look to see if the general direction makes sense, so as to not let the author waste a lot of time polishing a change that will be turned down because it is orthogonal to how it should be done - I will then wait until tests are implemented: it is by reading them that I will know what and how the change is done. I will ask the author to verify that coverage is good (because there is a lack of tooling for that in most cases but it should be automated). - After the tested behavior makes logical sense to me and appears to address the change that is proposed, I will read the code looking for significant problems (I'm not nitpicking on style, lint does it well enough). If the change is broken up in individual commits to help the reviewer, I will look at those commits one by one. If this is not the case, I will not ask the author to do that. If someone prefers to submit their work as a single change, I'm fine with that. The short answer is that I review in conceptual layers and not all at once. There also are variations depending on the author. If I'm used to how someone writes code, I will have a very different review process than if I review a change from someone I've never worked with before. > #### Problems > > 8. Tell me about a time when managing a large change went wrong. What happened? As a reviewer https://codeberg.org/forgejo/forgejo/pulls/5564 was a frustrating experience (for the author as well I suspect). Primarily because it went on for a very long time with weeks in between reviews. I struggled to figure out what changed in between reviews. I also struggled to rebuild the mental model that I created in my head but that was no longer in short term memory. An an author I enthusiastically engaged in the [implementation of a large feature](https://codeberg.org/forgejo/forgejo/pulls/6979/files) and spent days working on it. Only to be kindly told by reviewers that it was not in scope for a Forgejo feature. This is a mistake I did a lot years ago. Nowadays I'm more careful to first discuss about the idea of a large change and be ready to change my point of view entirely based on other people inputs. This failed attempt at a large change was a reminder of the sanity of this approach. I also think that by doing so the magnitude of the change gets reduced to something more manageable. If only because a discussion about a very large change is likely to stall and lead nowhere. I don't know really, it is just a gut feeling. > 9. What makes you avoid breaking down a large change, even if you know you should? I don't remember that happening. I sure it did happen, only I don't remember that and I don't think it ever was deliberate. A large change should be broken down if it breaks the promises of the codebase. - If a codebase is only ever used by you or a few people you know (e.g. https://code.forgejo.org/f3/gof3), a large change that breaks everything is not really a problem. If it is difficult to break down the change, that's ok to have it as is. - However, in a codebase such as Forgejo the promise is that the codebase is improved upon in ways that is manageable for reviewers and other authors. In that case the change should be broken down to manageable pull requests. And that is what happens, if only because very large changes struggle to get the attention of reviewers because they are too time consuming. > 10. What's missing from your current tools that would make large changes easier to manage? - As an author I frequently have to copy/paste the URL to the change I made in response to a review comment - I sometime struggle to find the pull requests that relate to particular topic or codebase. For instance there are a number of pull requests related to the Forgejo Actions notifications, even after the initial ones were merged. I remember they are related but there is nothing that groups them. And then there is the big question of the roadmap that does not exist as such but exists in various shapes in the mind of every Forgejo contributor for instance. The items in those roadmaps are big changes (Federation, moderation, migrations, etc.). But they are at a higher level than what you mean by big change I suppose. In short I don't miss much but I'm also very aware that this is because I'm so used to this workflow. Not because it is perfect. Just that I'm now well adjusted to it and do not feel there are roadblocks creating recurring frustrations.

jerger commented 2025年07月15日 14:56:59 +02:00

FYI:

This is how we manage large PRs in federation: https://codeberg.org/forgejo-contrib/federation/src/branch/main/doc/merge-large-pr.md

On example PR is: forgejo/forgejo#4767

How / where can I contribute my findings?

FYI: This is how we manage large PRs in federation: https://codeberg.org/forgejo-contrib/federation/src/branch/main/doc/merge-large-pr.md On example PR is: https://codeberg.org/forgejo/forgejo/pulls/4767 How / where can I contribute my findings?

mfenniak commented 2025年07月15日 19:16:08 +02:00

@jerger Very interesting link to the existing process, thanks for sharing that. I might be missing some context to fully understand the process; is it roughly correct to summarize this as: "Start a feature (my-fork/new-federation-feature). Recognize a chunk that can be done merged independently. Implement that chunk (my-fork/p99-1) and get it merged. Build on that chunk more feature, or more chunks`?

How / where can I contribute my findings?

Any one of these approaches would be great:

• As Earl did, if you're willing to answer the proposed 10 survey questions, you can do so here in a PR comment and I'll incorporate the responses into the PR contents.
• As your experience is much more direct with problems and solutions in the Forgejo context, if prefer to skip the survey and provide your overall feedback on pains and wins with your current tooling, you could provide a PR comment of that nature.
• I'd be happy to do an audio-or-video chat and interview you to inquire more, if that would be more to your liking; we can coordinate a time through Matrix if you want to find me there (@mathieu:kainnef.com)

@jerger Very interesting link to the existing process, thanks for sharing that. I might be missing some context to fully understand the process; is it roughly correct to summarize this as: "Start a feature (`my-fork/new-federation-feature`). Recognize a chunk that can be done merged independently. Implement that chunk (`my-fork/p99-1`) and get it merged. Build on that chunk more feature, or more chunks`? > How / where can I contribute my findings? Any one of these approaches would be great: - As Earl did, if you're willing to answer the proposed 10 survey questions, you can do so here in a PR comment and I'll incorporate the responses into the PR contents. - As your experience is much more direct with problems and solutions in the Forgejo context, if prefer to skip the survey and provide your overall feedback on pains and wins with your current tooling, you could provide a PR comment of that nature. - I'd be happy to do an audio-or-video chat and interview you to inquire more, if that would be more to your liking; we can coordinate a time through Matrix if you want to find me there (`@mathieu:kainnef.com`)

jerger commented 2025年07月16日 17:46:34 +02:00

Motivations

1. When you're planning a large development effort, what influences whether you tackle it as one change or multiple?

When developing completely new stuff, I need room for experiments & have to prove the result is working. In such a situation submitting small & well planned serial PRs is no option.

2. Can you tell me about a time when you decided that a software change was 'too big'? What made you realize that?

If there are more than 15 files involved.

3. What's the most frustrating thing about reviewing large software changes?

As you can see in forgejo/forgejo#4767 there are many comments during a long time development process. In role contributor it is very hard to track, what is still an open TODO.

As reviewer:

1. it is also hard to track the state of open TODOs.
2. In addition there are many aspects to consider like architecture, side effects, testcoverage. From first PR till all aspects are covered there can be long time & much dev work. For PR named above the initial effort was about 250h, to fix all non functional stuff took additional 400h.
3. It is very hard to decide when it is a good point in time for a re-review.

Activities

4. Referencing your answer from question 2, what did you do when the software change was too big?

Split into smaller parts, as isolated as possible to create small PRs.

5. How do you typically organize commits in your changes?

For federation features there are many devs involved. This means:

1. Commits are used to exchange code in case of mob-programming,
2. Commits might carry semantic useful feature increments in order to communicate progress between devs
3. We use no rebase & squash - force push would break history for devs involved.

Commits in small PRs might be more meaningfull for reviewers.

6. If you had to split a change up into two or more separate changes, what steps would you take?

See process lined out above.

7. When you're reviewing a complex change, what's your approach? Do you review it all at once or break it up somehow?

This evolves. For federation we've review guidelines: https://codeberg.org/forgejo-contrib/federation/src/branch/main/doc/federation-pr-review-guidelines.md

Problems

8. Tell me about a time when managing a large change went wrong. What happened?

1. An other merger merged but there was open todos. Responsibility was unclear.
2. I accepted a large PR but there was side effects / not aligned function signatures / duplicated code . Fix was up to me ...

9. What makes you avoid breaking down a large change, even if you know you should?

I feel responsible to help contributors & sometimes I was not aware of the hidden complexity.

10. What's missing from your current tools that would make large changes easier to manage?

Management of TODOs & triggering a re review in a improved & defined way would help me a lot.

> #### Motivations > 1. When you're planning a large development effort, what influences whether you tackle it as one change or multiple? When developing completely new stuff, I need room for experiments & have to prove the result is working. In such a situation submitting small & well planned serial PRs is no option. > 2. Can you tell me about a time when you decided that a software change was 'too big'? What made you realize that? If there are more than 15 files involved. > 3. What's the most frustrating thing about reviewing large software changes? As you can see in https://codeberg.org/forgejo/forgejo/pulls/4767 there are many comments during a long time development process. In role contributor it is very hard to track, what is still an open TODO. As reviewer: 1. it is also hard to track the state of open TODOs. 2. In addition there are many aspects to consider like architecture, side effects, testcoverage. From first PR till all aspects are covered there can be long time & much dev work. For PR named above the initial effort was about 250h, to fix all non functional stuff took additional 400h. 3. It is very hard to decide when it is a good point in time for a re-review. > #### Activities > 4. Referencing your answer from question 2, what did you do when the software change was too big? Split into smaller parts, as isolated as possible to create small PRs. > 5. How do you typically organize commits in your changes? For federation features there are many devs involved. This means: 1. Commits are used to exchange code in case of mob-programming, 2. Commits might carry semantic useful feature increments in order to communicate progress between devs 3. We use no rebase & squash - force push would break history for devs involved. Commits in small PRs might be more meaningfull for reviewers. > 6. If you had to split a change up into two or more separate changes, what steps would you take? See process lined out above. > 7. When you're reviewing a complex change, what's your approach? Do you review it all at once or break it up somehow? This evolves. For federation we've review guidelines: https://codeberg.org/forgejo-contrib/federation/src/branch/main/doc/federation-pr-review-guidelines.md > #### Problems > 8. Tell me about a time when managing a large change went wrong. What happened? 1. An other merger merged but there was open todos. Responsibility was unclear. 2. I accepted a large PR but there was side effects / not aligned function signatures / duplicated code . Fix was up to me ... > 9. What makes you avoid breaking down a large change, even if you know you should? I feel responsible to help contributors & sometimes I was not aware of the hidden complexity. > 10. What's missing from your current tools that would make large changes easier to manage? Management of TODOs & triggering a re review in a improved & defined way would help me a lot.

jerger commented 2025年07月16日 17:54:49 +02:00

If there are questions open feel free to contact me on matrix: @jerger:matrix.org

If there are questions open feel free to contact me on matrix: @jerger:matrix.org

mfenniak commented 2025年07月16日 18:27:52 +02:00

Thanks @jerger, wonderful feedback and greatly appreciated. 👍

Thanks @jerger, wonderful feedback and greatly appreciated. 👍

mfenniak commented 2025年08月09日 04:20:12 +02:00

Regarding the user research results...

I still have one outstanding survey response from a volunteer, but it has been pending for a while and I'm not receiving any response to follow-ups. I believe this PR contains all the collected data that it will ever have. In terms of user research, there are some survey results attached that are hit and miss, and include some interesting insights into areas for improvement in Forgejo.

The primary insight for me is that most of the best practices and advice for managing large change is to avoid it. So, developing new tools in Forgejo to support those use-cases may not be overall advisable. I'm kinda on the fence over this, but I think for my future plans it would be more practical to target small tactical improvements that support all kinds of change management.

Regarding this PR...

I don't think the content collected here seems well aligned with the intent of the existing content in this repo, which is notes from experiential research sessions. So, I'll close this PR unless there's other feedback, but the survey responses will remain attached here in case they have future value in supporting work.

Regarding the user research results... I still have one outstanding survey response from a volunteer, but it has been pending for a while and I'm not receiving any response to follow-ups. I believe this PR contains all the collected data that it will ever have. In terms of user research, there are some survey results attached that are hit and miss, and include some interesting insights into areas for improvement in Forgejo. The primary insight for me is that most of the best practices and advice for managing large change is to avoid it. So, developing new tools in Forgejo to support those use-cases may not be overall advisable. I'm kinda on the fence over this, but I think for my future plans it would be more practical to target small tactical improvements that support all kinds of change management. Regarding this PR... I don't think the content collected here seems well aligned with the intent of the existing content in this repo, which is notes from experiential research sessions. So, I'll close this PR unless there's other feedback, but the survey responses will remain attached here in case they have future value in supporting work.

earl-warren commented 2025年08月10日 13:03:46 +02:00

I think there is value it not loosing the content of this PR. Could it be linked somewhere in the README or elsewhere? Even a oneliner in the current repository content would make sure it is not burried forever. 🙏

I think there is value it not loosing the content of this PR. Could it be linked somewhere in the README or elsewhere? Even a oneliner in the current repository content would make sure it is not burried forever. 🙏