Patman patch manager
This tool is a Python script which:
Creates patch directly from your branch
Cleans them up by removing unwanted tags
Inserts a cover letter with change lists
Runs the patches through checkpatch.pl and its own checks
Optionally emails them out to selected people
Links the series automatically to Patchwork once sent
It also has some Patchwork features:
Manage local series and their status on patchwork
Show review tags from Patchwork and allows them to be gathered into commits
List comments received on a series
It is intended to automate patch creation and make it a less error-prone process. It is useful for U-Boot and Linux work so far, since they use the checkpatch.pl script.
It is configured almost entirely by tags it finds in your commits. This means that you can work on a number of different branches at once, and keep the settings with each branch rather than having to git format-patch, git send-email, etc. with the correct parameters each time. So for example if you put:
Series-to: fred.blogs@napier.co.nz
in one of your commits, the series will be sent there.
In Linux and U-Boot this will also call get_maintainer.pl on each of your patches automatically (unless you use -m to disable this).
Installation
You can install patman using:
pip install patch-manager
The name is chosen since patman conflicts with an existing package.
If you are using patman within the U-Boot tree, it may be easiest to add a symlink from your local ~/.bin directory to /path/to/tools/patman/patman.
How to use this tool
This tool requires a certain way of working:
Maintain a number of branches, one for each patch series you are working on
Add tags into the commits within each branch to indicate where the series should be sent, cover letter, version, etc. Most of these are normally in the top commit so it is easy to change them with ‘git commit –amend’
Each branch tracks the upstream branch, so that this script can automatically determine the number of commits in it (optional)
Check out a branch, and run this script to create and send out your patches. Weeks later, change the patches and repeat, knowing that you will get a consistent result each time.
How to configure it
For most cases of using patman for U-Boot development, patman can use the file ‘doc/git-mailrc’ in your U-Boot directory to supply the email aliases you need. To make this work, tell git where to find the file by typing this once:
git config sendemail.aliasesfile doc/git-mailrc
For both Linux and U-Boot the ‘scripts/get_maintainer.pl’ handles figuring out where to send patches pretty well. For other projects, you may want to specify a different script to be run, for example via a project-specific .patman file:
# .patman configuration file at the root of some project [settings] get_maintainer_script: etc/teams.scm get-maintainer
The get_maintainer_script option corresponds to the –get-maintainer-script argument of the send command. It is looked relatively to the root of the current git repository, as well as on PATH. It can also be provided arguments, as shown above. The contract is that the script should accept a patch file name and return a list of email addresses, one per line, like get_maintainer.pl does.
During the first run patman creates a config file for you by taking the default user name and email address from the global .gitconfig file.
To add your own, create a file ~/.patman like this:
# patman alias file [alias] me: Simon Glass <sjg@chromium.org> u-boot: U-Boot Mailing List <u-boot@lists.denx.de> wolfgang: Wolfgang Denk <wd@denx.de> others: Mike Frysinger <vapier@gentoo.org>, Fred Bloggs <f.bloggs@napier.net>
As hinted above, Patman will also look for a .patman configuration file at the root of the current project git repository, which makes it possible to override the project settings variable or anything else in a project-specific way. The values of this "local" configuration file take precedence over those of the "global" one.
Aliases are recursive.
The checkpatch.pl in the U-Boot tools/ subdirectory will be located and used. Failing that you can put it into your path or ~/bin/checkpatch.pl
If you want to avoid sending patches to email addresses that are picked up by patman but are known to bounce you can add a [bounces] section to your .patman file. Unlike the [alias] section these are simple key: value pairs that are not recursive:
[bounces] gonefishing: Fred Bloggs <f.bloggs@napier.net>
If you want to change the defaults for patman’s command-line arguments, you can add a [settings] section to your .patman file. This can be used for any command line option by referring to the "dest" for the option in patman.py. For reference, the useful ones (at the moment) shown below (all with the non-default setting):
[settings] ignore_errors: True process_tags: False verbose: True smtp_server: /path/to/sendmail patchwork_url: https://patchwork.ozlabs.org
If you want to adjust settings (or aliases) that affect just a single project you can add a section that looks like [project_settings] or [project_alias]. If you want to use tags for your linux work, you could do:
[linux_settings] process_tags: True
How to run it
First do a dry run:
./tools/patman/patmansend-n
If it can’t detect the upstream branch, try telling it how many patches there are in your series
./tools/patman/patman-c5send-n
This will create patch files in your current directory and tell you who it is thinking of sending them to. Take a look at the patch files:
./tools/patman/patman-c5-s1send-n
Similar to the above, but skip the first commit and take the next 5. This is useful if your top commit is for setting up testing.
How to install it
The most up to date version of patman can be found in the U-Boot sources. However to use it on other projects it may be more convenient to install it as a standalone application. A distutils installer is included, this can be used to install patman:
cdtools/patman&&pythonsetup.pyinstall
Where Patches Are Sent
Once the patches are created, patman sends them using git send-email. The whole series is sent to the recipients in Series-to: and Series-cc. You can Cc individual patches to other people with the Patch-cc: tag. Tags in the subject are also picked up to Cc patches. For example, a commit like this:
commit 10212537b85ff9b6e09c82045127522c0f0db981 Author: Mike Frysinger <vapier@gentoo.org> Date: Mon Nov 7 23:18:44 2011 -0500 x86: arm: add a git mailrc file for maintainers This should make sending out e-mails to the right people easier. Patch-cc: sandbox, mikef, ag Patch-cc: afleming
will create a patch which is copied to x86, arm, sandbox, mikef, ag and afleming.
If you have a cover letter it will get sent to the union of the Patch-cc lists of all of the other patches. If you want to sent it to additional people you can add a tag:
Cover-letter-cc: <list of addresses>
These people will get the cover letter even if they are not on the To/Cc list for any of the patches.
Patchwork Integration
Patman has a very basic integration with Patchwork. If you point patman to your series on patchwork it can show you what new reviews have appeared since you sent your series.
To set this up, add a Series-link tag to one of the commits in your series (see above).
Then you can type:
patmanstatus
and patman will show you each patch and what review tags have been collected, for example:
... 21 x86: mtrr: Update the command to use the new mtrr Reviewed-by: Wolfgang Wallner <wolfgang.wallner@br-automation.com> + Reviewed-by: Bin Meng <bmeng.cn@gmail.com> 22 x86: mtrr: Restructure so command execution is in Reviewed-by: Wolfgang Wallner <wolfgang.wallner@br-automation.com> + Reviewed-by: Bin Meng <bmeng.cn@gmail.com> ...
This shows that patch 21 and 22 were sent out with one review but have since attracted another review each. If the series needs changes, you can update these commits with the new review tag before sending the next version of the series.
To automatically pull into these tags into a new branch, use the -d option:
patmanstatus-dmtrr4
This will create a new ‘mtrr4’ branch which is the same as your current branch but has the new review tags in it. The tags are added in alphabetic order and are placed immediately after any existing ack/review/test/fixes tags, or at the end. You can check that this worked with:
patman-bmtrr4status
which should show that there are no new responses compared to this new branch.
There is also a -C option to list the comments received for each patch.
Example Work Flow
The basic workflow is to create your commits, add some tags to the top commit, and type ‘patman’ to check and send them.
Here is an example workflow for a series of 4 patches. Let’s say you have these rather contrived patches in the following order in branch us-cmd in your tree where ‘us’ means your upstreaming activity (newest to oldest as output by git log –oneline):
7c7909c wip 89234f5 Don't include standard parser if hush is used 8d640a7 mmc: sparc: Stop using builtin_run_command() 0c859a9 Rename run_command2() to run_command() a74443f sandbox: Rename run_command() to builtin_run_command()
The first patch is some test things that enable your code to be compiled, but that you don’t want to submit because there is an existing patch for it on the list. So you can tell patman to create and check some patches (skipping the first patch) with:
patman-s1send-n
If you want to do all of them including the work-in-progress one, then (if you are tracking an upstream branch):
patmansend-n
Let’s say that patman reports an error in the second patch. Then:
gitrebase-iHEAD~6 # change 'pick' to 'edit' in 89234f5 # use editor to make code changes gitadd-u gitrebase--continue
Now you have an updated patch series. To check it:
patman-s1send-n
Let’s say it is now clean and you want to send it. Now you need to set up the destination. So amend the top commit with:
gitcommit--amend
Use your editor to add some tags, so that the whole commit message is:
The current run_command() is really only one of the options, with hush providing the other. It really shouldn't be called directly in case the hush parser is bring used, so rename this function to better explain its purpose:: Series-to: u-boot Series-cc: bfin, marex Series-prefix: RFC Cover-letter: Unified command execution in one place At present two parsers have similar code to execute commands. Also cmd_usage() is called all over the place. This series adds a single function which processes commands called cmd_process(). END Change-Id: Ica71a14c1f0ecb5650f771a32fecb8d2eb9d8a17
You want this to be an RFC and Cc the whole series to the bfin alias and to Marek. Two of the patches have tags (those are the bits at the front of the subject that say mmc: sparc: and sandbox:), so 8d640a7 will be Cc’d to mmc and sparc, and the last one to sandbox.
Now to send the patches, take off the -n flag:
patman-s1send
The patches will be created, shown in your editor, and then sent along with the cover letter. Note that patman’s tags are automatically removed so that people on the list don’t see your secret info.
Of course patches often attract comments and you need to make some updates. Let’s say one person sent comments and you get an Acked-by: on one patch. Also, the patch on the list that you were waiting for has been merged, so you can drop your wip commit.
Take a look on patchwork and find out the URL of the series. This will be something like http://patchwork.ozlabs.org/project/uboot/list/?series=187331 Add this to a tag in your top commit:
Series-links: 187331
You can use then patman to collect the Acked-by tag to the correct commit, creating a new ‘version 2’ branch for us-cmd:
patmanstatus-dus-cmd2 gitcheckoutus-cmd2
You can look at the comments in Patchwork or with:
patmanstatus-C
Then you can resync with upstream:
gitfetchorigin# or whatever upstream is called
gitrebaseorigin/master
and use git rebase -i to edit the commits, dropping the wip one.
Then update the Series-cc: in the top commit to add the person who reviewed the v1 series:
Series-cc: bfin, marex, Heiko Schocher <hs@denx.de>
and remove the Series-prefix: tag since it it isn’t an RFC any more. The series is now version two, so the series info in the top commit looks like this:
Series-to: u-boot Series-cc: bfin, marex, Heiko Schocher <hs@denx.de> Series-version: 2 Cover-letter: ...
Finally, you need to add a change log to the two commits you changed. You add change logs to each individual commit where the changes happened, like this:
Series-changes: 2 - Updated the command decoder to reduce code size - Wound the torque propounder up a little more
(note the blank line at the end of the list)
When you run patman it will collect all the change logs from the different commits and combine them into the cover letter, if you have one. So finally you have a new series of commits:
faeb973 Don't include standard parser if hush is used 1b2f2fe mmc: sparc: Stop using builtin_run_command() cfbe330 Rename run_command2() to run_command() 0682677 sandbox: Rename run_command() to builtin_run_command()
so to send them:
patman
and it will create and send the version 2 series.
Series Management
Sometimes you might have several series in flight at the same time. Each of these receives comments and you want to create a new version of each series with those comments addressed.
Patman provides a few subcommands which are helpful for managing series.
Series and branches
‘patman series’ works with the concept of a series. It maintains a local database (.patman.db in your top-level git tree) and uses that to keep track of series and patches.
Each series goes through muliple versions. Patman requires that the first version of your series is in a branch without a numeric suffix. Branch names like ‘serial’ and ‘video’ are OK, but ‘part3’ is not. This is because Patman uses the number at the end of the branch name to indicate the version.
If your series name is ‘video’, then you can have a ‘video’ branch for version 1 of the series, ‘video2’ for version 2 and ‘video3’ for version 3. All three branches are for the same series. Patman keeps track of these different versions. It handles the branch naming automatically, but you need to be aware of what it is doing.
You will have an easier time if the branch names you use with ‘patman series’ are short, no more than 15 characters. This is the amount of columnar space in listings. You can add a longer description as the series description. If you are used to having very descriptive branch names, remember that patman lets you add metadata into commit which is automatically removed before sending.
This documentation uses the term ‘series’ to mean all the versions of a series and ‘series/version’ to mean a particular version of a series.
Updating commits
Since Patman provides quite a bit of automation, it updates your commits in some cases, effectively doing a rebase of a branch in order to change the tags in the commits. It never makes code changes.
In extremis you can use ‘git reflog’ to revert something that Patman did.
Series subcommands
Note that ‘patman series ...’ can be abbreviated as ‘patman s’ or ‘patman ser’.
Here is a short overview of the available subcommands:
- add
Add a new series. Use this on an existing branch to tell Patman about it.
- archive (ar)
Archive a series when you have finished upstreaming it. Archived series are not shown by most commands. This creates a dated tag for each version of the series, pointing to the series branch, then deletes the branches. It puts the tag names in the database so that it can ‘unarchive’ to restore things how they were.
- unarchive (unar)
Unarchive a series when you decide you need to do something more with it. The branches are restored and tags deleted.
- autolink (au)
Search patchwork for the series link for your series, so Patman can track the status
- autolink-all
Same but for all series
- inc
Increase the series number, effectively creating a new branch with the next highest version number. The new branch is created based on the existing branch. So if you use ‘patman series inc’ on branch ‘video2’ it will create branch ‘video3’ and add v3 into its database
- dec
Decrease the series number, thus deleting the current branch and removing that version from the data. If you use this comment on branch ‘video3’ Patman will delete version 3 and branch ‘video3’.
- get-link
Shows the Patchwork link for a series/version
- ls
Lists the series in the database
- mark
Mark a series with ‘Change-Id’ tags so that Patman can track patches even when the subject changes. Unmarked patches just use the subject to decided which is which.
- unmark
Remove ‘Change-Id’ tags from a series.
- open (o)
Open a series in Patchwork using your web browser
- patches
Show the patches in a particular series/version
- progress (p)
Show upstream progress for your series, or for all series
- rm
Remove a series entirely, including all versions
- rm-version (rmv)
Remove a particular version of a series. This is similar to ‘dec’ except that any version can be removed, not just the latest one.
- scan
Scan the local branch and update the database with the set of patches in that branch. This throws away the old patches.
- send
Send a series out as patches. This is similar to ‘patman send’ except that it can send any series, not just the current branch. It also waits a little for patchwork to see the cover letter, so it can find out the patchwork link for the series.
- set-link
Sets the Patchwork link for a series-version manually.
- status (st)
Run ‘patman status’ on a series. This is similar to ‘patman status’ except that it can get status on any series, not just the current branch
- summary
Shows a quick summary of series with their status and description.
- sync
Sync the status of a series with Pathwork, so that ‘patman series progress’ can show the right information.
- sync-all
Sync the status of all series.
Patman series workflow
Here is a run-through of how to incorporate ‘patman series’ into your workflow.
Firstly, set up your project:
patman patchwork set-project U-Boot
This just tells Patman to look on the Patchwork server for a project of that name. Internally Patman stores the ID and URL ‘link-name’ for the project, so it can access it.
If you need to use a different patchwork server, use the –patchwork-url option or put the URL in your Patman-settings file.
Now create a branch. For our example we are going to send out a series related to video so the branch will be called ‘video’. The upstream remove is called ‘us’:
git checkout -b video us/master
We now have a branch and so we can do some commits:
<edit files> git add ... <edit files> git add -u git commit ... git commit ...
We now have a few commits in our ‘video’ branch. Let’s tell patman about it:
patman series add
Like most commands, if no series is given (patman series -s video add) then the current branch is assumed. Since the branch is called ‘video’ patman knows that it is version one of the video series.
You’ll likely get a warning that there is no cover letter. Let’s add some tags to the top commit:
Series-to: u-boot Series-cc: ... Cover-letter: video: Improve syncing performance with cyclic
Trying again:
patman series add
You’ll likely get a warning that the commits are unmarked. You can either let patman add Change-Id values itself with the -m flag, or tell it not to worry about it with -M. You must choose one or the other. Let’s leave the commits unmarked:
patman series add -M
Congratulations, you’ve now got a patman database!
Now let’s send out the series. We will add tags to the top commit.
To send it:
patman series send
You should send ‘git send-email’ start up and you can confirm the sending of each email.
After that, patman waits a bit to see if it can find your new series appearing on Patchwork. With a bit of luck this will only take 20 seconds or so. Then your series is linked.
To gather tags (Reviewed-by ...) for your series from patchwork:
patman series gather
Now you can check your progress:
patman series progress
Later on you get some comments, or perhaps you just decide to make a change on your own. You have several options.
The first option is that you can just create a new branch:
git checkout -b video2 video
then you can add this ‘v2’ series to Patman with:
patman series add
The second option is to get patman to create the new ‘video2’ branch in one step:
patman inc
The third option is to collect some tags using the ‘patman status’ command and put them in a new branch:
patman status -d video2
One day the fourth option will be to ask patman to collect tags as part of the ‘patman inc’ command.
Again, you do your edits, perhaps adding/removing patches, rebasing on -master and so on. Then, send your v2:
patman series send
Let’s say the patches are accepted. You can use:
patch series gather patch series progress
to check, or:
patman series status -cC
to see comments. You can now archive the series:
patman series archive
At this point you have the basics. Some of the subcommands useful options, so be sure to check out the help.
Here is a sample ‘progress’ view:
Patman showing the progress view
General points
When you change back to the us-cmd branch days or weeks later all your information is still there, safely stored in the commits. You don’t need to remember what version you are up to, who you sent the last lot of patches to, or anything about the change logs.
If you put tags in the subject, patman will Cc the maintainers automatically in many cases.
If you want to keep the commits from each series you sent so that you can compare change and see what you did, you can either create a new branch for each version, or just tag the branch before you start changing it:
gittagsent/us-cmd-rfc # ...later... gittagsent/us-cmd-v2If you want to modify the patches a little before sending, you can do this in your editor, but be careful!
If you want to run git send-email yourself, use the -n flag which will print out the command line patman would have used.
It is a good idea to add the change log info as you change the commit, not later when you can’t remember which patch you changed. You can always go back and change or remove logs from commits.
Some mailing lists have size limits and when we add binary contents to our patches it’s easy to exceed the size limits. Use "–no-binary" to generate patches without any binary contents. You are supposed to include a link to a git repository in your "Commit-notes", "Series-notes" or "Cover-letter" for maintainers to fetch the original commit.
Patches will have no changelog entries for revisions where they did not change. For clarity, if there are no changes for this patch in the most recent revision of the series, a note will be added. For example, a patch with the following tags in the commit:
Series-version: 5 Series-changes: 2 - Some change Series-changes: 4 - Another change
would have a changelog of::
(no changes since v4) Changes in v4: - Another change Changes in v2: - Some change
Other thoughts
This script has been split into sensible files but still needs work. Most of these are indicated by a TODO in the code.
It would be nice if this could handle the In-reply-to side of things.
The tests are incomplete, as is customary. Use the ‘test’ subcommand to run them:
$tools/patman/patmantest
Note that since the test suite depends on data files only available in the git checkout, the test command is hidden unless patman is invoked from the U-Boot git repository.
Alternatively, you can run the test suite via Pytest:
$cdtools/patman&&pytest
Error handling doesn’t always produce friendly error messages - e.g. putting an incorrect tag in a commit may provide a confusing message.
There might be a few other features not mentioned in this README. They might be bugs. In particular, tags are case sensitive which is probably a bad thing.