Sam asked me to post here after we talked about this topic on IRC.
A lot of projects have found it beneficial to spider d.o to retrieve information from the project pages. I think it would be great to have a public REST API around d.o projects instead. This would reduce load on d.o and also increase the accuracy of the information being retrieved. vThis would include a lot of the information from the git repo, but also general project information like description, maintainers, usage stats, etc. I am format-agnostic, although it would be nice to support a variety of formats dependent on accepts headers. Obviously I'd love to see Services run this but whatever on that really. Once the project information is in place, this could be expanded to issues which could open up some kick ass possibilities. Remote Views query backend for issue queues anyone? It would also then lead to an issue posting API which would be A++. Git integration there could include the commit the issue was fixed with, the branch the issue is currently being worked on in, whatever. Many many possibilities for awesomesauce. Discuss!
Here is a halfassed representation of how this could look.
Title
Short Name
Description
Maintainer
Committers
Name
Name
Name
Maintenance Status
Development Status
Categories
Category
Category
Reported Installs
Usage History
Date
Version
Version
Date
Version
Version
Recommended Releases
Release
Date
Zip Download
Tarball Download
Release
Date
Zip Download
Tarball Download
Recent Commits
Hash
Who
Comment
Files
File
File
Branches
Branch
Branch
Clone URL
Comments
AWESOME!!!The issues api
AWESOME!!!
The issues api would really help my git-release-notes, which pulls apart the issue page to get the category for sorting.
BTW, there is already an issue for remote issue access, see http://drupal.org/node/1002712
I go by Josh The Geek.
Drupal.org
GitHub
When heyrocker first asked me
When heyrocker first asked me about this, I chuckled a bit because a) I would LOVE to write some excellent APIs that dive into our project/Git data, and b) because we already provide some of this for internal infra purposes. I pastebinned the following as an example:
{"project":"Drupal core","project_nid":"3060","repository_name":"drupal","repo_id":"4","users":{"drumm":{"uid":"3064","repo_id":"4","access":"2","branch_create":"0","branch_update":"0","branch_delete":"0","tag_create":"0","tag_update":"0","tag_delete":"0","per-label":[],"name":"drumm","pass":"","ssh_keys":{" ":" "},"global":1},"webchick":{"uid":"24967","repo_id":"4","access":"2","branch_create":"0","branch_update":"0","branch_delete":"0","tag_create":"0","tag_update":"0","tag_delete":"0","per-label":[],"name":"webchick","pass":"","ssh_keys":{" ":" "},"global":1},"rfay":{"uid":"30906","repo_id":"4","access":"2","branch_create":"0","branch_update":"0","branch_delete":"0","tag_create":"0","tag_update":"0","tag_delete":"0","per-label":[],"name":"rfay","pass":"","ssh_keys":{" ":" "},"global":0},"dries":{"uid":"1","repo_id":"4","access":"2","branch_create":"0","branch_update":"0","branch_delete":"0","tag_create":"0","tag_update":"0","tag_delete":"0","per-label":[],"name":"Dries","pass":"","ssh_keys":{" ":" "},"global":0},"":{"uid":"3","repo_id":"4","access":"2","branch_create":"0","branch_update":"0","branch_delete":"0","tag_create":"0","tag_update":"0","tag_delete":"0","per-label":[],"name":"Drupal","pass":"","ssh_keys":{" ":" "},"global":1},"goba":{"uid":"4166","repo_id":"4","access":"2","branch_create":"0","branch_update":"0","branch_delete":"0","tag_create":"0","tag_update":"0","tag_delete":"0","per-label":[],"name":"G\u00e1bor Hojtsy","pass":"","ssh_keys":{" ":" "},"global":1},"killes":{"uid":"227","repo_id":"4","access":"2","branch_create":"0","branch_update":"0","branch_delete":"0","tag_create":"0","tag_update":"0","tag_delete":"0","per-label":[],"name":"killes@www.drop.org","pass":"","ssh_keys":{" ":" "},"global":1}},"protected_labels":{"tags":["DRUPAL-4-3-0-RC","DRUPAL-4-1-1","DRUPAL-4-5-0-RC","4.4.0","DRUPAL-4-6-0-RC","4.0.0","4.4.2","4.5.0","4.1.0","4.3.1","4.5.2","4.2.0","4.4.1","4.3.0","4.5.1","x_y_z","4.3.2","4.6.0","4.5.3","4.6.1","4.4.3","4.5.4","4.6.2","4.5.5","4.6.3","4.5.6","4.6.4","DRUPAL-4-7-0-BETA-1","4.5.7","4.6.5","DRUPAL-4-7-0-BETA-2","4.7.0-beta-3","4.7.0-beta-4","4.7.0","4.7.0-rc-4","4.7.0-rc-3","4.7.0-beta-5","4.5.8","4.6.6","4.7.0-beta-6","4.7.0-rc-1","4.7.0-rc-2","4.7.1","4.6.7","4.7.2","4.6.8","4.7.3","4.6.9","4.7.4","4.6.10","5.0-beta-1","5.0-beta-2","5.0-rc-1","4.6.11","4.7.5","5.0-rc-2","5.0","4.7.6","5.1","5.2","4.7.7","6.0-beta-1","5.3","4.7.8","6.0-beta-2","6.0-beta-3","4.7.9","5.4","6.0-beta-4","5.5","4.7.10","6.0-rc-1","5.6","4.7.11","6.0-rc-2","5.7","6.0-rc-3","6.0-rc-4","6.0","6.1","6.2","6.3","5.8","5.9","5.10","6.4","5.11","6.5","6.6","5.12","6.7","5.13","5.14","6.8","5.15","6.9","6.10","5.16","5.17","6.11","6.12","5.18","6.13","5.19","6.14","5.20","5.21","6.15","7.0-alpha1","7.0-alpha2","6.16","5.22","7.0-alpha3","7.0-alpha4","7.0-alpha5","6.17","7.0-alpha6","6.18","6.19","5.23","7.0-alpha7","7.0-beta1","7.0-beta2","7.0-beta3","7.0-rc-1","7.0-rc-2","6.20","7.0-rc-3","7.0-rc-4","7.0"],"branches":["5.x","6.x","4.6.x","master","HEAD"]},"repo_group":1}That's JSON representing all the current information about a project (core, in this case), its maintainers, and its releases. This data is cached by varnish so it's highly available, and selectively clears the cache on any web-frontend-driven events that alter the underlying dataset. We currently use this data in the twisted ssh daemon (that your request passes through every time you clone/push over ssh) and in our serverside git hook scripts. It's a bit rough, but is nevertheless a functioning API. It'll never be public-facing (it contains password MD5s when properly populated) and is pretty sensitive to our internal needs...but the underlying point here is, this is a family of problems we've already solved once. And I'd be excited - once we're in phase 3 - to solve it again, and for public consumption.
Without the password md5s,
Without the password md5s, maintainers (maybe just the names). Maybe also include the project node content?
EDIT: Maybe i should read the whole post, not just comments. :)
I go by Josh The Geek.
Drupal.org
GitHub
Oh, and one fun point - note
Oh, and one fun point - note that there's a lot of granular ACL data in there - per-branch & per-tag permissions. I baked support in for that to the base architecture a months ago. All it really needs is a UI and some additions in our serverside git hooks, and we'll have it.
AWESOME!!!
AWESOME!!! That would be cool. Where should I open a feature request?
I go by Josh The Geek.
Drupal.org
GitHub
http://drupal.org/project/ver
http://drupal.org/project/versioncontrol_project , and file it with the 'git phase 3' tag.
http://drupal.org/node/106000
http://drupal.org/node/1060000
I go by Josh The Geek.
Drupal.org
GitHub
+1 to per branch/per tag
+1 to per branch/per tag
I go by Josh The Geek.
Drupal.org
GitHub
I think the idea of exposing
I think the idea of exposing d.o. data via an API is fantastic. Ideally, this would just be the first set of data. Things like commit logs by user would also be quite useful.
Yes!
This would open the door for direct issue queue integration with various IDEs, 3rd party project management tools, and so forth. The awesomeness of that should not be under-stated.
Indeed it should not be. +1.
Indeed it should not be. +1.
WizOne Solutions - https://wizone.solutions - Drupal module development, theme implementation, and more
FillPDF Service - https://fillpdf.io - Hosted solution for FillPDF
+1
+1
Would love to see this
Currently localize.drupal.org integrates with d.o projects and releases via some custom means, which is not too nice and does not lend well for changes in the d.o infrastructure. We do have some XML data about projects and releases in fact for update status, that is openly available. It does not include some of this data, but it includes releases and most basic info about the project. We do not have an API to get a list of projects though, and I assume the update status data should not be soaped up with all kinds of other data, because that would penalize the performance of update status requests.
Yeah, I have no interest in
Yeah, I have no interest in bloating update status any further. I also have no interest in publishing more XML APIs unless the verbosity is actually necessary. JSON & such like it are very effective, lightweight transports.
A public REST API with OAuth
A public REST API with OAuth authentication for interacting with it would be awesome. I would be happy to contribute towards such an API and would love to see it based on Services 3.x and its REST Server.
The "only" thing needed to create an API for Project.module based on Services would be to create some custom resources for the project module and bundle it up with an exported endpoint and if we want OAuth also bundle it up with an exported context. I think OAuth should be a step 2 - first a read only API and after that add write features to it.
Where do we start? :)
Oh, and one fun point - note
Oh, and one fun point - note that there's a lot of granular ACL data in there - per-branch & per-tag permissions. I baked support in for that to the base architecture a months ago. All it really needs is a UI and some additions in our serverside git hooks, and we'll have it.
++
This smells awesomeness to me! :D
Existing issue
I think: http://drupal.org/node/112805
I'm thinking there was at least one other one about exposing Project* data to the outside world, but I can't find it atm.
Also, for people who want to start hacking on this...
I've been puttering away on http://drupal.org/project/drupalorg_testing in my "spare" time, and it's kind of working now, for basic stuff. It's an installation profile that gets you set up with all the various modules Drupal.org uses, and configures some basic starter data like projects, issues, releases, etc.
+1
+1
+1
Sounds awesome! I am really looking forward to write some integration for our internal project management tool :)
drush issue queue commands
This will also be very useful for the upcoming drush issue queue commands, which currently scrape html to find patches.
Very much agreed. drush issue
Very much agreed. drush issue queue interaction is one of the things I personally want a better project API for, actually. been on my list for a long time.