In discussions with Perry Greenfield at STScI, it seems we are in the position of being able to get some financial support to pay for a continuous integration system. Travis has shown itself to be rather glitchy for matplotlib. The tight integration with Github PRs is really convenient. However, matplotlib has longer test runs and more dependencies than many projects. The frequent false positives start to wear down at a certain point. And we still aren't testing everything because we can't aren't installing Inkscape and other things in the Travis VM. So we're looking to add another paid, hosted continuous integration service that will better meet our needs. A hosted service is nice because it's easy to give multiple developers access to the system so anyone can tweak it and keep it going -- the bottleneck of a single developer responsible for maintenance of the build machine was a problem years ago when we were using buildbot. This may remain "in addition to" rather than "instead of" Travis for some time. An obvious first choice to me is ShiningPanda, as I have experience using it for astropy. Jenkins (the software Shining Panda is based on), is a really easy-to-use system, for those not familiar with it. And we can store the output of the tests (i.e. the result_images) for later inspection. I think this feature alone is huge for matplotlib. They also offer Windows build slaves. There is no OS-X (probably because Apple licensing doesn't really allow for use in a VM), but results can be "published" to their Jenkins instance. Are there any other similar alternatives we should consider before we move forward? Mike
On Wed, Jan 30, 2013 at 8:44 AM, Michael Droettboom <md...@st...> wrote: > In discussions with Perry Greenfield at STScI, it seems we are in the > position of being able to get some financial support to pay for a > continuous integration system. > > Travis has shown itself to be rather glitchy for matplotlib. The tight > integration with Github PRs is really convenient. However, matplotlib > has longer test runs and more dependencies than many projects. The > frequent false positives start to wear down at a certain point. And we > still aren't testing everything because we can't aren't installing > Inkscape and other things in the Travis VM. > > So we're looking to add another paid, hosted continuous integration > service that will better meet our needs. A hosted service is nice > because it's easy to give multiple developers access to the system so > anyone can tweak it and keep it going -- the bottleneck of a single > developer responsible for maintenance of the build machine was a problem > years ago when we were using buildbot. This may remain "in addition to" > rather than "instead of" Travis for some time. > > An obvious first choice to me is ShiningPanda, as I have experience > using it for astropy. Jenkins (the software Shining Panda is based on), > is a really easy-to-use system, for those not familiar with it. And we > can store the output of the tests (i.e. the result_images) for later > inspection. I think this feature alone is huge for matplotlib. They > also offer Windows build slaves. There is no OS-X (probably because > Apple licensing doesn't really allow for use in a VM), but results can > be "published" to their Jenkins instance. > > Are there any other similar alternatives we should consider before we > move forward? > > Mike I think hosted infrastructure is the right choice. I was initially going to suggest that we try and work with a bespoke solution. That way we could roll our own build architectures. On reflection I think the maintenance headache of managing our own build slaves outweighs the convenience of having it hosted, as you point out. -- Damon McDougall http://www.damon-is-a-geek.com Institute for Computational Engineering Sciences 201 E. 24th St. Stop C0200 The University of Texas at Austin Austin, TX 78712-1229
On Wed, Jan 30, 2013 at 9:27 PM, Damon McDougall <dam...@gm...> wrote: > On Wed, Jan 30, 2013 at 8:44 AM, Michael Droettboom <md...@st...> wrote: >> In discussions with Perry Greenfield at STScI, it seems we are in the >> position of being able to get some financial support to pay for a >> continuous integration system. >> >> Travis has shown itself to be rather glitchy for matplotlib. The tight >> integration with Github PRs is really convenient. However, matplotlib >> has longer test runs and more dependencies than many projects. The >> frequent false positives start to wear down at a certain point. And we >> still aren't testing everything because we can't aren't installing >> Inkscape and other things in the Travis VM. >> >> So we're looking to add another paid, hosted continuous integration >> service that will better meet our needs. A hosted service is nice >> because it's easy to give multiple developers access to the system so >> anyone can tweak it and keep it going -- the bottleneck of a single >> developer responsible for maintenance of the build machine was a problem >> years ago when we were using buildbot. This may remain "in addition to" >> rather than "instead of" Travis for some time. >> >> An obvious first choice to me is ShiningPanda, as I have experience >> using it for astropy. Jenkins (the software Shining Panda is based on), >> is a really easy-to-use system, for those not familiar with it. And we >> can store the output of the tests (i.e. the result_images) for later >> inspection. I think this feature alone is huge for matplotlib. They >> also offer Windows build slaves. There is no OS-X (probably because >> Apple licensing doesn't really allow for use in a VM), but results can >> be "published" to their Jenkins instance. >> >> Are there any other similar alternatives we should consider before we >> move forward? >> >> Mike > > I think hosted infrastructure is the right choice. I was initially > going to suggest that we try and work with a bespoke solution. That > way we could roll our own build architectures. > > On reflection I think the maintenance headache of managing our own > build slaves outweighs the convenience of having it hosted, as you > point out. Of course, you're still going to need people who are willing/able to maintain OSX build slaves if you want to do testing on OSX (which obviously you should). It's easy with Jenkins to run your own instances and have it push results to the Shining Panda instance or any other hosted service. But otherwise you're back to square one in terms of having to rely on the people running the OSX builds. We've had this problem on Astropy in that I'm maintaining our current only Windows machine (ShingingPanda does have Windows support but a a cost) and so whenever the build bot itself breaks on Windows it's up to me to do anything about it. It also makes it hard for other admins to make tweaks to the build configuration. Unfortunately you're probably going to have that problem with OSX and probably Windows with any other hosted service as well. Erik