On Ruby

I wrote about puppet and how we're starting to look into it at work a couple of days ago. Yesterday, I learned about an awesome little tool that puppet provides — ralsh, the Ruby Abstraction Layer SHell. To understand just how cool ralsh is, you need to know a little bit more about how puppet works. puppet allows you to describe how a server should 'look' using a declarative language (this is a hurdle a lot of people have problems with, but it's too big to go into here). It then provides the plumbing to enact those declarations on different kinds of servers (OS X, Solaris, Linux, etc.) using its 'Ruby Abstraction Layer'. This allows you to define something like:

user { 'postfix':
 ensure => 'present',
 uid => '51',
 comment => 'Postfix Daemon',
 gid => '51',
 home => '/var/spool/postfix',
 shell => '/bin/false'
}

Which says that you should have a user called postfix, defined as listed. If that user doesn't exist on a system where this is defined, puppet will do the right thing to add them for that kind of box. ralsh provides commandline access to this abstraction layer so that you can query the current configuration of the system, or modify that configuration. When used to query a system ralsh dumps out the current configuration in puppet's declarative language. For example:

$ ralsh service apache2
service { 'apache2':
 ensure => 'running',
 enable => 'false'
}
$

If you think about this for a minute, you'll see that it's a lot cooler than it first appears. Let's say you've got 15 servers that have been 'built' piecemeal. You're really not sure what's installed on them, or what's running there. Now, let's make a bad situation even worse, you need to build out a duplicate set of servers, only you're going to migrate from SLES to debian. ralsh suddenly looks like a great way to audit those servers and to provide a basic puppet configuration to build out the new ones. Don't expect this process to be perfect — debian and SLES name their packages differently enough, for example, that you're going to have to sort some things out by hand. It's a great way to start though. puppet and ralsh look like great tools to help you get out the door before five.

I've been trying to introduce Ruby at work for a long time, and have had some success here. I've been able to show many of the developers the value of Ruby, and it's made it's way into the organization as a valuable tool. I've never been able to make many inroads into the systems administration team that I work with though, and this has bothered me because I see myself as much more of a sys admin that a programmer. Something's changed recently though. We're working on some new projects, and Puppet is a good model for us to understand how we can automate the configuration of the systems involved. I've been watching Puppet for a while, and even interviewed Luke Kanies (or see his blog, the maintainer of Puppet, and James Turnbull, the author of Pulling Strings with Puppet: Configuration Management Made Easy. Now that some of my co-workers are looking at Puppet, I think they're starting to see the beauty that drew me into Ruby many years ago. I'm looking forward to working with them as we start to build out tools for our new project ... in Ruby. Who knows, maybe I'll even get one or two of them to come to MountainWest RubyConf 2009 with me.

[フレーム]

Puppet is a great looking configuration management tool written in Ruby, from Luke Kanies at Reductive Labs. Recently, James Turnbull has written a book, Pulling Strings with Puppet, about it.

I’ve interviewed both Luke and James about Ruby, Puppet, and James’ book. So, it’s only fair that I turn an eye to the book as well. In the interest of full disclosure I should mention that Apress gave me a PDF of this book for review purposes.

Pulling Strings with Puppet was published as an eBook, through Apress’ imprint firstPress. It’s a really big book to see published this way, coming in at 187 pages, two or three times the size of a lot of ebooks. Like other books in the firstPress line, this one is also available in print (the amazon links in this review point to the paper copy, if you want the PDF, look here and follow the link).

James is an accomplished Apress author, having also published Pro Nagios 2.0 and Hardening Linux (both of which come highly regarded). As such, he’s certainly proven his chops in the sysadmin world, so his book on Puppet can be seen as something of a vote of confidence in the system.

The book contains seven chapters, as follows:

1. Introducing Puppet
2. Installing and Running Puppet
3. Speaking Puppet
4. Using Puppet
5. Reporting on Puppet
6. Advanced Puppet
7. Extending Puppet

The first chapter should whet your appetite for the rest of the book, chapters three and four will get you running along smoothly, but I think it’s the last three chapters that really make this book worthwhile. In chapter five the discussion of the built in report tools and the mechanism for building custom reporting will be of interest. I really liked chapter six’s coverage of scalability (both the admission that Puppet still needs work and the workarounds that can help scale it until the work gets done). All of chapter seven was interesting reading, and shows the ease of extending Puppet and Facter (one of the underlying libraries that makes Puppet work).

This book is filled with helpful code samples and pointers to external resources that look very useful. It’s well written and easy to understand. As good a tool as Puppet looks to be, this looks like an equally good book to get you going. If you’re doing configuration management for anything more than a box or two, run, don’t walk, and pick up your copy of Pulling Strings with Puppet.

[フレーム]

Recently, I’ve been reading about Puppet because of James Turnbull’s excellent Pulling Strings with Puppet. James has been good enough to do a short interview with me as well. I hope you enjoy reading it as much as I enjoyed chatting with James (you might want to check out my interview with Luke Kanies too).

---

I normally think of you as a Pythonista, yet Puppet comes from the land of Ruby. How much does that matter to you? How much should it matter to an end user?

Actually I’d probably be classed a Perl Monger rather than a Pythonista. Either way I am a “hack & slash” programmer in both languages – functionality over elegance.

I took my first serious look at Ruby about the same time I found Puppet. I initially thought that the language wouldn’t matter to me at all but I confess Ruby has sucked me in a great deal more than I had anticipated. It is a very elegant language and one well suited to beginners and system administrators. I find Ruby’s syntax and flow control very easy to grasp and yet a few simple extensions and you are doing some very powerful things. I think when more system administrators discover Ruby we’ll see a slow movement away from Perl and Python as the core “sysadmin scripting” languages.

As a Puppet end user Ruby almost shouldn’t matter to you though. The “almost” relates to exactly where Puppet is in its life cycle. Puppet is still a young product. It is highly flexible and featured and if you’re using Puppet in serious anger about 80% of what you need is there. But there are still some resources you can’t manage. The fastest way to fix that? Develop these resources yourself. Luke Kanies, Puppet’s author, has built a very simple framework that with some Ruby knowledge allows you to do this. We are also seeing more development input as the community expands and people are starting to get more involved. This may provide most of the additional resource types needed right now.

How did you discover Puppet?

By accident mostly. I was researching configuration management tools and I was reading a blog post and someone had commented on this upstart new tool called Puppet. I went over to the Reductive Labs site and downloaded it. Fifteen minutes later I had a master and 5 clients running managing my DMZ servers. Two days after that I was starting to think – “Gee it’d be nice if Puppet did…” And then I was hooked.

Any recommendations on books, blogs, or websites for sysadmins who want to learn more about Ruby?

I like Apress' (disclosure – I am obviously an Apress author) Beginning Ruby. I it found a very useful book – easy to use as a reference and easy to read through. I also found that the Ruby documentation and the basic tutorial at tryruby.hobix.com are good places to start.

Why not CFEngine, one of the other systems management tools, or even a roll-your-own solution?

I have used CFEngine in the past and I know CFEngine has a very loyal user base. It’s a solid tool for many purposes but it has weaknesses and a lot of things that I personally don’t like about it. It also has a long development cycle and a limited number of people contributing to its development. Alternatively I see Puppet as having a quite dynamic community with developers who are responsive to community requirements. ~ Hence for me Puppet is a better choice of tool and one I felt it was important to support and develop.

Using “roll-your-own” tools I think defeats the whole purpose of configuration management. The scripts usually require trees of case statements to cater for varying operating systems and platforms, they usually aren’t portable and they require a communications mechanism that usually defaults to ssh and a for loop. Puppet makes all of that go away – it is a configuration management abstraction language combined with a secure client-server mechanism to configure your hosts. It takes all the pain out of systematically and efficiently managing your hosts.

Puppet seems like a good answer on the configuration management end of the sysadmin house. Is there room for a Ruby solution on the monitoring side as well? Or do you think nagios, mon, or some other solution have that all sewn up? (If so, which one?)

There is a Ruby solution that I quite like for monitoring called “god”. It is very immature compared to Nagios or others of that ilk but it has some interesting concepts. You can see it at god.rubyforge.org. I think there is always room for new ideas and new approaches in system administration tools.

Since you were learning Ruby and Puppet at the same time, what kinds of things did Puppet teach you about the language?

Well I had little understanding of classes, modules and related concepts—actually very little OO experience. Understanding Puppet required gaining an understanding of Ruby’s OO nature. That has also made a big difference to how I code overall.

What kinds of things did writing the book teach you about Ruby and Puppet?

Well I hadn’t looked at storing hosts in LDAP or had much experience with Mongrel. Writing the book meant I had to delve quite deeply into both. Also explaining to people how to create your own extensions to Puppet to manage other resources also meant I had to ensure I learnt enough Ruby to comfortably explain the concepts involved.

Can you show us a little bit of Puppet in action?

Well I can show you a very simple example of managing a resource. Let’s say you wanted to ensure the sshd service was enabled and running on hosts. In Puppet you would define the following resource:

service { "sshd":
~ enable => true,
~ ensure => running,
}

>

And that’s it! You can then assign this resource to the hosts which need it and Puppet will automatically ensure that the service is enabled and started. The best thing is that this same resource definition will work on Red Hat, Debian, Solaris, BSD and others. This is the magic of Puppet’s configuration abstraction – you only need to define what the resource should look like. Puppet takes cares of the how.

What’s the coolest thing you’ve done (or seen someone else do) with Puppet?

That is a hard choice because lots of people are doing lots of cool things. There is a large multi-coloured search engine company who are managing many thousands of OS X desktops with Puppet and the Fedora Project who use it to manage all of their infrastructure.

Probably the coolest thing I see is how people use Puppet in ways I hadn’t considered. I regularly see people on the mailing list and IRC channel say “Wow – I just found out the syntax can do x”. I look at what they have posted and they have often found some new way to take advantage of the language to configure hosts. It is being part of this sort of community that makes working with Puppet very cool.

What non-Puppet Ruby projects are you watching/using?

I’ve recently started playing with rbot—an IRC bot written in Ruby. That also led me to another IRC bot framework called Autumn Leaves. But generally due to a distinct lack of time most of my focus is on Puppet and Facter.

Who gets the credit (Or is it blame?) for the title of your book, “Pulling Strings with Puppet”?

That’d be my editor and the marketing guys at Apress. Do you know how excited marketing people are when a product allows amusing alliteration and puns? :) But I like it – it’s both kitsch and catchy.

I’m currently reading Pulling Strings with Puppet, the new book from Apress. So far I’m really enjoying it, look for a review in a week or two. The book whetted my appetite to learn more about Puppet though, so I decided to go to the source and talk to Luke Kanies of Reductive Labs, the development team behind Puppet.

Hey, I've just posted my interview with James Turnbull too.

Here’s my interview with Luke.

---

How did you get started with Ruby?

About 4 years ago, I had a pretty clear idea in my head of how I wanted to implement a tool to manage resources, and specifically how each resource type would basically just be a collection of attributes, each with their own behaviour and with the majority of the resource’s behaviour coming from the attributes, not the resource itself. I’m not saying this was a good idea, just that I had it. :)

I was a sysadmin who did a lot of development at the time, which basically meant I was a perl developer, and I was mostly doing OO in my perl. I tried to implement my idea in perl, but I just couldn’t get the class relationships to work (the attributes and resource types each needed to be classes, according to the design in my head). This was back when Python was the shiznit, so I naturally tried it, but Python just makes my eyes bleed (and no, it wasn’t the whitespace, it was things like the fact that ‘print’ was a statement instead of a function, and ‘len’ was a function instead of a method).

I had a friend who had heard Ruby was cool but hadn’t actually tried it himself. Since I was just messing around at the time, I figured I’d give it a go. Four hours in, never having seen a line of Ruby previously, I had a functional prototype.

When I decided to go full time on Puppet, I spent a lot of time agonizing over whether to stick with Ruby (I can’t seem to find a link to the original discussion), because I was rightly concerned that not many people had Ruby deployed and in fact it was a very niche language at the time (this was long before Rails had ever been released).

In the end, I figured developer productivity trumped nearly everything, so I jumped.

How did you get started with systems management?

I am apparently fantastically good at breaking computers, and not so good at fixing them. Thus, when I did fix them, I tended to write code to allow me to repeat that fix. Even in m earliest computer-using days (I’m a Johnny-come-lately—my first computer had a CD-ROM drive and a 500MB hard drive), I could rebuild my computer by booting from a different drive and moving a few directories around.

As I used this ability to get computer jobs, I found that life was always easier when I could teach the computer to do the work rather than doing it myself—the computer doesn’t get bored, and if it’s complaining, I can’t tell. Things got really going for me when I switched from MacOS to *nix (yes, I was doing near-unattended MacOS installs in 1997, using AppleScript and everything).

As the tools got bigger, so did my dreams, and eventually I realized that the only person who was going to create the tool I wanted was me.

Why Puppet?

Puppet’s fundamental advance is its concept of a resource—it builds an abstraction over the things we need to manage, and then uses this abstraction as a kind of API into the operating system. This abstraction allows us to build truly portable configurations that work across multiple platforms, and it also enforces a declarative view of the world instead of the traditional procedural scripting view. See my most recent LCA presentation¹ for more information.

The goal for Puppet’s resource abstraction layer is that it can be the lowest layer in a configuration management stack, but so far we haven’t had the time to get much beyond Puppet itself.

What are your future plans for Puppet?

I’m pushing toward a 1.0 this year, hopefully, as soon as I can get the critical APIs stable. I’m also hoping to add a lot of interesting functionality around making each host’s resource catalog more useful outside of Puppet—e.g., you could have all of your resource relationships set up in it, modify /etc/ssh/sshd_config, and then tell Puppet to figure out what services need to restart because of that change.

As we move toward a more database-backed catalog, vs. the current YAML-dumped version, we’ll get a lot more functionality out of it yet, and I can’t really even see most of that functionality right now.

What are some cool things that people are doing with Puppet?

Google’ s managing more than 6000 Mac OS X laptops and desktops with it, which I think is pretty cool. Red Hat has built a tool called cft² , which goes some way toward converting traditional-style work into Puppet code. Julian Simpson at Thoughtworks has done some cool integration with Cruise Control that uses Puppet and virtualization to automate testing Puppet configuration³ .

Puppet seems like a great solution on the front end of the sysadmin space (configuration management). Is there room for (or a need for) a good Ruby monitoring framework? If not, why not?

I’m not really sure that the language of the framework matters all that much. I chose Ruby because it did the job the best for me during development, not because I wanted a Ruby version of an existing piece of software.

If someone can identify clear shortcomings in existing monitoring tools, to the point where a new tool should be created, it might make sense to implement such a tool in Ruby, but I think the monitoring space is pretty congested right now, so I certainly wouldn’t want to try to product a competitive product there right now.

Do you have a monitoring app that you’re happy with?

Fortunately, I haven’t been an operational admin in years, so I haven’t had to deal with monitoring in a long time. Thus, no, I have no app I’m particularly fond of. I hear a lot about monit, and god (which is apparently pretty new).

Apress has just released the first book on Puppet, Pulling Strings with Puppet. Have you read it yet? What do you think of it?

I haven’t read the book yet, but I worked pretty closely with James Turnbull as he wrote it, and he’s had a huge impact on the community during its writing. He updated many docs online with the info he gleaned from me, and it seems like the community is very fond of the book.

I just got back from Melbourne, Australia, where James lives, and he’s a great guy; it’s definitely nice to get to meet the various community members in person.

¹ LCA Presentation (search for luke and download the ogg)

²Red Hat’s cft

³Thoughtworks Puppet stuff