Ola Bini: Programming Language Synchronicity

I have been involved in several discussions about programming languages lately and people have assumed that since I spend lots of time in the Ruby world and with dynamic languages in general I don't like static typing. Many people in the dynamic language communities definitely expresses opinions that sound like they dislike static typing quite a lot.

I'm trying to not sound defensive here, but I feel the need to clarify my position on the whole discussion. Partly because I think that people are being extremely dogmatic and short-sighted by having an attitude like that.

Most of my time I spend coding in Java and in Ruby. My personal preference are to languages such as Common Lisp and Io, but there is no real chance to use them in my day-to-day work. Ruby neatly fits the purpose of a dynamic language that is close to Lisp for my taste. And I'm involved in JRuby because I believe that there is great worth in the Java platform, but also that many Java programmers would benefit from staying less in the Java language.

I have done my time with Haskell, ML, Scala and several other quite statically typed languages. In general, the fact that I don't speak that much about those languages is that I have less exposure to them in my day-to-day life.

But this is the thing. I don't dislike static typing. Absolutely not. It's extremely useful. In many circumstances it gives me things that really can't have in a dynamic language.

Interesting thought: Smalltalk is generally called a dynamic language with very late binding. There are no static type tags and no type inference happening. The only type checking that happens will happen at runtime. In this regard, Smalltalk is exactly like Ruby. The main difference is that when you're working with Smalltalk, it is _always_ runtime. Because of the image based system, the type checking actually happens when you do the programming. There is no real difference between coding time and runtime. Interestingly, this means that Smalltalk tools and environments have most of the same features as a static programming language, while still being dynamic. So a runtime based image system in a dynamic, late bound programming language will actually give you many of the benefits of static typing at compile time.

So the main take away is that I really believe that static typing is extremely important. It's very, very useful, but not in all circumstances. The fact that we reach for a statically typed programming language by default is something we really need to work with though, because it's not always the right choice. I'm going to say this in even stronger words. In most cases a statically typed language is a premature optimization that gets extremely much in the way of productivity. That doesn't mean you shouldn't choose a statically typed language when it's the best solution for the problem. But this should be a conscious choice and not by fiat, just because Java is one of the dominant languages right now. And if you need a statically typed language, make sure that you choose one that doesn't revel in unnecessary type tags. (Java, C#, C++, I'm looking at you.) My current choice for a static language is Scala - it strikes a good balance in most cases.

A statically typed language with type inference will give you some of the same benefits as a good dynamic language, but definitely not all of them. In particular, you get different benefits and a larger degree of flexibility from a dynamic language that can't be achieved in a static language. Neal Ford and others have been talking about the distinction between dynamic and static typing as being incorrect. The real question is between essence and ceremony. Java is a ceremonious language because it needs you to do several dances to the rain gods to declare even the simplest form of method. In an essential language you will say what you need to say, but nothing else. This is one of the reasons dynamic languages and type inferenced static languages sometimes look quite alike - it's the absence of ceremony that people react to. That doesn't mean any essential language can be replaced by another. And with regads to ceremony - don't use a ceremonious language at all. Please. There is no reason and there are many alternatives that are better.

My three level architecture of layers should probably be updated to say that the stable layer should be an essential, statically typed language. The soft/dynamic layer should almost always be a strongly typed dynamic essential language, and the DSL layers stays the same.

Basically, I believe that you should be extremely pragmatic with regards to both static and dynamic typing. They are tools that solve different problems. But out industry today have a tendency to be very dogmatic about these issues, and that's the real danger I think. I'm happy to see language-oriented programming and polyglot programming get more traction, because they improve a programmers pragmatic sensibilities.

Someone asked in a comment where my Scala exploration were going. It's a fair question.

I did a first implementation of the core of the system I was using. Most of if was tested using Specs (which incidentally is very good, and even more importantly - have a responsive creator). Now, I realized a while back that the implementation weren't really using Scala properly. I had mostly used the imperative way of doing things. So I'm planning a rewrite of these parts, using pattern matchin, case classes and more recursive algorithms correctly. It's kinda funny. I didn't expect this to happen since I have so much experience with functional languages. But apparently, the fact that Scala looks like Java in many regards makes it very easy to slip into the Java imperative mode. Not good.

Right now I'm spending some time working on the next release of JtestR, writing on a book, and doing some pesky day time work. =) Also, JRuby needs to have its 1.1 release. I expect to get back to Scala within one or two weeks, though.

This post is basically a follow up to Language explorations. That post had several statements about what I believe, and some of them have been a bit misunderstood, so I'll try to substantiate them a bit more here. Specifically, I want to talk about the viability of the Java language and whether "the stable layer" should be a static or dynamic language.

Lets begin with the stable layer. First, notice that I didn't have any specific definitions for the different layers when I wrote the post. In fact, I think that they are inherently a bit fuzzy. That's OK. DSL's are also quite fuzzy. The attributes I put at the stable layer is first of all that it shouldn't need to change much. In a smaller application, you can actually look at supporting structures as the stable layer, things like application servers or web servers. But for larger applications I find that you usually need your own stable kernel.

The reasons I choose static typing for stable layer is for several reasons. Performance is absolutely one of these, since everything will run on this base it's an advantage if that part is as performant as possible. Secondly, I am no dynamic language, TDD purist that says that static typing is totally unnecessary. Static type checking can be very helpful, but it stops you from having a malleable language. So the kernel should be stable, this means that as much checking as possible is necessary. None of these requirements explicitly preclude a dynamic language, but the specific features of dynamic languages doesn't really come to their rights at this level, and also leading to performance and static type checking suffering.

The more contentious part about my post seems to have been my off hand comment that I don't believe Java is really good for anything. I didn't give much reason in that post, and people have reacted in different ways, from saying that "Java is really good" to "Why are you involved in JRuby if you think Java is so bad?". So lets start with why Java is a bad language. (I am only talking about the language here, not the platform). Also, that Java is a bad doesn't say anything about the worse alternatives, so no comments along the lines of "if you think Java is so bad, why don't you use X instead".

In short:

• Java is extremely verbose. This is really a few different problems in one:
• No type inference
• Verbose generic type signatures
• Anonymous class implementations
• There is no way to create new kinds of abstractions:
• No closures
• Anonymous classes are a really clunky way to handle "block" functionality
• Broken generics implementation
• Language is becoming bloated

A few notes. That Java is becoming bloated is one of those things that people have been blogging about lately. After 1.5 came out, the complexity of the language got very much larger, without actually adding much punch for it. The improvements of generics were mostly negated by the verbosity of them. Fixing all the problems above will bloat the language even more. And at the same time, programming in Java is needlessly painful right now.

So, lets look at the question about why I'm working on JRuby. Well, first, I believe the JVM is a very good place to be at. It's the best platform out there, in my opinion. The libraries are great, the runtime is awesome, and it's available basically everywhere. The bytecodes of the JVM spec is good enough for most purposes. There is some tweaking that can be done (which we are looking at in JSR292), but mostly it's a very nice place. And working on JRuby is really one of the ways I've seen how bad Java as a language is. We are employing several different tricks to get away from the worst parts of it, though. Code generation is used in several places. We are generating byte codes at runtime. We are using annotations to centralize handling of JRuby methods. And we are moving parts of the implementation to Ruby. I believe that JRuby is important because it can run in the same environment as Java, but without the problems of Java.

What are the solutions to the problem with Java? There are basically two different ways. Either define a subset of Java, not necessarily totally compatible, that takes the best parts of Java syntax, does away with the problems and so on. That should be possible, but I don't know anyone who has done it. Or, you can go with an existing static language on the JVM. Here you have two choices - either one of the ports of existing extremely static languages (like ML or Haskell), or you can go with something like Scala. I haven't decided on the best solution here yet. The only thing I'm sure of is that Java in itself is a problem. I'm investigating Scala, but maybe Scala isn't the right answer either. We'll see.

Oh, right. This is a perfect example of Scala magic that I REALLY don't understand yet. If there are any Scala wizards reading, please explain!

This line of code is from a test cases written in specs:

 runtime.ev("\"hello world\"")._with(interpreter)
 must be_==(runtime.stringFor("hello world"))

Runtime is a Scala class and so is interpreter. I added println()'s before and after, and at different methods in this statement. Now, the baffling thing, the really strange thing (at least in my mind), is that the left hand side of the must, gets evaluated THREE times. But this code only run once. How can that happen? What does Scala do here? (And must doesn't seem to be a special method. I looked at the implementation, and it's just a regular method on a trait, mixed into the Scala Object.

Someone please explain this! =)

It's nice to see how the amount of people looking into Scala has really exploded lately, based on the rush of blog posts an discussion.

One of the things I find a bit annoying about Scala is the proliferation of keywords. Actually, this is something I really don't like in any languages. A language should be as keywordless as possible. Of course, such a vision goes against ease of implementation for language implementers, so there always needs to be a balance here. Coming from languages such as Lisp and Io, it's amazing how clear a language can be with a well chosen message passing or invocation model. In fact, both of those languages have zero keywords. That makes it incredibly nice to implement whatever you want.

Actually, Java has been quite good at not taking much more keywords than they had from the beginning, so I found it a bit annoying when I tried to build a fluent interface in Scala, and found out that the word "with" is a keyword. And it's a keyword in the strictest sense, meaning you can't use it in places where you can't use the "with" keyword anyway. So there is no way to implement a method named "with" in Scala. Annoying. It's just, the English connection words are so much more useful for method names, especially when you can use the methods in "operator" position. Then you just want to be able to use all these words.

So. If you design a language, make sure that you take care to actually add every keyword extremely carefully. If you can, make sure that keywords can actually be used for some things when there is no ambiguity. Of course, I'm not proposing the kind of madness you can do in some languages, where a statement such as "IF IF THEN THEN" is valid, since the first IF is a keyword, the next is a variable name, etc. But be reasonable about keywords. They are sometimes necessary, but not always as often as people believe.

So. The story about unit testing is over for now. I will use specs. Eric made an incredible job and got the JUnit support working with JUnit 4 too very quickly. Today I reintegrated it, and everything works fine.

So, if you want working Ant integration for your Scala testing, I recommend specs.

Of course, this still doesn't explain while all the other alternatives failed so miserably. Hopefully there will be a bit more testing in the community soon. Testing frameworks need competition to evolve well.

It's funny. One of the points to my original post about Scala unit testing was this quote: "Now some lovely Ruby people are looking at Scala, and the very first thing they must do (of course) is write the sacred unit tests:".

I'm not sure about you, but I would say that that's a good statement about Ruby people in general, if that's the way people view us. =)

I blogged about looking at languages a while back. At that point I didn't know what my next language to explore would be. I got lots of excellent suggestions. In the end I decided to try OCaML, but gave that up quickly when I found out that half of the type system exists to cover up deficiencies in the other half of it. So I went back and decided to learn Scala. I haven't really had time to start with it though. Until now, that is.

So let's get back to the motivation here? Why do I want to learn another language? Aren't I happy with Ruby? Well, yes and no. But that's not really the point. You can always point to the Prags one-language-a-year, but that's not it either. I mean, it's really good advice, but there is a more urgent reason for me to learn Scala.

I know many people have said this before, but it bears repeating. Everyone doesn't share this opinion, but have a firm belief that the end of big languages is very close. There won't be a next big language. There might be some that are more popular than others, but the way development will happen will be much more divided into using different languages in the same project, where the different languages are suited for different things. This is the whole Polyglot idea. And my take on it is this: the JVM is the best platform there is for Polyglot platform, and I think we will see three language layers emerge in larger applications. Now, the languages won't necessarily be built on top of each other, but they will all run on the JVM.

The first layer is what I called the stable layer. It's not a very large part of the application in terms of functionality. But it's the part that everything else builds on top off, and is as such a very important part of it. This layer is the layer where static type safety will really help. Currently, Java is really the only choice for this layer. More about that later, though.

The second layer is the dynamic layer. This is where maybe half the application code resides. The language types here are predominantly dynamic, strongly typed languages running on the JVM, like JRuby, Rhino and Jython. This is also the layer where I have spent most of my time lately, with JRuby and so on. It's a nice and productive place to be, and obviously, with my fascination for JVM languages, I believe that it's the interplay between this layer and the stable layer that is really powerful.

The third layer is the domain layer. It should be implemented in DSL's, one or many depending on the needs of the system. In most cases it's probably enough to implement it as an internal DSL within the dynamic layer, and in those cases the second and third layer are not as easily distinguishable. But in some cases it's warranted to have an external DSL that can be interacted with. A typical example might be something like a rules engine (like Drools).

I think I realized a long time ago that Java is not a good enough language to implement applications. So I came up with the idea that a dynamic language on top of Java might be enough. But I'm starting to see that Java is not good enough for the stable layer either. In fact, I'm not sure if Java the language is good enough for anything, anymore. So that's what my language exploration is about. I have a suspicion that Scala might be a good language at the stable layer, but at this point the problem is there aren't any other potential languages for that layer. So what I'm doing is trying to investigate if Scala is good enough for that.

But I need to make one thing clear - I don't believe there will be a winner at any of these layers. In fact, I think it would be a clearly bad thing if any one language won at any layer. That means, I'm seeing a future where we have Jython and JRuby and Rhino and several other languages coexisting at the same layer. There doesn't need to be any rivalry or language wars. Similarly, I see even less point in Scala and Ruby being viewed as competing. In my point of view they aren't even on the same continent. And even if they were, I see no point in competing.

I got accused of being "religious" about languages yesterday. That was an interesting way of putting it, since I have always been incredibly motivated to see lots of languages coexisting, but coexisting on the JVM in a productive way.

This post is a bit of a follow up to my rant about Unit testing in Scala two days back. First let me tell you that that story actually has a happy ending. Eric Torreborne (creator of the Specs framework) immediately stepped up and helped me, and the problem seemed to be a lack of support for JUnit4 test running, which he subsequently implemented. I'm going to reintegrate
specs into my test suite later today. So that makes me happy. I might still retain JtestR. Actually, it would be a bit interesting to see the differences in writing tests in Ruby and Scala.

I spent some time on #scala on FreeNode yesterday. Overall it was an interesting experience. We ended up talking a bit about the unit testing bit, and Ant integration in particular. I'll get back to that conversation later. But this sparked in me the question why I felt that it was really important to have Ant integration. Does it actually matter?

I have kind of assumed that for a tool running on the JVM, that people might need during their build process, integration with Ant and Maven is more or less a must. It doesn't really matter what I think of these tools. If I want anyone to actually use the tool in question, this needs to work. In many cases that means it's not enough to just have a Java class that can be called with the Java-task in Ant. The integration part is a quite small step, but important enough. Or at least I think it is.

I am thinking that no matter what you think of Ant, it's one of those established tools that are here to stay for a long time. I know that I reach for Ant by default when I need something built. I know there are technologically better choices out there. Being a Ruby person, I know that Rake is totally superior, and that there is Raven and Buildr who both provide lots of support for building my project with Ruby. So why do I still reach for Ant whenever I start a new project?

I guess one of the reasons is that I'm almost always building open source projects. I want people to use my stuff, to do things with it. That means the barrier to entry need to be as low as possible. For Java, the de facto standard build system is still Ant, so the chance of people having it installed is good. Ant is easy enough to work with. Sure, it can be painful for larger things, but for smaller projects there really is no problem with Ant.

What do you think? Does it matter if you use established tools, that might be technologically inferior? Or should you always go for the best solution?

Sometimes I'm considering using other tools for build my own personal projects, but I never know enough to say with certainty I will never release it. That means I have two choices - either I use something else first and then convert it if I release it, or I just go with Ant directly.

Now, heading back to that conversation. It started with a comment about "an ant task for failing unit tests is severely overrated". Then it went rapidly downhill to "Haskell shits all over Ant for a build script for example", at which point I totally tuned out. Haskell might be extremely well suited for build scripts, but it's not an established tool that anyone can use from their tool chain. And further, I got two examples of how these build scripts would look, later in the conversation, and both of them were barely better than shell scripts for building and testing. Now there is a very good reason people aren't using shell scripts for their standard building and testing tool chain.

Or am I being totally unreasonable about this? (Note, I haven't in any way defended the technological superiority of Ant in this post, just to make it clear.)

I wish this could be a happy story. But it's really not. If I have made any factual errors in this little rant, don't hesitate to correct me - I would love to be proved wrong about this.

Actually, I wrote this introduction before I went out to celebrate New Years. Now I'm back to finish the story, and the picture have changed a bit. Not enough yet, but we'll see.

Let's tell it from the start. I have this project I've just started working on. It seemed like a fun and quite large thing that I can tinker on in my own time. It also seemed like a perfect match to implement in Scala. I haven't done anything real in Scala yet, and wanted to have a chance to do it. I like everything I've seen about the language itself. I've said so before and I'll say it again. So I decided to use it.

As you all probably know, the first step in a new project is to set up your basic structure and getting all the simple stuff working together. Right, for me that means a simple Ant script that can compile Java and Scala, package it into a jar file, and run unit tests on the code. This was simple. ... Well, except for the testing bit, that is.

It seems there are a few options for testing in Scala. The ones I found was SUnit (included in the Scala distribution), ScUnit, Rehersal and specs (which is based on ScalaCheck, another framework). So these are our contestants.

First, take SUnit -- this is a very small project, no real support for anything spectacular. The syntax kinda stinks. One class for each test? No way. Also, no integration with Ant. I haven't even tried to run this. Testing should be painless, and in this case I feel that using Java would have been an improvement.

ScUnit looked really, really promising. Quite nice syntax, lots of smarts in the framework. I liked what the documentation showed me. It had a custom Ant task and so on. Very nice. It even worked for a simple Hello, World test case. I thought that this was it. So I started writing the starting points for the first test. For some reasons I needed 20 random numbers for this test. Scala has so many ways of achieving this... I think I've tried almost all of them. But all failed with this nice class loading exception just saying InstantiationException on an anonymous function. Lovely. Through some trial and error, I found out that basically any syntax that causes Scala to generate extra classes, ScUnit will fail to run. I have no idea why.

So I gave up and started on the next framework. Rehersal. I have no idea what the misspelling is about. Anyway, this was a no show quite quickly since the Ant test task didn't even load (it referenced scala.CaseClass, which doesn't seem to be in the distribution anymore). Well then.

Finally I found specs and ScalaCheck. Now, these frameworks look mighty good, but they need better Google numbers. Specs also has the problem of being the plural of a quite common word. Not a good recipe for success. So I tried to get it working. Specs is built on top of ScalaCheck, and I much preferred the specs way of doing things (being an RSpec fan boy and all). Now specs doesn't have Ant integration at all, but it does have a JUnit compatibility layer. So I followed the documentation exactly and tried to run it with the Ant JUnit task. KABOOM. "No runnable methods". This is an error message from JUnit4. But as far as I know, I have been able to run JUnit3 classes as good as JUnit4 classes with the same classpath. Hell, JRuby uses JUnit3 syntax. So obviously I have JUnit4 somewhere on my classpath. For the world of me I cannot find it though.

It doesn't really matter. At that point I had spent several hours getting simple unit testing working. I gave up and integrated JtestR. Lovely. Half of my project will now not be Scala. I imagine I would have learned more Scala by writing tests in it, than with the implementation. Apparently not. JtestR took less than a minute to get up and working.

I am not saying anything about Scala the language here. What I am saying is that things like this need to work. The integration points need to be there, especially with Ant. Testing is the most important thing a software developer does. I mean, seriously, no matter what code you write, how do you know it works correctly unless you test it in a repeatable way? It's the only responsible way of coding.

I'm not saying I'm the worlds best coder in any way. I know the Java and Ruby worlds quite well, and I've seen lots of other stuff. But the fact that I can't get any sane testing framework in Scala up and running in several hours, with Ant, tells me that the Scala ecosystem might not be ready for some time.

Now, we'll see what happens with specs. If I get it working I'll use it to test my code. I would love that to happen. I would love to help make it happen - except I haven't learned enough Scala to actually do it yet. Any way or another, I'm not giving up on using Scala for this project. I will see where this leads me. And you can probably expect a series of these first impression posts from me about Scala, since I have a tendency to rant or rave about my experiences.

Happy New Years, people!