Is Scala Too Complex?

Is Scala too complex to become the 'New Java'?  This question has been debated before, and there are some good existing posts including:
I tend to mostly agree with their arguments that Scala is not inherently more complex, but instead is different and has complexity in different ways.  The syntax is different, and you need to learn a few different rules, and some features like Traits (multi-inheritance) and implicit conversions can make tracking down how things work a bit more difficult.  In the end though, I don't believe Scala is itself more complex than Java.

But I want to take a different angle on this question.  I believe that the use of Scala to build Domain Specific Languages illustrates how some of Scala's features can be used to create code that can be complex to understand, but that may be OK.

Domain Specific Languages
Domain Specific Languages, or DSLs, provide a syntax suited to a specific problem domain.  Scala provides a set of features that enable the development of Scala libraries that enable a DSL or pseudo-DSL.

Lift is a web framework written in Scala.  It provides what I consider a Scala based DSL for creating web applications.  Let's examine some sample code from a simple Lift Snippet.  (A Lift Snippet is somewhat related to a Servlet):
class A {
  def snippet (xhtml : NodeSeq) : NodeSeq = bind("A", xhtml, "name" -> Text("The A snippet"))
}
This example (from Exploring Lift site - pdf) provides a trivial snippet implementation.  It simply replaces the <A:name /> XML tag with "The A snippet" text and would be called by the following XHTML:
<lift:A.snippet>
  <p>Hello, <A:name />!</p>
</lift:A.snippet> 
There are a few things going on here to make this work. First, the bind method is statically imported. The import looks like:
import net.liftweb.util.Helpers._
This imports all the methods defined in the Helpers class into your class, enabling bind to be called without prefix. The Helpers class itself is just a roll-up of 9 other Helper class, so you could instead:
import net.liftweb.util.BindHelpers._
or even:
import net.liftweb.util.BindHelpers.bind
if you want strict control over what you are including.  However, the point here is not that you can do static imports, which are possible in Java as well, but that the Lift framework makes heavy use of static imports to help create its DSL.

The second part requires two important concepts, implicit functions and operator overloading.  The bind method has a few overloaded versions, but a typical case uses the following signature:
def bind(namespace : String, xml : NodeSeq, params : BindParam*)
In our simple example, the first two parameters are pretty straight forward, but how does:
"name" -> Text("The A snippet")
Get converted to a BuildParam?  The answer is implicit functions and operator overloading.  The BindHelpers Object (statically imported with Helpers._), contains an implicit method with the following signature:
implicit def strToSuperArrowAssoc(in : String) : SuperArrowAssoc
The SuperArrowAssoc object defines a method (operator overload) ->.  This operator/method is overloaded to take different parameters, including scala.xml.Text. So in our simple example:
"name" -> Text("The A snippet")
The "name" string is implicitly converted to a SuperArrowAssoc, and the -> method is executed with a scala.xml.Text instance as the parameter, and returns a BindParam, which is then passed to the statically imported bind method.

Got all that?

It is a bit complicated, and it took me a little while to track down how it all worked.  However...  There are two important points to make before concluding that this is 'bad'.

First, the disparity between IDE functionality for Scala and Java is enormous.  It is reasonably easy to understand any piece of code quickly in Java, in large part because of the IDE.  It is trivial to understand inheritance trees, all references to a method/variable,  all subclasses, etc.  What I would like to see in Scala is easy decoding/indication of implicit methods and static imports.  The Scala 2.8 plugin for Eclipse still fails at handling basic Class and import statement auto-complete most of the time, let alone handling more complex values(1).

Second, and I think more interestingly, I'm not sure it matters if you 'understand' the code.  The point of a DSL is to make it easy to build something in a specific domain.  Yes, you are coding Scala, but do you really need to know where/how everything is defined/wired?  If I'm building a web site, do I care how "x" -> "y" gets converted and handled by Lift?  What you have is a case where you end up learning the DSL syntax.  I think this is different than learning a library API in Java.  A library API uses traditional Java syntax, so you are just learning what methods exit.  In Lift, you have to learn syntax itself, because there is no obvious way to 'discover' what is possible with the Lift library.  An example:

Using Java and a traditional Library, I can use auto-complete to find the 'primary' classes, and use method auto-complete to look at the methods.  Assuming it is a reasonably designed API, I can probably figure out how to use it without much more.  Of course reference thing JavaDoc will help and is reccomended to make sure everything works as you assume, but you can get a long way on your own.

In Lift, there is no easy way to auto-discover how to interact with the framework using the DSL.  You must read documentation/samples to understand what is available, or dig through endless pages of ScalaDoc to deduce the possible options.

In the end, I don't believe Scala is inherently more complex than Java.  That said, I think it is easier to write hard to understand Scala code than it is with Java.  Writing a DSL in Scala can make for code that is nearly impossible to 'infer proper usage' from the traditional means, but that isn't surprising.  It is a DOMAIN SPECIFIC LANGUAGE, in essence, you created a new syntax, and you have to acknowledge that that learning that is different than learning the language it is written in.

I remain a fan of Lift and Scala.  Even with these issues, I feel that the value they deliver is worth the learning curve necessary to become proficient. 

(1) - But I am still very appreciative that the plugin exists at all, and hope they keep up the hard work!

11 comments:

  1. You raise a good point about Scala DSLs. Because they are often implemented using implicits and other "advanced" features, their ScalaDocs are often useless. In fact, I find ScalaDoc to provide very poor documentation of APIs, because it cannot show you how the API/DSL should be used.

    ReplyDelete
  2. Well in a way Java suffers from the same issue only worse: there people often choose to implement (external) DSLs in some XML variant (Maven, Spring, Ant to name a few). This may result in unreadable syntax which is overly verbose, vague and poorly documented semantics, partial loss of IDE support, poor performance due to the interpreted nature of the DSL...

    ReplyDelete
  3. The market has decided - Scala is not the successor to Java. A successor has yet to be built. And I suspect Java 7 will bury such talk for years to come.

    I dedicated a few weeks to the study of Scala. I found it interesting. But until I can make a living using it, Scala is of little value to me.

    ReplyDelete
  4. That you are even raising this issue, suggests Scala is too complex - too much. It will remain an elitist's language, but never become mainstream.

    ReplyDelete
  5. @Mike Funk
    I think you may hope too much from Java 7.

    This version will bring highly desirable closures, if it get past the process. But with this addition, Java will come very close to collapsing under its own complexity. Maybe, it will even get past this point. Combined with generics, prepare yourself to a massive wave of rants.

    With generics, it seemed more and more obvious that Java needed type inference. With closures, this lack will feel even more painful.

    But let's face it. It is now impossible to add type inference to Java without breaking backward compatibility, and it isn't even an option.

    Simply using type inference in Scala, even without any other advanced language constructions, makes coding a breathe! Everything becomes shorter and much more readable. Everything flows, intent doesn't drown itself in an ocean of type declaration madness. And if you need to know the type, you just hover with the mouse in Eclipse.

    Don't forget that. This is probably the last time the Java language will be able to evolve. Next time this will be a dinosaur.

    ReplyDelete
  6. @Nicolas
    I agree with you that Java 7 is not 'the savior'. Each new 'bolt on' is showing its age. And with closures, it isn't so much the language's support as the support of the libraries, and I don't expect them to all quickly adopt closures.

    One of the points I tried to make with the article is that with Scala, you don't have to understand how everything works to build applications. If you look at Lift as a DSL, the fact that it uses 'complex' aspects of Scala is irrelevant to the everyday developer building a web application. They need to learn the Lift syntax, not necessarily the Scala that makes it work.

    I think in Scala you will have a bigger disparity between people's level of understanding, but I'm not convinced that is 'bad'.

    ReplyDelete
  7. @Mike Funk
    if java 7 provides all the good stuff that scala has to offer at the moment, i'd be really surprised. the scala engineers seem to be very good at pinpointing what's 'wrong' with Java and they provide an elegant solution for many of these issues. for years now. concepts of functional programming, options, comprehensions, mixins, ... and i almost forgot, a concise and elegant syntax.

    the fact that scala is interesting is of high value to me and makes me want to learn more about it although i'm not using it to make a living. this is an interesting post and defintely worth the effort for me as a reader and i guess also for the writer, without making money.

    i think 'java-ish' scala is not hard or complicated. it's the extra features that make it look like it's complicated (at first sight). in this regard i have to agree with one of my colleagues who thinks that the full power of scala is currently too hard for most of the java developers. ask a java developer about currying, yielding, folding, list comprehensions, pattern matching, ... it takes time to learn new concepts and take benefit from them. i'm still learning new stuff everyday playing with scala.

    ReplyDelete
  8. The average developer doesn't have to understand how a framework works in the background, as others already said. But what is, if a developers needs to extend the framework? It would be really hard then to understand how the API works internally!

    But what I fear most, is that mediocre developers misuse powerful concepts to write horrible code. My experience says: They will do so!

    ReplyDelete
  9. Having just gotten through learning enough Scala and Lift to do a very simple web app; here are my thoughts.

    First, with a DSL approach the issue becomes one of documentation. Because the APIs are difficult to track down the onus becomes on having excellent documentation. The Lift wiki is good, but not excellent, and that causes a lot of confusion. As the author notes this can possibly be addressed both by improving the IDE integration and by improvements to the Scala docs mechanisms - but only to a degree.

    Second, Lift doesn't provide a "pure" DSL, and I'm not sure you'd want one as you need to tie into backend logic of some sort in most cases. The methods / syntax on the web side are pretty cool, but you still need to know some amount of Scala to get it to work (by "pure DSL" I'm thinking of something that would essentially hide completely the underlying implementation). So to do anything "real" you do need to know some amount of Scala, again I don't think this is bad its just the way things are.

    Finally, in my experience as soon as you do start needing to do something "real" in any framework then you run up against limitations in the framework. In some cases these can't be overcome using the framework you're working with; in others it is simply a matter of understanding the framework well enough to write the custom code you need for your project. This is one reason I don't think a totally pure DSL is a good idea. So, one of the first things I look for when looking at a framework is to say to myself "where are the edges" and "can I add code to move past them if needed". In the case of Lift I think it is possible, but with any framework the author's simply can't build in direct support for every possibly need, at the end of the day you (or someone) almost always still has to write some amount of custom code - its just a question of how hard the framework makes it. At this point I am not yet familiar enough with Lift to know the answer to this question, but from what I have seen so far it seems to be well designed system, so my guess is that it shouldn't be too hard for most things. Again, not a knock on Lift - I can already see it is a heck of a lot easier to add custom logic than with something like say JSF 1.

    ReplyDelete
  10. Maybe I'm the only one who thinks Scala is less complex than Java (or even that mixture of Java, XML config, expression language, annotations, dependency injection which is "standard" in the Java world)?

    Arguing that Scala is complex when the critized feature doesn't even exist in Java is just cheap.
    Of course an existing features is more complex than a non-existing feature.

    But overall, Scala is much simpler.

    - Better object-orientation
    - sane syntax for arrays
    - no things like operators (only methods)
    - no special exceptions for primitives
    - no statics
    - no seperation between float-Float/int-Integer
    - saner generics
    - better working inheritance
    - possibility to "put objects together" with traits instead of repeating your code implementing interfaces
    - dead-easy handling of XML with XML literals plus an XPath-like API
    - a superior collection API which just works
    - case classes, named and default arguments which reduce the boilerplate by a factor of 10

    Scala is a simpler Java. If it is considered complex because it makes additional things easy which are not even possible in Java then I can live with that.

    ReplyDelete
  11. Yes. Scala is too complex. Learn Clojure instead.

    ReplyDelete