Is Scala Too Complex?

Is Scala too complex to become the 'New Java'?  This question has been debated before, and there are some good existing posts including:
I tend to mostly agree with their arguments that Scala is not inherently more complex, but instead is different and has complexity in different ways.  The syntax is different, and you need to learn a few different rules, and some features like Traits (multi-inheritance) and implicit conversions can make tracking down how things work a bit more difficult.  In the end though, I don't believe Scala is itself more complex than Java.

But I want to take a different angle on this question.  I believe that the use of Scala to build Domain Specific Languages illustrates how some of Scala's features can be used to create code that can be complex to understand, but that may be OK.

Domain Specific Languages
Domain Specific Languages, or DSLs, provide a syntax suited to a specific problem domain.  Scala provides a set of features that enable the development of Scala libraries that enable a DSL or pseudo-DSL.

Lift is a web framework written in Scala.  It provides what I consider a Scala based DSL for creating web applications.  Let's examine some sample code from a simple Lift Snippet.  (A Lift Snippet is somewhat related to a Servlet):
class A {
  def snippet (xhtml : NodeSeq) : NodeSeq = bind("A", xhtml, "name" -> Text("The A snippet"))
}
This example (from Exploring Lift site - pdf) provides a trivial snippet implementation.  It simply replaces the <A:name /> XML tag with "The A snippet" text and would be called by the following XHTML:
<lift:A.snippet>
  <p>Hello, <A:name />!</p>
</lift:A.snippet> 
There are a few things going on here to make this work. First, the bind method is statically imported. The import looks like:
import net.liftweb.util.Helpers._
This imports all the methods defined in the Helpers class into your class, enabling bind to be called without prefix. The Helpers class itself is just a roll-up of 9 other Helper class, so you could instead:
import net.liftweb.util.BindHelpers._
or even:
import net.liftweb.util.BindHelpers.bind
if you want strict control over what you are including.  However, the point here is not that you can do static imports, which are possible in Java as well, but that the Lift framework makes heavy use of static imports to help create its DSL.

The second part requires two important concepts, implicit functions and operator overloading.  The bind method has a few overloaded versions, but a typical case uses the following signature:
def bind(namespace : String, xml : NodeSeq, params : BindParam*)
In our simple example, the first two parameters are pretty straight forward, but how does:
"name" -> Text("The A snippet")
Get converted to a BuildParam?  The answer is implicit functions and operator overloading.  The BindHelpers Object (statically imported with Helpers._), contains an implicit method with the following signature:
implicit def strToSuperArrowAssoc(in : String) : SuperArrowAssoc
The SuperArrowAssoc object defines a method (operator overload) ->.  This operator/method is overloaded to take different parameters, including scala.xml.Text. So in our simple example:
"name" -> Text("The A snippet")
The "name" string is implicitly converted to a SuperArrowAssoc, and the -> method is executed with a scala.xml.Text instance as the parameter, and returns a BindParam, which is then passed to the statically imported bind method.

Got all that?

It is a bit complicated, and it took me a little while to track down how it all worked.  However...  There are two important points to make before concluding that this is 'bad'.

First, the disparity between IDE functionality for Scala and Java is enormous.  It is reasonably easy to understand any piece of code quickly in Java, in large part because of the IDE.  It is trivial to understand inheritance trees, all references to a method/variable,  all subclasses, etc.  What I would like to see in Scala is easy decoding/indication of implicit methods and static imports.  The Scala 2.8 plugin for Eclipse still fails at handling basic Class and import statement auto-complete most of the time, let alone handling more complex values(1).

Second, and I think more interestingly, I'm not sure it matters if you 'understand' the code.  The point of a DSL is to make it easy to build something in a specific domain.  Yes, you are coding Scala, but do you really need to know where/how everything is defined/wired?  If I'm building a web site, do I care how "x" -> "y" gets converted and handled by Lift?  What you have is a case where you end up learning the DSL syntax.  I think this is different than learning a library API in Java.  A library API uses traditional Java syntax, so you are just learning what methods exit.  In Lift, you have to learn syntax itself, because there is no obvious way to 'discover' what is possible with the Lift library.  An example:

Using Java and a traditional Library, I can use auto-complete to find the 'primary' classes, and use method auto-complete to look at the methods.  Assuming it is a reasonably designed API, I can probably figure out how to use it without much more.  Of course reference thing JavaDoc will help and is reccomended to make sure everything works as you assume, but you can get a long way on your own.

In Lift, there is no easy way to auto-discover how to interact with the framework using the DSL.  You must read documentation/samples to understand what is available, or dig through endless pages of ScalaDoc to deduce the possible options.

In the end, I don't believe Scala is inherently more complex than Java.  That said, I think it is easier to write hard to understand Scala code than it is with Java.  Writing a DSL in Scala can make for code that is nearly impossible to 'infer proper usage' from the traditional means, but that isn't surprising.  It is a DOMAIN SPECIFIC LANGUAGE, in essence, you created a new syntax, and you have to acknowledge that that learning that is different than learning the language it is written in.

I remain a fan of Lift and Scala.  Even with these issues, I feel that the value they deliver is worth the learning curve necessary to become proficient. 

(1) - But I am still very appreciative that the plugin exists at all, and hope they keep up the hard work!

Recursive Snippet Processing with Lift

I really like how Lift's 'template' engine works.  In short, you define XML tags that map to a Class and Method for execution.  For instance, a basic HTML template looks like:
<lift:MyClass.myMethod>
  <div>Hello, <my:name/>.  Welcome to my sample web app</div>
</lift:MyClass.myMethod>
This will result in the myMethod function on MyClass being called, which can then easily replace <my:name/> with a dynamic value.

The real power comes from the fact that the Lift framework will continue to (re)process the XML until all Lift tags have been resolved.  This means that a call to one snippet can produce a call to one or more snippets.

I came across an example of this on a recent project.  I wanted to produce the same HTML block for multiple snippets.  My first effort at refactoring produced something similar to this:
<lift:MyClass:showAttr1 eager_eval="true" name="Attribute 1">
  <lift:embed what="attribute" />
</lift:MyClass>
My MyClass looked like:
class MyClass extends AttributeHelper {

  private val attributeDefinitnion = ...
  val name = S.attr("name").openOr("Unnamed Attribute")

  def showattr1(xml:NodeSeq) : NodeSeq = {
    attrHelperBind(attributeDefinition, name, xml)
  } 
}
The AttributeHelper trait defined the attrHelperBind method which took the attributeDefinition and used the bind method to replace the XML tags defined in the attribute template that was embedded in the body.  Note, I needed the eager_eval="true" attribute so that the embed tag would be executed before the showAttr1 tag.

This worked well and greatly reduced the amount of boiler plate code needed for each attribute.  However, since Lift will continue to evaluate the XML until all the tags are processed, I realized I could further improve it.  I created a generic snippet that simply returned the following block:
<lift:MyClass.myMethod eager_eval="true">
   <lift:embed what="attribute" />
</lift:MyClass.myMethod>
This allowed me to have a very generic entry in my HTML:
<lift:Myhelper.helper snippet="MyClass.myMethod"/>
The implementation of this Snippet is:
def helper(xml:NodeSeq) : NodeSeq = {
  val snippet = S.attr("snippet").openOr("Helper.default")
  new Elem("lift", snippet, Attribute("eager_eval", Text("true"), Null), TopScope,
    <lift:embed what="attribute" />)
}
This simply produces the original XML block, which will then be processed normally. The Elem call produced a element named <lift:{snippet}> with the body
<lift:embed what="attribute" />. You must use the Elem object to create the XML because you cannot have dynamic tag names in XML literals.  IE:
def myXml(name:String) = {
  <lift:{name}>Body</lift:{name}>
}
is not legal as the XML literal will not be parsed correctly.

This ability to 'recursively' process the Lift XML tags enables the development of easy helper methods to allow the final XHTML templates to be very concise and readable.

Using Comet with Lift to Dynamically Update Pages

Lift, a web framework written in Scala, provides easy integration with Comet, a server side HTML push model.

The Lift Demo site provides a good overview of the basic Comet usage.  A CometActor has two main parts.  The render method, which is executed when the page is first requested, generates any initial content required for the page.  The actor message method, called when the CometActor receives a message, is responsible for triggering an update on the page.

The render method is similar to the render method on any snippet.  It usually has a bind method that replaces XML tags in the template with dynamic content.  This can be visible HTML, JavaScript methods that will be used during updates, or a combination of the two.

The Comet Chat example has the following render method:

// display a line
private def line(c: ChatLine) = bind("list", singleLine,
                "when" -> hourFormat(c.when),
                "who" -> c.user,
                "msg" -> c.msg)

// display a list of chats
private def displayList(in: NodeSeq): NodeSeq = 
chats.reverse.flatMap(line)

override def render = 
  bind("chat", bodyArea,
       "name" -> userName,
       AttrBindParam("id", Text(infoId), "id"),
       "list" -> displayList _)
Here, the render method sets the chat:name tag to userName, and the chat:list tag is processed by the line method (via displayList).

The actor methods are where it gets interesting.  The actual method can be one of low(med/high)Priority or messageHandler.  These methods define PartialFunctions that can be used to handle incoming messages.  There are two basic approaches that can be used from here.  The first approach is to update some internal state and force the render method to replace the existing content.  This would look something like:

var messageText : List[String] = Nil 

override def messageHandler = {
    case message : String => messageText ::= message; reRender(false)
}

Assuming you have a render method that takes the messageText and renders it out as HTML, the page will reflect the contents of messageText, and will be updated each time a message is received.  However, if the amount of changes are small relative to the rendered block, or you are creating an ever-increasing list of content, this can be inefficient.

The second approach for the actor message methods is to call a JavaScript method on the client with the updated information.  This would look something like:

override def messageHandler = {
  case message : String => partialUpdate(OnLoad(AppendHtml("logContents", <div>{message}</div>) & JsRaw("autoScroll();")))
}

In this example, the partialUpdate method is used to push a string of JavaScript Commands to the web page.  AppendHtml is a shortcut to a jQuery method to add HTML to a specific element. This is combined with a call to the autoScroll method which we assume is already defined on the page and acts on the newly updated information in some way (i.e. automatically scrolling the div to the bottom so the updated HTML is visible).  This does not trigger a new call to the render method.

Note that the appendHtml method is defined net.liftweb.http.js.jquery.JqJsCmds instead of net.liftweb.http.js.JsCmds as it is a bridge to jQuery (hence the Jq prefix). 

The combined power of server side pushes, Actors, and convenient helper methods makes it very easy to build a page that is updated in real time.

Deploying a Scala Lift Application in an OSGi Container

My current project involves building a Lift web application and deploying it in our OSGi application container.  I've been working with Scala on and off for a while, and I've always been interested in Lift.  With the release of Scala 2.8 and Lift 2.0, I decided it was time to give Lift a real try on my current project. 

The easiest way to deploy a WAR file is using Pax Web's War Extender. This allows you to simply deploy a WAR file with an updated MANFIEST.MF file (making it an OSGi Bundle) in the same container as Pax Web. In my example I will build a WAR file as a standard OSGi Plugin and build it using Eclipse, but you could also build a normal WAR file using Maven or SBT and add the OSGi attributes to the MANIFEST.MF file and deploy it with Pax Web.

The following steps assume:
  • Eclipse is the IDE
  • Scala-IDE (Scala Plugin) is installed.
  • Scala 2.8
  • Lift 2.0 (Need the Scala 2.8 Snapshot from Scala-Tools)
The first step was to create a standard OSGi Plug-In Project.  Then edit the project file to add the Scala Build Command
<buildcommand> 
  <name>ch.epfl.lamp.sdt.core.scalabuilder</name>
</buildcommand>
to buildSpec, and remove the Java Builder

Then add the Scala Nature
<nature>ch.epfl.lamp.sdt.core.scalanature</nature>
to natures. I added both to the top of the respective sections.

You will then need to reload the Project (Close/Reopen) and it should be Scala-enabled.

I then merged the Lift Template project into my Eclipse Project.  I copied the src/main/scala directory into my src directory, and src/main/webapp into the project root.

At this point Eclipse should see the Scala source files, but they will not compile as the Lift libraries are not yet in the classpath.

I downloaded the required dependencies from the various Maven repositories and added them to the WEB-INF/lib directory.  For my initial project I needed:
  • joda-time
  • lift-actor
  • lift-common
  • lift-json
  • lift-util
  • lift-webkit
  • paranamer-generator
  • slf4j
  • org.apache.commons.fileupload
Remember, if you are using Scala 2.8 to download the 2.8.0 version of the lift libraries (which are currently under snapshots instead of releases).  Once you have all the dependencies, the Lift sample code should compile.

You can now work on deploying the bundle.

You will need the Pax Web WAR Extender bundle and all its dependencies.  They can be found in this Maven Repo, and are outlined here for version 0.7.1.

Once the Pax Web bundles are deployed, you should be able to deploy your bundle.  The Pax Web WAR Extender will scan all bundles for a web.xml file and attempt to deploy them if it finds one.

By default it uses the Bundle's Symbolic Name (Bundle-SymbolicName: xxx) as the context root, but you can specify your own by adding the following line to your MANIFEST.MF:
Webapp-Context: /
to deploy as the root context or
Webapp-Context: mycontext
to deploy as /mycontext

That was all it took to get a sample Lift application up and running. I can now use the OSGi container to reference other dependencies and continue to build the application in Eclipse.

A couple of notes:
You can deploy the Lift libraries and dependencies as OSGi bundles instead of in the WEB-INF/lib directory of the bundle. At the time of this writing, the Lift OSGi Bundles were not available for 2.8 yet, but all they need are an updated MANIFEST.MF file.

As I noted at the beginning, you can simply edit the MANIFEST.MF file of any WAR and deploy it this way. If your Lift app does not depend on other OSGi bundles this may be the easiest approach.

Hiking with a DSLR Camera

Update: I've posted an updated version of this here: Hiking with a DSLR Part 2.

I just spent a week in Rocky Mountain National Park hiking with my family. It was a great week, and it allowed me to give my new hiking setup a full workout.

My camera is a Nikon D300, and my primary hiking lens is the Nikkor 17-55 2.8.  Since I have small kids, I carry a child carrier, either a Kelty FC3 or FC1.  None of my normal carrying solutions (outlined below) worked for hiking, so I needed something different.

I came across the Cotton Carrier, which is a chest harness for the camera.  You screw a small round attachment into the bottom of your camera, and it slides into the chest harness.  It also provides a Velcro strap to immobilize the camera further, although I only use this if I need to scramble up rocks or jog.  It also provides a safety strap that I attach to where a normal camera strap would attach.

In this setup, I don't use a camera strap at all.  Between the harness and the safety strap, the camera isn't going anywhere, and it provides for clean access to the camera.

When I first got my carrier, it didn't come with the safety strap, and I kept a very short camera strap on the camera to provide for a little extra safety when I had it out of the carrier.  When they added this feature they offered it for free (plus postage) which was a nice touch.

This setup worked pretty well, but it was frustrating to use the Tripod which I sometimes carry with me on hikes.  To address this, I moved to a solution where I 'always' used a quick-release solution.

I use Arca Swiss compatible plates and clamps.  Initially, I had a generic plate for my D300 and a Kirk QRC-2 clamp on my tripod.  This worked alright, but the plate was bulky so I always took it off the camera when I wasn't using it, which was time consuming.

I looked into using quick release plates with my Cotton Carrier and realized all I needed was an additional clamp.  You simply screw the Cotton Carrier round attachment to the Clamp, and then you simply clamp the clamp to the camera.  This approach increases the distance from the camera to the carrier a little, but I didn't find it to be a problem.

I also upgraded my generic plate to a Kirk plate made specifically for the D300, the Kirk PZ-122.  This plate was MUCH better than the generic plate I used before.  It is much more low profile, and has a 'stopper' you can put on one side to the plates can only go on and off on one side.  Additionally, it has a second screw hole, so you can attach the camera to other things that use this attachment without removing the plate.  Since I regularly use a Rapid Strap, that screws into that hole, it turned out to be a great feature.

All in all, I'm very happy with the Cotton Carrier + Kirk Plates/Clamp system in general, and for Hiking in particular.

The harness looks like it covers your entire chest, but only the bottom band (about 1 1/2 inches) is really snug against your chest.  It does get a little sweaty, but not bad considering, and nothing compared to my hiking backpack.

When I'm not hiking, I use one of the following (all of which I like for specific uses):
The SlingShot is a great 'all day' bag for most outings.  The 200 can carry my D300 with the 17-55 lens attached in the main hold, with two other (modestly sized) lenses.  I also carry an extra battery, data card, the R Strap, Filters, and other miscellaneous gear I may need throughout the day.  It provides easy access to the camera without taking the pack off, and is reasonably light.

The R Strap works great when you are shooting a lot with the same lens and a 'casual' level of movement.  It is not good for walking on non-level ground, walking long distances, or doing a lot of squatting.

I rarely use the camera with a normal camera strap, but it is the traditional approach.