Skip to content

Lessons Learnt Fommil

Jump to bottom Edit New page
janm399 edited this page Apr 13, 2013 · 5 revisions

The akka-patterns project is a dumping ground for lessons learnt on a variety of Scala / Akka / Spray topics.

At the end of 5 months working on real world (commercial) projects, that were originally based on the akka-patterns architecture, Sam Halliday (@fommil) was asked to document the lessons learnt:

This short document is a summary of the highlights from the pull request.

Not everything made the cut, so super-keen readers are advised to read the commits or get in touch.

Akka

We initially went a little overboard with Akka, and it really bit us in the backside.

A common pattern in our Spray endpoints went like this:

    // HTTP POST a JSON User object to /user/register
    path("user" / "register") {
      post {
          handleWith {user: User =>
            (userActor ? RegisteredUser(user)).mapTo[Either[NotRegisteredUser, RegisteredUser]]
          }
      }
    }

What we're doing here is forwarding the endpoint input as a message to an actor, which will hopefully respond with Either a typesafe endpoint response or a domain specific error message (which could inherit from Exception).

It is necessary to use the Either, because it is not possible to propagate exceptions from Actors. And of course, the logs for such exceptions are very hard to link with specific endpoint requests... which is why we spent so much effort getting the logging to produce good - traceable - output (see below).

However, there are a couple of real PITAs here:

  • we have lost type safety and have to manually code our mapTos to agree with actor messages. The only way to be sure we catch everything is with 100% code coverage. Messy. Not to mention this little gotcha https://groups.google.com/d/msg/akka-user/P5NadnLjrh4/frPlweBGdi0J
  • NotRegisteredUser is now semantically just like a checked exception... which we all agree were a failed experiment in Java. We've brought them back, but we've done much much worse: all other exceptions are only detected when this ask times out and that can be confused with legitimate timeouts.

Furthermore, we've now funnelled all our processing into Actors, which are by default single threaded. Now imagine we have a network of Actors in the background and there are a few in there which become bottlenecks across endpoints (e.g. a database actor).

We had a workaround to the first point, discussed on akka-user but the general consensus was that Actors aren't designed to be asked too many questions.

We opened up our Actors to parallel processing of requests by allowing Akka to decide how many threads to associate to each Actor, as demonstrated here (this is called "routing" in Akka speak):

The general feeling in the team was that traditional services / controllers are more appropriate. Here the pattern looks like

    path("customers" / JavaUUID) { id =>
      get {
        complete {
          // when using classes instead of actors, we have to explicitly create the Future here
          // it is not necessary to add the T information, but it helps with API documentation.
          Future[Customer] {
            customerController.get(id)
        }
      }
    }

The "customerController" is very terse, whereas a "customerActor" is just riddled with boilerplate to define the input and output messages.

Traditional controllers / services then bring back all the traditional problems of dependency graphs and so on, which means there is still room for Spring-style dependency injection in Scala (and, unfortunately, mutable references).

Incidentally, Akka Actors aren't much better at resolving the dependency problems either. Although it is possible to depend on other Actors as either peers or children, many Akka experts claim that naming dependencies by their String path is bad practice... the only other alternative is dependency injection. For me, this is one of the biggest pains when dealing with Akka... I'm never quite sure how to depend on other Actors. Perhaps there is scope for a typesafe Akka Injection library to do for Akka what Spring does for Java.

All in all, we thought that the middle tier was best implemented by solid classes... perhaps moving to an Actor ask pattern if the middle tier needs to be distributed (however, there are other alternatives available for scaling before having to separate front and middle tiers).

Not shown in akka-patterns is how we found a home for Akka Actors in the Akka/Spray stack. This fell into two categories:

  • scheduled tasks, performing a variety of database cleanup and periodic calculations
  • fire-and-forget opportunistic calculations. There were a few scenarios when it was beneficial to perform a calculation, but it was not necessary to wait for the result before returning the endpoint.

Arguably, the latter could be handled equally well within a Future (which has logging in the case of failure!), but when we use Actors we can throttle how much of the CPU is spent in such tasks (using low priority dispatchers, see application.conf) AND we have an easy route to distribution.

Logging in Scala

Logging (i.e. developer logging, not service logging) is a mess in Scala: there is no standard API.

In Java, many logging APIs and backends exist. SLF4J is used by sensible people to route the various interfaces into their preferred backend at runtime. Nevertheless, some NIH-prone teams still insist on rolling their own logging API: I'm looking at you, JBoss.

The main reasons why folk don't simply use JUL (standard J2SE Logging) are:

  1. the API is ugly (in the opinion of some)
  2. the output is ugly
  3. it is inefficient

I can live with 1 (see @Log) and I've already written an OSS project (java-logging) to deal with point 2. I have never seen any performance tests for point 3, so I'm inclined to ignore the FUD. I always use JUL because I see no good reason to reach for a third party library.

Scala comes with JUL, but JUL is a Java API and risks making Scala code look unsightly. Avoiding direct use is understandable.

The closest thing to a Scala Logging API is LoggingAdapter, brought into scope with a simple with ActorLogging when using Akka. Unfortunately, this doesn't apply to non-actor code.

Furthermore, the logging requirements in Scala can be much more demanding than in Java. Functional programming tends to result in very complex stack traces. It is not uncommon to see a stacktrace of over 100 lines when something goes wrong in the Akka / Spray stack. And there are usually multiple exceptions when this happened in an Actor responding to an ask. This absolutely must be cleaned up because it obfuscates errors.

Furthermore, in testing, in would appear that Specs2 (an otherwise pure library) has decided that exception reporting is worth tinkering with the logging chain, so it needs a little TLC of its own.

We addressed these problems by writing a JUL backend for Akka's logging infrastructure (contributed to Akka) with a corresponding "not in an Actor" logging trait JavaLogging. Enabling this "Scala Logging API" requires several fiddly things:

but it now means that there is a field named log available with the same API in any Scala class.

In our test support module, we created NoActorSpecs (and ActorSpecs) as an alternative to plain-old Specification, setting up the correct logging.

This logging lesson will be a lot easier to implement in future projects when I get around to updating and releasing scala-java-logging, containing all the relevant parts from akka-patterns bundled into a standalone (and tiny) dependency.

Other things that would help would be if the following external tickets were given more attention / votes:

and if Akka used the JUL instead of System.out by default :-D

MongoDB with ScalaD

When we started our projects, the officially supported driver for MongoDB (a very fast NoSQL database), Casbah, didn't support Scala 2.10. So we were forced to write our own, and released it OSS: ScalaD.

Much of what we accomplished surpasses Casbah, so we've continued to use it.

All our projects involved marshalling of case classes into JSON. For this, we use Spray JSON, because at the time it was the only JSON marshaller supported by Spray. Since MongoDB uses the vaguely similar BSON, we thought it best to re-use the marshallers at endpoints and the database. Not all data stores have this advantage in their favour.

The ScalaD API was originally designed to appeal to users of JPA, but with a strongly implicit direction. Nearing the end of the projects, we learnt that exposing the Mongo query language in a JSON DSL is much more powerful. Type safety is provided as well as the developer's database queries are correct - there should be no need to ever call the dreaded .asInstanceOf[T]. In particular, ScalaD makes it really easy to do incremental updates and to issue Aggregation Framework queries.

BTW, don't ever use MapReduce in MongoDB: it acquires a global lock on the database and is therefore infeasible to use in production environments.

The best place to learn more is the ScalaD project page.

For those who want a really nice way to set up MongoDB connections from Scala, have a look at our setup pattern across these files:

This pattern might make its way into ScalaD if there are enough requests.

We'll discuss the Settings object and test support (especially fixtures) below.

Oh, and please remember that your objects are to be persisted when designing domain objects ... it is possible to handle hierarchies (e.g. sealed traits) but you'll incur a lot of pain when it comes to writing unmarshallers. Prefer flat hierarchies to avoid all the usual ORM issues ... NoSQL has no advantage over SQL here, even though it persists OneToMany nested collections without creating additional tables.

SBT for multi-module projects

Other than the cleaner configuration afforded by SBT, we found that the SBT incremental compiler radically sped up our development cycle: from 10 minutes to 1 minute in some cases.

We had a fairly epic commit log when we moved from Maven to SBT:

"Showing 12 changed files with 125 additions and 1,107 deletions."

The entire multi-module build configuration is stored in one folder: project. The canonical reference for multi-module projects is typically Akka, but perhaps this build file (being simpler) will be of use to somebody.

It took one day to convert our real projects the first time, which went down to 3 hours for the second project. I have personally spent 3 hours lost in Maven config files when something goes wrong, so converting is a no-brainer.

One thing to note, however, is that we haven't gotten publishing to Sonatype working yet (we didn't need it for commercial projects). Any help here would be greatly appreciated.

ConfigFactory and Settings

TypeSafe really nailed runtime configuration with config.

But it is a Java API and that means it can look a little ugly when used in Scala.

Our approach is to have one file where we dump all our module-specific configuration and then expose a typesafe object which is super-clean to access as a global object: Settings.

This file is pretty much the rug that we dust everything under.

It is really important to remember that this global object is storing immutable, runtime config, and should never be used as a resource store.

Maybe, one day, someone will extend HOCON to auto-generate files such as our Settings.scala.

Spray

Spray is an Akka-based framework that we found to be good for writing REST servers.

The biggest single issue we had with Spray wasn't actually in Spray itself. Basically, IDE's hate spray. It is an implicit-heavy framework, which makes for beautiful code, but IDEs don't have a clue what is going on. IntelliJ IDEA, our preferred IDE, just cried like a baby and froze for minutes at a time. We all learnt to drop to TextMate when writing Spray code.

Alexander (author of the Scala plugin) has been really fantastic at responding to performance issues in this area and is now tracking this as part of ticket #5460, but OMG this was such a frustration for us.

Mathias (author of Spray) was also a great help when it came to making API suggestions. Many of our lessons have actually been adopted as part of the next API, so there should be no need to discuss them here. For example, current releases of Spray are prone to DOS attacks, but the upcoming release will allow for caps, which presumably will allow for dynamic runtime configurations to help prevent against such attacks without restricting legitimate users.

Endpoints in Spray are very modular, the general pattern is demonstrated in:

the code is fairly self-documenting. Basically, boot.scala sets up all the endpoints and pulls them together, authentication.scala provides an authentication backend for the endpoints, services.scala and tracking.scala provide error handling (with developer logging) and service logging to MongoDB (perfect for KPI reports)... everything else is a REST endpoint and can potentially be handed to an external developer to code against as the API definition.

The failure handling afforded by services.scala allows one to code to "the happy path" and then have a series of domain exceptions which are then converted to error codes and custom user-readable messages (with more detailed server logs).

We aren't very big fans of the test API in Spray, so we wrote our own little layer that captured our usual test patterns, this is in the support.scala test file. Boilerplate killer, and worth looking at.

We also used Spray Client for accessing third party services, this is coded up in io.scala but I understand the API is going to undergo a large rewrite so (unless you need it right now), there isn't much point looking at it.

Database Fixtures in Testing

Testing applications that depend on a database can be frustrating, because as much time needs to be dedicated to "fixtures" (i.e. the definition of the world state) as to the actual test itself.

Hopefully our ScalaD pattern makes it easy for you to implement database fixtures. The specs2 pattern goes something like this (from TestDataSpecs.scala):

    "customers fixture should attach" in new Fix("customers") {
      mongo.count[Customer]() must beGreaterThan(0L)
      mongo.findOne[Customer]("id" :> TestCustomerJanId) must beLike {
        case Some(customer) if customer.firstName == "Jan" => ok
      }
      mongo.findOne[Customer]("id" :> UUID.randomUUID()) === None
    }

what happens here is that the Fix will wipe the test database and apply all the scripts contained in customers.js. It takes a varargs, so you can mix and match fixture scripts.

Unfortunately, the Fix has to be applied at the level of individual examples... it doesn't work for collections of examples.

Mixing in CleanMongo to the entire file will simply ensure that the database is cleaned up before any specs are run. That might actually be all that is needed for some types of tests.

Testing and Performance

Here are a few really good tools that might be news to some people, useful for testing and performance testing:

  • code coverage with SCCT, although it is not great with multi-module projects and seems to spit all reports out at the top level at the time of writing
  • gatling, some more instructions to set this up are in our gatling folder
  • when doing any kind of performance testing, make sure you CPU profile! Otherwise you're guessing. We discovered some real howlers by profiling. There is no excuse, reach for VisualVM... which admittedly is severely limited when profiling remotely (why they didn't create a headless version that could run against a fixed PID and save to an XML snapshot, we'll never know... but is pretty much top priority feature in my world.)
Add a custom footer
Add a custom sidebar
Clone this wiki locally