What’s the Problem?

Given a function that takes multiple arguments A,B,… that returns a Result, how can that function by applied to arguments M[A], M[B],… and get M[Result]?

where M is an Option, List, etc.

Applicative Functors

To understand the mechanics of how applicative functors solve our problem, please refer to the references below, in particular Heiko Seeberger’s Applicatives are generalized functors.

What I present below is an example of use. I hope you can take it away and benefit from it without necessarily understanding the theory. However, I thoroughly recommend further reading and study, its fun and you will gain much deeper insights.

So without further ado, here’s the code:

import scalaz._
import Scalaz._
 
object Applicatives extends App {
 
  /*
   * Lets assume some Options which must be applied to a function.
   */
  val x:Option[Int] = 2.some // scalaz enrichment for options
  val y:Option[Int] = 3.some
  val z:Option[Int] = 5.some
 
  // Lets add them without applicative functors
  val usingFor = for (theX <- x; theY <- y; theZ <- z) 
                   yield theX + theY + theZ
 
  val usingMaps = x flatMap 
                    (theX => y flatMap 
                      (theY => z map 
                        (theZ => theZ + theY + theX)))
 
  /* With scalaz we can do the following instead of for or maps
   * First we need to put the function in the right form, curried.
   * To understand why please read the references I've given below.
   */
  val addInts = ( (a:Int, b:Int, c:Int) => a + b + c ).curried
 
  // apply the function to x, y and z
  val sum = x <*> (y <*> (z map addInts)) // Some(10)
 
  // Scalaz offers an alternative syntax that is easier to use
  (x |@| y |@| z) {_ + _ + _}   // Some(10)
 
  /*
   * If one of the options is a none
   * then the result of the whole expression will be none.
   */
  (x |@| none[Int]) {_ + _} // None
 
  /*
   * The function can be any method, including 'apply'
   */
  case class Person(age: Int, height:Double, name: String)
 
  /*
   * Person.apply method is a function (Int, Double, String) => Person
   */
  (some(2) |@| some(1.1) |@| none[String]) {Person.apply _}  
  // none
 
  (some(4) |@| some(1.1) |@| "Angelica".some) {Person.apply _} 
  // Some(Person(4, 1.1, Angelica))
 
  /*
   * The beauty of this is that it works with ANY higher kind, eg. List
   * or your own types!
   */
  val l1 = 1 :: 2 :: Nil
  val l2 = 3 :: 4 :: Nil
 
  (l1 |@| l2) {_ + _} // List(4, 5, 5, 6)
  (<xml/> |@| <xml2/>){_ ++ _} // List(<xml></xml><xml2></xml2>)
}

Further Reading

• Learn You a Haskell’s Chapter on Functors, Applicative Functors and Monoids
• Heiko Seeberger’s Applicatives are generalized functors
• Scalaz Applicative Examples

What’s the Problem?

We often want to use a function that takes an instance of A and returns an instance of B in a context where we have an instance of M[A] and need an instance of M[B]. The most common example is the map method on List, Option, etc.

So the problem is:

Given a function f that takes an A and returns a B, how can we construct a new function that takes an M[A] and returns an M[B] for any M?

Functors

A Functor accepts a function, A ⇒ B, and returns a new function M[A] ⇒ M[B], where M is any kind.

So without further ado, here’s the code:

object Functorise extends App {
 
  // This is a Functor
  trait Functor[M[_]] {
 
    /* convert f into a function mapping M[A] to M[B]
     * eg. if M were List, and f was Int ⇒ String
     * fmap would yield List[Int] ⇒ List[String]
     */
    def fmap[A, B](f: A ⇒ B): M[A] ⇒ M[B]
  }
 
  /* Here are a couple of examples for Option and List Functors
   * They are implicit so they can be used below in enrichWithFunctor
   */
 
  implicit object OptionFunctor extends Functor[Option] {
    def fmap[A, B](f: A ⇒ B): Option[A] ⇒ Option[B] 
      = option ⇒ option map f
  }
 
  implicit object ListFunctor extends Functor[List] {
    def fmap[A, B](f: A ⇒ B): List[A] ⇒ List[B] 
      = list ⇒ list map f
  }
 
  /* enrichWithFunctor is an implicit to enrich any kind 
   * with an fmap method.
   * List, Option and any other Foo[X] can be enriched with the
   * new method.
   */
  implicit def enrichWithFunctor[M[_], A](m: M[A]) = new {
 
    /* fmap requires an implicit functor, whose type is M, to which it
     * delegates to do the real work
     */
    def mapWith[B](f: A ⇒ B)(implicit functor: Functor[M]): M[B] 
      = functor.fmap(f)(m)
  }
 
  // some examples
 
  println(List(1, 2) mapWith (_ + 1)) // List(2, 3)
 
  println(some(1) mapWith (_ + 1) mapWith (_ * 3)) // Some(6)
 
  println(none[Int] mapWith (_ + 1)) // None
 
  def some[A](a: A): Option[A] = Some(a)
  def none[A]: Option[A] = None
}

What is Going On?

Lets take the List(1, 2) mapWith (_ + 1) example and go through it.

There is no mapWith method on List, so an implicit method is searched for that can turn the List into something that does have a mapWith method. Thus, enrichWithFunctor is found and called with the List(1, 2) which returns an anonymous class with the mapWith method required.

By the way, to understand more about that M[_], watch Daniel Spiewak’s High Wizardry in the Land of Scala talk.

That anonymous class also has a type parameter, M, which has now been fixed to List, and so mapWith in this class looks like this:

def mapWith[B](f: A ⇒ B)(implicit functor: Functor[List]): List[B] 
  = functor.fmap(f)(m)

The mapWith method requires an implicit Functor[List] to be available, and the ListFunctor object is just that. So when functor.fmap(f) is called, it is the ListFunctor’s fmap method being called.

ListFunctor.fmap(f) takes a function, in our example its (i:Int) ⇒ i + 1, and returns a new function (list:List[Int]) ⇒ list.map(_ + 1).

Finally, m, the List(1, 2) we started with that was enriched with the mapWith method, is applied to the new function which results in a new List.

But Why?

All we’ve done here is create a complicated looking bunch of code that just ends up calling methods on Option and List that already exist, specifically the map method.

Yes, in these examples thats true.

The real idea is that we’ve created a general abstraction for all kinds, including those that don’t have a map method. All a user of this code needs to do is have an implicit Functor in scope for the kind they would like to map over.

Furthermore, this is a building block for some interesting things that come next like Applicative Functors.

Finally, you don’t need to implement this yourself, Scalaz has it already for you to exploit.

Further Reading

• Learn You a Haskell’s Chapter on Functors, Applicative Functors and Monoids
• Heiko Seeberger’s fantastic Introduction to Category theory in Scala
• Scalaz Functor Examples

The system enables HSBC to handle the processing of all cleared interest rate and credit derivatives from multiple market participants. The first release provided full connectivity to the London Clearing House, with the Chicago Mercantile Exchange and the Intercontinental Exchange following shortly.

We had a tight deadline and quite a lot of things to figure out. The project had been difficult to analyse in the past as a result of changes in legislation and many other unknowns. So how did we do it?

Talking

The first thing we always do is talk to the business and figure out what is needed. Not in abstract business terms or strategic direction, but what the actual user sitting in front of our app is actually going to see, do and expect. Whilst the general business background is useful context and necessary, it is woefully inadequate for building a system.

Having the people who are going to actually build your system talk, in person, to the people that are actually, physically, going to use that system is critical. Not managers talking to managers but the people doing the work. And we do that every day, for every feature!

Minimal Viable Product

The next critical step is establishing the absolute minimum that will enable the system to be useful, and deliver that. Nothing else! (until next week)

It is surprisingly difficult to help stakeholders become accustomed to focussing on what is needed for the next release. IT has done a really good job of telling people they need to write down all their requirements for the next year or they will not get them. We have quite a bit of work to do to undo that ridiculous notion. But when the business understand that we aren’t saying we won’t let them have a feature, just that we want them to prioritise what they want for the next release (next week), and they can change their mind whenever they like, they are very happy.

Who Else Cares?

I have talked about the business as if they are the only interested party. Not so. We talk to every stakeholder we can find: senior management, BAs, support people, release people, release boards, architects; everyone who might ever have some kind of interest in the system. They all have needs that have to be addressed and prioritised. On one project we were able to name 23 different stakeholder roles for a project, which was surprising to everyone who thought there was only one stakeholder.

Planning

Having established a backlog of stuff, we maintain it by planning weekly with all stakeholders, and marvel at how much priorities change, new work appears and old work disappears from week to week. But thats all good! Every change is welcome, and shows how badly wrong things would have gone had we tried to plan-it-up-front.

Where’s the Code?

Every day, the team has a stand up to discuss what they are going to do. Then, we implement some features.

1. pair up with another team member in the stand up
2. talk to the user and find out what they want, sketch ideas out on paper or mock them up until enough is understood
3. write a failing acceptance test with the user. We use Specs2 but you can use whatever you like so long as it produces some kind of user visible result
4. make the acceptance test pass by typing some code in your IDE, writing unit tests as necessary

However, a lot of what we did involved communication with other systems too for which we required real samples of messages since nothing had schemas etc. This was where a lot of time was spent, again in dialog with stakeholders to ensure that the messages we process and respond with were correct.

Good, all done then!

No. When we think we’ve finished a feature we demo it to the user as soon as possible – we don’t wait to the end of the iteration. They can play with it and change things which they often do when they see something working for real.  And thats all good, we want them to have something useful.

Builds?

Part of the development practice is to commit code frequently ensuring tests always pass. People are surprised by how often we mean by frequently. We mean every single time you complete a piece of work and all the tests pass locally. That happens about once an hour but can be more frequent.

We also use TeamCity to keep us sane but again there are many alternatives.

Finally, deployment. At the start of all project’s we always have a single mantra about deployment: Thou must automate the deployment. More than that, the same deployment script should work on every dev, test, uat and production environment. Sometimes, the production environment needs to be subtly different, but we try to make the other environments as close to production as possible.

Technology

We built the system in Scala with Liftweb. It was tested with Specs2 and we used sbt for builds. Scala is significantly more productive than Java, and Liftweb enabled us to build a robust, secure, dynamic web application very quickly.

Summary

Is this starting to sound familiar? Yup, its basically an agile method using practices from XP, Scrum, Kanban, anything that works for this client in this project with this team. And no, you can’t transplant it to another team! They will have to figure out what works for them, and when we move to another project, we have to figure out what will work for the new environment.

I present the example without explanation, the web does not need another set of explanations about this subject but I have found the following blogs useful, with a mix of theoretical explanation and practical examples:

• Functional IO in Scala with Scalaz
• The IO Monad for People who Simply Don’t Care
• Towards an Effect System in Scala, Part 2: IO Monad

All this example does is print out the names of files in a directory.

import scalaz._
import Scalaz._
import scalaz.effects._
import java.io._
 
object TheIO extends App {
 
  val dirInIO = io { new File(System.getProperty("user.dir")) }
 
  def filesInIO = for (dir <- dirInIO) yield dir.listFiles()
 
  def namesInIO = for (theFiles ← filesInIO) yield theFiles.map(_.getName())
 
  def printNamesInIO = for (all ← namesInIO) yield all.map(println(_))
 
  val result = printNamesInIO
 
  println("Nothings happened yet, lets go!")
  result.unsafePerformIO
}

Here’s some code which is based on an example on Scalaz’s website.

import scalaz._
import Scalaz._
 
object Applicatives extends App {
 
  /*
   * Lets pretend you have two optional values, but you want to apply some
   * function to their contents safely.
   * You also don't want to fiddle about with flatmap / map / for / yield
   * Then do this:
   */
  val x:Option[Int] = Some(2)
  val y:Option[Int] = Some(3)
 
  (x |@| y) {_+_} 	// Some(5)
 
  /*
   * The |@| method is a pimp (enrichment! sorry), on Option in this case,
   * that builds 'something' that applies the function _ + _
   */
 
  /*
   * If one of the options is a none
   * then the result of the whole expression will be none.
   * (I am using scalaz's some and none because it means less boilerplate)
   */
  (some(2) |@| none[Int]) {_+_} // None
 
  /*
   * The function can be any method, including 'apply'
   */
  case class Person(age: Int, height:Double, name: String)
 
  /*
   *  ⊛ = |@| just for fun
   */
  (some(2) ⊛ some(1.1) ⊛ none[String]) {Person.apply _}  // none because an arg was none
  (some(4) ⊛ some(1.1) ⊛ "Angelica".some) {Person.apply _} // Some( Person(4, 1.1, Angelica) )
 
  /*
   * did you catch the "Angelica".some - another bit of enriching by scalaz
   */
 
  /*
   * It works with ANY higher kind, eg. List
   */
  val l1 = 1 :: 2 :: Nil
  val l2 = 3 :: 4 :: Nil
 
  (l1 ⊛ l2) {_ + _} // List(4, 5, 5, 6)
 
}

References

1. Eric Torrebore has a very nice article explaining much of this.
2. Learn You a Haskell has a great chapter on applicative functors amongst other things.
3. Heiko Seeberger blog post

Having obtained estimates for all of the work, and with contracts duly signed, etc., the work was begun. And so, we tracked the burndown:

The chart above shows the first 43 days of burndown.

One common question that we try to answer with the burndown charts is: ‘What day do we expect the burndown to reach 0?’. That is: ‘When do we expect to finish?’ The assumption is that historical performance predicts future performance. In our case, the question is more like: ‘Is there any reason to suspect we might not finish by the contracted date?’

One simple way of answering this question is to look at ratios: we have completed (551 - 302) estimated days of work in 43 days, so we can expect to take 43 * 551 / (551 - 302) = 96 days to finish everything.

A more ‘sophisticated’ approach is to put a linear regression line through the burndown and extrapolate. This tells us that we will reach 0 remaining estimate on day 85, but it fails to give an indication of the confidence we can have about this prediction. We can use the the correlation coefficient to give some idea of this, but it is difficult to know what to do with it.

In fact, the questions above are not really the right ones. The question we should be asking is: ‘How confident can we be that the burndown will reach zero on or before a given date?’.

In order to answer this, we need a different approach. One such approach is based on the Monte Carlo Method, which in this case translates to asking: ‘Assuming the future burndown is similar to the historical data, what fraction of the possible futures result in the burndown reaching zero on or before a given date (or between two dates)?’

We can apply this method by starting with the estimate on the last burndown day for which we have an estimate (day 43), choosing a day from the history randomly, determining the change in burndown between this day and the preceding one, and then applying that change to the estimate we have for day 43. This then becomes the estimate for day 44 and we repeat the process, until the estimate reaches 0. This then represents one simulated burndown, and we repeat the whole process thousands of times to obtain a distribution of estimates for each future day.

The chart below shows the first result from applying this method:

The line extending the historical burndown is the ‘Median Line’. This is obtained by determining the estimate for each day for which half of the simulations yield a higher estimate and half yield a lower estimate. When this line intersects the Day Number axis (day 97), it tells us that half of the simulated burndowns have reached a zero estimate on or before day 97 and half have not. Based on this information (and speaking somewhat informally), we have a half chance of delivering on or before day 97 and a half chance of delivering afterwards.

Knowing this is interesting information, but it is reasonable to ask what is the spread of simulation results. After all, if all simulations terminate within a very tight spread centred around day 97, we can still reasonably accurately predict the end day. If however, the spread is very wide, some other action might be suggested.

We can obtain an idea of the spread in the same way as we obtained the median line. If, for each day, we ask ‘What value of estimates (centred around the median value) contain x% of the simulated results on each simulated day?’. This will give us ‘confidence intervals’ around the median:

The chart above shows the 50%, 90% and 99% confidence intervals. Consider the 90% confidence interval, which is enclosed by the dark green curves starting at the final known burndown day (43, 302) and terminating at (74, 0) and (124, 0). On each simulated day, the set of estimates between these two curves contain 90% of the simulated burndowns. Because we placed the intervals symmetrically around the median line, the space underneath the lower curve represents 5% of the simulated estimates (those that represent the team performing above expectations), and the space above the upper curve represents the final 5% of the simulated estimated (those that represent the team performing below expectations).

Clearly the spread is quite wide! If we wanted to give a date that represented 95% confidence, we should choose choose day 124, which is 27 days after the median line predicts, or a difference of 25%. It is also 28 days (also about 25%) after our simple ratio based estimate given at the beginning of this article and 39 days (40%) after the the linear regression estimate.

The astute reader might notice two important things:

1. The day at which the confidence interval curves reach a Remaining Estimate of 0 spreads out alarmingly as the required confidence increases. i.e. the 50%, 90% and 99% upper confidence envelopes require 10%, 28% and 45% respectively more days to reach 0 than the median line does.
2. The confidence intervals are distributed symmetrically around the median line when a vertical slice is taken through the chart. Indeed, the histograms giving rise to these confidence intervals are near-Gaussian (courtesy of the Central Limit Theorem), at least until some of the simulations are terminated when they reach 0 remaining estimate. This is not true when a horizontal slice is taken. In particular, when those curves reach the Remaining Estimate of 0, it is plain to see that, for example, the lower envelope for the 90% confidence interval is 97 - 74 = 23 days below the median, whereas the corresponding upper curve is 124 - 97 = 27 days above the median.

The first of these points suggests that, even at this advanced state of the project, requiring high confidence about the delivery date demands significant ‘contingency’, with the 99% upper confidence envelope prediction being more than a factor of 2 larger than the corresponding lower confidence envelope prediction. Similarly, the ratio between the upper 99% confidence envelope prediction and the median prediction is just under 1.5.

The second point suggests that it is easier to run late than it is to run early, which will come as no surprise to development teams and their customers everywhere. Indeed, the skew also implies that when things are running late, it is possible for even more tardiness to result. This is intuitively true since, with less work remaining to be done, it is less likely that significant estimation ‘errors’ will occur.

Of course, we should not take all of this too seriously; a great deal has been made of very little historical data. Attention to a burndown chart often causes teams to adjust scope and other factors to bring the project in on time. The date is far more likely to be missed because of some unmitigated risk, such as the absence of a team member for an extended period of time.

In a subsequent article, once the delivery is complete, I will return to this topic with the remainder of the burndown data and see how well the confidence intervals modelled the future.

Update: About a week after I wrote this article, a series of events occurred that made it impossible to complete the graph in the way that I’d hoped.

First, one of the ‘key’ developers left the team. The notion of ‘key’ developer (due to functional silos) is something that I’d been trying to eliminate since starting with the organisation, but the team had not had sufficient time to get to grips with each other’s work.

Since the team was not large, the developer leaving implied that timescales would change. This caused the client to panic somewhat, leading to a review of the planning. During this review, it was discovered that the original requirements were hopelessly inadequate in term of both ‘completeness’ and ‘correctness’. After more analysis, the total scope required was estimated at six times (!) the original estimate and much of the remaining original work envisaged was de-prioritised or deemed to be no longer required.

Given the significantly extended timescales, a couple of the developers who had been intending to hang on until the delivery decided to resign, since the delivery was now clearly much more remote.

Shortly after this, a member of the executive team who I had a good relationship with was replaced by someone else (not because of this project). It very quickly became obvious that this new executive had a strong belief in process over people, big requirements up front and all manner of things I have railed against for years. This clearly spelled the end of my time with this client.

Although I had warned of much of this risk early in the project, I take no pleasure in its materialisation. It could have all been completely different.

As I said in the first paragraph above: This is not the mode in which we usually like to work.

This is the question I frequently ask myself when searching for the names of domain concepts.

The typical architectures and layering of code in ‘the enterprise’ frequently exhibit poor systems of naming which causes developers to jump through hoops to try to maintain the layers because of lack of abstraction. This in turn results in an unwieldy codebase that is more concerned with accidental implementation complexity than codifying domain knowledge.

As an example consider the ‘data access object’, usually referred to as a ‘DAO’. It is common in enterprise Java systems to have a number of classes whose names include a DAO suffix to handle the persistence needs for the system. The problem with this naming scheme, if this is the extent of the abstraction, is that it is an implementation concern; there is nothing in the business domain called a ‘DAO’.

Once the DAO naming scheme has been adopted, it has a profoundly limiting effect on the use of the classes that are so named. Developers treat the DAO classes as something that must only be used at the edges of the system, frequently requiring a ‘service layer’ to be bought into existence, partially in order to keep the domain model free of coupling to the persistence layer.

There is nothing wrong with a service layer in principle if its purpose is to orchestrate behaviour at a high level. But far too much behaviour is usually found in these service layers because domain model objects are rightly not allowed to become aware of the DAOs. The effect of this is that the service layer needs to find precisely those things required by the domain logic before finally handing control over to the domain model. The service layer therefore becomes far too familiar with the details of the domain logic which means that at least two places have to change if the domain logic changes.

One lamentably common solution to this problem is to move all domain logic into the  service layer, resulting in an anaemic domain model.

However, there is frequently something in the real world that does represent the place where certain types of objects are stored and this should be represented directly in code with a type named after the domain concept.

If there is no such thing, which often occurs because the modern world has replaced that mechanism with a set of database tables or some equivalent, then I appeal to my opening question: How would this be done in an earlier age?

For example, consider the order management system I have used for examples in other articles. Somewhere in such a system, there is the concept of an ‘order’. Then, we must have a mechanism for persisting orders. If we refer to this as an ‘OrderDAO’ and take the abstraction no further, we give ourselves all of the problems mentioned earlier. However, in an earlier age, orders would have been written down, modified and deleted in an ‘order book’, and this is how the concept can be named in the order management system.

Now, the OrderBook may be, in Java, an interface whose implementation in the running system uses all of the machinery that enterprise Java developers love: Hibernate, Spring, etc. That implementation might even be called something like HibernateOrderBook or, god help us, OrderDAO. The point is that the rest of the order management system knows only about an OrderBook, which is a strong domain concept and hence may be packaged with the domain model and passed around the domain model without reservation. More of the domain logic can therefore be kept in the domain model rather than in services.

Interestingly, this approach frequently clarifies another hot issue at the moment: when and what to mock when unit testing (in the specific sense of interaction testing).

From the point of view of a class that must make use of the OrderBook, all that matters is that the OrderBook is sent something like write(…), delete(…) or find(…) messages with the appropriate arguments. How it writes, deletes or finds is irrelevant, because that is a delegated concern whose implementation should be completely unknown to the subject under test. Bear in mind it could be Hibernate or JDBC, an in memory collection, an invocation of a Rest service using XML over HTTP or teleportation using tangled quantum effects. Any test that verifies more than the message send is overspecified, and hence suffers from the well documented, numerous effects of over-specified definitions. Therefore mocking is entirely appropriate.

Conclusion

Naming is important; it is a fundamental aspect of abstraction. The wrong name can profoundly warp the way people think about a problem which frequently results in a profoundly warped solution. Sometimes, it is hard to find a name because nothing like the thing you are trying to name exists in the appropriate domain.

In such circumstances, I often find the name by thinking about how the problem was solved before the invention of computers: How would this be done in an earlier age?

Sometimes this is due to organisational constraints. In some very hierarchical, command-and-control organisations, by the time the message reaches those who have the ideas, it has been corrupted beyond recognition, and appears to be an admonishment to maintain the status quo or instruction to move in a different direction completely. Even without these confusions, and within a relatively shallow local hierarchy (one or two layers of management), team members can inhibit themselves because of a sense of impotence, with each team member thinking that others have greater authority because of a whole host of factors.

In most organisations hierarchical rank is a major component of a person’s authority and power. But the concept of rank is a complex, multi-dimensional notion that frequently speaks to our primate brains. Furthermore, rank is dependent on context and is relational; a person only has rank with respect to others and in a particular situation – there is no absolute rank.

The Problem

One such organisation we worked with recently suffered from just this issue. Various well-intentioned management attempts to help the team had actually reinforced this sense of lack of authority.

We ran retrospectives to try to get the team to cohere around certain issues and solutions. Almost every retrospectives followed a pattern of identifying issues, coming up with solutions and then declaring the solutions impossible because the team lacked the authority to apply the solutions.

An Exercise

Several years ago, I attended a course run by Ben Fuchs and Joseph Pelrine of CATeams entitled ‘The Deep Dynamics of Agile Teams’. A large part of this course was about rank, and this is where the ideas and material came from for this exercise.

After a (too) brief introduction to the concept of rank, we gave everybody a questionnaire which asked them to assess their own rank on 25 distinct dimensions (e.g. position in the formal hierarchy, education, gender, intelligence, self-confidence, etc.) placed into 3 categories (Situational, Social and Personal).

Having done that, we then asked everybody to pair up. We gave each team member another copy of the questionnaire, asking them to assess their partner’s rank in the same way. Having done that, we asked the pair to discuss the differences in the two assessments of each member of the pair.

What is striking about this process is that usually, individuals will assess their own rank lower than their partner’s assessment on many dimensions. In this instance, almost everybody afforded their partners greater rank than their partner had afforded themselves.

This is quite common and was actually the point of the exercise when Fuchs and Pelrine asked us to do it on the course.

Taking it a Bit Further

Somewhat on a whim, we asked each person to provide us with two post-it notes.

On the first, we asked them to write ‘<’, ‘=’ or ‘>’ if they felt that, in general, their partner had assessed their rank lower, equal to or higher respectively, than their own assessment of their rank.

On the second, we asked them to write ‘<’, ‘=’ or ‘>’ if they felt that, in general, they assessed their partner’s rank lower, equal to or higher than their partner had assessed his or her own rank.

Of course, given that software development teams (including testers, BAs and project managers) frequently believe that they have impeccable logic, this request was regarded with incredulity; ‘surely the results will be symmetrical?’ they asked.

However, once the team had done as we asked, we put the post it notes on display and found that in response to the first question, just over half of the people present felt that their partners had ranked them higher than they had ranked themselves. However, in response to the second, almost all believed that they had ranked their partners higher than than their partners had ranked themselves.

So to some extent the participants’ assessment of their own rank even coloured their assessment of the questionnaire results.

Conclusion

The message is that we frequently assess our own rank lower than other people assess it. The gap between our assessment and the assessment others make of us represents rank, authority and power that we can use if only we are prepared to wield it.

It has become clear to me, as it did with Java 15 years ago, that Scala is an incredibly important language and almost certainly Java’s successor. There is no reason for anyone intending to build applications for the JVM to choose Java over Scala.

Any Java developer is capable of writing Scala in a basic sense, that is, write exactly what they would write in Java but in Scala’s syntax. They can even start by mixing Java and Scala if they feel uncomfortable about jumping all the way in. Developers can choose to use Scala on parts of a system or even just in tests. Scala’s tight integration with Java libraries and frameworks enables developers to make use of the Java ecosystem as they normally would which is incredibly valuable.

However, Scala has a great deal of headroom. It enables much more concise, simpler, reusable and elegant software to be written for those that have the will to learn functional programming and a little practical type theory. Importantly, Scala offers Java developers a way to move from idiomatic Java to idiomatic Scala in small steps, there is no need for a wholesale leap into the unknown. In short, Scala offers a migration path to a better way to build robust software.

A reason why Scala is important is that it offers mechanisms for controlling complexity. In The Structure and Interpretation of Computer Programs, Abelson and Sussman said:

In our study of program design, we have seen that expert programmers control the complexity of their designs with the same general techniques used by designers of all complex systems. They combine primitive elements to form compound objects, they abstract compound objects to form higher-level building blocks, and they preserve modularity by adopting appropriate large-scale views of system structure.

Scala facilitates this abstraction and combination that results in simpler, more robust software in a number of ways: functional programming techniques, object oriented techniques, a rich type system and mechanisms to build components. In Scalable Component Abstractions, Martin Odersky and Matthias Zenger describe abstractions for the construction of reusable components:

abstract type members, explicit selftypes, and modular mixin composition. Together, these abstractions enable us to transform an arbitrary assembly of static program parts with hard references between them into a system of reusable components.

Making full use of Scala does not come for free, there is a lot to learn. Fortunately, the Scala community is very friendly which is in large part due to Martin and his team’s efforts. The community is full of very smart people who are very willing to help novices. It is not unusual for questions asked on mailing lists to be answered by Martin or other leaders in the Scala community. This friendly environment is actively fostered so that beginners feel comfortable asking basic questions without fear of angry replies to RTFM.

There are those that think that Scala is a complex language and they are right to a certain extent, but you can be productive in Scala without using advanced features, just not as productive as you would be if you learnt about those features.

Most objections to new languages like Scala and Clojure can be reduced to, “I don’t understand this after 2 mins of skimming the website so its too academic”. Not very good really and not worthy of further argument.

Learning Scala has real benefits in understanding more about computing in general, and it lets you do it gently whilst being productive. Give it a try!

To be sure, this kind of automated test will run slower than unit tests, because almost all of the machinery of the system under test is going to be involved. But I have seen many implementations that are much slower than they need to be because of one or two technical choices.

The advice that I offer here does not guarantee fast and robust tests, but if you don’t follow this advice and you don’t somehow deal with the issues raised in this article, I am pretty sure your tests will be slow and time-consuming to maintain as the test suite grows.

By way of example, let us consider an online book seller. The kind of test that can be built with the aid of many libraries might well look like this:

public void testSearchAndOrder() {
    type("username", "JRHartley");
    type("password", "verysecret");
    click("Submit");

    type("searchField", "Fly Fishing Hartley");
    click("Search");

    assertEquals("Fly Fishing"), innerHtml("searchresults/tr[1]/td[1]");
    assertEquals("J.R.Hartley"), innerHtml("searchresults/tr[1]/td[2]");

    click("searchresults/tr[1]/td[1]");
    click("Order");

    assertPageTitle("Your order has been placed");
}

This will certainly test the functionality. However, there are two things wrong with it:

• Who is ‘JRHartley’? Why is he buying ‘Fly Fishing’? i.e. Is there something significant about that author or that book in the ‘standard test dataset’? No? Then this test is overspecified. Yes? Then the test is dependent on something that is not stated clearly. In either case, it is dependent on a ‘standard dataset’.
• Is JRHartley really interested in clicking buttons, typing into text fields, etc.? i.e. is the logic we are testing really about button clicks and typing? This test is specifying mechanisms rather than outcome.

These issues contribute to brittleness and difficulties with maintenance. In addition, the dependency on standard data frequently leads to slow tests because there is more data than required.

My general approach to these problems is to focus on abstraction and composability. The solutions shown below achieve these fundamental aspects of software development partially by using a domain specific language in fluent interface style (although the fluency is limited and I usually go considerably further). By good fortune, Debasish Ghosh describes an approach to building DSLs here that is precisely what I am advocating.

However, this choice is largely irrelevant. You could achieve the results with, for example, FIT, Fitness, Concordion or some other tool instead. I have never used those tools and suspect that I don’t really want to, but if you like them, go for it.

Dependency on a ‘Standard Dataset’

Note: Although this section refers to databases, the same argument applies if, for example, your system accepts feed files from other systems and processes them.

My preference is for each test to start with an empty database schema and to populate it with exactly what is needed. This tends to:

• prevent one test from destroying the data that another test depends on
• prevent tests from becoming dependent on an incidental aspect of the data
• prevent more and more data being added to the database in order to accomodate new tests without breaking previous tests
• make tests run quickly because there is very little data in the database

However, I do not want each test to have a bunch of database inserts, because doing so will make the test fragile with respect to database changes and in any case, doing this specifies the test at the wrong level. Therefore the database setup must be abstracted.

In order to truly achieve this, the abstraction must be above the level of database tables; for example, if tests need to know that in order to add a book to the database, a publisher must be created first, each test:

• has structural knowledge of the database leading to duplication of information
• contains information that is not relevant to the test leading to overspecified tests

In the example above, as far as the test is concerned, the book needs two attributes: a name and an author. Undoubtedly, the book will have many more attributes and relationships in the database, but none of these are relevant to this test and so should not be part of the setup. My DSL will therefore start to look like this:

public void testOrderBook() {
    given()
        .aBook()
            .title("Fly Fishing")
            .author("Hartley");

    ...
}

The given() method is an entry point into the fluent interface and this expression should be read ‘Given a book with title “Fly Fishing” and author “Hartley”‘. The important aspect of this is that the book() method initialises all of the book’s fields and knows how to build a book with integrity in the database. I usually initialise fields to random values.

This deals with the dependency on the standard data issue. However, it does not eliminate the over-specificity of the test. To put it simply, while the test is interested in the title and author, it should be non-specific about what title and author is actually used.

Overspecificity

In order to solve this problem, we need to allow the title and author to be given to us, rather than prescribed. As said earlier, my convention is to randomise any unspecified fields. So not specifying an author and title will result in what we want.

public void testOrderBook() {
    Book book = given().aBook();

    ...
}

We can now express the rest of the test using the book’s properties. We will see how to do that later.

Some might feel uncomfortable that something important has been lost from this fixture; the fact that the test does depend on an author and title and yet we are not setting up the book with a particular author and title. If that really bothers you, it is easily remedied (I’ll use the more terse form above for the rest of the article though):

public void testOrderBook() {
    String author = given().anAuthor();
    String title = given().aBookTitle();

    Book book = given()
        .aBook()
            .author(author)
            .title(title);

    ...
}

The important point is that the test is now free of pre-existing fixture data and is minimally specific.

Specify Tests In Business Terms

The other concern in the original test was that it was expressed in terms of what to type into form fields and what buttons to click. We should hide this away:

public void testOrderBook() {
    Book book = given().aBook()

    ...

    then()
        .searchingFor(book.getTitle() + " " + book.getAuthor());

    resultsIn()
        .searchResults(book);

    ...
}

The logic of how to do the search is hidden behind the searchingFor(…) method. The only relevant thing passed is the search term. Subsequently, we make sure that the results returned from the search contain the book. We could be more specific and assert that the search results contain only one book. Of course, for that to have any real relevance, more than one book would have to be created in the database.

This test now does not directly depend on button clicks and, apart from a few artifacts of the Java language, is pretty easy to understand at the level of business interactions.

Finally, we want to place the order:

public void testOrderBook() {
    Book book = given().aBook();
    User user = given().aUser();

    loginAs(user);

    then()
        .searchingFor(book.getTitle() + " " + book.getAuthor())

    resultsIn()
        .searchResults(book);

    then()
        .select(book)
        .order()

    resultsIn()
        .anOrder(book, user);
}

And once again, details about how the order is placed and how we verify that the order has been placed are abstracted away. Of course, if the user was able to check their orders on a web-page, we could use that page to determine that the order has been placed. Otherwise, we might go to the database to make sure that an entry has been added to an appropriate table. We might also expect an email to be sent, so the anOrder(Book, User) method might verify that too. The point is that all of that can be hidden in anOrder(Book, User) and can be changed over time if necessary.

Emergent Goodness

The approach described above brings enough benefits as it is; fixture data independence and tests specified at the level of the business process. However, there are now two things that also emerge:

• Because the DSL is written in fluent interface style in Java, as more tests are written, the fluent interface helps to guide the writing of tests. New tests can be composed very quickly.
• The test is not tied to an implementation. The fact that this started life as a test for a web-app does not mean that it will always be so. An entire test suite can be reused to test, for example, a REST api, or a B2B messaging system for book supplies. All that is needed is to reimplement the model behind the DSL appropriately. This may not be a small task, but the fact remains that the tests themselves can be used in many ways.

This latter point has been exercised on one of our recent projects in which there were multiple ways of using the system (web-app, REST API and message queues each offering the same functionality to clients).

Conclusion

In order to keep automated acceptance tests maintainable, fast and flexible, focus on those staples of solid development: abstraction and composability. Avoid ‘standard data sets’ and express tests in terms of business processes.