What is a type class and why do I need it?

Type classes are useful to solve several fundamental challenges in software engineering and programming languages. In particular type classes support retroactive extension: the ability to extend existing software modules with new functionality without needing to touch or re-compile the original source. [1]

The Scala library includes a few typeclasses such as scala.math.Numeric and scala.math.Ordering, and Scalaz is all typeclasses.

Scala enables the typeclass pattern using traits and implicits, and whilst Scala’s implementation is more verbose than Haskell’s, it comes with greater flexibility. Haskell only allows a single typeclass instance globally, whereas Scala allows any number to be available.
Furthermore, Scala allows default implementations to be made available if no others can be found.

For a deeper understanding see the references at the bottom of the blog.

Show me the code!

(You can copy and paste this into your favourite IDE)

/* Here is the first piece of the Scala typeclass puzzle, its just a trait.
 * The trait defines a concept which in this case is a transformer 
 * that transforms a type T into a R.
 */
trait Transformer[T, R] {
  def transform(t: T): R
}
 
/* 
 * This is a companion object for the typeclass giving default implementations for the typeclass.
 * These implementations are found after local implicits, so you can still override the default
 * behaviour. For more about the search order see [3].
 */
object Transformer {
  implicit object IntToStringTransformer extends Transformer[Int, String] { def transform(t: Int) = t.toString }
 
  /* This one is interesting. It transforms a List[T] into a String, but to do that it
   * needs a transformer for T to String. So it requires such a transformer implicitly.
   */
  implicit def ListToStringTransformer[T](implicit tToString: Transformer[T, String]) = new Transformer[List[T], String] {
    def transform(t: List[T]) = t.map(tToString.transform(_)).mkString(",")
  }
}
 
// This is something that makes use of the typeclass
trait Transform {
 
  // The implicit Transformer, transformer, supplies an appropriate transformer for the method
  def transform[T, R](t: T)(implicit transformer: Transformer[T, R]): R = transformer.transform(t)
}
 
// These examples will drop back to the default implementations in the Transformer's companion object
object ExampleWithDefaults extends App with Transform {
  println(transform(2)) // prints 2
  println(transform(List(1, 2, 3))) // prints 1,2,3
}
 
// I want my own special transformers!
object ExampleWithMyTransformers extends App with Transform {
 
  implicit object MyIntToXmlTransformer extends Transformer[Int, xml.Node] { def transform(t: Int) = <aNumber>{ t.toString }</aNumber> }
 
  // As with the default List transformer, this one needs a transformer of T to Node so that it can perform the transform for lists.
  implicit def MyListToXmlTransformer[T](implicit transformer: Transformer[T, xml.Node]) = new Transformer[List[T], xml.NodeSeq] {
    import xml._
    def transform(t: List[T]): NodeSeq = t.foldLeft(NodeSeq.Empty)((accumulator, next) ⇒ accumulator ++ transformer.transform(next))
  }
 
  println(transform(2)) // prints <aNumber>2</aNumber>
  println(transform(List(1, 2, 3))) // prints <aNumber>1</aNumber><aNumber>2</aNumber><aNumber>3</aNumber>
 
  // Here is a transformer from Boolean to String
  implicit object BooleanToStringTransformer extends Transformer[Boolean, String] { def transform(b: Boolean) = b.toString }
 
  // This is the clever bit. 
  // Now we can now transform List[Boolean] to String. The List transformer defined in the default Transformer trait
  // will be used but with the new BooleanToStringTransformer. So the default transformers and the new ones
  // work together.
  println(transform(List(true, false))) // prints true,false
 
  // Finally, if there are no implicits available then the code fails to compile.
  // transform("3")
}

Summary

So we defined a typeclass which has been extended with new instances. If the transformer typeclass were part of a library, we can retroactively extend the library with new behaviour supporting new types not originally conceived of in the library. Furthermore, the library supplied types and the new ones work together perfectly.

Further Study

[1] “Type Classes as Objects and Implicits”; Bruno C. d. S. Oliveira, Adriaan Moors, Martin Odersky
[2] “How to make ad-hoc polymorphism less ad-hoc”; Philip Wadler, Stephen Bolt, (POPL ’89)
[3] Implicit resolution order
[4] Tutorial: Typeclass with Dan Rosen

/**
 * Functions that do the flattening
 */
object Loans {
 
  def loan[T1, T2](f: (T1, T2) ⇒ Unit)(implicit f1: (T1 ⇒ Unit) ⇒ Unit, f2: (T2 ⇒ Unit) ⇒ Unit)
  	= f1 { a ⇒ f2 { b ⇒ f(a, b) } }
 
  def loan[T1, T2, T3](f: (T1, T2, T3) ⇒ Unit)(implicit f1: (T1 ⇒ Unit) ⇒ Unit, f2: (T2 ⇒ Unit) ⇒ Unit, f3: (T3 ⇒ Unit) ⇒ Unit) 
  	= f1 { a ⇒ f2 { b ⇒ f3 { c ⇒ f(a, b, c) } } }
 
}
 
object LoanExample extends App {
 
  import Loans._
 
  // create some fake types for the purpose of illustration
  type Connection = Unit
  type Session = Int
  type User = String
 
  // implicits calling functions with loans
  implicit def withDatabase(f: Connection ⇒ Unit) = f(1)
  implicit def withSession(f: Session ⇒ Unit) = f(1)
  implicit def withUser(f: User ⇒ Unit) = f("Bob")
 
  // conventional loan pattern results in deep nesting
  withDatabase { connection ⇒
    withSession { session ⇒
      withUser { user ⇒
        println("Hi from the depths.")
      }
    }
  }
 
  // flattened
  loan { (c: Connection, s: Session, u: User) ⇒ 
    println("hi from flatland") 
  }
 
}

The nice thing about this is that the implicit argument lists in Loans enable you to call a loan method without passing those functions in explicitly. A problem will be if you need to loan two or more of the same type which is not likely since one is usually borrowing unique resources.

What’s the Problem?

Given a function that takes multiple arguments A,B,… that returns a Result, how can that function by applied to arguments M[A], M[B],… and get M[Result]?

where M is an Option, List, etc.

Applicative Functors

To understand the mechanics of how applicative functors solve our problem, please refer to the references below, in particular Heiko Seeberger’s Applicatives are generalized functors.

What I present below is an example of use. I hope you can take it away and benefit from it without necessarily understanding the theory. However, I thoroughly recommend further reading and study, its fun and you will gain much deeper insights.

So without further ado, here’s the code:

import scalaz._
import Scalaz._
 
object Applicatives extends App {
 
  /*
   * Lets assume some Options which must be applied to a function.
   */
  val x:Option[Int] = 2.some // scalaz enrichment for options
  val y:Option[Int] = 3.some
  val z:Option[Int] = 5.some
 
  // Lets add them without applicative functors
  val usingFor = for (theX <- x; theY <- y; theZ <- z) 
                   yield theX + theY + theZ
 
  val usingMaps = x flatMap 
                    (theX => y flatMap 
                      (theY => z map 
                        (theZ => theZ + theY + theX)))
 
  /* With scalaz we can do the following instead of for or maps
   * First we need to put the function in the right form, curried.
   * To understand why please read the references I've given below.
   */
  val addInts = ( (a:Int, b:Int, c:Int) => a + b + c ).curried
 
  // apply the function to x, y and z
  val sum = x <*> (y <*> (z map addInts)) // Some(10)
 
  // Scalaz offers an alternative syntax that is easier to use
  (x |@| y |@| z) {_ + _ + _}   // Some(10)
 
  /*
   * If one of the options is a none
   * then the result of the whole expression will be none.
   */
  (x |@| none[Int]) {_ + _} // None
 
  /*
   * The function can be any method, including 'apply'
   */
  case class Person(age: Int, height:Double, name: String)
 
  /*
   * Person.apply method is a function (Int, Double, String) => Person
   */
  (some(2) |@| some(1.1) |@| none[String]) {Person.apply _}  
  // none
 
  (some(4) |@| some(1.1) |@| "Angelica".some) {Person.apply _} 
  // Some(Person(4, 1.1, Angelica))
 
  /*
   * The beauty of this is that it works with ANY higher kind, eg. List
   * or your own types!
   */
  val l1 = 1 :: 2 :: Nil
  val l2 = 3 :: 4 :: Nil
 
  (l1 |@| l2) {_ + _} // List(4, 5, 5, 6)
  (<xml/> |@| <xml2/>){_ ++ _} // List(<xml></xml><xml2></xml2>)
}

Further Reading

• Learn You a Haskell’s Chapter on Functors, Applicative Functors and Monoids
• Heiko Seeberger’s Applicatives are generalized functors
• Scalaz Applicative Examples

What’s the Problem?

We often want to use a function that takes an instance of A and returns an instance of B in a context where we have an instance of M[A] and need an instance of M[B]. The most common example is the map method on List, Option, etc.

So the problem is:

Given a function f that takes an A and returns a B, how can we construct a new function that takes an M[A] and returns an M[B] for any M?

Functors

A Functor accepts a function, A ⇒ B, and returns a new function M[A] ⇒ M[B], where M is any kind.

So without further ado, here’s the code:

object Functorise extends App {
 
  // This is a Functor
  trait Functor[M[_]] {
 
    /* convert f into a function mapping M[A] to M[B]
     * eg. if M were List, and f was Int ⇒ String
     * fmap would yield List[Int] ⇒ List[String]
     */
    def fmap[A, B](f: A ⇒ B): M[A] ⇒ M[B]
  }
 
  /* Here are a couple of examples for Option and List Functors
   * They are implicit so they can be used below in enrichWithFunctor
   */
 
  implicit object OptionFunctor extends Functor[Option] {
    def fmap[A, B](f: A ⇒ B): Option[A] ⇒ Option[B] 
      = option ⇒ option map f
  }
 
  implicit object ListFunctor extends Functor[List] {
    def fmap[A, B](f: A ⇒ B): List[A] ⇒ List[B] 
      = list ⇒ list map f
  }
 
  /* enrichWithFunctor is an implicit to enrich any kind 
   * with an fmap method.
   * List, Option and any other Foo[X] can be enriched with the
   * new method.
   */
  implicit def enrichWithFunctor[M[_], A](m: M[A]) = new {
 
    /* fmap requires an implicit functor, whose type is M, to which it
     * delegates to do the real work
     */
    def mapWith[B](f: A ⇒ B)(implicit functor: Functor[M]): M[B] 
      = functor.fmap(f)(m)
  }
 
  // some examples
 
  println(List(1, 2) mapWith (_ + 1)) // List(2, 3)
 
  println(some(1) mapWith (_ + 1) mapWith (_ * 3)) // Some(6)
 
  println(none[Int] mapWith (_ + 1)) // None
 
  def some[A](a: A): Option[A] = Some(a)
  def none[A]: Option[A] = None
}

What is Going On?

Lets take the List(1, 2) mapWith (_ + 1) example and go through it.

There is no mapWith method on List, so an implicit method is searched for that can turn the List into something that does have a mapWith method. Thus, enrichWithFunctor is found and called with the List(1, 2) which returns an anonymous class with the mapWith method required.

By the way, to understand more about that M[_], watch Daniel Spiewak’s High Wizardry in the Land of Scala talk.

That anonymous class also has a type parameter, M, which has now been fixed to List, and so mapWith in this class looks like this:

def mapWith[B](f: A ⇒ B)(implicit functor: Functor[List]): List[B] 
  = functor.fmap(f)(m)

The mapWith method requires an implicit Functor[List] to be available, and the ListFunctor object is just that. So when functor.fmap(f) is called, it is the ListFunctor’s fmap method being called.

ListFunctor.fmap(f) takes a function, in our example its (i:Int) ⇒ i + 1, and returns a new function (list:List[Int]) ⇒ list.map(_ + 1).

Finally, m, the List(1, 2) we started with that was enriched with the mapWith method, is applied to the new function which results in a new List.

But Why?

All we’ve done here is create a complicated looking bunch of code that just ends up calling methods on Option and List that already exist, specifically the map method.

Yes, in these examples thats true.

The real idea is that we’ve created a general abstraction for all kinds, including those that don’t have a map method. All a user of this code needs to do is have an implicit Functor in scope for the kind they would like to map over.

Furthermore, this is a building block for some interesting things that come next like Applicative Functors.

Finally, you don’t need to implement this yourself, Scalaz has it already for you to exploit.

Further Reading

• Learn You a Haskell’s Chapter on Functors, Applicative Functors and Monoids
• Heiko Seeberger’s fantastic Introduction to Category theory in Scala
• Scalaz Functor Examples

The system enables HSBC to handle the processing of all cleared interest rate and credit derivatives from multiple market participants. The first release provided full connectivity to the London Clearing House, with the Chicago Mercantile Exchange and the Intercontinental Exchange following shortly.

We had a tight deadline and quite a lot of things to figure out. The project had been difficult to analyse in the past as a result of changes in legislation and many other unknowns. So how did we do it?

Talking

The first thing we always do is talk to the business and figure out what is needed. Not in abstract business terms or strategic direction, but what the actual user sitting in front of our app is actually going to see, do and expect. Whilst the general business background is useful context and necessary, it is woefully inadequate for building a system.

Having the people who are going to actually build your system talk, in person, to the people that are actually, physically, going to use that system is critical. Not managers talking to managers but the people doing the work. And we do that every day, for every feature!

Minimal Viable Product

The next critical step is establishing the absolute minimum that will enable the system to be useful, and deliver that. Nothing else! (until next week)

It is surprisingly difficult to help stakeholders become accustomed to focussing on what is needed for the next release. IT has done a really good job of telling people they need to write down all their requirements for the next year or they will not get them. We have quite a bit of work to do to undo that ridiculous notion. But when the business understand that we aren’t saying we won’t let them have a feature, just that we want them to prioritise what they want for the next release (next week), and they can change their mind whenever they like, they are very happy.

Who Else Cares?

I have talked about the business as if they are the only interested party. Not so. We talk to every stakeholder we can find: senior management, BAs, support people, release people, release boards, architects; everyone who might ever have some kind of interest in the system. They all have needs that have to be addressed and prioritised. On one project we were able to name 23 different stakeholder roles for a project, which was surprising to everyone who thought there was only one stakeholder.

Planning

Having established a backlog of stuff, we maintain it by planning weekly with all stakeholders, and marvel at how much priorities change, new work appears and old work disappears from week to week. But thats all good! Every change is welcome, and shows how badly wrong things would have gone had we tried to plan-it-up-front.

Where’s the Code?

Every day, the team has a stand up to discuss what they are going to do. Then, we implement some features.

1. pair up with another team member in the stand up
2. talk to the user and find out what they want, sketch ideas out on paper or mock them up until enough is understood
3. write a failing acceptance test with the user. We use Specs2 but you can use whatever you like so long as it produces some kind of user visible result
4. make the acceptance test pass by typing some code in your IDE, writing unit tests as necessary

However, a lot of what we did involved communication with other systems too for which we required real samples of messages since nothing had schemas etc. This was where a lot of time was spent, again in dialog with stakeholders to ensure that the messages we process and respond with were correct.

Good, all done then!

No. When we think we’ve finished a feature we demo it to the user as soon as possible – we don’t wait to the end of the iteration. They can play with it and change things which they often do when they see something working for real.  And thats all good, we want them to have something useful.

Builds?

Part of the development practice is to commit code frequently ensuring tests always pass. People are surprised by how often we mean by frequently. We mean every single time you complete a piece of work and all the tests pass locally. That happens about once an hour but can be more frequent.

We also use TeamCity to keep us sane but again there are many alternatives.

Finally, deployment. At the start of all project’s we always have a single mantra about deployment: Thou must automate the deployment. More than that, the same deployment script should work on every dev, test, uat and production environment. Sometimes, the production environment needs to be subtly different, but we try to make the other environments as close to production as possible.

Technology

We built the system in Scala with Liftweb. It was tested with Specs2 and we used sbt for builds. Scala is significantly more productive than Java, and Liftweb enabled us to build a robust, secure, dynamic web application very quickly.

Summary

Is this starting to sound familiar? Yup, its basically an agile method using practices from XP, Scrum, Kanban, anything that works for this client in this project with this team. And no, you can’t transplant it to another team! They will have to figure out what works for them, and when we move to another project, we have to figure out what will work for the new environment.

I present the example without explanation, the web does not need another set of explanations about this subject but I have found the following blogs useful, with a mix of theoretical explanation and practical examples:

• Functional IO in Scala with Scalaz
• The IO Monad for People who Simply Don’t Care
• Towards an Effect System in Scala, Part 2: IO Monad

All this example does is print out the names of files in a directory.

import scalaz._
import Scalaz._
import scalaz.effects._
import java.io._
 
object TheIO extends App {
 
  val dirInIO = io { new File(System.getProperty("user.dir")) }
 
  def filesInIO = for (dir <- dirInIO) yield dir.listFiles()
 
  def namesInIO = for (theFiles ← filesInIO) yield theFiles.map(_.getName())
 
  def printNamesInIO = for (all ← namesInIO) yield all.map(println(_))
 
  val result = printNamesInIO
 
  println("Nothings happened yet, lets go!")
  result.unsafePerformIO
}

Here’s some code which is based on an example on Scalaz’s website.

import scalaz._
import Scalaz._
 
object Applicatives extends App {
 
  /*
   * Lets pretend you have two optional values, but you want to apply some
   * function to their contents safely.
   * You also don't want to fiddle about with flatmap / map / for / yield
   * Then do this:
   */
  val x:Option[Int] = Some(2)
  val y:Option[Int] = Some(3)
 
  (x |@| y) {_+_} 	// Some(5)
 
  /*
   * The |@| method is a pimp (enrichment! sorry), on Option in this case,
   * that builds 'something' that applies the function _ + _
   */
 
  /*
   * If one of the options is a none
   * then the result of the whole expression will be none.
   * (I am using scalaz's some and none because it means less boilerplate)
   */
  (some(2) |@| none[Int]) {_+_} // None
 
  /*
   * The function can be any method, including 'apply'
   */
  case class Person(age: Int, height:Double, name: String)
 
  /*
   *  ⊛ = |@| just for fun
   */
  (some(2) ⊛ some(1.1) ⊛ none[String]) {Person.apply _}  // none because an arg was none
  (some(4) ⊛ some(1.1) ⊛ "Angelica".some) {Person.apply _} // Some( Person(4, 1.1, Angelica) )
 
  /*
   * did you catch the "Angelica".some - another bit of enriching by scalaz
   */
 
  /*
   * It works with ANY higher kind, eg. List
   */
  val l1 = 1 :: 2 :: Nil
  val l2 = 3 :: 4 :: Nil
 
  (l1 ⊛ l2) {_ + _} // List(4, 5, 5, 6)
 
}

References

1. Eric Torrebore has a very nice article explaining much of this.
2. Learn You a Haskell has a great chapter on applicative functors amongst other things.
3. Heiko Seeberger blog post

Having obtained estimates for all of the work, and with contracts duly signed, etc., the work was begun. And so, we tracked the burndown:

The chart above shows the first 43 days of burndown.

One common question that we try to answer with the burndown charts is: ‘What day do we expect the burndown to reach 0?’. That is: ‘When do we expect to finish?’ The assumption is that historical performance predicts future performance. In our case, the question is more like: ‘Is there any reason to suspect we might not finish by the contracted date?’

One simple way of answering this question is to look at ratios: we have completed (551 - 302) estimated days of work in 43 days, so we can expect to take 43 * 551 / (551 - 302) = 96 days to finish everything.

A more ‘sophisticated’ approach is to put a linear regression line through the burndown and extrapolate. This tells us that we will reach 0 remaining estimate on day 85, but it fails to give an indication of the confidence we can have about this prediction. We can use the the correlation coefficient to give some idea of this, but it is difficult to know what to do with it.

In fact, the questions above are not really the right ones. The question we should be asking is: ‘How confident can we be that the burndown will reach zero on or before a given date?’.

In order to answer this, we need a different approach. One such approach is based on the Monte Carlo Method, which in this case translates to asking: ‘Assuming the future burndown is similar to the historical data, what fraction of the possible futures result in the burndown reaching zero on or before a given date (or between two dates)?’

We can apply this method by starting with the estimate on the last burndown day for which we have an estimate (day 43), choosing a day from the history randomly, determining the change in burndown between this day and the preceding one, and then applying that change to the estimate we have for day 43. This then becomes the estimate for day 44 and we repeat the process, until the estimate reaches 0. This then represents one simulated burndown, and we repeat the whole process thousands of times to obtain a distribution of estimates for each future day.

The chart below shows the first result from applying this method:

The line extending the historical burndown is the ‘Median Line’. This is obtained by determining the estimate for each day for which half of the simulations yield a higher estimate and half yield a lower estimate. When this line intersects the Day Number axis (day 97), it tells us that half of the simulated burndowns have reached a zero estimate on or before day 97 and half have not. Based on this information (and speaking somewhat informally), we have a half chance of delivering on or before day 97 and a half chance of delivering afterwards.

Knowing this is interesting information, but it is reasonable to ask what is the spread of simulation results. After all, if all simulations terminate within a very tight spread centred around day 97, we can still reasonably accurately predict the end day. If however, the spread is very wide, some other action might be suggested.

We can obtain an idea of the spread in the same way as we obtained the median line. If, for each day, we ask ‘What value of estimates (centred around the median value) contain x% of the simulated results on each simulated day?’. This will give us ‘confidence intervals’ around the median:

The chart above shows the 50%, 90% and 99% confidence intervals. Consider the 90% confidence interval, which is enclosed by the dark green curves starting at the final known burndown day (43, 302) and terminating at (74, 0) and (124, 0). On each simulated day, the set of estimates between these two curves contain 90% of the simulated burndowns. Because we placed the intervals symmetrically around the median line, the space underneath the lower curve represents 5% of the simulated estimates (those that represent the team performing above expectations), and the space above the upper curve represents the final 5% of the simulated estimated (those that represent the team performing below expectations).

Clearly the spread is quite wide! If we wanted to give a date that represented 95% confidence, we should choose choose day 124, which is 27 days after the median line predicts, or a difference of 25%. It is also 28 days (also about 25%) after our simple ratio based estimate given at the beginning of this article and 39 days (40%) after the the linear regression estimate.

The astute reader might notice two important things:

1. The day at which the confidence interval curves reach a Remaining Estimate of 0 spreads out alarmingly as the required confidence increases. i.e. the 50%, 90% and 99% upper confidence envelopes require 10%, 28% and 45% respectively more days to reach 0 than the median line does.
2. The confidence intervals are distributed symmetrically around the median line when a vertical slice is taken through the chart. Indeed, the histograms giving rise to these confidence intervals are near-Gaussian (courtesy of the Central Limit Theorem), at least until some of the simulations are terminated when they reach 0 remaining estimate. This is not true when a horizontal slice is taken. In particular, when those curves reach the Remaining Estimate of 0, it is plain to see that, for example, the lower envelope for the 90% confidence interval is 97 - 74 = 23 days below the median, whereas the corresponding upper curve is 124 - 97 = 27 days above the median.

The first of these points suggests that, even at this advanced state of the project, requiring high confidence about the delivery date demands significant ‘contingency’, with the 99% upper confidence envelope prediction being more than a factor of 2 larger than the corresponding lower confidence envelope prediction. Similarly, the ratio between the upper 99% confidence envelope prediction and the median prediction is just under 1.5.

The second point suggests that it is easier to run late than it is to run early, which will come as no surprise to development teams and their customers everywhere. Indeed, the skew also implies that when things are running late, it is possible for even more tardiness to result. This is intuitively true since, with less work remaining to be done, it is less likely that significant estimation ‘errors’ will occur.

Of course, we should not take all of this too seriously; a great deal has been made of very little historical data. Attention to a burndown chart often causes teams to adjust scope and other factors to bring the project in on time. The date is far more likely to be missed because of some unmitigated risk, such as the absence of a team member for an extended period of time.

In a subsequent article, once the delivery is complete, I will return to this topic with the remainder of the burndown data and see how well the confidence intervals modelled the future.

Update: About a week after I wrote this article, a series of events occurred that made it impossible to complete the graph in the way that I’d hoped.

First, one of the ‘key’ developers left the team. The notion of ‘key’ developer (due to functional silos) is something that I’d been trying to eliminate since starting with the organisation, but the team had not had sufficient time to get to grips with each other’s work.

Since the team was not large, the developer leaving implied that timescales would change. This caused the client to panic somewhat, leading to a review of the planning. During this review, it was discovered that the original requirements were hopelessly inadequate in term of both ‘completeness’ and ‘correctness’. After more analysis, the total scope required was estimated at six times (!) the original estimate and much of the remaining original work envisaged was de-prioritised or deemed to be no longer required.

Given the significantly extended timescales, a couple of the developers who had been intending to hang on until the delivery decided to resign, since the delivery was now clearly much more remote.

Shortly after this, a member of the executive team who I had a good relationship with was replaced by someone else (not because of this project). It very quickly became obvious that this new executive had a strong belief in process over people, big requirements up front and all manner of things I have railed against for years. This clearly spelled the end of my time with this client.

Although I had warned of much of this risk early in the project, I take no pleasure in its materialisation. It could have all been completely different.

As I said in the first paragraph above: This is not the mode in which we usually like to work.

This is the question I frequently ask myself when searching for the names of domain concepts.

The typical architectures and layering of code in ‘the enterprise’ frequently exhibit poor systems of naming which causes developers to jump through hoops to try to maintain the layers because of lack of abstraction. This in turn results in an unwieldy codebase that is more concerned with accidental implementation complexity than codifying domain knowledge.

As an example consider the ‘data access object’, usually referred to as a ‘DAO’. It is common in enterprise Java systems to have a number of classes whose names include a DAO suffix to handle the persistence needs for the system. The problem with this naming scheme, if this is the extent of the abstraction, is that it is an implementation concern; there is nothing in the business domain called a ‘DAO’.

Once the DAO naming scheme has been adopted, it has a profoundly limiting effect on the use of the classes that are so named. Developers treat the DAO classes as something that must only be used at the edges of the system, frequently requiring a ‘service layer’ to be bought into existence, partially in order to keep the domain model free of coupling to the persistence layer.

There is nothing wrong with a service layer in principle if its purpose is to orchestrate behaviour at a high level. But far too much behaviour is usually found in these service layers because domain model objects are rightly not allowed to become aware of the DAOs. The effect of this is that the service layer needs to find precisely those things required by the domain logic before finally handing control over to the domain model. The service layer therefore becomes far too familiar with the details of the domain logic which means that at least two places have to change if the domain logic changes.

One lamentably common solution to this problem is to move all domain logic into the  service layer, resulting in an anaemic domain model.

However, there is frequently something in the real world that does represent the place where certain types of objects are stored and this should be represented directly in code with a type named after the domain concept.

If there is no such thing, which often occurs because the modern world has replaced that mechanism with a set of database tables or some equivalent, then I appeal to my opening question: How would this be done in an earlier age?

For example, consider the order management system I have used for examples in other articles. Somewhere in such a system, there is the concept of an ‘order’. Then, we must have a mechanism for persisting orders. If we refer to this as an ‘OrderDAO’ and take the abstraction no further, we give ourselves all of the problems mentioned earlier. However, in an earlier age, orders would have been written down, modified and deleted in an ‘order book’, and this is how the concept can be named in the order management system.

Now, the OrderBook may be, in Java, an interface whose implementation in the running system uses all of the machinery that enterprise Java developers love: Hibernate, Spring, etc. That implementation might even be called something like HibernateOrderBook or, god help us, OrderDAO. The point is that the rest of the order management system knows only about an OrderBook, which is a strong domain concept and hence may be packaged with the domain model and passed around the domain model without reservation. More of the domain logic can therefore be kept in the domain model rather than in services.

Interestingly, this approach frequently clarifies another hot issue at the moment: when and what to mock when unit testing (in the specific sense of interaction testing).

From the point of view of a class that must make use of the OrderBook, all that matters is that the OrderBook is sent something like write(…), delete(…) or find(…) messages with the appropriate arguments. How it writes, deletes or finds is irrelevant, because that is a delegated concern whose implementation should be completely unknown to the subject under test. Bear in mind it could be Hibernate or JDBC, an in memory collection, an invocation of a Rest service using XML over HTTP or teleportation using tangled quantum effects. Any test that verifies more than the message send is overspecified, and hence suffers from the well documented, numerous effects of over-specified definitions. Therefore mocking is entirely appropriate.

Conclusion

Naming is important; it is a fundamental aspect of abstraction. The wrong name can profoundly warp the way people think about a problem which frequently results in a profoundly warped solution. Sometimes, it is hard to find a name because nothing like the thing you are trying to name exists in the appropriate domain.

In such circumstances, I often find the name by thinking about how the problem was solved before the invention of computers: How would this be done in an earlier age?

Sometimes this is due to organisational constraints. In some very hierarchical, command-and-control organisations, by the time the message reaches those who have the ideas, it has been corrupted beyond recognition, and appears to be an admonishment to maintain the status quo or instruction to move in a different direction completely. Even without these confusions, and within a relatively shallow local hierarchy (one or two layers of management), team members can inhibit themselves because of a sense of impotence, with each team member thinking that others have greater authority because of a whole host of factors.

In most organisations hierarchical rank is a major component of a person’s authority and power. But the concept of rank is a complex, multi-dimensional notion that frequently speaks to our primate brains. Furthermore, rank is dependent on context and is relational; a person only has rank with respect to others and in a particular situation – there is no absolute rank.

The Problem

One such organisation we worked with recently suffered from just this issue. Various well-intentioned management attempts to help the team had actually reinforced this sense of lack of authority.

We ran retrospectives to try to get the team to cohere around certain issues and solutions. Almost every retrospectives followed a pattern of identifying issues, coming up with solutions and then declaring the solutions impossible because the team lacked the authority to apply the solutions.

An Exercise

Several years ago, I attended a course run by Ben Fuchs and Joseph Pelrine of CATeams entitled ‘The Deep Dynamics of Agile Teams’. A large part of this course was about rank, and this is where the ideas and material came from for this exercise.

After a (too) brief introduction to the concept of rank, we gave everybody a questionnaire which asked them to assess their own rank on 25 distinct dimensions (e.g. position in the formal hierarchy, education, gender, intelligence, self-confidence, etc.) placed into 3 categories (Situational, Social and Personal).

Having done that, we then asked everybody to pair up. We gave each team member another copy of the questionnaire, asking them to assess their partner’s rank in the same way. Having done that, we asked the pair to discuss the differences in the two assessments of each member of the pair.

What is striking about this process is that usually, individuals will assess their own rank lower than their partner’s assessment on many dimensions. In this instance, almost everybody afforded their partners greater rank than their partner had afforded themselves.

This is quite common and was actually the point of the exercise when Fuchs and Pelrine asked us to do it on the course.

Taking it a Bit Further

Somewhat on a whim, we asked each person to provide us with two post-it notes.

On the first, we asked them to write ‘<’, ‘=’ or ‘>’ if they felt that, in general, their partner had assessed their rank lower, equal to or higher respectively, than their own assessment of their rank.

On the second, we asked them to write ‘<’, ‘=’ or ‘>’ if they felt that, in general, they assessed their partner’s rank lower, equal to or higher than their partner had assessed his or her own rank.

Of course, given that software development teams (including testers, BAs and project managers) frequently believe that they have impeccable logic, this request was regarded with incredulity; ‘surely the results will be symmetrical?’ they asked.

However, once the team had done as we asked, we put the post it notes on display and found that in response to the first question, just over half of the people present felt that their partners had ranked them higher than they had ranked themselves. However, in response to the second, almost all believed that they had ranked their partners higher than than their partners had ranked themselves.

So to some extent the participants’ assessment of their own rank even coloured their assessment of the questionnaire results.

Conclusion

The message is that we frequently assess our own rank lower than other people assess it. The gap between our assessment and the assessment others make of us represents rank, authority and power that we can use if only we are prepared to wield it.