Sometimes this is due to organisational constraints. In some very hierarchical, command-and-control organisations, by the time the message reaches those who have the ideas, it has been corrupted beyond recognition, and appears to be an admonishment to maintain the status quo or instruction to move in a different direction completely. Even without these confusions, and within a relatively shallow local hierarchy (one or two layers of management), team members can inhibit themselves because of a sense of impotence, with each team member thinking that others have greater authority because of a whole host of factors.

In most organisations hierarchical rank is a major component of a person’s authority and power. But the concept of rank is a complex, multi-dimensional notion that frequently speaks to our primate brains. Furthermore, rank is dependent on context and is relational; a person only has rank with respect to others and in a particular situation – there is no absolute rank.

The Problem

One such organisation we worked with recently suffered from just this issue. Various well-intentioned management attempts to help the team had actually reinforced this sense of lack of authority.

We ran retrospectives to try to get the team to cohere around certain issues and solutions. Almost every retrospectives followed a pattern of identifying issues, coming up with solutions and then declaring the solutions impossible because the team lacked the authority to apply the solutions.

An Exercise

Several years ago, I attended a course run by Ben Fuchs and Joseph Pelrine of CATeams entitled ‘The Deep Dynamics of Agile Teams’. A large part of this course was about rank, and this is where the ideas and material came from for this exercise.

After a (too) brief introduction to the concept of rank, we gave everybody a questionnaire which asked them to assess their own rank on 25 distinct dimensions (e.g. position in the formal hierarchy, education, gender, intelligence, self-confidence, etc.) placed into 3 categories (Situational, Social and Personal).

Having done that, we then asked everybody to pair up. We gave each team member another copy of the questionnaire, asking them to assess their partner’s rank in the same way. Having done that, we asked the pair to discuss the differences in the two assessments of each member of the pair.

What is striking about this process is that usually, individuals will assess their own rank lower than their partner’s assessment on many dimensions. In this instance, almost everybody afforded their partners greater rank than their partner had afforded themselves.

This is quite common and was actually the point of the exercise when Fuchs and Pelrine asked us to do it on the course.

Taking it a Bit Further

Somewhat on a whim, we asked each person to provide us with two post-it notes.

On the first, we asked them to write ‘<’, ‘=’ or ‘>’ if they felt that, in general, their partner had assessed their rank lower, equal to or higher respectively, than their own assessment of their rank.

On the second, we asked them to write ‘<’, ‘=’ or ‘>’ if they felt that, in general, they assessed their partner’s rank lower, equal to or higher than their partner had assessed his or her own rank.

Of course, given that software development teams (including testers, BAs and project managers) frequently believe that they have impeccable logic, this request was regarded with incredulity; ’surely the results will be symmetrical?’ they asked.

However, once the team had done as we asked, we put the post it notes on display and found that in response to the first question, just over half of the people present felt that their partners had ranked them higher than they had ranked themselves. However, in response to the second, almost all believed that they had ranked their partners higher than than their partners had ranked themselves.

So to some extent the participants’ assessment of their own rank even coloured their assessment of the questionnaire results.

Conclusion

The message is that we frequently assess our own rank lower than other people assess it. The gap between our assessment and the assessment others make of us represents rank, authority and power that we can use if only we are prepared to wield it.

It has become clear to me, as it did with Java 15 years ago, that Scala is an incredibly important language and almost certainly Java’s successor. There is no reason for anyone intending to build applications for the JVM to choose Java over Scala.

Any Java developer is capable of writing Scala in a basic sense, that is, write exactly what they would write in Java but in Scala’s syntax. They can even start by mixing Java and Scala if they feel uncomfortable about jumping all the way in. Developers can choose to use Scala on parts of a system or even just in tests. Scala’s tight integration with Java libraries and frameworks enables developers to make use of the Java ecosystem as they normally would which is incredibly valuable.

However, Scala has a great deal of headroom. It enables much more concise, simpler, reusable and elegant software to be written for those that have the will to learn functional programming and a little practical type theory. Importantly, Scala offers Java developers a way to move from idiomatic Java to idiomatic Scala in small steps, there is no need for a wholesale leap into the unknown. In short, Scala offers a migration path to a better way to build robust software.

A reason why Scala is important is that it offers mechanisms for controlling complexity. In The Structure and Interpretation of Computer Programs, Abelson and Sussman said:

In our study of program design, we have seen that expert programmers control the complexity of their designs with the same general techniques used by designers of all complex systems. They combine primitive elements to form compound objects, they abstract compound objects to form higher-level building blocks, and they preserve modularity by adopting appropriate large-scale views of system structure.

Scala facilitates this abstraction and combination that results in simpler, more robust software in a number of ways: functional programming techniques, object oriented techniques, a rich type system and mechanisms to build components. In Scalable Component Abstractions, Martin Odersky and Matthias Zenger describe abstractions for the construction of reusable components:

abstract type members, explicit selftypes, and modular mixin composition. Together, these abstractions enable us to transform an arbitrary assembly of static program parts with hard references between them into a system of reusable components.

Making full use of Scala does not come for free, there is a lot to learn. Fortunately, the Scala community is very friendly which is in large part due to Martin and his team’s efforts. The community is full of very smart people who are very willing to help novices. It is not unusual for questions asked on mailing lists to be answered by Martin or other leaders in the Scala community. This friendly environment is actively fostered so that beginners feel comfortable asking basic questions without fear of angry replies to RTFM.

There are those that think that Scala is a complex language and they are right to a certain extent, but you can be productive in Scala without using advanced features, just not as productive as you would be if you learnt about those features.

Most objections to new languages like Scala and Clojure can be reduced to, “I don’t understand this after 2 mins of skimming the website so its too academic”. Not very good really and not worthy of further argument.

Learning Scala has real benefits in understanding more about computing in general, and it lets you do it gently whilst being productive. Give it a try!

To be sure, this kind of automated test will run slower than unit tests, because almost all of the machinery of the system under test is going to be involved. But I have seen many implementations that are much slower than they need to be because of one or two technical choices.

The advice that I offer here does not guarantee fast and robust tests, but if you don’t follow this advice and you don’t somehow deal with the issues raised in this article, I am pretty sure your tests will be slow and time-consuming to maintain as the test suite grows.

By way of example, let us consider an online book seller. The kind of test that can be built with the aid of many libraries might well look like this:

public void testSearchAndOrder() {
    type("username", "JRHartley");
    type("password", "verysecret");
    click("Submit");

    type("searchField", "Fly Fishing Hartley");
    click("Search");

    assertEquals("Fly Fishing"), innerHtml("searchresults/tr[1]/td[1]");
    assertEquals("J.R.Hartley"), innerHtml("searchresults/tr[1]/td[2]");

    click("searchresults/tr[1]/td[1]");
    click("Order");

    assertPageTitle("Your order has been placed");
}

This will certainly test the functionality. However, there are two things wrong with it:

• Who is ‘JRHartley’? Why is he buying ‘Fly Fishing’? i.e. Is there something significant about that author or that book in the ’standard test dataset’? No? Then this test is overspecified. Yes? Then the test is dependent on something that is not stated clearly. In either case, it is dependent on a ’standard dataset’.
• Is JRHartley really interested in clicking buttons, typing into text fields, etc.? i.e. is the logic we are testing really about button clicks and typing? This test is specifying mechanisms rather than outcome.

These issues contribute to brittleness and difficulties with maintenance. In addition, the dependency on standard data frequently leads to slow tests because there is more data than required.

My general approach to these problems is to focus on abstraction and composability. The solutions shown below achieve these fundamental aspects of software development partially by using a domain specific language in fluent interface style (although the fluency is limited and I usually go considerably further). By good fortune, Debasish Ghosh describes an approach to building DSLs here that is precisely what I am advocating.

However, this choice is largely irrelevant. You could achieve the results with, for example, FIT, Fitness, Concordion or some other tool instead. I have never used those tools and suspect that I don’t really want to, but if you like them, go for it.

Dependency on a ‘Standard Dataset’

Note: Although this section refers to databases, the same argument applies if, for example, your system accepts feed files from other systems and processes them.

My preference is for each test to start with an empty database schema and to populate it with exactly what is needed. This tends to:

• prevent one test from destroying the data that another test depends on
• prevent tests from becoming dependent on an incidental aspect of the data
• prevent more and more data being added to the database in order to accomodate new tests without breaking previous tests
• make tests run quickly because there is very little data in the database

However, I do not want each test to have a bunch of database inserts, because doing so will make the test fragile with respect to database changes and in any case, doing this specifies the test at the wrong level. Therefore the database setup must be abstracted.

In order to truly achieve this, the abstraction must be above the level of database tables; for example, if tests need to know that in order to add a book to the database, a publisher must be created first, each test:

• has structural knowledge of the database leading to duplication of information
• contains information that is not relevant to the test leading to overspecified tests

In the example above, as far as the test is concerned, the book needs two attributes: a name and an author. Undoubtedly, the book will have many more attributes and relationships in the database, but none of these are relevant to this test and so should not be part of the setup. My DSL will therefore start to look like this:

public void testOrderBook() {
    given()
        .aBook()
            .title("Fly Fishing")
            .author("Hartley");

    ...
}

The given() method is an entry point into the fluent interface and this expression should be read ‘Given a book with title “Fly Fishing” and author “Hartley”‘. The important aspect of this is that the book() method initialises all of the book’s fields and knows how to build a book with integrity in the database. I usually initialise fields to random values.

This deals with the dependency on the standard data issue. However, it does not eliminate the over-specificity of the test. To put it simply, while the test is interested in the title and author, it should be non-specific about what title and author is actually used.

Overspecificity

In order to solve this problem, we need to allow the title and author to be given to us, rather than prescribed. As said earlier, my convention is to randomise any unspecified fields. So not specifying an author and title will result in what we want.

public void testOrderBook() {
    Book book = given().aBook();

    ...
}

We can now express the rest of the test using the book’s properties. We will see how to do that later.

Some might feel uncomfortable that something important has been lost from this fixture; the fact that the test does depend on an author and title and yet we are not setting up the book with a particular author and title. If that really bothers you, it is easily remedied (I’ll use the more terse form above for the rest of the article though):

public void testOrderBook() {
    String author = given().anAuthor();
    String title = given().aBookTitle();

    Book book = given()
        .aBook()
            .author(author)
            .title(title);

    ...
}

The important point is that the test is now free of pre-existing fixture data and is minimally specific.

Specify Tests In Business Terms

The other concern in the original test was that it was expressed in terms of what to type into form fields and what buttons to click. We should hide this away:

public void testOrderBook() {
    Book book = given().aBook()

    ...

    then()
        .searchingFor(book.getTitle() + " " + book.getAuthor());

    resultsIn()
        .searchResults(book);

    ...
}

The logic of how to do the search is hidden behind the searchingFor(…) method. The only relevant thing passed is the search term. Subsequently, we make sure that the results returned from the search contain the book. We could be more specific and assert that the search results contain only one book. Of course, for that to have any real relevance, more than one book would have to be created in the database.

This test now does not directly depend on button clicks and, apart from a few artifacts of the Java language, is pretty easy to understand at the level of business interactions.

Finally, we want to place the order:

public void testOrderBook() {
    Book book = given().aBook();
    User user = given().aUser();

    loginAs(user);

    then()
        .searchingFor(book.getTitle() + " " + book.getAuthor())

    resultsIn()
        .searchResults(book);

    then()
        .select(book)
        .order()

    resultsIn()
        .anOrder(book, user);
}

And once again, details about how the order is placed and how we verify that the order has been placed are abstracted away. Of course, if the user was able to check their orders on a web-page, we could use that page to determine that the order has been placed. Otherwise, we might go to the database to make sure that an entry has been added to an appropriate table. We might also expect an email to be sent, so the anOrder(Book, User) method might verify that too. The point is that all of that can be hidden in anOrder(Book, User) and can be changed over time if necessary.

Emergent Goodness

The approach described above brings enough benefits as it is; fixture data independence and tests specified at the level of the business process. However, there are now two things that also emerge:

• Because the DSL is written in fluent interface style in Java, as more tests are written, the fluent interface helps to guide the writing of tests. New tests can be composed very quickly.
• The test is not tied to an implementation. The fact that this started life as a test for a web-app does not mean that it will always be so. An entire test suite can be reused to test, for example, a REST api, or a B2B messaging system for book supplies. All that is needed is to reimplement the model behind the DSL appropriately. This may not be a small task, but the fact remains that the tests themselves can be used in many ways.

This latter point has been exercised on one of our recent projects in which there were multiple ways of using the system (web-app, REST API and message queues each offering the same functionality to clients).

Conclusion

In order to keep automated acceptance tests maintainable, fast and flexible, focus on those staples of solid development: abstraction and composability. Avoid ’standard data sets’ and express tests in terms of business processes.

To do so, we will break a recent client’s confidence and show you the whole system we built. This system was an ‘enterprise’ system in that it was bought to life within an ecosystem that contained the usual array of multiple messaging technologies, databases (legacy and new) and bizarre mandated in-house technologies that performed functions we didn’t have a need for.

The system in question is an order management system, the back end of which is written in Java and a front end is written in C#. The back end takes orders in the form of messages from multiple sources. Each source has its own message format and sometimes its own messaging technology. Examples of the different message formats and messaging technology are: the C# GUI which uses REST over HTTP to exchange XML messages, an electronic sales system that sends messages over JMS using a proprietary message format, and another electronic system that sends messages over a proprietary messaging system in FIXML.

In addition, we had to back onto a legacy database for reference data, but used our own new database for storing our specific data.

I think it’s reasonable to say that this is not a toy problem. It is exactly the kind of problem that might set an expectation of low code quality, high defect rates, etc.

Now, without further ado, I will show you the whole system. Brace yourselves:

public class MessageConsumer {
    private MessageTranslator translator;
    private OrderCommandProcessor processor;
    private Responder responder;

    public MessageConsumer(MessageTranslator translator,
                           OrderCommandProcessor processor,
                           Responder responder) {
        this.translator = translator;
        this.processor = processor;
        this.responder = responder;
    }

    public void consume(T message) {
        try {
            processor.process(translator.translate(message));
            responder.success(message);
        } catch (Exception ex) {
            responder.error(message, ex);
        }
    }
}

There it is. Nine months of work!

This class represents the entire function of the system. It accepts messages, translates them into a canonical form, processes them and informs the sender of the result.

So where is the complexity? This code is pretty clean; there’s no duplication, no excessive cyclomatic complexities, no global variables, no spooky action at a distance, etc.

What’s that? You’re saying I’m cheating. That all of the complexity must be in the MessageTranslator, OrderCommandProcessor and Responder. Yes, OK, there’s some more code there. As an irrelevant side note, there are actually multiple instances of the MessageConsumer, each configured with different MessageTranslators and Responders but the same OrderCommandProcessor. This is because, as stated earlier, there were multiple messaging technologies and message protocols in play and the abstractions represented by MessageTranslators and Responders adapt each of those to the one internal form that we want to handle.

But since you insisted, let’s look at the OrderCommandProcessor to see if we can find some of these ‘large system’ problems:

public class OrderCommandProcessor {
    private OrderCommandLogger logger;
    private OrderBook orderBook;
    private ProductCatalog productCatalog;
    private OrderValidator validator;

    public OrderCommandProcessor(OrderBook orderBook,
                                 ProductCatalog productCatalog,
                                 OrderValidator validator,
                                 OrderCommandLogger logger) {
        this.orderBook = orderBook;
        this.productCatalog = productCatalog;
        this.validator = validator;
        this.logger = logger;
    }

    public void process(OrderCommand orderCommand)
                throws ValidationException {
        orderCommand.validate(validator, productCatalog);
        orderCommand.execute(orderBook);
        logger.success(orderCommand);
    }
}

Hmmm…. That doesn’t look too bad either. One question the astute reader might ask is why we have the OrderCommandProcessor at all. Why don’t we just have those three lines of code and the necessary dependencies in MessageConsumer? The Single Responsibility Principle notwithstanding, it looks like the introduction of an unnecessary class. Right?

Well, the MessageConsumer is one of many entry points into the system. All but one entry point is a message queue of one kind of another. The C# GUI is the other one and that sends XML over HTTP. Our corresponding servlet turns the XML into OrderCommands and then dispatches to the OrderCommandProcessor for the rest. But, messages received from the GUI also required some additional processing so we couldn’t simply have a MessageTranslator and Responder suitable for HTTP messages (actually, we could have done that and then decorated the OrderCommandProcessor with the extra stuff – this would have required an extra class anyway). In any case, separating out this functionality lets us process the OrderCommands uniformly, regardless of where they come from.

But still, it’s not complicated; we have a bunch of instances of MessageConsumer and our servlet, all appropriately configured to translate messages and hand them over to the OrderCommandProcessor to do the rest. That’s all fairly straightforward, so far.

However, validation always makes things more complicated. Maybe we can find some complexity in there. OrderCommand is an interface, because there are different kinds of command (create, update, cancel, execute, book, etc). So to see the validation we’ll have to look at a particular implementation:

public class CreateOrderCommand implements OrderCommand {
    private Integer salesPersonId;
    private Integer clientId;
    private SourceSystem sourceSystem;
    private Integer productId;
    private Quantity quantity;
    private Price price;
    private Side clientSide;

    public void validate(OrderValidator validator,
                         ProductCatalog productCatalog,
                         ClientBook clientBook)
                throws ValidationException {
        validateRequiredFields(validator);
        validateClient(validator, clientBook);
        validateSalesPerson(validator);
        validateProduct(validator, productCatalog);
        validateQuantity(validator, productCatalog);
        validatePrice(validator, productCatalog);
    }

    private void validateRequiredFields(OrderValidator validator)
                 throws ValidationException {
        validator.notNull(salesPersonId, "salesPersonId");
        validator.notNull(clientId, "clientId");
        validator.notNull(sourceSystem, "sourceSystem");
        validator.notNull(productId, "productId");
        validator.notNull(quantity, "quantity");
        validator.notNull(side, "side");
    }

    private void validateClient(OrderValidator validator,
                                ClientBook clientBook)
                 throws ValidationException {
        validator.validate(clientBook.hasClient(clientId), "client does not exist");
    }

    ...
}

Oh well, not too much complexity there either.

OK. I have shown enough of this system. What is my point? Well, it all looks the same! Yes… code at each level of this system looks the same. It is very hard to tell by looking at the code that one class is ‘enterprisey’, whereas another is ‘domainy’, other than by looking at their names. Where is the ‘large-system’ code? This code all looks ’small-system’ to me. The complexity of code in each location is no more or less complex than in any other location, and it’s all pretty simple stuff.

All code, no matter what size of system it appears in, consists of the same language constructs to which is added some library usage plus whatever application specific abstractions are required. There is no difference between the code in ‘high-level’ classes and that in ‘low-level’ classes. A class doesn’t need to know that it is in a large system or a small system. The level of complexity at any point can be about the same, if you choose to distribute it appropriately.

What is true is that systems usually (but not always) become large over a longer period of time than small systems, giving them more time to accumulate bad decisions. There are however some practices that make it more likely that a system will suffer in this way. For example, ubiquitous use of language primitives to represent rich concepts invites complexity, subsequent attempts to work around the complexity, and introduction of ‘clever hacks’ to avoid disturbing too much code. These ‘clever hacks’ are often themselves the source of need for future workarounds.

The ‘large-system’ problems start as choices about the small stuff. As an example consider the Side type in the CreateOrderCommand above. This represents the notion that an order can be a ‘buy’ or a ’sell’ order. When we introduced this type, we had a choice. We could represent the concept of side with a language primitive (a char with value ‘B’ or ‘S’ maybe, a String with value ‘buy’ or ’sell’, a boolean named ‘buy’ implying a sell if its value was false, etc.) or we could introduce something whose meaning was unequivocal. We chose the latter and in so doing we aid future development because we leave little room for misunderstanding, and we also allow safe extension of the notion of side. In this system, this concept did end up being extended with two more cases and dealing with those cases at all the decision points in the code (some polymorphic, some explicit) simply became an issue of looking at the compiler errors and understanding what the business wanted in each case, rather than having to track down each case, often in response to a defect found in production.

Another example of a ‘big system problem’ is that in order to add a new feature, a new piece of data needs to be passed through a large number of interfaces before it is finally used. Sometimes, developers decide to circumvent the need to touch a large number of classes by using some shared mutable state. This is ‘chewing-gum and string’ programming. While it preserves the appearance of the interfaces, it profoundly changes its semantics, introduces non-locality, hides something essential to understanding and will most likely result in defects (even without concurrency). It certainly results in something that has to be remembered or documented. Often this problem is due to a reluctance to add new abstractions – ‘primitive obsession’ and concern about ‘proliferation of classes’; it is frequently the case that if various domain specific abstractions are being passed as parameters, then the data required for new features will fit into existing abstractions. If the new data does not fit, then something significant has happened and a team that thinks in terms of abstractions will not hesitate to make use of that new information to make their code more accepting of changes in the future.

When we maintain ‘big’ systems, we are often involved with third party components. For example, the order management system above uses messaging systems, has an embedded HTTP server, persistence layers, etc. The degree to which these components pollute the application specific code is entirely a matter of choice. These components could have been allowed to run amok through the codebase above but instead are tightly contained and as a result, the code that deals with them amounts to a few dozen lines at the most, expressed in one or two classes in each case.

Collectively, the amount of code represented by these components completely dwarfs the order management system codebase. So the vast majority of the system is code that wasn’t even written by us. But all of those components do not require us to deal with ‘big system’ problems. Why? Because most developers would rightly shy away from using a third party component if it invaded the rest of their codebase – consider the reaction of developers to using an ‘ORM’ framework that requires all persistent classes to implement a framework interface – so the implementers of those components have taken great care to make sure that their stuff can be used in simple, non-invasive ways.

Given that at any one time, a developer sees a very small fraction of most codebases and it hardly matters if that fraction is 0.01% or 0.0001%, the difference between small systems and large systems is simply that we deem a system to be large when we can no longer keep track of the idiosyncrasies. And that can happen in very small codebases indeed. These ‘large system’ problems are actually nothing to do with the size of the system, but instead entirely to do with their quality.

In Kirk’s article he states two things, amongst others, which caught my attention:

1. Over time, at least a portion of every enterprise software system experiences the problem of rotting design.
2. Fred Brooks: “All repairs tend to destroy the structure, to increase the entropy and disorder of the system. Less and less effort is spent on fixing the original design flaws; more and more is spent on fixing flaws introduced by earlier fixes. As time passes, the system becomes less and less well-ordered. Sooner or later the fixing ceases to gain any ground. Each forward step is matched by a backward one. Although in principle usable forever, the system has worn out as a base for progress.”

My initial reaction to both of these was to be both irritated and depressed, particularly with Fred’s statement. However, my irritation is not with Kirk or Fred but rather the sad fact that they are right and that most people involved in software development use it as an excuse not to do better. Its a self-perpetuating phenomenon.

I find it doubly irritating because over the last 20 years I have seen a number of systems that do not rot in the manner described. Indeed, not only do they not rot they actually become easier to work with over time and have incredibly low bug counts.

So how did those teams do it?

Trust and Respect

Mutual trust between management, developers, users and analysts is of fundamental importance. When a developer says “no we can’t just add a column to the table” then people back off and respect the expert’s opinion. That’s not to say that questions shouldn’t be asked – quite the opposite. Because we trust each other we can ask questions in a non-confrontational, exploratory environment, but ultimately the expert’s opinion must be respected.

Trust also enables people to explore and develop in new and interesting ways without fear of failure, more of which below.

People

You cannot build high quality software with poor quality people.

Writing code is easy. Writing code that is easy to maintain, extend, doesn’t rot, doesn’t crash, doesn’t require a production support team, demands highly skilled, experienced people. I am not just talking about developers here; I include management, analysts, developers and anyone else involved in the system you are building.

Process

You cannot build high quality software with a poor quality process.

I’ve never seen a waterfall project succeed. Ever. Don’t do it. Royce never intended the wholesale adoption of the waterfall process which he described as ‘high risk and invites failure’. If you don’t know who I’m talking about then you have another problem. Go and learn something about process and where it comes from and where it’s going. Find out who Deming was while you’re at it. Read about iterative and incremental methods, Agile methods, Kanban and Lean, etc.

Accepting Acceptable Failure

By acceptable failure I do not mean that the system was deployed and it didn’t do what the users asked for or it fell over in a sorry heap. That is unacceptable failure.

Acceptable failure is to explore a solution to a problem and find that the solution does not work or is not good enough. This kind of acceptable failure must be going on as part of the discovery and learning process of the team. If it isn’t then I would be concerned that the system as a whole is not as good as it could be.

Accepting this kind of failure requires a deep respect and humility between all members of the team.

Technical Practices

Test everything. Everything!!! That means production code, scripts, batches, anything that makes your system run. Test everything without fail and without compromise.

Refactor everything. Everything!!! That means production code, unit tests, acceptance tests, scripts, batches, anything that makes your system run. Refactor everything without fail and without compromise.

If you code is difficult to refactor then you have a big problem. Stop everything and fix it. Today. Now. No, now!

Define DONE. I hear “its development complete, we just need to test it and fix it” far too often and its always in a failing system. To me, DONE means ready to go to production. Nothing less. Not ready for testing, or staging, or whatever else.

The deployment process should be fully automated. By that I mean you are able to go to a command line or equivalent, type a single command or press a button and sit back whilst the system is deployed. Don’t argue with me, I’ve done it often enough to know its possible. I’ll accept that retrofitting this to a legacy system is very, very difficult, but not impossible.

The Real World

I am sure a lot of readers are now thinking, “Yeah but in the real world blah blah blah”. First, YOU created your world!!! Second, this IS my real world. I am describing my experience and the experience of many successful developers I’ve worked with in a range of industries. Don’t bother arguing with me about whether this is feasible.

No Time

I am sure some readers are thinking, “We don’t have the time for all this testing and refactoring nonsense”. Time. What is meant by ‘time’ here?

The lack of time argument usually surfaces when the development process values arbitrary targets, deadlines, measurements and, worst of all, maximum work in progress. This leads people to minimise the time taken for individual tasks today, rather than maximising the delivery of value overall. They never spend time investing in the system as a whole and, therefore, never reach the point where the costs of writing tests and refactoring is low.

In my experience, systems developed as I have described actually facilitate changes and additions rather than impede them. The code is malleable, understandable and a pleasure to work with.

The Culture

I have revisited a number of teams that have adopted good practices and have found something very remarkable. Even when the original team members have left and been replaced by new people, the culture of the original team survives and the quality of software produced is sustained.

The investment in good people, processes, and technical practices pays dividends for the life of the project with no sign of rot at all.

The design industry has a lot to say about this kind of thing and no one more so than Massimo Vignelli, who summarised his core ideas in The Vignelli Canon. I am following each part of the Canon, this part being Pragmatics.

Whatever we do, if not understood, fails to communicate and is wasted effort. – Vignelli

No matter how cleverly we design something, if it is not understood then that design has failed. Software is complex and some decisions will need explanation, but if the design is still not understood then the effort in producing it is wasted.

Complicated, confused designs are very common in our industry. They are often the result of confusion in the minds of their developers, analysts and users. A confused mind will produce a confused product. These projects usually limp along before collapsing, mercifully, under their own weight, but not after many years of very expensive and wasted effort.

We should be careful not to confuse a complex solution with an accidentally complicated one. Some systems are unavoidably complex and what we expect to find there, as in all systems, is a set of core principles that underpin the whole edifice. Without clear, strong and forceful decisions that provide structure to systems, there will be no clarity and plenty of confusion. This is not to say that those decisions are carved in stone. Constantly changing environments and requirements are a fact of life in software and so those decisions will need to be revisited and reworked. However, it is much easier to move from one set of clear and strong design decisions to another, than it is to rework a confused system.

Having clear, strong and forceful decisions is the only pragmatic choice we can make. Allowing ambiguity and confusion is not pragmatic, it is simply sloppy and is the easy way out in the short term.

Vignelli sums it beautifully:

• We like Design to be forceful
• We do not like limpy Design
• We like Design to be intellectually elegant – that means elegance of the mind, not one of manners, elegance that is the opposite of vulgarity
• We like Design to be beyond fashionable modes and temporary fads
• We like Design to be as timeless as possible
• We despise the culture of obsolescence
• We feel the moral imperative of designing things that will last for a long time

This is pragmatism in any field, and no more so than in software.

Indeed, any system with feedback can become unstable if the delay around the loop is significant. There are things that one can do to prevent instability but the best thing to do is remove the delay if possible.

In software, there are many feedback loops. When you write code, often the first feedback you get is whether it compiles or not. Can you imagine the pain if that wasn’t the case? Waiting a few weeks to find out whether the code actually compiled would be ridiculous. Right?

Fortunately, most software developers have a go at making sure their software actually compiles. Some don’t, but thats a different article.

Testing is another feedback mechanism. So how is it sane to wait months before finding out that your code actually works? It isn’t, thats crazy talk. TDD helps in a number of ways: it helps you to think about what your code will do before you think about how it will do it; it provides a safety-net for all those refactorings you will need to do; and it tells you that the code you thought you wrote is what you actually wrote.

The instant feedback obtained by TDD is a stabilising mechanism to ensure that what is written is correct and that you haven’t broken anything else inadvertently because all that other code in the system is also tested.

But what is meant by stability in this context?

The main instability is the code – wait – test – fix – repeat cycle. During the wait and test phases, new code is being built on top of the broken code, and the fixes themselves often introduce new defects. This results in increasingly painful fix phases. What is worse is that there may be many overlapping loops like this which is utterly chaotic!

This chaotic process results in developers working in a panicked, crisis mode which leaves no time for refactoring, reflection about improving the codebase, or writing tests. It’s very demoralising as the deterioration in the codebase cannot be stopped.

The end result is a brittle, sprawling mess of a system that is now not in any state to be tested or repaired without heroic effort from the developers, and buy in from business stakeholders. And we know how that story ends…

This problem occurs no less frequently when there are conversations, often by phone, intended to clarify or analyse an issue but the participants leave with varying views of what has been decided or agreed. It is more frequent between team members with different roles (developers, business analysts, users, project managers, testers, etc).

The main reason this happens is that there is a lack of large scale feedback in the conversation.

The solution is usually to summarise the conversation at the end. The key is that the person who summarises it should be the person who initiated the conversation, i.e. the person who has been trying to find out the information. This now puts the questioner in the position of knowledgeable authority and gives the other party a chance to correct any misunderstandings or supply further information that they feel has been missed.

When I am the initiator of a discussion and I believe I have acquired all the information I need, I prefer to start the summary formally by making a statement such as ‘OK. Now let me summarise of all this and you can correct anything that I’ve got wrong’. This indicates to the other participants that you believe that the interrogative part of the conversation is finished, that the roles are changing, and that I am inviting corrections and won’t be offended if they are given. It also gives the other person a chance to indicate that they do not think they’ve finished giving you all the information you need.

The key part of this, as with any good summary, is that the information is presented much more concisely than it has in the preceding conversation and without any extraneous side paths. Remember to include decisions about what must be done, what is not going to be done and what still needs some decisions and who is going to do these things. If necessary, follow up with written notes about what has been discussed.

By following this simple protocol, I have seen extraordinary improvements in the effectiveness of several teams and their working relationships.

“In the long run, men hit only what they aim at. Therefore, though they should fail immediately, they had better aim at something high.”  – Henry David Thoreau

I recently had occassion to phone Sky because my ’set-top box’ was misbehaving. While I was waiting in the queue for an operator, I was entertained by the requisite generic music, which was interrupted every 15 seconds or so in order that I may be given some useful piece of information about Sky or their services. One such interruption provided the following illumination: “Your set-top box is like a computer and so will need to be rebooted from time to time.”

I believe that this message crystalises the expectation that most people have of software; it is unreliable, mystical, needs manual support and is generally unsatisfactory.

On the other hand, Casual Miracles has recently completed an eleven month piece of work in a large investment bank. One of the reasons that we were asked to take part in this project was to ‘get it started’ since the incumbent team had been unable to do so for four months. Another reason was that the organisation has trouble producing high quality software. Business users are frequently dissatisfied with work done for them and a significant support team is usually required. Nine months after we started, the software was put into production. In the four months since that release the users have expressed pleasure with the software, no bugs have been reported and the support team have not had a single event to deal with. The only reason it has not been available continuously is that it has been taken down for new releases.

While the first story is an indication of expectations held by the general population about software, the second is a story of possibility. Keep in mind that this is not a theoretical possibility; Casual Miracles achieves these kinds of results as a matter of course. They are actually not very hard to achieve. A small fraction of developers and teams actually seem to find it easier to achieve these results than the alternative low quality stuff that normally limps over the finishing line of a software development project.

In this article I am not going to talk about the mechanics of this achievement. Instead I want to talk about one  essential precondition. That is, the expectation of quality.

As evinced by the first story above, expectation is astonishingly low with respect to the quality of software. Although projects often start with grand ambitions (the grandness of which is frequently to do with size rather than quality), this usually rapidly subordinates to getting *something* done under a mass of bad approaches to management, implementation, analysis, politics, resourcing, etc.

In order to correct this problem a line needs to be drawn in the sand. Expectations need to be raised and a clear goal set. A system that performs the functions required by the business is necessary, but usually not sufficient. There are infrequent cases for which a piece of software can be said to have attained an appropriate level of quality merely because it doesn’t crash too frequently and its bugs are suitably hidden from the user but such circumstances are few and far between and should not be the aspiration of most development efforts.

At the very least we need a sort of Hypocratic Oath of software development – ‘Above all, do no harm’. But a better expectation is that the software will do its work in ways that bring joy rather than pain to the users and other stakeholders (including those who developed it). It must do this without need for significant support staff (if the developers are nervous about supporting it, pay attention!).  Planning to rely on support staff to handle the inevitable crashes, data integrity issues and other failures is woefully inadequate. The notion of bug triage is to be deplored.

However, along with the expectation of high quality comes a responsibility. It behooves ill of stakeholders, whatever their function, to expect one thing and then behave as if other things are more important. No man can serve two masters. 

When Casual Miracles was asked to take part in the project described above, senior management specified a few things that should be achieved. First and most obvious, deliver the software in some reasonable time. Second, the project was a replacement for another system that had been in operation for almost two decades and the software that we were providing should be capable of being maintained for an equally long time. Third, help the incumbent team to adapt to our ‘new’ way of doing things if we could, but do not let this requirement cause the first two to be threatened; if it could not be achieved, let senior management know so that they could take necessary action.

These were clear, non-conflicting goals. When the third goal did prove too difficult, senior management did what they had promised, thus affirming their commitment to the first two goals. These actions helped us to deliver the result that we did.

One of the most significant distractions from a high quality software product is the notion that time is excessively constrained and that quality should be sacrificed. This apprehension is often caused and maintained by business analysts, project managers and developers, rather than users or other, more senior stakeholders. But poor quality work is future waste. The appropriate thing to do, as Deming showed so many years ago, is to find and eliminate the cause of inadequate quality. This may be achieved simply by having the stakeholders confirm that quality is the goal and making adjustments elsewhere as necessary. If change does not follow, retraining may be necessary. If there is still no change, then more radical approaches are needed.

In short, if the expectation is low, so too will be the result. If you want a better result, expect it and admonish all stakeholders to expect it also. Then act consistently with those expectations.