Growing an XP team

April 9th, 2009

I was invited to speak at Better Software 2009, a conference that will happen in Florence on May 6 and 7. I will talk about “Growing a Team”, which is what happened in my professional life in the last two years. I will talk about what happens when you decide you *really* want a team that works well. I will not say “when you decide you want an *agile* team”; that is not the goal. The goal is to have a *great* team, in the sense that James Shore explains. Of course, the best way I know to go in that direction is to do Extreme Programming.

My team started in 2007 with 3 people, and has grown to 12-13 people today. The growth has been along more dimensions other than size, of course. We’ve all learned a lot in these two years; the way we worked two years ago is not the same as how we work now.

I’m flattered that I was invited, given the company I will be in: look at the speakers! I’m very much looking forward to this conference. I hope to see you too.

No Essap this Summer

April 6th, 2009

We had three great editions from 2006 to 2008. But this year I no longer have the time to help organize the School. Vieri, who was an organizer last year, moved to Dublin. Federico is not, understandably, willing to organize without our help.

What happened? Well, what happened to me is that I’m coaching a team of 12 people, and it’s more than enough to keep me busy. On top of that, I’m teaching a brand new class at Insubria this year.

I hope I will have more time to do an Essap in 2010.

Not wishful thinking

March 17th, 2009

I read in this blog post many comments along the line of “In my experience, you don’t need to do X to succeed”, where X is any agile practice such as standup meetings or TDD. Well that’s true. In fact, you don’t need to stand upright to get from A to B. Toddlers around the age of 12 months often are quite skilled at zipping around the room on all fours. But around that age they start experimenting with standing up. At first they fall, and it takes quite a while for them to feel confident, walking on two feet. Yet, once they learn, they find they can cover much greater distances with ease.

Yesterday I was chatting with a collegue around the coffee machine. We were talking about the difference between proper unit tests, and integration tests. The view about testing that most outsiders have, is that all you need is machinery that allows you to record what your code does and play it back. Things around the lines of Selenium. My point is that the most important thing a programmer needs is to learn how to write clean, testable code. Selenium and similar tools are useful, but secondary; what a programmer needs the most is to learn how to do it right, not write a new tool.

My collegue said to me that writing clean code in the sense that I mean is wishful thinking. I told him no, just look into the code that our team is checking in… it’s not wishful thinking, it’s about learning a new level of skill. I’m on that journey.

Interesting job opportunity in Oxford

February 5th, 2009

It’s nice to hear that some companies are growing and hiring, despite the current economic troubles. I know Oege de Moor since my time as a Ph.D. student, working in mathematical techniques for program construction. Oege is an extremely smart guy and a positive person. So here is the message I got from him last week:

Semmle is taking off fast, and we urgently need more top-notch software design engineers to join. The main criterion is that they’re smart and love building great software. Skills that are immediately relevant include:

  • experts in the C and C++ language spec (for static analyses)
  • visualisation tools (like yGraph)
  • all forms of declarative programming

.. but we expect people who work with us to pick these up in their stride.

Do you know anyone suitable who might like to join Semmle as a student intern, as a consultant or as a full-time engineer? Then please let me know, or pass on this message.

Thanks,

-Oege

Results Day 2009 — Il mio contributo

January 30th, 2009

English summary: during the last Italian Agile Day the one concrete action that was proposed during the retrospective was to post what concrete successes we would be able to obtain in the next two months. This is my contribution.

Che cosa è cambiato fra il 21 novembre 2008 e il 30 gennaio 2009?

Tommaso Torti ha proposto, durante l’ultimo Agile Day, di raccontarci di lì a due mesi che risultati concreti abbiamo ottenuto nel miglioramento della nostra vita lavorativa.

Una cosa che ho fatto è stato di ridurre i miei impegni di prevendita e altro, e concentrarmi con più attenzione nelle vicende del nostro team in Sourcesense. Il team a questo punto è cresciuto a 12 persone (oggi 13), ed è stato impegnato in questo periodo su due clienti.

Read the rest of this entry »

I value simplicity

January 28th, 2009

I’ve been trying for some time to put my finger on the uneasy feeling I have when I talk with or read from good, skilled, mainstream Java programmers. I think my uneasiness comes from a fundamental, basic, philosophic difference in values. I value simplicity. Of the 5 (or 7) XP values, this is the one that for me does not even need explaining. I think “surely everybody values simplicity, don’t they?” In fact, it seems not everybody does.

It’s always difficult for me to keep my blood pressure down when we’re talking about Maven, for instance. Maven solves a difficult problem, namely the problem of mantaining a collection of Java application modules and libraries across different source code repositories and servers. As a result, Maven is extremely complex. From the complexity come the many problems that Maven has: it’s slow; it’s difficult to understand; it’s difficult to debug when things go wrong. For that matter, things often go wrong with Maven, again as a result of it being so complex; in fact, complexity leads to fragility.

Given this preamble, I think you understand why I regard Maven’s complexity as a major, show-stopping problem. But, to my unending surprise, Maven fans don’t regard this as a problem. They regard Maven’s complexity as a minor problem, if even that. A necessary evil, perhaps, which they don’t think it necessary to spend much effort to reduce.

Now Maven is just an example; a controversial one at that, since in fact many good Java programmer dislike it. But the same arguments could be made against most if not all Java major frameworks; Spring comes to mind, and Hibernate, too. They’re way more complex than necessary; they fail for reasons that require divine, or at least guru, intervention to solve.

I have different values. I value simplicity. Less parts mean less things that can go wrong, less things to understand. I like it when things work correctly from the start. I like it when the defaults allow me to use something in a basic way with no configuration at all. I like it when simple things can be done in a simple way, so that difficult things become possible.

We stand on the shoulders of giants. Me, I admire the works of the Algol patriarchs, Dijkstra, Wirth and Hoare. And I admire the works of the Unix patriarchs, Kernighan, Ritchie, Pike and McIlroy. And the works of the XP patriarchs: Beck, Jeffries, Cunningham. And the works of the Smalltalk patriarch, Kay. All of them valued simplicity.

Hoare, Wirth and Dijkstra deserted the IFIP Working Group 2.1 because they thought the specification for Algol 68 was way too complex; it ruined the beauty and simplicity of Algol 60. They produced a famous minority report about their disagreement.

Kent Beck explicitly named simplicity as one of the 4 (now 5) values of XP. The XP method is simple to explain; it does not try to write down every single detail of what I as a programmer should do. Yet it contains just enough guidance; it’s precise and clear enough.

The Unix patriarchs valued simplicity. The VMS patriarchs didn’t, and as a consequence neither did Windows NT. A clear example is the difference in the API for creating a new process. In Windows you have

BOOL WINAPI CreateProcess(
  __in_opt     LPCTSTR lpApplicationName,
  __inout_opt  LPTSTR lpCommandLine,
  __in_opt     LPSECURITY_ATTRIBUTES lpProcessAttributes,
  __in_opt     LPSECURITY_ATTRIBUTES lpThreadAttributes,
  __in         BOOL bInheritHandles,
  __in         DWORD dwCreationFlags,
  __in_opt     LPVOID lpEnvironment,
  __in_opt     LPCTSTR lpCurrentDirectory,
  __in         LPSTARTUPINFO lpStartupInfo,
  __out        LPPROCESS_INFORMATION lpProcessInformation
);

It has 10 parameters, many of which are structures with tens of other parameters. In Unix you have

pid_t fork(void);

It has zero parameters. All the things you can do with the complex and monolithic CreateProcess, in Unix you can do with smaller, simpler, modular calls.

It’s difficult to work together when we have a disagreement on fundamental values. For this reason it’s important that we recognize what values are important to us. Let’s talk about them, put them in the open, respectfully discuss about them. At least we will know what to expect of each other, so that working together in a fruitful way becomes a possibility, in spite of our differences.

What is a coach doing all the time?

January 19th, 2009

Lately I was chatting with a collegue, an experienced agile developer who is just now beginning to work as an XP coach. He said to me “It’s difficult to picture what a coach does all day — it is much simpler when I am a developer.”

I also felt that way at times. When I first spent so much time away from the code, that I was not able to provide any significant technical help anymore, it hurt. There is also a danger that the coach drifts away from doing useful work for the team. There’s so much email to be done, so many things to coordinate, so many things that beg your attention. It’s important to keep your focus on the work of the team.

So, what is it that a coach is doing all the time? I wrote down a list, so that whenever I find myself doing something not on this list, I know I may not be doing my job well.

  • Coach developers, customers and other collegues about agile values, principles and practices.
  • Write iteration reports.
  • Update and tidy up the team documentation (aka wiki gardening).
  • Mantain data about team performance: stories, effort, code metrics.
  • Train junior developers.
  • Ensure the team keeps up the practices they gave themselves.
  • Encourage developers. Invent or find new things to keep work fresh and fun.
  • Provide feedback.
  • Lead retrospectives.
  • Pay attention. Observe the process. See what the developers are doing, and how they are doing it.

Anything else?

The birthday greetings kata

January 13th, 2009

Purpose

To learn about the hexagonal architecture, which is a good way to structure an application, and how to shield your domain model from external apis and systems.

Prerequisites

This is not a basic exercise. I suppose you are already familiar with TDD and refactoring. You will need a computer with Java ≥ 1.5 and a Java IDE (I assume you will use Eclipse).

Ingredients

This kata can be done in two ways; if you want to try the refactoring way, then import the code in your IDE. If you want to try the TDD way, start with a blank Java project.

Problem: write a program that

  1. Loads a set of employee records from a flat file
  2. Sends a greetings email to all employees whose birthday is today

The flat file is a sequence of records, separated by newlines; this are the first few lines:

last_name, first_name, date_of_birth, email
Doe, John, 1982/10/08, john.doe@foobar.com
Ann, Mary, 1975/09/11, mary.ann@foobar.com

The greetings email contains the following text:

Subject: Happy birthday!

Happy birthday, dear John!

with the first name of the employee substituted for “John”

The program should be invoked by a main program like this one:

public static void main(String[] args) {
    ...
    BirthdayService birthdayService = new BirthdayService(
        employeeRepository, emailService);
    birthdayService.sendGreetings(today());
}

Note that the collaborators of the birthdayService objects are injected in it. Ideally domain code should never use the new operator. The new operator is called from outside the domain code, to set up an aggregate of objects that collaborate together.

Goals

The goal of this exercise is to come up with a solution that is

  • Testable; we should be able to test the internal application logic with no need to ever send a real email.
  • Flexible: we anticipate that the data source in the future could change from a flat file to a relational database, or perhaps a web service. We also anticipate that the email service could soon be replaced with a service that sends greetings through Facebook or some other social network.
  • Well-designed: separate clearly the business logic from the infrastructure.

An optional complication

If you want to develop further the domain logic, you can take into account the special rule for people born on a 29th of February: they should be sent greetings on the 28th of February, except in leap years, when they will get their greetings on the 29th.

Testability

A test is not a unit test if:

  • It talks to a database
  • It communicates across the network
  • It touches the file system
  • You have to do things to your environment to run it (eg, change config files, comment line)
    Tests that do this are integration tests.

Integration tests have their place; but they should be clearly marked as such, so that we can execute them separately. The reason we draw this sharp distinction is that unit tests should be

  1. Very fast; we expect to run thousands of tests per second.
  2. Reliable; we don’t want to see tests failing because of random technical problems in external systems.

One way to make code more testable is to use Dependency Injection. This means that an object should never instantiate its collaborator by calling the new operator. It should be passed its collaborators instead. When we work this way we separate classes in two kinds.

  1. Application logic classes have their collaborators passed into them in the constructor.
  2. Configuration classes build a network of objects, setting up their collaborators.
    Application logic classes contain a bunch of logic, but no calls to the new operator. Configuration classes contain a bunch of calls to the new operator, but no application logic.

The hexagonal architecture

The traditional way to structure an application in layers is

+--------------+
| presentation |
|--------------|
|    domain    |
|--------------|
| persistence  |
+--------------+

The meaning of a layer diagram is that

  1. a layer is a set of classes;
  2. a class cannot reference classes in layers above; a class can only reference other classes in the same layer, or in the lower layers.

In other words, it should be possible to compile a layer without access to the source code of the layers above it, as long as we have the source code of the layers below.

The traditional three-layers architecture has many drawbacks.

  1. It assumes that an application communicates with only two external systems, the user (through the user interface), and the database. Real applications often have more external systems to deal with than that; for instance, input could come from a messaging queue; data could come from more than one database and the file system. Other systems could be involved, such as a credit card payment service.
  2. It links domain code to the persistence layer in a way that makes external APIs pollute domain logic. References to JDBC, SQL or object-relational mapping frameworks APIs creep into the domain logic.
  3. It makes it difficult to test domaain logic without invoving the database; that is, it makes it difficult to write unit tests for the domain logic, which is where unit tests should be more useful.

The hexagonal architecture avoids these problems by treating all external systems as equally external. The system is not seen as a pipe with user interface as one end, and the database at the other. We model the system as a kernel of application code, surrounded by ports and adapters to the external systems.

Every external systems is hidden behind a facade that:

  1. Provides a simplified view of the external system, with only the operations that we need to do with it.
  2. Is expressed in terms of the domain model.

The domain model does not depend on any other layer; all other layers depend on the domain model.

+-----+-------------+----------+
| gui | file system | database |
|-----+-------------+----------+
|          domain              |
|------------------------------+

How can we make the domain independent, for instance, of the database? We should define a repository interface that returns domain objects. The interface is defined in the domain layer, and is implemented in the database layer.

The Facade-Adapter combo

The way to implement the hexagonal architecture is to abstract external systems and APIs with a Facade. A facade is a simplified view of the external system. For instance, there are a million of things that I could do with SQL and JDBC and my relational database. But my application only needs to do a few, specific things with the DB; for instance, retrieve all the employees that match some criterion. In that case, I can write a simple interface that exposes just that operation:

interface EmployeeRepository {
  List<Employee> findEmployeesBornOn(int month, int day);
}

Note that the interface is written in terms of domain objects: the Employee object belongs to the domain. A case could be made that the specification for the date (month, day) should also be a domain object; I leave that to you, the reader.

So that was the facade part. The domain logic will only deal with the facade, and can be tested thoroughly using stubbed and mocked versions of that interface. But what about the real implementation?

The code that talks with the real database (or flat file, in our case) implements the facade and correspond to the adapter pattern. The adapter can be unit tested as well, by mocking the external APIs, but this is usually not worth the effort. Mocking the JDBC APIs is in most cases very complicated. It’s perhaps more effective to just test it on a copy of the real database, or maybe with an in-memory version of the real database. There are no hard-and-fast rules here; there is a mixed bag of tricks. One thing most proponents of mock objects agree on is that you should only mock your own interfaces (see section 4.1 of Mock Roles, Not Objects).

Acknowledgment

The idea for the exercise of sending email I got from someone I overheard at an XP Day; I think it was Willem van den Ende but I’m not sure.

Thanks to the Orione team for beta-testing and commenting on this kata, in particular Marco Gulino and Roberto Albertini who performed it more than once.

References

The standard layered architecture is described in Patterns of Enterprise Application Architecture by Martin Fowler, and in Domain Driven Design by Eric Evans.

The Hexagonal architecture is due to Alistair Cockburn. Here is a video of Alistair explaining it.

My collection of resources on the Hexagonal Architecture

A pattern similar to the hexagonal architecture is the Onion architecture by Jeffrey Palermo.

Miško Hevery explains the adapter-facade combo in his post Interfacing with hard-to-test third party code.

Again Miško Hevery talks on Google Video on Dependency Injection, part of the Clean Code Talks Series.

The Dependency Injection pattern is one form of the Dependency Inversion Principle (pdf) by Bob Martin. Thanks to Jacopo for pointing out that this comment of mine was wrong.

Test Driven by Lasse Koskela contains plenty of tricks about how to do integration tests in Java.

Appendix: Useful snippets

How to convert a String to an InputStream:

new ByteArrayInputStream(string.getBytes());

Creative Commons License
This work is licensed under a Creative Commons Attribution-Share Alike 2.5 Italy License.

Update 2009/01/17: restored images, added CC license.
Update 2009/12/09: add link to my Hexagonal Architecture page
Update 2020/01/07: fix the url for Alistair Cockburn’s page, add video

Reuse is overrated

December 28th, 2008

I read today somewhere in the session notes from XP Day London 2008

Rails ControllerViews discourage re-use all the way up to the UI.

I disagree. It reflects a point of view that is wrong in a fundamental way. Rails controllers tend to be very simple. The code in the controller that responds to an action (a user clicking on a button, for instance) is usually very short, from 1 to 6 lines of code. The code being so short, there is little value in reusing it; it’s much faster to just rewrite it. Not copy-and-paste it, mind you: just rewrite it. There are plenty of ways in Rails to factor out code that is common to more methods, when you want. But the real joy of Rails is that writing the code that responds to an action is so simple, that it usually takes less than a minute to write it from scratch. So it’s true that in a well-written Rails application many controller methods will look quite similar. It’s OK, because readability is more important than refactoring every microscopic bit of duplication.

So why the concern about reuse? Well, it comes out of the idea that writing software is difficult. So when I finally get some piece of software really working, I hang on to it. It’s precious. It was so difficult to write that I’m afraid it would take at least as much time to rewrite it. And I’m afraid I would not even be able to write again correctly. And more often than not, it’s so tangled and unreadable that I’m not sure I even understand it.

It’s not wise to let fear be the main driver of our actions. Let’s revert this. Let’s start throwing away code! The second and third time I write something, it will come out different and better. Sure, writing software is difficult; like playing the piano is difficult, like handwriting, like riding a bike, like baking pizza. They are all difficult activities, and they all get better with practice. With good, proper practice.

Let’s practice rewriting code until it becomes so easy and so quick that the old concerns about “reuse” will become a quaint thing of the past!

XP Day Benelux 2008

November 24th, 2008

I attended for the third year in a row the XP Day Benelux.

Prologue

I arrived on Wednesday 19 early, so I spent the whole afternoon in my cozy hotel room, working on my presentation. I was worried that it would not be a success, and rightly so, as I will explain later.

At about 19:00 I moved downstairs to the hotel pub. Now, northern European are true masters of the art of the Drinking Tavern. The tavern in question was beautiful, comfortable, with huge old tables of solid wood. I had some great Belgian beer (Brigand). I met Olivier, an acquaintance from last year edition, a really nice guy to talk with. Later I met Pascal, Vera, Portia and some more people I didn’t know before. We had a nice dinner at the hotel.

The next day, I was still in a hurry about my slides. Had them printed at the hotel (well organized!) Then I presented my talk at the Official Half Minute Presentation (ohmp), where presenters have a half minute each to entice people to come to their session.

Responsibility-driven Design with Mocking

I attended first Marc, Willem and Rob‘s presentation on interaction-based testing. I was a bit disappointed that there was little interaction in this session; I was hoping for some hands-on training, because I always end up with some new insight then. It was basically a demo of the ideas in the excellent Mock Roles, Not Objects paper. They concluded with a list of potential problems in interaction-based testing.

One interesting opinion came from a coach I don’t know the name of, who said that he likes to use mocks heavily as a design tool, then he usually throws away the tests and write new, state-based tests, because they are more readable and less brittle. I have no idea if it’s a sensible thing. I’m still not sold on the mock everything thing (I *love* my value objects.)

Agile Metrics

Next I attended Agile Metrics, by Dave Nicolette. This also was not hands-on, except that Dave kept the audience engaged by asking us questions. It was a nice recap of the agile thinking with regards to metrics. I always hear that you should measure the outcome of actions chosen in the retrospective; only I don’t do that in practice. I guess it’s one of those things you should just start doing; there’s no secret ingredient. One nice thing about Dave’s presentation was that the use of metrics was linked to principles from the Agile manifesto. For instance, the velocity and burndown metrics are of interest because “Working software is the primary measure of progress”.

Mirror, mirror on the wall…Why me?

Next the nice session by Portia and Pascal. It was mostly about thinking about the people around you, about their strengths and weaknesses (the magic mirror), and what our perception of these strengths and weaknesses tells us about what we fear and what we value (the mirror again). And to decide one simple action that will improve our relationship with these human beings. I got some food for thought, like how different personalities contribute to teamwork. They concluded by revealing the Two Secrets of Teamwork, which, being secrets, I will not divulge :-)

The Invariant Game

And we get to my session. I’m afraid the material I tried to present was too difficult. As usual when you’re beginning to teach some material, you tend to try to cover too much ground. This was an interactive session, but the exercises were a bit too tough for the audience (except for one). If there is a single thing that I think was a success, is the suggestion that there’s a vast space of learning that most developers, agile or not, are not aware of. I don’t think I succeeded in giving the participants a useful tool to bring home.

Invariant Game Slides

Invariant Game Problembook

I still want to make progress in applying math to make programming more effective. Someday I will find out how.

Conclusion

Olivier organized a demonstration of Aikido with his master, a kindly looking man who is probably quite deadly with his hands. It was nice to relase the tension and do something with my body.

We had a fine dinner; I had a nice chat with my friend Philippe. Then I took the train to Amsterdam with my Sourcesense collegues Marijn and Amelia, and another participant. Amsterdam by night is beautiful: the clean air over the dark water in the channels; the Christmas lights decorating the arches of the bridges; the Dutch houses with large windows without curtains, so that you see nice paintings and bookshelves from the street. I’m sad I could not stay the whole two days. I’m puzzled why my session was not so useful. I’m happy I was breathing again the energetic atmosphere of the Benelux Agile community.