The empirical tradition (the book calls it hermeneutics, but I will stick to this less formal name :-) values experimentation. Empiricists hold that the meaning of a document is a shared, temporary convention between the author and the readers. Think “user story”; think CRC cards; think “quick design session on the whiteboard.”

The empiriricists brought us Lisp; the formalists brought us Haskell. The formalists brought us Algol, Pascal, Ada. The empiricists brought us C, Perl, Smalltalk.

Empiricists like to explain things with anthropomorphism: “this object knows this and wants to talk to that other object…” The formalists detest anthropomorphism; see these quotes from Dijkstra.

As a former minor student of the best formalist tradition there is, and a current student of the Object Thinking tradition, I think I’m qualified to comment. Please don’t take my notes as meaning that the formalist tradition sucks; I certainly don’t think this. I’m interested in highlighting differences. I think a good developer should learn from both schools.

Formalists aim to bring clarity of thought by leveraging mathematical thinking.

Object thinking aims to bring clarity of thought by leveraging spatial reasoning, metaphor, intuition, and other modes of thinking.

It is well known that mathematical thinking is powerful. It’s also more difficult to learn and use. One example that was a favourite of Dijkstra is the problem of covering a chessboard with dominoes when the opposite corners of the chessboards were removed. If we try to prove that it’s impossible by “trying” to do it or simulating it, we’d quickly get bogged down. On the other hand, there’s a very simple and nice proof that shows that it’s impossible. Once you get that idea, you have power :-)

An even more striking example is in this note from Dijkstra on the proof method called “pigeonhole principle”. Dijkstra finds that the name “pigeon-hole principle” is unfortunate, as is the idea to imagine “holes” and a process of filling them with “pigeons” until you find that some pigeon has no hole. The process is vivid and easy to understand; yet it is limiting. Dijkstra shows in this note how to define the principle in a more simple and powerful way:

For a non-empty, finite bag of numbers, the maximum value is at least the average value.

This formulation is simple (but not easy!) Armed with this formulation, Dijkstra explains how he used this principle to solve on the spot a combinatorial problem about Totocalcio that a collegue of his could not solve with pen and paper. He also explains how he used it to solve a generalization of the problem, which would not be easy to prove with the “object-oriented” version of the principle.

I think this note presents the contrast between formalism and empiricism vividly. If you put in the effort to internalize the formal tool, that which was difficult becomes easy, and you can solve a whole new level of problems.

On the other hand, the formalists do now always win :-) Formalists reject the idea of making tests the cornerstone of software development. In my opinion they are squarely wrong; examples are the primary tools to do software development, and you can’t even understand if a specification is correct until you *test* it with examples.

The one thing that boths camps have in common is that they are both minority arts. Real OOP is almost as rare as Dijkstra-style program derivation. The common industrial practice is whateverism :-)

A design is simple when

1. Runs all the tests.
2. Contains no duplication
3. Expresses all the ideas you want to express.
4. Minimizes classes and methods

in this order.

Rule 2 is important, as it pushes us to invent abstractions that capture recurring patterns. But rule 3 is also imporant, as it pushes us to invent abstractions that correspond to the ideas that we want to express.

The other day I saw this post by Luca about a fun kata: implementing the scoring rules for a dice game called “Greed”. This exercise is part of the Ruby Koans, but its use as a programming exercise dates at least from the OOPSLA ’89 conference, when Tom Love proposed a contest to show how a program could be written in different ways and in different languages.

A little research shows many solutions for this problem. As this problem is presented in the context of a Ruby programming exercise, people usually tries clever tricks that exploit peculiar Ruby idioms. For instance:

def score(dice)
  (1..6).collect do |roll|
    roll_count = dice.count(roll)
    case roll
      when 1 : 1000 * (roll_count / 3) + 100 * (roll_count % 3)
      when 5 : 500 * (roll_count / 3) + 50 * (roll_count % 3)
      else 100 * roll * (roll_count / 3)
    end
  end.reduce(0) {|sum, n| sum + n}
end

http://stackoverflow.com/a/6742129/164802

There’s a place for this sort of exercises, but it’s not the sort of programming that I would like my collegues to practice! If we apply the rule 3, I expect to see in the source cose some mention of the *rules* of the game. I expect that there’s a programming element that corresponds to the rule that “three ones are worth 1000 points”, etc. Really, it does not take all that much more effort, and I assert that it’s more fun to code expressively!

This is my solution:

class Array
  def occurrences_of(match)
    self.select{ |number| match == number }.size
  end

  def delete_one(match)
    for i in (0..size)
      if match == self[i]
        self.delete_at(i)
        return
      end
    end
  end
end

def single_die_rule(match, score, dice)
  dice.occurrences_of(match) * score
end

def triple_rule(match, score, dice)
  return 0 if dice.occurrences_of(match) < 3
  3.times { dice.delete_one match }
  score
end

def score(dice)
  triple_rule(1, 1000, dice) +
  triple_rule(2, 200, dice) +
  triple_rule(3, 300, dice) +
  triple_rule(4, 400, dice) +
  triple_rule(5, 500, dice) +
  triple_rule(6, 600, dice) +
  single_die_rule(1, 100, dice) +
  single_die_rule(5, 50, dice)
end

There's some more duplication that could be removed (the five similar rules could be expressed as a single rule) and the names could be improved, but I think this is the way to go. Make your code look like a model of the problem!

A similar thing happens in the code base of our current project. Sometimes I see a class that strikes me as odd. Perhaps it has a name that does not communicate; more often it is simply a class that should not exist.

One such example is the UserAuthenticationContainer. WTF?!? you are saying? That’s what I also thought when I saw it. It turns out that a UserAuthenticationContainer contains a hashed password and a User.

class UserAuthenticationContainer {
    public final User user;
    public final String password;

    public UserAuthenticationContainer(User userFrom, String password) {
        this.user = userFrom;
        this.password = password;
    }
}

It was used to authenticate the user:

public User authenticate(String username, String password) {
    try {
        UserAuthenticationContainer userAuthentication =
          registry.getUserAuthenticationContainer(username);
        if (!encode(password).equals(userAuthentication.password)) {
            throw new WrongPasswordException(username);
        }
        return userAuthentication.user;
    } catch (RecordNotFoundException e) {
        throw new UserNotFoundException(username);
    }
}

This is unnecessary. This class existed only for the purpose of returning two items from a call. We got rid of it by letting User contain its hashedPassword. Furthermore, we moved the job of matching the user input with the hashed password inside the User object itself. This is how the authentication code looks now:

public User authenticate(String username, String password) {
    try {
        User user = registry.findByName(username);
        if (!user.matchesPassword(password)) {
            throw new WrongPasswordException(username);
        }
        return user;
    } catch (RecordNotFoundException e) {
        throw new UserNotFoundException(username);
    }
}

The matchesPassword method in turn delegates the matching to its PasswordDataItem.

public boolean matchesPassword(String userInput) {
  return this.password.matches(userInput);
}

Another example is this FilterByUserShopsCriteria:

public class FilterByUserShopsCriteria implements Criteria {
    private final Criteria criteria;

    public FilterByUserShopsCriteria(User user, Criteria criteria) {
        Collection<String> shopIds
          = transform(user.getShops(), toShopIds());
        this.criteria = and(criteria, in("ShopID", shopIds));
    }

    @Override
    public String condition() {
        return criteria.condition();
    }

    @Override
    public Object[] values() {
        return criteria.values();
    }
}

Its purpose is to return a Criteria (it represent a Sql “where” condition) that matches the Shops that belong to a user. This class was made unnecessary by moving the generation of that Criteria in the User class itself.

I think that this sort of classes happens when we detach “object-oriented” from “modeling”. An object should be a part of a model of the problem domain. It might also be a “technical” class, like an HtmlForm; this is not part of the core domain of the application, but still has a clearly defined role in the system. When we keep applying OOP in a “syntactical” way, we risk creating objects that may be decoupled, but are not cohesive. Functionality is dispersed far away from the center of responsibility.

The “semantic” clue is that a FilterByUserShopsCriteria is not a natural part of the problem domain. See also this post by Carlo Pescio about how the names we choose for classes have a deep influence on design.

When I read Object Thinking, I was intrigued by this quote by Grady Booch:

Let there be no doubt that object-oriented design is fundamentally different from traditional structured design approaches: it requires a different way of thinking about decomposition, and it produces software architectures that are largely outside the realm of the structured design culture.

So it seems that OOD decomposes a problem in a way that is essentially different from what you would arrive at with other design methods. I was intrigued: I wonder what are these different, elusive ways of decomposing things.

One important hint came from reading this highly recommended paper by Alan Kay, also quoted by Carlo Pescio in a very interesting post:

“The basic principle of recursive design is to make the parts have the same power as the whole.” For the first time I thought of the whole as the entire computer and wondered why anyone would want to divide it up into weaker things called data structures and procedures. Why not divide it up into little computers, as time sharing was starting to? But not in dozens. Why not thousands of them, each simulating a useful structure?

So one main idea is to recurse. When you break up a system in data structures and procedures, what you get are pieces that are less powerful than the whole. A data structure has no behaviour, a procedure has no state. Computers have both state and behaviour, and so do objects. Let’s see how this idea applies to a common programming task: a web-based registration form.

Tedium and repetition

Suppose we must implement a registration form, with the usual abundance of of data: first name, last name, email, username, password, etc. The common way to do this in Java is more or less along these lines:

<form>
  <label for="firstName">First Name</label>
  <input id="firstName" name="firstName" value="${firstName}" />
  <label for="lastName">Last Name</label>
  <input id="lastName" name="lastName" value="${lastName}" />
  <label for="email">Email</label>
  <input id="email" name="email" value="${email}" />
  <!-- and so on and so forth for 100 other fields -->
</form>

class RegistrationRequest {
  private String firstName;
  private String lastName;
  private String email;
  // and so on and so forth ...

  public void setFirstName(String firstName) {
    this.firstName = firstName;
  }

  public String getFirstName() {
    return firstName;
  }
  // and so on and so forth ...

  public ErrorList validate() {
    ErrorList errors = new ErrorList();
    if (isEmpty(firstName)) {
      errors.add("First Name is required");
    }
    if (isEmpty(lastName)) {
      errors.add("Last Name is required");
    }
    // and so on and so forth ....
    return errors;
  }
}

class RegistrationRequestController {
  public void handleRequest(
    HttpServletRequest request, HttpServletRequest response
  ) {
    RegistrationRequest rr = new RegistrationRequest();
    rr.setFirstName(request.getParameter("firstName"));
    rr.setLastName(request.getParameter("lastName"));
    rr.setEmail(request.getParameter("email"));
    // etc etc ...
    if (rr.isEmpty()) {
      registrationDao.save(rr);
      response.redirectTo(THANKYOU_PAGE);
    } else {
      registrationView.setErrors(errors);
      response.getWriter().write(registrationView.toHtml());
    }
  }
}

class RegistrationDao {

  public void save(RegistrationRequest rr) {
    String sql = "insert into registration_requests" +
      " (firstName, lastName, email, ...) " + // etc.
      " values (?,?,?,....)"; // etc.
    PreparedStatement statement = connection.prepareStatement(sql);
    statement.setObject(1, rr.getFirstName());
    statement.setObject(2, rr.getLastName());
    // ... arrrgh!
    statement.execute();
  }
}

And so on and so forth. The list of the attributes: firstName, lastName, email… is repeated dozens of times in the source code. Talk about “remove duplication”! Rails has an approach to solve this problem. Rails’ ActiveRecord objects turn down the number of repetitions considerably, but not in a completely satisfactory way. Hibernate removes some of this tedium, but not all of it and definitely not in a satisfactory way. At least for me.

So is there a way to code this very simple and very common scenario without killing your soul with the tedium of repeating the same list of attributes over and over? Is there a programming technique that can save us without having to resort to the mysterious and treacherous magicks of Rails, Spring or Hibernate?

I hinted at such a way in my CodeMotion presentation in 2011. One simple way to get started is to stop treating the RegistrationRequest attributes as code and just let them be what they are: data. If we substitute the RegistrationRequest class with a simple HashMap we can remove many of the repetitions. When I showed slide 22 to the audience, they were surprised :-) Noted blogger and DDD authority Alberto Brandolini was so surprised that he tweeted about it. Many would say that this is a dirty trick, or the mark of an inexperienced programmer. But a Rails’ ActiveRecord object is not too dissimilar to a Map. In fact the heart of a Rails’ ActiveRecord object is a Hash of attributes. Using Maps this way is natural for people coming from PHP programming, like me and like David Heinemeier Hansson.

More inspiration

The HashMap kinda works, but even I admit it’s not the most elegant solution. The next flash of inspiration came from the Object Thinking book, where he presents the example of a mortgage application form. The form, he says, is a collection of data items.

Another flash of insight from Object Thinking: holding information is a responsibility. We budding OO programmers are constantly being warned against the anemic domain model. “Don’t confuse a data structure with an object! Stay away from objects without behaviour!” But Object Thinking says that some objects have as their primary responsibility the one of holding information. This is liberating :-) There are more responsibilities that are closely related with the one of holding information: presenting and validating the information. Now this is an interesting set of responsibilities.

If the application as a whole has a user interface, behaviour and persistence, then the principle of recursive design should imply that most objects should also have user interfaces, behaviour and persistence. Persistence itself is a kind of user interface, where the user is a database. So the registration form should have a user interface, behaviour and persistence. And since the registration form is essentially a collection of data items, its user interface should be composed of the user interfaces of its data items; the form’s behaviour should be a composition of the behaviours of its data items.

All this leads to the conclusion that storing a first name in a String is folly. A String cannot defend itself! It cannot encapsulate its state, it cannot validate itself, it cannot persist itself. Suppose we rewrote the RegistrationRequest like this:

class RegistrationRequest extends CompositeDataItem {
  private PrimaryKeyDataItem requestId =
    new PrimaryKeyDataItem("requestId");

  private StringDataItem firstName =
    new StringDataItem("First Name", "firstName");

  private StringDataItem lastName =
    new StringDataItem("Last Name", "lastName");

  private StringDataItem email =
    new StringDataItem("Email", "email") {{
      add(new EmailValidationRule());
    }}
  // ...
}

Now this list of data items could be the one and only representation in the system of the list of attributes of the RegistrationRequest. A lot of duplication disappears. For instance, how do we load the HttpServletRequest parameters into the RegistrationRequest? Easy, by delegating the work to the DataItems. Every DataItem knows its name, so it can find the parameter(s) it needs.

class StringDataItem extends DataItem {
  // ...
  void loadFrom(HttpServletRequest request) {
    this.value = request.getParameter(this.name);
  }
  // ...
}

The RegistrationRequest loads itself from the parameters by delegating to its data items:

class RegistrationRequest extends CompositeDataItem {
  // ...
  void loadFrom(HttpServletRequest request) {
    for (DataItem item: dataItems()) {
      item.loadFrom(request);
    }
  }
  // ...
}

If I give an HtmlForm to the RegistrationRequest, then it can add its fields to the form:

class RegistrationRequest extends CompositeDataItem {
  // ...
  void presentYourselfOn(HtmlForm form) {
    for (DataItem item: dataItems()) {
      item.presentYourselfOn(form);
    }
  }
  // ...
}

Every Data Item knows how to present itself on a form:

class StringDataItem extends DataItem {
  // ...
  void presentYourselfOn(HtmlForm form) {
    form.addTextField(this.label, this.name, this.value);
  }
  // ...
}

Does this violate the principle of separating the domain from the user interface? Not really. We’re not concatenating strings to produce html here. We’re interacting with a view object.

Validation will also be handled recursively.

All of the various responsibilities that we’re giving the DataItems can make them increasingly complex. But we can deal with this complexity with delegation. For instance, validation rules could be separate objects that are added to the DataItems, like the EmailValidationRule in the example above.

Conclusion

So we’ve arrived to something that is quite different from what you would get with procedural thinking. Procedural thinking leads you to use a String to represent a first name. This may lead you to use Lombok to generate the setters and getters automatically, and then Hibernate to generate the SQL statements automatically, and so on. An apparently simple and obvious choice leads to a lot of complexity…

The non-entirely-obvious choice of storing the attributes in DataItems leads to a simple way to implement all of the basic functionality of a Crud, with little or no need for frameworks. This helps to keep accidental complexity low.

The approach shown here is not a framework; not even a library. it’s just an idea. You can implement it yourself in very little time, and it will probably come out different every time. That’s right; the DataItem you need in one project is not necessarily the same that you need in another.

I’ve been reading Object Thinking. It’s an unusual book, in that it talks mainly about the philosophy of OOP rather than technicalities like Dependency Injection. There are many gems in this book. One of the main things I got out of it is that Object Thinking is a revolutionary break with respect to established software engineering practice. Another thing is that the view of objects as Abstract Data Type is basically the establishment’s way to dilute and incorporate the grand new thing and make it look like a variation of the old thing.

Again, one thing that I changed my mind about is to take seriously the idea of modeling the domain. I mean, the old OOP books all said things like “take the description of your problem, underline the nouns in the description, and these are your candidate objects”. I used to think that this is a silly exercise! And I’m not alone. I heard many times the phrase: “objects are good, but not for the domain.”

In the last few years since I attended Francesco Cirillo‘s workshop on Emergent Design in 2009, I have been studying OOP. I learned about the GOF patterns. I learned about the SOLID principles. I learned how to write testable code. I learned how to do TDD for the infrastructure. I learned how to test-drive Chicago style and London style. I invented exercises for teaching the Hexagonal Architecture and the Open/Closed principle.

All of these things are good, and very useful to learn. Yet they are not the heart of Object Thinking. All these things are based on the shape of programs. For instance: I write a Long Method (bad), then I Extract Method a few times, then I Extract Class, and voila, I have done an Object Decomposition. But this is a mechanical exercise, that takes into account mostly the shape of the methods and not their meaning. If I’m lucky, my decomposed class will turn out to be something significant in terms of the domain. If I’m less lucky, I’ll have extracted some boring bit of infrastructure, like the IndoorSessionInitializer.

What I was missing on is decomposition based on Object Thinking. Decomposition based on writing the Domain Model. Thinking about the Domain Model as a simulation of the problem domain, as if the domain objects were little computer people sending messages to each other. Ding!

The incredible compressing brain of Kent Beck

Kent Beck’s brain contains probably the most efficient compression algorithm ever. Kent is able to compress in a couple of lines what others do take volumes to explain. Take Kent Beck’s four rules of simple design:

The right design for the software at any given time is the one that

1. Runs all the tests.
2. Has no duplicated logic. [...]
3. States every intention important to the programmers.
4. Has the fewest possible classes and methods.

Kent Beck, Extreme Programming Explained

Consider item 2. It basically means: look at the shape of your code, and wherever you see the same information encoded twice, find a way to generalize or abstract so that it is encoded just once. In a few words he is implying the contents of the Refactoring book, of SOLID, and all approaches that look at the shape of code for guidance on how to improve its design.

Then consider item 3. It basically means: look at the meaning of your code, and make sure that the meaning is apparent in the names and the structure. This implies that you should make your code a model of the problem domain.

(Items 1 and 4 are just safeguards against making refactorings that break functionality, or going wild with OOD ideas and making the design worse by making it overcomplicated.)

Am I crazy or…

As an aside, I find myself buying lots of out-of-print books. It seems the things that I find most interesting are not what the world of programming in general finds interesting. Even my beloved Extreme Programming seems out of fashion lately. Nobody wrote about XP since Kent Beck wrote the last of the Coloured Books in 2004. But I still think that OOP and evolutionary design and XP are crucial things to learn. Yet everyone wants to learn about the latest technology fad. The latest technology fad I’m getting into is SmallTalk, and it’s something that stopped being developed around 1980. This means that either I’m crazy, or everyone else is.

In conclusion

When I was young I really believed in objects. My first falling in love with objects was over the cover of an issue of Byte Magazine dedicated to OOP.

I remember now that in nineteen-eighty-something I wrote an object-oriented program for an AI assignment. It was a program for symbolic integration that took inspiration from some crazy idea by Douglas Hofstadter. It was a community of software agents, each one specialized in some symbolic solution technique, that worked at the problem without any central coordination. It was written in some variant of Lisp that had no Object-Oriented extensions; so I wrote my own OOP system on top of Lisp. It was fun :-) and it was not difficult at all to do.

Then, in the nineties, I got enamored of formalist thinking; not the brand that did UML, but the brand that did Formal Methods. By then, OOP seemed such a scruffy and ineffective thing to me. But by that time, when I thought of OOP I was thinking of C++; I was thinking of large, unwieldy frameworks based on very long inheritance chains. Small wonder that I didn’t like it.

In the eighties, when I thought of objects, I thought of letting artificial intelligence emerge out of the collaboration of many simple autonomous agents. What a silly and romantic idea! Yet this idea is at the heart of OOP as was originally imagined by Alan Kay and the others.

This is probably a good place to comment on the difference between what we thought of as OOP-style and the superficial encapsulation called “abstract data types” that was just starting to be investigated in academic circles. … To put it mildly, we were quite amazed at this, since to us, what Simula had whispered was something much stronger than simply reimplementing a weak and ad hoc idea. What I got from Simula was that you could now replace bindings and assignment with goals. … the objects should be presented as sites of higher level behaviors more appropriate for use as dynamic components.
Alan Kay, The Early History of SmallTalk

Uberto showed us some production code, that he thought was a neat application of FP in a OOP context. The funny thing is that to my eyes it seemed that was neither FP nor OOP; it seemed procedural to me. Uberto argued that it was functional and not procedural, because there was no global state involved; it was a self-contained transformation of data.

That got me thinking. Do I like that definition of “procedural”? I don’t think so. I think that you can tell procedural code when the thinking that goes behind that code is something like:

First I do this.
Then I do that.
If Zot Then I do Blah.
Else I do Blorgh.

When you think of a recipe, or a sequence of steps, then for me it’s procedural. Essentially “procedural” is to model the solution as a process.

On the other hand, functional programming, in my humble opinion, is about mathematical models. You model the problem and the solution with functions, sets, maps, trees and all those abstract and precise mathematical concepts.

What about Object-Oriented Programming then? My thinking about OOP has been in the past that OOP is a natural consequence of writing modular programs. An object is a hidden data structure with a visible collection of operations. Then I understood that this is just one way to look at OOP, and probably it’s the wrong one :-) This point of view is heavily influenced by mathematics (the idea that the operation on an object define a sort of algebra) and by procedural thinking (the idea that programs = data + procedures, just like the title of an old favourite textbook.)

I learned from reading GOOS that OOP is mostly about the messages. There are some intesting and tantalizing quotes from Alan Kay about that. Even more recently I read Object Thinking, and I came to understand that OOP is about modeling the problem with a community of autonomous agents. The way to make OOP shine is to build a simulation of the problem. And I remembered the thrill I once had when I first read about OOP. About imagining these little software robots going about their business inside my programs. An object that can “think” for itself and has a behaviour. Now *that* is what OOP is about!

Back to the original theme, now was the code that Uberto showed to us OOP, FP or procedural? It turns out that it depends! You can’t say by just looking at the code. It depends on the thinking that goes behind the code. Uberto thinks functionally when he thinks about this code. That makes it functional code for him.

Suppose that years later I inherit his codebase. If I have no access to Uberto, it’s pretty difficult for me to reconstruct his mental process. I will probably treat the code as procedural; or maybe OO depending on my way of thinking about this code.

So this is a pretty shocking thought, isn’t it? I thought you could look at a program and tell its paradigm, functional or procedural or OO, just by looking at the code. But I now think you can’t; the three paradigms are not different ways to code; they are different ways of thinking! Procedural thinking is about recipes: follow these steps. Functional thinking is about math: functions and sets. Object-Oriented is about simulations: many independent agents. And this explains why it’s difficult to do OO well. You have to learn a different way of thinking. This is something that I heard Francesco Cirillo say many times. Now I understand it a bit better :-)

Quindi vi do la mia personale interpretazione, senza pretendere di parlare per Francesco. Fatto: se proviamo a leggere un qualsiasi libro in tema Metodi Agili, si dà per assodato che usando i Metodi Agili si diventa più bravi.

La realtà dei fatti, che ho personalmente riscontrato nelle mie esperienze, è che spesso questo non è vero. Diversi team agili di mia conoscenza hanno fallito progetti, o comunque hanno conseguito una fama di essere troppo cari. Fama immeritata? Che importa! Se il risultato finale è che il cliente non ti sceglie più hai fallito. E poi è facile che la fama non sia poi così immeritata.

E’ vero che molte pratiche agili producono un beneficio immediato; quel problema che facevi fatica a debuggare si risolve brillantemente con gli unit test, la comunicazione con il cliente migliora se andiamo a chiedergli che cosa pensa veramente, ecc. ecc. Ma c’è un grosso MA. Ci vuole molta, molta fatica per rendere questi benefici permanenti. E’ sulla distanza che si vede la differenza.

Tanti che credono di programmare a oggetti scrivono invece codice procedurale, e la differenza sulla distanza si traduce in codice ingarbugliato. Tanti che si rifiutano di fare big design upfront, non hanno capito che devono fare invece tanto design in maniera continua. E magari non sarebbero in grado di farlo neanche se volessero, il design upfront, né big né small. E poi, ci vuole tanto coraggio per continuare a mantenere un vero contatto con il cliente nel lungo periodo.

Kent Beck ha detto “What does it mean to be agile? My definition is that you accept input from reality and you respond to it”. Allora facciamola per davvero questa follia di accettare l’input dalla realtà! Misuriamo quanto ci costa sviluppare. E’ da qui che si vede se quello che facciamo funziona veramente.

E già che ci siamo, da quanto tempo non andiamo dal cliente a chiedergli “sei contento di come lavoriamo?”

Update 2011/12/11: updated link to HD video

Sessions are the Achilles heel of every application server.
Michael T. Nygard, Trampled By Your Own Customers (pdf)

Earlier this year I started a big Java development project with my team. I did mostly Rails development for a long time, so when I came back to Web applications in Java I had some expectations. I expected that Tomcat would treat user sessions in much the same way as Rails does; but it doesn’t.

A bit of background. In a RESTful web application, all session data is maintained in the memory of clients, and nothing at all is saved in server-side sessions. In a well-written web application you might want to save just a few bytes on the server; ideally you should save no more than the id of the authenticated user. On the other hand, poorly-written application and frameworks tend to save lots of data in server-side sessions.

In a Rails application, there are essentially three ways to implement session data:

a) in encrypted cookies, which are entirely client-side;

b) in server-side files;

c) in a database table.

In a single-server application you might want to use a) for maximum efficiency, or b) to avoid the risk of someone tampering the cookies (not easy to do; it is necessary to steal the encryption key.) In a multi-server application you might choose a) for maximum efficiency, or c) for added security.

Now let me clarify this: servers-side sessions in a database table are extremely efficient. Most web applications make dozens of database queries per page view; adding one or two simple queries for getting the session data will *not* add a significant overhead.

This was my point of view when I came from Rails to Java again. Now we are writing an application that will run on Tomcat, and I was expecting that Tomcat, being an established product, would offer similar functionality; but it does not.

Tomcat options

When it comes to session management, Tomcat offers three options:

i) The standard session manager, that keeps sessions in the Java heap.

ii) The “persistent” session manager, that keeps sessions in the Java heap and and might swap “inactive” sessions on file or on a database.

iii) The “clustering” session manager, that keeps sessions in the Java heap and keeps them synchronized with other instances of Tomcat using a custom protocol over TCP. In order to cater for the needs of poorly-written applications that save lots of data in sessions, this manager has a sophisticated algorithm for moving “deltas” of session data instead of moving full session contents.

At first I thought that I could use option ii) to replicate the functionality of Rails option c). But it turns out that it’s not possible; the “persistent” session manager insists that sessions are saved to disk in batches, not individually. This means that there is no guarantee that users will not lose their session, if they click quickly from a page to another.

I briefly considered option iii), but it requires to open new TCP ports. For my application, this would require complex firewall adjustments, since our servers are located in different geographic locations. I would feel like a stupid to ask people in Operations to open this many holes in their firewalls. And I would not bet a dime on the security implications. Besides, the whole concept seems a hugely complicated way to do stuff that is best left to specialized services like memcached. The fact that the whole thing is optimized for poorly-written applications makes me like it even less.

In addition, all three options store session data in the Java heap. This is an incredibly bad idea. Memory is the most important resource for a Java application. You don’t want to waste Java heap memory for session data that will be used at best once every few minutes, and at worst will never be needed again. Doing this means that memory usage will grow linearly with the number of sessions, and most of that memory will be unused for most of the time. As a consequence, we will be forced to set a maximum number of concurrent users.

A RESTful application, on the other hand, will not waste memory on sessions, so its memory consumption will depend on the number of concurrent *accesses*, not sessions. The application only allocates the memory it needs to fulfill the requests it is currently serving, and no more. As a result, a RESTful application can handle many more concurrent requests, all other things being equal.

I can imagine that the reasoning behind the Tomcat session managers is that keeping sessions in memory is “more efficient”. For most common applications, this reasoning is wrong. For a scalable (and robust) application, the efficiency that matters most is memory efficiency; adding a few milliseconds of latency per request does not matter. Tomcat reminds me of what this very good article calls “1975 programming”; adding complications when the simpler algorithm would be much better.

Conclusion

In conclusion, how do I think I will solve the problem with Tomcat? I could write my own session manager; but that requires to interact with complex interfaces that I’m not confident to be able to understand correctly in a short time. I could use one of the several alternative session managers that are based on alternative stores; but I don’t really need to save milliseconds on latency, and I don’t want to install new services, and what’s more important I don’t know how reliable these things are.

The solution I decided to use is to roll my own session management in the application. I will apply to all requests a filter that does the following:

1. Look for a session cookie. If it’s missing, redirect to the authentication page; otherwise find the session id in the cookie and use it to fetch session data from the database.
2. If the session data is not found or if it’s expired, then redirect to the authentication page.
3. Otherwise find the id of the authenticated user in the session.

The authentication page asks the user for credentials; if they are correct, then it

1. Generates a session id using a secure random number generator;
2. save the session id in a cookie;
3. save the user id in the database.

It’s simple and very efficient. There’s not much more needed. This scheme has a nice property: it only creates a session for users who supplied valid credentials. This avoids the problem of generating lots of useless sessions for requests coming from search engines. It’s also more secure, because it makes it much more difficult for a sophisticated attacker to guess existing session numbers. This is better than what most application servers do :-)

Another nice property is that session data never changes. The only thing we store in it is the user id, and there is no way or reason for it to change.

When the design of the THE Multiprogramming System neared its completion the University’s EL X8 was getting installed, but it had not been paid yet, and we hardly had access to it because the manufacturer had to put it to the disposal of an American software house that was supposed to write a COBOL implementation for the EL X8. They were program testing all the time, and we let it be known that if occasionally we could have the virgin machine for a few minutes, we would appreciate it. They were nice guys, and a few times per week we would get an opportunity for our next test run. We would enter the machine room with a small roll of punched paper tape, and a few minutes later we would leave the machine room with the output we wanted. I remember it vividly because when they realized what we were achieving, our minimal usage of the machine became more and more frustrating for them. I don’t think their COBOL implementation was ever completed.
EWD1303, My recollections of operating systems design

I can picture Dijkstra and his collegues working with paper and blackboards and thinking hard about how they were writing their software. They came into the room and their software just worked. And I can picture the Cobol crew in a furious vicious circle of code-and-fix; their growing frustration and despair. That was about 1960: no books on software design existed back then. For that matter, no books on compiler writing existed either.

Could the Cobol crew have done better? Absolutely. In 1954, the FORTRAN team led by John Backus produced a compiler that reportedly wrote code almost as good as hand-written. How could the poor Cobol crew have done something as good or at least good enough?

My answer: by designing their software. By breaking the thing into parts (modules) and developing each one separately. By using analogies from other engineering disciplines; by using metaphors. I know this is all obvious to us in 2011 as we all know about module decomposition and the use of metaphors and coupling and cohesion. But is it really obvious? Really?

What I see is that modern-day software teams today can *still* be divided in two kinds: the Dijkstra-crew kind and the Cobol-crew kind. Those who are in control and produce reasonably good software within reasonable resources and whose code is reasonably clean; and those who toil away late hours and produce late and buggy software, with no design, or perhaps with an ineffective, we-dont-really-believe-in-it design and crappy code.

Which do you want to be? What will you do to become it?

http://codebetter.com/iancooper/2011/10/06/avoid-testing-implementation-details-test-behaviours

Re-reading Kent Beck’s TDD book, Ian Cooper recovers some important things that are often overlooked.