Extreme Enthusiasm

Il programma è stato pubblicato (anche se ci saranno piccoli aggiustamenti) e le iscrizioni sono aperte: affrèttati perché sono già spariti quasi 300 dei 400 posti disponibili!!!

I know a developer who says that he enjoys discussion during pair programming. He says that it’s one of the ways to learn and grow on the job.

I disagree. Here’s why. Discussion is waste. In the sense of Lean, discussion is a non-value adding activity. Consider coding and discussing. Do they add value? How do you tell what is waste and what is value-adding, anyway? After all, if you don’t discuss things with the customer and with your collegues, you can’t arrive at a functioning release, right?

David Anderson in his Kanban book explains well, IMHO, how to tell waste from value-adding. Suppose that discussing is value-adding. It that is so, let’s do it to the max! The more, the better. Do the same thought experiment with coding. If coding is value-adding, then let’s do it to the max! Which of the two rings more true?

I have this image in my mind. Consider when you release a feature to the customer. The release is made of lines of code you added, changed or deleted. The value-adding activity was in writing those lines of code. Everything else is waste.

I understand that programming is not typing, and that thinking is the most important activity that happens while we work. Yet, this thinking is waste. Suppose that you can code a feature without much thinking, because (for instance) your team has standard ways of dealing with this sort of features, or you did similar things years ago, and you know how to code a solution. This position is better than the position of someone who does not know how to proceed, and has lots of choices and doubts and has to think.

So if you don’t know how to solve a problem, by all means, think! Step away from the keyboard, consult books, discuss at the whiteboard, ask in forums, look in your codebase to see how this problem was solved before. All these activities are necessary waste. Just don’t charge heads down and try to code yourself out by brute force. That would be probably more wasteful.

You see, not all coding is value adding. Only the coding that produces the lines of codes that will make the release is value adding. Spikes, refactorings, false starts are all (perhaps necessary) waste.

I have not become convinced that iterative development, small steps, refactoring are to be avoided. I’m not advocating doing some big upfront design to make sure we can hit the release code without iterating, without incrementing functionality bit by bit, without refactoring. I still think that iterative and incremental design is the only reliable way to produce good software. But we must remember that refactoring per se is waste. Francesco Cirillo is fond of saying that emergent design is not “doing some messy work now, then I will refactor”. It’s more efficient to write incremental code that can be extended without refactoring; it’s more efficient to exploit the Open-Closed Principle!

Let’s get back to the subject of discussions while coding. My opinion is that programmers either know how to proceed, or they don’t.

• If they don’t, they should do spikes, study, think, stop pretending they are doing a pomodoro of coding and call it a pomodoro of studying.
• If one of the developer thinks he knows how to proceed, he should be the driver, explain to make sure that the navigator understands, and then do it. The navigator’s job is to make sure that the driver has a reasonable understanding of the problem, and he’s not doing something that is probably/obviously/previously shown to be a dead end.
• If both developers have a differing idea, then my friend says they should discuss the relative merit of both points of view and come to an agreement to what to do. I disagree. This is pure waste. I think that it’s more productive to choose one of the two ideas and see how it turns out. That is worth more than all the discussion in the world. Choose quickly; challenge the other saying things like “I bet I can solve it cleanly in 4 pomodori!”.

The navigator’s job is still to support the driver, not to challenge the driver’s idea. That would take energy away from both. If the navigator thinks that the driver’s idea might work, he or she should support the driver and accept that the thing will be done differently. Now this will help both grow and learn, more than discussing. Just remember to respect the other, and always explain clearly what your idea is.

Summary: my friend Federico Gobbo argues about the wisdom of pushing hard for Italian citizens to learn English.

Segnalo questo post del mio amico Federico Gobbo, che affronta l’argomento dell’insegnamento della lingua inglese, e di come, quanto e come sia saggio spingere in quella direzione.

I’m reading The Toyota Way by Jeffrey K. Liker. It’s changing my mind.

When I started this book, I thought, well, this book is a compilation of interviews and stories, so there is no real meat, it’s not really teaching the methods they are using in Toyota. I thought it was a pretty fluffy book. Was I wrong!

It’s true that the meat of the book are stories that the author experienced himself, or were related to him. It’s not true that there are no practical hints on how to apply Lean Production; there are many examples of how to do that. But that’s not the point. I mean, Toyota makes cars, I make software. The actual practices are meant to solve their particular problem. What this book shows is how getting good at management comes from a shift in the way you think. The wealth of examples in the book pushes the reader to a different way of thinking.

The Toyota Way is also a depressing book. Perhaps half the stories in the book are about how American companies tried to apply The Toyota Way and failed miserably. This speaks about the importance of being serious in what you do. The Toyota people are dead serious. Which is not to say they don’t have fun… If your idea of fun is to stretch your capabilities to the limit, keep learning and getting better.

There is another feeling that comes to me while reading. The Toyota company is what in Italy is called “primo della classe”. This expression means “the boy or girl with the best grades in their class”, and carries a negative connotation. This says a lot about the way people thinks in Italy; the idea that someone can be better than me, or maybe *a lot* better than me is met with hostility. Our thinking is, “they are first because they spend countless hours on the books; if I did that I would get the same grades, but I won’t bother”. Ha! Do I have what it takes to really spend this effort? I probably don’t.

So these Toyota people come across as “primi della classe” because they are really good. And there are companies out there that are way better than the norm, and still are not as good as Toyota; which shows that there is a long path of improvement. This realization can be either exciting or depressing; it’s our choice :-)

As I keep reading the book, it takes me a lot of time as I’m taking notes, and all the while I think how the principles of the Toyota Way would apply in my situation. For instance, the book says that Taiichi Ohno insisted that when you walk to a plant, you should see immediately if everything is OK, or there are problems.

Translate that to software development. How would you do that? Good question! Agile developers do many things to make the development process visible; the cardwall, the burndown chart, the velocity chart are good visual indicators. But can we do better? Translate “plant” to “a workstation with a pair of programmers at work”. Can you see that their work is processing smoothly? One valuable insight is recognizing that the pair might block on a problem, just like an assembly line can jam. How do you see when a pair is blocked? (Hint.) What do you do when a pair is blocked? Do you treat this block as seriously as you would for a defect in the product? Do you try to understand the root cause of the block, so that it doesn’t happen again? Do you measure how many such “incidents” happen in a week? Do you measure how much time is spent working smoothly, and how much time is spent blocked on a problem?

One day I was discussing with Josh (not his real name) a problem we have with the customer. The problem is a very common one: priorities keep changing, every day we receive new “urgent” tasks, we have lots of work-in-progress, and it’s difficult to get the customer to approve or give feedback on what we did.

Josh did not agree with me on the nature of the problem. He said that he respects our customer as Product Owner; he accepts the priorities, and any new task they give us is properly recorded in the backlog. But Josh, I said, I’m afraid that the day-to-day priorities they give us are not what the customer really needs. The boss of the person that gives us priorities may not be happy with the results we are getting. Josh replied, “if what we do is not what their boss wants, then let it be. It will come up at some point.”

What is Josh thinking? It may be that his reasoning is “it’s not the best course of action for us to rock the boat with their boss; your worries might be wrong, perhaps the priorities are right after all. And if you are right and the priorities are wrong, well, the best course of action is to let this problem emerge.” That’s my best interpretation of what Josh thinks.

The other interpretation is that Josh is delegating responsibility to the customer’s product owner. If the PO gets priorities wrong, we’ll fail, but it will not be our fault. It’s another way to say “You just tell me what I have to do and then I do it”. This goes against the principle of taking responsibility for the outcome of a project. The reasoning is that if the project fails, we can show that we did all that was required of us. But being “right” in a failing project is not a great outcome.

What’s my conclusion? I found two interpretations for Josh’s reaction. They are both valuable; I resolved to talk to Josh further, to see if there is anything else behind his words. And I resolved to discuss the issue of priorities with the POs.

I won’t bore you with the story of how long it took for people to recognize that zero is a number. Without zero it would be difficult to explain what is the value of, say, 3 minus 3; we’d be forced to say that it’s a “meaningless” expression. Funny huh? Yet some developers seem to be stuck to medieval thinking in this respect.

Have you ever seen code like this?

public List findAllEmployeesByDepartment(int departmentId) {
  String sql = "select * from employees where department_id = ?";
  ResultSet rs = select(sql, department_id);
  if (rs.size() == 0) {
    return null;
  } else {
    // ... convert the recordset to a List and return it
  }
}

This developer seems to think that an empty List is not a regular list, so he thinks he should return a special value like null to signal that the query returned no values. This is totally unnecessary. No, I take it back: this is totally wrong. You are forcing all callers of findAllEmployeesByDepartment to check for null. Not only that; this code seem to say that it’s a totally unnatural and unexpected thing for this query to return no rows. Soon developers will forget to check for null, and the application will throw NullPointerExceptions.

A related example is:

Foo[] foos = ...;
if (foos.length > 0) {
  for (int i=0; i < foos.length; i++) {
    // do something with foo[i]
  }
}

Here the developer thinks that they have to treat the case of an empty array separately. In fact the IF is totally unnecessary. If the array is empty, the loop would execute zero times anyway. Java (and C) arrays use asymmetric bounds, which make it easier to write code that does not need to treat a zero-size interval as a special case.

In conclusion: empty collections are perfectly valid collections, and empty arrays are perfectly valid arrays. It’s a good idea to write code that doesn’t treat “zero” as a special case.

This post is part of a series on development fundamentals.

Last time at the Milano XP User Group we had our first retrospective. We gathered lots of useful insights. One of the selected actions was “if you propose a Kata, you must publish your solution“.

Guilty as charged: when I proposed Rock-Scissors-Paper for an evening of coding, we got an acute case of analysis-paralysis. It was a humbling evening: we didn’t manage to implement the first requirement, “rock beats scissors.” I provide here my solution. It’s brand new code, written today, so it benefits from the experience.

This is the list of requirements. I didn’t get it right immediately.

Context: we want to implement a gaming engine for the game rock-scissors-paper. We don’t care about players or rounds; we only want to implement the “beats” relation.

1. – rock beats scissors
2. – scissors does not beat rock
3. – rock does not beat rock
4. – scissors beats paper
5. – paper beats rock

The clarifying statement should kill most of the discussion that we had that evening. We are not interested in modelling players or rounds or winning. Only the “beats” relations.

This kata was meant to be executed with the OCP Dojo rules. I will use Ruby today, because I want to take a break from Java. But the solution will not use any trick that can’t be done in Java. I promise :)

Rock beats scissors

This one is easy. That is, easy if you decide how to implement the “rock” and the “scissors”. I decided for the simplest thing, that is a Ruby “symbol”. In Java I would have used a string.

require "test/unit"

class Rsp
  def beats(first, second)
    true
  end
end

class TestRsp < Test::Unit::TestCase
  def setup
    @rsp = Rsp.new
  end

  def test_rock_beats_scissors
    assert @rsp.beats(:rock, :scissors)
  end
end  

Note that I have substituted the “factory” of the OCP rules with a simple setup method. The @-prefixed variable is Ruby’s way to define a member variable.

Scissors does not beat rock

This one almost escaped me at first. If your requirements only mention “X beats Y”, then return true will always be a valid implementation! Also, note that we don’t need to represent “ties”. Winning or tieing are not part of our requirements; but it should be clear that the “beats” relation can be used to determine victory or tie.

So this is the famous “second test” that in most OCP Dojos forces the creation of design. If it were not for the rules, I’d be tempted to write something as boring as

  return true if first.equals(:scissors) and second.equals(:rock)

inside the Rsp#beats method. Instead I decide I want to solve the problem with a rule-based style. Let’s delegate the decision of “beating” to a rule.

class Rule
  def initialize first, second
    @first, @second = first, second
  end

  def beats(first, second)
    @first == first and @second == second
  end
end

class Rsp
  def initialize(rule)
    @rule = rule
  end

  def beats(first, second)
    return true if @rule.beats(first, second)
    false
  end
end

class TestRsp < Test::Unit::TestCase

  def setup
    @rsp = Rsp.new Rule.new(:rock, :scissors)
  end

  def test_rock_beats_scissors
    assert @rsp.beats(:rock, :scissors)
  end

  def test_scissors_does_not_beat_rock
    assert_false @rsp.beats(:scissors, :rock)
  end
end    
  

Now if we wanted to be strict I’d have to show that I can make the first test pass and the second test fail by simply passing in a different Rule. I could do that by defining

class AlwaysTrueRule
  def beats(a, b); return true; end
end

This way if I pass an instance of AlwaysTrueRule, it will make the first test pass and the second fail. If I pass in the correct rule, both tests will pass.

Rock does not beat rock

Now this works with no modification needed to the code. From this point on, it will be clear that we don’t need to test all cases where “beats” does not hold. The implementation makes it obvious that only the cases that are explicitly covered by the rules pass.

Scissors beats paper

This forced me to change the single rule to a list of rules.

class Rsp
  def initialize(rules)
    @rules = rules
  end

  def beats(first, second)
    for rule in @rules
      return true if rule.beats(first, second)
    end
    false
  end
end

class TestRsp < Test::Unit::TestCase

  def setup
    @rsp = Rsp.new [
      Rule.new(:rock, :scissors), 
      Rule.new(:scissors, :paper),
      ]
  end
  
  # ...
  
  def test_scissors_beats_paper
    assert @rsp.beats(:scissors, :paper)
  end
end  

The rest

At this point the design is more or less complete. The remaining test, “paper beats rock” only needs to add a new Rule to the list. No “coding” needed, only configuration of existing objects.

The test of a design is how well it resists to changes. I would say that at this point, implementing Rock-scissors-paper-spock-lizard or even RSP-15 would be trivial.

I can imagine even wierder variations, for instance:

• Peace beats anything other than peace
• Random has a 0.5 chance of beating anything other than random
• Thrice beats anything other than thrice on the third time it’s played

and I think that by writing a custom Rule, these could also be easily done. Therefore I think that the design so far is a success. The amazing thing is that I did all the coding in 1 pomodoro (25 minutes)! That includes taking notes and saving intermediate versions, for blogging about later.

Why did we take so long when we tried this at the coding dojo? I think that the trick is in defining the scope more precisely:

We don’t care about players or rounds; we only want to implement the “beats” relation.

By thinking of “beats” as a mathematical relation, that is, a subset of Stuff × Stuff for you mathematically inclined, we simplify our job. Also, giving up on the idea of defining a Rock class helps.

Know your stuff

A while ago we had a fun evening at the Milano XPUG writing a Sudoku solver. I blogged about my solution. I’m not particularly proud of it, in retrospect. The code and the tests are not obvious. I can’t read any of it and be certain that it works. It does not speak.

It is true that solving puzzles like Sudoku is quite different from what application programmers do everyday at work. Why is it that? The problems that we solve in business applications do not have that mathematical crispness that puzzles have. Perhaps it’s because we’re not good enough at analyzing them and expressing them abstractly. That would explain why business code is so long, convoluted and expensive.

Anyway, the point I want to make is that it is not satisfying to use the tests in TDD as a crutch for constructing hapazard code that, with a kick here and a few hammer blows there seem to work. The point of TDD is to *design* code; and a good design shows how and why a solution works.

I often see people doing katas that involve problems with well-known solutions. We usually disregard, forget, or ignore the well-known solution! And we keep writing tests and production code until we rig together something that passes the tests. It’s painful. I know. I too did some of that.

TDD does not work well when we don’t know what we’re doing. Some high-profile XPers failed to ship when TDDing their way with unfamiliar APIs or disregarding known solutions. TDD is no substitute for analyzing a problem, and finding abstractions that make it easy to solve. TDD without thinking and analyzing and abstracting is not fun!.

It’s for this reason that there is the XP practice of “spiking solutions”, that is, take time to learn how to do something, make experiments, then apply what you learned. If you know how to do things, you will not spend hours discussing with your pair; you and your pair will grab the keyboard from each other, as Francesco Cirillo often says.

A better way

Consider Sudoku again. Peter Norvig solves it in two different ways by using known techniques. The first solution is depth-first search, which is gueranteed to terminate as the graph of Sudoku states is acyclic. The other is by constraint propagation. If I were to do the exercise again, I would try to make the analysis apparent from the code.

Say we want to solve it by depth-first search. That entails two sub-problems:

• a) writing a depth-first algorithm
• b) writing something that enumerates legal moves in a given Sudoku board

I would start by testing the depth-first search algorithm. I would drive it with an abstract “tree” implementation. This means I could concentrate on the search algorithm without being distracted by the complex details of the Sudoku rules.

Then I would test-drive the generation of next-moves from a Sudoku position. That could also be done incrementally. Can we imagine a simplified Sudoku board? The full Sudoku is too complex for the first test! A year ago I would have started by defining a 9 by 9 array of numbers, but now the sheer boredom of defining it would stop me. Is there a better way?

Relax. Think. Dream.

Think about the game terminology. As Norvig says, the game is about units (either a row, a column or a box). A unit with no empty spaces has no descendant moves. A unit where a number is missing has the full unit as a descendant move. A unit where two numbers are missing… You get the point.

Then work out how to combine descendant moves of two units that share a square. Think a row and a column. If the common square is empty, than valid solutions for that square must be valid for both units…

The point is to work the problem incrementally. Try smaller scales first. Try 2×2 boards. Make the size of units and the board parametric. Add the constraint rules one by one, so that you can test them separately.

Conclusions

One important principle to apply is “separation of concerns”. Enumerating moves is a problem, and search strategy is another. By solving them separately, our tests become more clear and to the point. We gain confidence that we know how and why our code works.

Another way to see this is to decompose a problem in smaller problems; prove with tests that you can solve the subproblems, then prove with tests that you can solve the composition of the subproblems.

When you have a problem that is of fixed size 42, turn that constant into a parameter and solve the problem for N=1, N=2, … Imagine if the Sudoku board was 100×100 instead of 9×9; would you define a 100×100 matrix in your tests? Turning these constants into parameters make your code more general, your tests more clear, while making the problem *easier* to solve!

To summarize, what I think is important is

• Learn data structures, algorithms, known solutions, the proper way of doing things.
• Apply separation of concerns.
• Solving a slightly more general problem sometimes is much easier than solving the actual problem
• It’s more fun to work when you know what you’re doing!

Update

Charlie Poole recently posted this on the TDD mailing list (Emphasis is mine):

I’ve written elsewhere that I believe attempting to get TDD to “drive” the invention of a new algorithm reflects an incorrect understanding of what TDD is for.

TDD allows us to express intent (i.e. design) in a testable manner and to move from intention to implementation very smoothly – I know of no better way.

OTOH, you have to start out with an intent. In this context, I think that means you need to have some idea of the algorithm you want to implement. TDD will help you implement it and even refine the details of the idea. Writing tests may also inspire you to have further ideas, to deepen the ones you started with or to abandon ideas that are not working out.

Vlad Levin blogs thoughtfully:

one of the first rules I teach my students when I am doing a TDD workshop or teaching a course is precisely that TDD is not an algorithm generator! Solving sudoku is just the kind of problem you want to find an algorithm for first, then implement that algorithm

[…]

So what is the purpose of TDD then? One goal of TDD is to reduce the need to determine ahead of time which classes and methods you’re going to implement for an entire story. There’s a large body of shared experience in the developer community that trying to anticipate such things tends to lead to paralysis where nothing useful gets done and/or produces bloated, over-designed code. Instead, you can develop one aspect of the story at a time, using each test to keep yourself moving forward and refactoring the design as you go along

In a comment to my post about testing data serialization, Jacopo says:

well, if you’re lucky enough, both client and webservice will be under your control, so yes: focus on data (structure) sent/received, to be sure they’re as expected.

but is far more common a scenario where webservice aren’t under your control, they represent and integration point with external systems: that is, XML document syntax _is_ what you’re trying to test. for example, (I can think of at least 2 cases that happened to me in the last 6 months) switching from one parsing library to another or simply testing interoperabily (such as Java/.NET).

of course, not as much domain logic there ;) but still, webservices aren’t supposed to encapsulate domain knowledge, so focusing on syntax would be good enough.

Hi Jacopo,

Even in your case, the users of your service will be interested in doing something with your data, not in its syntax.

Consider this modified test: there’s an error in it, but it’s not easy to see.

@Test public static void serializesToCorrectXml() {
  DataStructure structure = new ....;
  XmlSerializer s = new XmlSerializer();
  String xml = s.serialize(structure);
  assertEquals("<?xml version=\"1.0\"><foo><bar>zot</bar></foo>",
    xml);
}

Can you see it? The xml prolog is broken. Now if your code passes this test, it will break in production with your paying customers.

Consider this other test:

@Test public static void serializesArray() {
  XmlSerializer s = new XmlSerializer();
  String xml = s.serialize(new String[] {"A", "B"});
  assertEquals("<list><el>A</el><el>A</el></list>", xml);
}

this one is also wrong, for a different reason. It’s syntactically valid, but does not contain the right information.

Even when I am not writing the client side, I would still write a client for my tests. The client should do something with my data that is similar to what a typical real client would do. At the very least, it should parse the XML to make sure it’s valid! That would take care of the first broken test. Then I would make the client extract the information from the serialized form, and make sure it contains the same information.

@Test public static void serializesArray() {
  String [] anArray = { "A", "B" };
  XmlSerializer s = new XmlSerializer();
  XmlClient c = new XmlClient();
  String xml = s.serialize(anArray);
  
  assertArrayEquals(anArray, c.parse(xml).toStringArray());
}

Another thing that happened when I was last coaching a team on XP techniques. We were working on a real application, and one thing that we had to do was to serialize a data structure to XML for publishing on a web service. There are a ton of different ways to do that, but whatever you do, don’t write tests like this:

@test public static void serializesToCorrectXml() {
  DataStructure structure = new ....;
  XmlSerializer s = new XmlSerializer();
  String xml = s.serialize(structure);
  assertEquals("<?xml version=\"1.0\"?><foo><bar>zot</bar></foo>", 
    xml);
}

Can you see what’s bad with this code?

I can name a few things:

• It doesn’t scale. When the data structure grows, the xml string becomes big and difficult to handle
• It’s brittle. A tiny, insignificant difference in white space will break the test.
• It does not express intent. Look at the name of the test. We could not come up with anything better than this.

The crux of the matter is that we don’t care at all what the xml string is. What really matters is what we do with it, and in this case what we do is build the XML in one process, and reconstruct an equivalent data structure in another process. We should test that the behaviour works, not the details of an intermediate structure.

@test public static void serializesAndRebuildsFoobars() {
  DataStructure structure = new DataStructure("foo", "bar");
  XmlSerializer s = new XmlSerializer();
  XmlParser p = new XmlParser();
  
  assertEquals(structure, p.parse(s.serialize(structure)));
}

Now this tests the behaviour we’re interested in. Note that this test is robust, as changes of the intermediate representation will not break it. We don’t really even care what the serialized data type is: we could change it to byte array, or anything else, with no need to change this test.

This sort of tests supports incremental development. If my data structure was, for instance, a list, I could write a nice sequence of tests:

• serializesAndRebuildsEmptyLists
• serializesAndRebuildsSingletonLists
• serializesAndRebuildsListsOfManyElements
• …

One final note: let the pair who writes the serializer also write the parser. This will save a lot of arguments :-) Vertical slices of functionality always work best.