Extreme Enthusiasm

Questo sabato parlerò al Code Motion di TDD. Io sono alle 15.20 nella sessione “Tool and Processes.”

http://www.codemotion.it/programma-talks

After the fact, it’s easy to find better solutions. I present two examples from my experience with an ecommerce Rails application.

How to avoid joins, part I

We needed to make both product categories and discounts appear or disappear at specific times. The canonical solution seemed, at the time, to declare

class Schedule < ActiveRecord::Base
  belongs_to :schedulable, :polymorphic => true

  def active?
    Time.now.between?(self.start, self.end)
  end  
end

class Discount < ActiveRecord::Base
  has_one :schedule, :as => :schedulable
  
  # ...
end

class Category < ActiveRecord::Base
  has_one :schedule, :as => :schedulable
  
  # ...
end

The schedules table contains the two columns “start” and “end”. The logic for a schedule is pretty simple: it’s “active?” if “now” is between “start” and “end”.

A lot of thought and discussion went into deciding when to save a Schedule object in the schedules table; we wanted to write

class Category < ...
  def active?
    self.schedule.active?
  end
end

but this presumes that there indeed is a schedule saved in the database for a_product, so it had to be

class Category < ...
  def active?
    self.schedule && self.schedule.active?
  end
end

and similar complication happened when adding or changing the schedule. Selecting all the active categories, got more complicated for we needed an extra join, and a left outer join at that. That impacted all queries on categories or discounts, in particular the free search query that really didn't need to get more complicated. We decided to add a post_create callback so that every new discount or category would always have an associated schedule. The decision to go with the "polymorphic has-one association" led to complification and increased coupling.

So what is a simpler, caveman's solution? Well, why not add "validity_start" and "validity_end" columns to both categories and discounts tables? Our Ruby code would become:

module Schedulable
  def active?
    Time.now.between?(self.validity_start, self.validity_end)
  end  
end

class Discount < ActiveRecord::Base
  include Schedulable
  
  # ...
end

class Category < ActiveRecord::Base
  include Schedulable
  
  # ...
end

So all it takes to make a model "schedulable" is to include the Schedulable module, and add two more columns to the model table.

Analysis of the "caveman" solution:

• Duplication? In the definition of the two tables, perhaps a bit. But no duplication in Ruby code, and far less code to write.
• Denormalized? Actually no. A "schedule" is not something relevant to the business; it doesn't need an identity. It is not a business entity so it's proper that we implement it as a collection of attributes rather than with its own table.
• Good Rails design? I think it is. The decision to make Schedule a model was bad design, as a Schedule is not a business entity.

So the "caveman" solution actually is what a good data modeler would have chosen from the start. We were fascinated by how easy it was to use the "polymorphic association" to remove the duplication of the two extra columns, that we ended up complicating our life for no good reason.

Lesson learned: always consider what a caveman would do. He might be smarter than you!

Lesson learned: Rails is an effective way to put a web GUI in front of a database. Think like a data modeler. Think Entity-Relationship.

How to avoid joins, part II

Later in the same project, we needed to make the website respond in English or Italian. Rails 2 is well equipped for localizing the GUI out-of-the-box; but it will not deal with the problem to translate the properties of your model. In our case, we needed to translate the names and descriptions of products.

The canonical Rails solution at the time was to use the Globalize2 gem. It's actually a good gem, but in retrospect it was not a good fit for our problem.

Fact: we didn't need to support 1000 languages. Just 2. Maybe 3 or 4 in the next 5 years.

Fact: we didn't need to translate 100 attributes in 100 models. Just 3 properties in 2 models.

The Globalize2 gem adds a join to a "globalize_translations" table to every query. That didn't do any bit of good to the free search query, that was awfully complicated already! So while in theory Globalize2 is transparent, in practice you have to modify many queries to take it into account.

Once again, what would our friend the caveman do? You guessed it, add extra columns instead of a join. Replace columns "name" and "description" with "name_en", "name_it", "description_en", "description_it". You add a bit of drudgery to the schema definition, but your queries turn out to become simpler. Taking advantage of the fact that schema migrations in Rails are very easy, it would not be a big problem to add a new language by adding the few extra columns.

And all the Ruby code it would take to produce the globalized descriptions in the web pages would be to define

class Product < ...
  def globalized_name
    if self.attributes.include? "name_#{current_locale}"
      self.attributes["name_#{current_locale}"]
    else
      # fallback to Italian
      self.name_it
    end
  end
end    
  

My solution is not transparent, but it's simple and easy to understand; while Globalize2 tries to be transparent but does not quite succeed.

Lesson learned: less magic please. If you can't achieve complete transparency, then go for being explicit *and* simple.

What I’ve been up to?

It’s been some time. This last eight months or so I haven’t been doing much software development. Most of my work has been in support. I’ve done consulting work.

This autumn I was hired to help a team with an application that was constantly crashing. The application was closed source, so we could not easily change it or trace how it worked. The first thing we did was to start monitoring stats. I wrote a simple monitor that tracks the health of a Tomcat process. This gives a feeling for what’s happening inside the application, for what happens in normal times and in times when it’s stressed. We kept a monitor open with baretail open on these log files and watched what was happening.

Every two or three days, the count of busy threads in Tomcat started rising, from the normal level of 2-3 up to the maximum of 150. At that point there was nothing to do but restart the server.

Lesson 0: set up watch dogs. It’s easy to write a custom status page sniffer. Nagios keeps an eye over many variables (memory, CPU load, failing services and the like.) New Relic is a very good all-round solution when you don’t know where to start.

The usual suspect in such cases is the database, and watching the process monitor on SQL Server revealed that many connections were blocking each other. We called in one consultant, then another, and finally a third one who knew how to deal with this problem.

Lesson 1: a consultant may know how to deal with your problem, or he may be out of his depth. You better make sure that any consultant you call in know what they’re doing. Warning signs is when they try stuff at random (I read in a blog that changing parameter XYZ could be helpful!) or just give you canned responses (uh, better install the latest service pack…)

The last SQL Server consultant really knew his stuff, and was able to solve the SQL Server hangups problems by cleaning up and optimizing the SQL Server installation. It turned out that this database has been running with no maintenance for years.

Lesson 2: if you run a business that depends on database-backed applications, you better have a full-time DBA looking over maintenance.

After the SQL Server tune up the application ran considerably better. But there were more problems on the horizon. We still had a few occasions when the application did block. I didn’t know what to do to get insight on what happened in those cases. I wanted a thread dump of the Java Virtual Machine, but on Windows it turns out it’s not an easy thing to do.

Lesson 3: Windows works. You can’t really say it doesn’t. And this is the most charitable thing I can say about it. All the same, Unix is so much better than Windows in every which way. It has always been.

In the end I found a neat utility called psexec that helped us solve that problem. We set up a script that checked the number of busy threads on Tomcat every 30 seconds, and created a thread dump for us. This showed that our Tomcat threads were busy trying to deliver a mail message to a SMTP server. A quick investigation found that the application was somehow configured to use a temporary, slow SMTP server nobody remembered about. We changed the configuration to point to a fast server and this problem was fixed.

The last failure mode we encountered happened rarely; at times the memory consumption in Tomcat started growing from the usual 100-150MB up to its configured maximum of 500MB, and stay there. At that point, the application was slowing to a crawl, and there was nothing but, again, reboot it. Try to imagine our feelings of helplessness while this was happening!

Our next tool here was to run JMap to get a memory dump, and then analyzing it with the Eclipse Memory Analyzer. The culprit seemed to be some data objects that we knew the application was allocating to contain the result of queries. Whenever a user was requesting a report of, say, 50K records, the app allocated memory to hold all of the result in memory, and saved it… guess where…. in the user session! Then it showed the first 100 rows to the user. Since that query was slow, the user was staring at a blank screen for a minute or two, and then reload (F5!), which caused Tomcat to perform another query. Each of these 50K rows queries was allocating 50MB of ram in the JVM. Boom, boom, boom, once this started it brought Tomcat to its knees.

Lesson 4: Don’t store more than a few bytes in the user session. Being stateless is the key to scaling. Which brings us to the next…

Lesson 5: Pagination is best done in the database. Caching is best done in the database. Filtering is best done in the database. Heavy computations with the data are best done in the database. Brush up on your SQL skills! If you’re doing business software, your app’s performance and capabilities will be dominated by how well you use your database.

We could not change the application code, but luckily there was a configuration parameter that limited the maximum number of returned rows. Setting it to 1000 solved the problem.

Lesson 6: whenever you perform a query, you must be certain that the number of rows it will return is bounded.

And so our job with this application was finished. We went on tackling the next… One amazing thing that I learned in the process is that placing Apache in front of Tomcat is not always gueranteed to work well. We had some really rough time when we discovered we had this devastating Apache bug, and later we found a different bug in a different app with the same setup.

Lesson 7: try to use as few pieces (processes, computers, things) as possible. Every piece you have may be a cause of failure. Whomever says “… and by using XYZ, we get ZYX for free” is a fool! Nothing comes for free.

One of the collegues who valiantly supported me in this adventure was so much overwhelmed by the stressful situation that he resigned. I don’t know what to think of this; certainly this was not the sort of job that we were trained to deal with. This was Operations, not Software Development. And this was the life of the consultant. Diving into a problem not knowing the solution, but with the confidence that we will find the means to find the means of solving it. This is the opposite of “you tell me what I must do and I do it”, which is an unfortunately common misinterpretation of “customer collaboration”.

Lesson 8: Act as a consultant, not as a contractor (as @p_pugliese would say). Find your own ways of gathering information, making plans, trying solutions. Find your own ways of checking if you’re making progress.

In the end, my collegue will have to decide for himself if this is a crisis that leads to coming back to this job stronger than before, or a signal that his deep desire is to do something else.

Mercoledì 1 dicembre al Milano XP User Group, ci sarà Francesco Cirillo a rispondere alle nostre domande su “10 anni di XP in Italia.” Dove? A Sesto San Giovanni, presso XPeppers. Francesco ha portato Extreme Programming in Italia, formando il primo team XP italiano, di cui ho la fortuna di avere alcuni membri come colleghi. Ma Francesco non si è limitato a importare in Italia le cose apprese da Kent Beck; Francesco ha aggiunto le sue idee originali, come la Tecnica del Pomodoro, la sua maniera molto metodica di misurare la performance dei team, il suo stile di design emergente che è un po’ diverso dal TDD standard.

Contribuisci alla serata postando le tue domande sulla pagina dell’evento.

OCP Dojo at the XP Days Benelux 2010

Last week I presented with Antonio Carpentieri The Open/Closed Principle Dojo at the XP Days Benelux 2010. 19 people attended; I’m happy that we received positive feedback and good criticism.

In the feedback for our session, someone wrote in the “what I like”:

• Refactor before implementing new stuff
• Can apply this on Monday

In the general feedback about the conference someone wrote, in the “Which (new) things are you going to apply or investigate further?”

• The Open/Closed Principle
• Pomodoro technique
• IFs are evil

I’m also happy that Marc Evers and Rob Westgeest did a demo of the OCP Dojo at the NLJug last month.

Here are my slides for my talk at the Italian Agile Day 2010.

I’m happy about how the presentation turned out. I shared my hour with Carlo Bottiglieri. While I came only recently to realize that html templates are evil and objects rule, Carlo has been doing something like this for years. I expecially liked one thing he said:

These days you often hear people say that TDD is only good for the “domain model”. When I started learning TDD I didn’t know that :-) so I started using it for everything, including user interface, persistence and other infrastructure.

Some people say that they do TDD, but when they start a new project they do something different: choose technology, choose application servers, choose frameworks… TDD is meant to help us make decisions. If we make important decisions before starting with TDD, we lose a great deal of its power.

I’m quoting from memory and paraphrasing. I hope I got the meaning right.

Update: Carlo has published his slides.

Update The video for our session has been published.

Quest’anno il programma dell’Italian Agile Day è più ricco che mai. Vorrei poter seguire tutte le sessioni… e immagino che tutti i partecipanti abbiano questo dilemma. Per questo ho scritto alcune note per orientare chi magari non ha dimestichezza con tutti gli argomenti presenti.

Tipi di sessione

La maggior parte delle sessioni quest’anno sono di tipo frontale; cioè c’è un tizio che parla, gesticola e mostra delle slide sul proiettore, e il pubblico ascolta o al massimo risponde alle domande del tizio.

Ci sono però anche alcune sessioni di stampo più pratico: i famosi workshop. In un workshop il pubblico deve partecipare attivamente. Può essere molto faticoso ma è anche molto interessante. Le sessioni di tipo workshop quest’anno sono:

• Agilità come evoluzione sistemica di Pierluigi,
• Due uomini e una lavagna di Alberto,
• Guelfi versus Ghibellini di Sergio,
• The Dot Game di Alberto e Jacopo.

Il Dot Game è anche un gioco: c’è una lunga tradizione nel mondo Agile di giochi concepiti per imparare. Uno dei primi è stato lo XP Game di Pascal Van Cauwenberghe. Io l’ho organizzato decine di volte; penso che sia molto utile. Per approfondire: iscriviti a questa mailing list.

Alcune delle sessioni sono kata o performance di programmazione dal vivo (ne parlo più avanti.) Nella sessione BDD Live Show gli autori si dicono pronti a ricevere l’input dal pubblico nel caso si incontri un ostacolo imprevisto…

Tema: le soft skill

Molti sono attratti dal mondo Agile per le pratiche tecniche: il TDD, il Continuous Deployment, ecc. Ma le pratiche tecniche non sono sufficienti: il software lo fanno le persone, e le persone sono tutte diverse. Non solo diverse l’una dall’altra, ma anche diverse da un giorno all’altro :-)

Per lavorare bene in team bisogna coltivare le cosiddette soft skill. Le sessioni Note to managers: IT is different, get over it di Andrea Provaglio e la keynote di Paolo Perrotta parleranno del fattore umano nello sviluppo software.

Le sessioni Agilità come evoluzione sistemica di Pierluigi Pugliese, Guelfi versus Ghibellini di Sergio Berisso e The Dot Game di Jacopo Romei e Alberto Brandolini sono centrate su come lavorare in team.

Un tema che corre sotto tutte queste sessioni è che lo sviluppo software non assomiglia per niente alla produzione industriale. Anche parlare di “fabbricare” software è errato, a mio avviso. “Sviluppare”, “fare crescere” sono termini più appropriati. Lo sviluppo software assomiglia molto più al giardinaggio che alla manifattura.

Tema: Lean

Il Lean manufacturing è una corrente di pensiero che nasce e fiorisce in Giappone ed ha consentito alla Toyota di diventare il primo produttore di automobili nel mondo. Non lasciatevi ingannare dai problemi recenti per cui hanno preso in giro la Toyota; è come quando per una volta il primo della classe prende un brutto voto, e tutti gli asini gioiscono! Il pensiero Lean mette al primo posto la creazione di valore per il cliente. Da questo concetto apparentemente semplice nascono potenti conseguenze. (Per inciso, la manifattura occidentale mette al primo posto il profitto e una malintesa “produttività”; e i risultati sono sotto gli occhi di tutti.)

In una fabbrica Lean gli operai sono responsabili del loro modo di lavorare. Sono essi stessi che scrivono il manuale di operazione per le loro postazioni. I manager di una fabbrica Lean passano più tempo negli impianti che nei loro uffici. Al centro del Lean c’è il miglioramento continuo (kaizen). Leggi The Toyota Way per saperne di più.

Non so se da questo brutale sommario si capisce che c’è una profonda affinità fra Lean e Metodi Agili. Anche nei metodi agili il fondamento è la consegna di valore reale per il cliente. Anche nei metodi agili le persone sono l’asset più importante. Anche nei metodi agili si privilegiano la concretezza e lo sporcarsi le mani piuttosto che le gerarchie e le teorie tayloriste.

Uno degli strumenti usati dal Lean è il kanban, che è una maniera di limitare la quantità cose che sono contemporaneamente in lavorazione. Il kanban usato dagli agilisti prende la forma di un tabellone su cui il processo di lavoro viene visualizzato. Recentemente il Kanban (con la K maiuscola) è stato presentato come un metodo per introdurre l’agilismo in azienda. Al contrario di Scrum o XP, che sono in aperta rottura con il modo di lavoro corrente, il Kanban ha un’introduzione più “morbida”. Si inizia analizzando e visualizzando sul tabellone il metodo di lavoro corrente, identificando il “cliente”. Questo è il trampolino per introdurre il miglioramento continuo: si misura e si ottimizza il tempo che ci mette un task a passare attraverso il processo. Il vantaggio è che, almeno in prima istanza, non vengono minacciati gli equilibri esistenti. Leggi Kanban per saperne di più.

Le sessioni IAD dedicate al Lean e al Kanban sono

• Scrumban, a methodology fusion, di Fabio Armani, che ha applicato in azienda una combinazione dei due metodi Scrum e Kanban.
• Complexity vs. Lean (in inglese), di Jurgen Appelo, un consulente che pubblica un blog molto seguito.
• La Lean Startup e il customer development model. In Italia. Di Nicola Junior Vitto. Il “Lean Startup” è un’applicazione dei principi del Lean alla nuova micro-imprenditoria. Leggi il blog di Eric Ries per saperne di più.
• Lean e Kanban: la nuova rivoluzione Agile di Claudio Perrone. Claudio ci racconta la sua esperienza in Irlanda come agente di cambiamento.
• The Dot Game. Una delle teorie affini al Lean è la Teoria dei Vincoli (Theory of Constraints) di Eli Goldratt. Sospetto che questa sessione (che consiste in un gioco) presenterà delle riflessioni legate a questo argomento. (Leggete The Goal, è un libro interessantissimo sia per la forma che per il contenuto.)

Tema: il Test-Driven Development

Il TDD, e la sua variante Behaviour-Driven Development (BDD) sono una pratica tecnica centrale per gli agilisti. Se vogliamo partire a sviluppare senza una fase iniziale di progettazione, e pretendiamo di riuscire a mantenere basso il costo di manutenzione dell’applicazione, dobbiamo essere in grado di progettare l’applicazione strada facendo. Altrimenti rischiamo di produrre il classico gomitolo di spaghetti, e quello che facciamo non è agile ma un becero code-and-fix.

Quindi, per raggiungere gli obiettivi dei Metodi Agili, ci serve una base tecnica solida. Il TDD è la tecnica base che ci permette di progettare l’applicazione in maniera incrementale e di fare evolvere il design. Per approfondire questi temi le sessioni Affiliamo i nostri strumenti: un test driver fatto in casa di Jacopo e The secret art of Agile Javascripting
di Luca.

Uno dei sotto-temi che si è sviluppato negli ultimi anni è quello delle abilità artigianali. Per sviluppare queste abilità, non c’è come la pratica! Così come i musicisti, i chirurghi, i cultori di arti marziali, anche gli sviluppatori possono imparare tantissimo dall’eseguire esercizi in maniera consapevole. La sessione di Roberto e Paolo, The BDD Live Show rientra in questo tema, e anche Code Kata Live di Gabriele, Antonio e Giordano. Se volete vedere all’opera questi artisti marziali del codice, potete vedere alcuni loro exploit su katacasts. Oppure partecipare a una serata in uno XPUG.

Per ora…

Ci sono altre cose che vorrei raccontare; altri temi che emergono quest’anno sono “come introdurre l’Agilità in azienda”, “Come misurare come stiamo andando”, più altri temi tecnologici come Javascript e NoSQL. Tempo permettendo posterò un seguito domani.

I presented “Framework-free web programming” at the Webtech conference yesterday. Here are the slides. There’s enough text in them to make them understandable, I hope. If you understand Italian, that is :-) I will translate them sooner or later.

TDD is about design. It’s not about testing.

Last week an interesting thing happened while I was working with a team. They had been writing a new application with TDD since last June, and they were successful in that the application was nearly ready, on time and on budget. But there was one thing that was worrying them.

They had been following an article on the Model-View-Controller pattern, and had structured their application accordingly. There was a controller for every window, and there was a model too. (It’s not a web application, it’s Swing.) The view object was extremely thin, as it only contained code for constructing and laying out widgets; absolutely no other logic in there. The controller hooked listeners to the appropriate UI widgets.

So everything was OK, no? A clear architecture, a clear separation of concerns, everything tested. There was a problem: the controllers were big objects, with intricate logic. The tests were difficult to read and difficult to write. It was difficult to understand what was being tested. My worry was that maintenance of that GUI could quickly turn into a mess. Also my customer was worried about that. The fact is, whenever anything had to be changed about the logic of that window, the controller would have to be changed, and the view would have to be changed, and the model would have to be changed. We had three big monoliths that were involved in everything. This ran against the open-closed principle.

My first reaction was to call upon my refactoring skills. Take the unreadable tests and make them readable. Use extract method and expressive names to make them read like a story about what was being tested. Some success there, but not a clear victory. It was still difficult to find the code being tested, as those were end-to-end tests. Then I found the bit that contained the important logic and wrote proper unit tests for it. Some success, as I was able to test with precision all the corner cases that the end-to-end test was not exercising. But still not a clear victory. Then I started to work on the big controller. I took the worst one with me in my hotel room, and I spent a chunk of my evening trying to refactor it. Extract method, rename, extract class. Extract, extract, extract. It was still big and ugly. With a bit of polish, but still big and ugly. Still running against the OCP.

The next day I pulled a different set of tools. I remembered the lessons in Francesco Cirillo‘s Emergent Design Workshop. I showed the team how to use Francesco’s technique, which is to start with a design of the communicating objects on the whiteboard. After one day of practice with this technique, we attacked the real problem. We started designing the main application window from scratch.

The team was surprised, and I was surprised as well, because the big controller evaporated. It was clear, when working at the whiteboard, that it was not necessary. The model object evaporated too. We designed the main scenarios, and we saw that a simple design could accommodate all of them. At first I sketched the design for them, but they were soon criticizing my design and changing it to suit their needs and tastes. Then we started coding; it went smoothly because the design was already done, and we were taking advantage of all the low-level objects that they had implemented already.

We learned is that when the design is wrong, no amount of refactoring moves will get you out of the hole.

My next speaking engagements:

• Webtech Italy on November 10. I will speak on a topic dear to my heart, “Framework-free web programming”. Why do we assume any (web) programming job should start by selecting the framework? My opinion is that what you *really* need is an understanding of the fundamental protocols of the web, and an understanding of object-oriented programming.
• Italian Agile Day, November 19. I will speak about “TDD for web views”, or how to apply TDD directly to the GUI of web applications, rather than avoiding it with tricks like Humble Dialog Box, or substituting it with end-to-end Selenium tests.
• XP Days Benelux, November 25. Together with Antonio Carpentieri I will present a workshop on the Open-Closed Principle Dojo. It’s a way to learn an important principle of object-oriented programming.

Hope I will make it through all of this…