Archive for September, 2017

No Frameworks, Part II

Sunday, September 24th, 2017

Context is king

Engineering is about making tradeoffs. It is about delivering useful solutions at an acceptable cost. As Carlo Pescio likes to say, a construction engineer knows the properties of materials; she knows when to use steel, concrete, glass or bricks. Every material has particular properties that make it more or less appropriate for achieving a desired effect.

In the world of software, application frameworks are a broad category of “construction materials” for application developers. What are, generally speaking, their properties? In which contexts are they useful or harmful? What kind of results are made easier, or more difficult, by using frameworks?

These are all relevant questions, IF you have a choice of using frameworks or not. All too often, developers assume that frameworks are necessary; the idea of not using one is not taken seriously. Indeed, not using frameworks can be scary. If I don’t use a framework, will my design be maintainable? Will my performance be good enough?

Like everything, it takes practice. I argue that it’s possible to have a very maintainable design and a rock solid performance without frameworks. When you learn to do it, you discover that it is not that difficult, and it does not requires you to write mountains of boilerplate code.

Once you know how to do stuff without frameworks, you can take an informed decision whether to use one or not. If you don’t know how, you don’t have a choice.

Frameworks are good for learning

Frameworks are a condensation of proven techniques and design solutions. You can learn a lot by learning frameworks. I, for one, learned a lot from Ruby on Rails. I think we should keep learning frameworks, just for the purpose of learning. When I was developing my first Mac application, back in 1990, I was struggling, as at the time I didn’t know any other Mac programmer. There was no Internet! Apple back then had an FTP server, that was maintained by an Apple employee in his spare time. So at some point I got hold of the MacApp framework; it was one of the first application frameworks to come out. It helped me in many ways; unfortunately, it increased the startup time of my app from 4 to 30 seconds. I was *so* pissed off by this added delay, but at the time, I thought I was better off keeping the framework. I remember I was constantly thinking “the authors of MacApp are very experienced Mac programmers, and they cristallized their knowledge in this framework. If only they had written a *book* instead! Then I could take the bits I need of their knowledge, and leave the ones I don’t need.”

Now I understand that writing an effective book is hard, expecially if your primary skill is programming and not book writing. I still think, though, that frameworks are good for learning techniques and design solutions.

Viewed in this lights, frameworks can be useful training wheels. For me, the goal is to be able to get rid of them and be able to do as good and even better. Once you know how to do that, then using frameworks can be a conscious choice.

Most frameworks are DB-centric

You can generally set up a barebones project much faster with a framework. It’s hard to beat Ruby on Rails on this; with `rails new` you create a working web app, and with generators you can quickly set up a CRUD REST system. Same goes with Spring Boot REST controllers.

Too bad that CRUD does not cut it. Organizing your app around your DB tables is not the way to design an app for a complex domain.

Rails and all the frameworks inspired by it, including Spring Boot, Symfony, Play, etc are all *DB-centric*. They assume that you will design your system primarily around an entity-relationship diagram. This leads to designing for data first, which leads to doing procedural programming. Let me stress this: DB-centric design leads to procedural programming, which is much different from object-oriented programming. Yes, there are “classes” and “objects” in Spring programs; however, “entities” in a Spring program are usually data structures without behaviour, while most behaviour is in service objects that have behaviour but no data. Which is exactly the opposite of object-oriented programming.

Like everything else, procedural programming has a place. It’s generally speaking easier to understand a procedural program than an object-oriented program. Even a bad procedural program is generally easier to understand and fix than a bad object-oriented program. However, procedural programming does not lead to a clean model of the domain, in the DDD sense.

Automating a complex domain in procedural code leads to complex procedural logic, with nested conditionals, and more often than not, with persistence code mixed in with business logic.

True object-oriented code, however, is good for automating complex domains, if you take care to keep persistence well separated from business logic. Moreover, true OO, free from persistence concerns, is good for TDD.

Talk is cheap

Talk is cheap. Show me the code.
Linus Torvalds

In fact, replacing what frameworks do for you may require less than 1000 lines of code. The things that are required of you are:

  • knowing the fundamental technologies of the web: how HTTP works
  • knowing some fundamental design patterns: the share-nothing architecture and MVC.
  • knowing the fundamentals of databases: SQL and database design
  • TDD

The way I do it, is by developing the application from scratch with TDD, focusing on delivering early an end-to-end scenario that is relevant to the customer. Perhaps a heavily simplified scenario, but still something that makes sense for the customer.

I cannot show you the code that went in projects done for paying customers. However, I have some code developed for trainings that I’m happy to share.

The main thing is: these are not meant to be production code. They are versions, sometimes simplified, of stuff I did in production code. Even if I could share production code from real projects, it wouldn’t be wise to copy it in your own projects. Good software design is in how the code evolves; you cannot really see it in a snapshot of the code tree. What I suggest you to do is to try to TDD toy projects until you can do it without too much effort. You can steal some ideas from my training projects, but when you will be faced with real problems, you will have to find your own solutions.

When to use third-party code

Am I advocating that we write everything from scratch then?

This is a question worth answering. There is a world of third-party systems, libraries and frameworks out there: let’s classify along two axes, the first axis being how focused to a single task it is, versus how general-purpose.

The second axis is how easy or hard it is to reproduce the service it does to you; how easily you can do without it.

In the upper-left quadrant we have things that are focused on doing a single thing, but that are easy to reproduce in your own code. A good example would be the infamous left-pad function.

In the upper-right quadrant we have things that are focused, but difficult to reproduce. For instance, crypto libraries like openssl take a huge amount of skill and knowledge to write. Web servers such as Jetty also go up there. Yes, you can write a simple HTTP server in 10 lines of Java, but it’s not likely to be robust enough for production.

In the lower-right quadrant we have stuff that’s general purpose, yet difficult to reproduce in-house. Operating systems, programming language interpreter and compilers, DBMS. All system software goes there.

Now, what would go in the lower-left quadrant, where we find general-purpose things that are more or less easily reproduced in-house? If you have the skill, you can reproduce most services that application frameworks do for you without a lot of effort. Mind you, I’m not saying that I can easily rewrite Spring! I’m saying something very different: that I can reproduce with ease what Spring does for me.

  • Routing http requests to the appropriate controllers: it can be done in-house, as simple as chain of IFs, or as sophisticated as a ~100 line router.
  • Injecting dependencies in objects: I can very well instantiate objects explicitly and pass their dependencies in constructors. It does not take a rocket scientist.
  • Automated CRUD “repositories”: thank you, I don’t need them. A real repository in the DDD sense is something that encapsulates custom persistence logic. I’m better off implementing it with a simple adapter over JDBC. Carlo Pescio suggests to use dbutils; I generally TDD my own.

Etc. etc. In this sense, Spring and similar frameworks are things that you can do without, if you want to.

Application servers such as JBoss too are general-purpose, also in the sense that they provide a set of different, not-that-much-cohesive services. Most of these services can be obtained in better ways:

  • HTTP is best implemented with an embedded web server, as the popularity of Dropwizard and Spring Boot shows.
  • Clustering as traditionally provided by application servers is best avoided. It is built on the assumption that web apps maintain expensive session state between one HTTP request and the other. In fact, only poorly designed web apps do that; well designed web apps are stateless or nearly stateless, and if this is the case, they are best served by non-clustered, independent servers. This, in fact, is the “share nothing” architecture.
  • I argued in the past that distributed session management in Tomcat is broken. The same arguments is likely valid for most Java application servers.

For these reasons, I place JBoss and other JEE application servers in the lower-left quadrant: you can do very well without them.

                    focused on a single task
                               |
                               |
                     dbutils   |   Jetty
                    left-pad   |   openssl
easy to                        |                    difficult
do without  -------------------+------------------- to do without
                               |
                      Spring   |   Linux  JVM
                      JBoss    |   PostgreSQL
                               |
                               |
                        general purpose

Now what should be our strategy? Of course, everything on the right hand side should be bought and not done in-house. Unless the value of your system depends, let’s say, on advances in cryptography or DB technology that are not generally available, doing system software or crypto in-house is madness.

Stuff in the upper-left quadrant is a candidate for writing in-house. You can save some time by using dbutils over TDD-ing your own JDBC adapter. However, every dependency on an external library is a liability, as the left-pad incident shows.

Stuff in the lower-left quadrant is probably best avoided, if you have the skills. They have the highest impact on your design and reduce significantly your control over the fate of your system.

Other objections

You make the customer pay for reinventing the wheel

In the end, what really matters is to delivery quality software in reasonable time. I found that by avoiding stuff in the left half of the easy-focused diagram, teams are able to do so. If that involves some rewriting of common logic, so be it. It turns out that universal wheels such as Spring do not make me go faster.

By your reasoning, we shouldn’t use compilers or standard libraries

See the section above

You’ll write your own quirky framework and it will suck

No. I will not write my own framework. I write just the code that I need to solve this particular customer’s problem in this particular moment. I will spend zero time writing code with the expectation that it will be reused elsewhere, not even in other projects that I will do for this customer, not code that I expect to reuse in this same project next month.

I take YAGNI very seriously. I TDD the simplest code that solves the problem I have today. It’s counter-intuitive, but this makes me go faster. Every time I wrote code for tomorrow, I found I wasted my time.

This does not mean that I write the first thing that works and then move on. I tend to refactor quite a bit before I decide I’m finished with a bit of code. My code is generally very simple and clean as a result of quite a bit of effort. A side effect of this is that my code tends to be flexible and easy to extend.

I repeat: the simplest thing is often the result of sustained research and effort. It’s not the first thing and certainly not the easiest thing.

New developers will have a hard time understanding the codebase

That depends on how good my code is, isnt’t it? If you write simple code (see above answer), it shouldn’t be too hard to understand. On the other hand, even apps written with frameworks are not always that easy to understand.

Much wiser and skilled developers wrote the frameworks that you’re despising

Their circumstances may be different. Their programming habits and processes are probably quite a bit different from mine. I really don’t care how experienced, respected or influential framework authors may be; all I care about is the results I get. I evaluate the quality of a solution with respect to how well it helps me reach my goals.

Conclusions?

Engineering is about evaluating options. It seems to me that developers discard the “no-frameworks” option without giving it a lot of thought. In my experience, without frameworks, I was able to help team deliver high performance, high quality code in a short time.

Your circumstances may be different. Don’t trust me; think for yourself. Think critically about technologies and design solutions; and if you care about your craft, try to de-framework an app and see what happens!

Posted in Agile | 1 Comment »

No Frameworks, Part 1

Tuesday, September 19th, 2017

Twitter is not good for deep conversations

In a recent talk, I told the story of a very successful software project where I was the tech leader. In this project we built a web service + web application, in Java, without using any framework: no Spring, no Hibernate, no MVC framework, nothing like that. Contrary to prevalent opinion, we delivered quickly, faster than even I had believed.

And the code was reasonably clean and maintainable: five years after I left the project, many developers told me that they liked working on that product, and it was a formative experience.

At the conference, I quipped that “not having to spend time on Stackoverflow to solve framework configuration problems saved us a lot of time,” and that indeed was part of the reason why we delivered fast, but it is not the whole story. Common reactions to #noframeworks include:

  • You make the customer pay for reinventing the wheel
  • By your reasoning, we shouldn’t use compilers or standard libraries
  • You’ll write your own quirky framework
  • Developers will have a hard time understanding the codebase

All of the above objections are wrong, if you do things the way I recommend, which is, by doing Extreme Programming for real. Doing XP for real, for me, includes focusing and investing on simplicity, in a way that most developers are not usually willing to do.

Applying lean thinking to software development

Lean thinking says that we should optimize the end-to-end value stream, from when a customer need is identified, to when that need is solved; if that need is to be solved with software, the value is obtained when we deliver working software in a production environment.

What should we optimize? Obviously, for cost and time. However, since costs in software development are dominated by personnel salaries, the cost is dominated by the number of person/days spent to obtain value: hence, we should optimize for delivery time.

Once you know what to optimize for, lean thinking suggests that you observe the software delivery process, looking for waste, and act to remove that waste.

Terminology: In Lean, an activity is classified as “value adding” if it directly contributes to building the value that customers want; it is “waste” if it does not. For instance, if you’re building a chair, cutting and shaping the wood are value-adding activities; moving the wood from the warehouse to the worktable is waste. Sometimes it’s difficult to decide if an activity is waste or not. In this case, a simple thought experiment can help: if we did the activity more and more, would the customer be happier? Or not?

[Waste is further classified in “necessary waste”, something that if we don’t do, we will end up making the customer pay more, and “pure waste”.]

In the world of software development, writing the lines of code that will end up as part of the solution is certainly value-adding. On the other hand, a standup meeting is waste: it does not directly contribute to the delivery of software. If we imagine the standup meeting being extended to two hours or a full day, we clearly see that it would not make the customer any happier :) The standup meeting is probably *necessary waste*, because if we don’t do it, we reduce communication within the team and we end up creating more waste elsewhere. However, it is still waste, and we should try to keep it short.

There is a world of waste in building the wrong product, or the wrong features: you should certainly look in the disciplines of Lean Startup and service design, to make sure you are solving the right problem. I will not talk about these because they’re well covered elsewhere.

The kinds of waste that I want to talk about are the things that take away the best time from developers. Picture this: the developer comes to work; perhaps he’s 20 minutes late because of traffic. He comes in, settles down, checks email, has coffee with his collegues, he chats a little. Nothing wrong with this. Then he has a standup meeting, and perhaps other meetings for discussing planning or technical choices. All of this is still acceptable; they are not value-adding activities, but often they are necessary. Then comes the magic moment when the developers are in front of the keyboard, with their code open in the editor or IDE, and they are ready to do the value-adding activities that we care about. And they write a line of code, which requires, say, one minute, and they need to check if it works; so they start the application, and they wait; because the application takes one or two minutes to compile and start up. They spend another minute logging in the application and clicking through to get to the screen they are working on, and they confirm that their line of code works (or not.) Then they write another line or two of code, (time required: one minute) and again they need to check if they work, so again they waste three minutes restarting the app and testing it manually.

They could try to save time by batching changes: write 10 or 100 lines of code, and then test if they work. Alas, this does not usually save time, as the number of mistakes increases, and the mistakes interact with each other, and the end result is that debugging 10 or 100 lines of code takes significantly longer than debugging 1 or 2.

The point I’m trying to make is that the compile-link-start-login-navigate-to-the-right-screen cycle eats minutes from the best time that developers have: the time when they are sitting at their workstation, ready to write the lines of code that they know they have to write, having completed all the meeting obligations and coffee breaks and all other necessary wastes.

When you start looking at the work of developers in this way, you start to see the value of dynamic languages, where building the app is istantaneous, and the time saved in this way may trump the supposed gains obtained by static typing. You also start looking at the long compile times of C++ or Scala, and you see that those languages were not designed with the need of fast software delivery in mind.

You also start to see other wasteful activities. When a framework does not do what you want, there is often not a lot of reasoning you can do. The framework is too complex for you to debug it; the documentation is too vast, or too incomplete, and you know from experience that looking there is not likely to provide a solution quickly. So you go to Stackoverflow and hope that someone else has solved your problem. Then you try a possible solution, and if it does not work, you try another. All of this is, of course, pure waste, borne from the fact that you are using complex frameworks that may behave in unforeseen ways.

If, on the other hand, you have a problem with a simple app that does not use a framework, you can most likely solve it by reading the code, or stepping through the code in the debugger. All of the code of the app is your own code, and you can step through it, with no jumps into framework code, or AOP-generated code like that, for instance, Spring or Hibernate put into your app and that make debugging much more difficult. The solution to your problem is in your own code, not on Stackoverflow, and the size of your own code is many orders of magnitude less than the code in frameworks.

Getting back to the length of the compile-and-startup cycle, we see that frameworks do not help: Spring apps, for instance, take minutes to start, because they search the classpath for configuration classes. On the other hand, a no-framework Java web app with an embedded Jetty starts in under a second, and when you debug it in the IDE it has nearly-instant reload of changed code. No need to install JRebel. You don’t have to believe me; just try.

Other framework-related wastes include:

  • The time required to learn it.
  • The time required to upgrade your app when a new version comes up (this is work that the framework developer inflict on you, without your consent)
  • The time required to upgrade an old app that was using an outdated version of the framework (this is expecially bad when vulnerabilities are discovered in an old framework, forcing the customer to pay for upgrading a framework that is old and by this time poorly supported. Pure waste, as there is no business reason to change the app)
  • Time required to argue on the relative merits of framework A vs framework B, which leads to the mother of all framework-related wastes, the Meeting To Decide Which Frameworks To Use :)

Now this is all about the wastes of frameworks, and we haven’t even started to look into the risks generated by them. In the next part, I will talk about how the benefits of frameworks, and how you can get the same benefits without using them.

Posted in Agile | 9 Comments »