The Right Tool for the Job?
The seemingly self-evident advice to “use the right tool for the job” is more subtle and more complex than it first appears
What I want to explore is this idea that you should “use the right tool for the job.” We see this phrase used in software development. We also see it used elsewhere, in other disciplines, but I’m going to focus on software development for now.
The intent of it is self-evident: if you’re doing something, then do it the right way — do the right thing, in the right way. The problem is that once you scratch the surface of that advice and deconstruct it, you discover it is anything but self-evident. It is not at all obvious.
Assumptions and the limits of knowledge
So, the first question, the first challenge, the first assumption we encounter is the idea of the right tool. When we say ‘the right tool’ it makes it sound like there is uniquely one — a chosen one that it is self-evident from all the other possibilities.
Perhaps, if what you’re doing is simple enough, and your options are reduced enough — maybe two — then there is one of those that is more ‘right’ than the other. But we should already be on our guard: right is not the right term; there is more appropriate or less appropriate. What defines appropriateness is based on a number of factors.
If you’re dealing with tools that you know versus tools that you don’t know, how do you know that the things that you don’t know may be better? And even if somebody urges you and tells you the tool you don’t know is better, you may have preconceptions that prevent from taking that step into the unknown.
You also have knowledge-acquisition time. If acquiring that knowledge is going to be more expensive and take longer than is proportionate for the task in hand, then that’s probably not the right tool for the task.
When we think about ecosystems of software, when we think about programming languages — when we think about all of these things — it is not obvious that the right tool is an obvious thing.
Recognising it when you see it
A number of years ago, when I was working in a bank, one guy came to me with a particular problem related to text. He had been told by one of my colleagues that he should be thinking about preprocessing.
“Yeah, that sounds reasonable,” he said, to which I agreed. “I was recommended to use the C preprocessor.” Which I did not, however, think so reasonable.
It felt like the wrong tool: he was dealing with free text — text in English — that had a semi-formal structure. The C preprocessor has a relatively narrow scope of applicability that is ill-suited to general text preprocessing. Indeed, people working in C and C++ have horror stories about how appropriate — or inappropriate — it is even in those contexts, and how troublesome it can be.
I recommended he use something different, the lesser-known Unix tool m4, which is actually designed for more general preprocessing, is highly configurable and will work with regular text. He’d never come across it, which is not that surprising — how would he have known there was a better option available unless he asked?
What was more interesting was the reaction of my colleagues who’d made the first recommendation. They came to me and said, disapprovingly, “Somebody’s gone and recommended m4.” It turns out that of my two colleagues, one of them had never used it and the other one had had a bad experience. Understandably, we are often influenced by our experience and preconceptions. The bad experience — and false negative — probably came from using it for something it wasn’t appropriate for, or they had gone in ill-prepared, underestimating the learning curve.
The self-evidentness of what is the right tool is not always obvious. First, you may not know there are other tools that could be better. Second, you may not know what ‘better’ looks like.
The wrong tool
In some cases, the right tool is only obvious in hindsight, and with that greater and learned context that comes from time.
There are some things, however, we can say with confidence. I’m old enough that the first proper paying job in programming I had after university was in a place that used some very traditional languages — Fortran, COBOL, some variant of BASIC and PL/1 (but not so much).
This place did not use C when I started. One of the main things we did, however, was systems programming — the work involved factory automation, among other things. One thing I can say with hindsight is that Fortran really was the wrong tool for this job. It is a slow and painful way of doing something that is far better done in other languages. Of the languages that were available, it did seem the obvious one. During my time there, however, I did start using C.
The sensation of using C, having used Fortran, was the sensation of no longer hitting my head against a brick wall. In this context, C was a perfect match. It’s not only a better designed language overall; its design is intentional rather than cobbled together. If you think I’m being hard on Fortran, I recommend looking at its history — it was put together by John Backus and his team to prove a point (that a compiled language could rival the efficiency of hand-coded assembler), with little regard for anything that did not contribute to that goal. The point about Fortran is that it was singularly unsuited to a number of things — including structured programming, data structuring and systems programming — all of which were relevant to the context we were in.
Once you’ve broadened your scope, you can say “Oh yeah, there are better options,” but they are not immediately obvious. That’s defined partly by our experience or inexperience. That’s why we should draw on the knowledge of others — certainly those we can trust — but we also need to weigh up against other factors. For example, what is the training and learning cost? I’ll be quite honest that, back when I started, my C looked like C-tran — C spoken with a very heavy Fortran accent. Nonetheless, the improvement was huge.
Stepping back
In many cases we can step back to make a clear-headed judgement based on what we already know, but we often don’t allow ourselves to do that.
A team of four, that I visited a few years ago, had spent six or seven months with some Java code, screen-scraping mainframe output. This was all text manipulation. This is not Java’s strength, which is in its ecosystem, its universality and a number of other aspects; text processing is not where I would put its strengths.
After spending some time with them I came to the conclusion that “You guys would have been a lot better off had you just had a team of two and used Python. You’d have been done in just a handful of weeks.”
Although aware of Python, they didn’t have experience in it. That was one obstacle, but their culture was the larger obstacle. The culture of the place was large projects, so people were immediately suspicious of anything that was small — four people over some months was already small by their standards. They were also not as familiar with dynamically typed languages or anything that feels and looks like a shell language. That was not their culture: Java/C#/C++ was their culture. Shell and dynamic languages were only glue languages; they were not considered to be principal, first-class engineering languages. Theirs was not a culture of exploration and experimentation — “I wonder if Python might be better for this? Let’s spend a couple of days to find out.”
When we reduce the job to something simple, sometimes it is easy to tell what is needed and what is not, i.e., what are the characteristics that we want. We often don’t pause to make the evaluation that leads to a trade-off.
Jobs versus tasks
There’s another problem we encounter with the right tool for the job. Let’s be very clear: tools are not for jobs; tools are task-oriented. That’s a very subtle difference. Most of the time the distinction doesn’t matter. There are indeed occasions when we use the term job and task interchangeably, and some jobs are reducible to one or more tasks.
An example of that, many years ago, was a simple, small application that I developed. It was reducible into three basic parts, three basic paradigms, and there was actually a tool for each ‘job’, i.e., task. I used shell scripting as the co-ordination language — the argument and file handling, the options handling. That was the entry point. I used Awk for the text formatting and handling the content and structure of the file. For the numerical and computational part, I used C. All of these could be arranged in a simple pipeline; it was very task-oriented.
Sometimes our tasks really do dominate the job. But in many cases, they don’t.
When we say a programming language is a tool, we need to understand that’s a metaphor, because programming languages have typically many degrees of freedom. When people talk about tools in the real world — things like hammers and screwdrivers and so on— they’re talking about things that have relatively few degrees of freedom and apply to simple domains where the consequences are obvious. Software development, on the whole, is almost exactly unlike this. Of course, there are certain cases where there are things that are more or less obvious and we can equate a particular domain with a particular shape of problem and, therefore, correspond it to a tool that is finely tuned to that. Most programming languages have multiple dimensions of possibility. They are general purpose: they are not simple tools; they are collections of tools.
On the job
The right thing is looking harder and harder by the second — there’s a lot of context we have to take into account.
But there’s also one other aspect: even when we’ve recognised that tasks are not necessarily jobs — jobs are composed of many tasks, tools are primarily geared towards tasks — and that our design challenge is far larger than this apparently self-evident statement suggests, there’s something else we need to account for: how do we know what the job is?
A lot of things in software development are based on the fact that we don’t know what the job is in advance. The very act of development is what reveals the job. If we try and make all these decisions up front, we are doing so from the position of greatest ignorance. When do we know the least? Right at the start. When do know the most? Later. And, if it is actually a project, then it has an end point — that’s when we know the most. But we made all the big decisions back at the start about the right tool for the job.
Our preconceptions about what the job involved have shaped — perhaps distorted — our work and prevented us from using the right tool.
