Friday, October 23, 2009

What is Wrong with Local Optimization Anyway?

How could it be wrong to optimize anything, local or not?
Well, if by local optimization we mean having a resource in our system utilize an optimum amount of its inputs, to produce timely, sufficient, but not excessive, output to subsequent steps in the process, then there is nothing wrong, as long as this optimization contributes positively to the system's goal.
Note that timely, sufficient, and not excessive, output is defined by subsequent steps in the process. As such, this output might, at times, be zero.
Note also that optimizing the whole system may call for one step or process to be removed altogether.
If this is how we are approaching the problem of efficiency, then we are not actually doing local optimization.

Consider, however, the following approaches to optimization:
  • Increasing the resource utilization to 100%.
  • Getting the maximum possible throughput out of every resource.
  • Keeping everyone busy all the time.
  • Removing all idle time.
If this is our focus, then we are heading for trouble, and we are introducing a significant waste in our system.

To see why this is the case, consider the following consequences of increasing a resource's throughput in our system to the maximum:
  • More inventory to manage in subsequent processes, If these subsequent processes are not ready, or capable, of consuming all the output.
  • More load on subsequent processes, since now they will have more input to process.
  • Delays in getting urgent work done, since there is no slack in the system to handle occasional spikes, resulting from natural statistical variations.
  • More work being stuck at bottlenecks.
  • Increasing demand artificially on up-stream processes, since this demand is not driven by the needs of the market or the ultimate customers.
  • Increasing demand on resources required to maintain the high efficiency.
  • The process of optimization itself will consume resources. The overall gain may not exceed the cost.

Local efficiency, then, is a waste. One has to look for alignment with the system's goals to define what, where, how, and how much to optimize, weighing costs against benefits.

"But wait," you may remark, "how about local optimization at bottlenecks?", which is, granted, a nice try. But this will have to wait for another post.

Wednesday, October 7, 2009

Does Waterfall Make More Sense?

I came in contact with a few people who were actually content with waterfall.
A senior dev explained to me that waterfall is simple, everyone gets it, and it's easy to implement. There are well defined, easy, consecutive steps to be followed.
An upper manger was very keen to find ways to convince his company's leadership that waterfall fits his department really well, thus avoiding the drive to adopt agile. From his perspective, waterfall provided predictability. He new at the beginning of the year what his budget is, what projects he will be working on, and what the duration of each projects will be.

It also makes sense to design something before building it. If you don't design it before hand, how do you know what you will be building? How do you know how much it will cost? How do you decide if it's worth it? How do you compare it to other options?

In our day-to-day life, we demand predictability. Before we offer a job to a carpenter to install new kitchen cabinets, or ask a mechanic to service our car, for example, we want to know before hand how much it will cost, and how long it will take. We are really disturbed when either of these estimates are not met, although we know they are just estimates.

So what is the problem in expecting the same from software projects?
We can always give the example of an apparently simple job gone badly, as when an air conditioning engineer starts asking you when was the last time you cleaned your air vents, or changed the air filters, only to discover that you'd have to pay more and wait longer to have your air system fixed. Let's put this example aside for now.
Instead, consider the how likely is the change in your project, from inception to project end, in the following areas:
  • The business needs from your application.
  • The specified requirements from your application.
  • Your understanding of the requirements.
  • Technology.
  • The project team's mastery of the technology.
  • The people who are doing the work.
If your meter reads anything other than low for all the above, you should rethink waterfall.
Because waterfall makes sense for certain types of projects. Software development projects are a breed of their own, with a lot of sub-varieties within.
And change is inevitable in software projects, because you'll never build the same application twice, for the same business need, with the same people, who have the same experience, using the same technology, will you?