July 30, 2006

Re: The "Snakes on a Plane" Problem

Found an interesting article on Reddit:
The "Snakes on a Plane" Problem


I worked in newspapers for eight years, right when that industry was starting to disintegrate. As such, we spent a lot of time talking with focus groups, forever trying to figure out what readers wanted. And here is what they wanted: everything. [...] When it comes to mass media, it's useless to ask people what they want; nobody knows what they want until they have it.
Why this sounds all so familiar ?

July 27, 2006

One other reason why new software is released

Because the users keep using the old one in the wrong way.

July 24, 2006

Reusable business-specific abstraction is an oxymoron

In this article I'd like to talk about abstractions in software development, focusing specifically on business-specific reuse.

Abstractions come in many flavours. At the most general level, abstraction is the single most valuable method of analysis. Mathematics is all about it. Abstraction is also the result of that analysis. I will be using the term "abstraction" as if it meant "a concept relevant in particular context". In simple terms abstraction is "something that ..."

In programming, abstractions appear in many places. It's impossible to discuss all of them, so here is a few random examples before we proceed to the points of the discussion.

The simplest abstraction in programming is the byte. The concept of byte suggests that (1) all data we manipulate with can be converted to numbers and (2) the numbers can be encoded using fixed width integers. Although bytes with size different from 8 bits have been experimented with, they were not successful (not that I saw CDC Cyber myself, but the books say so). Let's think why all the bytes have converged to 8 bits. My answer is - because there is no difference. They could have been of any size, 8 bits were just a good balance of range/processing power vs. costs to produce at that time. As the costs kept falling, the bytes actually grew (although under a name of machine word now). Anyhow, the byte became a universally accepted abstraction, reused literally everywhere.

Abstractions also appear in OOP where they hide in the shadows of the objects themselves. After all what is an object if not an abstraction of some real-world entity ? OOP is very successful in giving developers abstraction producing machinery which they apply to their problems yielding abstractions directly translatable to programming languages. Not all the created abstractions are reusable, leave alone useful, but OOP itself is.

As we can see, low level abstractions can become quite successful and widely adopted. This is because they are so "abstract" they apply everywhere out of the box, and changing them would make no difference, and so nobody even thinks about why they are there and what's their purpose. They are just there, they've always been (for an eternity of fifty years), they are the truth. Although this approach is far from scientific, it works well in production. After all, why do I need to know the details of the chemical process for producing plastic, as long as all I need is to wrap my groceries in a plastic bag ?

And so, my first point - easily reusable abstractions are invisible. We use them without ever noticing. "Why ? Of course our data is encoded in bytes, what a stupid suggestion !" Low level abstractions are such, but hardly any others, at least right now.

As we move farther towards the "real business" side of programming scale, it's increasingly difficult not to notice the abstractions arising more as obstacles than helpers. One step up, look at what building blocks we have as we build real systems. No matter if it's 1980s or 2000s, the abstractions that your programming environment offers are the same:

  • User. Session. Context. Authentication with plaintext username and password (anything else is still out of reach because of interop problems).
  • Database (whatever make, hard to design). Connection. Pool (not always useful). SQL query (hardly anyone knows how to write one that won't bring the server to its knees). Transactions (requires even more knowledge).
  • Component framework, networking, distributed facilities, RPC, messaging (various compatibility, performance and scalability problems difficult to address).
  • I18N: Encoding (Latin-1 as the mostly useful one, interop hell again).
And on and on it goes. All those "development" abstractions are useful and widely accepted but still are technical, low-level and because of their complexity only reusable as a mindset. From this point of view, they are patterns more than real tools.

Two steps up, when it comes to real business, there are no abstractions. It has to be done in exact particular way. Nobody's going to use an application if it doesn't make precisely what the user wants. That's why all the software is to some degree configurable and adjustable. The rigid abstract part of the software is clearly separated from the flexible configuration part. The abstraction therefore has nothing to do with the business specifics - it's all in the configuration. And if you are lucky, you've got enough flexibility to cover everything.

Now, I'm asking you this - wouldn't then a "reusable business-specific abstraction" be an oxymoron ?

For one, "business-specific abstraction" is already nonsence. Abstract has its details stripped off and replaced with imaginary concepts. But details are the heart of the particular business, no matter if it's called "valuable proprietary know-how" or "ugly poorly understood mess". And it makes difference. And it's highly visible. If you are switching from DCOM to SOAP, nobody cares. But if you mishandle customer's data for the sake of abstraction that you might have - well, they won't approve it. It has to be done their way or no way. If your abstraction is too rigid - too bad for it.

For two, "reusable business-specific" is also nonsence, as soon as you are crossing enterprise boundaries. Unless Big Brother takes over, every single business is going to be to some extent unique and as such will not be covered by the abstraction. Even if you are borrowing a clearly separated network service, not burdened with thousand dependencies, it's likely to be useless without tweaking.

And so my second point is - business specific software can hardly be abstract and hardly reusable. In fact, the most valuable property of such software is flexibility. It has to allow easy changes, no matter what abstraction has been put in it.

That's one of the points behind the Pythomnic project - allowing developers to build network services that can to any extent be changed on the fly, without service restart and as soon as you have read this far I invite you to check it out.

July 22, 2006

Always write what you really think ...

... because if your thinking is right, so will be your writing, but if your thinking is wrong, it doesn't matter what you write anyway.

July 06, 2006

Want to automate your business ? Want a single-click software ? Prepare for changes.

Very few people actually love the work they do. If it's somehow done magically with no concern of theirs, hey, they'd rather be fishing ! People have always tended to avoid work. Having somebody else to do one's job is the best strategy under the sun. Even for those who are not very sick of their job today, and pretend to be doing things, the less actual action - the better. Laziness is a much better driver than fictional pursue of productivity, urge for business success or corporate loyalty.

Now, people don't need software at all. Like I said, they need their job done, preferrably with no load on themselves. But software was giving such promises about how much easier it would be (if you purchase product X) since day one. Not that many still believe that new version of Microsoft Word will boost their productivity, but hey, this is how documents are prepared in 21st century, ain't it ? It's just that you install it, boom ! and your documents "practically manage themselves" (a ridiculous sentence if you ask me).

Therefore users dream of software with one big button reading "Get my job done now" (even then there would be some moaning about how much hassle it is to push it). It is indeed crucial to design software in such a manner that it requires as little user intervention as possible. But doing so is very tricky.

Let's say we are developing an accounting package. The first problem we are going to face is ubiquitous lack of knowledge. It probably wouldn't surprise you the least that someone with degree in CS knows nothing of accounting, leave alone specific customer's business practices. What would surprise you for the first few times is that the users themselves don't know what they are doing, no matter if they've got a degree in finance.

We are likely to see that everyone knows a little bit but nobody sees the big picture. "It's just how it's done here, that guy over there probably knows something" is the sort of speech you may hear. Blind leading the blind, at its worse.

A good business analyst can save the day, no doubt. Since a good business analyst essentially is a person to some extent trained in both areas, experienced in exactly that kind of projects and (which is crucial) clever, fluent and clear in his thoughts and talks, he could definetely talk things out.

But the business analyst will actually be busy recognizing an elephant from descriptions given by three blind wisemen. Or, sorting out all the quirks of the customer's business practices, which is essentially the same thing. Even if (and it's a big if) he succeeds all the team gets is a 10 sq. ft. diagram with lines going in all directions. Better understanding, perhaps, but of what ? And this is where the second problem hits - ugly and tangled business practices.

Well, the customer is always right and all that, but what's the point in one-to-one mapping software functions to existing business practices ? Nobody understands how it works, I see. Now you have software that automates that ! I mean, it's just outrageous. You take a can of worms, sort of peek into it, then, using whatever perfect (huh ?!) software development process, implement the automated can of worms and hand it over to the customer.

"Eeew !" - that's what the user says - "I thought this software thing would be simpler ! Where is the "Get my job done now" button ?!" Oops, there is no such button. What you get is a button for every chaotic activity you have performed before the software extravaganza started.

And so my point is - for a business to benefit from software automation, it must work well enough even before and without any software. Chaos cannot be automated, that's what I say. As there is hardly a perfectly organized business out there, well, it needs to be changed before it can be automated.

The good news is - those two processes can be performed in parallel and greatly support each other - straightening (uh, streamlining is so much a buzzword) the existing business practices, and analysis of the same for the sake of software development. If there is a perfect moment for changes, it's now - there is a lot of efforts put into it from the software developers camp, and people genuinely interested in understanding and making changes will be around, not the stupidest people too. All the customer needs is to allocate decent resources and get ready to change.

The bad news is - a customer is more likely to remain heavy and unresponsive. "There, we've paid you, now where is our button ?" Still, as you know, changes come, now or later, and it's better to change it right now than to postpone.

To conclude, I believe we as software developers should be working not just on software but also on improving our customer's business rules, it's in our own best interest. Moreover, we should be pushing the customers to changes, not just accepting their requirements.

July 04, 2006

Open source, closed source, can anyone tell a good piece of software from a bad one ?

To start the discussion, let me ask you this - shall we release early and often ?

If we release early, we are likely to release crap, but have better exposure. But people are already used to software being crap. Besides, crap or not, there is no easy way for users to tell.

But if we go the long way of making the software better as we see it, and release later, the quality might be better, but it's still expected to be crap (release 1.0, anyone ?) and even though the experience might prove otherwise, the impression is still about the same. Besides, whi will care ?

Paradoxically, we are pushed towards producing crap faster.

Again, the key problem here is that we have no way of evaluating software quality. The closed source used to rely on the quality signals that, once been in favour have long since sold out - brands, advertisement, experts, reviews. The open source may rely on community judgement, popularity and zealotry, but what is it if not bandwagon effect ? Is it any better then ? I don't think so. Both sides rely on experience, but then again, experience is nothing but a way of convincing oneself.

Anyhow, my point is - open source, closed source, both parties have their reasons, politics and propaganda, both are eager to sell, but not only neither knows a way of producing high quality software, nobody can even know for sure if any given piece of software is good or not.

Think about it. We make stuff. We are proud artists (each a Picasso, no less). But when presented with somebody else's work, we cannot tell its quality. Isn't it outright ridiculous ? What kind of an industry is that ?

We thus are tackling the wrong problem. We need to build quality things. Instead we invent different ways of building things of unknown quality.

We desperately need reliable indicators of software quality.