December 30, 2006

Why having processing power wasted ?

I don't know about you, but seeing unused hardware troubles me. Doesn't matter if it's an old 386 desktop or a 8 years old Compaq Proliant 7000 which I've recently acquired.

Hardware is always fun to set up, see it running and play around with. Folklore about 100s refurbished Pentium's making up a cluster in a garage fascinates me. I do realize that a single modern server will eat that hundred for breakfast, but still, wouldn't that be fun, running that garage ?

And so, my working PC and the servers that are in testing or spare, are normally loaded up high most of the time, doing what - running workload tests of course ! Right now I'm leaving it running a stress test for Pythomnic, specifically its new capabilities of distributed transactions with recovery. The test will be running for a few days now, and I hope when I return from holidays it's still up and running.

December 25, 2006

JMS: lock on Java

I have to admit, I have only a basic knowledge of Java. And I wouldn't ever touch it. For some reason, when writing in Java, I feel like swimming in tar - possible, but hardly enjoyable. There may be some hidden beauty beyond my knowledge, but I simply don't want to spend time on getting deep into it, so deep that that beauty-of-possibility is revealed. And frankly, I don't believe there is one.

But then "industrial standard" kind of arguments bite, and all of a sudden there is a significant development around in Java. It's totally beyond me why solutions have to be adapted to technologies, not the other way around, but hey, it's real world.

And so, here comes another industrial standard - JMS. But, if you ask me, JMS is no industrial standard. It's a standard way of implementing messaging in Java world. In other words, JMS is a perfect example of a platform-specific technology, created with one thing in mind - to lock the developers to that platform. Many of the J-things are such, don't you think ?

See, JMS is Java Messaging Service, which asserts right from the start that anything non-Java is out. Should its designers have cared about interoperability and ease of integration, they would have standardized an open network protocol. How much easier life would have been if there was simply generic Message Queueing Protocol, say MQP based on TCP, optionally UDP, with very simple and clear semantics for swapping plaintext (i.e. XML) messages. By now we would have had tons of implementations in different languages, of better quality too. Not the case. Not only JMS transport may be based on RMI, Java-specific thing by definition, it's also focused on transferring instances of Java classes for messages and/or using messages for Java object to Java object communications. A perfect lock on Java. Pity.

Here is one more weirdness - in order to connect to a JMS server you have to know precisely what kind of JMS server software is run on it. Imagine that you'd have to install different software for sending your e-mail via SMTP server running Sendmail vs. Exchange. Sounds awful, but not for industrial standard it's not. What it means is two things - first, the JMS specification failed to provide implementation independence (I'm not sure that it was even supposed to do so), and two, with JMS you have to lock yourself not only to Java, but to particular server implementation, which is even worse. And I mean - theoretically, you should be able to modify your classpath, client configuration file - pf-f-f-t - and switch to a different implementation transparently, but then again - you have to explicitly switch to a particular server implementation, and different servers may have slight differences in behaviour (just love it), hence your clients may break anyhow.

What kind of industry requires such standards I wonder ?

December 06, 2006

Moscow: exploding buildings

They indeed failed to explode demolish a condemned building in Moscow. I told you.

December 05, 2006

Experience matters

Everything in software can be done in so many different ways. As my favourite saying goes - there are many ways to skin a cat. And just like with the cat, any solution suffices as soon as the problem is solved. The "any solution goes" maxim is valid and very powerful tool for a pragmatic software developer. But there are points to consider.

For one, it's nearly always that we see the problem wrong. It's either lack of knowledge, perception bias or just plain misunderstanding. In this case we are simply solving the wrong problem.

For two, the problem may be a part of something bigger and the way we build the part may render impossible building the whole.

For three, and this is almost universally true, the problem at hand may be a special case of some other, more abstract problem. Now the solution that we use should be evaluated against this concrete/abstract scale.

Besides, solutions are very fragile. You make one wrong decision and it falls apart.

Given all the above, how one manages to ever build anything ?!

My answer to this question and the subject of this post is that experience matters.

To me, a "I tried it and failed and this is why" is better than "I did it by the book and it worked". The former is a sign of a creative mind that just took a wrong turn, but (and it is very important) was able to accept the failure and find out the cause. The latter is someone who believes in universal solutions and silver bullets and never asks questions. The paradox of the situation is that the he still has a working solution this time around.

It's not like failures are valuable all by themselves. It's the experience you bring out of them. Then, like I said before, you have to actually work on your experience. Experience is not the same as the sum of all your failures (and successes for that sake). Deep analysis is required to turn individual failures and successes into valuable experience.

One other problem is that experience is very personal, even intimate thing. The mentioned analysis is often subconscious and results in something that one may call intuition or "gut feeling".

But you rarely work alone and for the team, what's experience of its individual members ? It cannot be transferred from one to another and it cannot be enforced. Try convincing somebody else in your solution using "my experience is that" or "I feel this is right" kind of arguments. "But I feel different" - this is what you are going to hear.

Individual experience then can only be effectively used within a team if you have worked an essence out of each of your successes or failures and are capable of reasoning and giving arguments that never mention experience at all.

The other use for experience of individuals is have the team highly specialized, or even have single leader and support team (a-la Frederick Brooks' surgical team). This may lead to better results faster but yields less for the group development.

Besides, the team can and should have experience of its own as a whole, but it's another story...

October 07, 2006

"Software as a floor mosaic" metaphor

Here is a "software as a floor mosaic" metaphor: let's say we need to inlay a floor mosaic. We have a sketch or may be even a highly detailed plan of how the thing has to look like, and even the vision of it and clear and perfect, something like this:

We also have at our disposal assorted pieces of stone, ceramic patches, coloured sand and all the tools we might need. But no matter what tools do we have, it takes experience and above that a lot of time to make such a mosaic. Yes, experience helps but we still need to take each piece, adjust it, finish it and put it in place. Sometimes the piece will crack in our hands and we have to start over. Sometimes, a piece will be very different from what's needed in that particular spot and require much more work than another.

But then there is time pressure - the worst of all curses. Pressed with time we have the following alternative: to make it faster, we might take bigger stones and throw them in quickly yielding something like this:

And it may even sort of look similar, but I just can't force myself to see the plan in the implementation. Aesthetically, at least, it's obvious that mosaic laid out in this fashion is no match to the envisioned one. But how did our big stone practice made it break ?

In other words, what can we say about big stone mosaics ?


Bigger stones are by definition fewer, so we have to invest less in learning. The less stones we have to learn how to deal with, the more confidence we have in that we know how to handle them properly.

There is less variance between big stones, consequently there is no choice as to which stone pick for this or that spot - one less burdening decision to make.

Takes fewer pieces to lay the mosaic out. And it saves time, this is precisely the reason we choose this way.

The less pieces, the less interaction between pieces (although I must admit, this is really more a software insight).


UGLY. There are situations in which big stones fit in precisely, but those are exceptions, rather than rules. If we are up to a piece of art, not to the standard kitchen tile. This is the aesthetics-breaking reason. This is why the first one is amazing and the second one is not even satisfactory.

Along the same lines of aesthetics and taste goes this - if someone is always doing big stones, how could she tell what's beautiful and what's not ? It's still a mosaic, isn't it ?

Bigger stones obviously not always fit and we have no control over it. If we want control after all, we need to apply much more work compared to what would have needed if we were combining smaller ones from the start (that's exactly the reason why mosaic exists - we take small patches and combine them in a such a way that the whole thing looks beautiful from some distance, and don't spend forever cutting).

A mosaic built of bigger stones is less reliable (alhtough again, it's more of a software insight). We may be confident, but there is no way we can tell a crack inside a big stone, and if it happens to exist, we have no control.

And so, what's the average stone size modern software mosaics are being built of, what do you think ?

October 03, 2006

Security and usability of a real-life elevator

Security problems often appear from unforeseen interactions between different system parts. Likewise it happens when new features are added or existing ones are modified. Here is another example of that.

The office where I work occupies a five-storey building. All entrances to any level were initially equipped with card locks, and so to enter or to go from one level to another you had to have your card with you. This brought a certain amount of security, in terms of authentication and audit.

The problem was, there was initially no elevator, only stairs to access the levels. As time went on, convenience considerations prevailed and so an elevator has been installed. As such thing hasn't been predicted in the first place, the elevator shaft now couldn't be locked with card locks, simply because there is no doors on the elevator compartments, and such doors can't be installed (easily, or at all).

What happened is that security has been breached - once you have access to one level, you could take an elevator to any other without a card. This effectively defeated the entire system.

Anyhow, this is not the end of the story yet although the rest of it adds nothing from the security perspective. It's just that the elevator seems to attract user interaction problems as well.

And so one of the levels is also undergoing heavy repairs. You can enter it with or without a card now, but it's a mess therefore noone is supposed to. But the feature of entering any level without a card (in a form of elevator) is already there. Now, what's been done to prevent unaware people from using it in the wrong way, i.e. going to the second floor ?

The elevator button panel has been patched (literally, with a piece of paper and tape) so that you couldn't go to level two at all.

Now every day I come to work, this patched button reminds me of how difficult and unpredictable security and usability really is where it meets real life uses.

September 05, 2006

Building bridges

Just heard it in the news - they decided not to explode the remains of the flyover bridge collapsed yesterday, here, in the city where I live. Thank God they wouldn't. Because they haven't manage to build it in the first place. With all their industrial-grade materials and world-class technologies, the thing crashed two months before it was supposed to be opened. Can you imagine what happens if they use world-class explosives ? Stop doing what you know not how to do, will you ?

Same thing happens in software and it makes me sick. Guess what ? Technologies and products are worth nothing unless you actually know how to build stuff. You have to actually get to it and do it over and again before you know how to do it. If you decide to use explosives to bring a building down, you'd better be damn proficient in exactly that sort of things. If all you previously have done was paving the roads, but you occasionally have a pack of industrial-grade high explosives, your attempts to follow world-class procedures in demolition will likely go down in flames.

The worst mistake of all when you want to build quality stuff is that everything is simple. No damn it, it's not. It's damn hard. You have to know ins and outs and you have to learn by your own mistakes, and never assume that bying Oracle will solve all your application performance problems.

September 02, 2006

Experience is the name ...

About this quote by Oscar Wilde:
Experience is the name everyone gives to their mistakes

I thought it had an interesting language twist to it. Although the apparent interpretation suggests that everyone is hiding their mistakes behind the word "experience", there is another way of looking at it:

Experience is the name everyone gives to their mistakes
so that it means that everyone finds a name for their mistakes, and that name becomes the experience. Thus by picking good or bad word for referring to their mistakes, everyone can totally change the way they perceive their experience.

August 25, 2006

All systems are real-time and need extra processing power

Real-time systems have much hype on them and as always need a special form of speech to cover the shortcuts. In case of real-time such speech is about "soft real-time".

The problem is - "hard real-time systems" (the ones that ALWAYS meet execution deadlines) are fiendishly difficult to build, especially on the mainstream hardware, using mainstream practices, for any kind of realistic load.

"Soft real-time" systems are thus not required to always meet execution deadlines, but rather have a probabilistic process imposed on them. Yes, it's bad if you fail to react in time, but it's still better if you react 3 ms late 40% of time than 10 ms late 10% of time (depends on the application of course).

But then, aren't all systems real-time ? After all, noone needs a system which never finishes processing, and it's better doing it sooner than later. Systems may have more or less relaxed requirements, but it's the same probabilistic process, for example 90% of users would wait 5 seconds for a transaction to complete, but only 10% wouldn't be calling tech support upon a 30 seconds delay.

Look closely at the word "real-time" itself. Everything is executed in real time, all processes have references to time flowing by, executions have starts and stops and whether or not they've been a success is determined by an external entity (a user in most cases) and timing is not the last criteria.

Anyhow, I'd like to point out one other thing. There is this theorem which says that for a real-time system to meet its deadlines under certain conditions, it must have a significant fraction of CPU processing power free. If I remember correctly, the exact top load percentage is around 70%, which means that if the CPU is busy more than that, the system starts missing deadlines.

If this result is blindly (or philosophically) applied to all the systems, it leads to an interesting conclusion - no system should be functioning under stress load (perhaps above the mentioned 70% CPU), otherwise it starts slipping behind the schedule and, being real-time, degrades its service significantly.

It would thus be a mistake to believe that an application quality of service would remain steady across all of the CPU load scale. Instead, as CPU usage climbs up to 70%, the application may scale perfectly linearly, but as the load keeps rising, the scaling breaks. It's therefore more safe to consider 70% to be the top acceptable load and start adding more processing power as this line is crossed.

July 30, 2006

Re: The "Snakes on a Plane" Problem

Found an interesting article on Reddit:
The "Snakes on a Plane" Problem


I worked in newspapers for eight years, right when that industry was starting to disintegrate. As such, we spent a lot of time talking with focus groups, forever trying to figure out what readers wanted. And here is what they wanted: everything. [...] When it comes to mass media, it's useless to ask people what they want; nobody knows what they want until they have it.
Why this sounds all so familiar ?

July 27, 2006

One other reason why new software is released

Because the users keep using the old one in the wrong way.

July 24, 2006

Reusable business-specific abstraction is an oxymoron

In this article I'd like to talk about abstractions in software development, focusing specifically on business-specific reuse.

Abstractions come in many flavours. At the most general level, abstraction is the single most valuable method of analysis. Mathematics is all about it. Abstraction is also the result of that analysis. I will be using the term "abstraction" as if it meant "a concept relevant in particular context". In simple terms abstraction is "something that ..."

In programming, abstractions appear in many places. It's impossible to discuss all of them, so here is a few random examples before we proceed to the points of the discussion.

The simplest abstraction in programming is the byte. The concept of byte suggests that (1) all data we manipulate with can be converted to numbers and (2) the numbers can be encoded using fixed width integers. Although bytes with size different from 8 bits have been experimented with, they were not successful (not that I saw CDC Cyber myself, but the books say so). Let's think why all the bytes have converged to 8 bits. My answer is - because there is no difference. They could have been of any size, 8 bits were just a good balance of range/processing power vs. costs to produce at that time. As the costs kept falling, the bytes actually grew (although under a name of machine word now). Anyhow, the byte became a universally accepted abstraction, reused literally everywhere.

Abstractions also appear in OOP where they hide in the shadows of the objects themselves. After all what is an object if not an abstraction of some real-world entity ? OOP is very successful in giving developers abstraction producing machinery which they apply to their problems yielding abstractions directly translatable to programming languages. Not all the created abstractions are reusable, leave alone useful, but OOP itself is.

As we can see, low level abstractions can become quite successful and widely adopted. This is because they are so "abstract" they apply everywhere out of the box, and changing them would make no difference, and so nobody even thinks about why they are there and what's their purpose. They are just there, they've always been (for an eternity of fifty years), they are the truth. Although this approach is far from scientific, it works well in production. After all, why do I need to know the details of the chemical process for producing plastic, as long as all I need is to wrap my groceries in a plastic bag ?

And so, my first point - easily reusable abstractions are invisible. We use them without ever noticing. "Why ? Of course our data is encoded in bytes, what a stupid suggestion !" Low level abstractions are such, but hardly any others, at least right now.

As we move farther towards the "real business" side of programming scale, it's increasingly difficult not to notice the abstractions arising more as obstacles than helpers. One step up, look at what building blocks we have as we build real systems. No matter if it's 1980s or 2000s, the abstractions that your programming environment offers are the same:

  • User. Session. Context. Authentication with plaintext username and password (anything else is still out of reach because of interop problems).
  • Database (whatever make, hard to design). Connection. Pool (not always useful). SQL query (hardly anyone knows how to write one that won't bring the server to its knees). Transactions (requires even more knowledge).
  • Component framework, networking, distributed facilities, RPC, messaging (various compatibility, performance and scalability problems difficult to address).
  • I18N: Encoding (Latin-1 as the mostly useful one, interop hell again).
And on and on it goes. All those "development" abstractions are useful and widely accepted but still are technical, low-level and because of their complexity only reusable as a mindset. From this point of view, they are patterns more than real tools.

Two steps up, when it comes to real business, there are no abstractions. It has to be done in exact particular way. Nobody's going to use an application if it doesn't make precisely what the user wants. That's why all the software is to some degree configurable and adjustable. The rigid abstract part of the software is clearly separated from the flexible configuration part. The abstraction therefore has nothing to do with the business specifics - it's all in the configuration. And if you are lucky, you've got enough flexibility to cover everything.

Now, I'm asking you this - wouldn't then a "reusable business-specific abstraction" be an oxymoron ?

For one, "business-specific abstraction" is already nonsence. Abstract has its details stripped off and replaced with imaginary concepts. But details are the heart of the particular business, no matter if it's called "valuable proprietary know-how" or "ugly poorly understood mess". And it makes difference. And it's highly visible. If you are switching from DCOM to SOAP, nobody cares. But if you mishandle customer's data for the sake of abstraction that you might have - well, they won't approve it. It has to be done their way or no way. If your abstraction is too rigid - too bad for it.

For two, "reusable business-specific" is also nonsence, as soon as you are crossing enterprise boundaries. Unless Big Brother takes over, every single business is going to be to some extent unique and as such will not be covered by the abstraction. Even if you are borrowing a clearly separated network service, not burdened with thousand dependencies, it's likely to be useless without tweaking.

And so my second point is - business specific software can hardly be abstract and hardly reusable. In fact, the most valuable property of such software is flexibility. It has to allow easy changes, no matter what abstraction has been put in it.

That's one of the points behind the Pythomnic project - allowing developers to build network services that can to any extent be changed on the fly, without service restart and as soon as you have read this far I invite you to check it out.

July 22, 2006

Always write what you really think ...

... because if your thinking is right, so will be your writing, but if your thinking is wrong, it doesn't matter what you write anyway.

July 06, 2006

Want to automate your business ? Want a single-click software ? Prepare for changes.

Very few people actually love the work they do. If it's somehow done magically with no concern of theirs, hey, they'd rather be fishing ! People have always tended to avoid work. Having somebody else to do one's job is the best strategy under the sun. Even for those who are not very sick of their job today, and pretend to be doing things, the less actual action - the better. Laziness is a much better driver than fictional pursue of productivity, urge for business success or corporate loyalty.

Now, people don't need software at all. Like I said, they need their job done, preferrably with no load on themselves. But software was giving such promises about how much easier it would be (if you purchase product X) since day one. Not that many still believe that new version of Microsoft Word will boost their productivity, but hey, this is how documents are prepared in 21st century, ain't it ? It's just that you install it, boom ! and your documents "practically manage themselves" (a ridiculous sentence if you ask me).

Therefore users dream of software with one big button reading "Get my job done now" (even then there would be some moaning about how much hassle it is to push it). It is indeed crucial to design software in such a manner that it requires as little user intervention as possible. But doing so is very tricky.

Let's say we are developing an accounting package. The first problem we are going to face is ubiquitous lack of knowledge. It probably wouldn't surprise you the least that someone with degree in CS knows nothing of accounting, leave alone specific customer's business practices. What would surprise you for the first few times is that the users themselves don't know what they are doing, no matter if they've got a degree in finance.

We are likely to see that everyone knows a little bit but nobody sees the big picture. "It's just how it's done here, that guy over there probably knows something" is the sort of speech you may hear. Blind leading the blind, at its worse.

A good business analyst can save the day, no doubt. Since a good business analyst essentially is a person to some extent trained in both areas, experienced in exactly that kind of projects and (which is crucial) clever, fluent and clear in his thoughts and talks, he could definetely talk things out.

But the business analyst will actually be busy recognizing an elephant from descriptions given by three blind wisemen. Or, sorting out all the quirks of the customer's business practices, which is essentially the same thing. Even if (and it's a big if) he succeeds all the team gets is a 10 sq. ft. diagram with lines going in all directions. Better understanding, perhaps, but of what ? And this is where the second problem hits - ugly and tangled business practices.

Well, the customer is always right and all that, but what's the point in one-to-one mapping software functions to existing business practices ? Nobody understands how it works, I see. Now you have software that automates that ! I mean, it's just outrageous. You take a can of worms, sort of peek into it, then, using whatever perfect (huh ?!) software development process, implement the automated can of worms and hand it over to the customer.

"Eeew !" - that's what the user says - "I thought this software thing would be simpler ! Where is the "Get my job done now" button ?!" Oops, there is no such button. What you get is a button for every chaotic activity you have performed before the software extravaganza started.

And so my point is - for a business to benefit from software automation, it must work well enough even before and without any software. Chaos cannot be automated, that's what I say. As there is hardly a perfectly organized business out there, well, it needs to be changed before it can be automated.

The good news is - those two processes can be performed in parallel and greatly support each other - straightening (uh, streamlining is so much a buzzword) the existing business practices, and analysis of the same for the sake of software development. If there is a perfect moment for changes, it's now - there is a lot of efforts put into it from the software developers camp, and people genuinely interested in understanding and making changes will be around, not the stupidest people too. All the customer needs is to allocate decent resources and get ready to change.

The bad news is - a customer is more likely to remain heavy and unresponsive. "There, we've paid you, now where is our button ?" Still, as you know, changes come, now or later, and it's better to change it right now than to postpone.

To conclude, I believe we as software developers should be working not just on software but also on improving our customer's business rules, it's in our own best interest. Moreover, we should be pushing the customers to changes, not just accepting their requirements.

July 04, 2006

Open source, closed source, can anyone tell a good piece of software from a bad one ?

To start the discussion, let me ask you this - shall we release early and often ?

If we release early, we are likely to release crap, but have better exposure. But people are already used to software being crap. Besides, crap or not, there is no easy way for users to tell.

But if we go the long way of making the software better as we see it, and release later, the quality might be better, but it's still expected to be crap (release 1.0, anyone ?) and even though the experience might prove otherwise, the impression is still about the same. Besides, whi will care ?

Paradoxically, we are pushed towards producing crap faster.

Again, the key problem here is that we have no way of evaluating software quality. The closed source used to rely on the quality signals that, once been in favour have long since sold out - brands, advertisement, experts, reviews. The open source may rely on community judgement, popularity and zealotry, but what is it if not bandwagon effect ? Is it any better then ? I don't think so. Both sides rely on experience, but then again, experience is nothing but a way of convincing oneself.

Anyhow, my point is - open source, closed source, both parties have their reasons, politics and propaganda, both are eager to sell, but not only neither knows a way of producing high quality software, nobody can even know for sure if any given piece of software is good or not.

Think about it. We make stuff. We are proud artists (each a Picasso, no less). But when presented with somebody else's work, we cannot tell its quality. Isn't it outright ridiculous ? What kind of an industry is that ?

We thus are tackling the wrong problem. We need to build quality things. Instead we invent different ways of building things of unknown quality.

We desperately need reliable indicators of software quality.

June 24, 2006

Perpetual movement of crap

Software is crap. All of it. Or, may be you can prove otherwise ? But I'm warning you - I'm not buying a name underneath the ®, a web-site full of buzzwords, a stellar magazine review or even a community size of China. Why not ? Because neither is reliable. Now, all the above left out, how can you make me believe it's good ?

We desperately lack means to evaluating software quality. Not just quality, but all the -ilities aka non-functional requirements. It's so bad - it's not even funny. We cannot examine the software as such, we never look at it directly, all we have is different directions from which we observe its reflections. Software is incredibly opaque - it's as if you had to evaluate an actor performance on a stage veiled by not one but many curtains. Difficult casting problem, isn't it ?

What exactly are those curtains the software is hidden behind ? I believe they are exactly the steps we take to make it - to come from an idea or perhaps a set of basic requirements to the working machine code. It looks innocent, even proud - all those philosophical aspects of OOP, applied math with its abstractiveness so profound, the development processes with metrics and iterations and so on.

But each step takes us further into the land of not understanding, and the sad truth is - our methods don't work. Somewhere on this glorious path, something slips and all we get at the end is crap. A wonderfully abstract crap. A beatifully modelled crap. A crap produced with the process tailored by developers at a CMM5 level.

And not just that. Now that we have our methods applied, it's impossible to see the thing we have been building in this pile of crap. It's shredded crap in a box - it's impossible to see through. Isn't it funny how testing speaks about black box vs. white box testing ? The box - note to yourself.

Being compared so often to the construction, software development is far worse in that it's a land of fairies - it's difficult to apply your common sense in a world where buildings can be started from their roofs and left hanging in the air.

Imagine that you are buying a house but you can't see it - it's in a box - granted a big one. All you can do is to give it tests - does the water start when you open the tap ? Yes, says the developer, it does - trust me, we have installed the tap complying with the latest industrial standards. Can I see it, you ask ? No, it's in the box, but it works ! You tap something which sort of looks like one and hear the sounds of water. Uhm, ok. Does it have windows ? Yes, sure it does. Can I see it ? No, it's in the frikin box ! How can I be sure the windows are installed in walls, not in floors or are not even left leaning to a basement wall ? Trust me on this, we are professionals here. So much for a black box house testing. Now, white box testing is not much better. When applied to houses, white box testing wouldn't mean you are allowed in to look around - instead you are allowed to look at the blueprints. Oh, now it's so much better. As a home owner I couldn't tell the mentioned tap on the blueprint. Besides, having a finest project of all still doesn't mean the windows are where they are supposed to be.

Unable to see for yourself, how do you estimate the quality of software ? You go for much less reliable sources - the talk of the people who want to sell it to you. Now, that's who you can trust - the salesmen ! Their cheap talk..., but I don't have to tell you that. The words they are using, please, don't let me start - "industrial standard" - that's my favourite... Trade shows, web site reviews, free T-shirts, it's all part of the selling game.

But think - what speaks louder than all the advertisements, and is closer to the real situation ? Hint - it literally yells in your face, sometimes several times a day, you are so used to it, you are skipping it right on the spot by clicking "Next". What is it ? It's the software license. No matter if it's a proprietary license or whatever orthodox OSI certified open source one - the most important part of it says that the people who did it have no confidence in the thing they are selling you. It comes with no guarantee for god's sake. Where else you can buy a thing and rightfully expect it not to work right out of the box ? And you can't return it for a working one, he he he...

Don't ever expect the software not to be crap. You can write it yourself (that's what I personally always prefer), but it'd still be crap, although you'd have a much better hold of it and earn experience. Besides, you can't write everything from the ground up, it's insane. You may just as well pick your software at random and then clench your teeth and stick with it through years until the fact that it's crap is so obvious that you cannot bear it any more. I've seen it a few times when a 3rd party software (complying with industrial standards, no doubt there) bought for ridiculous amounts of money left everybody engaged for a while, harder and harder, until eyes pop out and somebody decides that it needs to be replaced with another one (complying with newer industrial standards).

There is one other aspect I'd like to bring out.

It's ever so cool to proclaim that "lives depends on the software" - makes you so proud and self-assured, like hey ! the software that we write here - everybody depends on it ! We are cool ! It's an exaggeration at the very least. Lives depend on software no more than they do on Hollywood movies - yes, they want it, and there is an outrage if theaters are shut down, somebody may even get killed in a riot, but it only concerns those who actually allowed the movies to take so important place in their lifes. Likewise with the software - if your word processor crashes, there is an Internet outage, your bank's ATM network is out of service, your flight is cancelled while you are in the middle of nowhere, or even if your plane goes down due to a software malfunction - does that mean your life depends on it ? No, it's your habits, your ways, your convenience that do. What's true to the point is that despite having no real importance to anyone's life, once it's been given the right to shape the lives - the software may not stop. We are used to it - it's all around, it's uncomfortable to live without it.

And so, where does this discussion leave me as a software developer ? I build crap, nobody really needs it, but everybody sort of expects that the crap is there every time they look. And so I have to keep the crap running. I come to work every day and keep pushing the wagon full of crap so that it doesn't stop. If I'm lucky, I can take a few hours free of pushing and try to build something different, something which is either less crap, or at least a better wagon someone will have to push. Still.

Perpetual movement of crap, that's what our discipline is at its current stage of development.

June 19, 2006

The difference between a programmer and an administrator

A good programmer knows how to do something. A good administrator knows how somebody else has already done something.

May 28, 2006

Acme Corp. Do something.

Take any big company ad slogan:
  • HP. Invent.
  • IBM. Think.
  • Apple. Think different.
  • Samsung. Imagine.
whenever it has instructions in it - to think, invent or do anything in particular. What a heck does thinking (even thinking differently) or being able to invent has to do with the commodities those companies produce ? What does it mean at all ?

Do they really want to say - "Buy our stuff and then do whatever nice things we are telling you to do" ? That would be ridiculous. Do I need permission to think ? Do I have to buy something IBM to apply ?

"Buy our stuff and then you will start doing it no matter if you want it or not" ? Buying a Mac unlikely will make you think differently. Acquiring different GUI habits may be, but not different way of thinking, otherwise nobody would ever be able to interact with Apple owner from the moment of the purchase on.

"Buy our stuff and then doing those things would be easier" ? Nah, I doubt if inventing with HP is easier than with anything else.

Oh, I get it, I'm probably supposed to associate HP with inventions as such ? IBM with thinking ? Whatever company with whatever pleasant activity ? Now, THAT would be really stupid, just to think that there is an association between buying some piece of equipment and engaging in any pleasant activity. Otherwise, where are "IBM. Think, do you really need it ?", "Samsung. Imagine something not made of semiconductors" or even "Don't believe everything you hear in commercials" or "Trash your TV and go see the sunset" ? That last one, wouldn't it be a pleasure to do ?

Anyhow, TV commercials, magazines ads and equipment cartons would be the last things I would use for motivation, thank you very much. Except for blogging.

May 24, 2006

So, how does it feel writing infallible code ?

I think many people feel uncomfortable about the code they write or maintain, systems they support or administer. The stuff is just bad, let's face it. But people (including myself) tend to find a comforting spot in which they are most emotionally stable and least disturbed about the reality they have to live in. But this comfort is often based on ignoring the truth. Let's see how it may work in one particular case.

I had this idea when thinking about moving one very small but very common piece of functionality into a network service. We have each system to determine whether a given day is a workday or a holiday. As there is no such stable rule, each system has its own volatile piece of data in whatever format which it consults, and every administrator of every system has not to forget to keep it in good shape. It repeats in tens of variations, burdens the developers and is a constant administrative pain. Sounds like a good idea to factor it out into a network service ?

Paradoxically, it does not, precisely from the point of view of those troubled systems' developers and administrators. But why not ? It seems odd. Before the refactoring they have

function workday(date):
return consult_own_database(date) == "W"
if workday(today) then do_stuff()

whereas after refactoring they have

function workday(date):
return consult_magic_service(date) == "W"
if workday(today) then do_stuff()

(mind you that this a pretty real-life looking code). It looks like nothing changes at all, except for now they have one less problem. To simplify, let's assume such change wouldn't require any work from them, will be done in a snap and cause no troubles in itself. Still they refuse. What is the likely reason they give to cancel the change ?

- It's less reliable. A call to a local database (or filesystem) is less likely to fail than some network service.
Reliability, that I understand. And it's also reasonable to assume that the probability of local database failure is less than that of a remote network service. But I argue that it's not the different degree of reliability that matters here. After all the above chunks of code are identical, why would one be less reliable than the other ?

The first (existing) piece of code does not indicate a less probable failure. In fact it indicates a zero probability of failure. The call may fail, but the code does nothing about it. The developer consciously or unconsciously knows that there is a possibility of failure, but as soon as it's perceived below certain threshold, she's still comfortable ignoring it. Now with the subtlest change comes an increased failure probability which is too uncomfortable to ignore.

And what are the options ? Rewrite the code so that it accepts failures ? Impossible, too difficult, can't be done, no way, period. Thus we don't need the change, it will break the system which works fine. Ironical, isn't it ?

And so the thing remains unchanged, with regards to this switch to a network service or not, it becomes too fragile - touching it is dangerously uncomfortable. Therefore the point of this argument is that developer's comfort based on false assumptions, such as improbability of failures, forbids any major changes in the software.

May 06, 2006

The pattern to rescue other patterns: The Golden Hammer

Patterns look like they can solve all of your real-life problem. To me, this is awfully wrong and can do more harm than good. Yes, patterns allow for organizing your knowledge and also for developers to find commond ground easier and they do awesome job there. But they are nothing more than a mnemonic constructs.

I sometimes hear sentences like

- We use pattern X here, because it's the right thing to do in this kind of project.

Not because the solution has the appropriate structure, and pattern X somehow emerges from it, but because such and such problems require such and such patterns. What a heck ? I thought problems require solutions, not patterns. To me, such talk is a bad sympthom - it means that people can't see forest for the trees. Or, which is also possible, that I can't understand what they are saying, but let's forget about this insane assumption at once :)

Now, for pattern lovers - the pattern to the rescue - The Golden Hammer. It says that if you are good at hammer work, all your problems appear to you as nails. Likewise, if you are good at patterns, you find the patterns you already know everywhere. The key point here is that you do not attempt to apply thought and find other kind of patterns, but blindly say - hey, this obviously asks for a bridge, and this is undoubtedly a mediator here.

It appears that most beautiful and influential things appear with reflection - a tendency to analyze oneself. The Golden Hammer is most reflectional as it implies you actually think about the process of applying patterns (including itself). And if you are more conscious in applying patterns - you benefit more, don't you think ?

May 03, 2006

Expecting software errors, not fixing one and pretending none left

There is a problem with your software, something strange is going on, some error appeared that has never had appeared before, or should have never appeared in the first place, the system produced insane output or just died ?

It's unwise to think about anything you are experiencing as of "unbelievable". You see it, you have hard evidence - it exists. Why is that - you have no idea, but it exists. Finding out the origin of such a problem requires a totally different approach to debugging. As now you have no idea why it happened, you likely could have never predicted it, tested for it or even thought about it.

Stop and think about it. You could have never thought your software would fail in that particular way. Taken to the extreme this would mean that you never thought your software could fail at all.

But this is exactly how novice programmers think. As they become more experienced, the amazing world of software failures appears before them, and they start applying the typical code/debug cycle, error handling and/or try/catch patchwork, which still limits them to the errors they are experiencing at that single moment.

It seems logical that if the programmer's experience is taken to another extreme, she would expect failures everywhere, just for the sake of vigilance. No matter if a failure of some kind has never happened before - it's there, waiting to happen.

The point of this argument is that the key thing to writing stable software is being failure-proactive and expecting everything to fail. Just ask yourself pointing finger to any piece of your system: "what happens if this fails (for whatever reason) ?" The results can be far reaching.

I therefore argue that the common debugging is far less important a process than it's usually thought. Of course, finding an error and fixing it is important and must be done, but if you had expected it, your whole system is by now prepared to any error anywhere, and this one error only challenges this fact.

Moreover, I also argue that trying to break your own software is at least as valuable a practice as the forementioned debugging.

Yes, it's a well known maxim that all testing is about breaking things, not about seeing it working. But there is a catch. Tester can only test against problems that he can think of, so much like the developer. Moreover, as soon as developers and testers work side by side (unless it's the same person), they tend to share the same narrow field of view, which greatly reduces testing efficiency.

That's why I like stress testing. Not because it shows me impressive transactions per second, but because it tends to reveal another layer of problems usually never seen. Stress testing is very useful, but it shines whenever a failure leads to a cascade of other failures. Whoa ! I couldn't have thought of THAT !

Now, how else you can intentionally break your system except by giving it a stress load ?

Failure injection comes to mind, but it contradicts to the normal development cycle (as there is simply no place for it), and requires the mentioned failure-ready system structure from day one. I also tend to remove the injected failures in production system, after all being a failure-vigilant software developer won't save you from black cats and broken mirrors.

Unplugging cables from the working system is also fun and can reveal yet another unexpected problems, although again, it's probably not applicable to a production system. For another joke of this kind see "The China Syndrome Test" in "Blueprints for High Availability" by Evan Marcus & Hal Stern.

In conclusion, this discussion again turns me to the PITOMNIK principle: as the system grows, its small errors grow with it and therefore lead to inevitable failure. You cannot ignore the presence of the errors, nor you can fix them any significant fraction of them, so you'd better be ready.

April 29, 2006

Software: too many layers, too few tiers

Layers and tiers are both ways of enforcing structure onto a software system. I will be using term layer = "logically isolated group of modules within a process" and tier = "physically isolated group of modules within a system". The difference is basically logical vs. physical.

Now, layers are way more popular and friendly to any development environment, because they
  • are directly supported with the language and/or runtime
  • allow greater flexibility and freedom of interaction between different layers
  • are accustomed way of thinking about software
OOP with classes and interfaces are all about layers.

Tiers, in turn, are more difficult to deal with, because they
  • require more upfront design
  • require complex and/or restrictive and/or expensive (un)marshalling
  • require different way of thinking
There also are obvious upsides to tiers, mostly about independence, ex.
  • deployment and execution independence
  • development, language independence
  • reuse independence
CBD with components and interfaces are all about tiers. Right now the choice is basically limited to a few popular component-based environments, such as DCOM, CORBA or (somewhat differently) web services and SOAP.

I sincerely believe that having more tiers is beneficial and it would be great if there was a way of making tiers easier to use in a way similar to layers. And so, guess what, this is one of the ideas behind the Pythomnic (Python framework for building reliable services) - to allow fast and easy way of converting layers into tiers - to mix and match modules in any way.

For example, if there is a cleanly separable function (ex. CPU intensive XSLT transformation) currently allocated to a module or a set of modules in a layer, you may also take one step ahead and declare that this function can possibly be separated to a different tier. For example, instead of calling
you do
the key thing to note is that the way execute.on works, if "xslt_transformer" RPC channel is not configured in an appropriate configuration file, the pmnc call will still be local, but as soon as you modify the configuration file and save it, the very next call will go to the specified server. There is no need to restart Pythomnic on this machine, all you need to do is to copy the xslt module to a separate server and start it there in its own Pythomnic thus turning a layer into a tier.

I do believe that such a feature is beneficial to a middleware development framework.

April 23, 2006

One note on Python simplicity in handling phoenix problems

One of the major problems in developing long running applications (and applications in general really) is cleanly handling shutdown. Among them a particular subproblem of phoenix singletons - what happens sometimes if one entity references the other (typically a global singleton), which has already been unloaded.

For instance, consider the following shutdown code in moduleA (using simplified Python syntax):

def shutdown():
there is no guarantee that moduleB has not been shut down yet. Now, if moduleB is also written in a delayed initialization fashion, ex:

impl = Impl()
def callme():
if not impl.initialized():
return impl.callme()
def shutdown():
then what happens upon moduleA's shutdown is a reinitialization of the impl - one sort of a phoenix. It just went on me that instead of building a complex synchronization schemes to handle this cleanly, all I need to do to prevent this is just

def shutdown():
del impl
and now as impl is just not there, the moduleA's attempt to reference it at shutdown will fail and throw - a much more appropriate behaviour in a given situation. This is less clean a solution, but how simple it is !

April 19, 2006

Shortcuts to icons to shortcuts: what happens if you click that ?

It seems that desktop icons have rather confusing nature. Let's say there is an object, e.g. executable file. Whenever it physically presents on the desktop, it appears as an icon. Whenever it's not on the desktop itself, which is a more frequent situation, but a shortcut, an object of different kin is, the shortcut appears as the same icon albeit with a little arrow in the lower left corner.

Hence problem #1. There are different icons on the desktop with and without arrows, but single-clicking on them reveals identical behaviour - the target object is invoked. What arrow is for then ? For the user to see which is "real thing" and which is a representation of a concept she can't quite grasp anyway ? Wouldn't it be more logical if non-shortcuts couldn't be placed on desktop at all ?

Next, surprisingly, there are other places on the desktop where icons appear - task bar, quick launch bar and system tray. Those add to the confusion, because although their icons look identical, their behaviour is even more different:

Icon on the quick launch bar represents a possibility to start an application, clicking it starts another instance. Icon on the task bar represents an already running application, clicking it brings it to front. Icon in the tray represents an application willing to talk, but what happens if you click it is not known beforehand.

Hence problem #2: wouldn't it be appropriate if all these icons were different to some degree, or behaved in more consistent fashion ?

April 18, 2006

On (information security) audit: giving money to the developers

Auditing information systems is fiendishly difficult. Think about it - a typical situation for a developer is discovering problems in the _small_ pieces of code that she's working on _right_now_. Few days later other problems may be discovered. Half a year from that - yet other.

Then, as the system is assembled, parts developed by different people come together, a whole new world of problems emerge. The people who built it have scattered knowledge of the system themselves.

Now, to audit. Suits come in, unpack their laptops, run standard tests, look (!) at everything and ask tough questions. A week after they conclude whether the system the very authors have no complete knowledge of is good or not. And then they leave.

Hence my point - a good team should be doing internal audits as it goes. A good developer should be running custom-tailored tests, looking at the thing, asking tough questions no worse than the auditors. And the knowledge remains with the company.

Therefore, why not investing the same money into team education, so that they become their own auditors ? It's the old "give fish" vs. "teach to fish" thing.

I realize there are PR and sometimes legal aspects to audit, but to a developer PR along with legalities don't make much sense.

March 26, 2006

Note to self - Python properties are non-polymorphic.

Properties are just one application of Python ubiquitous proxying. The properties are created as wrappers over get/set methods:
class C(object):

def __init__(self, value):
self._value = value

def _get_value(self):
return self._value

value = property(_get_value)

# all is fine, value calls C._get_value:

assert C(10).value == 10

Here is what's happening: an instance of a property class is created and a reference to C._get_value is stored in it. Next time c.value is referenced, the property calls the original method.

The problem is, the reference is bound to the particular class C, and so if you inherit from it, you should not expect the property to be rebound to the ancestor class'es method:
class D(C):

def _get_value(self):
return self._value + 1

# now, this still calls C._get_value:

assert D(10).value != 11

The properties are thus non-polymorphic. There is one way to make them such by introducing late binding,
class C(object):


value = property(lambda self: self._get_value())

but I personally find it cumbersome.

March 24, 2006

Moore's law as a social amplifier

The hardware world lives by the Moore's law of constant speed up. The software world is following the same trail, but there is significant difference - hardware measurably increases performance while staying on the same line. Dragster style if you like. Compared to this software is a bunch of beer cans hanging off the racer's bumper:

It makes a lot of noise. So much noise, 90% of the humanity can hear it, for God's sake.

More stuff can be added as the car's engine gets more powerful. What for ? Heck, why not ? After all, people prefer dancing pigs.

It can hardly choose the way. As the principal architecture remains the same, the algorithms remain the same, people have no need in extending body of knowledge and software thus becomes a self-fulfilling prophecy.

It passes everything at such a great speed, there's no way to ever see what's in it. Who cares, next year pack will be better. Likewise, when you are inside the can, all you can see is wind. Speed, more speed !

Anyhow, the question here is - are people strong enough, knowledgeable enough and not the least - moral enough to deal with Moore's law, to use such a powerful engine ? Are they using it and for what ?

My answer is - Moore's law is a great social amplifier.

It gives you incomprehensible powers. And then it's up to you. Dancing pigs, be it. Just riding the can, good. Sending a billion worth of ads with a single click ? Yes, sadly you can do that. Moreover, as computers control everything, everything becomes a beer can. Attitudes change in all industries, and this is sad, as there are areas which cannot be sped up at that rate (if at all).

But !

You can always take off and observe the way where it's all going. Or may be even forget about the race and do some beer can art (that's what I personally prefer).

On average though, having more power means less thinking and more chaos. Secure about the future ?

March 21, 2006

Do you really need Python ?

I love Python, it's wonderful. But what does in do in fact ? It limits the things you can do in different, previously unseen ways. It shifts your development paradigm big time. It gives you different ways to express your ideas.

But what happens if you stick to Python (or any other language or technology in fact) ? You become dependent on it to get something done. Yes, in this developer-tool symbiosis you do gain a lot, in a short run. But in the long run it ends with useless comparisons like "Ruby is better than Python", or "Java is better than C++", where each party protects its own investments.

I argue that languages and technologies do not matter, it's your development virtue which makes things, not the syntactic constructs or deployment schemes. It's like with math - you have a handful of simple facts and more or less simple rules, and how far you can go only depends on you. There is no reason why you can't build good stuff in Python, C++, Forth or Visual Basic, although the particular schematics will be different for sure.

Therefore, instead of praising Python, why not simply enjoying it while it lasts ?

March 19, 2006

What makes the code unmanageable ?

And so you keep writing good code and everybody else do, and then bah ! you look at (somebody else's) code and do feel that you would never want to touch that. The question is - what exactly makes the code such - horrible, scary, ugly and in general unmaintainable ?

To simplify, let's say we have a team that has set up the formatting, naming and other style rules beforehand. Will it save from ugly code ? No (but it undoubtedly helps). And so we have a group of professionals who have agreed on what a good code is, looking at the bad code they wrote.

If there is something wrong with the code itself (which basically means that all the members agree on its low quality, including the author), then the rules must have not covered everything and legal code is bad.

Now, given the code is not idiotic, it has indentation, theOnlyTrueNames, interfaces and whatnot, what about it turns others away ?

March 18, 2006

Out of all project artifacts, what is the last thing you want to lose ?

Consider this - there are a lot of things subject to configuration management. If there is a disaster and all the artifacts start disappearing one by one - what's the last thing you want to lose ?

Well, source code is one of the players. As this is the ultimate goal of the software project, it's clearly a winner, so let's pick the next runner up.

Source code history, versions repository ? Not unless you do some serious branching.

Backups ? Who needs them if there are originals ?

Architecture ? Documentation ? Can be recovered more or less easily as soon as the team is intact.

Test results, planning, meeting records ? Unlikely...

And so I argue that the next worst thing to lose would be the list of bugs and here is why:

  1. There are hundreds of unrelated items. There is no way anyone could keep any significant part of the list in one's head.
  2. There are no rules by which they can possibly be recovered.
  3. Each item was produced by more or less hard knock, sometimes even from an angry customer. Going through this again would be painful.
  4. Each defect is a ...uhm... defect, something that's not right with the project, something that should bug you until you fix it. And now you know that the thing just became a can of worms and there is no way to fix it.