What eternal have we produced ?

For any system, living or artificial to span a long time it ought to change. Evolution, change over time are a sure signs of life. Rigidness, immutability and stasis are a no less sure signs of death.

Now, there is an interesting perspective to this. The strive to perfection leads us to creating something absolute, things that will exist forever. Any artist or craftsman would certainly like his creation to last for thousand years. But is eternal unchanged presence good ? By the way, have you ever seen anything eternal recently ?

Look around. Look for perfect. Indestructible. Eternal. See anything like this ?

Plastic, glass, and radiation.

Pretty much anything man-made will eventually disappear. Any piece of art will die out. Any building or construction will collapse. But not those three, not in any foreseeable future.

Every single grocery bag will stay forever. Empires rise and fall but the empty milk bottle will swamp around. Radiation will remain invisible, but the half-decay principle will ensure it will only vanish in asymptotically distant future.

Isn't it ironic ? Do we need eternal plastic bags ? No. Perfect milk bottles ? No. But this is what we've got. To make it worse, nothing really complex and useful can be done out of plastic or glass alone. Anything constructed with it will decay and leave around useless pieces of eternal stuff. Somehow we have finally managed to produce something that will outlast us million times. Something perfect. Something perfectly useless and dead.

Why having processing power wasted ?

I don't know about you, but seeing unused hardware troubles me. Doesn't matter if it's an old 386 desktop or a 8 years old Compaq Proliant 7000 which I've recently acquired.

Hardware is always fun to set up, see it running and play around with. Folklore about 100s refurbished Pentium's making up a cluster in a garage fascinates me. I do realize that a single modern server will eat that hundred for breakfast, but still, wouldn't that be fun, running that garage ?

And so, my working PC and the servers that are in testing or spare, are normally loaded up high most of the time, doing what - running workload tests of course ! Right now I'm leaving it running a stress test for Pythomnic, specifically its new capabilities of distributed transactions with recovery. The test will be running for a few days now, and I hope when I return from holidays it's still up and running.

JMS: lock on Java

I have to admit, I have only a basic knowledge of Java. And I wouldn't ever touch it. For some reason, when writing in Java, I feel like swimming in tar - possible, but hardly enjoyable. There may be some hidden beauty beyond my knowledge, but I simply don't want to spend time on getting deep into it, so deep that that beauty-of-possibility is revealed. And frankly, I don't believe there is one.

But then "industrial standard" kind of arguments bite, and all of a sudden there is a significant development around in Java. It's totally beyond me why solutions have to be adapted to technologies, not the other way around, but hey, it's real world.

And so, here comes another industrial standard - JMS. But, if you ask me, JMS is no industrial standard. It's a standard way of implementing messaging in Java world. In other words, JMS is a perfect example of a platform-specific technology, created with one thing in mind - to lock the developers to that platform. Many of the J-things are such, don't you think ?

See, JMS is Java Messaging Service, which asserts right from the start that anything non-Java is out. Should its designers have cared about interoperability and ease of integration, they would have standardized an open network protocol. How much easier life would have been if there was simply generic Message Queueing Protocol, say MQP based on TCP, optionally UDP, with very simple and clear semantics for swapping plaintext (i.e. XML) messages. By now we would have had tons of implementations in different languages, of better quality too. Not the case. Not only JMS transport may be based on RMI, Java-specific thing by definition, it's also focused on transferring instances of Java classes for messages and/or using messages for Java object to Java object communications. A perfect lock on Java. Pity.

Here is one more weirdness - in order to connect to a JMS server you have to know precisely what kind of JMS server software is run on it. Imagine that you'd have to install different software for sending your e-mail via SMTP server running Sendmail vs. Exchange. Sounds awful, but not for industrial standard it's not. What it means is two things - first, the JMS specification failed to provide implementation independence (I'm not sure that it was even supposed to do so), and two, with JMS you have to lock yourself not only to Java, but to particular server implementation, which is even worse. And I mean - theoretically, you should be able to modify your classpath, client configuration file - pf-f-f-t - and switch to a different implementation transparently, but then again - you have to explicitly switch to a particular server implementation, and different servers may have slight differences in behaviour (just love it), hence your clients may break anyhow.

What kind of industry requires such standards I wonder ?

Moscow: exploding buildings

They indeed failed to explode demolish a condemned building in Moscow. I told you.

Experience matters

Everything in software can be done in so many different ways. As my favourite saying goes - there are many ways to skin a cat. And just like with the cat, any solution suffices as soon as the problem is solved. The "any solution goes" maxim is valid and very powerful tool for a pragmatic software developer. But there are points to consider.

For one, it's nearly always that we see the problem wrong. It's either lack of knowledge, perception bias or just plain misunderstanding. In this case we are simply solving the wrong problem.

For two, the problem may be a part of something bigger and the way we build the part may render impossible building the whole.

For three, and this is almost universally true, the problem at hand may be a special case of some other, more abstract problem. Now the solution that we use should be evaluated against this concrete/abstract scale.

Besides, solutions are very fragile. You make one wrong decision and it falls apart.

Given all the above, how one manages to ever build anything ?!

My answer to this question and the subject of this post is that experience matters.

To me, a "I tried it and failed and this is why" is better than "I did it by the book and it worked". The former is a sign of a creative mind that just took a wrong turn, but (and it is very important) was able to accept the failure and find out the cause. The latter is someone who believes in universal solutions and silver bullets and never asks questions. The paradox of the situation is that the he still has a working solution this time around.

It's not like failures are valuable all by themselves. It's the experience you bring out of them. Then, like I said before, you have to actually work on your experience. Experience is not the same as the sum of all your failures (and successes for that sake). Deep analysis is required to turn individual failures and successes into valuable experience.

One other problem is that experience is very personal, even intimate thing. The mentioned analysis is often subconscious and results in something that one may call intuition or "gut feeling".

But you rarely work alone and for the team, what's experience of its individual members ? It cannot be transferred from one to another and it cannot be enforced. Try convincing somebody else in your solution using "my experience is that" or "I feel this is right" kind of arguments. "But I feel different" - this is what you are going to hear.

Individual experience then can only be effectively used within a team if you have worked an essence out of each of your successes or failures and are capable of reasoning and giving arguments that never mention experience at all.

The other use for experience of individuals is have the team highly specialized, or even have single leader and support team (a-la Frederick Brooks' surgical team). This may lead to better results faster but yields less for the group development.

Besides, the team can and should have experience of its own as a whole, but it's another story...

"Software as a floor mosaic" metaphor

Here is a "software as a floor mosaic" metaphor: let's say we need to inlay a floor mosaic. We have a sketch or may be even a highly detailed plan of how the thing has to look like, and even the vision of it and clear and perfect, something like this:


We also have at our disposal assorted pieces of stone, ceramic patches, coloured sand and all the tools we might need. But no matter what tools do we have, it takes experience and above that a lot of time to make such a mosaic. Yes, experience helps but we still need to take each piece, adjust it, finish it and put it in place. Sometimes the piece will crack in our hands and we have to start over. Sometimes, a piece will be very different from what's needed in that particular spot and require much more work than another.

But then there is time pressure - the worst of all curses. Pressed with time we have the following alternative: to make it faster, we might take bigger stones and throw them in quickly yielding something like this:


And it may even sort of look similar, but I just can't force myself to see the plan in the implementation. Aesthetically, at least, it's obvious that mosaic laid out in this fashion is no match to the envisioned one. But how did our big stone practice made it break ?

In other words, what can we say about big stone mosaics ?

Good:

Bigger stones are by definition fewer, so we have to invest less in learning. The less stones we have to learn how to deal with, the more confidence we have in that we know how to handle them properly.

There is less variance between big stones, consequently there is no choice as to which stone pick for this or that spot - one less burdening decision to make.

Takes fewer pieces to lay the mosaic out. And it saves time, this is precisely the reason we choose this way.

The less pieces, the less interaction between pieces (although I must admit, this is really more a software insight).

Bad:

UGLY. There are situations in which big stones fit in precisely, but those are exceptions, rather than rules. If we are up to a piece of art, not to the standard kitchen tile. This is the aesthetics-breaking reason. This is why the first one is amazing and the second one is not even satisfactory.

Along the same lines of aesthetics and taste goes this - if someone is always doing big stones, how could she tell what's beautiful and what's not ? It's still a mosaic, isn't it ?

Bigger stones obviously not always fit and we have no control over it. If we want control after all, we need to apply much more work compared to what would have needed if we were combining smaller ones from the start (that's exactly the reason why mosaic exists - we take small patches and combine them in a such a way that the whole thing looks beautiful from some distance, and don't spend forever cutting).

A mosaic built of bigger stones is less reliable (alhtough again, it's more of a software insight). We may be confident, but there is no way we can tell a crack inside a big stone, and if it happens to exist, we have no control.

And so, what's the average stone size modern software mosaics are being built of, what do you think ?

Security and usability of a real-life elevator

Security problems often appear from unforeseen interactions between different system parts. Likewise it happens when new features are added or existing ones are modified. Here is another example of that.

The office where I work occupies a five-storey building. All entrances to any level were initially equipped with card locks, and so to enter or to go from one level to another you had to have your card with you. This brought a certain amount of security, in terms of authentication and audit.

The problem was, there was initially no elevator, only stairs to access the levels. As time went on, convenience considerations prevailed and so an elevator has been installed. As such thing hasn't been predicted in the first place, the elevator shaft now couldn't be locked with card locks, simply because there is no doors on the elevator compartments, and such doors can't be installed (easily, or at all).

What happened is that security has been breached - once you have access to one level, you could take an elevator to any other without a card. This effectively defeated the entire system.

Anyhow, this is not the end of the story yet although the rest of it adds nothing from the security perspective. It's just that the elevator seems to attract user interaction problems as well.

And so one of the levels is also undergoing heavy repairs. You can enter it with or without a card now, but it's a mess therefore noone is supposed to. But the feature of entering any level without a card (in a form of elevator) is already there. Now, what's been done to prevent unaware people from using it in the wrong way, i.e. going to the second floor ?



The elevator button panel has been patched (literally, with a piece of paper and tape) so that you couldn't go to level two at all.

Now every day I come to work, this patched button reminds me of how difficult and unpredictable security and usability really is where it meets real life uses.

Building bridges

Just heard it in the news - they decided not to explode the remains of the flyover bridge collapsed yesterday, here, in the city where I live. Thank God they wouldn't. Because they haven't manage to build it in the first place. With all their industrial-grade materials and world-class technologies, the thing crashed two months before it was supposed to be opened. Can you imagine what happens if they use world-class explosives ? Stop doing what you know not how to do, will you ?

Same thing happens in software and it makes me sick. Guess what ? Technologies and products are worth nothing unless you actually know how to build stuff. You have to actually get to it and do it over and again before you know how to do it. If you decide to use explosives to bring a building down, you'd better be damn proficient in exactly that sort of things. If all you previously have done was paving the roads, but you occasionally have a pack of industrial-grade high explosives, your attempts to follow world-class procedures in demolition will likely go down in flames.

The worst mistake of all when you want to build quality stuff is that everything is simple. No damn it, it's not. It's damn hard. You have to know ins and outs and you have to learn by your own mistakes, and never assume that bying Oracle will solve all your application performance problems.

Experience is the name ...

About this quote by Oscar Wilde:

Experience is the name everyone gives to their mistakes

I thought it had an interesting language twist to it. Although the apparent interpretation suggests that everyone is hiding their mistakes behind the word "experience", there is another way of looking at it:

Experience is the name everyone gives to their mistakes

so that it means that everyone finds a name for their mistakes, and that name becomes the experience. Thus by picking good or bad word for referring to their mistakes, everyone can totally change the way they perceive their experience.

All systems are real-time and need extra processing power

Real-time systems have much hype on them and as always need a special form of speech to cover the shortcuts. In case of real-time such speech is about "soft real-time".

The problem is - "hard real-time systems" (the ones that ALWAYS meet execution deadlines) are fiendishly difficult to build, especially on the mainstream hardware, using mainstream practices, for any kind of realistic load.

"Soft real-time" systems are thus not required to always meet execution deadlines, but rather have a probabilistic process imposed on them. Yes, it's bad if you fail to react in time, but it's still better if you react 3 ms late 40% of time than 10 ms late 10% of time (depends on the application of course).

But then, aren't all systems real-time ? After all, noone needs a system which never finishes processing, and it's better doing it sooner than later. Systems may have more or less relaxed requirements, but it's the same probabilistic process, for example 90% of users would wait 5 seconds for a transaction to complete, but only 10% wouldn't be calling tech support upon a 30 seconds delay.

Look closely at the word "real-time" itself. Everything is executed in real time, all processes have references to time flowing by, executions have starts and stops and whether or not they've been a success is determined by an external entity (a user in most cases) and timing is not the last criteria.

Anyhow, I'd like to point out one other thing. There is this theorem which says that for a real-time system to meet its deadlines under certain conditions, it must have a significant fraction of CPU processing power free. If I remember correctly, the exact top load percentage is around 70%, which means that if the CPU is busy more than that, the system starts missing deadlines.

If this result is blindly (or philosophically) applied to all the systems, it leads to an interesting conclusion - no system should be functioning under stress load (perhaps above the mentioned 70% CPU), otherwise it starts slipping behind the schedule and, being real-time, degrades its service significantly.

It would thus be a mistake to believe that an application quality of service would remain steady across all of the CPU load scale. Instead, as CPU usage climbs up to 70%, the application may scale perfectly linearly, but as the load keeps rising, the scaling breaks. It's therefore more safe to consider 70% to be the top acceptable load and start adding more processing power as this line is crossed.