Perpetual movement of crap

Software is crap. All of it. Or, may be you can prove otherwise ? But I'm warning you - I'm not buying a name underneath the ®, a web-site full of buzzwords, a stellar magazine review or even a community size of China. Why not ? Because neither is reliable. Now, all the above left out, how can you make me believe it's good ?

We desperately lack means to evaluating software quality. Not just quality, but all the -ilities aka non-functional requirements. It's so bad - it's not even funny. We cannot examine the software as such, we never look at it directly, all we have is different directions from which we observe its reflections. Software is incredibly opaque - it's as if you had to evaluate an actor performance on a stage veiled by not one but many curtains. Difficult casting problem, isn't it ?

What exactly are those curtains the software is hidden behind ? I believe they are exactly the steps we take to make it - to come from an idea or perhaps a set of basic requirements to the working machine code. It looks innocent, even proud - all those philosophical aspects of OOP, applied math with its abstractiveness so profound, the development processes with metrics and iterations and so on.

But each step takes us further into the land of not understanding, and the sad truth is - our methods don't work. Somewhere on this glorious path, something slips and all we get at the end is crap. A wonderfully abstract crap. A beatifully modelled crap. A crap produced with the process tailored by developers at a CMM5 level.

And not just that. Now that we have our methods applied, it's impossible to see the thing we have been building in this pile of crap. It's shredded crap in a box - it's impossible to see through. Isn't it funny how testing speaks about black box vs. white box testing ? The box - note to yourself.

Being compared so often to the construction, software development is far worse in that it's a land of fairies - it's difficult to apply your common sense in a world where buildings can be started from their roofs and left hanging in the air.

Imagine that you are buying a house but you can't see it - it's in a box - granted a big one. All you can do is to give it tests - does the water start when you open the tap ? Yes, says the developer, it does - trust me, we have installed the tap complying with the latest industrial standards. Can I see it, you ask ? No, it's in the box, but it works ! You tap something which sort of looks like one and hear the sounds of water. Uhm, ok. Does it have windows ? Yes, sure it does. Can I see it ? No, it's in the frikin box ! How can I be sure the windows are installed in walls, not in floors or are not even left leaning to a basement wall ? Trust me on this, we are professionals here. So much for a black box house testing. Now, white box testing is not much better. When applied to houses, white box testing wouldn't mean you are allowed in to look around - instead you are allowed to look at the blueprints. Oh, now it's so much better. As a home owner I couldn't tell the mentioned tap on the blueprint. Besides, having a finest project of all still doesn't mean the windows are where they are supposed to be.

Unable to see for yourself, how do you estimate the quality of software ? You go for much less reliable sources - the talk of the people who want to sell it to you. Now, that's who you can trust - the salesmen ! Their cheap talk..., but I don't have to tell you that. The words they are using, please, don't let me start - "industrial standard" - that's my favourite... Trade shows, web site reviews, free T-shirts, it's all part of the selling game.

But think - what speaks louder than all the advertisements, and is closer to the real situation ? Hint - it literally yells in your face, sometimes several times a day, you are so used to it, you are skipping it right on the spot by clicking "Next". What is it ? It's the software license. No matter if it's a proprietary license or whatever orthodox OSI certified open source one - the most important part of it says that the people who did it have no confidence in the thing they are selling you. It comes with no guarantee for god's sake. Where else you can buy a thing and rightfully expect it not to work right out of the box ? And you can't return it for a working one, he he he...

Don't ever expect the software not to be crap. You can write it yourself (that's what I personally always prefer), but it'd still be crap, although you'd have a much better hold of it and earn experience. Besides, you can't write everything from the ground up, it's insane. You may just as well pick your software at random and then clench your teeth and stick with it through years until the fact that it's crap is so obvious that you cannot bear it any more. I've seen it a few times when a 3rd party software (complying with industrial standards, no doubt there) bought for ridiculous amounts of money left everybody engaged for a while, harder and harder, until eyes pop out and somebody decides that it needs to be replaced with another one (complying with newer industrial standards).

There is one other aspect I'd like to bring out.

It's ever so cool to proclaim that "lives depends on the software" - makes you so proud and self-assured, like hey ! the software that we write here - everybody depends on it ! We are cool ! It's an exaggeration at the very least. Lives depend on software no more than they do on Hollywood movies - yes, they want it, and there is an outrage if theaters are shut down, somebody may even get killed in a riot, but it only concerns those who actually allowed the movies to take so important place in their lifes. Likewise with the software - if your word processor crashes, there is an Internet outage, your bank's ATM network is out of service, your flight is cancelled while you are in the middle of nowhere, or even if your plane goes down due to a software malfunction - does that mean your life depends on it ? No, it's your habits, your ways, your convenience that do. What's true to the point is that despite having no real importance to anyone's life, once it's been given the right to shape the lives - the software may not stop. We are used to it - it's all around, it's uncomfortable to live without it.

And so, where does this discussion leave me as a software developer ? I build crap, nobody really needs it, but everybody sort of expects that the crap is there every time they look. And so I have to keep the crap running. I come to work every day and keep pushing the wagon full of crap so that it doesn't stop. If I'm lucky, I can take a few hours free of pushing and try to build something different, something which is either less crap, or at least a better wagon someone will have to push. Still.

Perpetual movement of crap, that's what our discipline is at its current stage of development.

The difference between a programmer and an administrator

A good programmer knows how to do something. A good administrator knows how somebody else has already done something.

Acme Corp. Do something.

Take any big company ad slogan:

• HP. Invent.
• IBM. Think.
• Apple. Think different.
• Samsung. Imagine.

whenever it has instructions in it - to think, invent or do anything in particular. What a heck does thinking (even thinking differently) or being able to invent has to do with the commodities those companies produce ? What does it mean at all ?

Do they really want to say - "Buy our stuff and then do whatever nice things we are telling you to do" ? That would be ridiculous. Do I need permission to think ? Do I have to buy something IBM to apply ?

"Buy our stuff and then you will start doing it no matter if you want it or not" ? Buying a Mac unlikely will make you think differently. Acquiring different GUI habits may be, but not different way of thinking, otherwise nobody would ever be able to interact with Apple owner from the moment of the purchase on.

"Buy our stuff and then doing those things would be easier" ? Nah, I doubt if inventing with HP is easier than with anything else.

Oh, I get it, I'm probably supposed to associate HP with inventions as such ? IBM with thinking ? Whatever company with whatever pleasant activity ? Now, THAT would be really stupid, just to think that there is an association between buying some piece of equipment and engaging in any pleasant activity. Otherwise, where are "IBM. Think, do you really need it ?", "Samsung. Imagine something not made of semiconductors" or even "Don't believe everything you hear in commercials" or "Trash your TV and go see the sunset" ? That last one, wouldn't it be a pleasure to do ?

Anyhow, TV commercials, magazines ads and equipment cartons would be the last things I would use for motivation, thank you very much. Except for blogging.

So, how does it feel writing infallible code ?

I think many people feel uncomfortable about the code they write or maintain, systems they support or administer. The stuff is just bad, let's face it. But people (including myself) tend to find a comforting spot in which they are most emotionally stable and least disturbed about the reality they have to live in. But this comfort is often based on ignoring the truth. Let's see how it may work in one particular case.

I had this idea when thinking about moving one very small but very common piece of functionality into a network service. We have each system to determine whether a given day is a workday or a holiday. As there is no such stable rule, each system has its own volatile piece of data in whatever format which it consults, and every administrator of every system has not to forget to keep it in good shape. It repeats in tens of variations, burdens the developers and is a constant administrative pain. Sounds like a good idea to factor it out into a network service ?

Paradoxically, it does not, precisely from the point of view of those troubled systems' developers and administrators. But why not ? It seems odd. Before the refactoring they have

function workday(date):
  return consult_own_database(date) == "W"
...
if workday(today) then do_stuff()
...

whereas after refactoring they have

function workday(date):
  return consult_magic_service(date) == "W"
...
if workday(today) then do_stuff()
...

(mind you that this a pretty real-life looking code). It looks like nothing changes at all, except for now they have one less problem. To simplify, let's assume such change wouldn't require any work from them, will be done in a snap and cause no troubles in itself. Still they refuse. What is the likely reason they give to cancel the change ?

- It's less reliable. A call to a local database (or filesystem) is less likely to fail than some network service.

Reliability, that I understand. And it's also reasonable to assume that the probability of local database failure is less than that of a remote network service. But I argue that it's not the different degree of reliability that matters here. After all the above chunks of code are identical, why would one be less reliable than the other ?

The first (existing) piece of code does not indicate a less probable failure. In fact it indicates a zero probability of failure. The call may fail, but the code does nothing about it. The developer consciously or unconsciously knows that there is a possibility of failure, but as soon as it's perceived below certain threshold, she's still comfortable ignoring it. Now with the subtlest change comes an increased failure probability which is too uncomfortable to ignore.

And what are the options ? Rewrite the code so that it accepts failures ? Impossible, too difficult, can't be done, no way, period. Thus we don't need the change, it will break the system which works fine. Ironical, isn't it ?

And so the thing remains unchanged, with regards to this switch to a network service or not, it becomes too fragile - touching it is dangerously uncomfortable. Therefore the point of this argument is that developer's comfort based on false assumptions, such as improbability of failures, forbids any major changes in the software.

The pattern to rescue other patterns: The Golden Hammer

Patterns look like they can solve all of your real-life problem. To me, this is awfully wrong and can do more harm than good. Yes, patterns allow for organizing your knowledge and also for developers to find commond ground easier and they do awesome job there. But they are nothing more than a mnemonic constructs.

I sometimes hear sentences like

- We use pattern X here, because it's the right thing to do in this kind of project.

Not because the solution has the appropriate structure, and pattern X somehow emerges from it, but because such and such problems require such and such patterns. What a heck ? I thought problems require solutions, not patterns. To me, such talk is a bad sympthom - it means that people can't see forest for the trees. Or, which is also possible, that I can't understand what they are saying, but let's forget about this insane assumption at once :)

Now, for pattern lovers - the pattern to the rescue - The Golden Hammer. It says that if you are good at hammer work, all your problems appear to you as nails. Likewise, if you are good at patterns, you find the patterns you already know everywhere. The key point here is that you do not attempt to apply thought and find other kind of patterns, but blindly say - hey, this obviously asks for a bridge, and this is undoubtedly a mediator here.

It appears that most beautiful and influential things appear with reflection - a tendency to analyze oneself. The Golden Hammer is most reflectional as it implies you actually think about the process of applying patterns (including itself). And if you are more conscious in applying patterns - you benefit more, don't you think ?

Expecting software errors, not fixing one and pretending none left

There is a problem with your software, something strange is going on, some error appeared that has never had appeared before, or should have never appeared in the first place, the system produced insane output or just died ?

It's unwise to think about anything you are experiencing as of "unbelievable". You see it, you have hard evidence - it exists. Why is that - you have no idea, but it exists. Finding out the origin of such a problem requires a totally different approach to debugging. As now you have no idea why it happened, you likely could have never predicted it, tested for it or even thought about it.

Stop and think about it. You could have never thought your software would fail in that particular way. Taken to the extreme this would mean that you never thought your software could fail at all.

But this is exactly how novice programmers think. As they become more experienced, the amazing world of software failures appears before them, and they start applying the typical code/debug cycle, error handling and/or try/catch patchwork, which still limits them to the errors they are experiencing at that single moment.

It seems logical that if the programmer's experience is taken to another extreme, she would expect failures everywhere, just for the sake of vigilance. No matter if a failure of some kind has never happened before - it's there, waiting to happen.

The point of this argument is that the key thing to writing stable software is being failure-proactive and expecting everything to fail. Just ask yourself pointing finger to any piece of your system: "what happens if this fails (for whatever reason) ?" The results can be far reaching.

I therefore argue that the common debugging is far less important a process than it's usually thought. Of course, finding an error and fixing it is important and must be done, but if you had expected it, your whole system is by now prepared to any error anywhere, and this one error only challenges this fact.

Moreover, I also argue that trying to break your own software is at least as valuable a practice as the forementioned debugging.

Yes, it's a well known maxim that all testing is about breaking things, not about seeing it working. But there is a catch. Tester can only test against problems that he can think of, so much like the developer. Moreover, as soon as developers and testers work side by side (unless it's the same person), they tend to share the same narrow field of view, which greatly reduces testing efficiency.

That's why I like stress testing. Not because it shows me impressive transactions per second, but because it tends to reveal another layer of problems usually never seen. Stress testing is very useful, but it shines whenever a failure leads to a cascade of other failures. Whoa ! I couldn't have thought of THAT !

Now, how else you can intentionally break your system except by giving it a stress load ?

Failure injection comes to mind, but it contradicts to the normal development cycle (as there is simply no place for it), and requires the mentioned failure-ready system structure from day one. I also tend to remove the injected failures in production system, after all being a failure-vigilant software developer won't save you from black cats and broken mirrors.

Unplugging cables from the working system is also fun and can reveal yet another unexpected problems, although again, it's probably not applicable to a production system. For another joke of this kind see "The China Syndrome Test" in "Blueprints for High Availability" by Evan Marcus & Hal Stern.

In conclusion, this discussion again turns me to the PITOMNIK principle: as the system grows, its small errors grow with it and therefore lead to inevitable failure. You cannot ignore the presence of the errors, nor you can fix them any significant fraction of them, so you'd better be ready.

Software: too many layers, too few tiers

Layers and tiers are both ways of enforcing structure onto a software system. I will be using term layer = "logically isolated group of modules within a process" and tier = "physically isolated group of modules within a system". The difference is basically logical vs. physical.

Now, layers are way more popular and friendly to any development environment, because they

• are directly supported with the language and/or runtime
• allow greater flexibility and freedom of interaction between different layers
• are accustomed way of thinking about software

OOP with classes and interfaces are all about layers.

Tiers, in turn, are more difficult to deal with, because they

• require more upfront design
• require complex and/or restrictive and/or expensive (un)marshalling
• require different way of thinking

There also are obvious upsides to tiers, mostly about independence, ex.

• deployment and execution independence
• development, language independence
• reuse independence

CBD with components and interfaces are all about tiers. Right now the choice is basically limited to a few popular component-based environments, such as DCOM, CORBA or (somewhat differently) web services and SOAP.

I sincerely believe that having more tiers is beneficial and it would be great if there was a way of making tiers easier to use in a way similar to layers. And so, guess what, this is one of the ideas behind the Pythomnic (Python framework for building reliable services) - to allow fast and easy way of converting layers into tiers - to mix and match modules in any way.

For example, if there is a cleanly separable function (ex. CPU intensive XSLT transformation) currently allocated to a module or a set of modules in a layer, you may also take one step ahead and declare that this function can possibly be separated to a different tier. For example, instead of calling

pmnc.xslt.transform(...)

you do

pmnc.execute.on("xslt_transformer").xslt.transform(...)

the key thing to note is that the way execute.on works, if "xslt_transformer" RPC channel is not configured in an appropriate configuration file, the pmnc call will still be local, but as soon as you modify the configuration file and save it, the very next call will go to the specified server. There is no need to restart Pythomnic on this machine, all you need to do is to copy the xslt module to a separate server and start it there in its own Pythomnic thus turning a layer into a tier.

I do believe that such a feature is beneficial to a middleware development framework.

One note on Python simplicity in handling phoenix problems

One of the major problems in developing long running applications (and applications in general really) is cleanly handling shutdown. Among them a particular subproblem of phoenix singletons - what happens sometimes if one entity references the other (typically a global singleton), which has already been unloaded.

For instance, consider the following shutdown code in moduleA (using simplified Python syntax):

moduleA:
...
def shutdown():
    moduleB.callme()

there is no guarantee that moduleB has not been shut down yet. Now, if moduleB is also written in a delayed initialization fashion, ex:

moduleB:
...
impl = Impl()
...
def callme():
    if not impl.initialized():
        impl.initialize()
    return impl.callme()
...
def shutdown():
     impl.cleanup()

then what happens upon moduleA's shutdown is a reinitialization of the impl - one sort of a phoenix. It just went on me that instead of building a complex synchronization schemes to handle this cleanly, all I need to do to prevent this is just

moduleB:
 ...
def shutdown():
    impl.cleanup()
    del impl

and now as impl is just not there, the moduleA's attempt to reference it at shutdown will fail and throw - a much more appropriate behaviour in a given situation. This is less clean a solution, but how simple it is !

Shortcuts to icons to shortcuts: what happens if you click that ?

It seems that desktop icons have rather confusing nature. Let's say there is an object, e.g. executable file. Whenever it physically presents on the desktop, it appears as an icon. Whenever it's not on the desktop itself, which is a more frequent situation, but a shortcut, an object of different kin is, the shortcut appears as the same icon albeit with a little arrow in the lower left corner.

Hence problem #1. There are different icons on the desktop with and without arrows, but single-clicking on them reveals identical behaviour - the target object is invoked. What arrow is for then ? For the user to see which is "real thing" and which is a representation of a concept she can't quite grasp anyway ? Wouldn't it be more logical if non-shortcuts couldn't be placed on desktop at all ?



Next, surprisingly, there are other places on the desktop where icons appear - task bar, quick launch bar and system tray. Those add to the confusion, because although their icons look identical, their behaviour is even more different:

Icon on the quick launch bar represents a possibility to start an application, clicking it starts another instance. Icon on the task bar represents an already running application, clicking it brings it to front. Icon in the tray represents an application willing to talk, but what happens if you click it is not known beforehand.

Hence problem #2: wouldn't it be appropriate if all these icons were different to some degree, or behaved in more consistent fashion ?

On (information security) audit: giving money to the developers

Auditing information systems is fiendishly difficult. Think about it - a typical situation for a developer is discovering problems in the _small_ pieces of code that she's working on _right_now_. Few days later other problems may be discovered. Half a year from that - yet other.

Then, as the system is assembled, parts developed by different people come together, a whole new world of problems emerge. The people who built it have scattered knowledge of the system themselves.

Now, to audit. Suits come in, unpack their laptops, run standard tests, look (!) at everything and ask tough questions. A week after they conclude whether the system the very authors have no complete knowledge of is good or not. And then they leave.

Hence my point - a good team should be doing internal audits as it goes. A good developer should be running custom-tailored tests, looking at the thing, asking tough questions no worse than the auditors. And the knowledge remains with the company.

Therefore, why not investing the same money into team education, so that they become their own auditors ? It's the old "give fish" vs. "teach to fish" thing.

I realize there are PR and sometimes legal aspects to audit, but to a developer PR along with legalities don't make much sense.