Things That Require Further Thinking

Displaying date/time values in relative human readable format

2011-12-07T16:06:00.000+06:00

A typical problem which universally appears in any interactive software is displaying date and time. A typical web page contains articles post times, a phone displays times of calls and short messages and so on and so forth.

Too often the developers don't bother with readability and we see something like

Posted Wednesday 1/12/2000 11:02:15 AM

whereas the American M/D/Y date and 12-hour AM/PM time format is even not universally recognized. Besides, more often it is not necessary to have it so detailed. Seconds, for instance, who ever needs them ? Security auditors ? Are you one of them ? A better way of displaying the same information is having only the relevant parts of it displayed:

Posted 11:02 AM (when posted the same day)
Posted 1/12/2000 (when posted days ago and time of day is not relevant)

And even better is to use relative user-readable form:

Posted 3 minutes ago
Posted 2 hours ago
Posted yesterday at 11:02 PM
Posted yesterday at 23:02 (tip to the hat for respecting locale)

Such format is user friendly, but why don't we see it often ? Surprise - because you have to implement it and it is not trivial. Instead of just

strftime("%M/%D/%Y");

you have to jump through hoops, the easiest being depending on external 3rd party library.

And this is my question: why don't we standardize some percent-format which would display date and time in printable relative form ? We already have strftime printing days of week names, month names (short and long), is it possible to have something like

strftime("%R");

to produce a human readable date-time relative to current time ? Or even have a small language in it, such as

strftime("%RD"); (produces "today" or "yesterday" or "2 days ago")
strftime("%RM"); (produces "this month" or "last month" or "2 months ago")

A cheap technical solution to control election vote counting

2011-11-29T13:41:00.000+06:00

The elections are coming in Russia. As always, its results are expected to be grossly falsified. But instead of politics I'd like to talk about a technical solution to the problem.

What if an independent party is allowed to observe the process of vote counting, in order to come up with its own results which later can be used to verify the official numbers. How could it be done without forfeiting the traits of the voting process such as privacy, and also so that it is reliable and cost effective and causes minimal disruption to the existing process ?

Voting process in Russia is like this. You present yourself to the voting site and you are given an anonymous paper ballot. You go to a booth, mark one square with a pen and drop the paper to the ballot box.

So I thought that the step which could be modified is "marking the square with a pen". Instead of putting a pen inside a booth, a "square marking device" should be installed by a supervising party. It is nothing but a simple plastic box with a thin slot for ballot and, say, 4 marked buttons. You go into the booth, draw a curtain behind you so nobody sees what you are doing. Then you slide your ballot into the slot, press a button, and the device prints a cross in a corresponding square. Then you take the ballot out and drop it into the box as usual.

The marking device adds 1 to the counter corresponding to the button you pressed. Later the counters can be read by an authorized supervisor with a key. The officials count the votes by hand as usual after taking the ballots out of the box.

This most significant threat to this scheme is double voting. It should be protected against malicious user just pressing buttons at will to push up the counters. This could be done by reading a pre-printed unique bar code and not allowing it to be used twice. Bar code is also useful to be sure the ballot is positioned correctly.

As the amount of memory on the device is limited, you should probably pair a device installed to each site with the series of ballots dispersed to that site. This way, in the morning, the supervisor will unpack the device, pick random ballot from the pack and go through "imprinting phase" by inserting a ballot and applying a key. The device will read the bar code and lock itself to the range of ballots it will accept from that moment on. This will also protect from using ballots from other sites.

The solution can even be made entirely mechanical, no electronics required - it can be made simply a paper cutting hole punching device known in Russia as "компостер". You force press a button, and it cuts a circle fragment from a ballot, the cut out circle falls into a tray and remains inside the device, protected by a regular mechanical lock and key. Later, a supervisor opens it and counts the confetti. Each piece may contain a number or be of different color and may be watermarked to prevent forgery.

Neither of the solutions is bullet-proof. But it is simple, cheap and reasonably effective. The devices need to be tamper-evident, they must last just one day, they need a 2D barcode reader, 4 buttons, a key slot (touch memory reader ?) and 4 static printing heads. To cut on the latter and avoid ink refilling, even the electronic version of the device can be cutting holes in the ballot instead of printing marks on it.

I'm not a production expert, but I'd say such a device would cost a few dozen bucks. And you need hundred thousands of them, so it'd be even more cheap.

Bullshit character

2011-09-09T15:00:00.000+06:00

On temporary patches

2011-08-25T19:07:00.000+06:00

Once again the other side of a client-server link is broken. Our side works perfectly, but the peer sends in something incompatible with the protocol. This time it sends money value with 3 decimal places whereas there could be 2 at most. Our parser throws upon

amount=1.001

and surprise ! They can't fix it. May be later. And it's definitely gonna take some time. Some indefinite time. Or forever. But they need the solution. Right now.

Reluctantly I go and remove the scale check, and add the abominable money rounding. It kind of works, but having that patch makes me feel uncomfortable. It doesn't matter that in case of a future problem we'll probably get away by pointing them to the case where they clearly state their wish. It simply doesn't feel right.

Worst of all, it's not going to be fixed. Ever. How about (I think to myself) having a temporary patch, a time bomb of a kind, which ceases working after, say, a month. Granted, not a good idea, for many reasons, but for a while it felt strangely good.

And it's dead simple to do in Python.

from datetime import datetime as dt



def fix_by(d):

  def _fix_by(m):

    def _fix_by_check(*args, **kwargs):

      if dt.now() > dt.strptime(d, "%Y-%m-%d"):

         raise Exception("TOO LATE !!!")

      return m(*args, **kwargs)

    return _fix_by_check

  return _fix_by



@fix_by("2011-09-25")

def foo():

  print("ugly patch reporting")



foo()

Sigh...

Security tokens for the masses

2011-08-01T17:07:00.000+06:00

Two new cryptographic devices have hit the market just recently. They are both of "cryptographic security tokens USB dongles" kind, and there would be nothing to speak about unless they haven't both been supporting GOST family of cryptoalgorithms. As a quick reminder, in Russia it is not only mandatory to be using GOST, but the implementations must also be certified by FSB. And until recently, there has been just one certified hardware device to support GOST, namely MS_KEY. It is supported in our product since mid-2010.

Suddenly there are three of them - the newcomers are Rutoken ECP and eToken GOST. And our customers (banks mostly) want them, like, yesterday ! So I had to stop whatever I was doing and write supporting plugins for them. Not a big deal really, once the documentation is there, a device sample is available, and support is ready to answer a stray stupid question. And so now we have support for 4 types of security tokens, including the eToken PRO Java not mentioned before because it only supports RSA keys.

A lot of interesting things I've learnt in the process. In this post I'd like to focus on usability.

The target audience of our product are average non-technical users working from home with their bank accounts and such. And so we have to keep the things really simple, ask no questions, to which they don't know answers anyway, don't mentor about what they needn't know, offer no choice and in general just walk them through hand in hand.

And here is where certain tokens features become an obstacle.

In most general terms, a cryptographic token is essentially a tiny computer with its own CPU, memory and private filesystem wrapped in a plastic USB dongle. It has a tiny operating system in the ROM which knows how to perform a handful of operations, mostly of cryptographic nature, such as "calculate hash" or "generate digital signature". It plugs to the PC through a standard USB port and when the right drivers are installed can be sent commands to. Seen this way a token simply is an external cryptographic device.

But it's not all. To have their product admissible to the market, token manufacturers must conform to certain standards. Those standards follow a certain procedure, a security protocol through which the tokens are supposed to be used. This procedure essentially implies that a token is used in a corporate environment, where tokens are distributed among the employees by security officers in orderly fashion, there is a well defined lifecycle to each token, and in general - there is a trusted administrator to see if there is any trouble.

Guess what ? Home users don't have administrators. They are their own security officers. And so the entire procedure goes down the drain.

Consider this - an average home user comes in to a bank, signs a few papers and is given a plastic dongle over the counter. All we need at this moment is to generate a new key inside the token, protect it with a user-provided PIN code and print out a certificate request. All we need later is to collect the PIN code for the key from the user and sign stuff.

Some tokens, specifically MS_KEY and eToken PRO Java which we have been using before, allow just that. You create a key, protect it with a PIN code and be done. Not the others, Rutoken ECP and eToken GOST. They conform. Which means that every time the owner wants to use the token for anything, he needs to provide the PIN code to the token.

This is precisely the same situation as with cell phones. In fact, there is a token inside a SIM card. The PIN code to it you enter to begin using the phone.

There is a lot of problems with token PINs (as opposed to the individual key PINs) for a home user:

It is 123456 or alike by default and nobody cares to change it. This is bad, because it's a key.
And if they would, the disaster is just around the corner, because they forget it. This is bad because there is no established procedure to recover the token contents, because there is no trusted administrator.
It can be recovered following manufacturer-specific procedure, and this is there another magic PIN code (security officer's) comes into play. It in turn is 654321 and you cannot even begin explaining what it is for to the user, leave alone having it changed, written down and kept in a safe place.
A bank can have employees setting random PINs to tokens before giving them out, but it complicates the procedure, is all expenses and not only doesn't make the threat of hacker knowing the PIN to go away, but makes it specifically targeting the bank. It was them who knew the PIN after all. You can automate the procedure and have PINs printed in envelopes similar to how it is done with credit cards, but it's way more expenses.

Again, as with cell phones, how many people have their phones protected with PIN codes ? And of those, how many have their PUKs kept in a safe place ?

So in the best case, the user never changes the PIN and the token contents is accessible to anyone. And in the worst case, the user forgets his password in 3 days and the token is as good as dead. Or the other way around depending which approach to security you take.

Compared to that, when you lose PIN code to an individual key inside the token, no big deal. Generate another one, come up with a new PIN and there you go.

Returning to the tokens at hand.

Both Rutoken ECP and eToken GOST have their mandatory PIN codes, but Rutoken ECP at least supports protecting keys with separate PINs. This way we don't care about the PIN code to the token at all. It may be 123456 or whatever. The user is required to enter it each time and it is a nuisance, but each individual key is protected with a separate PIN just as before.

With eToken GOST it is worse - it only supports one PIN code. There is no way you can protect individual keys. Anyone who knows the token PIN has access to everything. This approach makes sense too, but it renders useless some interesting features of our application such as having multiple active keys on the same token, or having a new key generated over an old one and the new certificate request signed with the old key, thus escaping from visiting the bank in person when the certificate is about to expire.

Why am I not blogging more ?

2011-08-01T13:45:00.000+06:00

This is kind of a meta-post, a blog entry about the process of writing a blog.

And so, I'm thinking about what is it that holds me back from writing more ? Things worth posting cross my mind frequently, I observe them, make a mental note if you like but do not write down.

Problem #1
The process of thinking the idea over is fun. I keep thinking about the thing until there no longer are interesting aspects. Once this is done, I quickly lose interest to the idea, sort of letting it go. And this is exactly the moment when I should start typing it in. Oh, the bother.

Problem #2
Similarly, once I had thought the idea over, it no longer fascinates me, and there is no indication of whether it would be fascinating to anyone else. Kind of a preliminary positive feedback, which is impossible.

Problem #3
Writing is hard on its own premises. It takes time and effort to write well, no matter in your native tongue or not. What are the benefits from doing it at all ?

The outcome ? The step between having a thought and having it written down is hard. It's not going to happen unless I actually make it.

Another usability pearl

2011-06-07T18:09:00.000+06:00

One of my all time favourites:

Press ENTER to exit.

So short but blows your mind right away.

Transactional memory - nothing but trouble

2011-05-20T22:31:00.000+06:00

This post is a followup to a review that I have recently written for a book called "Transactional memory". Here I focus not on the book and its qualities, but on the matter it discusses - transactional memory. Still, when I say "they" and "their" I refer to the authors and indirectly - to the researchers they represent.

Why at all ?

To begin with, what are the problems they say transactional memory solves ?

Arrival of multi-core chips makes programs parallel.
Writing effective parallel programs is hard.

My first point of criticism is right there - there is no evidence that parallel transactions make a good generic programming model at all. Even if everything else was perfect, why TM programs would be efficient ?

But there are transactions in databases !

They quote databases as a proven example of successful transactional systems and kind of go from there. Ok, I buy that, but there are huge differences -

Databases operate on very high level objects - tables, records and alike.
Databases are about well defined objects with a rather limited behavior. You only can do a handful of operations to them. You cannot invent relational operators.
Because the operations are at high level, databases can to a degree "understand" what you are doing, and control the semantics.

With memory bytes you can do anything. There is no way to tell what you are doing in that transaction. It's very low level.

Also, about their success, database transactions don't make the task of writing an efficient and/or correct program any simpler. The guarantee of an ACID system is not that it's effective or easy to use, but that it tolerates various screw ups and maintains data integrity. You normally say something like "it's in a transaction, so nothing bad can happen to the data". You don't say "it's in transaction, therefore it runs fast" or "it's in transaction therefore it is correct".

Anything good at all ?

The only thing that could possibly be good about transactional memory is that, just like any other enforcing formalism, it makes possible to reason about the correctness and behavior of a program. Proof of correctness, anyone ?

That'd be great, but proof of what ? Again, working on a very low level, analysis of a TM program will reveal that there is no conflict between accesses to memory addresses such and such. Uh-huh, thanks.

Transactional memory is shared

For a technology whose goal is supporting multi-core computers (and we are talking about hundreds of cores, right ?), TM is surprisingly insistent on shared memory model.

Although theoretically there is nothing wrong with 100 logical cores accessing the same memory, we live in a real world and in real world hardware rules. And hardware dictates that cores should be as independent as possible. Even in SMP architecture there are per-CPU caches, and you really should keep data accesses affine to a CPU, otherwise cache coherence would kill your performance. In NUMA architecture you don't have a privilege of a uniform view of a shared memory. And in clusters you don't have shared memory at all.

Now, why sticking to an architecture that already scales poorly ?

Transactional memory doesn't play well with anything

There hardly is anything in existing hardware or software which is not alienated by TM. Instead of having support from your commodity hardware and momentum from existing software development practices, you have to fight it.

Hardware:

TM doesn't work well with caches. Shared memory doesn't in general.
TM doesn't work well with CPU optimizations such as memory access reordering.
TM doesn't work well with non-shared bus architectures. Good bye NUMA, clusters, transputers.

Software:

TM doesn't work with locks and waits. Most of current parallel programs use locks and waits.
TM doesn't like global heap and especially garbage collection.
TM doesn't like exceptions, because it is unclear what to do in case of exception.
TM doesn't like I/O because it is irrevocable. Because you cannot un-print something from your output, you have to invent workarounds like "transactions that are guaranteed to commit".

Even worse, TM conflicts with itself. Consider having a transaction in a transaction. This can happen easily if you begin a transaction and then call a library which begins a transaction of its own. And it gets ugly fast. What if the inner transaction committed but the outer has to rollback ? What if there are multiple inner transactions ?

Is there any surprise that existing databases don't support arbitrarily nested transactions ? Are you saying there should be a single memory transaction per execution context, as in database ?

So what can you do in a transaction ?

Because of the mentioned hardware implications, there is not much you can do in a memory transaction.

1. The amount of memory that you can access in a single transaction is limited.

For many TM systems it is limited with hardware cache size, and this is not much at all. Caches contains lines and by touching a line you effectively poison it, so if you access 2 bytes in separate lines you use up 2 cache lines. Also, concurrent transactions have to share it. What good is a 4M cache if it is only 128K lines and there are 128 parallel transactions ?

2. The amount of time that you can spend in a transaction is limited.

For many systems the length of the transaction must be less then OS scheduling quantum. Put another way, you cannot have a context switch in a transaction. And this is how much ? 1/1000th of a second ?

There are such TM systems that allow dirty transaction data to spill from the cache to the main memory, and I presume there are such complex ones that allow context switching and arbitrary execution time, but their performance is to be seen.

So, what can you do in a transaction with such draconian restrictions ?

The answer is simple - memory transactions are only usable in low level libraries, implementing higher level objects such as data containers. For example you can have a multiple-producer multiple-consumer queue, a tree that allows access highly parallel, or some equally wonderful thing.

But is it worth it ? I agree, it takes an expert to write a tree using locks or some magic non-blocking technique, but it's only done once. Are you saying that TM is good because now anyone could write a queue in an hour and it will be correct and efficient ?

And you cannot use memory transactions in your applications. It is either impossible or grossly inefficient.

And the answer is ?

I do believe message passing is a better way to go. The technique goes under many names, but the idea is the same - there are separate processes with separate data and they interact only in a well-defined manner.

It has no ACID properties, but heck, so does the real world. Like I always say - there are no transactions in real world.

Notes on implementing TLS. #2: Version differences.

2011-05-17T12:58:00.000+06:00

The first question when starting with TLS is this: which version to implement ?

I started with version 1.2, the latest and presumably the best, initially using its specification as a reference. The other two versions I just skimmed over and since there were no apparent incompatibilities, proceeded with 1.2. Later on, when it was already working, I backported the differences.

1. What are the differences ? Are they significant ?

The real question is not about knowing what exactly those differences are, but about deciding whether it is feasible to support them all of the same codebase with minimal efforts. Luckily, it is.

There are two major formal differences between the versions.

1.0 vs. 1.1+

IV handling. When a block cipher is used for traffic encryption, the key remains the same but IV varies with each subsequent packet. Version 1.0 uses tail of previous ciphertext as next IV, while versions 1.1+ transmit the next IV encrypted inside the previous ciphertext.

1.1- vs. 1.2

Ciphersuite support. Version 1.2 allows using any cryptographic algorithm of your choice anywhere in the protocol, restricting only the packet parsing rules. With versions 1.1- you could replace some of the cryptographic primitives but some others were hardcoded.

There are minor differences in packet formats which are easily handled with a few if's here and there. On the other hand, these raise questions regarding interoperability between different versions client and servers.

2. How to test different versions ?

As far as testing goes, my main concern is correctness. In the initial phase, when my understanding of the protocol was still weak, it was invaluable to test against existing, presumably correct implementations. It allowed to quickly try the options which were unclear from the spec. Later on, as the product stabilized, I turned to loopback testing, leaving other implementations for regression testing.

The second testing concern is compatibility. Although our product is proprietary and compatibility is not a main goal, it is a sign of a good implementation and I pursue it. There are corner cases and behaviour quirks that you find out only when examining existing implementations and the more of them you try out - the better. Every successful connection to a product of others makes you more confident in yours.

Protocol versions pose a serious problem with testing. TLS 1.0 is the oldest and most widely supported, you can utilize pretty much any 3rd party TLS implementation to test against yours. On the other hand, not all of them support even 1.1 not to mention 1.2 and this hinders testing significantly. Most of the time I used OpenSSL which supports version 1.0 and GnuTLS which supports all versions.

3. GOST support.

Our top priority is using GOST family of cryptoalgorithms. Russian regulations mandate that GOST be used everywhere, and nothing else. It is unclear whether GOST is allowed to even co-exist with another cryptoalgorithm within the same system so to be on the safe side we have to replace all the TLS cryptographic primitives with their GOST versions. This is why TLS 1.2 is the only viable choice.

There are existing products that officially declare TLS 1.0 (and even SSL) with GOST support, and I have no idea how's that possible. My guess is that they did break the protocol by illegally inserting GOST where it was not supposed to be and to hell with compatibility.

My other blog

2011-05-05T17:07:00.000+06:00

I also blog in Russian now, here.

Notes on implementing TLS. #1: Reasons for.

2011-04-17T20:53:00.003+06:00

The last few months I've been busy implementing TLS protocol support for this product of ours. It's in fairly good shape by now and I'd like to share a few thoughts.

First of all, why having own implementation when there are existing ones ? Actually there are quite a few reasons.

1. We need it to support Russian GOST family of cryptoalgorithms.
2. In Russia one cannot freely implement GOST, only using of certified libraries produced by state accredited companies is allowed.
3. We need it tightly integrated with our existing product. For example different such crypto libraries may need to be used simultaneously.
4. To make things worse, one option is to have crypto operations done by hardware token or smart card.

Therefore in the worst case you have USB dongle with GOST support inside, and you have to pass all crypto operations through it. Not just the private key operations, but even simple hash you can't do outside.

One other thing is that the product is not crypto-centric. We need a client-server tunnel in which the server supports hundreds if not thousands connections and utilizes multiple cores effectively. This asks for a different approach in which you think about the server architecture upfront and leave a modest place for cryptography. It leaves a lot more room for maneuver.

Besides, doing things is fun !

Better programming is just a way of telling no later

2011-04-08T23:04:00.003+06:00

You work hard to build clean and extensible architecture to help your product withstand every possible requirement, hoping this will allow it to live long and prosper. And it will, but not for long. No matter how good your skills, one day some customer wants something so awkward that your only choice is to tell him no. No, this is not possible. No, this is not going to work. No, this is technically infeasible. You'd love to, but no.

But then, isn't good programming just a delaying that moment of telling your customer no ? Of reasoning finally backed with technical impossibility ? You had jumped through hoops implementing every possible wish while it was possible. Not anymore. You say no.

On the other hand, if you had your software written badly, you'd have this moment of truth ages ago. This could have provoked confrontation with the customer but perhaps would make both of you realize the limits earlier and come to a different, even non-technical solution, don't you think ?

Notes on implementing MongoDB driver

2011-01-10T12:56:00.012+05:00

It's been a week-long statutory holiday in Russia and I spent a few afternoons implementing MongoDB support for the Python 3 application framework Pythomnic3k.

It turned out to be simpler task than I thought, a few odd hours here and there, given that I wanted to implement everything from the ground up. Still, there are notes I'd like to share afterwards.

Note #1: BSON

BSON is "Binary jSON" - proprietary binary protocol for serializing pieces of JavaScript for transmitting or storing. It sets the byte representation rules for simple JavaScript types such as "Integer" or "String", and also for MongoDB-specific structures such as "Regex" or "JavaScript with scope".

Supporting BSON is #1 requirement for MongoDB driver. Essentially, BSON is all there is to it.

Its entire specification is just two pages long, describes formats for 20 different objects, and for such a small spec is notably awkward. There are different ways to serialize similar objects, there are deprecated objects, and there are mysterious objects with unspecified format, probably reserved for internal use.

For example, String is serialized as

(length)(content)(NULL)

but Regex is serialized as just

(content)(NULL)

Also key/value tuple is serialized as

(value type)(key)(NULL)(value)

which is strange because while deserializing you cannot simply read key, then value type and then switch to an appropiate type parser, which would be possible for

(key)(NULL)(value_type)(value)

input.

Note #2: No response to packets

Application communicates with MongoDB engine using request packets, OP_THIS, OP_THAT and OP_SO_ON. To some of them the database responds and to some it does not. In fact, it prefers to remain silent, responding only when it has to.

For example, if you want to find something, you send OP_QUERY then you do receive a response, OP_REPLY, containing the data you've been looking for. But if you want to insert data, you send OP_INSERT and (surprise) there is no response.

I understand that with MongoDB there are no guarantees of data integrity, no persistency and no transactions, but now there is even no ack from the database. It makes me feel uncomfortable.

Note #3: Explicit ordering of documents

This one is ugly. The main data structure used in BSON hence in MongoDB is a document which is a dict or associative array. And associative arrays have no order. They have iteration order which is arbitrary implementation dependent likely derived from internal hash function. That order may even be well defined, for example smallest key comes first, but that could only work with comparable keys. In any case dicts don't have n-th element.

The problem appears when the request packet containing document is serialized to be transmitted. MongoDB relies on the order in which the dict appears on the wire. Specifically, if you send a command to the database, the first key must contain the name of the command. And in some cases the command requires positional arguments to be transmitted in defined order. So you have to iterate over a dict in a specific order only known to the caller.

For example, if you have a dict

{ "foo": 1, "bar": 2 }

and you want to invoke

db.foo(bar)

with it you have to serialize it like this:

[foo=1, bar=2]

but if it is

db.bar(foo)

it is the other way around.

The abysmal

{ 1 => "first", 2 => "second" }

kind of dictarray has also found its way here, it is the way BSON serializes lists, but this is at least an implementation detail hidden from the application inside the driver.

Note #4: Similar things in different ways

This is easy to illustrate with the way MongoDB reports errors.

For one, as already mentioned, some of the requests have no response whatsoever even though they might fail.

Then, if you get an OP_REPLY response, it has bit flag QueryFailure, which if set should be accompanied with a single document with $err key in it containing error message. But it also has bit flag CursorNotFound, which apparently is also an error condition, but then it has no message.

Furthermore, if the request has been a command (which only the caller knows), the OP_REPLY is a success, but it contains a document with $ok key that can be 0 in which case there is also an $errmsg key containing error message.

So in the first case error reporting is done on packet level, and in the second case - on application level. And in some cases no error is reported at all.

All this leads to the final

Note #5: Ad Hoc-ish

The exercise of integrating with MongoDB leaves the impression of incompleteness. It feels like this database (and I suspect other currently existing NoSQL databases) is an experiment, an early prototype.

Well, they have 40 years of catching up with relational databases to be as well understood and implemented, it's a long way to go, so I do wish them good luck.

A simple way to tell whether a programmer is working

2010-09-14T11:53:00.002+06:00

If both hands are on the keyboard then he's working, but if one hand is on the mouse then he's not.

Walking with Pythons

2010-07-20T13:19:00.003+06:00

My previous posts about how long it takes for Python 3 to become mainstream had a few comments, both on- and off-site. Those comments written among others by members of Python team, pointed out that there may indeed be unspecified reasons for keeping two identical parallel versions for so long. Well, they know better.

But I still believe in two things:

No matter which version of Python 2 you decide to leave for Python 3, there is still an incompatibility gap you have to cross, and postponing doesn't make it go away.
Python 3 still offers no benefits compared to Python 2, even for new green field projects. And this means that new projects are still started in Python 2.

Anyway, we'll see. Python is a great language, no matter which version. Since the project I'm worrying about is itself a framework with very little dependencies, I will keep the development running easily, language not to blame.

Will Python 2 ever end ?

2010-07-06T18:39:00.003+06:00

It so happens that I have been an early adopter of Python 3. Ever since the first alpha was released back in 2008, I've started rewriting my middleware framework from Python 2 to Python 3. I have to admit, it was beneficial for the project. Not because the language was that better, but because of the ground-up redesign and rewrite, cleaning up the mess. What bothers me is that 2.5 years later I can't expect any significant number of users to even look at the project, because it is in the "future" language.

I'm totally confused. Python 2 is still mainstream and is undergoing active development. Python 2.7 may be the last of the 2.x branch, but still it is released in the middle of 2010. Which gives it at least another 1.5 years of joyous lifetime, even if is declared dead right now. Which means that Python 3 is still not there until 2012. Which means that for 4-5 years it has been a better language which very few actually use.

What I find absolutely ridiculous is the transition strategy. If you want to break compatibility with the previous version - for God's sake, just do it. And the new features - they are supposed to be the sweetener, the bait for the users to make the move. It makes no sense to keep backporting the new features into the already more popular legacy version. Absolutely no goddamn sense. What you get in the end is two versions, otherwise identical, but when the first one is active, popular and widely used, the other one is... uhm... incompatible with the first one ? Seriously, except for the broken compatibility, why Python 3 is different from Python 2.7 now ?

Are the users supposed to switch to the new version, when all they are facing is incompatibilities and no benefits ? This is so wrong...

Re: Cryptography

2010-06-13T13:25:00.008+06:00

This post is a response to a recent discussion on a "Russian Software Developer Network" forum. The thread was called "Cryptography".

Oh, the drama ! And professionalism level was unrivaled. It was there that I found a new addition to my personal hall of fame:

epileptic curves

Seriosly though, it somehow happens that cryptography becomes the easiest part of security. Easiest to know about, easiest to talk about, easiest to show off with.

Why ? I'd say it is because it is closely related to mathematics and mathematics brings in the safe harbour feeling to those who want certainty in the shaky world of security. Besides, many of those who talk passionately about cryptography (including myself) have mathematical background.

Surprise, the security-related feature of cryptography is not based on hard mathematics. See, the feature that we seek most in cryptography is called "strength". We want it for encryption, for hashes, for digital signatures, for everything. It is strength which causes holy wars on forums. But what is it ?

In cryptography, strength is the ability to withstand currently known attacks.

See the problem ?

The words "currently known" reduce all hopes for certainty to dust. You cannot "prove" strength in mathematical sense. Anything is strong as soon as it hasn't been demonstratively broken.

There is not much reason comparing strength as well. As seen on the Internet:

My kung-fu is stronger than yours by 2¹⁷

But it only makes sense if you compare identical or very similar algorithms - then you are essentially comparing their lifetimes. As we assume they both haven't been broken yet, the larger the power, the more time on average it takes the attacker to break it using some kind of brute-force attack.

Put simply, all cryptographical strength is based on one big assumption - that the good guys know better than the bad guys.

We believe something is strong because noone has published the way to break it. Even though such way may exists, and may be widely used against us, we still consider it strong until the contrary appears on paper.

The biggest paradox here is that we are even sure that there is a way to break it, it is just that noone (meaning the good guys) has found it yet. And we hope noone (meaning the bad guys) will while we are using it.

We believe that the respectable scientists work hard trying to break every known algorithm and we are somehow sure they break them first. And publish. Not for money, not for fame, just for the sake of it. What were the names of the people who published attacks against MD5 ?

And the bad guys have much better position. They need to attack just one algorithm, or even just one key. They have enormous resources and motivation to do it. They might have affected the design of the algorithm to put a backdoor in it in the first place. And they don't need to publish their results, but silently exploit it for years.

Well, the good guys seem to be winning so far. Or do they ? You never know. This is called security.

Python 3 frameworks anyone ?

2010-05-27T19:03:00.002+06:00

First, I'm happy to announce that I have just released the next version of Pythomnic3k, a Python 3 framework to develop SOA middleware.

But I'd also like to share with you the big question of this Python 3 framework.

I have been working on its predecessor Pythomnic (similar, but written in Python 2) in 2005-2007 using it for integrating various systems in some bank. It worked, but as any software being developed in ad-hoc manner became messy over time. Not to mention the fact that as I learned Python, the old code looked uglier every day.

And so, as of early 2008, along with the first Python 3 betas being released, I decided that Pythomnic needed a complete redesign and rewrite exclusively in brand new Python ! Pythomnic3k was in development ever since. It has a nice and clean upfront design, based on 3 years of experience with Pythomnic, it's written much better, and it is has extensive self-tests. Which is to say, it is a quality piece of software. I spent next 1.5 years polishing it, until release 1.0 was finally published in 2009. Release 1.1 which I believe I've already announced, came out after some 8 more months of refinement.

All this time I kept using it for what, for the same integration tasks - connecting point A to point B, transforming messages, supporting various protocols. In the company I work for, it is used for delivering bank transfers and billing payments, sending SMS notifications (contains full implementation of SMPP 3.4 among other things), providing cryptographic network services of various sorts, and just about anything. In short, it serves as a middleware glue, and if I'm allowed to judge, it fits the bill.

Now, the big question is - was it really beneficial to switch to Python 3 starting a new development ? Take a look at the list of Python 3 packages. The language is around for 2 years, and there is like what, 50 of them ? Out of which many are one-module utilities ? Give me a break.

Python 3 looked promising, although it was not immediately apparent, what new features are the killer ones. Frankly, I'm still not sure. I love the syntax improvements and the correct str/bytes, but what else ? Am I missing the wave or it is not there yet ?

Anyway, Pythomnic3k architecture has very little dependencies, it is a pretty much self-contained framework, which means that it doesn't suffer from the lack of anything in particular in Python 3 libraries, but I would still love to see more Python 3 libraries around to have them plugged to the framework.

You come to software market ...

2010-04-02T09:35:00.005+06:00

... and you want your software cheap, fast and of high quality. You my friend want to be fooled, and you will be fooled, because nature cannot be.

How are they going to shut the Internet down ?

2010-03-12T18:32:00.003+05:00

Well, I never would have thought that my first post after such a long period of silence would be like this, but this is what bothers me.

Given the current political situation in Russia, in which power belongs to totally corrupt organized crime, the Internet remains the only media where anyone can speak out. For doing so you still may be prosecuted, but this is the only place where one can at least publish an unpopular opinion.

For example, check out the Internet shit storm (available mostly in Russian) on the topic of outrageous stealing as much as $50 billion of budget money under state-approved "make drinking water clean" program.

And so my question is - how are they going to shut the Internet down and how soon ?

Faulty character decoding as the last line of anti-spam defense

2009-06-19T14:53:00.004+06:00

I receive spam every day. Filtering is in place and everything, but occasionally some garbage gets through. And then I may look through it, briefly, less than a second perhaps before I hit "Delete", but the eye is fast enough to read and understand more than I'd want to. Then you might say such kamikaze message still had succeeded.

Much of the spam I receive is in Russian. As a side note, Russian characters have multiple encodings - WIN1251, KOI8-R, CP866, ISO-8859-5 and the universal UTF-8 come to mind. This means that the mail client has to properly understand the encoding and decode the message so that it can be displayed correctly.

I use Thunderbird, and it is just awful in decoding Russian messages. I don't have any idea why is that, but I have to manually specify encoding for every last message, because they always appear garbled.

But then, the bug becomes an unexpected feature - the spam messages look undecipherable just like legitimate ones, and even though I look at it, nothing is imprinted in my mind, and I just hit "Delete".

Software architecture

2009-03-30T18:43:00.001+06:00

is what you explain to somebody else so that he understands the matter.

This is Python: context managers and their use

2009-02-05T12:15:00.006+05:00

Python allows the developer to override the behavior of pretty much everything. For example, as I explained before, the ability to override the "dot" operator makes all sorts of magic possible.

The topic of this post is similar magic enablers - "context managers", defined in PEP-343. I will also demonstrate one idiosyncratic context manager example.

To begin with, it is important to note that Python reasonably suggests that when a developer modifies the behavior (i.e. the semantics) of something, it is still done somewhat in line with the original syntax. The syntax therefore implies a certain direction in which a particular behavior could be shifted.

For instance, it would be rather awkward if you override the dot operator on some class in such way that it throws an exception upon attribute access:

class Awkward:
   def __getattr__(self, n):
       raise Exception(n)

Awkward().foo # throws Exception("foo")

It is a possible but very unusual way of interpreting the meaning of a "dot", which is originally a lookup of an instance attribute.

Having this in mind we proceed to the context managers. They originate from the typical resource-accessing syntactical pattern:

r = allocate_resource(...)
try:
   r.use()
finally:
   r.deallocate()

Such code is encountered so often, that it indeed was a good idea to wrap it into a simpler syntactical primitive. Context manager in Python is an object whose responsibility is to deallocate the resource when it comes out of its scope (or, context). The developer should only be concerned with allocating a resource and using it:

with allocated_resource(...) as r:
   r.use(...)

In simple terms, the above translates to:

ctx_mgr = ResourceAllocator(...)
r = ctx_mgr.__enter__()
try:
   r.use()
finally:
   ctx_mgr.__exit__()

I note a few obvious things first:

Context manager is any instance that supports __enter__ and __exit__ methods (aka context manager protocol).
A specific ResourceAllocator must be defined for a particular kind of resource. The syntactical simplification does not come for free.
Context managers are one-time objects, which are created and disposed of as wrappers around the resource instances they protect.

What is less obvious is that a class can be a context manager for its own instances, there need not be a separate class for that. For example, instances of threading.Lock are their own context managers, they provide the necessary methods and can be used like this:

lock = threading.Lock()
with lock:
   # do something while the lock is acquired

which is identical to

lock = threading.Lock()
lock.acquire()
try:
   # do something while the lock is acquired
finally:
   lock.release()

Finally, I proceed to an example of my own.

See, I tend to write a lot of self-tests and I love Python for forcing me to. And some of the tests require that you check for a failure. Long ago I used to write code like this:

try:
   test_specific_failure_condition()
except SpecificError, e:
   assert str(e) == "error message"
else:
   assert False, "should have thrown SpecificError"

which made my test code very noisy. I have even posted a suggestion that a syntactical primitive is introduced to the language just for that. It was rejected (duh !).

And then I wrote a simple "expected" context manager which makes exactly the same thing for me every day now:

with expected(SpecificError("error message")):
   test_specific_failure_condition()

See how much noise has been eliminated ? How much clearer the test code becomes ? It is not a particularly "resource-protecting" kind of thing, but still in line with the original syntax, just like I said above.

The "expected" context manager source code is available here, please feel free to use it if you like.

To be continued...

No sense in sensors

2008-12-28T22:27:00.002+05:00

I like buttons. I like handles. I like dials. I like doorknobs. I like doors for that matter. I like physical controls whose shape and feel suggests their usage and whose usage provides physical feedback. If it clicks, budges and moves, then it's good. When it is in expected position and its usage is apparent from its form, then it's good.

Sensor controls makes no sense to me. I hate smearing fingers against black glossy surface, with unclear outcome. Did it work ? Did I activate the right icon ? I hate it when controls are not really controls, but images on the glass. I hate it when controls change their places, look and functions depending on what I am doing.

Even my stove is black and glossy, with no buttons but tiny engraved white icons. Makes it easy to clean for sure, but using it feels nowhere like pressing a button. Oh well, at least the icons are always in same positions.

iPhone, yes I tried it. Could have spoken through a cigarette case instead. Doesn't feel like phone at all. Large flat nothing.

Now, why sensor controls are so popular these days then ?

The way I see it, sensor controls are cheap alternative to good interface design. See, if they knew what this thing would be used for, they could have spent time and money on design and give it a nice interface, specifically for its function.

But there is a problem - they don't know what the thing will be used for. Instead they plan to use it for something no one could imagine at the moment. And they don't want to cast it in stone. They want to leave their options open, so that the interface can be changed later through software update.

From the manufacturer point of view, the sensor panel is the ideal instrument to implement any interface they may need in the future. It is a way to secure investments, rather than make it more pleasant to use. And the rest is nothing but fashion, done through professional advertising, product placement and bandwagon effect.

Ideally, people need to be placed in a world with indifferent black walls, with the content dynamically downloadable from the BigCorp site. Virtual reality, that's what it is. Opaque screens instead of windows, so that you can choose a "view". Dumb sensor panels with fake buttons. Smaller packages with more useless contents. Things that you have no control over.

And I hate it. I like real things.

Pray to rest of the best remote banking solution in Russia

2008-12-08T11:31:00.004+05:00

Alas, the bank I've been working for for the last five years got a lower hand in a merger. What it means to IT, does not need explaining. Everything we made is slowly dying out.

While it's still boiling, perhaps it's time to look back, and think about what's been done.

The good:

1. Still, our remote banking solution for the last two years in a row has been rated the best in Russia, and the forum full of client complaints for future shutting it down is also a good indication of that. And I am honored to belong to the team to have made it.

2. The side open source project that I have been developing for years during this employment, Pythomnic, I have luckily had time starting early this year to completely rethink, redesign and rewrite from scratch in new Python 3.0. It is a framework for integration in enterprise network using distributed network services. SOA, EAI, you name it. Essentially, this is what I have been doing for the last five years in the Internetbank project. We have even managed to write a few production services with the new framework and port a few from its previous version. If you don't mind me saying, it is a high quality piece of software, well (re)designed and (re)written. This project I will be working on for the years to come.

The bad:

1. The recession is getting worse. Not too good having to look for a job at times like that. Mind to take a look at my slightly outdated resume ?

2. I still can't force myself to release software the quality of which I consider low. What it means is that I tend to work thoughtfully and thoroughly, but yes, slowly. I could have argued for and against such approach myself, but not in this post. Anyway, such habits don't play well with modern freelancing. Who needs quality today ?

Therefore, pray to rest of the wonderful Internetbank project and if you like pay attention to the Pythomnic3k framework - I hope it is worth your attention.

Why do e-mails have subject ?

2008-10-16T12:06:00.009+06:00

Real mail doesn't need subject, nor headers of any kind really. Could you imagine

From: Leo Tolstoy
To: Anton Chekhov
Subject: Re[2]: War and peace
Date: 16.03.1899

My dear Anton,
...

?

What is the point for e-mail to have headers anyway ? Some of them are transport level technical details. For example, To and From field serve about the same function as the physical letter envelope with handwritten addresses on it. But subject, what is in subject ?

It always takes me considerably more time to come up with a sound subject, and it still almost always says nothing about the contents of the letter. What's the point ?

Is it presumable e-mail volume, so that the user could just look over the long list of subjects without actually opening it ? Or is it limited space on 1970s terminal screens ? Or it is just a technical artefact for the sake of e-mail indexing, storing and referencing ?

Anyhow, right now, neither subject, nor From, nor To fields mean anything.

If a given e-mail is indeed a mail message sent to me, then I don't care about neither To (which is implicit - me), nor From (which is expected to be politely included in the body) nor subject (which, like I said is meaningless when written by a well-meaning sender). I simply open the message and read it entirely.

If, on the other hand, the e-mail is a spam, I care about From, To, or subject even less. I just trash it (in fact, my e-mail filter does it for me).

Then, either way, I care only about the contents, not about From, To or Subject. The key problem is really in separating letters from noise. But then From, To and Subject don't help it either. What's the point in having it ?

Google, DNS and finding stuff on the Internet

2008-08-26T13:12:00.004+06:00

What if you've encountered Internet for the first time ? World-wide-web for that matter. Someone opens you a browser and says

- This is Internet, it has everything. Just type in an address of a site you want to visit.

Er, excuse me ? An address of a site I want to visit ??? WTF is that supposed to mean ? Anyone remember the address of the Pyramids ? I wouldn't mind visiting that particular site.

But really, what is a site address ? It is merely a reflection of a technical detail of the physical network organization. It just so happens that for the sake of unambiguous data delivery each computer on the Internet needs its own unique address. Now, the techies that invented it in 1970s just chose such address to be an integer number. If it was for them, or shouldn't the count of connected computers have exploded, numbers could have been used just as well:

- Connect me to server 12345 !
- You got it.

But people are notoriously bad in remembering numbers, and so there emerged a service similar to the yellow pages where each address could be given a name, and conveniently looked up later. Then it went like this:

- The new server is at great.new.site.com

and the user never bothered to translate "great.new.site.com" into 12345. The responsible domain name system (DNS), the ubiquitous service for looking up pieces of information by name is quite fascinating. It is perhaps the biggest distributed database in the world, and its capabilities have been largely underutilized over the years. May be this is why it is still up and running.

Presence of the DNS became as important as physical network connectivity. If there is no DNS, the Internet might as well be down. If you care to notice, it is exactly DNS where mainstream operating systems have their like only built-in redundancy. You are actually encouraged to configure multiple DNS servers at once, just in case one dies.

Well, DNS being a nice thing, it still got it own idiosyncrasies. There is really no reason for the site names to be organized in a dot-separated hierarchial fashion. In other words, in

www.yahoo.com

there is no need for neither "www" nor "com". Yahoo is the name, but the rest is irrelevant. The whole "dot separated" thing and "com" are just technical nuisances which made the development of DNS technically feasible, so that the database could be distributed more effectively. And "www" is nothing but a habit, a meme introduced to the culture. The sounds of "double u, double u, double u" and perhaps the visual rhythm of letters www immediately prepare anyone familiar with the Internet that a site address is being transmitted. Synchronization bits if you like.

So, what matters is the "yahoo" part, right ? The name. But the name of what and what's in a name ?

First, I'll go about the "name of what" part. World wide web is de-facto a hypertext, a billion of files intertwined with mutual links. Accordingly, what you type in is but an entry to the web. Once inside, you neither type nor care to remember any more names nor addresses, you just keep following the links. Have you ever stared at a blank browser page trying to invent another name which to type in just to see what comes up ? That's the idea. Any name could be tried as entry gateway, but picking them at random is extremely ineffective. Whenever one has multiple entry points to the web, he has to write them down, which is a starting point for a personal bookmark catalogue, doubtfully a popular sport any more. Instead it happens that everyone has like ten favorite entry points to the web, the ones that are fashionable, familiar, have catchy names or refer to the person's location or interests. Ok, so each user has his own favorite entry points to the web and they are the only ones that need names.

What's in a name then ? Oh, it is then totally irrelevant what exactly the name is. www.google.com, www.wikipedia.org, www.reddit.com, www.e1.ru, www.kazna.ru whatever is meaningless but catchy or meaningful but easy to remember in connection to some relevant topic.

Google is a catchy name and it presents the most rich and the most poor entry page at the same time. See, it might look like it helps, when you type www.google.com and the simplest possible page pops up and says: hi there, just type in what you need. But it is the same question we have started from - just type in what you need ! The only difference is that before we had to type the name of a single site, presumably known beforehand. Now we have to try keywords until we find something.

One point here is that the DNS names of the sites are largely irrelevant. A name of a site used to be the single keyword available for finding it, but no more. Now you are far more likely to find a site through a right query to google.

Another point is that is that google and the likes perform the same function DNS was supposed to - for relieving the user from remembering addresses and looking up relevant sites. Truly distributed DNS mapping site names to addresses became the part of the physical network (on the right ISO layer if you care), and got replaced by centralized mammoth server farms that map keywords to pages.

Finally, this switch gave enormous power to a proficient user, but for the average user it is still a blank stare at

- This is Internet, it has everything. Just type in what you want to find.

Er, excuse me ?

This is Python, calling a spade a spade

2008-08-22T12:11:00.003+06:00

Python is a high level programming language, but what does this term mean ? What does it mean for a language to be high level or low level ? Can you compare height levels of different languages ?

The meaning for the term is nebulous and there is no single or final definition. Here is one approach - the more effectively the language allows you to handle things, the higher level it is. And by things I'm not meaning just objects as in classes instances. Things, you know, everything, even if I occasionally call them objects.

Enter the notion of first-class objects. Put simply, something is called first-class object in a programming language if it can be treated just like an instance of primitive type, such as int. For example, when you declare a variable (which is a valuable feature already, to be able to declare a variable of that kind)

int i;

you then can do all sorts of things with it, such as passing it as a parameter:

foo(i);

return it as a result of function:

return i;

and do other things, depending on the language. The point is that first-class objects can be handled more effectively and provide additional flexibility. Thus, the more objects in a language are first-class, the higher level that language is.

In Python pretty much everything is first-class. I won't be digging into language reference to find whether or not it is formally true, but in practice it is just like that. It is partly because Python is an dynamically typed language with referential variables semantics - as soon as something exists, you should be able to get a reference to it, and then, once you have a reference, you pass it around as a primitive, not caring about the nature of the object it points to. The language itself does not care what kind of an object is being referenced by the variable you pass. It is only when it comes to real work, such as access to the object's methods, it may turn out to be incompatible with the operation you throw at it. Such just-in-time type compatibility is a very old idea and is called "protocol compatibility" in Python.

Why is it good ? Because I can call a spade a spade. If I need to pass a class as a parameter, what a heck, I can do it:

def create(c, i):
   return c(i)

create(int, 0)

See ? Generic programming right there.

Or, why wouldn't I be able to pass in a method ?

def apply(f, x):
   return f(x)

def mul_by_2(x):
   return x * 2

print(apply(mul_by_2, 1)) # prints 2

Uhm, was it functional programming ?

One other curious and extremely useful first-class thing, which you wouldn't find in many other languages is the call arguments. Remember, I have said that before, there is no declarations in Python. Compatibility of a called function with the actually supplied arguments is checked just-in-time, just as anything else:

def foo(a):
   ...
foo(1, 2) # this throws at runtime

But nothing stops you from writing a function which accepts any arguments:

def apply(f, *args):
    return [f(arg) for arg in args]

apply(mul_by_2, 1)
apply(mul_by_2, 1, 2)
...

And the point is - inside the apply function args is a variable that references a tuple of the actually passed arguments:

def apply(*args):
    print(args)

apply(1, 2, 3) # prints (1, 2, 3)

there may be just a little stretch about calling args a first-class object being "arguments to the call", but practically it is just that. Imagine the flexibility of things you can do with it.

Anyway, in conclusion I will demonstrate another situation where calling a spade a spade is good. A state machine. An object with a state, and a set of state transition rules. What would it typically be ?

class C:

    def __init__(self):
        self._state = "A"

    def _switch(self, to):
        self._state = to

    def _state_A(self):
        print("A->B")
        self._switch("B")

    def _state_B(self):
        print("STOP")
        self._switch(None)

    def simulate(self):
        while self._state is not None:
            if self._state == "A":
                self._state_A()
            elif self._state == "B":
                self._state_B()

C().simulate() # prints A->B STOP

This is a quickly drawn together sample, so please don't be too picky. The problem with it, which I will try to eliminate is this - you have two kinds of way to represent the same thing - the state. What is the reason for aliasing _state_A by "A" and _state_B by "B" ? Oh, the last letter matches, I see... And what's the point in having the state-by-state switch in simulate ? Why don't we just call a spade a spade ?

class C:

    def __init__(self):
        self._state = self._state_A

    def _switch(self, to):
        self._state = to

    def _state_A(self):
        print("A->B")
        self._switch(self._state_B)

    def _state_B(self):
        print("STOP")
        self._switch(None)

    def simulate(self):
        while self._state is not None:
            self._state()

C().simulate()

In this second example, I don't have any arbitrary aliases for state, instead I use for a state its own handler. A method which handles a state is a state here. It simplifies things just a bit - the switch is gone, and it is overall more clean and consistent to my taste.

Well, that's about what I had to say.

Python being a high level language... Other factors, such as wide variety of built-in container types and huge standard library also help Python to be higher level than many other languages, but it's another story.

To be continued...

XML is like plankton in the information ocean

2008-08-20T14:45:00.004+06:00

as huge amounts of it float around to be consumed by everyone.

Bosons, my ass

2008-08-17T21:04:00.004+06:00

Higgs boson, they say, is the reason for wasting gazillions of euros on a high-tech circular tunnel.

So how come we still use portable energy sources that date back to 1800 and are only capable of only giving a 3000 mAh of power ? How come we can't purposely transfer a significant amount of energy wirelessly, through the air, without having to wear radiation-proof costume ? Speaking of which, why radiation protection is still 10m of lead ? Kind of limits space travel you know.

Higgs boson, when you discover it, you know what to do with it.

This is Python, dot operator and the magic "self"

2008-08-05T15:38:00.005+06:00

Although syntactically similar to "regular" imperative programming languages which support OOP and everything, Python offers extra semantical freedom short of being magic.

Consider you have a reference to some object, in

some variable. As soon as it contains a reference to an object (and it always does), you can access that object through the variable, by applying all sorts of operators to it:

x += 1
x["foo"] = "bar"
x(1, 2)
x.foo("bar")

and so on. Whether or not each of those accesses will succeed depends on the target object, but the worst thing that could happen if you mistreat an object is a runtime exception, for example:

x = 1
x()

results in

TypeError: 'int' object is not callable

(Note on samples: they are in Python3k, with Python2x theory is the same, but some of the samples may need to be slightly modified.)

Let's keep on looking. As soon as Python is an OOP-capable language (whatever on Earth that means), it supports classes and methods:

class C:
    def foo(self, x):
        print(x)

and allows overriding reaction to some of the operators, for example the following pieces of code have similar meaning:

class C:                      class C
    def __call__(self):       {
        pass            -vs-  public:
                                  void operator()(void) {}
                              };

and it might seem that there is no difference except for Python way of having a fancy double underscore method for anything advanced, but in fact Python offers more.

Python allows overriding of "dot" operator. For example, the following class (despite being a little unclean) appears to support just any method you throw at it:

class C:
    def __getattr__(self, name):
        def any_method(*args, **kwargs):
            print(name, args, kwargs)
        return any_method
    def i_exist(self):
        print("i would not budge")
c = C()
c.ping()
c.add(1, 2)
c.lookup([1, 2], key = 1)
c.i_exist()

prints out

ping () {}
add (1, 2) {}
lookup ([1, 2],) {'key': 1}
i would not budge

The magic method is apparently __getattr__, it is invoked when you apply dot operator to a class instance and it does not have such named attribute by itself, note how the i_exist method stepped up despite of having __getattr__ overriden.

x.foo
 ^---- __getattr__ is invoked when the dot is crossed

So what does it mean ? It means that you can override anything, including the dot operator, something not possible in static-typed compiled languages, and this feature makes it really simple to hide all sorts of advanced behavior behind a simple method access. For example, consider XMLRPC client in Python:

from xmlrpc.client import ServerProxy
p = ServerProxy("http://1.2.3.4:5678")
p.AddNumbers(1, 2, 3)

and see how straightforward the access to a network service with procedural interface is. ServerProxy class simply intercepts the method access and turns it into a network call. This is done transparently at runtime with no need to recompile any stub or anything - you can access any target service method without any preparation. Compare this to an XMLRPC client library of your choice.

Now take a look at the following fictional line:

foo.bar["biz"]("baz").keep.on("going")

Can you see now that every delimiter (except for literal string quoute) can be intercepted and have its behavior modified ? Given this, I can (and almost universally do) apply aesthetic thinking - how would I like my code to look ? One of the Python principles is to have code (pleasantly) readable. In each case, for each relation between program modules (whatever that means) I can have it

like["this"] -OR-
like("this") -OR-
like_this -OR-
like + "this" -OR-
like.this

and so on. Depending on the situation I can pick up whatever option that makes the code more clear. And guess what ? Overriding the dot is sometimes useful.

Anyhow, this is only half of the story.

The other half is told from the other side of the dot. See, __getattr__ notifies an instance that one of its methods is about to be accessed and allows for it to override. But Python also allows for the accessed member to be notified whenever it is being accessed as a member of some other instance. Sounds weird ? Take a look at this:

class Member:
    def __get__(self, instance, owner):
        print("I'm a member of {0}".format(instance))
        return self

class C:
    x = Member()

c = C()
c.x

prints out

I'm a member of <__main__.C object at ...>

See ? The Member instance being a member of some other class is notified whenever it is accessed. Where can it be useful you may ask ? Oh, it is the key to the magic "self" in Python.

Consider the following most simple piece of code:

class C:
    def foo(self):
        print(self)

Have you ever thought what "self" is ? I mean - it obviously is an argument containing a reference to the instance being called, but where did it come from ? It doesn't even have to be called "self", it is just a convention, the following will work just as well:

class C:
    def foo(magic):
        print(magic)

And so it turns out that somehow at the moment of the invocation the first argument of every method points to the containing instance. How is it done ?

What happens when you do

c = C()
c.foo()

anyhow ? At first sight, access to c.foo should return a reference to a method - something related to C and irrelevant to c. But it appears that the following two accesses to foo

c1 = C()
c1.foo
c2 = C()
c2.foo

fetch different things - c1.foo returns a method with its first argument set to c1 and c2.foo - to c2. How could that happen ? The key here is that you access a method (which is a member of a class) through a class instance. The class itself contains its methods in a half-cooked "unbound" state, they don't have any "self":

class C:
    def foo(self):
        pass
print(C.foo)
print(C().foo)

prints out

<function foo at ...>
<bound method C.foo of <__main__.C object at ...>>

See ? When fetched directly from a class, a method is nothing but a regular function, it is not "bound" to anything. You can even call it, but you will have to provide its first argument "self" by yourself as you see fit:

class C:
    def foo(self):
        print(self)
C.foo("123")

prints out

But as soon as you instantiate and fetch the same method through an instance, the magic __get__ method comes into play and allows the returned reference to be "bound" to the actual instance. Something like this:

class Method:
    def __init__(self, target):
        self._target = target
    def __get__(self, instance, owner):
        self._self = instance # <<<< binding ahoy !
        return self
    def __call__(self, *args, **kwargs):
        return self._target(self._self, *args, **kwargs)

class C:
    foo = Method(lambda self, *args, **kwargs:
                 print(self, args, kwargs))
c = C()
print(c)
c.foo(1, 2, foo = "bar")

prints out

<__main__.C object at 0x00ADA0D0>
<__main__.C object at 0x00ADA0D0> (1, 2) {'foo': 'bar'}

And so I could demonstrate a reimplementation of a major language feature in a few lines. May be not apparently useful most of the time, such experience certainly makes you understand the language better.

One more thing, have I told you Python was cool ? :)

To be continued...

This is Python, variable name lookup

2008-07-15T13:19:00.009+06:00

As I already noted there is no declarations in Python. In general, there is no way to tell in advance what an arbitrary piece of code means, whether it is semantically correct and whether it can be successfully executed. All you have is a syntactically correct code fragment, but the meaning for any symbol is undetermined until the code is finally executed. For example,

foo = bar

is syntactically correct, but you cannot tell whether variable bar is defined at that point or what kind of an object it refers to. What behavior do you have in the above simplest assignment ? It is that if variable bar is defined, a new local variable foo will reference the same object as bar. Something like this:

current_namespace["foo"] = reference_by_name("bar")

This may be a trivial example, except for the behavior of the fictional reference_by_name function. Where does the language look up for a variable ? Like in the other languages that support procedural programming, Python procedures are natural namespace compartments. For example:

def foo(a): # begins foo's local namespace
  b = 1     # modifies foo's namespace
print(a)    # fails because a is invisible here
print(b)    # same

Each procedure's individual namespace is in Python terms called "local namespace". Namespaces of nested procedures nest along with their frames, therefore a name inside of inner procedure may refer to the variable defined in an outer:

def foo():
  b = 1
  def bar():
    print(b) # prints 1
  bar()

On the other hand, presence or absence of a name in a namespace is determined dynamically, at the moment of access, unlike static lexical scoping, which welcomes all sorts of awkward ambiguities like

def foo():
  b = 1
  del b      # would have deleted b from foo's namespace,
  def bar(): # but could not be done, because this nested
    print(b) # reference to b would hang (ouch !)
  bar()

and

def foo():                    def foo():            
  def bar():                    def bar():          
    print(b) # prints 1  -VS-     print(b) # fails because b is
  b = 1                         bar()      # only almost there
  bar()                         b = 1

Unless you want to maintain such ugly code, you should minimize using foreign variables in nested scopes, resorting to argument passing instead. Procedure arguments automatically become part of its local namespace and all locally accessed variables thus explicitly become local:

def foo():
  b = 2
  def bar(b): # explicitly local, no possible ambiguity
    print b   # prints 1
  bar(1)

Nevertheless, it is convenient to visualize the name resolution as scanning chain of nested scopes upwards:

module.py:
5) is b here ?
def foo():
  4) is b here ?
  def bar():
    3) is b here ?
    def biz():
        2) is b here ?
        def baz():
          1) is b here ?
          a = b

Note that in step 5 the containing module becomes an implicit embracing namespace which is the last chance to find the name. In Python this module namespace is called "global namespace". Finally, in addition to local and global namespaces, there is a "built-in namespace" which contains the language primitives that are not explicitly defined anywhere.

Therefore, even the simplest access to a variable is a lookup in three namespaces - local, global and built-in in that order.

To be continued...

Block-drawing characters in Firefox. WTF ?

2008-06-30T16:08:00.004+06:00

Unicode defines a family of characters shaped like boxes of increasing height. Presumably useful for drawing diagrams in text. Something like this

x
xx
xxx

only fancier. The exact 8 characters in discussion have code points 0x2581-0x2588, and range from 1/8th to 8/8ths, i.e. full block. Here is a sample:

▁▂▃▄▅▆▇█

Now, correct me if I'm wrong, but those characters are only useful as soon as they are aligned with each other. You can't draw a diagram if one box is slightly offset - it turns out ugly. And so, can anyone tell why Firefox 2 renders the 4/8ths (half-block, code point 0x2584) and the 8/8ths (full block, code point 0x2588) shifted down a little ? Here, have a look:

This glitch makes it practically useless. WTF ?

Different ways to understand things in software engineering

2008-06-22T11:27:00.004+06:00

The proficiency of a software developer is determined not only by which technologies he used or for how long, but more importantly by how exactly he understood and interpreted the principles behind them. Because the basic principles of software engineering are so numerous and often not specified formally, the view of the actual developer means everything.

In the course of work, a developer adapts his understanding to the problems he is working at, this is somewhat similar to how shapes of key and lock match. For this reason two people may be using the same technology for the same amount of years but be totally unable to understand each other to a point of engaging religious wars over the simplest points.

Now I understand why whenever I have a chance to interview a job applicant, I ask rather unspecific questions even of philosophical kind - to see not what he knows, but how he actually understands it and whether his understanding matches mine. Because if it doesn't we'd probably have hard times working together.

The difficult part here is trying to keep your knowledge deep and broad at the same time, because both the details and the perspective are required to understand.

The set of good programmers is still very small, a great joke by David Parnas

2008-06-17T13:32:00.003+06:00

A reviewer explained his rejection of my best-known paper on the subject by writing, "Obviously Parnas does not know what he is talking about because nobody does it that way". Only a decade later, however, a textbook stated, "Parnas only wrote down what all good programmers did anyway". A logician would conclude that the set of good programmers was empty; that set is still very small.

-- David L. Parnas

This is Python, everything is executable

2008-06-17T11:23:00.009+06:00

Dynamically typed language is by definition the one where variables don't have type, but the actual values do. This is by all means true in Python where the following code works fine

x = 10
x = "ten"
print(x) # prints ten

but limiting dynamism to untyped variables only would be missing the point.

Like with many "scripting" languages a Python program is started by passing the name of its main module to the "interpreter", such as

c:> python main.py

The transition of Python source code to an actually executed program begins with loading and parsing the module file. This step succeeds as soon as the module does not contain any syntax errors fatal for the parser. Successful parsing only guarantees that the module is not totally broken - a weak guarantee, only useful for checking for unbalanced parentheses and such.

What happens next is magic - the parsed module file is executed as though it was just a chunk of a source code. Wait a minute ! It is a chunk of a source code ! Anyway, execution of every module at its first import is the major part of Python program run. This process is identical no matter if the module being loaded is the program's main module or some other module explicitly imported by demand.

I have arrived to Python from C++, it took me a long time to change the perspective and the change is this - in Python you should look at everything as though it is an executable statement, because it really is. To illustrate this principle, consider definitions vs. declarations.

In statically typed languages, declarations exist for the sake of separate compilation - for the compiler to be able to tell whether one part of code is compatible with another without having to dig through the entire program. In Python, which is a dynamic language, there is no compilation stage, therefore declarations are useless, and what's left only looks like definitions.

For example, where in C++ you have two files (if you do it properly)

// foo.h                  // foo.cpp                         
class Foo                 int Foo::GetX(void) const
{                         {
private:                    return x;
  int x;                  }
public:
  int GetX(void) const;
};

the .h file is a declaration - your promise to the compiler that you will provide the matching implementation and the .cpp file is that promise fulfilled. In Python there is no compiler so you don't have to feel obliged. Identical code in Python would be

class Foo:
  def get_x(self):
    return self.x

What you see in this Python code is neither a declaration nor a definition. It is a piece of executable code, which, when executed, introduces a new class to the containing module's namespace. Rewritten to its actual effect in pseudocode it would look like this:

class Foo:       temp1 = new class()

def get_x(self): temp2 = new method()
return self.x    temp2.__code__ = return self.x
                 temp1["get_x"] = temp2

                 module["Foo"] = temp1

What you just saw was an illustration that a Python class definition is an executable statement, just like anything else and it executes once when the module is first imported. For example, it is possible to do something like C++'s conditional compilation:

class Foo:
  if os.platform == "win32":
    def do_it(self): # windows way
      ...
  else:
    def do_it(self): # unix way
      ...

The effect of the above code is that when the module is imported, the compiled version of class Foo will contain method do_it matching the current environment. It is not the same as the straightforward approach, where the check would have been performed upon each call to do_it:

class Foo:
  def do_it(self):
    if os.platform == "win32": # windows way
       ...
    else: # unix way
      ...

In a similar vein, your class definition could fail to execute:

class Foo:
  1 / 0 # this throws at import time

and the module will fail to import, throwing an exception to the caller.

Now it should not surprise you the least that when one module imports the other it is again not a declaration. When module foo does

import bar

the described process repeats for module bar, unless it has already been imported, in which case the import statement does nothing (from the discussed point of view). Similarly, you can import modules as you need them at runtime:

if need_time:
  import time
  print time.time()

Python therefore does not have any declarative semantics, only executional - ask yourself - what does it do when executed ?

To be continued...

This is Python, language installation and program structure

2008-06-16T09:49:00.004+06:00

Installing Python is easy. If you use Windows, you have no choice at all - run setup.exe and you are done. Under Unix, Python can be preinstalled or you can install it manually. I always prefer to install from source on a clean machine, but if you have it preinstalled, you should be fine too.

Python installation is fully self-contained, and can be migrated to a different machine by copying all the files (or just the necessary ones) from c:\pythonXX or /usr/local/whatever/ to the destination. Multiple versions of Python coexist peacefully in different directories (although you should copy them around manually, because installation process registers stuff in the Windows registry and do other such things of global effect).

Python installation essentially contains the language parser+compiler and a huge and poorly structured standard library. The compiler itself along with a minimum set of libraries lives in pythonXX.dll or pythonXX.so.1, and the executable python.exe or bin/python is nothing but a simplest driver program of the (read line, execute, repeat) sort. The standard library lives in c:\pythonXX\lib + DLLs or /usr/local/lib/pythonXX/ and is just a heap of assorted utilities.

Python can be and is easily embedded into another application. It is a DLL, remember ? You take the DLL, zip the standard library and there you have it in two files - an embedded Python. In your application you create an instance of a compiler at runtime and start feeding it with stuff, that's all. Python can also be embedded into a diskless machine, it works just fine in a very restricted environment (such as the high security FreeBSD CD that I have here).

Python program consists of a set of separate modules, each module is a separate .py file containing some source code. The program is therefore available to the language in source, but Python nevertheless is not an interpreter. As each module is about to be used at runtime, it is loaded, parsed and compiled to an intermediate byte code for some virtual machine. The compiled byte code is saved alongside the original source file in an identically named .pyc file for future reuse. The outcome is the same as with Java or C# or any other language that translates source into byte code, and the difference is that in Python there is no separate compilation stage as such - the translation is performed at runtime and is in fact an important part of program execution.

To be continued...

This is Python, intro

2008-06-11T10:39:00.004+06:00

Writing in Python for a few years now, I still get a kick of it. Wonderful and very powerful language, if used in the right way (aren't they all like that ?). No matter if code base is in megabytes, I still occasionally sit back in silence, admiring the beauty of a little code fragment or the way an idea is expressed in code.

If there is one single snippet of Python code to introduce its power of simplicity, it would be swapping of two variables. When I found it long ago in Python cookbook, it hit me like thunder. Never was my understanding of Python the same as before.

Here goes. To swap two variables in Python you need to


a, b = b, a

This utilizes Python feature called automatic tuple packing/unpacking. What's actually going on is more like


a, b <<< unpacked <<< (b, a) <<< packed <<<  b, a

where (b, a) is a Python notion for an immutable sequence called tuple.

To be continued...

Slow pace of software development

2008-06-06T18:59:00.005+06:00

If you need it fast, make sure it *looks* good, because it won't be. Pay more attention to high quality advertisement than to high quality development, but note that once you start selling promises, you will unlikely deliver a product.

Delphi is a very useful tool (with case study)

2008-05-22T13:44:00.008+06:00

I've been working with Borland Delphi since 1996, when 2.0 was shiny new. Before that time I've been more or less formally taught Pascal and Delphi seemed a very welcome tool. And it still does, after 12 years. Strange thing is, whenever I mention using Delphi now, eyes are often rolled and "but it's not .NET" sounds. Know what ? I don't care about .NET. Pragmatically speaking, what matters is the product.

The very reason for this post is that the most recent GUI application that I've written was officially released a few days ago, and it is in Delphi. What's special about this particular application is that it comes really close to the ever sought perfect "one button" program. Ease of use was the major goal and what our group managed to produce you can see for yourself here:

Internetbank installer (in Russian!)

To give you the idea what it is for, it is a program to initiate access to remote banking web site for the bank of Severnaya kazna, which I happen to be working for right now. A client clicks the button, browser pops up and the client is brought to http://www.internetbank.ru/ from where he can manage his cards, accounts, transfers, payments etc.

Looks simple, except that before you can enter, you need to have started a cryptographically protected HTTP proxy server and provided with it a key, which is used for putting legally valid digital signatures under all your payments.

And that encryption key needs to be have been generated before. And the key expires and needs to be extended remotely. And CA certificates (the bank runs its own certification authority) also expire and need to be replaced remotely. And the user may have multiple keys to choose from.

And there might be problems with Internet access and the user needs a clear and concise yet technical enough diagnostics.

And it should run without installation from a removable drive with very restricted rights. And it should not require any runtime installation, such as .NET, but it must run on Windows 98.

And it must be remotely updatable.

And it must be simple to be usable by the most average Joe.

The result is the program works and really has just one button, which is also big and pretty. There is no confusion as to what to do next, because most of the time the user is simply not given any choice at all, clicking the only button starts the only process (of entering the protected web-site).

In a rare case when a user has to initiate some other process, such as generating a new key, the reasons for that are clearly explained right there on the program's face. Hints and careful label wording help at least those users who can and actually are reading.

Looking back I realize that the only reason why such a simple solution appeared is three years of refining the procedure. During last three years our customers had to use different software, clumsy and inconvenient (despite being also written by me). But in three years we learned all the moves and most of the problems. And then a strategic decision was made and we trashed the old software and rolled out the new one. And I like what we did.

Anyhow, returning to Delphi.

I'm using it for what it does best - pretty GUI with access to OS services or 3rd party libraries. Skin support (as well as good taste) makes programs really good looking. Being pretty is a major advantage for end-user software. Positive emotional reaction would actually help the user to deal with it.

Aside from that, Delphi programs don't require any *cough* runtime platform but run on Windows 98. And Delphi produces reasonably lightweight executables. And you don't need administrative rights to run them. And I like it.

Great stuff.

Just in: the benefits of compiling into platform-independent byte code

2008-05-18T09:11:00.003+06:00

After the existence of Pascal became known (in 1974), several people asked us for assistance in implementing Pascal on various other machines [...] Thereupon we decided to provide a compiler version that would generate code for a machine of our own design. This code later became known as P-code. [...] Had we possessed the wisdom to foresee the dimension of this movement, we would have put more effort and care into designing and documenting P-code.

-- Niklaus Wirth, 1985

Just in: the only way to reliability is through simplicity and that can't be bought

2008-05-13T19:43:00.004+06:00

Almost anything in software can be implemented, sold, and even used given enough determination. There is nothing a mere scientist can say that will stand against the flood of a hundred million dollars. But there is one quality that cannot be purchased in this way - and that is reliability. The price of reliability is the pursuit of the utmost simplicity. It is a price which the very rich find most hard to pay.

-- C.A.R. Hoare, 1981

О самописных программах

2008-05-13T09:38:00.003+06:00

There is a Russian word "самописный" [səmopisni], literally meaning - "written by oneself". The word has a strong disparaging tone in it and is typically used to describe the low quality software written by one's colleagues (or even oneself), as opposed to the high quality truly industrial commercial software written by people somewhere else, even better if purchased for a huge chunk of money.

The mere existence of such word and the attitude of programmers who use it in derogative way is totally beyond me. The programmers who don't appreciate and value the work they do, how good a software they really write ? Moreover, how can they evaluate the quality of somebody else's work if they despise their own ? Besides, the absurdity of the situation is in that all the programs are really of this sort as someone has ultimately written them.

A concluding quote from a nobleman:

Clearly IBM and MIT must be possessed of some secret of successful software design and implementation whose nature I could not even begin to guess at. It was only later that they realized they could not either.

-- C.A.R. Hoare, 1981

Just in: choose your language carefully

2008-05-11T23:32:00.003+06:00

I have regarded it as the highest goal of programming language design to enable good ideas to be elegantly expressed.

-- C.A.R. Hoare, 1981

Language designers also have an obligation to provide languages that encourage good style, since we all know that style is strongly influenced by the language in which it is expressed.

-- Donald Knuth, 1974

I have the feeling that one of the most important aspects of any computing tool is its influence on the thinking habits of those who try to use it.

-- E.W.Dijkstra, 1972

Just in: permanently low quality of software

2008-05-11T23:20:00.003+06:00

I feel that all too often we have been satisfied with such a low level of quality that we have done ourselves harm in the process. We seem not to be able to use the machine, which we all believe is a very powerful tool for manipulating and transforming information, to do our own tasks in this very field.

-- R. W. Hamming, 1969

Just in: debugging is unnecessary

2008-05-06T21:30:00.003+06:00

If you want more effective programmers, you will discover that they should not waste their time debugging - they should not introduce the bugs to start with.

-- E.W.Dijkstra, 1972

Just in: reusability as a central problem

2008-05-05T21:32:00.004+06:00

Perhaps the central problem we face in all of computer science is how we are to get to the situation where we build on top of the work of others rather than redoing so much of it in a trivially different way.

-- R. W. Hamming, 1969

Just in: no adequate programming teaching

2008-05-03T10:45:00.004+06:00

To parody our current methods of teaching programming, we give beginners a grammar and a dictionary and tell them that they are now great writers. [...] As a result, few programmers write in flowing poetry; most write in halting prose.

-- R. W. Hamming, 1969

Just in: domain-specific languages

2008-05-02T12:00:00.002+06:00

We have, in fact, two languages, one inside the other; an outer language that is concerned with the flow of control, and an inner language which operates on data. There might be a case for having a standard outer language - or a small number to choose from - and a number of inner languages which could be, as it were, plugged in. If necessary, in order to meet special circumstances, a new inner language could be constructed; when plugged in, it would benefit from the power provided by the outer language in the matter of organizing the flow of control.

-- Maurice V. Wilkes, 1967

Just in: standardization

2008-05-01T21:55:00.004+06:00

I am sorry when I hear well-meaning people suggest that the time has come to standardize on one or two languages. We need temporary standards, it is true, to guide us on our way, but we must not expect to reach stability for some time yet.

-- Maurice V. Wilkes, 1967

Method signature type checking decorator for Python 3000

2008-04-22T13:01:00.004+06:00

I have just published a Python 3000 decorator for method signature type checking using function annotations. Here:

http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/572161

It is much cleaner that the similar decorator I have previously written for Python 2.x, the used Python 3000 function annotations make it better for the following reasons:

1. The signature-related piece of syntax is right there where it belongs - next to the parameter. Where I used to write

@takes(int, str)
@returns(bool)
def foo(i, s):
    ...

I now write

@typecheck
def foo(i: int, s: str) -> bool:
    ...

2. I don't have to add checking to all the parameters simply because there was no way to skip one. Where it was

class Foo(object):
    @takes("Foo", str)
    def foo(self, s):
        ...

it is now

class Foo:
    @typecheck
    def foo(self, s: str):
        ...

3. It plays nicely with the default values. This one has no equivalent in 2.x version, but it is nice to have:

@typecheck
def foo(x: int = 10): # 10 is also checked
    ...

@typecheck
def foo(*, k: optional(str) = None):
    ...

Other than that, it is just a nice usable piece of code, extensible too. Here is a few more examples:

@typecheck
def foo(x: with_attr("write", "flush")):
    ...

@typecheck
def foo(*, k: by_regex("^[0-9]+$")):
    ...

@typecheck
def swap_tuple(x: (int, float)) -> (float, int):
    ...

@typecheck
def swap_list(x: [int, float]) -> [float, int]:
    ...

@typecheck
def extract(k: list_of(by_regex("^[a-z]+$")),
            d: dict_of(str, int)) -> list_of(int):
    ...

Approaching Python 3000: no more automatic tuple parameter unpacking

2008-04-21T09:30:00.009+06:00

Just keep in mind that

lambda (x, y): x + y

is no longer possible in Python 3000. You are supposed to write

lambda x_y: x_y[0] + x_y[1]

instead. This also applies to functions, not just to lambdas and the idea belongs to PEP 3113 which forbids automatic tuple unpacking in function parameters. Ugly and inconvenient if you ask me, but there apparently was somebody who kept shooting himself in the leg.

I think that automatic unpacking was rather useful (if used sparingly), especially when you had to do something like

map(lambda (i, (a, b)): i * (a + b),
    enumerate(zip(aa, bb))

which is now what ?

map(lambda i_a_b: i_a_b[0] * (i_a_b[1][0] + i_a_b[1][1]),
    enumerate(zip(aa, bb))

Eeew...

Approaching Python 3000: function annotations

2008-03-30T11:24:00.004+06:00

Python 3000 introduces function annotations, regulated by PEP-3107. This one is of particular interest to me as I've previously written one of the method signature checking decorators which this PEP is supposed to replace.

PEP-3107's has two major points:

1. Annotations can be anything. Any Python expression can be attached to a function argument or result value. For example, it is possible to write

def max(a: int, b: int) -> int:
def max(a: "first", b: "second") -> "maximum":
def foo(x: { ("no", "reason"): lambda x: x**2 }):

2. Annotations have no semantics and are not enforced, they are purely syntactical. For example, it's ok to write

def max(a: int, b: int) -> int:
...
max("foo", "bar") # nothing happens

The interpretation of annotations is left to 3rd party libraries. The language thus offers an unprecedented semantical freedom to the developers, but let's see what are the implications.

One problem is that you will have to choose your one true annotations. Specific interpretation of function annotations depends upon external module, library or application, and you acknowledge this dependence explicitly by either modifying your code or having it processed by external application.

For example, you could have chosen to use function annotations for method signature type checking, using typecheck decorator (resembling my type checking decorators)

@typecheck
def foo(a: int) -> int:

but once you have chosen the @typecheck implementation you have to stick with it and treat all your function annotations as type checks. Stacking multiple annotations is technically possible, but practically it is not, because the standard does not specify how multiple annotations should be multiplexed. Consider that you have

@typecheck
def foo(i: int):

in place and want to add a docstring kind of annotation to foo's first argument:

def foo(i: "comment"):

Should it be

def foo(i: (int, "comment")):

def foo(i: {"typecheck": int, "docstring": "comment"}):

?

No matter which way you choose, both typecheck and docstring must be prepared to extract their annotations from the actually encountered multiplexed construct. This means that two independent implementations must understand the same multiplexing format. Since such multiplexing is not standardized, it is impossible.

One seemingly reasonable way of such multiplexing could have been an iterable with instances of classes descended from some base class (Annotation?) for example

def foo(i: (typecheck(int), docstring("comment"))):

This example may be correct but it perfectly illustrates how having multiple semantically different annotations seriously hamper the visual quality and readability of the code.

Which opens a final question of whether the gains of the only particular annotations that you choose outweigh the loss of syntactical brevity.

Approaching Python 3000: string formatting

2008-03-10T18:37:00.005+05:00

One of the changes in Python 3000 applies to string formatting. PEP-3101 is the regulating document for the change. It basically says that the regular % operator is too limited and what we need is a powerful domain-specific language for string formatting.

To be frank, I never felt limited with what % had to offer, but there is no point in criticizing what's about to become standard. Let's see what's new:

1. What used to be a binary operator is now a str's method:

"{0}, {1}".format("A", 10) == "A, 10"
"{n} = {v}".format(n = "N", v = "V") == "N = V"

2. Formatting and alignment is applied in the same manner as before:

"{0:03d}".format(10) == "010"
"<{S:>5s}>".format(S = "foo") == "<  foo>"
"<{S:<5s}>".format(S = "foo") == "<foo  >"

3. Format can insert not just parameter values, but also their items and/or attributes:

d = dict(foo = 1, bar = "a")
"{0[foo]}, {0[bar]}".format(d) == "1, a"
"{0.__class__.__name__}".format(d) == "dict"

4. Recursive substitution is allowed to a degree, for example this works:

d = dict(value = 10, format = "03d")
"{0[value]:{0[format]}}".format(d) == "010"

but this doesn't:

d = dict(data = {"a": "A", "b": "B"}, key = "a")
"{0[data][{0[key]}]}".format(d)

5. Classes can control their own formatting:

class Foo():
def __format__(self, format):
    from re import match
    assert match("[0-9]+s", format)
    return "x" * int(format[:-1])
foo = Foo()

"{0:3s}".format(foo) == "xxx"
"{0:10s}".format(foo) == "xxxxxxxxxx"

I agree, the new way of string formatting (1 and 2) is cleaner and more straightforward. Substituting items and attributes (3) could be useful sometimes. Custom formatting (5) is Pythonic all right, but hardly practically useful, except when building a framework, a class framework perhaps.

What I don't buy is the attempt to make a statement from what is supposed to be an expression (4). If it is inconsistent, difficult to read and not apparently useful, it should not be there.

A programming language made of smileys

2008-03-04T12:04:00.006+05:00

Just imagine the possibilities of a language whereby programs are constructed from smileys:

Easy to write, all language objects can be arranged on a toolbar. No more typos !
Easy to read. You get an immediate emotional response by simply looking at the code.
Appealing to a programmer of any nationality. No internationalization required !
Easy to extend by adding smileys with hammers, flowers or database connectors.
Fun to work with !

The great effects of little imperfections

2008-02-03T14:36:00.000+05:00

Observing a chain of bubbles raising through a column of water I found it simply amazing how the same little fluctuations in water

that actually made the bubbles appear in the first place also make sure that they go straight up and don't deviate as they raise.

Re: Software Engineering Programs Are Not Computer Science Programs

2008-01-30T11:35:00.000+05:00

Written by David Lorge Parnas the article under that title was published in "IEEE Software", Nov/Dec 1999, and essentially says that computer science and software engineering need to be separated in the same way as theoretical physics is separated from its related engineering fields. For the sake of both. As far as the education goes at least.

He also advocates the mandatory accreditation of software engineering programs and points out the problems to be encountered. Among the problems mentioned by the author are the lack of knowledge how to teach and experienced staff.

Frankly, the article was a little tedious to me, biased by the magazine specifics perhaps. But just like the other works of this great man that I had a chance to read, it is truthful and inspirational. Although in the case of this article, the inspiration has driven me in a slightly unexpected direction.

And so I would like to criticize the article on the grounds that the analogy between physics/regular engineering and computer science/software engineering does not hold.

First, there is a historical difference. Between physics and its engineering fields, it all began with practice and experiment. The extreme case would be construction - people have been prototyping since probably 50 thousand years ago. Two thousand years ago selected craftsmen have already mastered wood, stone and iron construction. There was neither science nor engineering at that moment, all they had was observation and experience.

I am not an expert in history of science, but it seems plausible that same pattern repeated most of the time - experiments came first and the theory followed. To be sure, physics as a science is far ahead now setting up experiments that only a few understand, but at least at early stages practical considerations have prevailed.

Exactly opposite is true for computer science. What began as pure mathematical theory in 1930s couldn't even be supported by experiment until a decade later when some sort of electrical apparatus has been constructed. In fact, being a branch of mathematics, computer science didn't have to be supported by experiments in the first place.

The "mathematical" engineering therefore was not something that anyone practically required. All they needed was to speed up the calculations, and I seriosly doubt that anyone could see the consequences. As the story has it, at one time IBM predicted the computer world market to be in tens of installations. If it wasn't for semiconductors, software engineering wouldn't even be here today, but computer science would.

Over time, software engineering became an awkward crossover between mathematics and psychology, where people try to project mathematical abstractions onto real world. Remember how Knuth said: "I have only proved it correct, not tried it." The matter dealt with in software engineering is thus something that should work because it is theoretically perfect but doesn't work because we are not practically perfect.

Second, there is an economical difference. Software is intangible, software production does not respect political borders, it can easily be and is routinely outsourced. What would you say if a team of construction workers could fly from India with its own tools and materials to raise a house overnight ? Plus they would charge less and still get the job done with satisfactory quality. And they would not need to be certified. Similarly, if a doctor or a lawyer could consult over the Internet from a different country, and his services were just as good, wouldn't that nullify certification efforts ?

Besides, you can't strictly control telecommutable industry, you try to lock it down by regulations and it goes underground. And the last thing we need is the black software market in addition to a pirated software market. Besides, such regulatory inhibition of software engineering would hamper the scientific progress and thus have exactly the opposite effect to the desired.

You may try to enforce mandatory certification of products instead, but this brings in a totally different perspective and requires a definitive procedure of software quality assessment - something at least improbable at this moment.

Third, there is a natural difference. There is no laws of nature in software engineering.

Try hard as you may, you cannot build a house which levitates above the ground. Because physics provides its engineers with absolute laws - such as energy conservation law, thermodynamics laws or Newton laws. They all may be a reflection of some deeper principles, but in practice it is sufficient for an engineer to know that you are limited in energy and can't fight gravity. And this is not because a scientist said so, but because you simply can't.

Not the case with software. The world in which software lives is only restricted with hardware architecture. Von Neuman is literally the god of software and computer scientists are his prophets. But what absolute laws do they give to their poor engineers ?

None.

The hardware has its restrictions, that's true, but it is all in capacity. It is physics that limits the hardware, the computer science does not impose any restrictions above that. It is as though it was possible to build a house with the only restriction in mind - that its size should not exceed that of a planet. You can even start building from the roof, and it doesn't have to touch the ground when it's done. It's all imaginary.

The lack of unbreakable laws leaves all the arguments about how software has to be build to a degree open-ended. But then, how could you certify an industry in which there is still no consensus on how to do the simplest thing ?

To conclude, I believe that computer science and software engineering are indeed different, so different in fact, they can be treated as totally unrelated. But the relationship between them is not the same as with physics and engineering, and it would be wrong to approach it with established (educational) practice.

There are unnecessary details

2008-01-27T22:03:00.000+05:00

but there are no unimportant details.

What is the word LOVE associated with these days ?

2008-01-20T18:47:00.001+05:00

SPAM !!!

All we have is less good programmers

2008-01-19T21:16:00.000+05:00

The more I learn about programming, the more I want to ask: "why haven't I been taught this before ?". I mean - I graduated from a university, majored in "computer mathematics" or so it said, but I really got nothing useful from professional point of view. A few theoretical courses, such as graph theory is all. As the matter of fact, everything I know about programming I've learned from books and hard work.

The sad truth is that each generation takes a fresh start. I recall how confident I was in having known everything. May be it happens all the time, but programming is special because there is still no notion of software quality. It is surprisingly difficult to convince a beginning programmer that his program is bad. Because you simply have no reliable judgement basis except for your own expert opinion, but then what would you know ?

But then, there is no knowledge transfer and the entire industry is doomed to go around in circles, reinventing the same things every ten years or so, under different names.

I do realize that the software industry didn't get any better in the past decades, it even might have gotten worse by all accounts. The only reason why we could have possibly gotten more good programmers is because there simply appeared more just any programmers, because anyone could perform as one. Therefore it is statistically possible that the upper percentile also got more numerous.

It would also seem a valid guess that it becomes more and more difficult to find good programmers. Because the good ones tend to stick with a company, or a project, or a team, and bad ones may be changing places more often. This makes the problem of creating a strong team more difficult, and your typical team would in general be of lesser quality. As I believe that a strong team is the best thing that could happen to a programming project, this observation leaves even less hope in the future.

You write the program ...

2008-01-16T21:23:00.000+05:00

... and the program writes you.

Karlsson is from Sweden after all

2007-12-28T19:39:00.000+05:00

You know Karlsson who lives on the roof. Everybody knows him.

Being a soviet kid, I've always had this impression that the "true" Karlsson appears in the well known soviet short animated film "Malysh and Karlsson", created in 1968. I've watched it countless times, just like any other soviet kid at the time.

Now I've had a chance to watch a newer 2002 Swedish full length version: "Karlsson pa taket".

What was really surprising is how similar the images of Malysh (the little one, in Russian adaptation) and Karlsson himself were. And not just those two, the entire visual style is strikingly similar. See for yourself.

Russian version:

Swedish version:

Excuse me, but I don't believe that two different teams of animators working at different times in different countries can come up with something so similar. What's going on here ?

Here is what I found.

The first clue is right there in the titles to the Swedish film - it is declared to be based on the work by Ilon Wikland. As it turns out to be, she has illustrated the first edition of the book back in 1955.

The design of characters in Russian version is ultimately attributed to Anatoly Savchenko, although at least here he is said to have been inspired by illustrations to the first Swedish edition of the book. You know, judging by the look of it, I wouldn't even call it inspired, rather based upon.

This is it then, the one and only Karlsson belongs to Ilon Wikland and not to the biggest soviet animation studio Soyuzmultfilm.

What is sad though is that the Russian version doesn't indicate the borrowing. I can see that at the time the Russian film was made it would have been a suicide to refer to a western source in Russian product. Still, it is very sad and obscuring fact.

There are simple things which are hard to grasp.

2007-12-27T10:36:00.000+05:00

This is one of them.

Re: The end of America

2007-11-30T17:56:00.000+05:00

Having listened to Naomi Wolf as she speaks about "10 steps to fascism" here:

http://www.youtube.com/watch?v=RjALf12PAWc

and here is a supporting story in Guardian:

http://www.guardian.co.uk/usa/story/0,,2064157,00.html

I was applying the principles she suggests to current situation in Russia one by one, and surprisingly, they hardly applied:

1. Invoke a terrifying internal and external enemy.

Pass. None of those exist in modern Russia. Chechnen terrorists looked like a mixed ex/internal threat once, but then quickly diminished. No external power is considered a threat. There are "me too" kind of reactions to the America-declared global war of terrorism, such as the absurd requirement to take off one's shoes in the airports, but that looks like a totally random acts of power rather than a iron fist lead.

2. Create a gulag.

Well, I wouldn't know. In a country whose leader has invented gulag, I'm sure as hell there are secret prisons, but then they don't have to be secret, any would do. So, I'd say no, there is no gulag in modern Russia in the meaning of the word Naomi Wolf puts in it.

3. Develop a thug caste.

No such thing. Or, multiple such things, depending on what you mean. There is no single dedicated paramilitary force and none are emerging. There is army, state police, corporate security guards, and all sorts of criminal organisations, I'd presume. They all may apply force pursuing their arbitrary goals, but I don't think they orchestrate. This is not to obscure the fact that the police or army could at any point be given any orders.

4. Set up an internal surveillance system.

There easily could be files on anyone, just as with the mentioned Stasi, but I don't think there is a global network of surveillance and the percentage of informants has hopefully decreased since KGB time.

5. Harass citizens' groups.

Citizen groups ? What citizen groups ? None of active opposition to the power, that's for sure. Groups of political hobbyists and minorities of all sorts may be present, but noone worth mentioning with real power or a threat to power. Therefore there is no reason to persuasively harrass anyone. At any rate, they don't make a show out of it.

6. Engage in arbitrary detention and release.

It's not comforting, but I'd guess, yes, it's just like that. The stories of people being detained, kept in prisons, beaten and tortured appear every now and then. And the purpose could very well be the same - to terrorize and intimidate the entire population, even if in subtle way.

7. Target key individuals.

Check. Except, there is no key individuals. There are occassional public executions of someone who is sort of in opposition, but the truth is - there is no opposition. Oh, and nevermind the reporters murders.

8. Control the press.

Check. I mean, absolutely.

9. Dissent equals treason.

I'd answer this question if anyone could tell me what the today's Russia consent is ? Anything sacred ? Take Americans, they worship their democracy and freedom (ahem, given the steps to fascism title, this sounds awkward), but in Russia - what is the true way ? The way I see it, right now Russia is happily doing nothing, basically selling oil for food.

Actually, it's funny how I can't come up with anything that would sound plausible and treasonous at the same time. Brainwashed with no access to the facts - the most likely cause.

10. Suspend the rule of law.

Well, there is no martial law in Russia right now and I hope not to ever see it. We have laws and justice, right ? Although Russians are traditionally very sceptical to the laws and justice, but you can't argue that codices exist and trials work.

So, it sounded not so bad, right ? But wait until it comes to about minute 41 of the speech, and the answer should have been obvious in the context - the closed society doesn't look like one.

To quote Naomi Wolf:

We have this, like, wrong notion of what a closed society looks like. ... A closed society, even a violent military dictatorship, [looks a lot like] civil society, there are still elections, they are just corrupted. ... There is still a judiciary in a closed society, they are just not free to adjudicate freely. ... There's still academics, they're just watching what they're saying. There is still newspapers, you just know how far to push the enquiry.

Duh ! What can I say, welcome to the closed society. We'll have to see how it works out in America. Anyway, the speech was very thought-provocative.

On software reliability

2007-11-19T15:16:00.000+05:00

Despite the common prejudice and the name, for a system to be reliable is not the same as to have no errors or never crash (noone should ever be promising that). The way I see it, reliabilty is more of a predictability. A system is reliable if it behaves in a predictable fashion - so much that you can rely on that. Even if all it does reliably is crashing.

The systems we build, they don't exist in isolation. Again, despite a popular myth, programmers don't pull things out of a thin air. We base our work on the work of others - hardware, operating systems, servers, frameworks, libraries, compilers, you name it. Reuse is the boon and the bane of the industry. Client-server pairs, interfaces and contracts are not just about OOP, they are everywhere. Any interaction between software components is about grabbing something somebody else has.

I therefore take it for granted, that there are parts of the system not written by us or not belonging to us. This implies that they cannot be controlled in any way (actually, I mostly work on integration middleware, which makes my experience worse). Then this inability to control leads to inability to fix. More often than not, you cannot fix problems in systems you have to depend on.

What do you do when you encounter an unexpected behaviour from somebody else's system you have to depend on ? I do this: first - fetch a cookie for being lucky, second - understand the cause, third - find a workaround.

The point here is - a problem known is not a problem. Because if you know the exact circumstances under which it hits, you can work around. Figuratively, you have to walk the minefield, but every previously found mine can be sidestepped. Therefore,

For a component to be reliable, you must be able to work around any problem encountered, and for a system to be reliable, you need to know all the problems it can cause.

To conclude, here is a few examples of something which is broken and reliable at the same time, because there is a workaround:

Reliable is a broken CPU which always fails upon certain floating point division. The numeric libraries fall back to software emulation when encountering this particular kind of chip.
Reliable is the XML parsing library which crashes when you attempt to set attribute value to something with letter "Ё" in it. I replace "Ё" with "Е", which is a good enough if slightly ambiguous substitute.
Reliable is the compiler which chokes and dies upon too complex a template. I rearrange the angle brackets and it goes on fine.
Reliable is the provider's server which crashes every other Friday at 13:00. I schedule an offline gap at that time and noone complains.
Moreover, reliable is the provider's server which crashes sporadically, but will gracefully handle repeated access attempts. As you might guess, this last example is the warmest to my heart and is one of the bases for Pythomnic - the platform for developing network services I'm developing and using.

Another ridiculous sentence

2007-11-16T09:56:00.000+05:00

"Fighting stress"

The only way to get something done

2007-11-07T11:25:00.000+05:00

... is to start and then don't stop.

Note to self: default parameter values are mutable in Python

2007-10-19T17:59:00.000+06:00

Just hit a somewhat unexpected behaviour in Python code. What would the following code snippet print, what do you say ?

def foo(x = []):
    x.append("bar")
    print x

foo()
foo()

If you think about the def statement as a declaration, the answer is obvious - it should print

[ 'bar' ]
[ 'bar' ]

but in fact it prints

[ 'bar' ]
[ 'bar', 'bar' ]

Why ? Because in Python def is an executable statement, which means that the list of arguments for the method to be created (x) and most importantly their default values ([]) are themselves nothing but arguments to def. Something like this:

foo = def(x, default_x)

and when this gets executed, default_x is bound to something that at that point evaluates to an empty list, but remains mutable. Then, whenever the created foo is executed, the append method modifies the contents of default_x - effectively the "default value" of x.

This sounds strange, but is clearly documented in the language reference, quote from http://docs.python.org/ref/function.html

Default parameter values are evaluated when the function definition is executed. This means that the expression is evaluated once, when the function is defined, and that that same "pre-computed" value is used for each call. This is especially important to understand when a default parameter is a mutable object, such as a list or a dictionary: if the function modifies the object (e.g. by appending an item to a list), the default value is in effect modified.

The spec suggests using None for all the default parameters, but you should have no problem using any immutable objects as well. For example:

def foo(x = None):
    print (x or []) + [ "bar" ]

where x is None, or

def foo(x = ()):
    # oops, no append method
    print x + ( "bar", )

Phew, I've been lucky using None's so far...

Sympathy for Mr. Software

2007-10-08T13:34:00.000+06:00

No software does what the user is supposed to be doing, because that is not known to the user in the first place. Should that have become known, the person could have been replaced with a machine. Instead, software helps people by making small and routine parts of their jobs easier to do. Even that it does imperfectly.

The users constantly struggle to make the software do what they want, the way they want. Sometimes they find the features useful the way they are. Sometimes they adapt to their quirks. Sometimes they find ways around. And sometimes they dump the whole thing.

As such, software is a dumb servant. A very dumb one. It needs assistance by itself.

To be useful, software needs assistance from the user. There must be a way for the user to explain what he wants even though the software may not have this capacity.

But then, returning to the dumb servant metaphor, the user must be willing to help. Therefore, here is my point - the software should be appealing to the user's sympathy or even pity. The user should be empathically connected to the software. To help and not to throw it away should be its first reaction.

I don't have any recipes on how to build such software. Arousing human's sympathy can be difficult even for another human, not for a piece of software. The only answer that I have is that the user should like the software for what seems to be nothing in particular.

The way I see it now, software should be written in such way that the user likes it for no apparent reason from the first sight. If it's pretty and it behaves consistently and it doesn't jump in your face and it knows when to speak and when to shut up and it looks familiar and it looks novel and it has square buttons and it has round buttons, then perhaps the user likes it. But you never know.

Note to self: exc_info is only available after except handler

2007-09-27T12:30:00.000+06:00

It looks wrong to me, but in Python an exception is registered only when it hits the except handler, not immediately after it's thrown. In the following code snippet the first finally block is totally unaware of the exception.

import sys
try:
    try:
        try:
            raise SystemExit()
        finally:
            print "1:", sys.exc_info()
    except:
        print "2:", sys.exc_info()
        raise
finally:
    print "3:", sys.exc_info()

prints

1: (None, None, None)
2: (<class exceptions.SystemExit ... )
3: (<class exceptions.SystemExit ... )

This is not how I would expect it to work.

Upon closer inspection of the sys module's documentation:

exc_info()
This function returns a tuple of three values that give information about the exception that is currently being handled.
Here, 'handling an exception' is defined as 'executing or having executed an except clause.'

This contradicts to my common sense, but it is the way it is, so I'll have to find another solution for the problem at hand.

Re: Backpack

2007-09-25T11:50:00.000+06:00

Having listened to Jason Fried as he speaks about the process they use at 37signals:

http://itc.conversationsnetwork.org/shows/detail471.html

I couldn't agree more to nearly all of his points. Also while listening to it, I had this associaction of the process with nothing but Brownian motion.

See, Jason essentially suggests that you have a small team of positive people in a small and lightweight cart and then allow every single individual product user push it in whichever direction he or she feels appropriate. A single user's push may not matter much, but if many of them push in the same direction, the cart moves. Then, this ease of feedback will also encourage the users to push more.

There are many other fine points in his speech and so I would definetely recommend to listen to it. Anyhow, here is a few other thoughts.

For one, the word "architecture" doesn't come up, but I do believe in architectures. Although architecture wouldn't emerge from the process outlined above, I thought that it would still present at the end, as one of the artifacts Jason calls "embraced constraints". These are restrictions that you enforce on your project in pursuit for optimal solutions. And architecture is just that.

For two, the described process is perfect for exactly this particular niche - web-based collaboration-like projects open for mass public access. It wouldn't work in many other cases, for example, if you build a project which you target towards big companies. Or, if the audience is too few, so that there is not enough user mass to push the cart.

For three, it seems logical to me that such brownian motion wouldn't last forever, the project trajectory would converge to some ideal point. And having such ideal point in mind from day one may be beneficial.

Otherwise, there was may be one thing that I don't agree with, and even that was minor and taken out of the context, it's when Jason says "get rid of boxes and arrows". While he probably meant functional diagrams in the context of the speech, getting rid of my favourite boxes and arrows feels scary. I would never give up such powerful mental instruments as boxes and arrows.

Overall, a great speech.

Re: To mistake is human

2007-09-17T09:44:00.000+06:00

By programming we delegate the right to failure to the computer.

But the responsibility is still with us.

The most valuable pages of the World Wide Web

2007-09-14T13:28:00.001+06:00

The World Wide Web contains a huge amount of files. Of those, HTML pages can point to each other (and other files too) with hyperlinks. Thus created hypertext structure can be presented with a directed graph.

Technically, the World Wide Web graph can contain cycles, but this is only possible if a page has been modified after it has been referenced. Specifically a page which has never been modified after it was created, cannot participate in a cycle. Therefore there could exist leaf pages, which are only linked to and not contain links themselves.

It is also seems reasonable that if page A links to page B, then the owner of page A somehow values page B, even if in some negative kind of way. Otherwise, he wouldn't even bother to put it there.

Now, the question is - aren't the most valuable pages of the World Wide Web the ones that are only referenced to but do not reference other pages themselves ? Taking it one step further, is a value of a page a function of N/M (where N is the number of the links to this page and M is the number of links from this page) ? Then a page with no links in it will indeed have infinite value.

Re: Tono Bungay

2007-09-03T11:30:00.000+06:00

I've recently had a chance to read some Herbert Wells. It's been my opinion before that he was one of the greatest science fiction writers, a great visionary if you like. But it turns out that most of his novels are non-fiction. The ones that I've read are deeply inspirational and more interesting to read since they still have a strong connection to the real life, no matter that a hundred years have passed.

Quote from his Tono Bungay:

Always before these times the bulk of the people did not over-eat themselves, because they couldn't, whether they wanted to do so or not, and all but a very few were kept "fit" by unavoidable exercise and personal danger. Now, if only he pitch his standard low enough and keep free from pride, almost any one can achieve a sort of excess. You can go through contemporary life fudging and evading, indulging and slacking, never really hungry nor frightened nor passionately stirred, your highest moment a mere sentimental orgasm, and your first real contact with primary and elemental necessities, the sweat of your death-bed.

And this, mind you, is written in 1909.

Mirroring system drive under FreeBSD. Simply.

2007-07-03T11:30:00.000+06:00

When configuring a new server, one of the first thing I do is build a RAID-1 mirror across the two hard drives that present in nearly all entry-level to mid-range servers. Good if you have a hardware RAID controller, but the cheaper models don't - they just have two fast (often SATA, not SCSI) drives to live with.

The question is - how do you mirror the system drive once the system has been installed ? Oh, did I mention it is FreeBSD ?

I used to follow the instructions here:

http://people.freebsd.org/~rse/mirror/

and it is good and correct and worked for me many times. Ralf S. Engelschall has undoubtedly put a lot of effort having the script even better over time. It essentially suggests that whenever you need to build a mirror across ad0 and ad1, you copy ad0 to ad1, then build a "mirror" with a single drive ad1 and reboot from it. After a reboot you have a "mirrored" drive ad1 as system and ad0 left aside. Then you add ad0 to the mirror, wait for it to resynchronize and reboot again. You also have to modify a few configuration files, calculate sizes of partitions etc. Such hassle is necessary because you presumably cannot add the system drive ad0 to a mirror when you have booted from it.

Unfortunately something went wrong and it didn't work with the server I had to build just recently. After the first reboot the gmirror provider failed to recognize the mirror on ad1 and boot process failed. Tried a few times - no luck. I guess the problem was in that both ad0 and ad1 have previously participated in a mirror, and I didn't have the drives cleaned as well I should, and some meta information left lurking on ad0. Anyhow, after some googling I found this other article

http://www.onlamp.com/pub/a/bsd/2005/11/10/FreeBSD_Basics.html

It manages to do the same thing with no hassle whatsoever. You simply add /dev/ad0 to mirror/gm0 and that's it. Oh, the little problem about not being able to execute gmirror label on the drive the system is currently booted from ? Easy - an undocumented (I'd presume) sysctl:

# sysctl kern.geom.debugflags=16

will rid of this nuisance. You simply set this debug sysctl and the mirror can be built right away. Somewhat an unclean solution but how much simpler.

You can't write everything by yourself - the missing link

2007-05-28T10:35:00.000+06:00

I always tend to build software myself, not just coding, but also thinking about it, architecting it, experimenting with it and so on. It is always difficult to convince anyone in this approach compared to an off-the-shelf product reuse. Whenever such an argument comes up, I often hear this - "Well, you can't write everything by yourself, you have to use product X !" Indeed, it is impossible to rewrite an OS or a database engine, or a programming language, you have to reuse an existing piece of software. Not that I believe in a developer with a golden keyboard who writes infallible code (in Redmond, Bangalore or anywhere else) but doing so is just very impractical. Moreover, the truth is - you really can reuse just about anything, and doing so often gives good results. I never knew an answer to this, until now. I suddenly understood that it is this:

Even if you haven't written it yourself, you ought to know it so deep, as though you have.

The typical approach contradicts to this, you install something and just assume that it does what you need, exactly the way you need, that it will save you from all the problems you might possibly have. This is a mistake, a very convenient illusion. More often than not, it results in products being used in the wrong way, performing terribly, breaking on you, being cursed and thrown away only to be replaced with another of the same kind, only more expensive and fashionable.

It is therefore crucial to understand the working principles of the software that you reuse, to know the internals, what kind of problems to expect, and so on. And then, will you excuse me for repeating myself, but doing a lot of development on your own helps enormously in understanding the software written by somebody else.

Push a button, have ten bucks paid, repeat a thousand times

2007-04-13T12:08:00.000+06:00

Consider a situation, where a customer pays for something on the Internet. There appears to be a huge perception difference between the client who pays and the provider who collects the payments. The impression that the two sides have on the scale of the affair is completely different.

See, if you are a customer and all you want is to pay $10 for something on the Internet, to you it's a matter of efficiency - through which hoops you have to jump to have it done and how fast you get the stuff you pay for. In an online transaction like that the money themselves don't matter much to a client, for the following reasons -

1. The client thinks of the electronic payment as of the payment with real money, which cannot be mishandled or require any processing which can be delayed or refused;

2. It's actually the merchandise or the service which the client wants at the time. The necessity to pay money, even online, is the mandatory inconvenience, an obstacle to it;

3. The amount of money in question is not that large. Even in the worst case the customer's risk is nearly zero.

I'm not saying a customer will tolerate losing money in an online transaction. What I'm saying is that at the moment of such transaction the client will not worry much about what could happen. To a client, it's all in "push a button, have ten bucks paid, how complicated such a simple procedure could possibly be ?"

And it wouldn't unless there were thousands of customers. When serving a single customer, it's easy to take the money, even manually, over a counter and process the payment, but processing a stream of thousand payments is different. Then you have a different perspective at the same $10.

When you are a provider, the problems that you face all have the same root - that you are a money pipe - anyone can use your services to buy something for themselves. What are the outcomes ?

1. Responsibility before customers. You simply cannot afford to fail. When you fail to deliver, the customers will haunt you, even for the same lousy $10. Dealing with this requires certain investments in reliability of the solution.

2. Freedom to be abused. A strong incentive is presented to the entire world to hack you and profit from that. This asks for security-oriented thinking.

3. Overwhelming complexity. Unlike the customer, you understand the guts of the service, and see the great many places in which any given payment can fail. And you have to maintain it.

See, from the provider's point of view the same ten bucks payment becomes nothing short to a hand grenade.

As it is my primary job to develop such solutions, I'm obviously on the provider's side. And since I'm a software developer, I have one more problem to deal with - the deceitfully simple outside look of the solution, remember - one button, ten bucks... Should the management adopt a customer's view, then for them it's a similar question of "how difficult a development of something that simple could possibly be ?".

But then, indeed, how difficult could it be to develop something that simple ?

Re: The Illusion of Certainty

2007-03-13T09:30:00.000+05:00

Found this article on Design Observer:

The illusion of certainty

An interesting observation on how illusionary a structure imposed by form (or anything really) can be.

Quote:

... the rational side of our brains leads us to such solutions because they gesture to an odd kind of certainty. That tension — between structure and freedom, between form and its variation — is an essential characteristic of design thinking.

How new is the new Blogger ?

2007-03-13T09:22:00.000+05:00

A quick question before I proceed to another post I was originally up to.

How new is the new Blogger compared to the old one, if to user experience they differ only in the login form ?

Do radars have screensavers ?

2007-03-06T09:54:00.000+05:00

I was just wondering -

do radars have screensavers themselves ?

What difference does it make to know your people ?

2007-03-01T17:17:00.000+05:00

A banner has just popped up - "monster.com, 40 000 000 resumes". Or something like that.

Imagine a database of 40 000 000 documents prepared by people who have strong incentive to lie. What would level of noise be ? How reliable would it be ? What amount of sifting through is required to find anyone ?

Compare this to how this company works - Core Search Group

A recruiter from them once contacted me, but not with a standard template offer, heck, not even with an offer at all. The guy has actually read my blog and (I'd guess) my published resume and addressed me with remarks about it that made sense ! I responded with a detailed explanation of who I am and how I work and this, beside my resume may be somewhere on their file.

I realize this is their work, but still, what difference does it make !

What did you do today ?

2007-02-26T11:00:00.000+05:00

This one kept bugging me for a long time now.

It appears that many people hate the work they do. I can see how one can be forced to take a job he doesn't like, fate, bad luck, blah blah, but still, I totally don't understand this - how could you live if you cannot be proud of the work that you do ?

What do you say when you come home every day ? And I don't mean - to your family, but even to yourself, what do you say to yourself - I did what today ?

How comfortable it is when you can't point a finger and say "I made this" ?

When you are making crap, and everybody knows it, when people curse and spit when they encounter something that you've made, how do you feel ?

I do believe that most people still feel bad when then do their jobs wrong. Although they can find excuses and even blame somebody else, there has to be something, because if such behaviour was perfectly ok for them, they wouldn't have been looking for excuses and blaming others, and the more agression means more discomfort.

Which means - a lot of people hate their jobs, do them miserably and are at more or less permanent stress about that. Isn't that terrible ?

Never underestimate the power of randomness

2007-02-20T17:15:00.000+05:00

I've just returned from a deep testing and debugging session and all I can say is again - wow ! never underestimate the power of randomness !

The system I was testing is a complex of network services build on top of Pythomnic platform with multiple Python processes scattered across multiple servers and intertwined together in redundant and fault-tolerant fashion. When it's live, it's going to be the billing hub service for the bank where I work. It has to deal with all sorts of payments to all sorts of providers. And so my job is to build a system to which modules for specific providers will be plugged later. It is also transferring money, so it'd better be reliable.

Someday I'm going to describe the design of that system as a case-study for Pythomnic and publish it on its web site. That will be, but for now, here is my recipe for the best testing:

Stress + failure injection + randomness

Stress: don't spare the system you are testing. The users will not. Give it as high load as it can handle and then some. It's no problem if it breaks now, but the amount of problems (not always bugs) that are revealed under unbearable load is surprising.

Failure injection: don't expect the problems to happen just because you are testing. Make them happen. Break stuff. Insert something like:

if random() < 0.01:
raise Exception("failure before provider request")

if random() < 0.001:
sleep(3600)
raise Exception("provider request hangs")

result = provider_request()

if random() < 0.01:
raise Exception("failure after provider request")

Insert it all over the place. Well, there is no point inserting failures between each two statements, it quickly gets cumbersome, but you should decorate each "external" call with such injected failure frame. It may be a database request, a specific API call etc.

Randomness: that's my favourite part, in testing, you can't beat randomness. You would never make up such a combination of failures that random() would. Make sure your random switches cover all the major code paths and let it running for a while. If it succeeds you can be pretty certain the system is working. To be sure, such random testing may not catch all of the special border cases in each of the modules, but for load testing - it's invaluable.

What eternal have we produced ?

2007-01-18T09:40:00.000+05:00

For any system, living or artificial to span a long time it ought to change. Evolution, change over time are a sure signs of life. Rigidness, immutability and stasis are a no less sure signs of death.

Now, there is an interesting perspective to this. The strive to perfection leads us to creating something absolute, things that will exist forever. Any artist or craftsman would certainly like his creation to last for thousand years. But is eternal unchanged presence good ? By the way, have you ever seen anything eternal recently ?

Look around. Look for perfect. Indestructible. Eternal. See anything like this ?

Plastic, glass, and radiation.

Pretty much anything man-made will eventually disappear. Any piece of art will die out. Any building or construction will collapse. But not those three, not in any foreseeable future.

Every single grocery bag will stay forever. Empires rise and fall but the empty milk bottle will swamp around. Radiation will remain invisible, but the half-decay principle will ensure it will only vanish in asymptotically distant future.

Isn't it ironic ? Do we need eternal plastic bags ? No. Perfect milk bottles ? No. But this is what we've got. To make it worse, nothing really complex and useful can be done out of plastic or glass alone. Anything constructed with it will decay and leave around useless pieces of eternal stuff. Somehow we have finally managed to produce something that will outlast us million times. Something perfect. Something perfectly useless and dead.

Why having processing power wasted ?

2006-12-30T18:11:00.000+05:00

I don't know about you, but seeing unused hardware troubles me. Doesn't matter if it's an old 386 desktop or a 8 years old Compaq Proliant 7000 which I've recently acquired.

Hardware is always fun to set up, see it running and play around with. Folklore about 100s refurbished Pentium's making up a cluster in a garage fascinates me. I do realize that a single modern server will eat that hundred for breakfast, but still, wouldn't that be fun, running that garage ?

And so, my working PC and the servers that are in testing or spare, are normally loaded up high most of the time, doing what - running workload tests of course ! Right now I'm leaving it running a stress test for Pythomnic, specifically its new capabilities of distributed transactions with recovery. The test will be running for a few days now, and I hope when I return from holidays it's still up and running.

JMS: lock on Java

2006-12-25T10:59:00.000+05:00

I have to admit, I have only a basic knowledge of Java. And I wouldn't ever touch it. For some reason, when writing in Java, I feel like swimming in tar - possible, but hardly enjoyable. There may be some hidden beauty beyond my knowledge, but I simply don't want to spend time on getting deep into it, so deep that that beauty-of-possibility is revealed. And frankly, I don't believe there is one.

But then "industrial standard" kind of arguments bite, and all of a sudden there is a significant development around in Java. It's totally beyond me why solutions have to be adapted to technologies, not the other way around, but hey, it's real world.

And so, here comes another industrial standard - JMS. But, if you ask me, JMS is no industrial standard. It's a standard way of implementing messaging in Java world. In other words, JMS is a perfect example of a platform-specific technology, created with one thing in mind - to lock the developers to that platform. Many of the J-things are such, don't you think ?

See, JMS is Java Messaging Service, which asserts right from the start that anything non-Java is out. Should its designers have cared about interoperability and ease of integration, they would have standardized an open network protocol. How much easier life would have been if there was simply generic Message Queueing Protocol, say MQP based on TCP, optionally UDP, with very simple and clear semantics for swapping plaintext (i.e. XML) messages. By now we would have had tons of implementations in different languages, of better quality too. Not the case. Not only JMS transport may be based on RMI, Java-specific thing by definition, it's also focused on transferring instances of Java classes for messages and/or using messages for Java object to Java object communications. A perfect lock on Java. Pity.

Here is one more weirdness - in order to connect to a JMS server you have to know precisely what kind of JMS server software is run on it. Imagine that you'd have to install different software for sending your e-mail via SMTP server running Sendmail vs. Exchange. Sounds awful, but not for industrial standard it's not. What it means is two things - first, the JMS specification failed to provide implementation independence (I'm not sure that it was even supposed to do so), and two, with JMS you have to lock yourself not only to Java, but to particular server implementation, which is even worse. And I mean - theoretically, you should be able to modify your classpath, client configuration file - pf-f-f-t - and switch to a different implementation transparently, but then again - you have to explicitly switch to a particular server implementation, and different servers may have slight differences in behaviour (just love it), hence your clients may break anyhow.

What kind of industry requires such standards I wonder ?

Moscow: exploding buildings

2006-12-06T09:45:00.000+05:00

They indeed failed to explode demolish a condemned building in Moscow. I told you.

Experience matters

2006-12-05T09:43:00.000+05:00

Everything in software can be done in so many different ways. As my favourite saying goes - there are many ways to skin a cat. And just like with the cat, any solution suffices as soon as the problem is solved. The "any solution goes" maxim is valid and very powerful tool for a pragmatic software developer. But there are points to consider.

For one, it's nearly always that we see the problem wrong. It's either lack of knowledge, perception bias or just plain misunderstanding. In this case we are simply solving the wrong problem.

For two, the problem may be a part of something bigger and the way we build the part may render impossible building the whole.

For three, and this is almost universally true, the problem at hand may be a special case of some other, more abstract problem. Now the solution that we use should be evaluated against this concrete/abstract scale.

Besides, solutions are very fragile. You make one wrong decision and it falls apart.

Given all the above, how one manages to ever build anything ?!

My answer to this question and the subject of this post is that experience matters.

To me, a "I tried it and failed and this is why" is better than "I did it by the book and it worked". The former is a sign of a creative mind that just took a wrong turn, but (and it is very important) was able to accept the failure and find out the cause. The latter is someone who believes in universal solutions and silver bullets and never asks questions. The paradox of the situation is that the he still has a working solution this time around.

It's not like failures are valuable all by themselves. It's the experience you bring out of them. Then, like I said before, you have to actually work on your experience. Experience is not the same as the sum of all your failures (and successes for that sake). Deep analysis is required to turn individual failures and successes into valuable experience.

One other problem is that experience is very personal, even intimate thing. The mentioned analysis is often subconscious and results in something that one may call intuition or "gut feeling".

But you rarely work alone and for the team, what's experience of its individual members ? It cannot be transferred from one to another and it cannot be enforced. Try convincing somebody else in your solution using "my experience is that" or "I feel this is right" kind of arguments. "But I feel different" - this is what you are going to hear.

Individual experience then can only be effectively used within a team if you have worked an essence out of each of your successes or failures and are capable of reasoning and giving arguments that never mention experience at all.

The other use for experience of individuals is have the team highly specialized, or even have single leader and support team (a-la Frederick Brooks' surgical team). This may lead to better results faster but yields less for the group development.

Besides, the team can and should have experience of its own as a whole, but it's another story...

"Software as a floor mosaic" metaphor

2006-10-07T15:53:00.000+06:00

Here is a "software as a floor mosaic" metaphor: let's say we need to inlay a floor mosaic. We have a sketch or may be even a highly detailed plan of how the thing has to look like, and even the vision of it and clear and perfect, something like this:

We also have at our disposal assorted pieces of stone, ceramic patches, coloured sand and all the tools we might need. But no matter what tools do we have, it takes experience and above that a lot of time to make such a mosaic. Yes, experience helps but we still need to take each piece, adjust it, finish it and put it in place. Sometimes the piece will crack in our hands and we have to start over. Sometimes, a piece will be very different from what's needed in that particular spot and require much more work than another.

But then there is time pressure - the worst of all curses. Pressed with time we have the following alternative: to make it faster, we might take bigger stones and throw them in quickly yielding something like this:

And it may even sort of look similar, but I just can't force myself to see the plan in the implementation. Aesthetically, at least, it's obvious that mosaic laid out in this fashion is no match to the envisioned one. But how did our big stone practice made it break ?

In other words, what can we say about big stone mosaics ?

Good:

Bigger stones are by definition fewer, so we have to invest less in learning. The less stones we have to learn how to deal with, the more confidence we have in that we know how to handle them properly.

There is less variance between big stones, consequently there is no choice as to which stone pick for this or that spot - one less burdening decision to make.

Takes fewer pieces to lay the mosaic out. And it saves time, this is precisely the reason we choose this way.

The less pieces, the less interaction between pieces (although I must admit, this is really more a software insight).

Bad:

UGLY. There are situations in which big stones fit in precisely, but those are exceptions, rather than rules. If we are up to a piece of art, not to the standard kitchen tile. This is the aesthetics-breaking reason. This is why the first one is amazing and the second one is not even satisfactory.

Along the same lines of aesthetics and taste goes this - if someone is always doing big stones, how could she tell what's beautiful and what's not ? It's still a mosaic, isn't it ?

Bigger stones obviously not always fit and we have no control over it. If we want control after all, we need to apply much more work compared to what would have needed if we were combining smaller ones from the start (that's exactly the reason why mosaic exists - we take small patches and combine them in a such a way that the whole thing looks beautiful from some distance, and don't spend forever cutting).

A mosaic built of bigger stones is less reliable (alhtough again, it's more of a software insight). We may be confident, but there is no way we can tell a crack inside a big stone, and if it happens to exist, we have no control.

And so, what's the average stone size modern software mosaics are being built of, what do you think ?

Security and usability of a real-life elevator

2006-10-03T10:43:00.000+06:00

Security problems often appear from unforeseen interactions between different system parts. Likewise it happens when new features are added or existing ones are modified. Here is another example of that.

The office where I work occupies a five-storey building. All entrances to any level were initially equipped with card locks, and so to enter or to go from one level to another you had to have your card with you. This brought a certain amount of security, in terms of authentication and audit.

The problem was, there was initially no elevator, only stairs to access the levels. As time went on, convenience considerations prevailed and so an elevator has been installed. As such thing hasn't been predicted in the first place, the elevator shaft now couldn't be locked with card locks, simply because there is no doors on the elevator compartments, and such doors can't be installed (easily, or at all).

What happened is that security has been breached - once you have access to one level, you could take an elevator to any other without a card. This effectively defeated the entire system.

Anyhow, this is not the end of the story yet although the rest of it adds nothing from the security perspective. It's just that the elevator seems to attract user interaction problems as well.

And so one of the levels is also undergoing heavy repairs. You can enter it with or without a card now, but it's a mess therefore noone is supposed to. But the feature of entering any level without a card (in a form of elevator) is already there. Now, what's been done to prevent unaware people from using it in the wrong way, i.e. going to the second floor ?

The elevator button panel has been patched (literally, with a piece of paper and tape) so that you couldn't go to level two at all.

Now every day I come to work, this patched button reminds me of how difficult and unpredictable security and usability really is where it meets real life uses.

Building bridges

2006-09-05T11:16:00.000+06:00

Just heard it in the news - they decided not to explode the remains of the flyover bridge collapsed yesterday, here, in the city where I live. Thank God they wouldn't. Because they haven't manage to build it in the first place. With all their industrial-grade materials and world-class technologies, the thing crashed two months before it was supposed to be opened. Can you imagine what happens if they use world-class explosives ? Stop doing what you know not how to do, will you ?

Same thing happens in software and it makes me sick. Guess what ? Technologies and products are worth nothing unless you actually know how to build stuff. You have to actually get to it and do it over and again before you know how to do it. If you decide to use explosives to bring a building down, you'd better be damn proficient in exactly that sort of things. If all you previously have done was paving the roads, but you occasionally have a pack of industrial-grade high explosives, your attempts to follow world-class procedures in demolition will likely go down in flames.

The worst mistake of all when you want to build quality stuff is that everything is simple. No damn it, it's not. It's damn hard. You have to know ins and outs and you have to learn by your own mistakes, and never assume that bying Oracle will solve all your application performance problems.

Experience is the name ...

2006-09-02T18:43:00.000+06:00

About this quote by Oscar Wilde:

Experience is the name everyone gives to their mistakes

I thought it had an interesting language twist to it. Although the apparent interpretation suggests that everyone is hiding their mistakes behind the word "experience", there is another way of looking at it:

Experience is the name everyone gives to their mistakes

so that it means that everyone finds a name for their mistakes, and that name becomes the experience. Thus by picking good or bad word for referring to their mistakes, everyone can totally change the way they perceive their experience.

All systems are real-time and need extra processing power

2006-08-25T10:36:00.000+06:00

Real-time systems have much hype on them and as always need a special form of speech to cover the shortcuts. In case of real-time such speech is about "soft real-time".

The problem is - "hard real-time systems" (the ones that ALWAYS meet execution deadlines) are fiendishly difficult to build, especially on the mainstream hardware, using mainstream practices, for any kind of realistic load.

"Soft real-time" systems are thus not required to always meet execution deadlines, but rather have a probabilistic process imposed on them. Yes, it's bad if you fail to react in time, but it's still better if you react 3 ms late 40% of time than 10 ms late 10% of time (depends on the application of course).

But then, aren't all systems real-time ? After all, noone needs a system which never finishes processing, and it's better doing it sooner than later. Systems may have more or less relaxed requirements, but it's the same probabilistic process, for example 90% of users would wait 5 seconds for a transaction to complete, but only 10% wouldn't be calling tech support upon a 30 seconds delay.

Look closely at the word "real-time" itself. Everything is executed in real time, all processes have references to time flowing by, executions have starts and stops and whether or not they've been a success is determined by an external entity (a user in most cases) and timing is not the last criteria.

Anyhow, I'd like to point out one other thing. There is this theorem which says that for a real-time system to meet its deadlines under certain conditions, it must have a significant fraction of CPU processing power free. If I remember correctly, the exact top load percentage is around 70%, which means that if the CPU is busy more than that, the system starts missing deadlines.

If this result is blindly (or philosophically) applied to all the systems, it leads to an interesting conclusion - no system should be functioning under stress load (perhaps above the mentioned 70% CPU), otherwise it starts slipping behind the schedule and, being real-time, degrades its service significantly.

It would thus be a mistake to believe that an application quality of service would remain steady across all of the CPU load scale. Instead, as CPU usage climbs up to 70%, the application may scale perfectly linearly, but as the load keeps rising, the scaling breaks. It's therefore more safe to consider 70% to be the top acceptable load and start adding more processing power as this line is crossed.

Re: The "Snakes on a Plane" Problem

2006-07-30T13:57:00.000+06:00

Found an interesting article on Reddit:
The "Snakes on a Plane" Problem

Quote:

I worked in newspapers for eight years, right when that industry was starting to disintegrate. As such, we spent a lot of time talking with focus groups, forever trying to figure out what readers wanted. And here is what they wanted: everything. [...] When it comes to mass media, it's useless to ask people what they want; nobody knows what they want until they have it.

Why this sounds all so familiar ?

One other reason why new software is released

2006-07-27T14:18:00.000+06:00

Because the users keep using the old one in the wrong way.

Reusable business-specific abstraction is an oxymoron

2006-07-24T09:26:00.000+06:00

In this article I'd like to talk about abstractions in software development, focusing specifically on business-specific reuse.

Abstractions come in many flavours. At the most general level, abstraction is the single most valuable method of analysis. Mathematics is all about it. Abstraction is also the result of that analysis. I will be using the term "abstraction" as if it meant "a concept relevant in particular context". In simple terms abstraction is "something that ..."

In programming, abstractions appear in many places. It's impossible to discuss all of them, so here is a few random examples before we proceed to the points of the discussion.

The simplest abstraction in programming is the byte. The concept of byte suggests that (1) all data we manipulate with can be converted to numbers and (2) the numbers can be encoded using fixed width integers. Although bytes with size different from 8 bits have been experimented with, they were not successful (not that I saw CDC Cyber myself, but the books say so). Let's think why all the bytes have converged to 8 bits. My answer is - because there is no difference. They could have been of any size, 8 bits were just a good balance of range/processing power vs. costs to produce at that time. As the costs kept falling, the bytes actually grew (although under a name of machine word now). Anyhow, the byte became a universally accepted abstraction, reused literally everywhere.

Abstractions also appear in OOP where they hide in the shadows of the objects themselves. After all what is an object if not an abstraction of some real-world entity ? OOP is very successful in giving developers abstraction producing machinery which they apply to their problems yielding abstractions directly translatable to programming languages. Not all the created abstractions are reusable, leave alone useful, but OOP itself is.

As we can see, low level abstractions can become quite successful and widely adopted. This is because they are so "abstract" they apply everywhere out of the box, and changing them would make no difference, and so nobody even thinks about why they are there and what's their purpose. They are just there, they've always been (for an eternity of fifty years), they are the truth. Although this approach is far from scientific, it works well in production. After all, why do I need to know the details of the chemical process for producing plastic, as long as all I need is to wrap my groceries in a plastic bag ?

And so, my first point - easily reusable abstractions are invisible. We use them without ever noticing. "Why ? Of course our data is encoded in bytes, what a stupid suggestion !" Low level abstractions are such, but hardly any others, at least right now.

As we move farther towards the "real business" side of programming scale, it's increasingly difficult not to notice the abstractions arising more as obstacles than helpers. One step up, look at what building blocks we have as we build real systems. No matter if it's 1980s or 2000s, the abstractions that your programming environment offers are the same:

User. Session. Context. Authentication with plaintext username and password (anything else is still out of reach because of interop problems).
Database (whatever make, hard to design). Connection. Pool (not always useful). SQL query (hardly anyone knows how to write one that won't bring the server to its knees). Transactions (requires even more knowledge).
Component framework, networking, distributed facilities, RPC, messaging (various compatibility, performance and scalability problems difficult to address).
I18N: Encoding (Latin-1 as the mostly useful one, interop hell again).

And on and on it goes. All those "development" abstractions are useful and widely accepted but still are technical, low-level and because of their complexity only reusable as a mindset. From this point of view, they are patterns more than real tools.

Two steps up, when it comes to real business, there are no abstractions. It has to be done in exact particular way. Nobody's going to use an application if it doesn't make precisely what the user wants. That's why all the software is to some degree configurable and adjustable. The rigid abstract part of the software is clearly separated from the flexible configuration part. The abstraction therefore has nothing to do with the business specifics - it's all in the configuration. And if you are lucky, you've got enough flexibility to cover everything.

Now, I'm asking you this - wouldn't then a "reusable business-specific abstraction" be an oxymoron ?

For one, "business-specific abstraction" is already nonsence. Abstract has its details stripped off and replaced with imaginary concepts. But details are the heart of the particular business, no matter if it's called "valuable proprietary know-how" or "ugly poorly understood mess". And it makes difference. And it's highly visible. If you are switching from DCOM to SOAP, nobody cares. But if you mishandle customer's data for the sake of abstraction that you might have - well, they won't approve it. It has to be done their way or no way. If your abstraction is too rigid - too bad for it.

For two, "reusable business-specific" is also nonsence, as soon as you are crossing enterprise boundaries. Unless Big Brother takes over, every single business is going to be to some extent unique and as such will not be covered by the abstraction. Even if you are borrowing a clearly separated network service, not burdened with thousand dependencies, it's likely to be useless without tweaking.

And so my second point is - business specific software can hardly be abstract and hardly reusable. In fact, the most valuable property of such software is flexibility. It has to allow easy changes, no matter what abstraction has been put in it.

That's one of the points behind the Pythomnic project - allowing developers to build network services that can to any extent be changed on the fly, without service restart and as soon as you have read this far I invite you to check it out.

Always write what you really think ...

2006-07-22T13:13:00.000+06:00

... because if your thinking is right, so will be your writing, but if your thinking is wrong, it doesn't matter what you write anyway.

Want to automate your business ? Want a single-click software ? Prepare for changes.

2006-07-06T15:55:00.000+06:00

Very few people actually love the work they do. If it's somehow done magically with no concern of theirs, hey, they'd rather be fishing ! People have always tended to avoid work. Having somebody else to do one's job is the best strategy under the sun. Even for those who are not very sick of their job today, and pretend to be doing things, the less actual action - the better. Laziness is a much better driver than fictional pursue of productivity, urge for business success or corporate loyalty.

Now, people don't need software at all. Like I said, they need their job done, preferrably with no load on themselves. But software was giving such promises about how much easier it would be (if you purchase product X) since day one. Not that many still believe that new version of Microsoft Word will boost their productivity, but hey, this is how documents are prepared in 21st century, ain't it ? It's just that you install it, boom ! and your documents "practically manage themselves" (a ridiculous sentence if you ask me).

Therefore users dream of software with one big button reading "Get my job done now" (even then there would be some moaning about how much hassle it is to push it). It is indeed crucial to design software in such a manner that it requires as little user intervention as possible. But doing so is very tricky.

Let's say we are developing an accounting package. The first problem we are going to face is ubiquitous lack of knowledge. It probably wouldn't surprise you the least that someone with degree in CS knows nothing of accounting, leave alone specific customer's business practices. What would surprise you for the first few times is that the users themselves don't know what they are doing, no matter if they've got a degree in finance.

We are likely to see that everyone knows a little bit but nobody sees the big picture. "It's just how it's done here, that guy over there probably knows something" is the sort of speech you may hear. Blind leading the blind, at its worse.

A good business analyst can save the day, no doubt. Since a good business analyst essentially is a person to some extent trained in both areas, experienced in exactly that kind of projects and (which is crucial) clever, fluent and clear in his thoughts and talks, he could definetely talk things out.

But the business analyst will actually be busy recognizing an elephant from descriptions given by three blind wisemen. Or, sorting out all the quirks of the customer's business practices, which is essentially the same thing. Even if (and it's a big if) he succeeds all the team gets is a 10 sq. ft. diagram with lines going in all directions. Better understanding, perhaps, but of what ? And this is where the second problem hits - ugly and tangled business practices.

Well, the customer is always right and all that, but what's the point in one-to-one mapping software functions to existing business practices ? Nobody understands how it works, I see. Now you have software that automates that ! I mean, it's just outrageous. You take a can of worms, sort of peek into it, then, using whatever perfect (huh ?!) software development process, implement the automated can of worms and hand it over to the customer.

"Eeew !" - that's what the user says - "I thought this software thing would be simpler ! Where is the "Get my job done now" button ?!" Oops, there is no such button. What you get is a button for every chaotic activity you have performed before the software extravaganza started.

And so my point is - for a business to benefit from software automation, it must work well enough even before and without any software. Chaos cannot be automated, that's what I say. As there is hardly a perfectly organized business out there, well, it needs to be changed before it can be automated.

The good news is - those two processes can be performed in parallel and greatly support each other - straightening (uh, streamlining is so much a buzzword) the existing business practices, and analysis of the same for the sake of software development. If there is a perfect moment for changes, it's now - there is a lot of efforts put into it from the software developers camp, and people genuinely interested in understanding and making changes will be around, not the stupidest people too. All the customer needs is to allocate decent resources and get ready to change.

The bad news is - a customer is more likely to remain heavy and unresponsive. "There, we've paid you, now where is our button ?" Still, as you know, changes come, now or later, and it's better to change it right now than to postpone.

To conclude, I believe we as software developers should be working not just on software but also on improving our customer's business rules, it's in our own best interest. Moreover, we should be pushing the customers to changes, not just accepting their requirements.

Open source, closed source, can anyone tell a good piece of software from a bad one ?

2006-07-04T09:51:00.000+06:00

To start the discussion, let me ask you this - shall we release early and often ?

If we release early, we are likely to release crap, but have better exposure. But people are already used to software being crap. Besides, crap or not, there is no easy way for users to tell.

But if we go the long way of making the software better as we see it, and release later, the quality might be better, but it's still expected to be crap (release 1.0, anyone ?) and even though the experience might prove otherwise, the impression is still about the same. Besides, whi will care ?

Paradoxically, we are pushed towards producing crap faster.

Again, the key problem here is that we have no way of evaluating software quality. The closed source used to rely on the quality signals that, once been in favour have long since sold out - brands, advertisement, experts, reviews. The open source may rely on community judgement, popularity and zealotry, but what is it if not bandwagon effect ? Is it any better then ? I don't think so. Both sides rely on experience, but then again, experience is nothing but a way of convincing oneself.

Anyhow, my point is - open source, closed source, both parties have their reasons, politics and propaganda, both are eager to sell, but not only neither knows a way of producing high quality software, nobody can even know for sure if any given piece of software is good or not.

Think about it. We make stuff. We are proud artists (each a Picasso, no less). But when presented with somebody else's work, we cannot tell its quality. Isn't it outright ridiculous ? What kind of an industry is that ?

We thus are tackling the wrong problem. We need to build quality things. Instead we invent different ways of building things of unknown quality.

We desperately need reliable indicators of software quality.

Perpetual movement of crap

2006-06-24T16:10:00.000+06:00

Software is crap. All of it. Or, may be you can prove otherwise ? But I'm warning you - I'm not buying a name underneath the ®, a web-site full of buzzwords, a stellar magazine review or even a community size of China. Why not ? Because neither is reliable. Now, all the above left out, how can you make me believe it's good ?

We desperately lack means to evaluating software quality. Not just quality, but all the -ilities aka non-functional requirements. It's so bad - it's not even funny. We cannot examine the software as such, we never look at it directly, all we have is different directions from which we observe its reflections. Software is incredibly opaque - it's as if you had to evaluate an actor performance on a stage veiled by not one but many curtains. Difficult casting problem, isn't it ?

What exactly are those curtains the software is hidden behind ? I believe they are exactly the steps we take to make it - to come from an idea or perhaps a set of basic requirements to the working machine code. It looks innocent, even proud - all those philosophical aspects of OOP, applied math with its abstractiveness so profound, the development processes with metrics and iterations and so on.

But each step takes us further into the land of not understanding, and the sad truth is - our methods don't work. Somewhere on this glorious path, something slips and all we get at the end is crap. A wonderfully abstract crap. A beatifully modelled crap. A crap produced with the process tailored by developers at a CMM5 level.

And not just that. Now that we have our methods applied, it's impossible to see the thing we have been building in this pile of crap. It's shredded crap in a box - it's impossible to see through. Isn't it funny how testing speaks about black box vs. white box testing ? The box - note to yourself.

Being compared so often to the construction, software development is far worse in that it's a land of fairies - it's difficult to apply your common sense in a world where buildings can be started from their roofs and left hanging in the air.

Imagine that you are buying a house but you can't see it - it's in a box - granted a big one. All you can do is to give it tests - does the water start when you open the tap ? Yes, says the developer, it does - trust me, we have installed the tap complying with the latest industrial standards. Can I see it, you ask ? No, it's in the box, but it works ! You tap something which sort of looks like one and hear the sounds of water. Uhm, ok. Does it have windows ? Yes, sure it does. Can I see it ? No, it's in the frikin box ! How can I be sure the windows are installed in walls, not in floors or are not even left leaning to a basement wall ? Trust me on this, we are professionals here. So much for a black box house testing. Now, white box testing is not much better. When applied to houses, white box testing wouldn't mean you are allowed in to look around - instead you are allowed to look at the blueprints. Oh, now it's so much better. As a home owner I couldn't tell the mentioned tap on the blueprint. Besides, having a finest project of all still doesn't mean the windows are where they are supposed to be.

Unable to see for yourself, how do you estimate the quality of software ? You go for much less reliable sources - the talk of the people who want to sell it to you. Now, that's who you can trust - the salesmen ! Their cheap talk..., but I don't have to tell you that. The words they are using, please, don't let me start - "industrial standard" - that's my favourite... Trade shows, web site reviews, free T-shirts, it's all part of the selling game.

But think - what speaks louder than all the advertisements, and is closer to the real situation ? Hint - it literally yells in your face, sometimes several times a day, you are so used to it, you are skipping it right on the spot by clicking "Next". What is it ? It's the software license. No matter if it's a proprietary license or whatever orthodox OSI certified open source one - the most important part of it says that the people who did it have no confidence in the thing they are selling you. It comes with no guarantee for god's sake. Where else you can buy a thing and rightfully expect it not to work right out of the box ? And you can't return it for a working one, he he he...

Don't ever expect the software not to be crap. You can write it yourself (that's what I personally always prefer), but it'd still be crap, although you'd have a much better hold of it and earn experience. Besides, you can't write everything from the ground up, it's insane. You may just as well pick your software at random and then clench your teeth and stick with it through years until the fact that it's crap is so obvious that you cannot bear it any more. I've seen it a few times when a 3rd party software (complying with industrial standards, no doubt there) bought for ridiculous amounts of money left everybody engaged for a while, harder and harder, until eyes pop out and somebody decides that it needs to be replaced with another one (complying with newer industrial standards).

There is one other aspect I'd like to bring out.

It's ever so cool to proclaim that "lives depends on the software" - makes you so proud and self-assured, like hey ! the software that we write here - everybody depends on it ! We are cool ! It's an exaggeration at the very least. Lives depend on software no more than they do on Hollywood movies - yes, they want it, and there is an outrage if theaters are shut down, somebody may even get killed in a riot, but it only concerns those who actually allowed the movies to take so important place in their lifes. Likewise with the software - if your word processor crashes, there is an Internet outage, your bank's ATM network is out of service, your flight is cancelled while you are in the middle of nowhere, or even if your plane goes down due to a software malfunction - does that mean your life depends on it ? No, it's your habits, your ways, your convenience that do. What's true to the point is that despite having no real importance to anyone's life, once it's been given the right to shape the lives - the software may not stop. We are used to it - it's all around, it's uncomfortable to live without it.

And so, where does this discussion leave me as a software developer ? I build crap, nobody really needs it, but everybody sort of expects that the crap is there every time they look. And so I have to keep the crap running. I come to work every day and keep pushing the wagon full of crap so that it doesn't stop. If I'm lucky, I can take a few hours free of pushing and try to build something different, something which is either less crap, or at least a better wagon someone will have to push. Still.

Perpetual movement of crap, that's what our discipline is at its current stage of development.