Note to self: default parameter values are mutable in Python

Just hit a somewhat unexpected behaviour in Python code. What would the following code snippet print, what do you say ?

def foo(x = []):
    x.append("bar")
    print x

foo()
foo()

If you think about the def statement as a declaration, the answer is obvious - it should print

[ 'bar' ]
[ 'bar' ]

but in fact it prints

[ 'bar' ]
[ 'bar', 'bar' ]

Why ? Because in Python def is an executable statement, which means that the list of arguments for the method to be created (x) and most importantly their default values ([]) are themselves nothing but arguments to def. Something like this:

foo = def(x, default_x)

and when this gets executed, default_x is bound to something that at that point evaluates to an empty list, but remains mutable. Then, whenever the created foo is executed, the append method modifies the contents of default_x - effectively the "default value" of x.

This sounds strange, but is clearly documented in the language reference, quote from http://docs.python.org/ref/function.html

Default parameter values are evaluated when the function definition is executed. This means that the expression is evaluated once, when the function is defined, and that that same "pre-computed" value is used for each call. This is especially important to understand when a default parameter is a mutable object, such as a list or a dictionary: if the function modifies the object (e.g. by appending an item to a list), the default value is in effect modified.

The spec suggests using None for all the default parameters, but you should have no problem using any immutable objects as well. For example:

def foo(x = None):
    print (x or []) + [ "bar" ]

where x is None, or

def foo(x = ()):
    # oops, no append method
    print x + ( "bar", )

Phew, I've been lucky using None's so far...

Sympathy for Mr. Software

No software does what the user is supposed to be doing, because that is not known to the user in the first place. Should that have become known, the person could have been replaced with a machine. Instead, software helps people by making small and routine parts of their jobs easier to do. Even that it does imperfectly.

The users constantly struggle to make the software do what they want, the way they want. Sometimes they find the features useful the way they are. Sometimes they adapt to their quirks. Sometimes they find ways around. And sometimes they dump the whole thing.

As such, software is a dumb servant. A very dumb one. It needs assistance by itself.

To be useful, software needs assistance from the user. There must be a way for the user to explain what he wants even though the software may not have this capacity.

But then, returning to the dumb servant metaphor, the user must be willing to help. Therefore, here is my point - the software should be appealing to the user's sympathy or even pity. The user should be empathically connected to the software. To help and not to throw it away should be its first reaction.

I don't have any recipes on how to build such software. Arousing human's sympathy can be difficult even for another human, not for a piece of software. The only answer that I have is that the user should like the software for what seems to be nothing in particular.

The way I see it now, software should be written in such way that the user likes it for no apparent reason from the first sight. If it's pretty and it behaves consistently and it doesn't jump in your face and it knows when to speak and when to shut up and it looks familiar and it looks novel and it has square buttons and it has round buttons, then perhaps the user likes it. But you never know.

Note to self: exc_info is only available after except handler

It looks wrong to me, but in Python an exception is registered only when it hits the except handler, not immediately after it's thrown. In the following code snippet the first finally block is totally unaware of the exception.

import sys
try:
    try:
        try:
            raise SystemExit()
        finally:
            print "1:", sys.exc_info()
    except:
        print "2:", sys.exc_info()
        raise
finally:
    print "3:", sys.exc_info()

prints

1: (None, None, None)
2: (<class exceptions.SystemExit ... )
3: (<class exceptions.SystemExit ... )

This is not how I would expect it to work.

Upon closer inspection of the sys module's documentation:

exc_info()
This function returns a tuple of three values that give information about the exception that is currently being handled.
Here, 'handling an exception' is defined as 'executing or having executed an except clause.'

This contradicts to my common sense, but it is the way it is, so I'll have to find another solution for the problem at hand.

Re: Backpack

Having listened to Jason Fried as he speaks about the process they use at 37signals:

http://itc.conversationsnetwork.org/shows/detail471.html

I couldn't agree more to nearly all of his points. Also while listening to it, I had this associaction of the process with nothing but Brownian motion.

See, Jason essentially suggests that you have a small team of positive people in a small and lightweight cart and then allow every single individual product user push it in whichever direction he or she feels appropriate. A single user's push may not matter much, but if many of them push in the same direction, the cart moves. Then, this ease of feedback will also encourage the users to push more.

There are many other fine points in his speech and so I would definetely recommend to listen to it. Anyhow, here is a few other thoughts.

For one, the word "architecture" doesn't come up, but I do believe in architectures. Although architecture wouldn't emerge from the process outlined above, I thought that it would still present at the end, as one of the artifacts Jason calls "embraced constraints". These are restrictions that you enforce on your project in pursuit for optimal solutions. And architecture is just that.

For two, the described process is perfect for exactly this particular niche - web-based collaboration-like projects open for mass public access. It wouldn't work in many other cases, for example, if you build a project which you target towards big companies. Or, if the audience is too few, so that there is not enough user mass to push the cart.

For three, it seems logical to me that such brownian motion wouldn't last forever, the project trajectory would converge to some ideal point. And having such ideal point in mind from day one may be beneficial.

Otherwise, there was may be one thing that I don't agree with, and even that was minor and taken out of the context, it's when Jason says "get rid of boxes and arrows". While he probably meant functional diagrams in the context of the speech, getting rid of my favourite boxes and arrows feels scary. I would never give up such powerful mental instruments as boxes and arrows.

Overall, a great speech.

Re: To mistake is human

By programming we delegate the right to failure to the computer.

But the responsibility is still with us.

The most valuable pages of the World Wide Web

The World Wide Web contains a huge amount of files. Of those, HTML pages can point to each other (and other files too) with hyperlinks. Thus created hypertext structure can be presented with a directed graph.

Technically, the World Wide Web graph can contain cycles, but this is only possible if a page has been modified after it has been referenced. Specifically a page which has never been modified after it was created, cannot participate in a cycle. Therefore there could exist leaf pages, which are only linked to and not contain links themselves.

It is also seems reasonable that if page A links to page B, then the owner of page A somehow values page B, even if in some negative kind of way. Otherwise, he wouldn't even bother to put it there.

Now, the question is - aren't the most valuable pages of the World Wide Web the ones that are only referenced to but do not reference other pages themselves ? Taking it one step further, is a value of a page a function of N/M (where N is the number of the links to this page and M is the number of links from this page) ? Then a page with no links in it will indeed have infinite value.

Re: Tono Bungay

I've recently had a chance to read some Herbert Wells. It's been my opinion before that he was one of the greatest science fiction writers, a great visionary if you like. But it turns out that most of his novels are non-fiction. The ones that I've read are deeply inspirational and more interesting to read since they still have a strong connection to the real life, no matter that a hundred years have passed.

Quote from his Tono Bungay:

Always before these times the bulk of the people did not over-eat themselves, because they couldn't, whether they wanted to do so or not, and all but a very few were kept "fit" by unavoidable exercise and personal danger. Now, if only he pitch his standard low enough and keep free from pride, almost any one can achieve a sort of excess. You can go through contemporary life fudging and evading, indulging and slacking, never really hungry nor frightened nor passionately stirred, your highest moment a mere sentimental orgasm, and your first real contact with primary and elemental necessities, the sweat of your death-bed.

And this, mind you, is written in 1909.

Mirroring system drive under FreeBSD. Simply.

When configuring a new server, one of the first thing I do is build a RAID-1 mirror across the two hard drives that present in nearly all entry-level to mid-range servers. Good if you have a hardware RAID controller, but the cheaper models don't - they just have two fast (often SATA, not SCSI) drives to live with.

The question is - how do you mirror the system drive once the system has been installed ? Oh, did I mention it is FreeBSD ?

I used to follow the instructions here:

http://people.freebsd.org/~rse/mirror/

and it is good and correct and worked for me many times. Ralf S. Engelschall has undoubtedly put a lot of effort having the script even better over time. It essentially suggests that whenever you need to build a mirror across ad0 and ad1, you copy ad0 to ad1, then build a "mirror" with a single drive ad1 and reboot from it. After a reboot you have a "mirrored" drive ad1 as system and ad0 left aside. Then you add ad0 to the mirror, wait for it to resynchronize and reboot again. You also have to modify a few configuration files, calculate sizes of partitions etc. Such hassle is necessary because you presumably cannot add the system drive ad0 to a mirror when you have booted from it.

Unfortunately something went wrong and it didn't work with the server I had to build just recently. After the first reboot the gmirror provider failed to recognize the mirror on ad1 and boot process failed. Tried a few times - no luck. I guess the problem was in that both ad0 and ad1 have previously participated in a mirror, and I didn't have the drives cleaned as well I should, and some meta information left lurking on ad0. Anyhow, after some googling I found this other article

http://www.onlamp.com/pub/a/bsd/2005/11/10/FreeBSD_Basics.html

It manages to do the same thing with no hassle whatsoever. You simply add /dev/ad0 to mirror/gm0 and that's it. Oh, the little problem about not being able to execute gmirror label on the drive the system is currently booted from ? Easy - an undocumented (I'd presume) sysctl:

# sysctl kern.geom.debugflags=16

will rid of this nuisance. You simply set this debug sysctl and the mirror can be built right away. Somewhat an unclean solution but how much simpler.

You can't write everything by yourself - the missing link

I always tend to build software myself, not just coding, but also thinking about it, architecting it, experimenting with it and so on. It is always difficult to convince anyone in this approach compared to an off-the-shelf product reuse. Whenever such an argument comes up, I often hear this - "Well, you can't write everything by yourself, you have to use product X !" Indeed, it is impossible to rewrite an OS or a database engine, or a programming language, you have to reuse an existing piece of software. Not that I believe in a developer with a golden keyboard who writes infallible code (in Redmond, Bangalore or anywhere else) but doing so is just very impractical. Moreover, the truth is - you really can reuse just about anything, and doing so often gives good results. I never knew an answer to this, until now. I suddenly understood that it is this:

Even if you haven't written it yourself, you ought to know it so deep, as though you have.

The typical approach contradicts to this, you install something and just assume that it does what you need, exactly the way you need, that it will save you from all the problems you might possibly have. This is a mistake, a very convenient illusion. More often than not, it results in products being used in the wrong way, performing terribly, breaking on you, being cursed and thrown away only to be replaced with another of the same kind, only more expensive and fashionable.

It is therefore crucial to understand the working principles of the software that you reuse, to know the internals, what kind of problems to expect, and so on. And then, will you excuse me for repeating myself, but doing a lot of development on your own helps enormously in understanding the software written by somebody else.

Push a button, have ten bucks paid, repeat a thousand times

Consider a situation, where a customer pays for something on the Internet. There appears to be a huge perception difference between the client who pays and the provider who collects the payments. The impression that the two sides have on the scale of the affair is completely different.

See, if you are a customer and all you want is to pay $10 for something on the Internet, to you it's a matter of efficiency - through which hoops you have to jump to have it done and how fast you get the stuff you pay for. In an online transaction like that the money themselves don't matter much to a client, for the following reasons -

1. The client thinks of the electronic payment as of the payment with real money, which cannot be mishandled or require any processing which can be delayed or refused;

2. It's actually the merchandise or the service which the client wants at the time. The necessity to pay money, even online, is the mandatory inconvenience, an obstacle to it;

3. The amount of money in question is not that large. Even in the worst case the customer's risk is nearly zero.

I'm not saying a customer will tolerate losing money in an online transaction. What I'm saying is that at the moment of such transaction the client will not worry much about what could happen. To a client, it's all in "push a button, have ten bucks paid, how complicated such a simple procedure could possibly be ?"

And it wouldn't unless there were thousands of customers. When serving a single customer, it's easy to take the money, even manually, over a counter and process the payment, but processing a stream of thousand payments is different. Then you have a different perspective at the same $10.

When you are a provider, the problems that you face all have the same root - that you are a money pipe - anyone can use your services to buy something for themselves. What are the outcomes ?

1. Responsibility before customers. You simply cannot afford to fail. When you fail to deliver, the customers will haunt you, even for the same lousy $10. Dealing with this requires certain investments in reliability of the solution.

2. Freedom to be abused. A strong incentive is presented to the entire world to hack you and profit from that. This asks for security-oriented thinking.

3. Overwhelming complexity. Unlike the customer, you understand the guts of the service, and see the great many places in which any given payment can fail. And you have to maintain it.

See, from the provider's point of view the same ten bucks payment becomes nothing short to a hand grenade.

As it is my primary job to develop such solutions, I'm obviously on the provider's side. And since I'm a software developer, I have one more problem to deal with - the deceitfully simple outside look of the solution, remember - one button, ten bucks... Should the management adopt a customer's view, then for them it's a similar question of "how difficult a development of something that simple could possibly be ?".

But then, indeed, how difficult could it be to develop something that simple ?