Bosons, my ass

Higgs boson, they say, is the reason for wasting gazillions of euros on a high-tech circular tunnel.

So how come we still use portable energy sources that date back to 1800 and are only capable of only giving a 3000 mAh of power ? How come we can't purposely transfer a significant amount of energy wirelessly, through the air, without having to wear radiation-proof costume ? Speaking of which, why radiation protection is still 10m of lead ? Kind of limits space travel you know.

Higgs boson, when you discover it, you know what to do with it.

This is Python, dot operator and the magic "self"

Although syntactically similar to "regular" imperative programming languages which support OOP and everything, Python offers extra semantical freedom short of being magic.

Consider you have a reference to some object, in

x

some variable. As soon as it contains a reference to an object (and it always does), you can access that object through the variable, by applying all sorts of operators to it:

x += 1
x["foo"] = "bar"
x(1, 2)
x.foo("bar")

and so on. Whether or not each of those accesses will succeed depends on the target object, but the worst thing that could happen if you mistreat an object is a runtime exception, for example:

x = 1
x()

results in

TypeError: 'int' object is not callable

(Note on samples: they are in Python3k, with Python2x theory is the same, but some of the samples may need to be slightly modified.)

Let's keep on looking. As soon as Python is an OOP-capable language (whatever on Earth that means), it supports classes and methods:

class C:
    def foo(self, x):
        print(x)

and allows overriding reaction to some of the operators, for example the following pieces of code have similar meaning:

class C:                      class C
    def __call__(self):       {
        pass            -vs-  public:
                                  void operator()(void) {}
                              };

and it might seem that there is no difference except for Python way of having a fancy double underscore method for anything advanced, but in fact Python offers more.

Python allows overriding of "dot" operator. For example, the following class (despite being a little unclean) appears to support just any method you throw at it:

class C:
    def __getattr__(self, name):
        def any_method(*args, **kwargs):
            print(name, args, kwargs)
        return any_method
    def i_exist(self):
        print("i would not budge")
c = C()
c.ping()
c.add(1, 2)
c.lookup([1, 2], key = 1)
c.i_exist()

prints out

ping () {}
add (1, 2) {}
lookup ([1, 2],) {'key': 1}
i would not budge

The magic method is apparently __getattr__, it is invoked when you apply dot operator to a class instance and it does not have such named attribute by itself, note how the i_exist method stepped up despite of having __getattr__ overriden.

x.foo
 ^---- __getattr__ is invoked when the dot is crossed

So what does it mean ? It means that you can override anything, including the dot operator, something not possible in static-typed compiled languages, and this feature makes it really simple to hide all sorts of advanced behavior behind a simple method access. For example, consider XMLRPC client in Python:

from xmlrpc.client import ServerProxy
p = ServerProxy("http://1.2.3.4:5678")
p.AddNumbers(1, 2, 3)

and see how straightforward the access to a network service with procedural interface is. ServerProxy class simply intercepts the method access and turns it into a network call. This is done transparently at runtime with no need to recompile any stub or anything - you can access any target service method without any preparation. Compare this to an XMLRPC client library of your choice.

Now take a look at the following fictional line:

foo.bar["biz"]("baz").keep.on("going")

Can you see now that every delimiter (except for literal string quoute) can be intercepted and have its behavior modified ? Given this, I can (and almost universally do) apply aesthetic thinking - how would I like my code to look ? One of the Python principles is to have code (pleasantly) readable. In each case, for each relation between program modules (whatever that means) I can have it

like["this"] -OR-
like("this") -OR-
like_this -OR-
like + "this" -OR-
like.this

and so on. Depending on the situation I can pick up whatever option that makes the code more clear. And guess what ? Overriding the dot is sometimes useful.

Anyhow, this is only half of the story.

The other half is told from the other side of the dot. See, __getattr__ notifies an instance that one of its methods is about to be accessed and allows for it to override. But Python also allows for the accessed member to be notified whenever it is being accessed as a member of some other instance. Sounds weird ? Take a look at this:

class Member:
    def __get__(self, instance, owner):
        print("I'm a member of {0}".format(instance))
        return self

class C:
    x = Member()

c = C()
c.x

prints out

I'm a member of <__main__.C object at ...>

See ? The Member instance being a member of some other class is notified whenever it is accessed. Where can it be useful you may ask ? Oh, it is the key to the magic "self" in Python.

Consider the following most simple piece of code:

class C:
    def foo(self):
        print(self)

Have you ever thought what "self" is ? I mean - it obviously is an argument containing a reference to the instance being called, but where did it come from ? It doesn't even have to be called "self", it is just a convention, the following will work just as well:

class C:
    def foo(magic):
        print(magic)

And so it turns out that somehow at the moment of the invocation the first argument of every method points to the containing instance. How is it done ?

What happens when you do

c = C()
c.foo()

anyhow ? At first sight, access to c.foo should return a reference to a method - something related to C and irrelevant to c. But it appears that the following two accesses to foo

c1 = C()
c1.foo
c2 = C()
c2.foo

fetch different things - c1.foo returns a method with its first argument set to c1 and c2.foo - to c2. How could that happen ? The key here is that you access a method (which is a member of a class) through a class instance. The class itself contains its methods in a half-cooked "unbound" state, they don't have any "self":

class C:
    def foo(self):
        pass
print(C.foo)
print(C().foo)

prints out

<function foo at ...>
<bound method C.foo of <__main__.C object at ...>>

See ? When fetched directly from a class, a method is nothing but a regular function, it is not "bound" to anything. You can even call it, but you will have to provide its first argument "self" by yourself as you see fit:

class C:
    def foo(self):
        print(self)
C.foo("123")

prints out

123

But as soon as you instantiate and fetch the same method through an instance, the magic __get__ method comes into play and allows the returned reference to be "bound" to the actual instance. Something like this:

class Method:
    def __init__(self, target):
        self._target = target
    def __get__(self, instance, owner):
        self._self = instance # <<<< binding ahoy !
        return self
    def __call__(self, *args, **kwargs):
        return self._target(self._self, *args, **kwargs)

class C:
    foo = Method(lambda self, *args, **kwargs:
                 print(self, args, kwargs))
c = C()
print(c)
c.foo(1, 2, foo = "bar")

prints out

<__main__.C object at 0x00ADA0D0>
<__main__.C object at 0x00ADA0D0> (1, 2) {'foo': 'bar'}

And so I could demonstrate a reimplementation of a major language feature in a few lines. May be not apparently useful most of the time, such experience certainly makes you understand the language better.

One more thing, have I told you Python was cool ? :)

To be continued...

This is Python, variable name lookup

As I already noted there is no declarations in Python. In general, there is no way to tell in advance what an arbitrary piece of code means, whether it is semantically correct and whether it can be successfully executed. All you have is a syntactically correct code fragment, but the meaning for any symbol is undetermined until the code is finally executed. For example,

foo = bar

is syntactically correct, but you cannot tell whether variable bar is defined at that point or what kind of an object it refers to. What behavior do you have in the above simplest assignment ? It is that if variable bar is defined, a new local variable foo will reference the same object as bar. Something like this:

current_namespace["foo"] = reference_by_name("bar")

This may be a trivial example, except for the behavior of the fictional reference_by_name function. Where does the language look up for a variable ? Like in the other languages that support procedural programming, Python procedures are natural namespace compartments. For example:

def foo(a): # begins foo's local namespace
  b = 1     # modifies foo's namespace
print(a)    # fails because a is invisible here
print(b)    # same

Each procedure's individual namespace is in Python terms called "local namespace". Namespaces of nested procedures nest along with their frames, therefore a name inside of inner procedure may refer to the variable defined in an outer:

def foo():
  b = 1
  def bar():
    print(b) # prints 1
  bar()

On the other hand, presence or absence of a name in a namespace is determined dynamically, at the moment of access, unlike static lexical scoping, which welcomes all sorts of awkward ambiguities like

def foo():
  b = 1
  del b      # would have deleted b from foo's namespace,
  def bar(): # but could not be done, because this nested
    print(b) # reference to b would hang (ouch !)
  bar()

and

def foo():                    def foo():            
  def bar():                    def bar():          
    print(b) # prints 1  -VS-     print(b) # fails because b is
  b = 1                         bar()      # only almost there
  bar()                         b = 1

Unless you want to maintain such ugly code, you should minimize using foreign variables in nested scopes, resorting to argument passing instead. Procedure arguments automatically become part of its local namespace and all locally accessed variables thus explicitly become local:

def foo():
  b = 2
  def bar(b): # explicitly local, no possible ambiguity
    print b   # prints 1
  bar(1)

Nevertheless, it is convenient to visualize the name resolution as scanning chain of nested scopes upwards:

module.py:
5) is b here ?
def foo():
  4) is b here ?
  def bar():
    3) is b here ?
    def biz():
        2) is b here ?
        def baz():
          1) is b here ?
          a = b

Note that in step 5 the containing module becomes an implicit embracing namespace which is the last chance to find the name. In Python this module namespace is called "global namespace". Finally, in addition to local and global namespaces, there is a "built-in namespace" which contains the language primitives that are not explicitly defined anywhere.

Therefore, even the simplest access to a variable is a lookup in three namespaces - local, global and built-in in that order.

To be continued...

Block-drawing characters in Firefox. WTF ?

Unicode defines a family of characters shaped like boxes of increasing height. Presumably useful for drawing diagrams in text. Something like this

x
xx
xxx

only fancier. The exact 8 characters in discussion have code points 0x2581-0x2588, and range from 1/8th to 8/8ths, i.e. full block. Here is a sample:

▁▂▃▄▅▆▇█

Now, correct me if I'm wrong, but those characters are only useful as soon as they are aligned with each other. You can't draw a diagram if one box is slightly offset - it turns out ugly. And so, can anyone tell why Firefox 2 renders the 4/8ths (half-block, code point 0x2584) and the 8/8ths (full block, code point 0x2588) shifted down a little ? Here, have a look:

This glitch makes it practically useless. WTF ?

Different ways to understand things in software engineering

The proficiency of a software developer is determined not only by which technologies he used or for how long, but more importantly by how exactly he understood and interpreted the principles behind them. Because the basic principles of software engineering are so numerous and often not specified formally, the view of the actual developer means everything.

In the course of work, a developer adapts his understanding to the problems he is working at, this is somewhat similar to how shapes of key and lock match. For this reason two people may be using the same technology for the same amount of years but be totally unable to understand each other to a point of engaging religious wars over the simplest points.

Now I understand why whenever I have a chance to interview a job applicant, I ask rather unspecific questions even of philosophical kind - to see not what he knows, but how he actually understands it and whether his understanding matches mine. Because if it doesn't we'd probably have hard times working together.

The difficult part here is trying to keep your knowledge deep and broad at the same time, because both the details and the perspective are required to understand.

The set of good programmers is still very small, a great joke by David Parnas

A reviewer explained his rejection of my best-known paper on the subject by writing, "Obviously Parnas does not know what he is talking about because nobody does it that way". Only a decade later, however, a textbook stated, "Parnas only wrote down what all good programmers did anyway". A logician would conclude that the set of good programmers was empty; that set is still very small.

-- David L. Parnas

This is Python, everything is executable

Dynamically typed language is by definition the one where variables don't have type, but the actual values do. This is by all means true in Python where the following code works fine

x = 10
x = "ten"
print(x) # prints ten

but limiting dynamism to untyped variables only would be missing the point.

Like with many "scripting" languages a Python program is started by passing the name of its main module to the "interpreter", such as

c:> python main.py

The transition of Python source code to an actually executed program begins with loading and parsing the module file. This step succeeds as soon as the module does not contain any syntax errors fatal for the parser. Successful parsing only guarantees that the module is not totally broken - a weak guarantee, only useful for checking for unbalanced parentheses and such.

What happens next is magic - the parsed module file is executed as though it was just a chunk of a source code. Wait a minute ! It is a chunk of a source code ! Anyway, execution of every module at its first import is the major part of Python program run. This process is identical no matter if the module being loaded is the program's main module or some other module explicitly imported by demand.

I have arrived to Python from C++, it took me a long time to change the perspective and the change is this - in Python you should look at everything as though it is an executable statement, because it really is. To illustrate this principle, consider definitions vs. declarations.

In statically typed languages, declarations exist for the sake of separate compilation - for the compiler to be able to tell whether one part of code is compatible with another without having to dig through the entire program. In Python, which is a dynamic language, there is no compilation stage, therefore declarations are useless, and what's left only looks like definitions.

For example, where in C++ you have two files (if you do it properly)

// foo.h                  // foo.cpp                         
class Foo                 int Foo::GetX(void) const
{                         {
private:                    return x;
  int x;                  }
public:
  int GetX(void) const;
};

the .h file is a declaration - your promise to the compiler that you will provide the matching implementation and the .cpp file is that promise fulfilled. In Python there is no compiler so you don't have to feel obliged. Identical code in Python would be

class Foo:
  def get_x(self):
    return self.x

What you see in this Python code is neither a declaration nor a definition. It is a piece of executable code, which, when executed, introduces a new class to the containing module's namespace. Rewritten to its actual effect in pseudocode it would look like this:

class Foo:       temp1 = new class()

def get_x(self): temp2 = new method()
return self.x    temp2.__code__ = return self.x
                 temp1["get_x"] = temp2

                 module["Foo"] = temp1

What you just saw was an illustration that a Python class definition is an executable statement, just like anything else and it executes once when the module is first imported. For example, it is possible to do something like C++'s conditional compilation:

class Foo:
  if os.platform == "win32":
    def do_it(self): # windows way
      ...
  else:
    def do_it(self): # unix way
      ...

The effect of the above code is that when the module is imported, the compiled version of class Foo will contain method do_it matching the current environment. It is not the same as the straightforward approach, where the check would have been performed upon each call to do_it:

class Foo:
  def do_it(self):
    if os.platform == "win32": # windows way
       ...
    else: # unix way
      ...

In a similar vein, your class definition could fail to execute:

class Foo:
  1 / 0 # this throws at import time

and the module will fail to import, throwing an exception to the caller.

Now it should not surprise you the least that when one module imports the other it is again not a declaration. When module foo does

import bar

the described process repeats for module bar, unless it has already been imported, in which case the import statement does nothing (from the discussed point of view). Similarly, you can import modules as you need them at runtime:

if need_time:
  import time
  print time.time()

Python therefore does not have any declarative semantics, only executional - ask yourself - what does it do when executed ?

To be continued...

This is Python, language installation and program structure

Installing Python is easy. If you use Windows, you have no choice at all - run setup.exe and you are done. Under Unix, Python can be preinstalled or you can install it manually. I always prefer to install from source on a clean machine, but if you have it preinstalled, you should be fine too.

Python installation is fully self-contained, and can be migrated to a different machine by copying all the files (or just the necessary ones) from c:\pythonXX or /usr/local/whatever/ to the destination. Multiple versions of Python coexist peacefully in different directories (although you should copy them around manually, because installation process registers stuff in the Windows registry and do other such things of global effect).

Python installation essentially contains the language parser+compiler and a huge and poorly structured standard library. The compiler itself along with a minimum set of libraries lives in pythonXX.dll or pythonXX.so.1, and the executable python.exe or bin/python is nothing but a simplest driver program of the (read line, execute, repeat) sort. The standard library lives in c:\pythonXX\lib + DLLs or /usr/local/lib/pythonXX/ and is just a heap of assorted utilities.

Python can be and is easily embedded into another application. It is a DLL, remember ? You take the DLL, zip the standard library and there you have it in two files - an embedded Python. In your application you create an instance of a compiler at runtime and start feeding it with stuff, that's all. Python can also be embedded into a diskless machine, it works just fine in a very restricted environment (such as the high security FreeBSD CD that I have here).

Python program consists of a set of separate modules, each module is a separate .py file containing some source code. The program is therefore available to the language in source, but Python nevertheless is not an interpreter. As each module is about to be used at runtime, it is loaded, parsed and compiled to an intermediate byte code for some virtual machine. The compiled byte code is saved alongside the original source file in an identically named .pyc file for future reuse. The outcome is the same as with Java or C# or any other language that translates source into byte code, and the difference is that in Python there is no separate compilation stage as such - the translation is performed at runtime and is in fact an important part of program execution.

To be continued...

This is Python, intro

Writing in Python for a few years now, I still get a kick of it. Wonderful and very powerful language, if used in the right way (aren't they all like that ?). No matter if code base is in megabytes, I still occasionally sit back in silence, admiring the beauty of a little code fragment or the way an idea is expressed in code.

If there is one single snippet of Python code to introduce its power of simplicity, it would be swapping of two variables. When I found it long ago in Python cookbook, it hit me like thunder. Never was my understanding of Python the same as before.

Here goes. To swap two variables in Python you need to

a, b = b, a

This utilizes Python feature called automatic tuple packing/unpacking. What's actually going on is more like

a, b <<< unpacked <<< (b, a) <<< packed <<< b, a

where (b, a) is a Python notion for an immutable sequence called tuple.

To be continued...

Slow pace of software development

If you need it fast, make sure it *looks* good, because it won't be. Pay more attention to high quality advertisement than to high quality development, but note that once you start selling promises, you will unlikely deliver a product.