Method signature type checking decorator for Python 3000

I have just published a Python 3000 decorator for method signature type checking using function annotations. Here:

http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/572161

It is much cleaner that the similar decorator I have previously written for Python 2.x, the used Python 3000 function annotations make it better for the following reasons:

1. The signature-related piece of syntax is right there where it belongs - next to the parameter. Where I used to write

@takes(int, str)
@returns(bool)
def foo(i, s):
    ...

I now write

@typecheck
def foo(i: int, s: str) -> bool:
    ...

2. I don't have to add checking to all the parameters simply because there was no way to skip one. Where it was

class Foo(object):
    @takes("Foo", str)
    def foo(self, s):
        ...

it is now

class Foo:
    @typecheck
    def foo(self, s: str):
        ...

3. It plays nicely with the default values. This one has no equivalent in 2.x version, but it is nice to have:

@typecheck
def foo(x: int = 10): # 10 is also checked
    ...

@typecheck
def foo(*, k: optional(str) = None):
    ...

Other than that, it is just a nice usable piece of code, extensible too. Here is a few more examples:

@typecheck
def foo(x: with_attr("write", "flush")):
    ...

@typecheck
def foo(*, k: by_regex("^[0-9]+$")):
    ...

@typecheck
def swap_tuple(x: (int, float)) -> (float, int):
    ...

@typecheck
def swap_list(x: [int, float]) -> [float, int]:
    ...

@typecheck
def extract(k: list_of(by_regex("^[a-z]+$")),
            d: dict_of(str, int)) -> list_of(int):
    ...

Approaching Python 3000: no more automatic tuple parameter unpacking

Just keep in mind that

lambda (x, y): x + y

is no longer possible in Python 3000. You are supposed to write

lambda x_y: x_y[0] + x_y[1]

instead. This also applies to functions, not just to lambdas and the idea belongs to PEP 3113 which forbids automatic tuple unpacking in function parameters. Ugly and inconvenient if you ask me, but there apparently was somebody who kept shooting himself in the leg.

I think that automatic unpacking was rather useful (if used sparingly), especially when you had to do something like

map(lambda (i, (a, b)): i * (a + b),
    enumerate(zip(aa, bb))

which is now what ?

map(lambda i_a_b: i_a_b[0] * (i_a_b[1][0] + i_a_b[1][1]),
    enumerate(zip(aa, bb))

Eeew...

Approaching Python 3000: function annotations

Python 3000 introduces function annotations, regulated by PEP-3107. This one is of particular interest to me as I've previously written one of the method signature checking decorators which this PEP is supposed to replace.

PEP-3107's has two major points:

1. Annotations can be anything. Any Python expression can be attached to a function argument or result value. For example, it is possible to write

def max(a: int, b: int) -> int:
def max(a: "first", b: "second") -> "maximum":
def foo(x: { ("no", "reason"): lambda x: x**2 }):

2. Annotations have no semantics and are not enforced, they are purely syntactical. For example, it's ok to write

def max(a: int, b: int) -> int:
...
max("foo", "bar") # nothing happens

The interpretation of annotations is left to 3rd party libraries. The language thus offers an unprecedented semantical freedom to the developers, but let's see what are the implications.

One problem is that you will have to choose your one true annotations. Specific interpretation of function annotations depends upon external module, library or application, and you acknowledge this dependence explicitly by either modifying your code or having it processed by external application.

For example, you could have chosen to use function annotations for method signature type checking, using typecheck decorator (resembling my type checking decorators)

@typecheck
def foo(a: int) -> int:

but once you have chosen the @typecheck implementation you have to stick with it and treat all your function annotations as type checks. Stacking multiple annotations is technically possible, but practically it is not, because the standard does not specify how multiple annotations should be multiplexed. Consider that you have

@typecheck
def foo(i: int):

in place and want to add a docstring kind of annotation to foo's first argument:

def foo(i: "comment"):

Should it be

def foo(i: (int, "comment")):

or

def foo(i: {"typecheck": int, "docstring": "comment"}):

?

No matter which way you choose, both typecheck and docstring must be prepared to extract their annotations from the actually encountered multiplexed construct. This means that two independent implementations must understand the same multiplexing format. Since such multiplexing is not standardized, it is impossible.

One seemingly reasonable way of such multiplexing could have been an iterable with instances of classes descended from some base class (Annotation?) for example

def foo(i: (typecheck(int), docstring("comment"))):

This example may be correct but it perfectly illustrates how having multiple semantically different annotations seriously hamper the visual quality and readability of the code.

Which opens a final question of whether the gains of the only particular annotations that you choose outweigh the loss of syntactical brevity.

Approaching Python 3000: string formatting

One of the changes in Python 3000 applies to string formatting. PEP-3101 is the regulating document for the change. It basically says that the regular % operator is too limited and what we need is a powerful domain-specific language for string formatting.

To be frank, I never felt limited with what % had to offer, but there is no point in criticizing what's about to become standard. Let's see what's new:

1. What used to be a binary operator is now a str's method:

"{0}, {1}".format("A", 10) == "A, 10"
"{n} = {v}".format(n = "N", v = "V") == "N = V"

2. Formatting and alignment is applied in the same manner as before:

"{0:03d}".format(10) == "010"
"<{S:>5s}>".format(S = "foo") == "<  foo>"
"<{S:<5s}>".format(S = "foo") == "<foo  >"

3. Format can insert not just parameter values, but also their items and/or attributes:

d = dict(foo = 1, bar = "a")
"{0[foo]}, {0[bar]}".format(d) == "1, a"
"{0.__class__.__name__}".format(d) == "dict"

4. Recursive substitution is allowed to a degree, for example this works:

d = dict(value = 10, format = "03d")
"{0[value]:{0[format]}}".format(d) == "010"

but this doesn't:

d = dict(data = {"a": "A", "b": "B"}, key = "a")
"{0[data][{0[key]}]}".format(d)

5. Classes can control their own formatting:

class Foo():
def __format__(self, format):
    from re import match
    assert match("[0-9]+s", format)
    return "x" * int(format[:-1])
foo = Foo()

"{0:3s}".format(foo) == "xxx"
"{0:10s}".format(foo) == "xxxxxxxxxx"

I agree, the new way of string formatting (1 and 2) is cleaner and more straightforward. Substituting items and attributes (3) could be useful sometimes. Custom formatting (5) is Pythonic all right, but hardly practically useful, except when building a framework, a class framework perhaps.

What I don't buy is the attempt to make a statement from what is supposed to be an expression (4). If it is inconsistent, difficult to read and not apparently useful, it should not be there.

A programming language made of smileys

Just imagine the possibilities of a language whereby programs are constructed from smileys:

• Easy to write, all language objects can be arranged on a toolbar. No more typos !
• Easy to read. You get an immediate emotional response by simply looking at the code.
• Appealing to a programmer of any nationality. No internationalization required !
• Easy to extend by adding smileys with hammers, flowers or database connectors.
• Fun to work with !

The great effects of little imperfections

Observing a chain of bubbles raising through a column of water I found it simply amazing how the same little fluctuations in water



that actually made the bubbles appear in the first place also make sure that they go straight up and don't deviate as they raise.

Re: Software Engineering Programs Are Not Computer Science Programs

Written by David Lorge Parnas the article under that title was published in "IEEE Software", Nov/Dec 1999, and essentially says that computer science and software engineering need to be separated in the same way as theoretical physics is separated from its related engineering fields. For the sake of both. As far as the education goes at least.

He also advocates the mandatory accreditation of software engineering programs and points out the problems to be encountered. Among the problems mentioned by the author are the lack of knowledge how to teach and experienced staff.

Frankly, the article was a little tedious to me, biased by the magazine specifics perhaps. But just like the other works of this great man that I had a chance to read, it is truthful and inspirational. Although in the case of this article, the inspiration has driven me in a slightly unexpected direction.

And so I would like to criticize the article on the grounds that the analogy between physics/regular engineering and computer science/software engineering does not hold.

First, there is a historical difference. Between physics and its engineering fields, it all began with practice and experiment. The extreme case would be construction - people have been prototyping since probably 50 thousand years ago. Two thousand years ago selected craftsmen have already mastered wood, stone and iron construction. There was neither science nor engineering at that moment, all they had was observation and experience.

I am not an expert in history of science, but it seems plausible that same pattern repeated most of the time - experiments came first and the theory followed. To be sure, physics as a science is far ahead now setting up experiments that only a few understand, but at least at early stages practical considerations have prevailed.

Exactly opposite is true for computer science. What began as pure mathematical theory in 1930s couldn't even be supported by experiment until a decade later when some sort of electrical apparatus has been constructed. In fact, being a branch of mathematics, computer science didn't have to be supported by experiments in the first place.

The "mathematical" engineering therefore was not something that anyone practically required. All they needed was to speed up the calculations, and I seriosly doubt that anyone could see the consequences. As the story has it, at one time IBM predicted the computer world market to be in tens of installations. If it wasn't for semiconductors, software engineering wouldn't even be here today, but computer science would.

Over time, software engineering became an awkward crossover between mathematics and psychology, where people try to project mathematical abstractions onto real world. Remember how Knuth said: "I have only proved it correct, not tried it." The matter dealt with in software engineering is thus something that should work because it is theoretically perfect but doesn't work because we are not practically perfect.

Second, there is an economical difference. Software is intangible, software production does not respect political borders, it can easily be and is routinely outsourced. What would you say if a team of construction workers could fly from India with its own tools and materials to raise a house overnight ? Plus they would charge less and still get the job done with satisfactory quality. And they would not need to be certified. Similarly, if a doctor or a lawyer could consult over the Internet from a different country, and his services were just as good, wouldn't that nullify certification efforts ?

Besides, you can't strictly control telecommutable industry, you try to lock it down by regulations and it goes underground. And the last thing we need is the black software market in addition to a pirated software market. Besides, such regulatory inhibition of software engineering would hamper the scientific progress and thus have exactly the opposite effect to the desired.

You may try to enforce mandatory certification of products instead, but this brings in a totally different perspective and requires a definitive procedure of software quality assessment - something at least improbable at this moment.

Third, there is a natural difference. There is no laws of nature in software engineering.

Try hard as you may, you cannot build a house which levitates above the ground. Because physics provides its engineers with absolute laws - such as energy conservation law, thermodynamics laws or Newton laws. They all may be a reflection of some deeper principles, but in practice it is sufficient for an engineer to know that you are limited in energy and can't fight gravity. And this is not because a scientist said so, but because you simply can't.

Not the case with software. The world in which software lives is only restricted with hardware architecture. Von Neuman is literally the god of software and computer scientists are his prophets. But what absolute laws do they give to their poor engineers ?

None.

The hardware has its restrictions, that's true, but it is all in capacity. It is physics that limits the hardware, the computer science does not impose any restrictions above that. It is as though it was possible to build a house with the only restriction in mind - that its size should not exceed that of a planet. You can even start building from the roof, and it doesn't have to touch the ground when it's done. It's all imaginary.

The lack of unbreakable laws leaves all the arguments about how software has to be build to a degree open-ended. But then, how could you certify an industry in which there is still no consensus on how to do the simplest thing ?

To conclude, I believe that computer science and software engineering are indeed different, so different in fact, they can be treated as totally unrelated. But the relationship between them is not the same as with physics and engineering, and it would be wrong to approach it with established (educational) practice.

There are unnecessary details

but there are no unimportant details.

What is the word LOVE associated with these days ?

SPAM !!!

All we have is less good programmers

The more I learn about programming, the more I want to ask: "why haven't I been taught this before ?". I mean - I graduated from a university, majored in "computer mathematics" or so it said, but I really got nothing useful from professional point of view. A few theoretical courses, such as graph theory is all. As the matter of fact, everything I know about programming I've learned from books and hard work.

The sad truth is that each generation takes a fresh start. I recall how confident I was in having known everything. May be it happens all the time, but programming is special because there is still no notion of software quality. It is surprisingly difficult to convince a beginning programmer that his program is bad. Because you simply have no reliable judgement basis except for your own expert opinion, but then what would you know ?

But then, there is no knowledge transfer and the entire industry is doomed to go around in circles, reinventing the same things every ten years or so, under different names.

I do realize that the software industry didn't get any better in the past decades, it even might have gotten worse by all accounts. The only reason why we could have possibly gotten more good programmers is because there simply appeared more just any programmers, because anyone could perform as one. Therefore it is statistically possible that the upper percentile also got more numerous.

It would also seem a valid guess that it becomes more and more difficult to find good programmers. Because the good ones tend to stick with a company, or a project, or a team, and bad ones may be changing places more often. This makes the problem of creating a strong team more difficult, and your typical team would in general be of lesser quality. As I believe that a strong team is the best thing that could happen to a programming project, this observation leaves even less hope in the future.