On death march

With every commit, a question "the right way, or the usual way ?" sounds more and more sarcastic.

On improving movie-going experience

Do you know how people sometimes say that certain movies you just cannot watch without having a beer or two before ? So why not having alcohol bottled in different sizes and shapes, labeled with names of actors and/or directors and given out according to movie cast ? So that if you come to see "Suicide squad", you'd get yourself a large "Will Smith" beer, a small "Jared Leto" beer, and a shot of "Comix adaptation" vodka.

We have lost

I used to know every file on my computer. I used to open every single file in a hex editor to see what's inside. Actually, this is still my first reaction to understand what a file is.

In the programs that I wrote I used to know every line, every character even. Then they got bigger, but I still knew every file. Then they got bigger yet but at least I knew all the dependencies not only by name but by virtue.

Now a static single page web site in React comes with over 100000 files, some of which, as you know, are left pads and hot pockets and there are all new contests. Such complexity is beyond anyone's reach. Even the authors find it overwhelming.

We have lost the understanding of what's happening.

Today's software development is not engineering as such. Given the quality of the outcome it is not even a professional activity. Software development has become a recreational game for the young. The target audience for the programmer became not the users but other programmers. Stackoverflow and Github saw to that. This is now the most active social network in the world.

To impress one's peers it's no longer necessary to build some quality software. Bah ! You can't even tell what quality means. But to ride bicycle without hands - that's something ! And if you could do it blindfolded ! And backwards !

And so we see thousands of exercises in making things backwards. Without understanding the purpose or the reason, pick up a new tool, play for a month then move on to a new stimuli. Worse yet if it leaves behind another backwards-backwards-backwards-backwards tool. This adds another layer of useless and not understood complexity and provides positive feedback to the loop.

I remember well one day in 2005, when something out of the ordinary happened. At the time I was working under supervision of a great software engineer. He was always talking about "architecture", you know. Back then I didn't understand it at all, despite having already worked as a programmer for some 8-9 years. I thought it was all managerial talk. And then I was sitting in a conference room alone thinking how to organize a UI for some application and it dawned on me. I knew what architecture was, not burdened with details, my mind went to the next level, it was almost like I could fly. That I could not forget or unlearn, and I'm happy that this knowledge is with me, because it would not have happened today.

Today I would have just be dabbling in a Sea Of Complexity, pleasing my mind with details. May be I would have been happy about that, who knows.

On cookie consent

In case you didn't know what a "cookie consent" is, behold - according to EU legislation, web sites must ensure the user accepts their usage of cookies.

And I'm sick and tired of its effects.

It is an outstanding example of what happens when people that don't have the faintest come to control technology, in this case the Internet. For each web site to prompt the user about cookies is a terrible idea.

1. The users don't understand it. They don't know what cookies are or how they work, to them it is meaningless noise. It is not a consent, it is just another "some technical message, just click OK" moment. Also, the word "cookie" itself adds a little touch of insanity.

2. The users don't take it seriously. Even when the warning is straightforward (we need cookies to do something you may not like) it is a matter of a single click to close the annoying window.

3. It does not improve privacy. At all. From privacy standpoint, cookies are not the villains but the most innocent messengers that are being shot.

4. It makes the Internet more stressful. As if we had not enough banners, one-time offers, subscription popups, landing pages, paywalls and so on, now we have these noisy popups.

5. Technically, cookie consent is a catch-22 situation - to know whether to accept a cookie from a site you need to own a cookie from it. Therefore if you refuse, the site will ask again. Moreover, even if you accept, each browser on each device manages its own cookies, and only a limited number of them. So the questions will continue ad nauseam.

Python function guards

TL;DR: Rationale and thoughts behind the implementation of Python 3 function guards.

I really love Python, but unfortunately don't have to use it in my current daily job. So now I have to practice it in my spare time, making something generally useful and hopefully suitable for improving my Python application framework.

1. The idea

I already had a method signature checking decorator written years ago, and it turned out enormously useful, so in the same line I started thinking about whether it would be possible to implement declarative function guards that select one version out of many to be executed depending on the actual call arguments. In pseudo-Python, I would like to write something like this:

def foo(a, b) when a > b:
  ...

def foo(a, b) when a < b:
  ...

foo(2, 1) # executes the first foo

2. Proof of concept

At the first sight it looks impossible, because the second function kind of shadows the second one:

def foo(a, b):
  print "first"

def foo(a, b):
  print "second"

foo(2, 1) # second

but this is not exactly so. Technically, the above piece of code looks something like this:

new_function = def (a, b): print "first"
local_namespace['foo'] = new_function

new_function = def (a, b): print "second"
local_namespace['foo'] = new_function

and so the problem is not the function itself which is overwritten, but its identically named reference entry in current namespace. If you manage to save the reference in between, nothing stops you from calling it:

def foo(a, b):
  print("first")

old_foo = foo

def foo(a, b):
  if a > b:
    old_foo(a, b)
  elif a < b:
    print("second") 

foo(2, 1) # first
foo(1, 2) # second

so there you have it, what's left is to automate the process and it's done.

3. Syntax

There is no question as to how the guard should be attached to the guarded function - it would be done by terms of a decorator:

@guard
def foo(): # hey, I'm now being guarded !
  ...

@guard
def foo(): # and so am I
  ...

but the question remains where the guarding expression should appear. I see six ways of doing it:

A) as a parameter to the decorator itself:

@guard("a > b")
def foo(a, b):
  ...

B) as a default value for some predefined parameter:

@guard
def foo(a, b, _when = "a > b"):
  ...

C) as an annotation to some predefined parameter:

@guard
def foo(a, b, _when: "a > b"):
  ...

D) as an annotation to return value:

@guard
def foo(a, b) -> "a > b":
  ...

E) as a docstring

@guard
def foo(a, b):
  "a > b"
  ...

F) as a comment

@guard
def foo(a, b): # when a > b
  ...

Now I will dismiss them one by one until the winner is determined.

Method F (as a comment) is the first to go because implementing it would require serious parsing, access to source code and be semantically misleading as the comments are treated as something insignificant which can be omitted or ignored. The rest of the methods at least depend on the runtime information only and work on compiled modules.

Method A (as a parameter to the decorator) looks attractive, but is dismissed because it moves the decision from the function to the wrapper. So the function alone can't have guard expression and therefore it would not be possible to separate declaration from guarding:

def foo(a, b): # I want to be guarded
  ...
# but it is this guard here that knows how
foo = guard("a > b")(foo)

The rest of the methods are more or less equivalent and the choice is to personal taste. Nevertheless, I discard method E (docstring) because there is just one docstring per function and it has other uses. Besides, to me it looks like it describes the insides of the function, not the outsides.

So the final choice is between having the guarding expression as annotation and as default value. The real difference is this: a parameter with a default value can always be put last, but a parameter with annotation alone can not:

def foo(a, b = 0, _when: "a > b") # syntax error
  ...

This and the fact that aforementioned typecheck decorator already makes use of annotations tips the decision towards default value:

@guard
def foo(a, b, _when = "a > b"):
  ...

@guard
@typecheck 
def foo(a: int, b: int, _when = "a > b") -> int:
  ...

The choice of a name for the parameter containing the guard expression is arbitrary, but it has to be simple, clear and not conflicting at the same time. "_when" looks like a reasonable choice.

4. Semantics

With a few exceptions, the semantics of a guarded function is straightforward:

@guard 
def foo(a, b, _when = "a > b"):
  ...

@guard 
def foo(a, b, _when = "a < b"):
  ...

foo(2, 1) # executes the first version
foo(1, 2) # executes the second version
foo(1, 1) # throws

Except when there really is a question which version to invoke:

@guard 
def foo(a, b, _when = "a > 0"):
  ...

@guard 
def foo(a, b, _when = "b > 0"):
  ...

foo(2, 1) # now what ?

and if there is a default version, which is the one without the guarding expression:

@guard
def foo(a, b): # default
  ...

@guard
def foo(a, b, _when = "a > b"):
  ...

foo(2, 1) # uh ?

and the way it seems logical to me is this: the expressions are evaluated from top to bottom one by one until the match is found, except for the default version, which is always considered last.

Therefore here is how it should work:

@guard
def foo(a, b):
  print("default")

@guard 
def foo(a, b, _when = "a > 0"):
  print("a > 0") 

@guard 
def foo(a, b, _when = "a > 0 and b > 0"):
  print("never gets to execute")
 
@guard 
def foo(a, b, _when = "b > 0"):
  print("b > 0")

foo(1, 1)   # a > 0
foo(1, -1)  # a > 0
foo(-1, 1)  # b > 0
foo(-1, -1) # default

5. Function compatibility

So far we have only seen the case of identical function versions being guarded. But what about functions that have the same name but different signatures ?

@guard
def foo(a):
  ...

@guard
def foo(a, b):
  ...

Should we even consider to have these guarded as versions of one function ? In my opinion - no, because it creates an impression of a different concept - function overloading, which is not supported by Python in the first place. Besides, it would be impossible to map the arguments across the versions.

Another question is the behavior of default arguments:

@guard
def foo(a = 1, _when = "a > 0"):
  ...

@guard
def foo(a = -1, _when = "a < 0"):
  ...

Guarding these as one could work, but would be confusing as to which value the argument has upon which call. So this case I also reject.

What about a simplest case of different names for the same positional arguments ?

@guard
def foo(a, b):
  ...

@guard
def foo(b, a):
  ...

Technically, those have identical signatures, and can be guarded as one, but is likely to be another source of confusion, possibly from a mistake, typo or a bad copy/paste.

Therefore the way I implement it is this: all the guarded functions with the same name need to have identical signatures, down to parameter names, order and default values, except for the _when meta-parameter and annotations. The annotations are excused so that guard decorator could be compatible with typecheck decorator. So the following is about as far as two compatible versions can diverge:

@guard
@typecheck
def foo(a: int, _when = "isinstance(a, int)", *args, b, **kwargs):
  ...

@guard
@typecheck
def foo(a: str, *args, b, _when = "isinstance(a, str)", **kwargs):
  ...

Note how the _when parameter can be positional as well as keyword. This way it can be always put at the end of the parameter list in the declaration.

6. Function naming

Before we used simple functions, presumably declared at module level. But how about this:

@guard
def foo():
  ...

def bar():
  @guard 
  def foo():
    ...

class C:
  @guard 
  def foo(self):
    ...

those three are obviously not versions of the same function, but they are called foo() so how do we tell them apart ?

In Python 3.2 and later the answer is this: f.__qualname__ contains a fully qualified name of the function, kind of a "a path" to it:

foo
bar.<locals>.foo
C.foo

respectively. It doesn't matter much what exactly is in the __qualname__, but that they are different, just what we need. Prior to Python 3.3 there is no __qualname__ and we need to fallback to a hacky implementation of qualname.

7. Special cases

Lambdas are unnamed functions. Their __qualname__ has <lambda> in it but no own name. They would be impossible to guard:

foo = lambda: ...
foo = guard(foo)

bar = lambda: ...
bar = guard(bar)

because from the guard's point of view they are not "foo" and "bar", but the same "<lambda>".

An interesting glitch allows guarding classmethods and staticmethods. See, classmethod/staticmethod are not regular decorator functions but objects and therefore cannot be stacked with guard decorator

class C:
  @guard # this won't work
  @classmethod
  def foo(cls):
    ...

because classmethod can't be seen through to the original function foo. But it gets interesting when you swap the decorators around:

class C:
  @classmethod
  @guard
  def foo(cls, _when = "..."):
    ...
  @classmethod
  @guard
  def foo(cls, _when = "..."):
    ...

the way it works now is that guard decorator attaches to the original function foo, before it's wrapped with classmethod. Therefore the guarded chain of versions contains only the original functions, not classmethods. But when it comes to the actual call to it, it goes through a classmethod decorator before it gets to guard, the classmethod does it argument binding magic and whichever foo is matched by guard to be executed, gets its first argument bound to class as expected.

8. The register

Here is one final question: when a guarded function is encountered:

@guard
def foo(...):
  ...

where should the decorator look for previously declared versions of foo() ? There must exist some global state that maps function names to their previous implementations.

The most obvious solution is to attach a state dict to the guard decorator itself. The dict would then map (module_name, function_name) tuples to lists of previous functions versions. This approach certainly works but has a downside, especially considering I'm going to use it with Pythomnic3k framework. The reason is that in Pythomnic3k modules are reloaded automatically whenever source files containing them change. Having a separate global structure holding references to expired modules would be bad, but having a chain of function versions cross different identically named modules from the past would be a disaster.

There is a better solution of making the register slightly less global and attach the state dict to the module in which a function is encountered. This dict would map just function names to the lists of versions. Then all the information about the module's guarded functions disappear with the module with no additional effort.

9. Conclusion

The implementation works.

I'm integrating it with Pythomnic3k framework so that all public method functions are instrumented with it automatically, although it is tricky, because when you have a text of just a

def foo(...):
  ...
def foo(...):
  ...

and you need to turn it into

@guard
@typecheck
def foo(...):
  ...
@guard
@typecheck
def foo(...):
  ...

it requires modification of the parsed syntax tree. I will have to write a follow-up post on that.

That's all and thanks for reading.

Rent-a-battery ?

Why is there a service where you can instantly rent a car but no such service where you can instantly rent a precharged cell phone battery ? Every time your cell phone battery runs out it is a huge pain to find a charger and/or socket to plug it in. Wouldn't it be nice to have an over-the-counter rent-a-battery at every cashier ?

On Emoji

I'm having hard times grasping this originally Japanese thing. The word rhymes with "emotion", and although in Japanese it had no such meaning, it apparently was intended to express a notion, feeling or emotion in a single hieroglyphic depicting some character or other. It just so happened that after some time it became a (Unicode) standard alternative to smileys and other pictographs.
 
Speaking of smileys, I may have an emotional range of a teaspoon, but I can't tell what most of those emoji faces mean. Each time I pull up the emoji palette in an application, I'm always stuck at which to pick, despite of seemingly wide choice. They don't convey any emotion I can possibly want to express. Don't get me wrong, they may be perfectly suited to express a notion of a pile of shit but this is not what I need from an emotion. And even with faces, wtf ?

For the purpose of illustration I've picked a few, but you can imagine the rest. Here, see for yourself, and mind that it is an international standard, no less.

So I believe the entire emoji thing was a designer's experiment, a hip toy, which was backed by the Unicode consortium in order to keep filling their seemingly endless code pages, and from there application developers picked it as a "standard" way smileys should be done. Quite unfortunate really.

Please, PLEASE, use something like Kolobok, or hire a designer and at least make your smileys look like Skype's.

Монти Механик

Вот так, прослушав сотню случайных групп, наткнешься на нечто выдающееся. Монти механик - группа из Челябинска (что тоже приятно - соседи), играющая музыку, так ложащуюся на душу.

Helium shoelaces

Shoelaces should really be lighter than air, so that they float up and you never step on them. Inflatable with helium most likely.

Rethinking the cache

Caching is a very difficult problem. Not just invalidation and not because of some inherent algorithmic complexity, but because of the complex semantics of the process and its purpose of increasing performance.

In one of our products I had once written a specific application mini-ORM for caching objects persisted in a database. It was a Python 3 application in Pythomnic3k framework and a companion PostgreSQL database.

The database structure was really simple - one table per class named simply like "users" or "certificates" and a handful of stored procedures named like "users__load" or "certificates__save". The tables contained the actual data and a few special fields, like "checksum", which allowed to detect concurrent update conflicts optimistically.

So each time there was an ORM access in the code

user = pmnc.db_cache.User(user_id = 123)
user.passport_number = "654321"
user.save()

what happened behind the scene was that ORM implementation in db_cache.py executed

SELECT *, checksum FROM users__load(123)

then created an instance of class User based on returned data, one property per database column, cached the instance and returned it to the caller. After the passport_number property was modified the call to save executed

SELECT new_checksum
FROM users__save(123, ..., '654321', ..., checksum)

to flush changes to the database. Should another request for user 123 arrive in the meantime

user2 = pmnc.db_cache.User(user_id = 123)

it would have returned the cached instance and not go the database.

As any ORM, this one did not answer all the questions. It did not allow ad-hoc queries which did not map directly to object paradigm and it was impractical to create a separate method for every request. Therefore, ad-hoc queries started finding their way directly to the application code like

pmnc.transaction.db.execute(
"
  SELECT u1.last_name, u2.last_name
  FROM users u1 INNER JOIN users u2
  ON u1.passport_number = u2.passport_number
  WHERE u1.suspended = {suspended}
",
suspended = True)

So now there were requests that bypassed the ORM cache and went straight to the database. This was still normal as soon as they were read-only. But later there were more of them and they were getting more analytical and heavy (think history of changes of all objects belonging to a particular user), therefore a question of caching appeared again.

It was then that I implemented and released a universal read caching mechanism for Pythomnic3k (version 1.4), so that it was possible to enable cache on any resource, database being the most probable candidate of course. I wanted to put it to production and make all the requests including ORM's go through the resource cache. The first thing that surfaced immediately was that read caching implementation was pretty much useless as it was, because while caching reads it did not take into account the existence of concurrent writes.

Actually, I knew this before the release, but simply had not enough understanding of how such concurrent cache activity should behave. But I left a couple of callable hooks that allowed to customize the cache behavior on application level. So I hooked into the cache and made reads and writes coexist. Because the cache couldn't tell whether such and such SQL request had side effects, the fact had to be declared by the application with each call. In simple terms a read like this

pmnc.transaction.cached_db.execute(
"
  SELECT *, checksum FROM users__load({user_id})
", 
user_id = 123, 
pool__cache_read_keys = { "users/123" })

would later have its cached result invalidated by conflicting write

pmnc.transaction.cached_db.execute(
"
  SELECT new_checksum FROM users__save({user_id})
", 
user_id = 123, 
pool__cache_write_keys = { "users/123" })

This way it went to production and worked fine for about half a year. Until one fine after-deployment morning it didn't.

There was another thing not taken into account, a race condition between reads and writes. For example, if these two conflicting request executed concurrently:

pmnc.transaction.cached_db.execute(
"
  SELECT WHERE user_id = 123
")
pmnc.transaction.cached_db.execute(
"
  UPDATE WHERE user_id = 123
")

there was a chance that the read would start before the write but end after the write. In this case write wouldn't invalidate the result of read simply because there was none at the moment, but the result would still arrive and be cached containing already invalid data. The problem was resolved by patching in an industry-standard sleep() in the right place, and it indeed remedied the situation. But now I started to rethink the entire thing. Clearly, caching semantics needed to be improved.

So I went and made a lot of changes to the cache code, using new experience, focusing on concurrent reads and writes behaviour. In particular, the above race condition was fixed by registering affected cache keys before any request, read or write, is sent to the database. This way if a write arrives when a conflicting read is in progress or the other way around, both are allowed to proceed but read result is not cached when read returns. The result is still returned to the caller and it may or may not be invalid but now it is the responsibility of the database, not the cache, so we did not break the semantics.

Now, as I was overhauling the cache anyway, I also wanted to examine the evidence.

First, I picked up what log files with enough debug information we had from production installations of our product and read through the registered SQL queries. Predictably enough, they fell into two categories:

1. Ad-hoc queries. Fewer but slower.
2. ORM queries. A lot more numerous but also much faster.

Second, I checked the cache settings on database connections and they were not right. The cache sizes were too small and the "weight" eviction policy exclusively favored the requests of the first type, which produced less hits. By weight we mean the time it took to originally produce the actual value from the database.

So I thought it would be nice to improve the cache by accounting not only weight of the cached result, but also "usefulness", which would be weight multiplied by hit count. And so I added such eviction policy, only it was called "useless", as in "evict useless entries first".

Some time later I thought that since ORM produces a lot of identical parametrized queries:

SELECT * FROM users__load(?)

it would be reasonable to suggest that each entry cached from any such request has the same average usefulness. For example, if we have 1000 entries cache hits to which saved 1 second each, and there is a 10 new entries that have just been entered and did not have a chance yet, the newcomers should not be evicted right away. It would be better to let them stay hoping that each will come as helpful as the 1000 before them.

Therefore I added an optional "cache group" parameter for a query, the simplest kind of one is the literal SQL string itself. As entries produced by the same SQL string are entered to the cache, they are assigned to the same cache group and have their usefulness accounted combined. Even though the new entry may not have had any hits yet, it is under the umbrella of the high-ranked group with high average saving.

Eviction now had to work differently. At first I though that I would simply evict the low-ranked groups first, leaving the high-ranked ones intact if possible. But experiments indicated that one winning group simply took over the entire cache over time. So I had to implement eviction using weighted average amongst groups, where a high-ranked group has a better multiplier than a low-ranked one. This means that a value from a former group could still be evicted if it has low weight, and likewise a high weight value from a latter group could stay.

Fiction mode off. I have just committed all this code to Pythomnic3k SVN repository, and it will be in the next version, so anyone who is interested may check it out, the cache should now be usable. Although making it work right in an application may not be obvious, I will later include a sample specifically with cached access to a database.

How to make PF redirect traffic to localhost

Here in the company I work for, I also perform as a network administrator. Router, firewall, office VPNs, network segments, switches, cabling, that sort of thing.

The difficult part is that we have two separate internet connections through different providers, the perimeter router hence has two external IP addresses and three routing tables.

And we obviously want to

1. Spread outgoing traffic evenly through each provider.
2. Pass each provider's private networks through the owner.
3. Whenever either provider is down, pass the traffic through the other.
4. Redirect some incoming traffic to selected servers in the DMZ.
5. Redirect some other incoming traffic to the router itself.

I've had it all set up once four years ago using FreeBSD, ipfw and natd and it's been working ever since. The problem is maintenance. Having changed over time, ipfw ruleset got huge and unreadable, natd is troublesome to reconfigure and switches to userland consume CPU.

Now as the company is growing, we have new offices, new services, new hardware, and so I thought that rather than trying to shoehorn the changes, I'd do it over again using something else instead of ipfw/natd. For example, pf.

After reading man and a couple books it all seemed clear. Some things I indeed worked around easily. But not this one.

Say I want to redirect incoming internet traffic from either provider to a service running at the router itself listening at localhost. This should be something like

nat on ext -> (ext)
rdr on ext from any to (ext) tag FOO -> localhost
pass in on ext tagged FOO

Right ? Well, if ext is your only internet connection, yes. Otherwise you have ext1 and ext2, each having its own default gateway and it bites you. See, if ext1 is 1.1.1.1, ext2 is 2.2.2.2, a SYN packet arrives over ext1

ext1: 1.2.3.4 -> 1.1.1.1

it goes through the rdr rule first and becomes

ext1: 1.2.3.4 -> 127.0.0.1 tagged FOO

before it is seen by the pass rule. The routing information has been lost. Now even if the service responds with SYN/ACK, where should it be sent to ? Ideally we would need something like

rdr on ext1 from any to (ext) rtable 1 tag FOO -> localhost
rdr on ext2 from any to (ext) rtable 2 tag FOO -> localhost

to attach routing to a state as early as possible. Unfortunately, using rtable with rdr is not supported in FreeBSD 8. You can use rtable with pass, but it is too late a point.

My first reaction was to tag each provider's incoming traffic differently and play from there:

rdr on ext1 from any to (ext1) tag EXT1 -> localhost
rdr on ext2 from any to (ext2) tag EXT2 -> localhost
pass in on ext1 tagged EXT1
pass in on ext2 tagged EXT2
pass in on dmz route-to (ext1 gw1) tagged EXT1
pass in on dmz route-to (ext2 gw2) tagged EXT2

The trick here is that we intercept the packets from the service response at re-entry, they are automatically attached to an already tagged state and therefore we can determine where to route them. And it works, but not with loopback (note that I've used dmz for interface name). Specifically for the loopback interface the response packets avoid filtering, the

pass in on lo0 route-to (ext1 gw1) tagged EXT1

rule never fires and the matching packets go directly to routing. It doesn't work.

After some obligatory hair pulling, I've come up with the following working ruleset:

set skip on lo0
nat on ext1 -> (ext1)
nat on ext2 -> (ext2)
rdr on ext1 from any to (ext1) tag FOO -> localhost
rdr on ext2 from any to (ext2) tag FOO -> localhost
pass in on ext1 reply-to (ext1 gw1) tagged FOO
pass in on ext2 reply-to (ext2 gw2) tagged FOO

And an ever important addition to this:

route add default -iface lo0

Now, I have no idea why localhost needs to be specifically routed to lo0, but you can google up "route-to lo0" for a bunch of posts similar to mine. Without it the redirected packets somehow can't reach localhost. But as we already have reply-to clause on a pass rule, we cannot also include route-to (which is yet another obstacle). There appears no other way but to set a default route to lo0. It even makes sense, because on this machine I treat localhost as publicly accessible.

As a bonus this ruleset also works for redirected connections that reach out to DMZ, which is enough to keep me happy for the moment.

On interrupting application code

So, I've released a new version of Pythomnic3k framework recently. As usual, the changes that get incorporated are essentially answers to questions from real life usage. There is a few in current release too.

Now I'm thinking what to do next. Among the things that still make me uncomfortable is the real-time guarantee, or, specifically in Pythomnic3k, a guarantee that every request will return by deadline. Just anything, a failure perhaps, as soon as it won't hang.

The problem is, sometimes a developer writes something that may not look like an infinite loop, but behaves like one. A situation when execution hits a

while True:
    pass

is a disaster in any architecture. It never returns, it has to be executed, there goes a CPU. Even the perfect scheduler cannot decide that this code effectively does nothing and just prevent it from being scheduled.

In Python this is worse, because it is effectively single-threaded for CPU-bound code. And even if that would not have been so, the next request hits the same spot and before you know it your load average is over 26. Therefore a single mistake like this could bring entire service down.

There has to be a way of interrupting the application code.

And in Python there is. A thread can inject an exception to another thread's execution path and as soon as the victim gets scheduled next, that exception is thrown. This is very easy:

PyThreadState_SetAsyncExc(other_thread, Error)

As soon as the application has a watchdog (and in Pythomnic3k there is), it could interrupt worker threads using such injection. I tried it and it worked.

Now there appear other problems.

First is the problem I could easily turn blind eye to. A code which executes an OS call cannot be interrupted this way. Easy to see, because while it does it is outside the Python scheduler's reach. Therefore

time.sleep(86400)

will still return tomorrow.

Second is the bigger problem of unpredictability. You don't know when or where the deadline hence the exception hits you. This effectively means that no code can be considered exception-safe now.

As a framework author, I could protect sensitive fragments of code by essentially "disabling interrupts" for the moments when interruption would not be convenient. So I write something like this:

current_thread.can_interrupt = True
application_code()
current_thread.can_interrupt = False

That protects the framework, but not the application code. The same developer who wrote the infinite loop in the first place could very reasonably write something like this:

lock.acquire()
try:
    ...
finally:
    lock.release()

Now consider what happens if the exception is injected after acquire but before try. Although an opening try statement actually does something non-trivial, it is never considered to be a possible source of exceptions. Paranoid as I am, I read try as noop. If try started failing, all bets are off.

Similarly, it is common to put clean up code in finally and make that code exception-safe. For example:

d["foo"] = "I exist therefore I am"
try:
    print(d["foo"])
finally:
    del d["foo"] # this could not throw
    something very important

Now any code, however safe it looks could throw. One could write very defensive code wrapping every line in a try-finally block, but then again, it is still possible that in

finally:
    try:
        del d["foo"]
    finally:
        something very important

an exception is thrown just after the second finally statement and something very important is still not executed.

As it turns out to be, we have a "programming in presence of asynchronous signals" situation here. As soon as some external mechanism could interrupt execution, you have to always account for that. This was typical when programming interrupt handlers in assembler, where much of your code were cli and sti instructions. All hell rained down if you ever forgot one.

Granted, such programming is possible and could even be considered stylish and felt elitist. But it is entirely different style from application programming in high level dynamic language. Even if facilities to disable interrupts are provided by the framework, it would require much experience and care on developer's behalf, much more that could be expected. And it will be a lot of trouble to use correctly.

Therefore I don't think I will instrument Pythomnic3k with such deadline enforcing mechanism. A developer who wishes to make his code deadline-friendly could always do it in the same way it is done now, by explicitly checking for request expiration at well defined points. Something like

while not pmnc.request.expired:
    read_data(timeout = pmnc.request.remain)

And if someone makes a mistake and the service hangs... Well, you have to be ready for that too.

Oracle Application Integration Architecture (AIA) Foundation Pack 11gR1: Essentials

This is a review of a book Oracle Application Integration Architecture (AIA) Foundation Pack 11gR1: Essentials by Hariharan V. Ganesarethinam written by request of Packt publishing.

The book is not for me to begin with:

This book assumes that you have a fundamental knowledge of Oracle SOA suite and its components.

I don't. Integration is part of my job, that's true, but I've never used Oracle SOA suite. Therefore my only hope was that the book was "essential" enough to be worth reading.

Well, it was, but unfortunately the experience was not pleasant.

As far as published texts go, I'm a grammar nazi. If it is on paper, it should at the very least be syntactically correct. You do have editors and proofreaders in Packt, don't you ?

This book doesn't pass the grammar check, thereare wordsglued together, wrong prepositions used into it, and sequence of tenses has done wrong.

To me this greatly undermines the book's authority. If the authors didn't even bother to check the grammar, what are the chances that it contains correct information and in such form that I understand it right ?

And that would not be an easy challenge.

The author uses words such as "securities" (plural of security), "compatibilities", "upgradation", "inbuilt", "self-intelligent", "real-time" (supposedly instead of "real-life") and "product portfolio".

Writing style is less than perfect, but that would be nitpicking, because there are whole sentences that make no sense, for example:

review the test results, and correct the implementation if any.

or

we need to know the name of the operation where we are going to test and type.

And here is an example of an outright contradiction:

* EBF can only invoke or be invoked by another EBF or EBS. It never communicates with ABCS directly.
* EBF can be invoked by requester ABCS.

Even when the sentence structure is correct, one can encounter something like this:

AIA recommends extending these business processes at logical entry points. However, it does not recommend any four point extension locations.

WTF is "four point location" ?

The text is dense with acronyms. Safe would be to say that on average every sentence in this book contains about two. This makes it very hard to read. Besides, the purpose often seems to be not making it clear to the reader, but including every single one, as though omission would render the writing incorrect. Therefore trains of acronyms are repeated again and again.

Specifically in summary sections, same statements are repeated like 5 times in different variations.

Enough about the style, what about the content ?

The book essentially contains 3 types of information.

1. Architecture and components of Oracle AIA.

As we presumably have "a fundamental knowledge of Oracle SOA suite", there is no big picture, overview of purpose, problem, process, tools or methods. Instead, it jumps right to AIA components.

And it explains them well enough I guess. Upon reading this book I have an essential if vague understanding what AIA consists of.

There also are chapters about my favourite error handling, security and versioning, but they are shallow. What they say is that there is an XML in which you can configure everything. It is right there in a deep dark directory.

2. Screenshots.

Those are of the useless sort, where a shot of a maximized window takes half a page, the screen fonts are too small to be read, and you only need a single tiny button anyway, which is circled. And the text caption says something like "fill in the required information in all the fields and click save".

3. Guidelines.

Those are advices from the field experience, but given that the book does not talk about practical issues anyway, those advices come void. There is no structure, no reason why they are here. Out of the blue comes "and never do this !" What ? Why ?

A concluding quote from page 5:

Now that you are the proud owner of a Packt book, ...

Please, let me decide whether or not I'm proud, don't declare it before I've even read the book.

The book lives to its promise but just barely, therefore 3 out of 5.

A fantastic solution for falsifying elections in Russia

This has to do with presidential elections in Russia which are due tomorrow.

Just as with senate elections last December, Russians are getting ready for grossly falsified farce. Those expectations came out true last December and gave those who were watching tons of fun, such as total 146% of votes broadcast on TV, incriminating cellphone youtube flicks officially declared having been filmed by criminals in underground bunkers, and hundreds of what courts decided to be "technical counting mistakes" all in favour of the winning party.

Now the saga continues. This time the question is whether Putin becomes a dark lord for the next 12 years or something else happens.

Those of you who are interested in elections in a dictatorship state, can check out how army and police can be ordered to vote for such and such, how state employees (a synonym for "the poorest") can be threatened to do the same, how psychiatric clinics vote, how all the TV channels can ever display the same candidate, how protest rallies against unfair elections are mirrored the next day pro-Putin by having thousands of random people brought in by buses from all around, each paid $30 just to stand there, all the fun paid by budget money.

But the panacea emerged last month - web cameras !

A camera had to be installed at each voting site pointing at the slot box to which all the ballots are thrown. This is officially said to be the end of election corruption as "anyone can watch for himself that everything is in order with Russian elections".

The largest Russian communications company Rostelecom (of which Putin is presumably a stakeholder) has been assigned to install 300000 cameras, the total project cost being around half a billion dollars drained from the state budget.

I thought I would pass posting about how ineffective a solution this is, but as I'm on it, I'll say a few words.

Even in perfect conditions, the camera only shows the moment of throwing in the filled ballot. Even theoretically it thwarts two threats - one known as "throw in" when a single person throws in a stack of pre-filled ballots, and the other known as "carousel" when a group of person moves from one site to another voting at each using false id's (the faces can theoretically be cross-matched).

But really only the short-sighted can believe that cameras are an answer to anything.

First, it is proves nothing. Under Russian laws video footage is not an evidence. Furthermore, even if something is caught on tape and is passed down to a court, it is the same court as the last time, and it declares it to be a fake made by terrorists to discredit the state.

Second, noone is to ever watch it. It is 300000 days of hi-res video or about 5000 terabytes of data. Even such archive is made accessible to anyone over the Internet (and never has a wonder of a quality official web site appeared in Russian Internet before), it is 800 man/years to watch, therefore ten thousand people can watch in about a month, eating up 15 gigabit of traffic.

And if you were still unsure, the real stuff happens outside the camera views.

See, elections are nothing but their announced results. Therefore in the simplest case, in a few days after the elections the central election committee simply announces that Putin has won. How can anyone can disprove it ? For the results to be discarded and re-elections initiated, there must be sufficient evidence, and court must decide. And you know about Russian courts already.

The other thing about the elections is that the process of counting is an upstream aggregation of numbers. There are many places where those numbers can be manipulated.

Also, the paper trail is lost - the ballots are long since thrashed and the only remaining evidence is going to be the protocols signed by committees at each site after hand counting the ballots.

This is rather trivial excercise in applied security, so like I said I felt no obligation for this post. But then the Internet delivered something so striking that the great amazement forced me.

Practical security issues, I've seen a lot, but this takes the cake. It is simple, ingenious and extremely effective. There is a way to falsify all the counting protocols at once with noone ever noticing. And it has to do with web cameras. Interested ?

See, the web cameras are electronic devices and the members of the committees at each site all over Russia must be instructed how to use them. There is a printed manual, over which each member must be briefed and sign for it the last page. Everything fine so far ?

Now, behold !

The page at which they sign for the briefing is identical to the page at which they sign in the counting protocol. And accidentally, this page for signatures is actually a blank A3 sheet folded and clipped to a regular A4 instruction book. Therefore immediately after the briefing we have a blank A3 sheet with nothing on it except for the signatures of the counting committee at exact same place where they should be if that sheet was a counting protocol. You take the signed page, print over anything you want and there you have it - the official vote couting protocol signed by the election committee members in person.

This is fantastic.

The original report is here.

Displaying date/time values in relative human readable format

A typical problem which universally appears in any interactive software is displaying date and time. A typical web page contains articles post times, a phone displays times of calls and short messages and so on and so forth.

Too often the developers don't bother with readability and we see something like

Posted Wednesday 1/12/2000 11:02:15 AM

whereas the American M/D/Y date and 12-hour AM/PM time format is even not universally recognized. Besides, more often it is not necessary to have it so detailed. Seconds, for instance, who ever needs them ? Security auditors ? Are you one of them ? A better way of displaying the same information is having only the relevant parts of it displayed:

Posted 11:02 AM (when posted the same day)
Posted 1/12/2000 (when posted days ago and time of day is not relevant)

And even better is to use relative user-readable form:

Posted 3 minutes ago
Posted 2 hours ago
Posted yesterday at 11:02 PM
Posted yesterday at 23:02 (tip to the hat for respecting locale)

Such format is user friendly, but why don't we see it often ? Surprise - because you have to implement it and it is not trivial. Instead of just

strftime("%M/%D/%Y");

you have to jump through hoops, the easiest being depending on external 3rd party library.

And this is my question: why don't we standardize some percent-format which would display date and time in printable relative form ? We already have strftime printing days of week names, month names (short and long), is it possible to have something like

strftime("%R");

to produce a human readable date-time relative to current time ? Or even have a small language in it, such as

strftime("%RD"); (produces "today" or "yesterday" or "2 days ago")
strftime("%RM"); (produces "this month" or "last month" or "2 months ago")

?

A cheap technical solution to control election vote counting

The elections are coming in Russia. As always, its results are expected to be grossly falsified. But instead of politics I'd like to talk about a technical solution to the problem.

What if an independent party is allowed to observe the process of vote counting, in order to come up with its own results which later can be used to verify the official numbers. How could it be done without forfeiting the traits of the voting process such as privacy, and also so that it is reliable and cost effective and causes minimal disruption to the existing process ?

Voting process in Russia is like this. You present yourself to the voting site and you are given an anonymous paper ballot. You go to a booth, mark one square with a pen and drop the paper to the ballot box.

So I thought that the step which could be modified is "marking the square with a pen". Instead of putting a pen inside a booth, a "square marking device" should be installed by a supervising party. It is nothing but a simple plastic box with a thin slot for ballot and, say, 4 marked buttons. You go into the booth, draw a curtain behind you so nobody sees what you are doing. Then you slide your ballot into the slot, press a button, and the device prints a cross in a corresponding square. Then you take the ballot out and drop it into the box as usual.

The marking device adds 1 to the counter corresponding to the button you pressed. Later the counters can be read by an authorized supervisor with a key. The officials count the votes by hand as usual after taking the ballots out of the box.

This most significant threat to this scheme is double voting. It should be protected against malicious user just pressing buttons at will to push up the counters. This could be done by reading a pre-printed unique bar code and not allowing it to be used twice. Bar code is also useful to be sure the ballot is positioned correctly.

As the amount of memory on the device is limited, you should probably pair a device installed to each site with the series of ballots dispersed to that site. This way, in the morning, the supervisor will unpack the device, pick random ballot from the pack and go through "imprinting phase" by inserting a ballot and applying a key. The device will read the bar code and lock itself to the range of ballots it will accept from that moment on. This will also protect from using ballots from other sites.

The solution can even be made entirely mechanical, no electronics required - it can be made simply a paper cutting hole punching device known in Russia as "компостер". You force press a button, and it cuts a circle fragment from a ballot, the cut out circle falls into a tray and remains inside the device, protected by a regular mechanical lock and key. Later, a supervisor opens it and counts the confetti. Each piece may contain a number or be of different color and may be watermarked to prevent forgery.

Neither of the solutions is bullet-proof. But it is simple, cheap and reasonably effective. The devices need to be tamper-evident, they must last just one day, they need a 2D barcode reader, 4 buttons, a key slot (touch memory reader ?) and 4 static printing heads. To cut on the latter and avoid ink refilling, even the electronic version of the device can be cutting holes in the ballot instead of printing marks on it.

I'm not a production expert, but I'd say such a device would cost a few dozen bucks. And you need hundred thousands of them, so it'd be even more cheap.

Bullshit character

*

On temporary patches

Once again the other side of a client-server link is broken. Our side works perfectly, but the peer sends in something incompatible with the protocol. This time it sends money value with 3 decimal places whereas there could be 2 at most. Our parser throws upon

amount=1.001

and surprise ! They can't fix it. May be later. And it's definitely gonna take some time. Some indefinite time. Or forever. But they need the solution. Right now.

Reluctantly I go and remove the scale check, and add the abominable money rounding. It kind of works, but having that patch makes me feel uncomfortable. It doesn't matter that in case of a future problem we'll probably get away by pointing them to the case where they clearly state their wish. It simply doesn't feel right.

Worst of all, it's not going to be fixed. Ever. How about (I think to myself) having a temporary patch, a time bomb of a kind, which ceases working after, say, a month. Granted, not a good idea, for many reasons, but for a while it felt strangely good.

And it's dead simple to do in Python.

from datetime import datetime as dt

def fix_by(d):
  def _fix_by(m):
    def _fix_by_check(*args, **kwargs):
      if dt.now() > dt.strptime(d, "%Y-%m-%d"):
         raise Exception("TOO LATE !!!")
      return m(*args, **kwargs)
    return _fix_by_check
  return _fix_by

@fix_by("2011-09-25")
def foo():
  print("ugly patch reporting")

foo()

Sigh...

Security tokens for the masses

Two new cryptographic devices have hit the market just recently. They are both of "cryptographic security tokens USB dongles" kind, and there would be nothing to speak about unless they haven't both been supporting GOST family of cryptoalgorithms. As a quick reminder, in Russia it is not only mandatory to be using GOST, but the implementations must also be certified by FSB. And until recently, there has been just one certified hardware device to support GOST, namely MS_KEY. It is supported in our product since mid-2010.

Suddenly there are three of them - the newcomers are Rutoken ECP and eToken GOST. And our customers (banks mostly) want them, like, yesterday ! So I had to stop whatever I was doing and write supporting plugins for them. Not a big deal really, once the documentation is there, a device sample is available, and support is ready to answer a stray stupid question. And so now we have support for 4 types of security tokens, including the eToken PRO Java not mentioned before because it only supports RSA keys.

A lot of interesting things I've learnt in the process. In this post I'd like to focus on usability.

The target audience of our product are average non-technical users working from home with their bank accounts and such. And so we have to keep the things really simple, ask no questions, to which they don't know answers anyway, don't mentor about what they needn't know, offer no choice and in general just walk them through hand in hand.

And here is where certain tokens features become an obstacle.

In most general terms, a cryptographic token is essentially a tiny computer with its own CPU, memory and private filesystem wrapped in a plastic USB dongle. It has a tiny operating system in the ROM which knows how to perform a handful of operations, mostly of cryptographic nature, such as "calculate hash" or "generate digital signature". It plugs to the PC through a standard USB port and when the right drivers are installed can be sent commands to. Seen this way a token simply is an external cryptographic device.

But it's not all. To have their product admissible to the market, token manufacturers must conform to certain standards. Those standards follow a certain procedure, a security protocol through which the tokens are supposed to be used. This procedure essentially implies that a token is used in a corporate environment, where tokens are distributed among the employees by security officers in orderly fashion, there is a well defined lifecycle to each token, and in general - there is a trusted administrator to see if there is any trouble.

Guess what ? Home users don't have administrators. They are their own security officers. And so the entire procedure goes down the drain.

Consider this - an average home user comes in to a bank, signs a few papers and is given a plastic dongle over the counter. All we need at this moment is to generate a new key inside the token, protect it with a user-provided PIN code and print out a certificate request. All we need later is to collect the PIN code for the key from the user and sign stuff.

Some tokens, specifically MS_KEY and eToken PRO Java which we have been using before, allow just that. You create a key, protect it with a PIN code and be done. Not the others, Rutoken ECP and eToken GOST. They conform. Which means that every time the owner wants to use the token for anything, he needs to provide the PIN code to the token.

This is precisely the same situation as with cell phones. In fact, there is a token inside a SIM card. The PIN code to it you enter to begin using the phone.

There is a lot of problems with token PINs (as opposed to the individual key PINs) for a home user:

• It is 123456 or alike by default and nobody cares to change it. This is bad, because it's a key.
• And if they would, the disaster is just around the corner, because they forget it. This is bad because there is no established procedure to recover the token contents, because there is no trusted administrator.
• It can be recovered following manufacturer-specific procedure, and this is there another magic PIN code (security officer's) comes into play. It in turn is 654321 and you cannot even begin explaining what it is for to the user, leave alone having it changed, written down and kept in a safe place.
• A bank can have employees setting random PINs to tokens before giving them out, but it complicates the procedure, is all expenses and not only doesn't make the threat of hacker knowing the PIN to go away, but makes it specifically targeting the bank. It was them who knew the PIN after all. You can automate the procedure and have PINs printed in envelopes similar to how it is done with credit cards, but it's way more expenses.

Again, as with cell phones, how many people have their phones protected with PIN codes ? And of those, how many have their PUKs kept in a safe place ?

So in the best case, the user never changes the PIN and the token contents is accessible to anyone. And in the worst case, the user forgets his password in 3 days and the token is as good as dead. Or the other way around depending which approach to security you take.

Compared to that, when you lose PIN code to an individual key inside the token, no big deal. Generate another one, come up with a new PIN and there you go.

Returning to the tokens at hand.

Both Rutoken ECP and eToken GOST have their mandatory PIN codes, but Rutoken ECP at least supports protecting keys with separate PINs. This way we don't care about the PIN code to the token at all. It may be 123456 or whatever. The user is required to enter it each time and it is a nuisance, but each individual key is protected with a separate PIN just as before.

With eToken GOST it is worse - it only supports one PIN code. There is no way you can protect individual keys. Anyone who knows the token PIN has access to everything. This approach makes sense too, but it renders useless some interesting features of our application such as having multiple active keys on the same token, or having a new key generated over an old one and the new certificate request signed with the old key, thus escaping from visiting the bank in person when the certificate is about to expire.

Why am I not blogging more ?

This is kind of a meta-post, a blog entry about the process of writing a blog.

And so, I'm thinking about what is it that holds me back from writing more ? Things worth posting cross my mind frequently, I observe them, make a mental note if you like but do not write down.

Problem #1
The process of thinking the idea over is fun. I keep thinking about the thing until there no longer are interesting aspects. Once this is done, I quickly lose interest to the idea, sort of letting it go. And this is exactly the moment when I should start typing it in. Oh, the bother.

Problem #2
Similarly, once I had thought the idea over, it no longer fascinates me, and there is no indication of whether it would be fascinating to anyone else. Kind of a preliminary positive feedback, which is impossible.

Problem #3
Writing is hard on its own premises. It takes time and effort to write well, no matter in your native tongue or not. What are the benefits from doing it at all ?

The outcome ? The step between having a thought and having it written down is hard. It's not going to happen unless I actually make it.