On interrupting application code

So, I've released a new version of Pythomnic3k framework recently. As usual, the changes that get incorporated are essentially answers to questions from real life usage. There is a few in current release too.

Now I'm thinking what to do next. Among the things that still make me uncomfortable is the real-time guarantee, or, specifically in Pythomnic3k, a guarantee that every request will return by deadline. Just anything, a failure perhaps, as soon as it won't hang.

The problem is, sometimes a developer writes something that may not look like an infinite loop, but behaves like one. A situation when execution hits a

while True:
    pass

is a disaster in any architecture. It never returns, it has to be executed, there goes a CPU. Even the perfect scheduler cannot decide that this code effectively does nothing and just prevent it from being scheduled.

In Python this is worse, because it is effectively single-threaded for CPU-bound code. And even if that would not have been so, the next request hits the same spot and before you know it your load average is over 26. Therefore a single mistake like this could bring entire service down.

There has to be a way of interrupting the application code.

And in Python there is. A thread can inject an exception to another thread's execution path and as soon as the victim gets scheduled next, that exception is thrown. This is very easy:

PyThreadState_SetAsyncExc(other_thread, Error)

As soon as the application has a watchdog (and in Pythomnic3k there is), it could interrupt worker threads using such injection. I tried it and it worked.

Now there appear other problems.

First is the problem I could easily turn blind eye to. A code which executes an OS call cannot be interrupted this way. Easy to see, because while it does it is outside the Python scheduler's reach. Therefore

time.sleep(86400)

will still return tomorrow.

Second is the bigger problem of unpredictability. You don't know when or where the deadline hence the exception hits you. This effectively means that no code can be considered exception-safe now.

As a framework author, I could protect sensitive fragments of code by essentially "disabling interrupts" for the moments when interruption would not be convenient. So I write something like this:

current_thread.can_interrupt = True
application_code()
current_thread.can_interrupt = False

That protects the framework, but not the application code. The same developer who wrote the infinite loop in the first place could very reasonably write something like this:

lock.acquire()
try:
    ...
finally:
    lock.release()

Now consider what happens if the exception is injected after acquire but before try. Although an opening try statement actually does something non-trivial, it is never considered to be a possible source of exceptions. Paranoid as I am, I read try as noop. If try started failing, all bets are off.

Similarly, it is common to put clean up code in finally and make that code exception-safe. For example:

d["foo"] = "I exist therefore I am"
try:
    print(d["foo"])
finally:
    del d["foo"] # this could not throw
    something very important

Now any code, however safe it looks could throw. One could write very defensive code wrapping every line in a try-finally block, but then again, it is still possible that in

finally:
    try:
        del d["foo"]
    finally:
        something very important

an exception is thrown just after the second finally statement and something very important is still not executed.

As it turns out to be, we have a "programming in presence of asynchronous signals" situation here. As soon as some external mechanism could interrupt execution, you have to always account for that. This was typical when programming interrupt handlers in assembler, where much of your code were cli and sti instructions. All hell rained down if you ever forgot one.

Granted, such programming is possible and could even be considered stylish and felt elitist. But it is entirely different style from application programming in high level dynamic language. Even if facilities to disable interrupts are provided by the framework, it would require much experience and care on developer's behalf, much more that could be expected. And it will be a lot of trouble to use correctly.

Therefore I don't think I will instrument Pythomnic3k with such deadline enforcing mechanism. A developer who wishes to make his code deadline-friendly could always do it in the same way it is done now, by explicitly checking for request expiration at well defined points. Something like

while not pmnc.request.expired:
    read_data(timeout = pmnc.request.remain)

And if someone makes a mistake and the service hangs... Well, you have to be ready for that too.

Oracle Application Integration Architecture (AIA) Foundation Pack 11gR1: Essentials

This is a review of a book Oracle Application Integration Architecture (AIA) Foundation Pack 11gR1: Essentials by Hariharan V. Ganesarethinam written by request of Packt publishing.

The book is not for me to begin with:

This book assumes that you have a fundamental knowledge of Oracle SOA suite and its components.

I don't. Integration is part of my job, that's true, but I've never used Oracle SOA suite. Therefore my only hope was that the book was "essential" enough to be worth reading.

Well, it was, but unfortunately the experience was not pleasant.

As far as published texts go, I'm a grammar nazi. If it is on paper, it should at the very least be syntactically correct. You do have editors and proofreaders in Packt, don't you ?

This book doesn't pass the grammar check, thereare wordsglued together, wrong prepositions used into it, and sequence of tenses has done wrong.

To me this greatly undermines the book's authority. If the authors didn't even bother to check the grammar, what are the chances that it contains correct information and in such form that I understand it right ?

And that would not be an easy challenge.

The author uses words such as "securities" (plural of security), "compatibilities", "upgradation", "inbuilt", "self-intelligent", "real-time" (supposedly instead of "real-life") and "product portfolio".

Writing style is less than perfect, but that would be nitpicking, because there are whole sentences that make no sense, for example:

review the test results, and correct the implementation if any.

or

we need to know the name of the operation where we are going to test and type.

And here is an example of an outright contradiction:

* EBF can only invoke or be invoked by another EBF or EBS. It never communicates with ABCS directly.
* EBF can be invoked by requester ABCS.

Even when the sentence structure is correct, one can encounter something like this:

AIA recommends extending these business processes at logical entry points. However, it does not recommend any four point extension locations.

WTF is "four point location" ?

The text is dense with acronyms. Safe would be to say that on average every sentence in this book contains about two. This makes it very hard to read. Besides, the purpose often seems to be not making it clear to the reader, but including every single one, as though omission would render the writing incorrect. Therefore trains of acronyms are repeated again and again.

Specifically in summary sections, same statements are repeated like 5 times in different variations.

Enough about the style, what about the content ?

The book essentially contains 3 types of information.

1. Architecture and components of Oracle AIA.

As we presumably have "a fundamental knowledge of Oracle SOA suite", there is no big picture, overview of purpose, problem, process, tools or methods. Instead, it jumps right to AIA components.

And it explains them well enough I guess. Upon reading this book I have an essential if vague understanding what AIA consists of.

There also are chapters about my favourite error handling, security and versioning, but they are shallow. What they say is that there is an XML in which you can configure everything. It is right there in a deep dark directory.

2. Screenshots.

Those are of the useless sort, where a shot of a maximized window takes half a page, the screen fonts are too small to be read, and you only need a single tiny button anyway, which is circled. And the text caption says something like "fill in the required information in all the fields and click save".

3. Guidelines.

Those are advices from the field experience, but given that the book does not talk about practical issues anyway, those advices come void. There is no structure, no reason why they are here. Out of the blue comes "and never do this !" What ? Why ?

A concluding quote from page 5:

Now that you are the proud owner of a Packt book, ...

Please, let me decide whether or not I'm proud, don't declare it before I've even read the book.

The book lives to its promise but just barely, therefore 3 out of 5.

A fantastic solution for falsifying elections in Russia

This has to do with presidential elections in Russia which are due tomorrow.

Just as with senate elections last December, Russians are getting ready for grossly falsified farce. Those expectations came out true last December and gave those who were watching tons of fun, such as total 146% of votes broadcast on TV, incriminating cellphone youtube flicks officially declared having been filmed by criminals in underground bunkers, and hundreds of what courts decided to be "technical counting mistakes" all in favour of the winning party.

Now the saga continues. This time the question is whether Putin becomes a dark lord for the next 12 years or something else happens.

Those of you who are interested in elections in a dictatorship state, can check out how army and police can be ordered to vote for such and such, how state employees (a synonym for "the poorest") can be threatened to do the same, how psychiatric clinics vote, how all the TV channels can ever display the same candidate, how protest rallies against unfair elections are mirrored the next day pro-Putin by having thousands of random people brought in by buses from all around, each paid $30 just to stand there, all the fun paid by budget money.

But the panacea emerged last month - web cameras !

A camera had to be installed at each voting site pointing at the slot box to which all the ballots are thrown. This is officially said to be the end of election corruption as "anyone can watch for himself that everything is in order with Russian elections".

The largest Russian communications company Rostelecom (of which Putin is presumably a stakeholder) has been assigned to install 300000 cameras, the total project cost being around half a billion dollars drained from the state budget.

I thought I would pass posting about how ineffective a solution this is, but as I'm on it, I'll say a few words.

Even in perfect conditions, the camera only shows the moment of throwing in the filled ballot. Even theoretically it thwarts two threats - one known as "throw in" when a single person throws in a stack of pre-filled ballots, and the other known as "carousel" when a group of person moves from one site to another voting at each using false id's (the faces can theoretically be cross-matched).

But really only the short-sighted can believe that cameras are an answer to anything.

First, it is proves nothing. Under Russian laws video footage is not an evidence. Furthermore, even if something is caught on tape and is passed down to a court, it is the same court as the last time, and it declares it to be a fake made by terrorists to discredit the state.

Second, noone is to ever watch it. It is 300000 days of hi-res video or about 5000 terabytes of data. Even such archive is made accessible to anyone over the Internet (and never has a wonder of a quality official web site appeared in Russian Internet before), it is 800 man/years to watch, therefore ten thousand people can watch in about a month, eating up 15 gigabit of traffic.

And if you were still unsure, the real stuff happens outside the camera views.

See, elections are nothing but their announced results. Therefore in the simplest case, in a few days after the elections the central election committee simply announces that Putin has won. How can anyone can disprove it ? For the results to be discarded and re-elections initiated, there must be sufficient evidence, and court must decide. And you know about Russian courts already.

The other thing about the elections is that the process of counting is an upstream aggregation of numbers. There are many places where those numbers can be manipulated.

Also, the paper trail is lost - the ballots are long since thrashed and the only remaining evidence is going to be the protocols signed by committees at each site after hand counting the ballots.

This is rather trivial excercise in applied security, so like I said I felt no obligation for this post. But then the Internet delivered something so striking that the great amazement forced me.

Practical security issues, I've seen a lot, but this takes the cake. It is simple, ingenious and extremely effective. There is a way to falsify all the counting protocols at once with noone ever noticing. And it has to do with web cameras. Interested ?

See, the web cameras are electronic devices and the members of the committees at each site all over Russia must be instructed how to use them. There is a printed manual, over which each member must be briefed and sign for it the last page. Everything fine so far ?

Now, behold !

The page at which they sign for the briefing is identical to the page at which they sign in the counting protocol. And accidentally, this page for signatures is actually a blank A3 sheet folded and clipped to a regular A4 instruction book. Therefore immediately after the briefing we have a blank A3 sheet with nothing on it except for the signatures of the counting committee at exact same place where they should be if that sheet was a counting protocol. You take the signed page, print over anything you want and there you have it - the official vote couting protocol signed by the election committee members in person.

This is fantastic.

The original report is here.

Displaying date/time values in relative human readable format

A typical problem which universally appears in any interactive software is displaying date and time. A typical web page contains articles post times, a phone displays times of calls and short messages and so on and so forth.

Too often the developers don't bother with readability and we see something like

Posted Wednesday 1/12/2000 11:02:15 AM

whereas the American M/D/Y date and 12-hour AM/PM time format is even not universally recognized. Besides, more often it is not necessary to have it so detailed. Seconds, for instance, who ever needs them ? Security auditors ? Are you one of them ? A better way of displaying the same information is having only the relevant parts of it displayed:

Posted 11:02 AM (when posted the same day)
Posted 1/12/2000 (when posted days ago and time of day is not relevant)

And even better is to use relative user-readable form:

Posted 3 minutes ago
Posted 2 hours ago
Posted yesterday at 11:02 PM
Posted yesterday at 23:02 (tip to the hat for respecting locale)

Such format is user friendly, but why don't we see it often ? Surprise - because you have to implement it and it is not trivial. Instead of just

strftime("%M/%D/%Y");

you have to jump through hoops, the easiest being depending on external 3rd party library.

And this is my question: why don't we standardize some percent-format which would display date and time in printable relative form ? We already have strftime printing days of week names, month names (short and long), is it possible to have something like

strftime("%R");

to produce a human readable date-time relative to current time ? Or even have a small language in it, such as

strftime("%RD"); (produces "today" or "yesterday" or "2 days ago")
strftime("%RM"); (produces "this month" or "last month" or "2 months ago")

?

A cheap technical solution to control election vote counting

The elections are coming in Russia. As always, its results are expected to be grossly falsified. But instead of politics I'd like to talk about a technical solution to the problem.

What if an independent party is allowed to observe the process of vote counting, in order to come up with its own results which later can be used to verify the official numbers. How could it be done without forfeiting the traits of the voting process such as privacy, and also so that it is reliable and cost effective and causes minimal disruption to the existing process ?

Voting process in Russia is like this. You present yourself to the voting site and you are given an anonymous paper ballot. You go to a booth, mark one square with a pen and drop the paper to the ballot box.

So I thought that the step which could be modified is "marking the square with a pen". Instead of putting a pen inside a booth, a "square marking device" should be installed by a supervising party. It is nothing but a simple plastic box with a thin slot for ballot and, say, 4 marked buttons. You go into the booth, draw a curtain behind you so nobody sees what you are doing. Then you slide your ballot into the slot, press a button, and the device prints a cross in a corresponding square. Then you take the ballot out and drop it into the box as usual.

The marking device adds 1 to the counter corresponding to the button you pressed. Later the counters can be read by an authorized supervisor with a key. The officials count the votes by hand as usual after taking the ballots out of the box.

This most significant threat to this scheme is double voting. It should be protected against malicious user just pressing buttons at will to push up the counters. This could be done by reading a pre-printed unique bar code and not allowing it to be used twice. Bar code is also useful to be sure the ballot is positioned correctly.

As the amount of memory on the device is limited, you should probably pair a device installed to each site with the series of ballots dispersed to that site. This way, in the morning, the supervisor will unpack the device, pick random ballot from the pack and go through "imprinting phase" by inserting a ballot and applying a key. The device will read the bar code and lock itself to the range of ballots it will accept from that moment on. This will also protect from using ballots from other sites.

The solution can even be made entirely mechanical, no electronics required - it can be made simply a paper cutting hole punching device known in Russia as "компостер". You force press a button, and it cuts a circle fragment from a ballot, the cut out circle falls into a tray and remains inside the device, protected by a regular mechanical lock and key. Later, a supervisor opens it and counts the confetti. Each piece may contain a number or be of different color and may be watermarked to prevent forgery.

Neither of the solutions is bullet-proof. But it is simple, cheap and reasonably effective. The devices need to be tamper-evident, they must last just one day, they need a 2D barcode reader, 4 buttons, a key slot (touch memory reader ?) and 4 static printing heads. To cut on the latter and avoid ink refilling, even the electronic version of the device can be cutting holes in the ballot instead of printing marks on it.

I'm not a production expert, but I'd say such a device would cost a few dozen bucks. And you need hundred thousands of them, so it'd be even more cheap.

Bullshit character

*

On temporary patches

Once again the other side of a client-server link is broken. Our side works perfectly, but the peer sends in something incompatible with the protocol. This time it sends money value with 3 decimal places whereas there could be 2 at most. Our parser throws upon

amount=1.001

and surprise ! They can't fix it. May be later. And it's definitely gonna take some time. Some indefinite time. Or forever. But they need the solution. Right now.

Reluctantly I go and remove the scale check, and add the abominable money rounding. It kind of works, but having that patch makes me feel uncomfortable. It doesn't matter that in case of a future problem we'll probably get away by pointing them to the case where they clearly state their wish. It simply doesn't feel right.

Worst of all, it's not going to be fixed. Ever. How about (I think to myself) having a temporary patch, a time bomb of a kind, which ceases working after, say, a month. Granted, not a good idea, for many reasons, but for a while it felt strangely good.

And it's dead simple to do in Python.

from datetime import datetime as dt

def fix_by(d):
  def _fix_by(m):
    def _fix_by_check(*args, **kwargs):
      if dt.now() > dt.strptime(d, "%Y-%m-%d"):
         raise Exception("TOO LATE !!!")
      return m(*args, **kwargs)
    return _fix_by_check
  return _fix_by

@fix_by("2011-09-25")
def foo():
  print("ugly patch reporting")

foo()

Sigh...

Security tokens for the masses

Two new cryptographic devices have hit the market just recently. They are both of "cryptographic security tokens USB dongles" kind, and there would be nothing to speak about unless they haven't both been supporting GOST family of cryptoalgorithms. As a quick reminder, in Russia it is not only mandatory to be using GOST, but the implementations must also be certified by FSB. And until recently, there has been just one certified hardware device to support GOST, namely MS_KEY. It is supported in our product since mid-2010.

Suddenly there are three of them - the newcomers are Rutoken ECP and eToken GOST. And our customers (banks mostly) want them, like, yesterday ! So I had to stop whatever I was doing and write supporting plugins for them. Not a big deal really, once the documentation is there, a device sample is available, and support is ready to answer a stray stupid question. And so now we have support for 4 types of security tokens, including the eToken PRO Java not mentioned before because it only supports RSA keys.

A lot of interesting things I've learnt in the process. In this post I'd like to focus on usability.

The target audience of our product are average non-technical users working from home with their bank accounts and such. And so we have to keep the things really simple, ask no questions, to which they don't know answers anyway, don't mentor about what they needn't know, offer no choice and in general just walk them through hand in hand.

And here is where certain tokens features become an obstacle.

In most general terms, a cryptographic token is essentially a tiny computer with its own CPU, memory and private filesystem wrapped in a plastic USB dongle. It has a tiny operating system in the ROM which knows how to perform a handful of operations, mostly of cryptographic nature, such as "calculate hash" or "generate digital signature". It plugs to the PC through a standard USB port and when the right drivers are installed can be sent commands to. Seen this way a token simply is an external cryptographic device.

But it's not all. To have their product admissible to the market, token manufacturers must conform to certain standards. Those standards follow a certain procedure, a security protocol through which the tokens are supposed to be used. This procedure essentially implies that a token is used in a corporate environment, where tokens are distributed among the employees by security officers in orderly fashion, there is a well defined lifecycle to each token, and in general - there is a trusted administrator to see if there is any trouble.

Guess what ? Home users don't have administrators. They are their own security officers. And so the entire procedure goes down the drain.

Consider this - an average home user comes in to a bank, signs a few papers and is given a plastic dongle over the counter. All we need at this moment is to generate a new key inside the token, protect it with a user-provided PIN code and print out a certificate request. All we need later is to collect the PIN code for the key from the user and sign stuff.

Some tokens, specifically MS_KEY and eToken PRO Java which we have been using before, allow just that. You create a key, protect it with a PIN code and be done. Not the others, Rutoken ECP and eToken GOST. They conform. Which means that every time the owner wants to use the token for anything, he needs to provide the PIN code to the token.

This is precisely the same situation as with cell phones. In fact, there is a token inside a SIM card. The PIN code to it you enter to begin using the phone.

There is a lot of problems with token PINs (as opposed to the individual key PINs) for a home user:

• It is 123456 or alike by default and nobody cares to change it. This is bad, because it's a key.
• And if they would, the disaster is just around the corner, because they forget it. This is bad because there is no established procedure to recover the token contents, because there is no trusted administrator.
• It can be recovered following manufacturer-specific procedure, and this is there another magic PIN code (security officer's) comes into play. It in turn is 654321 and you cannot even begin explaining what it is for to the user, leave alone having it changed, written down and kept in a safe place.
• A bank can have employees setting random PINs to tokens before giving them out, but it complicates the procedure, is all expenses and not only doesn't make the threat of hacker knowing the PIN to go away, but makes it specifically targeting the bank. It was them who knew the PIN after all. You can automate the procedure and have PINs printed in envelopes similar to how it is done with credit cards, but it's way more expenses.

Again, as with cell phones, how many people have their phones protected with PIN codes ? And of those, how many have their PUKs kept in a safe place ?

So in the best case, the user never changes the PIN and the token contents is accessible to anyone. And in the worst case, the user forgets his password in 3 days and the token is as good as dead. Or the other way around depending which approach to security you take.

Compared to that, when you lose PIN code to an individual key inside the token, no big deal. Generate another one, come up with a new PIN and there you go.

Returning to the tokens at hand.

Both Rutoken ECP and eToken GOST have their mandatory PIN codes, but Rutoken ECP at least supports protecting keys with separate PINs. This way we don't care about the PIN code to the token at all. It may be 123456 or whatever. The user is required to enter it each time and it is a nuisance, but each individual key is protected with a separate PIN just as before.

With eToken GOST it is worse - it only supports one PIN code. There is no way you can protect individual keys. Anyone who knows the token PIN has access to everything. This approach makes sense too, but it renders useless some interesting features of our application such as having multiple active keys on the same token, or having a new key generated over an old one and the new certificate request signed with the old key, thus escaping from visiting the bank in person when the certificate is about to expire.

Why am I not blogging more ?

This is kind of a meta-post, a blog entry about the process of writing a blog.

And so, I'm thinking about what is it that holds me back from writing more ? Things worth posting cross my mind frequently, I observe them, make a mental note if you like but do not write down.

Problem #1
The process of thinking the idea over is fun. I keep thinking about the thing until there no longer are interesting aspects. Once this is done, I quickly lose interest to the idea, sort of letting it go. And this is exactly the moment when I should start typing it in. Oh, the bother.

Problem #2
Similarly, once I had thought the idea over, it no longer fascinates me, and there is no indication of whether it would be fascinating to anyone else. Kind of a preliminary positive feedback, which is impossible.

Problem #3
Writing is hard on its own premises. It takes time and effort to write well, no matter in your native tongue or not. What are the benefits from doing it at all ?

The outcome ? The step between having a thought and having it written down is hard. It's not going to happen unless I actually make it.

Another usability pearl

One of my all time favourites:

Press ENTER to exit.

So short but blows your mind right away.