Notes on implementing MongoDB driver

It's been a week-long statutory holiday in Russia and I spent a few afternoons implementing MongoDB support for the Python 3 application framework Pythomnic3k.

It turned out to be simpler task than I thought, a few odd hours here and there, given that I wanted to implement everything from the ground up. Still, there are notes I'd like to share afterwards.

Note #1: BSON

BSON is "Binary jSON" - proprietary binary protocol for serializing pieces of JavaScript for transmitting or storing. It sets the byte representation rules for simple JavaScript types such as "Integer" or "String", and also for MongoDB-specific structures such as "Regex" or "JavaScript with scope".

Supporting BSON is #1 requirement for MongoDB driver. Essentially, BSON is all there is to it.

Its entire specification is just two pages long, describes formats for 20 different objects, and for such a small spec is notably awkward. There are different ways to serialize similar objects, there are deprecated objects, and there are mysterious objects with unspecified format, probably reserved for internal use.

For example, String is serialized as

(length)(content)(NULL)

but Regex is serialized as just

(content)(NULL)

Also key/value tuple is serialized as

(value type)(key)(NULL)(value)

which is strange because while deserializing you cannot simply read key, then value type and then switch to an appropiate type parser, which would be possible for

(key)(NULL)(value_type)(value)

input.

Note #2: No response to packets

Application communicates with MongoDB engine using request packets, OP_THIS, OP_THAT and OP_SO_ON. To some of them the database responds and to some it does not. In fact, it prefers to remain silent, responding only when it has to.

For example, if you want to find something, you send OP_QUERY then you do receive a response, OP_REPLY, containing the data you've been looking for. But if you want to insert data, you send OP_INSERT and (surprise) there is no response.

I understand that with MongoDB there are no guarantees of data integrity, no persistency and no transactions, but now there is even no ack from the database. It makes me feel uncomfortable.

Note #3: Explicit ordering of documents

This one is ugly. The main data structure used in BSON hence in MongoDB is a document which is a dict or associative array. And associative arrays have no order. They have iteration order which is arbitrary implementation dependent likely derived from internal hash function. That order may even be well defined, for example smallest key comes first, but that could only work with comparable keys. In any case dicts don't have n-th element.

The problem appears when the request packet containing document is serialized to be transmitted. MongoDB relies on the order in which the dict appears on the wire. Specifically, if you send a command to the database, the first key must contain the name of the command. And in some cases the command requires positional arguments to be transmitted in defined order. So you have to iterate over a dict in a specific order only known to the caller.

For example, if you have a dict

{ "foo": 1, "bar": 2 }

and you want to invoke

db.foo(bar)

with it you have to serialize it like this:

[foo=1, bar=2]

but if it is

db.bar(foo)

it is the other way around.

The abysmal

{ 1 => "first", 2 => "second" }

kind of dictarray has also found its way here, it is the way BSON serializes lists, but this is at least an implementation detail hidden from the application inside the driver.

Note #4: Similar things in different ways

This is easy to illustrate with the way MongoDB reports errors.

For one, as already mentioned, some of the requests have no response whatsoever even though they might fail.

Then, if you get an OP_REPLY response, it has bit flag QueryFailure, which if set should be accompanied with a single document with $err key in it containing error message. But it also has bit flag CursorNotFound, which apparently is also an error condition, but then it has no message.

Furthermore, if the request has been a command (which only the caller knows), the OP_REPLY is a success, but it contains a document with $ok key that can be 0 in which case there is also an $errmsg key containing error message.

So in the first case error reporting is done on packet level, and in the second case - on application level. And in some cases no error is reported at all.

All this leads to the final

Note #5: Ad Hoc-ish

The exercise of integrating with MongoDB leaves the impression of incompleteness. It feels like this database (and I suspect other currently existing NoSQL databases) is an experiment, an early prototype.

Well, they have 40 years of catching up with relational databases to be as well understood and implemented, it's a long way to go, so I do wish them good luck.

A simple way to tell whether a programmer is working

If both hands are on the keyboard then he's working, but if one hand is on the mouse then he's not.

Walking with Pythons

My previous posts about how long it takes for Python 3 to become mainstream had a few comments, both on- and off-site. Those comments written among others by members of Python team, pointed out that there may indeed be unspecified reasons for keeping two identical parallel versions for so long. Well, they know better.

But I still believe in two things:

1. No matter which version of Python 2 you decide to leave for Python 3, there is still an incompatibility gap you have to cross, and postponing doesn't make it go away.
2. Python 3 still offers no benefits compared to Python 2, even for new green field projects. And this means that new projects are still started in Python 2.

Anyway, we'll see. Python is a great language, no matter which version. Since the project I'm worrying about is itself a framework with very little dependencies, I will keep the development running easily, language not to blame.

Will Python 2 ever end ?

It so happens that I have been an early adopter of Python 3. Ever since the first alpha was released back in 2008, I've started rewriting my middleware framework from Python 2 to Python 3. I have to admit, it was beneficial for the project. Not because the language was that better, but because of the ground-up redesign and rewrite, cleaning up the mess. What bothers me is that 2.5 years later I can't expect any significant number of users to even look at the project, because it is in the "future" language.

I'm totally confused. Python 2 is still mainstream and is undergoing active development. Python 2.7 may be the last of the 2.x branch, but still it is released in the middle of 2010. Which gives it at least another 1.5 years of joyous lifetime, even if is declared dead right now. Which means that Python 3 is still not there until 2012. Which means that for 4-5 years it has been a better language which very few actually use.

What I find absolutely ridiculous is the transition strategy. If you want to break compatibility with the previous version - for God's sake, just do it. And the new features - they are supposed to be the sweetener, the bait for the users to make the move. It makes no sense to keep backporting the new features into the already more popular legacy version. Absolutely no goddamn sense. What you get in the end is two versions, otherwise identical, but when the first one is active, popular and widely used, the other one is... uhm... incompatible with the first one ? Seriously, except for the broken compatibility, why Python 3 is different from Python 2.7 now ?

Are the users supposed to switch to the new version, when all they are facing is incompatibilities and no benefits ? This is so wrong...

Re: Cryptography

This post is a response to a recent discussion on a "Russian Software Developer Network" forum. The thread was called "Cryptography".

Oh, the drama ! And professionalism level was unrivaled. It was there that I found a new addition to my personal hall of fame:

epileptic curves

Seriosly though, it somehow happens that cryptography becomes the easiest part of security. Easiest to know about, easiest to talk about, easiest to show off with.

Why ? I'd say it is because it is closely related to mathematics and mathematics brings in the safe harbour feeling to those who want certainty in the shaky world of security. Besides, many of those who talk passionately about cryptography (including myself) have mathematical background.

Surprise, the security-related feature of cryptography is not based on hard mathematics. See, the feature that we seek most in cryptography is called "strength". We want it for encryption, for hashes, for digital signatures, for everything. It is strength which causes holy wars on forums. But what is it ?

In cryptography, strength is the ability to withstand currently known attacks.

See the problem ?

The words "currently known" reduce all hopes for certainty to dust. You cannot "prove" strength in mathematical sense. Anything is strong as soon as it hasn't been demonstratively broken.

There is not much reason comparing strength as well. As seen on the Internet:

My kung-fu is stronger than yours by 2¹⁷

But it only makes sense if you compare identical or very similar algorithms - then you are essentially comparing their lifetimes. As we assume they both haven't been broken yet, the larger the power, the more time on average it takes the attacker to break it using some kind of brute-force attack.

Put simply, all cryptographical strength is based on one big assumption - that the good guys know better than the bad guys.

We believe something is strong because noone has published the way to break it. Even though such way may exists, and may be widely used against us, we still consider it strong until the contrary appears on paper.

The biggest paradox here is that we are even sure that there is a way to break it, it is just that noone (meaning the good guys) has found it yet. And we hope noone (meaning the bad guys) will while we are using it.

We believe that the respectable scientists work hard trying to break every known algorithm and we are somehow sure they break them first. And publish. Not for money, not for fame, just for the sake of it. What were the names of the people who published attacks against MD5 ?

And the bad guys have much better position. They need to attack just one algorithm, or even just one key. They have enormous resources and motivation to do it. They might have affected the design of the algorithm to put a backdoor in it in the first place. And they don't need to publish their results, but silently exploit it for years.

Well, the good guys seem to be winning so far. Or do they ? You never know. This is called security.

Python 3 frameworks anyone ?

First, I'm happy to announce that I have just released the next version of Pythomnic3k, a Python 3 framework to develop SOA middleware.

But I'd also like to share with you the big question of this Python 3 framework.

I have been working on its predecessor Pythomnic (similar, but written in Python 2) in 2005-2007 using it for integrating various systems in some bank. It worked, but as any software being developed in ad-hoc manner became messy over time. Not to mention the fact that as I learned Python, the old code looked uglier every day.

And so, as of early 2008, along with the first Python 3 betas being released, I decided that Pythomnic needed a complete redesign and rewrite exclusively in brand new Python ! Pythomnic3k was in development ever since. It has a nice and clean upfront design, based on 3 years of experience with Pythomnic, it's written much better, and it is has extensive self-tests. Which is to say, it is a quality piece of software. I spent next 1.5 years polishing it, until release 1.0 was finally published in 2009. Release 1.1 which I believe I've already announced, came out after some 8 more months of refinement.

All this time I kept using it for what, for the same integration tasks - connecting point A to point B, transforming messages, supporting various protocols. In the company I work for, it is used for delivering bank transfers and billing payments, sending SMS notifications (contains full implementation of SMPP 3.4 among other things), providing cryptographic network services of various sorts, and just about anything. In short, it serves as a middleware glue, and if I'm allowed to judge, it fits the bill.

Now, the big question is - was it really beneficial to switch to Python 3 starting a new development ? Take a look at the list of Python 3 packages. The language is around for 2 years, and there is like what, 50 of them ? Out of which many are one-module utilities ? Give me a break.

Python 3 looked promising, although it was not immediately apparent, what new features are the killer ones. Frankly, I'm still not sure. I love the syntax improvements and the correct str/bytes, but what else ? Am I missing the wave or it is not there yet ?

Anyway, Pythomnic3k architecture has very little dependencies, it is a pretty much self-contained framework, which means that it doesn't suffer from the lack of anything in particular in Python 3 libraries, but I would still love to see more Python 3 libraries around to have them plugged to the framework.

You come to software market ...

... and you want your software cheap, fast and of high quality. You my friend want to be fooled, and you will be fooled, because nature cannot be.

How are they going to shut the Internet down ?

Well, I never would have thought that my first post after such a long period of silence would be like this, but this is what bothers me.

Given the current political situation in Russia, in which power belongs to totally corrupt organized crime, the Internet remains the only media where anyone can speak out. For doing so you still may be prosecuted, but this is the only place where one can at least publish an unpopular opinion.

For example, check out the Internet shit storm (available mostly in Russian) on the topic of outrageous stealing as much as $50 billion of budget money under state-approved "make drinking water clean" program.

And so my question is - how are they going to shut the Internet down and how soon ?

Faulty character decoding as the last line of anti-spam defense

I receive spam every day. Filtering is in place and everything, but occasionally some garbage gets through. And then I may look through it, briefly, less than a second perhaps before I hit "Delete", but the eye is fast enough to read and understand more than I'd want to. Then you might say such kamikaze message still had succeeded.

Much of the spam I receive is in Russian. As a side note, Russian characters have multiple encodings - WIN1251, KOI8-R, CP866, ISO-8859-5 and the universal UTF-8 come to mind. This means that the mail client has to properly understand the encoding and decode the message so that it can be displayed correctly.

I use Thunderbird, and it is just awful in decoding Russian messages. I don't have any idea why is that, but I have to manually specify encoding for every last message, because they always appear garbled.


But then, the bug becomes an unexpected feature - the spam messages look undecipherable just like legitimate ones, and even though I look at it, nothing is imprinted in my mind, and I just hit "Delete".

Software architecture

is what you explain to somebody else so that he understands the matter.