Alice, Bob, and Mallory

Leif in Clojure

Jonas Elfström — Tue, 01 Nov 2011 03:12:00 +0100

I've been meaning to learn a functional language for years. I don't know what put me off besides the fear of parentheses and the notion that maybe, just maybe, Ruby could be an acceptable Lisp. For all I know, Ruby might be just that, but there's one thing about Ruby that is starting to get more and more unacceptable to me.

Ruby is actually not very well fit to take advantage of the multi-core world we are living in. The GIL and the prevalent mutable state are two big problems. As web applications go, that's often not a problem since it's almost always possible to scale without threads there. But if you want to use the full potential of your multi-core machine in a singe Ruby process/VM, you are out of luck. There are workarounds that get you somewhere and there are implementations of Ruby without a GIL (JRuby and soon Rubinius). Even without the GIL we still have the problem of having a lot of mutable state. That will, probably, make it hard to scale over multiple cores.

A little more than a week ago I watched the presentation Simple Made Easy by Rich Hickey. It's an excellent presentation and one of the best I've seen in years. I was also aware that Hickey is the man who designed Clojure. Finally, I was able to make a choice. I chose to begin to learn Clojure.

I'm still very early in this endeavor. Actually, still trying to decide what books I should read. Anyway, the day after I saw the presentation I downloaded clooj (a simple IDE), read some blogs and some documentation, and started to hack on something. Just to try to get a feel for how hard this was going to be. The parentheses didn't bother me all that much, it was much harder to accept that operators are just functions and that they get no special treatment. It seems you simply have to (+ 31 11) to add 31 and 11. I think I can get over that.

As a side note, it turns out I wrote my first Clojure program on 2011-10-24 and that happened to be the day that John McCarthy, the father of Lisp, died. Yet another of those strange coincidences that seems to happens more often than they should.

The program I wrote is yet another implementation of the stupid simple "algorithm" that I've got to call Leif. If you take Conway's Game of Life, remove the rules and replace them with randomness, you could end up with something like Leif. Quite silly, actually.

Hopefully I will have something more interesting to tell in a couple of ~~weeks~~months, after I've got some more experience with Clojure. For now, Leif will have to do. Here it is.

This is what it looks like at the beginning

and after a couple of minutes it looks like this

One thing that surprised me is that it seems to use both of my cores even though I put no effort whatsoever in making it do so. Probably just the Java GC or something.

Inspired by randomness

Jonas Elfström — Thu, 27 Oct 2011 23:19:00 +0200

Inspired by Randomly not so random I decided to play around with the subject.

Both Java and C# uses a linear congruential generator as their pseudorandom number generators. I found this at MSDN

The current implementation of the Random class is based on a modified version of Donald E. Knuth's subtractive random number generator algorithm."

but I couldn't find any information on what values Microsoft uses for the m, a, and c parameters in their LGC implementation.

The German Federal Office for Information Security (BSI) has established four criteria for quality of deterministic random number generators. They are summarized here: K1 — A sequence of random numbers with a low probability of containing identical consecutive elements.

From the pseudorandom number generator article on Wikipedia. Let's see what we can do about that.

var random = new Random(116793166);
for (int i = 0; i < 9; i++)
{
    Console.Write(random.Next(10) + " ");
}

Outputs:

1 1 1 1 1 1 1 1 1

Did I just break the K1 criteria above? You know I didn't but you might be interested in how I found the seed number. It's easy but also quite computational intensive, that's why I used Parallel.For.

Parallel.For(0, int.MaxValue, i =>
{
    var rnd = new Random(i);
    bool allSame = true;
    for (int j = 0; j < 9; j++)
    {
        allSame = allSame && rnd.Next(10) == 1;
        if (!allSame) break;
    }
    if (allSame)
        Console.WriteLine(i); 
});

It took a couple of minutes for the number 116793166 to show up in my console.

To keep on playing I needed a random string generator. Mine looks like this.

private const string _chars = "abcdefghijklmnopqrstuvwxyz";

private static string RandomString(int size, Random rng)
{
    var buffer = new char[size];
    for (int i = 0; i < size; i++)
        buffer[i] = _chars[rng.Next(_chars.Length)];
    return new string(buffer);
}

If the random generators were the same it looks like it should give the same result as the Java one but RandomString(5, new Random(-229985452)) returns tfsld. Either my RandomString method works different than the Java one or it's the case that Java and .NET has slightly different settings of their LGCs.

Mathematically there's an infinite amount of inputs that results in a returned hello, for instance 1382472294, but here we are limited by the size of our integers.

I found the seed 1382472294 with the following little method and loop

private static bool CompareToRandomString(string str, Random rng) 
{ 
    for (int i = 0; i < str.Length; i++)
    {
        if (_chars[rng.Next(_chars.Length)] != str[i])
            return false;
    }
    return true;
}

...

const string lookingFor = "hello";

Parallel.For(0, int.MaxValue, i =>
{
    var rnd = new Random(i);
    if (CompareToRandomString(lookingFor, rnd))
        Console.WriteLine(i);
});

I leave it up to you to implement a break out of that loop.

Ruby does not use a LGC but a Mersenne twister instead. It's still only pseudorandom so there's no problem finding patters you like and being able to repeat them.

12	srand(2570940381)9.times { print "%d " % rand(10) }

Outputs:
1 2 3 4 5 6 7 8 9

123	charset=('a'..'z').to_asrand(124931)puts (0...4).map{ charset.to_a[rand(charset.size)] }.join

Outputs:
ruby

Sudoku solver in CoffeeScript

Jonas Elfström — Tue, 19 Apr 2011 15:46:00 +0200

CoffeeScript is inspired by Ruby and Python but what's most peculiar with it is that it compiles to JavaScript. The generated JavaScript isn't all that bad and it even passes JavaScript lint without warnings.

"Underneath all of those embarrassing braces and semicolons, JavaScript has always had a gorgeous object model at its heart." - Jeremy Ashkenas

About a week ago I stumbled on the very clever Sudoku solver by Peter Norvig. I have nothing (or at least not much) against Python but I pretty soon checked out the Ruby translations of the solver. I then refactored one of the solutions to get a chance to get to know the algorithm better.

Now that I finally installed CoffeeScript the Sudoku solver came to mind. I dived in head first and got in trouble pretty soon. It turns out that Array Comprehensions in CoffeeScript differs some from the List Comprehensions in Python.

123	def cross(A, B): "Cross product of elements in A and elements in B." return [a+b for a in A for b in B]

returns an one-dimensional array if you call it with two arrays (or strings).

But in CoffeeScript

cross = (A, B) -> (a+b for a in A for b in B)

returns a two-dimensional array.

The jury is still out on if this is intended or not but either way the array comprehensions in CoffeeScript are still very useful.

EDIT 2011-06-02
It's been decided that this is by design. The issue has been closed.

For the cross-function I ended up with

cross = (A, B) ->
  results = []
  for a in A
    for b in B
      results.push a + b
  results

EDIT 2011-06-02
Using `map` I could have got much closer to the Ruby version.

12	cross = (cols, rows) -> [].concat (cols.map (x) -> rows.map (y) -> y+x)...

I'm still more of a map/reduce guy than a list comprehension ninja.

You can find the CoffeScript Sudoku solver as a Gist. Compile it with
coffee -c sudoku.coffee
and then run it with
node sudoku.js

I don't know how I missed it but as Trevor pointed out below all you have to do is
coffee sudoku.coffee

    1 2 3 4 5 6 7 8 9
  |------+-----+-----|
A | 4 8 3|9 2 1|6 5 7|
B | 9 6 7|3 4 5|8 2 1|
C | 2 5 1|8 7 6|4 9 3|
  |------+-----+-----|
D | 5 4 8|1 3 2|9 7 6|
E | 7 2 9|5 6 4|1 3 8|
F | 1 3 6|7 9 8|2 4 5|
  |------+-----+-----|
G | 3 7 2|6 8 9|5 1 4|
H | 8 1 4|2 5 3|7 6 9|
I | 6 9 5|4 1 7|3 8 2|
  |------+-----+-----|

Here's the generated sudoku.js.

This is the only CoffeeScript I've ever written but I already like it (more than JavaScript). Please correct me if I strayed from the CoffeeScript way.

How to test IE9 if you run Windows XP

Jonas Elfström — Mon, 21 Mar 2011 22:20:00 +0100

I'm currently at a company still running Windows XP. Last week Internet Explorer 9 was released and it requires at least Windows Vista. That requirement was a problem because we absolutely had to test our web applications in IE9. Microsoft had released Internet Explorer Application Compatibility VPC Images for earlier versions of IE but not for IE9. At least they still offer a VPC running Vista and that was the path we took to solve our testing problems. If you know of an easier way then please tell!

Download and install Virtual PC.
Edit 2011-09-25 It seems Microsoft has changed the download page. You can get Virtual PC 2007 for older versions of Windows here.
Download and run/unpack the IE8-VIS VHD.
Create a new virtual machine and select the VHD as the harddisk.
In the virtual machine then download and install Vista SP2.
Finally, obviously still in the VPC, download IE9.

Lazy evaluation is no friend of mutable state

Jonas Elfström — Sat, 01 Jan 2011 17:08:00 +0100

A couple of days ago I accidentally landed on a page about implementing merge sort in C#. I thought it would be a nice exercise to try to implement that as a generic method and so I did.

I also wanted to learn more about the characteristics of IEnumerable so I used IEnumerable<T> instead of List<T>. That choice got me in trouble and I opted for help on Stack Overflow.

People said it was sorting correctly but Jon Skeet also asked if I tested it correctly and that I did not. I digged deeper into the problem and extended the question on SO. I had a hunch that it was the mutable state of List<T> and the lazy evaluation of IEnumerable that was the problem but I couldn't quite figure out how.

Along came Kobi and his answer finally made me understand why a .Sort() of the list messed up the result of my sorting.

I then changed the implementation to be fully lazy evaluated and now it looks like this.

public class MergeSort<T>
{
    public IEnumerable<T> Sort(IEnumerable<T> arr)
    {
        if (arr.Count() <= 1) return arr;

        int middle = arr.Count() / 2;
        var left = arr.Take(middle);
        var right = arr.Skip(middle);
        return Merge(Sort(left), Sort(right));
    }

    private static IEnumerable<T> Merge(IEnumerable<T> left, IEnumerable<T> right)
    {
        IEnumerable<T> arrSorted = Enumerable.Empty<T>();

        while (left.Count() > 0 && right.Count() > 0)
        {
            if (Comparer<T>.Default.Compare(left.First(), right.First()) < 0)
            {
                arrSorted=arrSorted.Concat(left.Take(1));
                left = left.Skip(1);
            }
            else
            {
                arrSorted=arrSorted.Concat(right.Take(1));
                right = right.Skip(1);
            }
        }

        return arrSorted.Concat(left).Concat(right);
    }
}

Please be aware that this is but an exercise and not very efficient.

Now to the problems that you can encounter with the above and an explanation to the title of the post. Let's MergeSort a simple List<int> followed by a call to the built-in .Sort() on List<T>.

var ints = new List<int> { 2, 3, 1 };
var mergeSortInt = new MergeSort<int>();
var sortedInts = mergeSortInt.Sort(ints);
// sortedInts.ToList() is {1, 2, 3}
ints.Sort();
// sortedInts.ToList() is {3, 1, 2}

So what's going on here? As far as I can tell it's something like this. sortedInts isn't evaluated until the first call to MoveNext() (or Tolist() or any of those). Before that it only has lazy pointers to the original enumerable values. ints.Sort() messes up the beauty of lazy evaluation by changing the underlying data structure. It can do that because List<T> is mutable (writeable).

But why {3, 1, 2} after ints.Sort()? The original sequence was { 2, 3, 1} and that is what the MergeSort sorted, not by creating a new sequence but only by lazy pointers.

At first the MergeSort sorts like this

but after the source list has been sorted/changed it will do this

What can you do to stop this to happening to you? The boring way is to never use lazy evaluation but that is kind of hard if you happen to use LINQ (and you should, it's great). Another way is to use immutable data structures and that is how functional languages tackles this problem. In C# we have the ReadOnlyCollection and it obviously has no Sort() method.

A nice feature of the MergeSort above is that it has no problems with an immutable collection.

var rints = new ReadOnlyCollection<int>(ints);
var sortedRints = mergeSortInt.Sort(rints);
// sortedRints.ToList() is {1, 2, 3}
ints.Sort();
// sortedRints.ToList() is {3, 1, 2}

What?! How could an immutable collection get messed up like this? It turns out that ReadOnlyCollection<T> is only a facade for mutable collections and that is what bit us here. You have to pass a copy of the list. Example follows.

123	var rints = new ReadOnlyCollection<int>(new List<int>(ints));var sortedRints = mergeSortInt.Sort(rints);ints.Sort();

That also works for List<T>.

var ints = new List<int> { 2, 3, 1 };
var mergeSortInt = new MergeSort<int>();
var sortedInts = mergeSortInt.Sort(new List<int>(ints));
// sortedInts.ToList() is {1, 2, 3}
ints.Sort();
// sortedInts.ToList() is {1, 2, 3}

Finally I would like to send a big thank you to Kobi for sorting out my problems with the original solution.

A strict directed graph

Jonas Elfström — Sun, 24 Oct 2010 00:39:00 +0200

Recently N. asked a question on a list that made my geek-senses tingle. He had a test with 12 levels of questions. The testee should answer 10 questions. If a question was answered correctly the level would go up and if it was not correct it would go down. If a level 12 was answered correctly, another level 12 would be asked and vice versa. The first question would be a level 6. N. wanted to know how many questions of each level he needed.

Pretty soon he got a correct answer and several wrong. One of the incorrect answers was, and I don't like to admit this, provided by me. I took the recursive route (which I very seldom do) and then tried to present the result in ASCII. I failed. The result from the algorithm in itself was correct but the presentation was simplified so much it failed to deliver the correct answer. Here it is:

             06
            05 07
          04 06 08
         03 05 07 09
       02 04 06 08 10
      01 03 05 07 09 11
    01 02 04 06 08 10 12
   01 02 03 05 07 09 11 12
 01 02 03 04 06 08 10 11 12
01 02 03 04 05 07 09 10 11 12

The problem is that this result can be read as if the testee could get two level 2 questions in a row but the rules forbids that.

Then I remembered reading about Graphviz and especially Ruby-Graphviz. That's what I used to generate the graph to the right. The graph itself is nothing but a visualization of all the possible question sequences. To find the answer that N. was looking for you can follow a level from top to bottom and count the number of times it can occur. I have to admit that it's far from the most convenient way but it works.

The result
5 of level 1, 5, 6 & 7.
4 of level 3, 4, 8, 9 & 12.
3 of level 2, 10 & 11.

I had to jump a few hurdles to get there. First my old Ubuntu-install, for unknown reasons, had some ancient version of libfreetype in /usr/local/lib/ that I had to remove. In the end an rm /usr/local/lib/libfreetype* would have been good. I also did a sudo ldconfig but I'm not sure that was necessary. Let's just hope you're on a modern system where a sudo apt-get install graphviz followed by sudo gem install ruby-graphviz will be enough.

The recursive code I wrote produces every single path the testee can take. A graph of that would be quite big since it's 1023 nodes. I then found out about strict digraph in the Graphviz DOT language. It sounded like a perfect fit and it was! One problem though, Ruby-Graphviz didn't seem to know about it. The power of open source software let me patch ruby-graphviz-0.9.18/lib/graphviz/constants.rb so that GRAPHTYPE included it

1234	## Const: graphs type GRAPHTYPE = [ "digraph", "graph", "strict digraph" ]

Another awesome power of open source is Grégoire Lejeune because only hours after me contacting him he added it to the GitHub repository.

Here's the code that produced the Rooted Binary Directed Acyclic Graph (or is it Rooted Directed Binary Acyclic Graph?).

require 'rubygems'
require 'graphviz'

@min_level=1
@max_level=12
@max_depth=10
start_level=6

@g = GraphViz.new(:G, :type => "strict digraph" )

def add_node(level, depth, parent)
  if depth<@max_depth
    current=[level, depth].join(",")

    sub=level<=>@min_level
    add=@max_level<=>level
    add_node(level-sub, depth+1, current)
    add_node(level+add, depth+1, current)

    @g.add_node(current).label=level.to_s
    @g.add_edge(parent, current) unless parent=="00"
  end
end

add_node(start_level, 0, "00")

@g.output( :png => "graph.png" )

With the current parameters this problem might not be the most exciting but I believe that by just modifying them slightly we would, again, reach a wall of complexity.

Encrypt images in JavaScript

Jonas Elfström — Thu, 14 Oct 2010 23:40:00 +0200

In browsers that supports the HTML5 canvas it's possible to read the raw image data. I figured that this makes it possible to encrypt images on the client. Why would one ever want to encrypt images client-side? One use case could be to send images to a server that only those who know the password can decrypt (host-proof hosting).

I used Crypto-JS to encrypt with AES and Rabbit.

First I get the CanvasPixelArray from the ImageData object.

1234	var ctx = document.getElementById('leif') .getContext('2d');var imgd = ctx.getImageData(0,0,width,height);var pixelArray = imgd.data;

The pixel array has four bytes for each pixel as RGBA but Crypto-JS encrypts a string, not an array. At first I used .join() and .split(",") to get from array to string and back. It was slow and the string got much longer than it had to be. Actually four times longer. To save even more space I decided to discard the alpha channel.

function canvasArrToString(a) {
  var s="";
  // Removes alpha to save space.
  for (var i=0; i<pix.length; i+=4) {
    s+=(String.fromCharCode(pix[i])
        + String.fromCharCode(pix[i+1])
        + String.fromCharCode(pix[i+2]));
  }
  return s;
}

That string is what I then encrypt. I sticked to `+=` after reading String Performance an Analysis.

var encrypted = Crypto.Rabbit.encrypt(imageString, password);

I used a small 160x120 pixels image. With four bytes for each pixels that gives 76800 bytes. Even though I stripped the alpha channel the encrypted image still takes up 124680 bytes, 1.62 times bigger. Using `.join()` it was 384736 bytes, 5 times bigger. One cause for it still being larger than the original image is that Crypto-JS returns a Base64 encoded string and that adds something like 37%.

Before I could write it back to the canvas I had to convert it to an array again.

function canvasStringToArr(s) {
  var arr=[];
  for (var i=0; i<s.length; i+=3) {
    for (var j=0; j<3; j++) {
      arr.push(s.substring(i+j,i+j+1).charCodeAt());
    }
    arr.push(255); // Hardcodes alpha to 255.
  }
  return arr;
}

Decryption is simple.

1234	var arr=canvasStringToArr( Crypto.Rabbit.decrypt(encryptedString, password));imgd.data=arr;ctx.putImageData(imgd,0,0);

Tested in Firefox, Google Chrome, WebKit3.1 (Android 2.2), iOS 4.1, and a very recent release of Opera.

Browser / Action in milliseconds	Enc. Rabbit	Dec. Rabbit	Enc. AES	Dec. AES
Google Chrome 6.0.472.62 C2D@1.86GHz	136	130	236	222
Opera 10.63 P4HT@3GHz	246	252	438	437
Google Chrome 6.0.472.63 P4HT@3GHz	280	648	303	547
Firefox 3.6.10 Phenom II X4 945@3GHz	494	321	1876	1745
Firefox 3.6.10 i5@3,5GHz	366	193	1639	1410
Firefox 3.6.10 C2D@1.86GHz	760	367	2417	1983
Firefox 3.6.10 P4HT@3GHz	880	440	4000	3500
Nokia N900 Chrome	1764	1975	2509	2508
WebKit 3.1 (HTC Desire)	2000	2200	3300	3400
iPhone 3GS iOS 4.1	2130	2071	7198	7131
N900 Firefox Mobile	3411	3508	19308	19466
N900 native (MicroB)	4681	4300	24560	20973
X10 Mini Pro, Android 1.6	7464	7747	timeout	timeout

A demo can be found here.

EDIT 2010-10-17

Warning I've noticed that the encrypted strings aren't compatible between browsers. At least not in between Google Chrome and Firefox. I don't know why.

I also tried to add deflate/inflate and that compresses my test image to a third of the raw size and in Google Chrome it also halved the execution time. In Firefox Rabbit got about 50% slower and AES about 50% faster with deflate/inflate.

Here's a demo of this.

EDIT 2010-10-21

Added a preprocessing filter of the raw image data inspired by PNG type 1 sub filter. Presented my idea to Fredrik and he returned a formula. Thanks!
The result wasn't what I had hoped for, a measly 6% smaller. I also tried to save the image as real PNG and that one is 20480 bytes. 20480*1.37=28057 bytes (Base64 overhead) and my homemade format is 38336 bytes. Not a fantastic result but not that horrible either.

Here's a demo of this.

To hack a Superpower

Jonas Elfström — Thu, 14 Oct 2010 21:57:00 +0200

I was interviewed in a swedish documentary called Att hacka en stormakt (To hack a Superpower). It's mostly in swedish but Brian Koref (US AF) and Tom Talleur (NASA) are interviewed in english. It can be streamed from SVTPlay (at least if you live in Sweden) until 2010-11-09.

It covers the cracking frenzy of a couple Swedish youths back in 1996. They happened to break into a server I was responsible for and from that server they logged in to a machine with a .mil-address. That's how we got involved.

The event has been covered before in a radio documentary called Svenska hackers (Swedish hackers).

Did Little Bobby Tables migrate to Sweden?

Jonas Elfström — Thu, 23 Sep 2010 22:36:00 +0200

As you may have heard, we've had a very close election here in Sweden. Today the Swedish Election Authority published the hand written votes. While scanning through them I happened to notice

R;13;Hallands län;80;Halmstad;01;Halmstads västra valkrets;0904;Söndrum 4;pwn DROP TABLE VALJ;1

The second to last field¹ is the actual text on the ballot². Could it be that Little Bobby Tables is all grown up and has migrated to Sweden? Well, it's probably just a joke but even so it brings questions since an SQL-injection on election data would be very serious.

Someone even tried to get some JavaScript in there:

R;14;Västra Götalands län;80;Göteborg;03;Göteborg, Centrum;0722;Centrum, Övre Johanneberg;(Script src=http://hittepa.webs.com/x.txt);1

I'm pleased to see that they published the list as text and not HTML. This hacker/joker voter seems to think³ they "censored" his vote/script. I'm not so sure about that, a more reasonable explanation is that they couldn't enter brackets, quotation marks, and so on.

There are also a couple of URL:s to online retailers and three votes on a conspiracy friendly site. I chose not to link to any of those.

This time the pen and paper scripting attack failed. Let's hope it stays that way.

PS. Someone noticed that there are no votes from Stockholm in there right now (2010-09-24). I asked the Swedish Election Authority about this and it turns out that The County Administrative Board (Länsstyrelsen) gets two months to register all the handwritten votes. There's a good chance that those will bring more attempts like the ones above. DS.

EDIT 2010-09-24
Links:
Aftonbladet DN SvD Expressen SVT - all in Swedish.
Slashdot BBC Wired

¹The name of the party, not a name of a person.
²Almost all Swedish voters use the preprinted ballots but you are allowed to write your own by hand.
³The site disappeared after this post was published.

A simple loop

Jonas Elfström — Mon, 21 Jun 2010 21:56:00 +0200

There's more than one way to skin a cat and the same is true for looping in Ruby. This is a silly post with a silly number of ways to

print the integers from 1 to 10.

If you're a BASIC-programmer and are getting your feet wet with Ruby, you might end up with something like this.

1234	while i<=10 do puts i i+=1end

That and the following for-loop is not the usual Ruby way of looping.

123	for i in 1..10 puts iend

Instead rubyists often iterates over ranges or arrays with each.

(1..10).each {|i| puts i }

But for simple integer loops like this, we also have upto

1.upto(10) {|i| puts i }

and times.

10.times {|i| puts i+1}

Here's where I should've stopped but I can't help myself, I just have to show off with some Symbol#to_proc "magic".

(0..10).inject(&:p)

The above works because p is an alias of puts and & converts the symbol :p to a proc that is called with the numbers in the range as parameters.

The alias p also gives us, what I think has to be, the shortest possible way.

p *1..10

You could argue that it's a bad thing that there are so many ways to do something as simple as this. But I see no big problem here, if any at all, even though these are hardly all possible ways to loop over integers in Ruby.