Links

Ben Laurie blathering

9 Oct 2012

I Hate Java, or How To Make Ant Run Javascript

Filed under: Open Source,Programming — Ben @ 18:29

I am writing this up because it took me all day to figure out, and the Interwebz was surprisingly unhelpful!

So, if you’re trying to configure EJBCA, or maybe something else whose author thought that including Javascript in a makefile was a really great idea, you may run into this error, when you run ant:

/disk1.1/usr/home/ben/software/unpacked/ejbca_4_0_12/propertyDefaults.xml:11: Unable to create javax script engine for javascript

If you Google for that, maybe you’ll now find this article, but as I write, you get a lot of hits on people with this problem, and a paucity of useful advice. So, here’s how I fixed it…

First off, the core problem is ant needs BSF (the Bean Scripting Framework) in order to run scripts at all, and also Rhino, in order to run Javascript. And to complicate matters further, BSF needs Jakarta Commons Logging. It turns out you can get these in various ways, some of which I’ve tested. In particular, you can generally download zips of jars (and other junk) from the projects’ sites (generally described as binaries, for some reason), or you might be able to install packages or ports. In practice I got BSF from its own site, logging by installing a port, and I tried Rhino both ways. So, once you’ve got the relevant jars, the next bit of magic is to let Ant know about them.

You can either install them in Ant’s library directory, wherever that is, or name them in a -lib argument on the command line. Like this:

ant -lib '../bsf-2.4.0/lib/bsf.jar:/usr/local/share/java/classes/commons-logging.jar:/usr/local/share/java/rhino/rhino.jar' <target>

And Robert, as they say, is a close relative. Update: putting them on the classpath does work, so long as you remember the classpath is separated by colons and not semicolons. On some platforms. However, this does not make jrunscript work.

So, this solved my immediate problem, but I’m now really curious to know how jrunscript can be made to understand JS. Coz right now, despite having all the stuff I need to make Ant do it, I get

script engine for language js can not be found

How about it, lazyweb?

BTW, I hate Java.

14 Aug 2012

Verifiable Logs: Solving The “Cryptocat Problem”

Filed under: Certificate Transparency,Open Source,Security — Ben @ 15:43

There’s been a lot of heat about Cryptocat lately. But not much light. In summary, one side says you can’t trust software you download from the ‘net to not reveal all your secrets, and the other side says that’s all we got, so suck it up. So, how do we fix this problem?

First off, lets take a look at the core of the problem: if you download something from the ‘net, how can you be sure what you got is what was advertised? One of the much-lauded benefits of open source is that it can be reviewed – experts can take a look and see whether it really does what it says. So, that deals with half the problem. But how do we know we got what the experts reviewed?

I propose that the answer is publicly verifiable logs. The idea is that anyone can operate a log of “stuff” that can be verified by anyone else. What do I mean by “verified”? I mean that if two people see the log, they can mutually check that they saw the same thing. Of course, this is trivial if you are prepared to send the whole log to each other – just check they’re identical. The trick is to do this verification efficiently.

Luckily we have a way to do that: Merkle Trees. These allow us to summarise the log with a short chunk of binary (the “root”). If we both get the same root, then we both have the same log. What’s more, they also allow an efficient proof that any particular item is in the log – given the item, the log can show a chain of hashes leading to the root. This chain proves that the item actually is in the log summarised by the root.

What’s more, with only a bit more cunningness, we can also efficiently show that any version of the log (with more data appended) contains any prior version. In other words, we can show that the log never deletes anything, but only grows by adding new things at the end.

Got it? To reiterate: it is possible to create a log that can demonstrate that everyone sees the same version, and that as it grows, everyone continues to see the same data added to it. What’s more, these things can be done efficiently[1].

Now we have that piece of machinery, how do we use it to solve the “Cryptocat problem”? Simple: every time Cryptocat does a new release, it pushes a copy of the source into the verifiable log. Every time you download Cryptocat, you verify that the version you are given is in the public log, and refuse to run it if not. And we’re done.

If Cryptocat ever decides to release a version that, say, reveals your keys, or decrypts your chats for a third party, then that version is on display for all to see. Cryptocat will get caught – and likely caught quite quickly. If Cryptocat tries to avoid this publication, then you won’t run it, so you’ll be safe.

Admittedly this does not actually _prevent_ Cryptocat from shafting you, but it does mean it is very unlikely to get away with it, and having done it once, it will probably not get the chance to do it to anyone again…

Note that it doesn’t matter if the author of Cryptocat is the one who made the change, or someone who hacked his site, or a man-in-the-middle. If they do not publish source, then you won’t run it. And if they do publish source, they get caught.

Incidentally, I originally proposed publicly verifiable logs for fixing PKI but they have many uses. Also, for Certificate Transparency, we are implementing a publicly verifiable log. I would be very happy to help with a version for logging software instead of certificates.

[1] To get an idea of what I mean by “efficiently” a proof that two log versions are consistent or that a particular item is in a particular log version consists of log_2(n) hashes, where n is the number of items in the log. So, for a log with a billion items, this proof would have around 30 entries, each, say, 32 bytes long. So, it takes me less than 1 kB for a proof about a log with a billion entries. How about a trillion? Just ten more entries, i.e. under 1,300 bytes.

22 May 2012

Factoring RSA

Filed under: Crypto,Open Source,Security — Ben @ 13:13

Apparently I have not blogged about factoring weak RSA keys before. Well, I guess I have now 🙂

One thing I’ve been wondering ever since that research was done is: is there anything OpenSSL could do about this? I’ve been assuming OpenSSL was used to generate at least some of those keys.

So, I was interested to read this analysis. First off, it shows that it is highly likely that the bad keys were generated by OpenSSL and one other (proprietary) implementation. However, I have to argue with some details in an otherwise excellent writeup.

Firstly, this canard irritates me:

Until version 0.9.7 (released on Dec 31, 2002) OpenSSL relied exclusively on the /dev/urandom source, which by its very definition is non-blocking. If it does not have enough entropy, it will keep churning out pseudo-random numbers possibly of very poor quality in terms of their unpredictability or uniqueness.

By definition? Whose definition? When did Linux man pages become “by definition”? In FreeBSD, which, IMO, has a much sounder approach to randomness, urandom does block until it has sufficient entropy. Is poor design of the OS OpenSSL’s fault?

Which brings me to

FreeBSD prior to version 5 posed its own problem, since its /dev/random source silently redirected to /dev/urandom.

Well. Modern FreeBSD versions link /dev/urandom to /dev/random. That doesn’t seem like a material change to me. I’m pretty sure that the implementation changed, too – perhaps that’s more important than filenames?

Finally, in the summary:

Some unfortunate choices by the OpenSSL library didn’t help either.

Oh really? So the fact that a 10-year-old version of OpenSSL used a device that in some OSes is not very well designed is contributing to this problem? I’m finding this a little hard to swallow. Also, “choices”? What choices? Only one choice is mentioned.

The real problem is, IMNSHO: if you provide a weak random number source, then people will use it when they shouldn’t. The problem here is with the OS that is providing the randomness, not the OpenSSL library. So, why is the OS (which I am prepared to bet is Linux) not even mentioned?

1 Mar 2012

Certificate Transparency: Spec and Working Code

Filed under: Certificate Transparency,Crypto,Open Source — Ben @ 17:29

Quite a few people have said to me that Certificate Transparency (CT) sounds like a good idea, but they’d like to see a proper spec.

Well, there’s been one of those for quite a while, you can find the latest version in the code repository, or for your viewing convenience, I just made an HTML version.

Today, though, to go with that spec, I’m happy to announce working code for a subset of the protocol. This covers the trickiest part – a fully backwards compatible SSL handshake between servers and clients. The rest of the protocol will necessarily all be new code for interacting with the log server and other new components, and so should not have these issues.

If you build the code according to the README, then you will find instructions in test/README for the demo.

What this does, in short, is the following:

  • Run a CT log server. Currently this has no persistence across runs, but does keep a full log in memory.
  • Issue a self-signed server certificate. A CA issued certificate would also be fine, but not so easy to automate for a demo.
  • Use the CT client to register that certificate with the log server and to obtain a log proof for it.
  • Use the CT client to convert that proof into a fake “certificate” which can be included in the certificate chain in the TLS handshake.
  • Run an Apache 2.2 instance to serve the self-signed certificate and the log proof certificate. Note that Apache is unmodified, all that is needed is appropriate configuration.
  • Use the CT client to connect to the Apache instance and verify the presented log proof.
  • You can also connect to Apache with an existing browser to check that you can still access the site despite the presence of the log proof.

There’s plenty more to be done, but this is the part that needs the earliest scrutiny, since we are bending the rules to get back compatibility and avoid the need to change server software. Client software has to change anyway to provide any benefit to users, so that’s less of a worry.

We welcome discussion, suggestions and questions on the mailing list.

How “Free” Leads to Closed

Filed under: Open Source — Ben @ 11:48

The FSF is fond of banging on about how the GPL is more “free” than other open source licences, even though it is actually a more restrictive licence than many others (for example, the Apache Licence).

So I find it ironic that the much anticipated Raspberry Pi is about as un-free as it is possible to be. Yes, it runs Linux. Can you run anything else? No, because the chipset is not documented, it is impossible to write drivers for any other OS. Its hard to imagine what would have happened if the dominant open OS was BSD or Apache licensed, but it is interesting to speculate: would this have happened in that world? Possibly not – one of the reasons the ASF adopted a more free licence was precisely because it is business-friendly. Would chipmakers obsessively protect their chip specs in that world? Who knows, but I like to think not.

In any case, if I were building a device like the Pi, I would not be using undocumented chips.

1 Oct 2011

Open Source Transcription Software Developer

Filed under: Open Data,Open Source,Programming — Ben @ 18:06

Since we set up FreeBMD, FreeREG and FreeCEN things have come a long way, and so we’re revisiting how we do transcription. Those great guys at Zooniverse have released their Scribe transcription software, which they developed to use with Old Weather and Ancient Lives (and more to come), as open source.

We are working with them to develop a new transcription platform for genealogical records, based on Scribe, and we want to hire a developer to help us with it. Scribe itself is written in Ruby, so some familiarity with that would help. We also use Python and EC2, so knowing about those would be good, too. And the front-end is sure to be using Javascript, so there’s another tickbox to tick.

Finally, we intend to open source everything, and so a developer used to working in an open source community would be helpful.

Everything is negotiable. FreeBMD does not have offices, so this would be “work from home” (or the beach, or whatever suits you).

If you’re interested, send email to freebmd-sd@links.org. Feel free to forward this post, of course.

27 Mar 2011

ZFS Part 3: Replacing Dead Disks

Filed under: Open Source — Ben @ 16:39

As discussed in my previous article, if a disk fails then a ZFS system will just carry on as if nothing has happened. Of course, we’d like to restore the system to its former redundant glory, so here’s how…

Once more, we simulate a failure by removing the primary disk, but this time replace it with a new unformatted disk (I guess if the new disk was already bootable you’d need to fix that first).

Let’s assume we’re several years down the line and no longer have any documentation at all. First off, find your disks by inspecting dmesg. As before we have ad4 and ad8. ad4 is the new disk.

# diskinfo -v ad4 ad8
ad4
        512             # sectorsize
        500107862016    # mediasize in bytes (466G)
        976773168       # mediasize in sectors
        0               # stripesize
        0               # stripeoffset
        969021          # Cylinders according to firmware.
        16              # Heads according to firmware.
        63              # Sectors according to firmware.
        S20BJ9AB212006  # Disk ident.

ad8
        512             # sectorsize
        500107862016    # mediasize in bytes (466G)
        976773168       # mediasize in sectors
        0               # stripesize
        0               # stripeoffset
        969021          # Cylinders according to firmware.
        16              # Heads according to firmware.
        63              # Sectors according to firmware.
        9VMYLC5V        # Disk ident.

This time they are conveniently exactly the same size, despite having diffferent manufacturers (Samsung and Seagate respectively). We already know from the first article in this series that we can deal with disks that don’t look the same, and in any case only 250GB is currently replicated. So, let’s partition the new disk as the old one…

# gpart show ad8
=>       34  976773101  ad8  GPT  (466G)
         34        128    1  freebsd-boot  (64K)
        162    4194304    2  freebsd-swap  (2.0G)
    4194466  484202669    3  freebsd-zfs  (231G)
  488397135  488376000    4  freebsd-zfs  (233G)

# gpart show -l ad8
=>       34  976773101  ad8  GPT  (466G)
         34        128    1  (null)  (64K)
        162    4194304    2  swap8  (2.0G)
    4194466  484202669    3  system8  (231G)
  488397135  488376000    4  scratch8  (233G)

# gpart create -s gpt ad4
ad4 created
# gpart add -b 34 -s 128 -t freebsd-boot ad4
ad4p1 added
# gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 ad4
bootcode written to ad4
# gpart add -s 4194304 -t freebsd-swap -l swap4 ad4
ad4p2 added
# gpart add -s 484202669 -t freebsd-zfs -l system4 ad4
ad4p3 added
# gpart add -t freebsd-zfs -l scratch4 ad4
ad4p4 added
# gpart show ad4
=>       34  976773101  ad4  GPT  (466G)
         34        128    1  freebsd-boot  (64K)
        162    4194304    2  freebsd-swap  (2.0G)
    4194466  484202669    3  freebsd-zfs  (231G)
  488397135  488376000    4  freebsd-zfs  (233G)

Now we’re ready to reattach the disk to the various filesystems.

First the swap. Since we can’t remove the dead disk from the gmirror setup, first we forget then add the new swap partition back in.

# gmirror forget swap
# gmirror insert -h -p 1 swap /dev/gpt/swap4
# gmirror status
       Name    Status  Components
mirror/swap  DEGRADED  gpt/swap8
                       gpt/swap4 (29%)

and after a while

# gmirror status
       Name    Status  Components
mirror/swap  COMPLETE  gpt/swap8
                       gpt/swap4

Next the main filesystem. In this case, since the new device has the same name as the old one, we can just write

# zpool replace system /dev/gpt/system4
If you boot from pool 'system', you may need to update
boot code on newly attached disk '/dev/gpt/system4'.

Assuming you use GPT partitioning and 'da0' is your new boot disk
you may use the following command:

        gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 da0

Once more we’ve already done this step, so no need to do it again. Note, this command took a little while, don’t be alarmed!

# zpool status
  pool: scratch
 state: ONLINE
 scrub: none requested
config:

        NAME            STATE     READ WRITE CKSUM
        scratch         ONLINE       0     0     0
          gpt/scratch8  ONLINE       0     0     0

errors: No known data errors

  pool: system
 state: DEGRADED
status: One or more devices is currently being resilvered.  The pool will
        continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
 scrub: resilver in progress for 0h0m, 9.77% done, 0h2m to go
config:

        NAME                   STATE     READ WRITE CKSUM
        system                 DEGRADED     0     0     0
          mirror               DEGRADED     0     0     0
            gpt/system8        ONLINE       0     0     0
            replacing          DEGRADED     0     0     0
              gpt/system4/old  UNAVAIL      0     0     0  cannot open
              gpt/system4      ONLINE       0     0     0  221M resilvered

errors: No known data errors

and after not very long

# zpool status
  pool: scratch
 state: ONLINE
 scrub: none requested
config:

        NAME            STATE     READ WRITE CKSUM
        scratch         ONLINE       0     0     0
          gpt/scratch8  ONLINE       0     0     0

errors: No known data errors

  pool: system
 state: ONLINE
 scrub: resilver completed after 0h1m with 0 errors on Sun Mar 27 13:04:02 2011
config:

        NAME             STATE     READ WRITE CKSUM
        system           ONLINE       0     0     0
          mirror         ONLINE       0     0     0
            gpt/system8  ONLINE       0     0     0
            gpt/system4  ONLINE       0     0     0  2.21G resilvered

errors: No known data errors

And we’re all good, back to where we were before. Reboot to check everything is fine.

Note, by the way, that all of this was done on a live system in multi-user mode. Apart from the occasional reboot there was no loss of service whatsoever.

Also, because the primary disk didn’t really fail, if I wanted I could put it in my other machine and end up with a working replicated system there without any need for setup.

There is one niggling question remaining: I started off with one 250 GB and one 500 GB disk. I now have two 500 GBs, which means the non-redundant scratch file system I had before could now become redundant. Or they could become part of the system pool. Or they could become a bigger non-redundant scratch filesystem.

In the end I decided to do the simplest thing, which is to make the scratch partitions part of the larger system partition. If I ever need to rearrange that is always possible either with the help of an additional disk or, even, with less safety, by taking one of the disks out of the pools and rearranging onto that (see a description of doing this kind of thing on freenas).

So, to make them part of the existing pool, first destroy the scratch filesystem (if I’d already used it I’d have to copy it before I started, but since I haven’t I can just blow it away). Since we mounted the pool direct, we destroy it with zpool:

# zpool destroy scratch

(and we can confirm it has gone with zpool list and zfs list). Just
for naming sanity, I rename the two scratch partitions:

# gpart modify -i 4 -l system8.p2 ad8
ad8p2 modified
# gpart modify -i 4 -l system4.p2 ad4
ad4p2 modified

and since those aren’t reflected in /dev/gpt, reboot. Then finally

# zpool add system mirror /dev/gpt/system4.p2 /dev/gpt/system8.p2

and presto

# zpool list
NAME SIZE USED AVAIL CAP HEALTH ALTROOT
system 463G 2.21G 461G 0% ONLINE -

ZFS Part 2: Disk Failure

Filed under: Open Source — Ben @ 16:12

Before I’m ready to trust ZFS I need to make sure I can replace a disk when it dies. With the setup described here, as a first experiment I removed the primary disk.

So, power down and remove the primary disk (ad4). Note that if you’re doing this on the Proliant system I mentioned, then you really should replace the drive mount (it is needed for cooling). Luckily I have a spare system so I just borrowed one.

Reboot. Comes up fine on the secondary disk without further intervention.

$ zpool status
  pool: scratch
 state: ONLINE
 scrub: none requested
config:

        NAME            STATE     READ WRITE CKSUM
        scratch         ONLINE       0     0     0
          gpt/scratch8  ONLINE       0     0     0

errors: No known data errors

  pool: system
 state: DEGRADED
status: One or more devices could not be opened.  Sufficient replicas exist for
        the pool to continue functioning in a degraded state.
action: Attach the missing device and online it using 'zpool online'.
   see: http://www.sun.com/msg/ZFS-8000-2Q
 scrub: none requested
config:

        NAME             STATE     READ WRITE CKSUM
        system           DEGRADED     0     0     0
          mirror         DEGRADED     0     0     0
            gpt/system8  ONLINE       0     0     0
            gpt/system4  UNAVAIL      0     0     0  cannot open

errors: No known data errors

Note that the system pool is now degraded. How would we have known if we hadn’t checked? Well, turns out we missed something from the previous setup.

We should have put

daily_status_zfs_enable="YES"
daily_status_gmirror_enable="YES"

in /etc/periodic.conf. Then in the daily mail we’d see:

Checking status of zfs pools:
  pool: system
 state: DEGRADED
status: One or more devices could not be opened.  Sufficient replicas exist for
	the pool to continue functioning in a degraded state.
action: Attach the missing device and online it using 'zpool online'.
   see: http://www.sun.com/msg/ZFS-8000-2Q
 scrub: none requested
config:

	NAME             STATE     READ WRITE CKSUM
	system           DEGRADED     0     0     0
	  mirror         DEGRADED     0     0     0
	    gpt/system8  ONLINE       0     0     0
	    gpt/system4  UNAVAIL      0     0     0  cannot open

errors: No known data errors

Checking status of gmirror(8) devices:
       Name    Status  Components
mirror/swap  DEGRADED  gpt/swap8

So remember, boys and girls, read your daily mails!

So far, so good. One disk failed, the system came back up without intervention, and would have alerted us in daily mails had we configured it correctly (of course it now is). So what happens if we put the disk back in? Since we’ve modified the other disk in the meantime, we’d hope that would get reconciled. Let’s see…

Power down and replace the missing disk, reboot.

Now we see

$ zpool status
  pool: scratch
 state: ONLINE
 scrub: none requested
config:

        NAME            STATE     READ WRITE CKSUM
        scratch         ONLINE       0     0     0
          gpt/scratch8  ONLINE       0     0     0

errors: No known data errors

  pool: system
 state: ONLINE
 scrub: resilver completed after 0h0m with 0 errors on Sat Mar 26 10:48:56 2011
config:

        NAME             STATE     READ WRITE CKSUM
        system           ONLINE       0     0     0
          mirror         ONLINE       0     0     0
            gpt/system8  ONLINE       0     0     0
            gpt/system4  ONLINE       0     0     0  345K resilvered

errors: No known data errors

$ gmirror status
       Name    Status  Components
mirror/swap  COMPLETE  gpt/swap4
                       gpt/swap8

and there we are, back to where we started. But suppose the disk had really failed, then what? See the next exciting installment!

18 Mar 2011

Completely Redundant Disks with ZFS and FreeBSD

Filed under: General,Open Source — Ben @ 15:00

A while back, I bought a ReadyNAS device for my network, attracted by the idea of RAID I can grow over time and mirrored disks.

Today I just finished building the same thing “by hand”, using FreeBSD and ZFS. At a fraction of the cost. Here’s how.

First off, I bought this amazing bargain: an HP ProLiant MicroServer. These would be cheap even at list price, but with the current £100 cashback offer, they’re just stupidly cheap. And rather nice.

Since I want to cater for a realistic future, I am assuming by the time I need to replace a drive I will no longer be able to buy a matching device, so I started from day one with a different second drive (the primary is 250 GB, secondary is 500 GB – both Seagate, which was not the plan, but I’ll remedy that in the next episode). I also added an extra 1GB of RAM to the machine (this is important for ZFS which is apparently not happy with less than 2GB of system RAM).

I then followed, more or less, Pawel’s excellent instructions for creating a fully mirrored setup. However, I had to deviate from them somewhat, so here’s my version.

The broad overview of the process is as follows

  1. Install FreeBSD on the primary disk, using a standard sysinstall.
  2. Create and populate gmirror and ZFS partitions on the secondary disk.
  3. Boot from the primary disk, but mount the secondary.
  4. Create and populate gmirror and ZFS partitions on the primary disk.
  5. Use excess secondary disk as scratch.

In my case the two disks are ad4 (primary, 250 GB) and ad8 (secondary, 500 GB). Stuff I typed is in italic.

Since we need identical size partitions for the mirror, we need to simulate the first disk (since it happens to be smaller). Get the disk’s size

# diskinfo -v /dev/ad4
/dev/ad4
        512             # sectorsize
        250059350016    # mediasize in bytes (233G)
        488397168       # mediasize in sectors
        0               # stripesize
        0               # stripeoffset
        484521          # Cylinders according to firmware.
        16              # Heads according to firmware.
        63              # Sectors according to firmware.
        9VMQN8T5        # Disk ident.

Create a memory disk the same size. Note that the sector sizes must match!

# mdconfig -a -t swap -s 488397168
md0

Verify they are the same.

# diskinfo -v /dev/ad4 /dev/md0
/dev/ad4
        512             # sectorsize
        250059350016    # mediasize in bytes (233G)
        488397168       # mediasize in sectors
        0               # stripesize
        0               # stripeoffset
        484521          # Cylinders according to firmware.
        16              # Heads according to firmware.
        63              # Sectors according to firmware.
        9VMQN8T5        # Disk ident.

/dev/md0
        512             # sectorsize
        250059350016    # mediasize in bytes (233G)
        488397168       # mediasize in sectors
        0               # stripesize
        0               # stripeoffset

Now partition the memory disk as we will the first disk later on.

# gpart create -s gpt md0
md0 created
# gpart add -b 34 -s 128 -t freebsd-boot md0
md0p1 added
# gpart add -s 2g -t freebsd-swap -l swap1 md0
md0p2 added
# gpart add -t freebsd-zfs -l systemx md0
md0p3 added

and show the resulting sizes

# gpart show md0
=>       34  488397101  md0  GPT  (233G)
         34        128    1  freebsd-boot  (64K)
        162    4194304    2  freebsd-swap  (2.0G)
    4194466  484202669    3  freebsd-zfs  (231G)

Now blow away the memory disk, we don’t need it any more.

# mdconfig -d -u 0

Create the partitions on the second disk.

# gpart create -s gpt ad8
ad8 created
# gpart add -b 34 -s 128 -t freebsd-boot ad8
ad8p1 added
# gpart add -s 2g -t freebsd-swap -l swap1 ad8
ad8p2 added
# gpart add -s 484202669 -t freebsd-zfs -l system8 ad8
ad8p3 added

And eat the rest of the disk as a scratch area (this area will not be mirrored, and so should only be used for disposable stuff).

# gpart add -t freebsd-zfs -l scratch8 ad8
ad8p4 added

Check it matches the md0 simulation

# gpart show ad8
=>       34  976773101  ad8  GPT  (466G)
         34        128    1  freebsd-boot  (64K)
        162    4194304    2  freebsd-swap  (2.0G)
    4194466  484202669    3  freebsd-zfs  (231G)
  488397135  488376000    4  freebsd-zfs  (233G)

And don’t forget to set up the bootloader

# gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 ad8
bootcode written to ad8

I realised as this point I had intended to label everything with an 8, to match the unit number, and had not done so for swap, so for completeness, here’s how you fix it

# gpart modify -i 2 -l swap8 ad8
ad8p2 modified
# gpart show -l ad8
=>       34  976773101  ad8  GPT  (466G)
         34        128    1  (null)  (64K)
        162    4194304    2  swap8  (2.0G)
    4194466  484202669    3  system8  (231G)
  488397135  488376000    4  scratch8  (233G)

Note that the label change is not reflected by the device names in /dev/gpt, which is needed for the next step, so at this point I rebooted.

Now set up the swap mirror.

# gmirror label -F -h -b round-robin swap /dev/gpt/swap8

Create the ZFS storage pool called system, consisting only of our system8 partition.

# zpool create -O mountpoint=/mnt -O atime=off -O setuid=off -O canmount=off system /dev/gpt/system8

And create a dataset – “mountpoint=legacy” stops ZFS from managing it.

# zfs create -o mountpoint=legacy -o setuid=on system/root

Mark it as the default bootable dataset.

# zpool set bootfs=system/root system

Mount it

# mount -t zfs system/root /mnt
# mount
/dev/ad4s1a on / (ufs, local)
devfs on /dev (devfs, local, multilabel)
system/root on /mnt (zfs, local, noatime)

And create the remaining mountpoints according to Pawel’s suggested layout…

# zfs create -o compress=lzjb system/tmp
# chmod 1777 /mnt/tmp
# zfs create -o canmount=off system/usr
# zfs create -o setuid=on system/usr/local
# zfs create -o compress=gzip system/usr/src
# zfs create -o compress=lzjb system/usr/obj
# zfs create -o compress=gzip system/usr/ports
# zfs create -o compress=off system/usr/ports/distfiles
# zfs create -o canmount=off system/var
# zfs create -o compress=gzip system/var/log
# zfs create -o compress=lzjb system/var/audit
# zfs create -o compress=lzjb system/var/tmp
# chmod 1777 /mnt/var/tmp
# zfs create -o canmount=off system/usr/home

And create one for each user:

# zfs create system/usr/home/ben

Now, at a slightly different point from Pawel, I edit the various config files. First /boot/loader.conf. Note that some of these are commented out: this is because, although they appear in Pawel’s version, they are already built into the kernel (this is because I use a GENERIC kernel and he uses a stripped-down one). Including them seems to cause problems (particularly geom_part_gpt, which causes a hang during boot if present).

geom_eli_load=YES
#geom_label_load=YES
geom_mirror_load=YES
#geom_part_gpt_load=YES
zfs_load=YES
vm.kmem_size=3G # This should be 150% of your RAM.

Enable ZFS

# echo zfs_enable=YES >> /etc/rc.conf

Change fstab for the new layout (note, you might want to edit these in – for example, my system had an entry for cd drives).

# cat > /etc/fstab
system/root / zfs rw,noatime 0 0
/dev/mirror/swap.eli none swap sw 0 0
^D

The .eli extension here is magic: geom_eli finds it at startup and automatically encrypts it.

Set the work directory for ports (so that it uses the faster compression scheme during builds).

# echo WRKDIRPREFIX=/usr/obj >> /etc/make.conf

These need to be done now because the next step is to copy the entire install to the new ZFS filesystem. Note that this particular command pastes completely incorrectly from Pawel’s blog post so be careful!

# tar -c --one-file-system -f - . | tar xpf - -C /mnt/

Tar can’t copy some types of file, so expect an error or two at this point:

tar: ./var/run/devd.pipe: tar format cannot archive socket
tar: ./var/run/log: tar format cannot archive socket
tar: ./var/run/logpriv: tar format cannot archive socket

Just for fun, take a look at the ZFS we’ve created so far…

# zfs list
NAME USED AVAIL REFER MOUNTPOINT
system 1.12G 225G 21K /mnt
system/root 495M 225G 495M legacy
system/tmp 30K 225G 30K /mnt/tmp
system/usr 652M 225G 21K /mnt/usr
system/usr/home 50K 225G 21K /mnt/usr/home
system/usr/home/ben 29K 225G 29K /mnt/usr/home/ben
system/usr/local 297M 225G 297M /mnt/usr/local
system/usr/obj 21K 225G 21K /mnt/usr/obj
system/usr/ports 190M 225G 159M /mnt/usr/ports
system/usr/ports/distfiles 30.8M 225G 30.8M /mnt/usr/ports/distfiles
system/usr/src 165M 225G 165M /mnt/usr/src
system/var 100K 225G 21K /mnt/var
system/var/audit 21K 225G 21K /mnt/var/audit
system/var/log 35K 225G 35K /mnt/var/log
system/var/tmp 23K 225G 23K /mnt/var/tmp

Unmount ZFS

# zfs umount -a

And the one we mounted by hand

# umount /mnt

And set the new ZFS-based system to be mounted on /

# zfs set mountpoint=/ system

And … reboot! (this is the moment of truth)

After the reboot, you should see

$ mount
system/root on / (zfs, local, noatime)
devfs on /dev (devfs, local, multilabel)
system/tmp on /tmp (zfs, local, noatime, nosuid)
system/usr/home/ben on /usr/home/ben (zfs, local, noatime, nosuid)
system/usr/local on /usr/local (zfs, local, noatime)
system/usr/obj on /usr/obj (zfs, local, noatime, nosuid)
system/usr/ports on /usr/ports (zfs, local, noatime, nosuid)
system/usr/ports/distfiles on /usr/ports/distfiles (zfs, local, noatime, nosuid)
system/usr/src on /usr/src (zfs, local, noatime, nosuid)
system/var/audit on /var/audit (zfs, local, noatime, nosuid)
system/var/log on /var/log (zfs, local, noatime, nosuid)
system/var/tmp on /var/tmp (zfs, local, noatime, nosuid)
$ zfs list
NAME USED AVAIL REFER MOUNTPOINT
system 1.79G 225G 21K /
system/root 763M 225G 763M legacy
system/tmp 43K 225G 43K /tmp
system/usr 1.04G 225G 21K /usr
system/usr/home 50.5K 225G 21K /usr/home
system/usr/home/ben 29.5K 225G 29.5K /usr/home/ben
system/usr/local 297M 225G 297M /usr/local
system/usr/obj 416M 225G 416M /usr/obj
system/usr/ports 190M 225G 159M /usr/ports
system/usr/ports/distfiles 30.8M 225G 30.8M /usr/ports/distfiles
system/usr/src 165M 225G 165M /usr/src
system/var 106K 225G 21K /var
system/var/audit 21K 225G 21K /var/audit
system/var/log 41.5K 225G 41.5K /var/log
system/var/tmp 23K 225G 23K /var/tmp
$ swapinfo
Device 1K-blocks Used Avail Capacity
/dev/mirror/swap.eli 2097148 0 2097148 0%

Note that system is not actually mounted (it has canmount=off) – it is used to allow all the other filesystems to inherit the / mountpoint. The one that is actually mounted on / is system/root, which is marked as legacy because it is mounted before zfs is up.

Now we’re up on the second disk, time to get the first disk back in the picture (we’re using it for boot but nothing else right now).

First blow away the MBR

# dd if=/dev/zero of=/dev/ad4 count=79
79+0 records in
79+0 records out
40448 bytes transferred in 0.008059 secs (5018970 bytes/sec)

and create the GPT partitions:

# gpart create -s GPT ad4
ad4 created
# gpart add -b 34 -s 128 -t freebsd-boot ad4
ad4p1 added
# gpart add -s 2g -t freebsd-swap -l swap4 ad4
ad4p2 added
# gpart add -t freebsd-zfs -l system4 ad4
ad4p3 added
# gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 ad4
bootcode written to ad4

No scratch partition on this one, there’s no room. Now the two disks should match

# gpart show
=>       34  976773101  ad8  GPT  (466G)
         34        128    1  freebsd-boot  (64K)
        162    4194304    2  freebsd-swap  (2.0G)
    4194466  484202669    3  freebsd-zfs  (231G)
  488397135  488376000    4  freebsd-zfs  (233G)

=>       34  488397101  ad4  GPT  (233G)
         34        128    1  freebsd-boot  (64K)
        162    4194304    2  freebsd-swap  (2.0G)
    4194466  484202669    3  freebsd-zfs  (231G)

apart from the scratch partition, of course.

Add the mirrored swap

# gmirror insert -h -p 1 swap /dev/gpt/swap4

And when rebuilding is finished, you should see

# gmirror status
       Name    Status  Components
mirror/swap  COMPLETE  gpt/swap8
                       gpt/swap4

Now add the second disk’s zfs partition

# zpool attach system /dev/gpt/system8 /dev/gpt/system4
If you boot from pool 'system', you may need to update
boot code on newly attached disk '/dev/gpt/system4'.

Assuming you use GPT partitioning and 'da0' is your new boot disk
you may use the following command:

        gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 da0

We already did this part, so no need to do anything. Wait for it to finish. Here it is partway through

# zpool status
  pool: system
 state: ONLINE
status: One or more devices is currently being resilvered.  The pool will
        continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
 scrub: resilver in progress for 0h0m, 39.20% done, 0h0m to go
config:

        NAME             STATE     READ WRITE CKSUM
        system           ONLINE       0     0     0
          mirror         ONLINE       0     0     0
            gpt/system8  ONLINE       0     0     0
            gpt/system4  ONLINE       0     0     0  718M resilvered

errors: No known data errors

and now done

# zpool status
  pool: system
 state: ONLINE
 scrub: resilver completed after 0h2m with 0 errors on Fri Mar 18 12:13:19 2011
config:

        NAME             STATE     READ WRITE CKSUM
        system           ONLINE       0     0     0
          mirror         ONLINE       0     0     0
            gpt/system8  ONLINE       0     0     0
            gpt/system4  ONLINE       0     0     0  1.79G resilvered

errors: No known data errors

And we’re done. Reboot one last time to check everything worked.

One final task not relevant to the mirroring is to mount the scratch disk area.

Create a mountpoint

# mkdir /scratch

And a pool

# zpool create -O mountpoint=/scratch -O atime=off -O setuid=off scratch /dev/gpt/scratch8

This filesystem has no redundancy, as previously mentioned. (edit: I am told that the mkdir and mountpoint are both redundant – zfs will create the directory as needed, and uses the pool name as the mount point by default)

In the next installment I will fail and replace one of the disks.

Edit:

daily_status_zfs_enable="YES"
daily_status_gmirror_enable="YES"

should be added to /etc/periodic.conf so checks are added to the daily mails.

16 Nov 2010

Apache vs. Oracle

Filed under: Open Source,Open Standards,Rants — Ben @ 11:49

As a founder of the Apache Software Foundation I have long been frustrated by the ASF’s reluctance to confront Sun and now Oracle head-on over their continued refusal to allow Apache Harmony to proceed as an open source Java implementation.

So I am very pleased to see the board finally draw a line in the sand, basically saying “honour the agreements or we block the next version of Java”. Oracle’s response is just ridiculous, including the blatant

Oracle provides TCK licenses under fair, reasonable, and non-discriminatory terms consistent with its obligations under the JSPA.

why even bother to say that in response to ASF’s charges? Do they think the ASF are suddenly going to say “oops, we see the light now, how wrong we were to expect a TCK licence under fair, reasonable and non-discriminatory terms consistent with your obligations under the JSPA, we see it all clearly now and humbly withdraw our unwarranted request”?

Well, whatever Oracle expected, the ASF’s response was short and sweet

Oracle statement regarding Java: “Now is the time for positive action (and) to move Java forward.”

The ball is in your court. Honor the agreement.

How this will play out is very uncertain, at least to me, but one thing is sure: delay and vacillation are over, at least from the ASF. Expect plenty of delay from Oracle, though.

23 May 2010

Nigori: Protocol Details

As promised, here are the details of the Nigori protocol (text version). I intend to publish libraries in (at least) C and Python. At some point, I’ll do a Stupid version, too.

Comments welcome, of course, and I should note that some details are likely to change as we get experience with implementation.

20 Mar 2010

Caja on Orkut

Filed under: Caja,Open Source,Security — Ben @ 18:19

If you’ve been living in a box for the last couple of years, you might not know that Caja is an open source project I am involved in at Google to make the web safer. Specifically, it allows untrusted Javascript, HTML and CSS to be sandboxed in a very fine-grained way. For example, the untrusted content can be limited to a subset of the whole DOM, primordial objects can be replaced or removed, properties of objects can be protected (from read, write or execution) and any method can be removed, replaced or attenuated. Yet it is still possible to write fully-featured Javascript applications in Caja. And, as a bonus, Caja hides the differences between browsers – any code you write will Just Work on any supported browser.

Caja has long been used by Yahoo!, ironically, to protect users from malicious gadgets on their Application Platform but until recently has been a bit of a poor relative at Google. So, I’m pleased to report that it is now in use to protect Orkut users.

Because Caja is open source, we don’t necessarily find out when people use it: do you know of someone using Caja? Leave a comment!

24 Feb 2010

Stupid Mailing List

Filed under: Open Source,Programming,Security — Ben @ 12:49

It seems Stupid just won’t go away – Ben Clifford and I have been gradually expanding it – it now has functions and even primitive I/O. I’m working on a new test harness and we have regular discussions on language philosophy. So I figured it was time for a mailing list for developers (and other interested parties). Here it is.

7 Feb 2010

Perhaps Not So Stupid, After All?

Filed under: Crypto,Open Source,Programming — Ben @ 17:04

Stupid now generates correct (single-block, still) SHA-256 code in C. It has functions. We’re starting to wonder about adding structures, and the semantics of arrays – particularly whether an array passed for output can also be used for input (or vice versa). I’m inclining towards making that illegal – if you want a function that, say, fills in every second entry in an array, then you’d need to pass in the array to be filled in, and return a second array which would be the result. The function would have to copy the input array to the output before filling in the new values (or copy the parts it isn’t going to fill in). It seems to me this makes analysis simpler, but can easily be optimised by smart compilers, too.

I guess its time we started writing some of this down! I’d also like to add generators for some common scripting languages, like Perl, Python and PHP.

The thing I’m a little scared of is that eventually, if I’m going to take this seriously, we’re going to need a bignum implementation – not too hard to do if you don’t care about efficiency, I guess.

30 Jan 2010

Stupid Haskell, Google Code

Filed under: Crypto,Open Source,Security — Ben @ 16:40

I can see the amusement I can derive from Stupid is going to be endless. If somewhat stupid.

More seriously, Ben Clifford wrote a Haskell plugin for Stupid. So, with his permission, I have added it to the source. I’ve also created a Google Code site for it – sadly someone already has stupid.googlecode.com, so you’ll find it at stupid-crypto.googlecode.com.

Ben also added a lot of test cases, which I haven’t yet pulled in because I want to move them into their own directory, but they may be there by the time you check the code out.

I still haven’t got around to testing the SHA-256 implementation, either. One day! Oh, and it seems the Haskell breaks, which may well be my fault. But I don’t really understand Haskell, so I might find it hard to fix.

24 Jan 2010

Stupid: A Metalanguage For Cryptography

Filed under: Crypto,General,Open Source,Programming,Security — Ben @ 20:32

Various threads lately have got me thinking about implementing cryptography and cryptographic protocols. As I have mentioned before, this is hard. But obviously the task itself is the same every time, by its very nature – if I want to interoperate with others, then I must implement effectively the same algorithm as them. So why do we ever implement anything more than once? There are various reasons, varying as to their level of bogosity. Here’s a few

  • Trust: “I don’t want to trust anyone else’s code”. This, in my view is mostly bogus. If you don’t trust others to write your crypto, then you’ve got some pretty big problems on your hands…
    • You’re likely to be using some pretty heavyweight stuff like SSL and/or X.509, and reimplementing those is a seriously major job. Are you really going to do that?
    • Unless you are also a cryptographer, then you’re trusting the guys that designed the crypto you’re implementing anyway.
    • Ditto protocol desginer.
  • Languages: an implementation in Java isn’t going to work so well in Python. And although its true that you can plug C implementations into almost everything, there are legitimate and not-so-legitimate reasons for not wanting to do so…
    • You don’t trust native code: see above.
    • It makes your distribution hard to build and/or use and tricky to install.
    • You are running on a platform that doesn’t allow native code, for example, a web hosting company, or Google’s App Engine.
    • Native code has buffer overflows and MyFavouriteLanguage doesn’t: true, but probably the least of your worries, given all the other things that you can get wrong, at least if the native code is widely used and well tested.
  • Efficiency: you are not in the long tail of users who’s transactions per second is measured in fractions. In this case, you may well want specialised implementations that exploit every ounce of power in your platform.

Of these, reimplementation for efficiency clearly needs a completely hand-crafted effort. Trust issues are, in my view, largely bogus, but if you really want to go that way, be my guest. So what does that leave? People who want it in their chosen language, are quite happy to have someone else implement it and are not in need of the most efficient implementation ever. However, they would like correctness!

This line of thinking let me spend the weekend implementing a prototype of a language I call “Stupid”. The idea is to create a language that will permit the details of cryptography and cryptographic protocols to be specified unambiguously, down to the bits and bytes, and then compile that language into the language of your choice. Because we want absolute clarity, Stupid does not want to be like advanced languages, like OCaml and Haskell, or even C, where there’s all sorts of implicit type conversions and undefined behaviour going on – it wants it to be crystal clear to the programmer (or reviewer) exactly what is happening at every stage. This also aids the process of compiling into the target language, of course. So, the size of everything wants to be measured in bits, not vague things like “long” or “size_t”. Bits need to be in known places (for example, big-endian). Operations need to take known inputs and produce known outputs. Sign extension and the like do not want to happen magically. Overflow and underflow should be errors, unless you specifically stated that they were not, and so on.

To that end, I wrote just enough compiler to take as input a strawman Stupid grammar sufficient to do SHA-256, and produce various languages as output, in order to get a feel for what such a language might look like, and how hard it would be to implement.

The result is: you can do something rough in a weekend 🙂

Very rough – but it seems clear to me that proceeding down this road with more care would be very useful indeed. We could write all the cryptographic primitives in Stupid, write relatively simple language plugins for each target language and we’d have substantially improved the crypto world. So, without further ado, what does my proto-Stupid look like? Well, here’s SHA-256, slightly simplified (it only processes one block, I was going cross-eyed before I got round to multiple blocks). Note, I haven’t tested this yet, but I am confident that it implements (or can be easily extended to implement) everything needed to make it work – and the C output the first language plugin produces builds just fine with gcc -Wall -Werror. I will test it soon, and generate another language, just to prove the point. In case the code makes your eyes glaze over, see below for some comments on it…

"This code adapted from Wikipedia pseudocode";

"Note 2: All constants in this pseudo code are in big endian";

"Initialize variables";
"(first 32 bits of the fractional parts of the square roots of the first 8 primes 2..19):";
uint32 h0 = 0x6a09e667;
uint32 h1 = 0xbb67ae85;
uint32 h2 = 0x3c6ef372;
uint32 h3 = 0xa54ff53a;
uint32 h4 = 0x510e527f;
uint32 h5 = 0x9b05688c;
uint32 h6 = 0x1f83d9ab;
uint32 h7 = 0x5be0cd19;

"Initialize table of round constants";
"(first 32 bits of the fractional parts of the cube roots of the first 64 primes 2..311):";
array(uint32, 64) k =
(0x428a2f98, 0x71374491, 0xb5c0fbcf, 0xe9b5dba5,
0x3956c25b, 0x59f111f1, 0x923f82a4, 0xab1c5ed5,
0xd807aa98, 0x12835b01, 0x243185be, 0x550c7dc3,
0x72be5d74, 0x80deb1fe, 0x9bdc06a7, 0xc19bf174,
0xe49b69c1, 0xefbe4786, 0x0fc19dc6, 0x240ca1cc,
0x2de92c6f, 0x4a7484aa, 0x5cb0a9dc, 0x76f988da,
0x983e5152, 0xa831c66d, 0xb00327c8, 0xbf597fc7,
0xc6e00bf3, 0xd5a79147, 0x06ca6351, 0x14292967,
0x27b70a85, 0x2e1b2138, 0x4d2c6dfc, 0x53380d13,
0x650a7354, 0x766a0abb, 0x81c2c92e, 0x92722c85,
0xa2bfe8a1, 0xa81a664b, 0xc24b8b70, 0xc76c51a3,
0xd192e819, 0xd6990624, 0xf40e3585, 0x106aa070,
0x19a4c116, 0x1e376c08, 0x2748774c, 0x34b0bcb5,
0x391c0cb3, 0x4ed8aa4a, 0x5b9cca4f, 0x682e6ff3,
0x748f82ee, 0x78a5636f, 0x84c87814, 0x8cc70208,
0x90befffa, 0xa4506ceb, 0xbef9a3f7, 0xc67178f2);

"For now, dummy in the message instead of declaring a function wrapper";
"Also, for now, allow enough room in the input for padding, etc, to simplify the loop";
uint32 message_bits = 123;
array(uint8, 64) message =
(0x12, 0x34, 0x56, 0x78, 0x9a, 0xbc, 0xde, 0xf0,
0x0f, 0xed, 0xcb, 0xa9, 0x87, 0x65, 0x43, 0x21);
uint32 pad_byte = 0;
uint32 pad_bit = 0;
uint32 tmp = 0;
uint32 tmp2 = 0;
array(uint32, 16) w = (0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0);
uint32 i = 0;
uint32 s0 = 0;
uint32 s1 = 0;
uint32 a = 0;
uint32 b = 0;
uint32 c = 0;
uint32 d = 0;
uint32 e = 0;
uint32 f = 0;
uint32 g = 0;
uint32 h = 0;
uint32 maj = 0;
uint32 t1 = 0;
uint32 t2 = 0;
uint32 ch = 0;

"Pre-processing:";
"append the bit '1' to the message";

"note that we're using a 32-bit length for now";
"all the op32, op8 etc are _without_ wrap (where applicable) - i.e. wrap is an error";
"they also require left and right to both be the correct type and size";
"also, we have no precedence, it is up to you to bracket things";
"rshift is with zero padding";

pad_bit = 7 minus32 (message_bits mod32 8);
pad_byte = (message_bits plus32 1) rshift32 8;
message[pad_byte] = message[pad_byte] or8 (1 lshift8 pad_bit);

"append k bits '0', where k is the minimum number >= 0 such that the
resulting message length (in bits) is congruent to 448 (mod 512)";

"eq32 and friends return a boolean value (which is not even a bit)";

if (pad_bit eq32 0) {
pad_bit = 7;
pad_byte = pad_byte plus32 1;
} else {
pad_bit = pad_bit minus32 1;
}

"bor is like C || (i.e. RHS is only executed if LHS is false)";

"448/8 = 56";
while (((pad_byte mod32 512) ne32 56) bor (pad_bit ne32 7)) {
message[pad_byte] = message[pad_byte] and8 (not8 (1 lshift8 pad_bit));
if (pad_bit eq32 0) {
pad_bit = 7;
pad_byte = pad_byte plus32 1;
} else {
pad_bit = pad_bit minus32 1;
}
}

"append length of message (before pre-processing), in bits, as 64-bit big-endian integer";

message[pad_byte] = 0;
message[pad_byte plus32 1] = 0;
message[pad_byte plus32 2] = 0;
message[pad_byte plus32 3] = 0;

message[pad_byte plus32 7] = mask32to8 message_bits;
tmp = message_bits rshift32 8;
message[pad_byte plus32 6] = mask32to8 message_bits;
tmp = message_bits rshift32 8;
message[pad_byte plus32 5] = mask32to8 message_bits;
tmp = message_bits rshift32 8;
message[pad_byte plus32 4] = mask32to8 message_bits;

"for each chunk (we only have one, so don't bother with the loop for now)";

" break chunk into sixteen 32-bit big-endian words w[0..15]";
tmp = 0;
while(tmp ne32 16) {
tmp2 = tmp lshift32 2;
w[tmp] = ((widen8to32 message[tmp2]) lshift32 24)
plus32 ((widen8to32 message[tmp2 plus32 1]) lshift32 16)
plus32 ((widen8to32 message[tmp2 plus32 2]) lshift32 8)
plus32 (widen8to32 message[tmp2 plus32 3]);
tmp = tmp plus32 1;
}

" Extend the sixteen 32-bit words into sixty-four 32-bit words";
i = 16;
while(i ne32 64) {
s0 = (w[i minus32 15] rrotate32 7) xor32 (w[i minus32 15] rrotate32 18) xor32 (w[i minus32 15] rshift32 3);
s1 = (w[i minus32 2] rrotate32 17) xor32 (w[i minus32 2] rrotate32 19) xor32 (w[i minus32 2] rshift32 10);
w[i] = w[i minus32 16] plus32 s0 plus32 w[i minus32 7] plus32 s1;
}

" Initialize hash value for this chunk:";

a = h0;
b = h1;
c = h2;
d = h3;
e = h4;
f = h5;
g = h6;
h = h7;

" Main loop:";

i = 0;
while(i ne32 64) {
s0 = (a rrotate32 2) xor32 (a rrotate32 13) xor32 (a rrotate32 22);
maj = (a and32 b) xor32 (a and32 c) xor32 (b and32 c);
t2 = s0 plus32 maj;
s1 = (e rrotate32 6) xor32 (e rrotate32 11) xor32 (e rrotate32 25);
ch = (e and32 f) xor32 ((not32 e) and32 g);
t1 = h plus32 s1 plus32 ch plus32 k[i] plus32 w[i];
h = g;
g = f;
f = e;
e = d plus32 t1;
d = c;
c = b;
b = a;
a = t1 plus32 t2;
}

" Add this chunk's hash to result so far:";

h0 = h0 plus32 a;
h1 = h1 plus32 b;
h2 = h2 plus32 c;
h3 = h3 plus32 d;
h4 = h4 plus32 e;
h5 = h5 plus32 f;
h6 = h6 plus32 g;
h7 = h7 plus32 h;

"end of outer loop (when we do it)";

"Obviously I can also do this part, but right now I am going cross-eyed";
"Produce the final hash value (big-endian):
digest = hash = h0 append h1 append h2 append h3 append h4 append h5 append h6 append h7";

Notice that every operator specifies the input and output sizes. For example plus32 means add two 32-bit numbers to get a 32-bit result, with wrap being an error (this probably means, by the way, that the last few plus32s should be plus32_with_overflow, since SHA-256 actually expects overflow for these operations). So far we only deal with unsigned quantities; some “overflows” are actually expected when dealing with negative numbers, so that would have to be specified differently. Also, I didn’t deal with the size of constants, because I wasn’t sure about a good notation, though I am leaning towards 23_8 to mean an 8-bit representation of 23 (subscripted, like in TeX).

Because Stupid really is stupid, it should be very easy to write static analysis code for it, enabling checks to be omitted sometimes – for example, the fact that we only subtract 1 from pad_bit if pad_bit is non-zero means that we would not have to check for underflow in that case.

Anyway, I’m a bit bemused after writing a lot of rather repetitive code for the compiler, so I think I’ll wait for reactions before commenting further – but it does seem to me that this is a project worth pursuing. The compiler itself, whilst somewhat crude, particularly since it doesn’t yet do most of the checks I suggest should be there, is pretty small and easily understood: less than 1,500 lines of Perl and YAPP. I won’t bore you with the details, but if you want a peek, here’s a tarball.

8 Jan 2010

TLS Renegotiation Fix: Nearly There

Filed under: Crypto,General,Open Source,Open Standards,Security — Ben @ 13:19

Finally, after a lot of discussion, the IESG have approved the latest draft of the TLS renegotation fix. It is possible it’ll still change before an RFC number is assigned, but it seems unlikely to me.

But that doesn’t mean there isn’t plenty of work left to do. Now everyone has to implement it (in fact, many have already done so, including tracking the various changes as the I-D was updated), interop test with each other and roll out to clients and servers. And even then it isn’t over, since until clients are (mostly) universally updated, servers will have to allow old clients to connect and clients may have to be prepared to connect to old servers. In the case of a new server and an old client, it doesn’t hugely matter that the client has not been updated because it is defended by the server, which should not allow a renegotiation to occur if the client is old. However, in the case of an old server and a new client, or an old server and an old client, then there’s a problem – the client could be attacked. Obviously a new client can detect it is talking to an old server, and decline to play, but for some transitional period, it seems likely that clients will have to tolerate this, perhaps warning their user.

We could summarise the situation like this:

Client
Old New
Server Old vulnerable vulnerable but client is aware, client should decline or at least warn
New not vulnerable if renegotiation is forbidden, client is unaware not vulnerable, everyone is aware

7 Jan 2010

Spamassassin FAIL

Filed under: Open Source,Troubleshooting — Ben @ 13:54

I use Spamassassin for spam filtering (anyone think there’s anything better out there these days?) and I noticed today that I’m getting a bunch of false positives. It turns out that this is because Spamassassin has a rule called FH_DATE_PAST_20XX which has kicked in now the date is 2010. Oops. Particularly oops because it has a score of 3.2 in my setup. Upgrading Spamassassin may or may not fix this, but for a quick fix, put

score FH_DATE_PAST_20XX 0

in your local.cf file, which, in my case at least, lives in /usr/local/etc/mail/spamassassin.

5 Jan 2010

Security Is Hard: Live With It

Filed under: Open Source,Open Standards,Programming,Rants,Security — Ben @ 17:59

I’ve been meaning to summon the energy to write about OAuth WRAP. It’s hard to do, because like OpenID, OAuth WRAP is just so obviously a bad idea, it’s difficult to know where to start. So I was pleased to see that Ben Adida saved me the trouble.

I understand. Security is hard. Getting those timestamps and nonces right, making sure you’ve got the right HMAC algorithm… it’s non-trivial, and it slows down development. But those things are there for a reason. The timestamp and nonce prevent replay attacks. The signature prevents repurposing the request for something else entirely. That we would introduce a token-as-password web security protocol in 2010 is somewhat mind-boggling.

Exactly. The idea that security protocols should be so simple than anyone can implement them is attractive, but as we’ve seen, wrong. But does the difficulty of implementing them mean they can’t be used? Of course not – SSL is fantastically hard to implement. But it is also fantastically widely deployed. Why? Because there are several open source libraries that do everything for you. Likewise every crypto algorithm under the sun is hard to implement, but there’s no shortage of libraries for them, either.

Clearly the way forward for OAuth is not to dumb it down to the point where any moron can implement it, the way forward is to write libraries that implement a properly secure version, and have everyone use them.

If the amount of effort that has been wasted on OAuth WRAP (and OpenID) had instead been put instead into writing code for the various platforms then we would probably now have pretty much universal support for OAuth and no-one would be whining that it’s too hard to implement.

Instead, we will spend the next decade or two clearing up the mess that we seem to be intent on creating. It makes me tired.

5 Nov 2009

Another Protocol Bites The Dust

Filed under: Crypto,Open Source,Security — Ben @ 8:03

For the last 6 weeks or so, a bunch of us have been working on a really serious issue in SSL. In short, a man-in-the-middle can use SSL renegotiation to inject an arbitrary prefix into any SSL session, undetected by either end.

To make matters even worse, through a piece of (in retrospect) incredibly bad design, HTTP servers will, under some circumstances, replay that arbitrary prefix in a new authentication context. For example, this is what happens if you configure Apache to require client certificates for one directory but not another. Once it emerges that your request is for a protected directory, a renegotiation will occur to obtain the appropriate client certificate, and then the original request (i.e. the stuff from the bad guy) gets replayed as if it had been authenticated by the client certificate. But it hasn’t.

Not that the picture is all rosy even when client certificates are not involved. Consider the attacker sending an HTTP request of his choosing, ending with the unterminated line “X-Swallow-This: “. That header will then swallow the real request sent by the real user, and will cause any headers from the real user (including, say, authentication cookies) to be appended to the evil request.

It’s obviously going to take a little while for the world to patch this – and since the news is spreading like wildfire I’ve put up a patch to OpenSSL that bans all renegotiation. I’m sure an official release will follow very shortly.

Note that the patch is against the head of the OpenSSL 0.9.8 development tree (that is, it is against 0.9.8l-dev). You may have to do a little work to patch against other versions. And if you intend to deploy this patch permanently, please change at least the textual version of the version number, which you can find in crypto/opensslv.h. Also note that if you need renegotiation for your site to work, I have no solution for you, other than you redesign your site. Sorry.

Next Page »

Powered by WordPress