Linux has a long way to go...


I'm a big fan of Linux but I'll be the first to admit it has a long way to go.  I came across this page which does a great job highlighting the shortcomings and providing a roadmap for improvement:

(updated) Distributed File System benchmark


Note: this is an update to my previous test

I'm investigating various distributed file systems (loosely termed here to include SAN-like solutions) for use in Docker, Drupal, etc. and couldn't find recent benchmark stats for some popular solutions so I figured I'd put one together.

Disclaimer: This is a simple benchmark test with no optimization or advanced configuration so the results should not be interpreted as authoritative.  Rather, it's a 'rough ballpark' product comparison to augment additional testing and review.

My Requirements:
  • No single-point-of-failure (masterless, multi-master, or automatic near-instantaneous master failover)
  • POSIX-compliant (user-land FUSE)
  • Open source (non-proprietary)
  • Production ready (version 1.0+, self-proclaimed, or widely recognized as production-grade)
  • New GA release within the past 12 months
  • Ubuntu-compatible and easy enough to set up via CloudFormation (for benchmark testing purposes)

Products Tested:

AWS Test Instances:
  • Ubuntu 14.04 LTS paravirtual x86_64 (AMI)
  • m1.medium (1 vCPU, 3.75 GB memory, moderate network performance)
  • 410 GB hard drive (local instance storage)

Test Configuration:

Three master servers were used for each test of 2, 4, and 6 clients.  Each client runs a small amount of background disk usage (file create and update):

(crontab -l ; echo "* * * * * ( echo \$(date) >> /mnt/glusterfs/\$(hostname).txt && echo \$(date) > /mnt/glusterfs/\$(hostname)_\$(cat /dev/urandom | tr -dc 'a-zA-Z0-9' | fold -w 25 | head -n 1).txt )") | sort - | uniq - | crontab -

Results of the three tests were averaged.  Benchmark testing was performed with bonnie++ 1.97 and fio 2.1.3.
Example Run:
$ sudo su -
# apt-get update -y && apt-get install -y bonnie++ fio 
# screen 
# bonnie++ -d /mnt/glusterfs -u root -n 1:50m:1k:6 -m 'GlusterFS with 2 data nodes' -q | bon_csv2html >> /tmp/bonnie.html
# cd /tmp
# wget -O crystaldiskmark.fio
# sed -i 's/directory=\/tmp\//directory=\/mnt\/glusterfs/' crystaldiskmark.fio
# sed -i 's/direct=1/direct=0/' crystaldiskmark.fio 
# fio crystaldiskmark.fio
Translation: "Login as root, update the server, install bonnie++ and fio, then run the bonnie++ benchmark tool in the GlusterFS-synchronized directory as the root user using a test sample of 1,024 files ranging between 1 KB and 50 MB in size spread out across 6 sub-directories.  When finished, send the raw CSV result to the html converter and output the result as /tmp/bonnie.html.  Next, run the fio benchmark tool using the CrystalDiskMark script by WinKey referenced here."

Important Notes: 

1.  Only GlusterFS and LizardFS could complete the intense multi-day bonnie++ test.  The others failed with these errors:
  • CephFS (both kernel and fuse)
    • Can't write block.: Software caused connection abort
    • Can't write block 585215.
    • Can't sync file.
  • SXFS
    • Can't write data.
2.  GlusterFS and LizardFS had significant differences in bonnie++ latency which couldn't be shown on the graph without distorting the scale:

Seq Create (sec)Rand Create (sec)
GlusterFS 3.7.6173164
LizardFS 3.9.4 33

3.  GlusterFS took at least twice as long as LizardFS to complete the bonnie++ tests (literally 48 hours!).  Switching to xfs out of curiosity helped performance significantly (less than 24 hours), however all tests were done with ext4 (Ubuntu default).

4.  CephFS did not complete the "Rand-Read-4K-QD32" fio test

Results (click to view larger image):

(Note: raw results can be found here)


Concluding Remarks:
  • Since GlusterFS and LizardFS were the only ones that could complete the more intense bonnie++ test, I would feel more confident recommending them as "production ready" for heavy, long-term loads.
  • Also (as mentioned above), LizardFS was much faster than GlusterFS (at the cost of higher CPU usage).
  • In terms of setup and configuration, GlusterFS was easiest, followed by LizardFS, then SXFS, and finally (in a distant last place) CephFS.
  • SXFS shows promise but they'll need to simplify their setup process (especially for non-interactive configuration) and resolve the bonnie++ failure.
  • My overall recommendation is currently LizardFS GlusterFS.  (Update: I have stopped recommending LizardFS because metadata HA is not currently supported out of the box -- see comments below).

What's the difference between KB and KiB?


I've always wondered what the big deal was between 1000 and 1024.  Some hard drive manufacturers and cloud providers stress they support one or the other but I never understood why.  It's just 24 bytes, right?  Well, it adds up and makes a big difference.  Here's a nice overview:

Securely share dynamic secrets between Linux computers


UPDATE: the service no longer works but the concept demonstrated below still works with a similar service like


I needed to set up password-less ssh access between a cluster of AWS Linux computers via CloudFormation.  Although ssh-copy-id was designed to help with this, it still presumes you have a login password which complicates things with design-time scripting, like CloudFormation.

Here was the solution I came up with (using a generic example of a random secret file):

On first server:


CURLIO=$( ( gpg --cipher-algo AES256 --symmetric --yes --batch --passphrase=${PRIVATEPASSWORD} -c ${PRIVATEFILE} && curl -F "file=@${PRIVATEFILE}.gpg"${PUBLICCURLIOTOKEN} ) 2>&1 | grep '^https' )

test -n ${CURLIO} && ( curl -s "${PUBLICTOKEN}&url=${CURLIO}" > /dev/null ) && rm "${PRIVATEFILE}.gpg"

On some other server(s):


curl -s $( curl -s "${PUBLICTOKEN}" | grep -oh 'https.*"' | head -1 | sed -e 's/"$//' ) | gpg --quiet --no-use-agent --yes --batch --passphrase=${PRIVATEPASSWORD} -o ${PRIVATEFILE}


  1. This is obviously best for sharing dynamic secrets that aren't known ahead of time when creating the CloudFormation script (like ssh keys).  Static secrets could have been simply hard-coded into the CloudFormation script directly.
  2. You'll want to protect your CloudFormation script since it will have the gpg password hard-coded.
  3. The space in front of the PRIVATEPASSWORD environment variable is to avoid saving it in the bash history.  Feel free to avoid the environment variable altogether and just insert the password into the commands where referenced.
  4. The PUBLICCURLIOTOKEN is randomly generated when you visit (right after "send/" in the example code snippet on the homepage).  Feel free to use the one in my example above -- I don't think it ever expires.
  5. For PUBLICTOKEN I recommend using the GUID from



crossfire game

In honor of the classic Milton Bradley game that my wife got me for Christmas (after spending a ridiculous amount on eBay), I thought I'd re-post this hilarious article from that is no longer available on their site (slightly edited):


This board game was created by Milton Bradley in 1971, though it was the indoctrination campaign of the 1990's which would ultimately etch this name into our minds for eternity. Playing requires only eyes, hands, and a will of steel.

Confused?  Let's ensure you're in the right place before moving on:

Just The Facts
  1. Crossfire has a 50% mortality rate
  2. Marbles + Crack x American Gladiators = Crossfire
  3. Many consider it a part of their childhood (even if they've never actually played it)

It's some time in the future...
You run through your mental checklist as you fly high above the elongated octagon of the Crossfire Arena on your triangle roller-puck.
Leather jacket? Check!
Fingerless gloves? Check!
Totally radical attitude? Raditude, check!
The chanting crowd of fist-pumping lost souls is drowned out only by the jarring barks of thunder. Lightning dances dramatically down the blackened skies as if to meet the rising flames half way, casting an eerie purple glow on the scene. The Arena shrinks down to combat size, and you are face to face with your opponent. "Poor wimp" you think, grasping the turret-mounted gun in front of you. It feels like shaking hands with an old friend.
"CROSSFIYAH!" is declared by the Overlord, signaling the beginning of the match. You load and immediately begin sending hot, chrome-laden doom toward your opponent. He responds with a torrent of well aimed silver retribution.The Arena is a blur of purple and silver; the organized chaos captivates the cheering hordes.
Tension grows as both pucks spin closer to their respective goals. Your blistered hands are on fire - you must ignore this for now. Loading. Shooting. Loading. Shooting. Eyes ablaze; no time to wipe away the dripping sweat. Tunnel vision sets in as adrenaline courses through your protruding veins. You know nothing--but to continue. Must continue!

Mom hates it when we Crossfire in the living room
Dazed, you hardly even notice as the puck sinks. A deafening "CROSSFIYAHHHH!!!!" signals the end of the match. You watch as your neighborhood friend, turned mortal enemy, is banished to a fiery, spiraling oblivion. "Yeah! Yeah!" you declare, boastfully thrusting your fist to the heavens as the exaltation of victory washes over you.
Welcome to the world of CROSSFIRE.

Cracked on Crossfire
Powerless in the face of pure marketing genius
If you lived through the '90s, chances are you've encountered Crossfire's iconic commercial, which saturated Nickelodeon's airwaves at the time. Representing one of the world's first successful ventures into "X-treme" style advertising, its promise of danger and glory quickly caught the attention of pre-pubescent boys everywhere. If you are anything like us, you wanted this "rapid fire shoot-out game" more than a Super Soaker 100, but didn't get it until your sadistic parents gave it to you as a sick joke when you turned seventeen.
"Remember that thing you wanted more than anything, son? Surprise! We bought it for you eight years ago, and thought it would be a riot to wait until you didn't want it anymore to give it to you!"
Irony makes a terrible birthday present.


The game's brilliance lies in the combination of ball grabbing and shooting; two of the most testosterone-fueled gestures known to man. Each player starts off with twenty marbles, and tries to shoot the polygonal pucks into their opponent's goal. It's a race to three points, and aggression is chiefly rewarded. The action is non-stop and fast-paced and other hyphenated words, consisting of constant reloading and shooting until a winner emerges. Gun jams are a common occurrence, but much like disobedient women and children, can be corrected easily with a little well placed smackage.
Tearing families apart since '71

The Legend:

Believe it or not, Crossfire wasn't always all lightning and flaming death. Far removed from the "I sent all my friends spinning into the ether, so now I wander this earth alone" style for which it is known, family togetherness was once the main theme.
Suburban, wholesome, family, togethernezzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzz
Not. One. Explosion.
In fact, it wasn't until 1992, twenty one years after its original release, that Milton Bradley began to change the image of Crossfire. The first attempt was quite impotent, with lame effects and a cheesy song. In truly artful protest, the lead singer pronounces "Crossfire" as "Fart fire" throughout the tune, displaying the self-aware contempt only a man stuck in a terrible '90s commercial could ever know.

In comparison, its better known counterpart has been called a thirty-second movie. The opening line "It's sometime in the future..." raises far more questions than it answers, and just seems unnecessary; but for kids, its ambiguity hinted at a deeper plot, and the open-ended storyline fired up their imaginations.
To brilliant effect, the ad was used merely as the primer for an epic to be continued in the young viewer's mind, lasting long after the ad itself ended. The unprecedented potency of the commercial is why Crossfire was so desired by kids at the time, and why it triggers such nostalgia in people today.
The same tactic was used in this classic:
It's like the more sophisticated big brother of the Crossfire ad in that it goes so far as to incorporate rudimentary metaphor (our "fire-demon" is "drinking alone", and it has yet to be slain), but there's something about the uncanny parallels that are almost.....suspicious.
[Updated on: Today at 6:07am]
Jesse Ventura has informed Cracked that Crossfire was a training & indoctrination device, set up by the Marine Corps to create a generation of American "super soldiers" by developing hand-eye coordination, tactical decision-making skills, and raw killer instinct early on in future prospects.
A random Ron Paul supporter we met online confirms this.
Good enough for us
Out of respect for the legions of people that grew up in the 1990s, still longing for that Crossfire glory which will never come, we'll fight the urge to be hilarious here and end on a serious note...

Where have all the space simulators gone?


2011 was a bad year for astronomy education.  It marked the end of development for Celestia, the first astronomy simulator that I ever fell in love with.  Spin-offs like Celestia.sci and Celestia161-ED slowly died on the vine after that.

A year before, Orbiter had gone dark.  Two years later, Digital Universe Atlas (demonstrated in the video above) development ended in favor of a commercial planetarium product.

Sky in Google Earth and Sky-Map only allow 2-D navigation and panning (flat-earth believers rejoice!).

Kerbal Space Program has mediocre graphics.  Likewise with Pioneer.

SpaceEngine looks promising but it's proprietary (owned by a single Russian developer) with an uncertain free beta status and very commercial aims [1] [2] [3].  Outerra, Starry Night, and Redshift are similarly commercial.

In addition to being commercial, Universe Sandbox by design is more about astrophysics than depicting the actual universe.

So, where have all the cool space simulators gone?  Where are all the brilliant retired astronomers, physicists, and OpenGL space animators congregating?  Is open source too unrealistic for this product category?  Please comment and vote for your favorite!

Update: WorldWide Telesecope looks promising (especially since Microsoft recently open sourced it and switched ownership to a neutral governing body) but it uses image stitching (similar to KStars) instead of vector 3D space so you don't get the star-flyby effect which I feel degrades the "space travel" experience.