Pete's Points | Peter Lyons

Node Summit

2012-01-31T00:06:04.787Z

So I flew out to San Francisco last week for the Node Summit conference. I wasn't sure exactly what to expect. The first event was the NodeUp live pod cast at the Bottom of the Hill bar. I was in the Mission immediately prior to that and it was a long walk (after 5.5 hours of walking) to get there. The crowd seemed to be almost all developers and it was surprisingly international. I got to meet some folks I admire from the Internet like Pedro Teixeira. Tim Caswell and TJ Holowaychuck (my current programming hero) were both there and Ryan Dahl eventually arrived as well. I learned an interesting thing about myself: even if you take a bar and fill it up with ubernerds, I still feel uncomfortable and count the minutes until I can make a quiet exit.

Tuesday morning I caught a shuttle from a hotel near my room to the event, which was a really nice service to have (although I guess I would have happily walked as well). The conference was a lot bigger and more extravagant than I was anticipating. Many large companies were represented including Microsoft, Ebay, Yahoo, LinkedIn, Sabre, Visa, Google, and more. The discussion panels were pretty hit or miss (mostly miss I guess), with a few being really interesting. I did a lot of networking and hopefully some of these connections will prove valuable in the near future. I went to both of the Nodejitsu workshops, which were packed beyond capacity.

Wednesday was more of the same but there was a good panel with Ryan Dahl and Brendan Eich on the future of JavaScript. I asked a question to the panel of Platform as a Service companies a question and the moderator, Jolie O'Dell said "and by the way you're nail polish is bitchin'" as I returned to my seat. :-)

I didn't manage to find any after parties via the twitters, although I probably would have opted to go home and nap anyway. I did get to hang out with the Nodejitsu crew Thursday evening though, which was fun.

Build a Consumer App With Node.js

2012-01-21T17:57:09.771Z

It's time to start building consumer apps with node

Troubleshooting responsive layout with :after

2012-01-11T00:34:06.000Z

So I'm working on a new design for my site and using the super-great stylus CSS preprocessor along with this great gist for simple responsive layout (again with stylus and the nib library).

Well, I wanted to see which of my CSS3 media queries were in effect. Here's a simple way to do it with the :before selector.

First, put some markup in your HTML near the top, like this (using jade template language.

body
  .content
    header
      p BUGBUG responsive layout max-width:
      h1
        a(href="/") Peter Lyons

Then in your .styl stylesheet, use :after to tell which media query is active.

////////// Responsive layout //////////
@media screen and (max-width: 960px)
  header
    p
      &:after
        content "960"

@media screen and (max-width: 720px)
  header
    p
      &:after
        content "720"

Now load that in your browser and resize the window. The CSS will change the text in the header telling you exactly which rules are firing. Nice!

Tips for MongoDB Migrations in Rails

2011-12-27T07:04:50.000Z

I wanted to share a quick and easy method for testing Rails migrations when using the MongoDB database. The flexibility of mongo and ruby makes this pretty straightforward. In this example, we'll be renaming a field in our example "books" collection from "isbn" to "book_number". This is a pretty common type of migration and once you get the hang of this simple case, more complex migrations follow the same pattern. First, lets generate our timestamped migration script boilerplate.

rails generate migration rename_isbn_to_book_number

Then we'll edit the file that generated under db/migrate. We'll split our code up into several sections as follows.

Some class constants shared by all methods
The self.up method that does the forward migration
A helper method to set up some sample data to test forward migration
The self.down method that does the rollback
A helper method to set up some sample data (if needed) to test rollback

First, just define some constants that our methods will use. We store the mongo update options hash MultiUpdate for convenience since most migration update operations want upsert false (don't create any new documents), multi true (update all matching documents), and safe true.

Then we also define a constant for the collection name. For the real migration, the collection is "books", but for testing, we'll create a new collection called "testbooksmigration" as we develop our code.

class RenameIsbnToBookNumber < Mongoid::Migration
  MultiUpdate = {:upsert => false, :multi => true, :safe=>true}
  Collection = db.collection("books") #Final production code
  Collection = db.collection("test_books_migration") #Just for testing on the console
  def up
  end

  def down
  end
end

OK, that's our initial boilerplate. The next step is to take a backup of our development database if there's any data in there we don't want to accidentally wreck. Then we start coding a little helper method to populate our test collection with fake documents resembling what we expect to see in production, but only focusing on the fields relevant to the migration. Add this method to your migration class.

def mock_data_for_testing_up
  3.times {|number| Collection.insert({"isbn" => "#{number}"})}
end

def show_collection
  Collection.find({}).each {|_| puts _}
end

This will create 3 dummy documents we can use for testing. We are putting this code in a method so it's easy to re-run as we tweak and test our migration code. For complex migrations, many rounds of tweaking to get all the edge cases might be needed.

We can now fire up a rails console and run this code by copying and pasting the 2 class constants and the mockdatafortestingup method into the console and then running mockdatafortestingup

$ bundle exec rails console
Loading development environment (Rails 3.1.1)
irb(main):001:0> #Paste the following into the console
MultiUpdate = {:upsert => false, :multi => true, :safe=>true}
Collection = db.collection("test_books_migration") #Just for testing on the console
def mock_data_for_testing_up
  3.times {|number| Collection.insert({"isbn" => "#{number}"})}
  Collection.find({}).each {|_| puts _}
end
def show_collection
  Collection.find({}).each {|_| puts _}
end
irb(main):008:0> mock_data_for_testing_up
mock_data_for_testing_up
MONGODB app_development['test_books_migration'].insert([{"isbn"=>"0", :_id=>BSON::ObjectId('4ef5f43b2a4397a5d7000001')}])
MONGODB app_development['test_books_migration'].insert([{"isbn"=>"1", :_id=>BSON::ObjectId('4ef5f43b2a4397a5d7000002')}])
MONGODB app_development['test_books_migration'].insert([{"isbn"=>"2", :_id=>BSON::ObjectId('4ef5f43b2a4397a5d7000003')}])
irb(main):009:0> show_collection
MONGODB app_development['test_books_migration'].find({})
{"_id"=>BSON::ObjectId('4ef5f3812a4397a5bd000001'), "isbn"=>"0"}
{"_id"=>BSON::ObjectId('4ef5f3812a4397a5bd000002'), "isbn"=>"1"}
{"_id"=>BSON::ObjectId('4ef5f3812a4397a5bd000003'), "isbn"=>"2"}
=> nil

So now we have a separate, well-understood test collection ready to test our simple migration. Let's code up our up method. To do our migration.

def up
  #We want to rename the book.isbn field to book.book_number
  Collection.find({"isbn" => {"$exists" => 1}}).each do |book|
    update_op = {
      "$unset" => {"isbn" => 1},
      "$set" => {"book_number" => book["isbn"]}
    }
  Collection.update({"_id" => book["_id"]}, update_op, MultiUpdate)
end

We can paste that into the console and run it to test our migration. We can verify the results with show_colletion. If we want to test other records for the rollback, we can create a mock_data_for_testing_down method.

This should give you a really quick way to experiment and get your migration code working. Mongo has some advanced query and modify capabilities that can do amazing things, and an easy way to do some trial and error is handy. If you make a mess of your test data, you can use Collection.drop to get a clean slate. Here's the final migration code for reference. Don't forget to remove the test collection constant and drop the test collection from your database when your ready to start running your code for real with rake db:migrate.

class RenameIsbnToBookNumber < Mongoid::Migration
  MultiUpdate = {:upsert => false, :multi => true, :safe=>true}
  Collection = db.collection("books") #Final production code

  def up
    #We want to rename the book.isbn field to book.book_number
    Collection.find({"isbn" => {"$exists" => 1}}).each do |book|
      update_op = {
        "$unset" => {"isbn" => 1},
        "$set" => {"book_number" => book["isbn"]}
      }
      Collection.update({"_id" => book["_id"]}, update_op, MultiUpdate)
    end
  end

  def down
    #We want to rename the book.book_number field to book.isbn
    Collection.find({"book_number" => {"$exists" => 1}}).each do |book|
      update_op = {
        "$unset" => {"book_number" => 1},
        "$set" => {"isbn" => book["book_number"]}
      }
      Collection.update({"_id" => book["_id"]}, update_op, MultiUpdate)
    end
  end

  #These methods are not called by the migration.  Just for manual testing
  #by copy/pasting into the console
  #To test (by copy/pasting from here to the console)
  #1. Set the MultiUpdate constant. Adjust "Collection" to be a test collection
  #2. Copy/paste the 2 methods below
  #2. Run mock_data_for_testing_up
  #3. Run the body of self.up
  #3b. Optionally run the body of self.up again to make sure it is idempotent
  def mock_data_for_testing_up
    3.times {|number| Collection.insert({"isbn" => "#{number}"})}
  end

  def show_collection
    Collection.find({}).each {|_| puts _}
  end
end

No Need for npm -g

2011-12-22T21:14:45.000Z

So npm has this "-g" switch to install "global" packages that bundle command line executable scripts. I've been on a strict project isolation kick lately after dealing with rbenv in the ruby world, and I just don't see any need for npm -g. I want each project to have its own version of node, coffeescript, mocha, or whatever else I need. Here's my principles for a harmonious multi-project system.

1. Install things under your project root

Node goes in project/node.

Install npm modules without -g. coffee becomes project/node_modules/.bin/coffee. mocha becomes project/node_modules/.bin/mocha. And so on.

2. Set your PATH

Add ./node/bin:./node_modules/.bin and to your PATH.

Done

Here's an example.

$ mkdir project1 project2
$ cd project1
$ npm install coffee-script@1.0.1
coffee-script@1.0.1 ./node_modules/coffee-script 
$ which coffee
./node_modules/.bin/coffee
$ coffee --version
CoffeeScript version 1.0.1
$ cd ../project2
$ npm install coffee-script@1.2.0
coffee-script@1.2.0 ./node_modules/coffee-script 
$ which coffee
./node_modules/.bin/coffee
$ coffee --version
CoffeeScript version 1.2.0

Same principle works for node and any scripts you get from npm modules.

While we're talking about PATH, here's how I set my PATH in my ~/.zshrc. It's nice because I can throw a bunch of crap in there that may or may not exist on any given computer and only directories that exist get into my PATH.

########## PATH ##########
PATH=
addPath() {
  if [ -d "${1}" ]; then
    export PATH=$PATH:"${1}"
  fi
}
addPath ./node_modules/.bin
addPath ./node/bin
#Repeat addPath lines for each directory...
export PATH

Security Caveat About Relative Paths in PATH

Having relative directory paths in your PATH is arguably a security vulnerability. See also slide 20 here. I'm not personally too concerned about this one for my personal interactive login shell. However, this practice is probably not suitable for shell scripts that are run as programs. Also note that in a production deployment you should launch your node or coffee executable via an absolute path when coding your upstart or SysV init script.

The destiny of complex languages

2011-12-04T01:33:01.000Z

I had an interesting thought about programming languages and complexity yesterday. I think of languages like Python and CoffeeScript (and even C ¹ aim to be small and consistent as a language. Things like Perl and Ruby are much more complex. I have been studying both Ruby and CoffeeScript for a while (I think about a year for Ruby and maybe 6 months for CoffeeScript guessing from unreliable memory), and I feel like my JavaScript/CoffeeScript knowledge is solid and with Ruby I'm mired in intermediate status. However, the fact is I write lots of code in each of these languages. So what does that indicate?

It seems to me that the more complex your language, the larger portion of the total extant codebase written in that language will have been authored by programmers that do not completely understand the language. I would venture that if you took all existing Perl code, about 80% of it was written by programmers or sysadmins with cursory knowledge of the language. And within the Perl community, finding truly competent programmers with solid mastery of the language is definitely the exception, not the rule. I think the combination of ruby's largeness and complexity coupled with the huge popularity of rails means the same is true for Ruby. But I don't think there's anything about Perl or Ruby that makes it this way other than just it's the nature of having complexity in your language.

I'm solidly in the camp of tiny, simple languages with programs authored by programmers with mastery, and this post has been interesting food for thought for me on this topic.

While C as a language is usually considered small and simple, the POSIX interfaces you need are so old and devilishly tricky that I wouldn't trust C code written by beginner or intermediate C programmers.

Hands off my window title!

2011-11-29T12:23:46.000Z

So I recently ssh'ed into a shiny new Ubuntu 11.10 server on EC2 and noticed some new things. Firstly, Ubuntu seems to have enabled the byobu terminal multiplexer configuration by default. This looks potentially handy, but just like Oh my zsh I don't feel motivated to futz with it just now. It's easy to disable byobu with just a quick byobu-disable, which will, somewhat surprisingly, fully exit your shell, log you out and close your ssh session. But next time you'll get a normal shell instead of a byobu/screen session.

Now one thing I noticed and finally got motivated with to "research" (and by that I mean ask smarter people on twitter) and fix was that when I connected via ssh, the window title in my iTerm2 tab was dynamically changed from what I had carefully set it to ("asset pipeline") to the supremely unhelpful "ubuntu@ip-10-11-12-13". In some non-cloud situations where the server has a meaningful hostname, this might be handy, but distinguishing dozens of servers by their internal EC2 IP is not appealing to me. Initial research suggested my PROMPT or PROMPT_COMMAND environment variables, but neither was set. I read through this DeveloperWorks article and found the responsible code in the ~/.bashrc file. Here's the offending excerpt.

# If this is an xterm set the title to user@host:dir
case "$TERM" in
xterm*|rxvt*)
    PS1="\[\e]0;${debian_chroot:+($debian_chroot)}\u@\h: \w\a\]$PS1"
    ;;
*)
    ;;
esac

So you can comment out that line that changes PS1 if you prefer your own manually-set window title.

Linkzie gets a stack upgrade

2011-11-24T08:17:08.000Z

So Linkzie has had a nice series of updates to the underlying software that runs it. It's now pretty much bleeding edge latest and greatest.

Ubuntu 11.10
rbenv
Ruby 1.9.3-p0
Rails 3.1.3
Unicorn 4.1.1
jQuery 1.7

It was an interesting process. I just went through a pretty challenging upgrade from Ruby 1.8.7 to 1.9.3 at work and it was pretty destabilizing and chaotic. So for this one I went slow and steady, changing one thing at a time, and keeping the app running and tests passing. First I left ruby at 1.8.7 and switched from the Ubuntu ruby to rbenv without changing anything else. Then I upgraded some of the supporting gems like Capistrano without messing with the key gems (rails). Then I upgraded to Rails 3.1.1 and got that working. Then I did a carte blanche bundle update and pulled everything else up to latest. Only then did I switch to ruby 1.9.3.

Ruby 1.9.3 still has bugs in the debugger, which I find a bit shocking, but I'm limping along with "pry" as a poor substitute. I'm a bit shocked by the issues with Ruby 1.9.2 and 1.9.3 given how old the project is. I don't remember any actual bugs going from Python 2.4 all the way up to 2.7, just the normal code evolution stuff.

In any case, if Linkzie had any users, in theory it might be loading and responding faster for them. I learned a lot in the process and got my staging server VM to very closely mirror production, which is good.

Thoughts on launching and scaling quickly

2011-10-24T22:10:40.000Z

I have new blog post over on the Dojo4 web site.

New Linkzie Release: Nice UI improvements

2011-07-30T21:09:10.000Z

I just pushed out a new release of https://linkzie.com. There is no longer any need to have an "Organize your links" button. You can always organize you categories by dragging them around by their name. You can always organize your links by dragging them by a little crosshairs icon that is displayed to the right of each link when you mouse over it. The commands to edit or delete items are now in a little pop-up context menu to the right of each link and they are more intuitive to use.

I think this UI makes Linkzie very easy to use and the code is much simpler now and the UI should respond more snappily. Try it out and let me know what you think!

Web Framework Woes

2011-07-05T23:10:46.000Z

So my attitude for most of this year has been one of reserving judgement in favor of direct experience. As part of this, I have been trying all the latest "Kool-Aid" web development technologies even if at first glance they didn't sit right with me. So I started building the Othenticate infrastructure on a shiny stack of mongodb, node.js, express, mongoose, jade, and stylus. For the most part, I love this stack. However, after I while I have come to the conclusion that for a micropreneur project such as mine, traditional RDBMS (PostgreSQL) is going to be more appropriate than dealing with the trade-offs that NoSQL is making. I won't need to scale to millions of objects, and I want ACID, a schema, and mature tools at the data layer. So I've decided I want to use PostgreSQL instead of MongoDB. However, along with this came the realization that it seems that there is no mature Object-Relational Mapper (ORM) on the node stack for PostgreSQL. There also does not seem to be a mature schema migration framework. Basically all of the other elements of the node stack are fantastic to work with, but these two might just be deal breakers for me.

So I started thinking along the lines of another stack I might use if I wanted a good PostgreSQL ORM and schema migrator. There seem to be two obvious candidates: Django and Rails, and maybe some smaller ones and probably the option of cobbling together a hybrid stack of python components. In contemplating learning another web development framework (Django) or doing another Rails project, I started doing some analysis. Below are some of my thoughts on these stacks.

First, Node.js and friends. I've been coding in node.js/express for a while and for the most part finding it fantastic. The frameworks are cleanly designed and easy to understand and work with. The environment is in general pretty straightforward (compared to rails). Node starts up instantly and has fantastic debugging support that work seamlessly with chrome's debugger via the node-inspector module. I can graphically debug the client side code in the browser, the server side code, my jasmine tests, even my server side node.js command line unit tests can be debugged in chrome. I LOVE the fact that I can re-use my model classes across the browser and server. The only real pain points are the lack of ORM, schema migrator, and the hassle of async callbacks (which is mild but it's still a hassle). Jade for templating with stylus for CSS is almost perfect.

I've also been working in rails for a while now. Overall the node/express environment feels like a better match for me. Ruby has been influenced by Perl and Perl is utter anathema to me. The fact that Rails is written in Ruby is pretty much a kiss of death for me. However, I do love ActiveRecord and the rails schema migration subsystem. But that's about it. Rails templating has decent choices although Jade is the clear king in my mind. I don't care much for the Rails controller model and routing mechanism either. But perhaps the paramount rails stopping point for me is just the way the community operates. It's a mess of gems and fads and hipsterism and uncontrolled software erosion. The documentation can't keep up with it so you end up in a sea of partially-clueful blog posts. My overall reaction to this just ends up being "this doesn't feel right".

So I've started to give serious consideration to Django. I do like python quite a lot. I was hoping Ruby would be the next major language I work in extensively, but I don't think I'm going to go down that path much farther. Working with CoffeeScript has made python feel much less elegant to me, but I can deal with it. I hear good things about the South subsystem in Django and the docs look good. I really don't want to recode my whole current effort though. So these days I'm torn. Here's a summary of my reactions in bullets.

Node.js/Express/Etc - the good

Love CoffeeScript
Express/Connect are elegant. They are libraries you use, not a framework you must inject your code into
Jade and Stylus are rapidly approaching Nirvana status
Debugging support is top-notch
The functional programming sits pretty well with me
High concurrency through solid design is pretty sweet

Node.js/Express/Etc - the bad

No good ORM for PostgreSQL
No good schema migrator for PostgreSQL
Event model means stack traces are often useless
No intelligent reloading of changed code on the fly
The async stuff, while manageable, is clearly a hinderance compared to synchronous code. It does force you to be a little more considered than you might otherwise be, though

Rails - the good

ActiveRecord is awesome
Good schema migrator
Lots of handy gems
Culture of thorough testing

Rails - the bad

Ruby is a kitchen sink language
Not quite optimized for modern web apps
Docs and blog culture don't suite me
A big freaking mess with ruby versions, RVM, gems, bundles
Not javascript means code duplication

Django - the predicted good

South/ORM/Schema are supposedly very good
Python
Docs look good

Django - the predicted bad

The python community has a long history of web frameworks that don't catch on. Django seems to be the winner, but I am still skeptical
Not JavaScript means code duplication
Seems like there might not be any intelligent autoreloading of code during development

Linkzie 0.7.1 released

2011-07-05T07:04:28.000Z

So after a fairly lengthy period of inattention to my Linkzie bookmark manager, I got frustrated on my current project and turned my focus to Linkzie today for some updates. There's a new menu bar shamelessly stolen from the new google bar. A brand new "beach house" color theme has been applied. The "organize" UI has been changed significantly. Lots of other minor UX tweaks have been made. For example, you can now click anywhere on the entire row for a link as opposed to before where you had to click on the actual link text itself. Perhaps the biggest improvement is session expiration detection. Since linkzie is designed to sit open in a browser tab for long periods of time, it was possible for a user's server side session to expire with the window open. The UI would let them attempt to make changes, only to fail with a login required situation when trying to save their changes. This resulted in lost work and an error message popping up. Now linkzie will poll the server periodically which will likely just keep your session alive longer, but if it does expire, the UI is updated to reflect that. You can still use your links, but all editing operations are disabled and you are notified to sign in again if you need to make any edits.

Have a look and let me know what you think!

Commander: Utility Automation

2011-07-02T04:24:54.000Z

Check out my new little automation utility belt app for the keyboard-centric power users. It is hosted on a github project here.

MicroConf

2011-06-05T09:11:46.000Z

Just a quick note that I'm off to MicroConf in Las Vegas tomorrow. A buddy of mine from Denver Hack Nite is also going. It should be a really fun time and hopefully I'll learn a lot. I've been coding away like a madman (an excruciating 4-6 hours a day!) preparing a little proof-of-concept for the startup idea I'm working on. Can't wait to have it scutinized and hopefully not completely blasted to bits. I'll be blogging and tweeting if there's anything interesting and I'm not too busy concentrating.

Leveling Up: Career Advancement for Software Developers

2011-05-06T00:56:18.000Z

I have written a piece to help software developers accelerate their career. It lives on its own page, over here, but please share your comments on this blog post. There are some other comments on Hacker News as well.

Great Design: Fat Brain Toys Order Confirmation

2011-05-02T23:04:19.000Z

Fat Brain Toys has the best-looking order confirmation page I have seen. Click the image for a larger version.

Good node.js things are happenning

2011-04-13T10:38:39.000Z

I hope to post some details on this soon, but just a note that my ongoing node.js hacking is turning up some real gems. I migrated my application level tests from Zombie to Phantom with great results. phantom.js is pretty rad. Also got remote graphical debugging in the browser working and learned how to properly drop in a REPL for quick and dirty investigations, which is cool. Maybe on Thursday I'll have time to post some of the details, but at this point my node.js stack is pretty fierce.

Node docs vs. Rails docs

2011-04-05T21:15:51.000Z

I've been working with Node.js a lot recently and enjoying it. However, I'm not sure entirely WHY I am enjoying it. I'm writing it in CoffeeScript, which is pleasant enough, but it doesn't change the fact that javascript is lacking is basic language/library features like a Set object or hashes with non-string keys and so forth. The other day I realized part of my good feelings about node are from my experience with the docs, which normally goes as follows.

Each project has a single authoritative home page
- often the main github repo or a unique domain that just links to github
The docs are usually a 1-pager that explains all you need to get started
The examples are clear enough
That's all it takes. You can get up and running with that.

I really enjoy that aspect of this. Case studies include underscore, backbone, jade, coffeescript, and express. Also, the docs are the right length (compared to rails). They aren't one-off blog posts on how to do X, nor are they a book-length comprehensive tutorial that builds an app from scratch. They are both full API references and guides coupled with enough tiny snippets to get you going.

Now rails on the other hand, and this might be my own fault, seems to be problematic here. For one, the gems you need often don't have enough documentation to get up and running on their github project. You are forced out into the harsh unpleasant jungle of umpteen out of date, contradictory or incorrect blog posts. One nice thing about having the main docs on github (in node land) is that all the info there is guaranteed to be current and the outdated stuff is not there (but it is still in the git history if you need it). The problem with information rot on rails related blog posts and screencasts is really annoying. Rails DOES have nice docs on the other end of the spectrum, the comprehensive gigantic tutorial, but for whatever reason these days I never read those end to end and I usually just get myself into trouble with impatient skimming. If I'm going to spend hours and hours reading something, I probably still prefer to get a physical book than read a gigantic web based tutorial.

This is just one taste of the different experience working with these two technologies. I still think I'll continue to view rails as basically the top web development framework out there (unless I try Django, maybe, but I learned rails first to force me to learn ruby), but there are definitely aspects of the rails ecosystem that rub me the wrong way.

Baby steps in node.js

2011-03-24T00:51:22.000Z

So I've been working on some toy projects in node.js for a few weeks. Node is pretty hot in the tech media these days (see this article, among many others), and I've been having fun learning it and opening my brain to the world of async and callbacks. This post will just note a handful of the interesting things I've found thus far.

OK, so one of the first things I got enticed by was CoffeeScript by Jeremy Ashkenas. It's a pretty little javascript dialect and does a good job of taking the JavaScript: The Good Parts book, smashing it together with python indentation aesthetics, and making a language out of it. It all just compiles to javascript though, so there's no deep magic here. Most node.js libraries and utilities can directly work with .coffee source files without needing a manual intermediate step of running coffee --compile to get from .coffee to .js source code, which is key for me. There is also coffee --compile --watch mydir which will use FSevents to watch all .coffee files under the mydir directory and regenerate the corresponding javascript instantly every time you save them. Currently, the only thing in my stack that actually needs the .js versions are my jasmine in-browser tests.

Next I stumbled upon the CSS compiler stylus, which is fantastic. I also looked at SASS/SCSS, but there's no good and complete node.js implementation at this time as well as less CSS. All of these are great and beyond adequate. I ended up choosing stylus because it had a utility to convert .css files to .styl files automatically, makes all of the extra CSS syntax optional, meaning it's easy to start with .css and gradually refactor to .styl, the command line utility is nice, and it works great with node.js and express.js. Again, any one of these will do just fine and improve greatly upon CSS. It seems at this point the CSS problem is pretty much solved and I don't have to spend much energy on it, which is great.

Next of course we need some templates. This is where most of the variation and choice seems to exist. I looked at several. I started with CoffeeKup because I was enamored with CoffeeScript and wanted to have my entire codebase in CoffeeScript. I eventually started to find all those "-" characters cumbersome though, as well as one bug that slowed my development. So I switched to Jade, which improves upon HAML. Jade is so far pretty good and very minimal. It's not perfect, but it's really pretty and I'm liking it. I also thought about Weld, which looks very cool but I haven't tried it yet. I watched this video on using server side YUI and honestly this looks like the shizzle. I've done some experimenting with jsdom and jquery but haven't gotten huge traction yet. I really like the idea of rendering the basic document and then after the fact doing some jquery-style changes before sending to the client.

On the middleware side I'm using Express which seems to be widely adopted. It works great with no fuss and no muss (so far, but my apps are still microscopic). For testing, it's jasmine-node on the server and jasmine in the browser. The jasbin command line utility is handy for this. I've been using zombie.js for application level testing with good success.

Today I did my first real async work on the server side. I used async to help with this successfully. The results are great, but not stellar. The code is still a bit foreign in its layout but overall I'm grokking the async world sufficiently to make forward progress. underscore.js has some handy utilities, but this is where coffeescript/javascript falls short. Consider taking a list of photo objects and extracting a list of captions from that in these languages:

#Javascript + underscore.js
captions = _.pluck(photos, "caption")

#CoffeeScript
captions = photo.caption for photo in photos

#python
captions = [photo.caption for photo in photos]

#Ruby, better than javascript but not as clean as python
#The block mechanism is more generically powerful though
captions = photos.collect {|photo| photo.caption}

I've written my business logic in backbone.js and so far so good. After some head scratching and namespace tweaking, I am able to run the exact same model code on the browser and in node on the server, which is sort of the holy grail that this whole effort was seeking. Hurray for only coding data validation and error messages once. So far so good, but again I'm still at the baby steps stage of this.

Overall my attitude about the node.js ecosystem is highly optimistic. I hope in the next year or two it gets the kind of mainstream support that rails has. Expect some more posts with more tech details sometime soon as my experiments move into the more complex guts of the applications.

Show HN: My bookmarking app

2011-03-17T10:21:11.000Z

So the Hacker News Guidelines tell me when submitting something I should write a blog post about it and link to that. OK, here it is. I'd like to get feedback from the HN community about my simple but effective bookmarking web application: Linkzie. I built it mostly during nights/weekends while holding down a day job doing enterprise software. I know there are a lot of apps out there for this problem, especially in the recent months following the delicious.com leaked "sunset" presentation. I missed that boat to announce the app, but anyway here it is. I think the interface is very clean and straightforward and the functionality is simple. There's nothing here that can't be explained in a handful of words. You put in your URLs, it lets you lay them out on one big page. There's drag and drop organization and in-place editing. You can try the app using sample data right on the home page without needing to sign in (click no the "Next" link in the tour to see the edit mode).

The app was written mostly as an "organic" project since I have been maintaining a home page of links using a little python script to generate static HTML since 2003 or thereabouts. I wanted to learn modern web development (spent 2004-2010 in back end enterprise software) with rails and jquery. My initial thought was this space was completely saturated but the success of Pinboard has made me rethink that assumption. Linkzie doesn't have any whiz-bang fancy features currently, but maybe the spacial layout and clean design will appeal to some users.

My Lifestyle Business Hero

2011-03-10T23:56:12.000Z

Brandon Pierce of Music Teacher's Helper has what looks to me like the ultimate lifestyle business. check out the case study on Tim Ferriss's blog.

Startup Weekend Boulder: Bridge My Path

2011-03-01T03:20:21.000Z

I spent all weekend at Startup Weekend Boulder. This is an event where people pitch their business ideas, vote on the top ideas, and then form teams and try to get the ideas from zero to launch in 54 intense hours. It was my first time participating in Startup Weekend. Overall, it was a really fun experience and I met a lot of really smart and fun people. Friday was pitch night where somewhere just shy of 40 people gave 1-minute pitches. There was good variety here from already established projects to things conceived just moments earlier. There were gaming ideas, social networking ideas, education, finance, retail, even a medical marijuana related business and an infrared steak griller. I pitched one of the ideas I have sitting around on my big long WorkFlowy business idea lists: ThreeShopper, a curated shopping site to end the paradox of choice and narrow things down to exactly three choices. My idea got zero votes. :-) I think there were around 60 total attendees including folks from Nova Scotia and South Africa.

After the top 7 or so pitches were elected we each had a chance to choose a project and join that team. I joined up with Marshall Hayes who was working on his mentoring networking idea. We ended up gathering a very large team. It was nine people even after a few joined and then switched to other projects. Folks were enthusiastic about promoting the concept of mentorship but I felt like the idea needed refinement. So we spent a lot of time just debating whether it was going to be consumer focused or licensed to universities for alumni networks amongst other variations. Saturday morning as these discussions continued and the idea seemed to be heading towards a profile-based "linkedin for mentoring", which was not at all appealing to me, I suggested we pivot the focus and instead of focusing on the mentors themselves, focus on a skill path and connect people along the same skill progression, facilitating mentorship from those ahead of you, connection with peers at the same place as you, and also helping those following you on the learning path. The team was receptive to the pivot, and we started refining that idea. Eventually we ended up with Bridge My Path as the company name and got to work on wireframing, graphic design, market research, copy writing, etc.

In the next day and a half we built the home page and refined a lot of the idea, as well as creating the demo pitch for Sunday evening. Our MVP was our graphic designer Carly Gloge from WarbWeb. She created a beautiful site and graphics. Nicole Glaros complemented her during the pitches and said we had the best design of all the teams. I think the aspiration of startup weekend is to actually get a functioning product up and working end to end, but given the lack of initial clarity in the idea and my years working in enterprise software, I am very skeptical of the utility of this goal. So we ended up just building a static HTML site to launch the concept. I put together most of the HTML/CSS using coffeekup templates and rendering them via node.js/express, then just posting the static HTML files.

Sunday evening teams had 5 minutes to pitch their product. Wow. What a difference a day and a half make! Everything was much more refined and lots of teams had at least a nice home page designed and live. Some teams had working mobile apps. I don't recall any of the web apps having much to show in terms of code written over the weekend (SnapGames had some pre-existing code). The panel of judges commented liberally on each idea and gave a lot of good feedback. Suzan Bond was there blogging and tweeting the event as well as conducting some video interviews (which I can't easily locate on her site just yet). After the event, I took a long overdue nap, grabbed some ribs at the Rib House, and then hung out at the Spark Palace after party.

Again, this was a really fun and valuable event. In general I like high-commitment events (like Burning Man), so the fact that all the participants sprung for a $75 ticket and committed to locking themselves in the bunker all weekend makes a great dynamic. A few lamers did pitch their idea and then jet when they didn't get selected, but what are you going to do? If you like those early days of product ideas, I definitely recommend giving it a try.

Linkzie gets a bookmarklet

2011-02-20T10:17:43.000Z

I have added a bookmarklet feature to Linkzie. In the "organize your links" area you will see a little link there you can drag to your browser's bookmarks bar. Then if you are looking at a web page and want to save it into Linkzie, just click the bookmarklet. You will be able to set the name and category for the link and then be taken right back to the original page you were looking at. This should make it easier to get lots of links into Linkzie. Enjoy!

House of Genius

2011-02-19T02:16:31.000Z

I got to participate in a cool new brainstorming event called House of Genius last night. I won't say too much about it since the founders seem to prefer if the participants don't know exactly what to expect. But it was populated by a great group of smart and interesting people from many different age groups and backgrounds. It was in a super-swank enormous apartment downtown and was overall very unusual, engaging, and fun. It's funny, last week's Denver Hack Nite meetup was also in an inappropriately-fancy mansion. Maybe all my tech meetups will gradually start shifting from coffee shops and garages to luxury homes with helipads....

Going all in on Google Calendar and Android

2011-02-18T22:00:53.000Z

So I bought an LG Optimus V Android phone this weekend. I said if I finished the redesigns for peterlyons.com and linkzie.com I would treat myself. I've been a fairly die-hard PalmOS fan since the early days of the PalmPilot Personal. I think I got my first palm while still in high school probably in 1997 or so. I learned the Palm Graffiti input method. I soaked it right up. I continued on with a Kyocera cell phone/PDA based on PalmOS which I loved passionately and then onto the Treo for work and finally the Centro which I bought in March 2008 and have been using for the past 3 years. Throughout this whole time I maintained the same contact database and calendar. I had phone numbers in there of high school friends from before 10-digit dialing with area code was required. I had 4-digit on-campus extensions for random people I met in college filed under their first name only. It was amusing to go and look through the archives. I was pretty anal-retentive about putting everything in there. Guy who conducted my Region Band in high school: in there. That girl who cut my hair for $5 one time in college: in there. The number for the King 101 CS lab: in there. The same goes for Calendar. I think during one of the numerous computer data migrations I did over the years, I abandoned the early years of calendar data, but I still have data going back to as early as September 1999. I can tell you that I had a saxophone quartet rehearsal at 6:30pm on Wednesday September 22, 1999. Some 7000+ calendar records have all been migrated into my google calendar. And now they're all synced to my phone. w00t!

So anyway for the contact list I had to go through and trim down the people I hadn't been in contact with for a decade and then I hired someone on odesk.com to figure out how to migrate it into my gmail contacts list. The whole keeping track of people's email addresses and phone numbers is pretty much moot at this point since tracking folks down by name online is usually straightforward, but we all have those handful of folks that still have little or no online presence. So at this point keeping my contact book maintained basically makes sure I can easily call the people in my immediate network and perhaps more importantly the caller ID works properly when they call. But clearly we're close to the point where any manual address book maintenance will be a thing of the past.

So far the Optimus V seems like an awesome deal, especially for mobile-phobes like me who are late late adopters. The phone costs $150 and you get a prepaid $25 plan with no contract that includes unlimited data and TXT plus 300 voice minutes. That's saving me $15/month over my already pretty cheap AT&T plan which has no data at all. The savings on the plan will pay for the phone in 10 months. Plus now that I'm already done with my AT&T contract I don't have to start a new one. We'll see how the coverage is: Virgin Mobile uses a subset of the Sprint towers, I'm told. So far no issues and AT&T is not great in this area anyway.

Perhaps the best part is how much functionality I get for my $150. I get a full GPS device with updated maps. No thanks, Garmin, I won't be paying you $75 to update the maps on my Nuvi. I get a new Sonos controller. No thanks, Sonos, I won't be buying a $300 dedicated extra controller. I get another Rhapsody/Pandora streaming music endpoint. Nope, don't need another Ibiza Rhapsody MP3 player for $160. Plus now I can develop and test Android apps, which I just might do. Plus since I can get a 32 GB microSD card for $100, I can _finally_ have an MP3 player that can hold pretty much all of my music. I bought and loved a Creative Nomad Xen Xtra around 2005 probably. It's the size of an old cassette tape player, and on the rare occasions when I bring it out into public, people think I'm a hipster listening to 80s cassettes or something. But it was cheap and good and it took years and years for Apple and other mainstream mp3 players to hit 64GB. If there was a 64GB microSD card I could fit all my ripped MP3s from my CD collection (30GB or so) plus a ton of Rhapsody stuff and still have room for some photos and videos. It seems microSD is topping out at 32GB though, so next year I'll probably buy another cheap Android phone with microSDHXC and then achieve portable device Nirvana.

On Tuesday I spent all day in Boulder working at Coffee shops in between my three (3!) meetups in one day. Thus I had to move my car about three or four times during the day to park for free and not get a ticket. What this meant is that when my last meetup concluded around 8:30pm I had forgotten which of the locations was the most recent one and spent about 20 minutes walking a grid in the dark looking for my car. Yesterday I downloaded the parked car locator app. Never again will I have to wonder around! (Assuming I remember to actually tag my car's location during the parking process).

Any other Optimus-V users out there? Let me hear from ya!

New web designs

2011-02-16T02:35:08.000Z

As you may notice, the peterlyons.com site has been redesigned. I worked with Stephanie Henderson to get a new look now that I am an independent professional. I'm pleased with the results, and the blogs are more seamlessly integrated now.

I have also released a new version of Linkzie today featuring a brand new web design. The original layout of categories was based on rows and that has been changed to columns. Check it out!

Going all in on gmail

2011-02-10T13:13:50.000Z

So somewhere in college around 2000 I started running my own IMAP mail server and storing my own mail. Around 2007 I got tired of fighting the spam battle and switched to gmail. I migrated some of my current messages into gmail using IMAP, but I didn't want to dump everything in there. Today I realized I no longer have convenient access to those messages sitting in mbox format files on the linux laptop I rarely use anymore, so I hooked Thunderbird up to gmail and dragged my mail archives into gmail. That was about 12,000 messages between 2000 and 2007. Now it's all in one happy, searchable, spam-free place. As long as Google doesn't spontaneously disappear gmail, I should be in good shape.

node.js and command line jQuery

2011-02-09T03:41:10.000Z

node.js combined with the new query utility which makes jQuery-like functionality available from the command line. This type of thing used to take a 30-line script. Brilliant!

for N in {1..10}; do curl --silent http://news.ycombinator.com | query td.title a get "${N}"; done


YC Partner Harjeet Taggar Gives Insights [Interview]
Update your timezone defs: Russia abolishes winter time (DST)
Want to be a leader? Wash the Dishes When Nobody Else Will. 
How to be 100% sure your startup idea is good
Introducing the Google Translate app for iPhone
In China, alpha males carry designer purses
How To Get Your First 1,000 Users
How Much Money I Made From Side Projects In 2010
Facebook Buys Old Sun Campus in Menlo Park
Government investigation: No evidence Toyota electronic throttles malfunctioned

The Perfect Exercise Headphones

2011-02-07T07:00:11.000Z

I believe I have now found a pair of headphones to wear while exercising and traveling that is optimal. I've gone through a lot of headphones over the years as I'm sure many of you have. I've tried both the extreme low end of the price spectrum (less than $10) and the costs-more-than-your-mp3-player range (I owned some $180 Shure E2C earbuds back when they were $180 instead of $90). Of course I have learned some lessons the hard way. The following list starts out with weightier arguments and then decays into personal preference

The $20 ballpark is about right where when they are inevitably lost or broken, you shake it off and move on, but they are still able to sound OK
- As an aside, my $180 Shure headphones started making static interference two weeks after the warranty expired.
- If you want a nicer pair for hi-fi listening at home, by all means go for it and invest in some nice cans, just don't bring them when traveling
Sound isolating earbuds are the way to go for travel and exercise. I know traditional full-ear headphones are back in popularity with the snowboarders, but in general I want something that will pack up to essentially zero size and block out the exterior noise. I don't wear these while skiing where taking them on and off frequently might be required.
It is better if there is a right angle between the plug and the cord. This is a sturdier design less likely to cause static or drop a stereo channel with wear and tear.
The combination of in-ear earbud with behind-the ear wire for stability and security is the best. You don't want your earpiece falling out and dangling there while you are working on your downward dog.
The cord should include a small clip you can slide up from the neck to the earbuds to zip up the "Y" section of the cable for storage to prevent tangling
The cord should have a woven fabric sheath instead of plastic or rubber. This makes it easier to untangle.

So to this end I think the MEElectronics M6-BK Sport Sound-Isolating In-Ear Headphones are pretty much perfect. The only item they lack from the above list is the woven fabric sheath. The behind the ear wire is minimal but effective and they come with both single and triple flange earpieces.

And the bots show up

2011-02-06T02:00:35.000Z

I set up a staging server for a web app I'm working on yesterday. This morning I find this in my rails log.

ActionController::RoutingError (No route matches "/scripts/setup.php"):
ActionController::RoutingError (No route matches "/sql/scripts/setup.php"):
ActionController::RoutingError (No route matches "/web/scripts/setup.php"):
ActionController::RoutingError (No route matches "/scripts/setup.php"):
ActionController::RoutingError (No route matches "/admin/scripts/setup.php"):
ActionController::RoutingError (No route matches "/PMA/scripts/setup.php"):
ActionController::RoutingError (No route matches "/phpmyadmin/scripts/setup.php"):

Thus the shotgun bot scanning has commenced. It's funny how organic and viral (viral in the bad sense) the web is given it's all just computer programs.

dayJob.quit()

2011-01-28T00:01:39.000Z

I've quit my job writing data center automation software for HP. I'm striking out on my own doing independent web application development. I'll be building some information products and then hopefully a subscription based web application targeted at a small niche audience. I had been working on the same product for 6.5 years and I am ready to switch gears from the world of enterprise software to consumer-facing web applications.

I've been learning ruby, rails, and jquery in my free time for the past six months or so. It's been pretty enjoyable (especially jQuery), and there is a stark contrast between the slow grind of enterprise software and the lightness and speed of web development. I'll have more time and spare brain cycles for blogging and hopefully some more open source contributions, to which I look forward. Subscribe to this blog's feed to see how things pan out for me in the coming weeks and months. My working plan is to give self-employment a good long try before considering outside work. I plan to work at it at least until Labor Day, which is a full 7 months away. Wish me luck!

Wiki migration from MoinMoin to gitit

2011-01-10T05:08:29.000Z

So I've been hosting my own MoinMoin wiki for a while. I use it pretty heavily to manage my own little projects, packing lists, and whatnot. I used MoinMoin initially because it is written in python and we were using it at work. It has served me reasonably well, but I decide to shut it down and migrate all my data over to gitit instead.

My reasoning for the migration includes the following factors.

MoinMoin seems as far as I can tell to be impossible if not difficult to run under nginx, and in this world of low-RAM VPSes where I keep all my online stuff, nginx beats apache soundly in this regard, so I've moved everything over to nginx
I never really liked the notion of WikiWords to create links. I wasted a lot of mental cycles worrying about how to prevent MoinMoin from making RedHat a hyperlink unintentionally, etc. Thus I wanted to move to a different markup, such as markdown, which seems to be pretty widely used, especially in github projects for README files.
Using git as the data store is just perfect and brilliant. I'm only now realizing the magic of git's "no single master repo" distributed nature and now I can confidently add stuff to my wiki from multiple machines via either a web browser or a text editor and know that everything will eventually be merged up perfectly. This is one less thing for me to worry about manually rsyncing around the next time I move to a different VPS host.
MoinMoin has several glaring warning signs about programmers running wild with no product sense. For example, they don't have configuration files. They have python code you edit. The ruby community is another big fan of this misguided approach. Folks, asking programmers to edit code in a language they know to achieve complicated configuration (a la routes.rb) is fine. Asking sysadmins to configure basic shit like which TCP port your program should listen on is not. Make a configuration file, preferably in dead simple key=value format.
Another glaring warning sign is how MoinMoin handles its default content (help pages, etc), of which there is quite a lot. It just dumps them into your wiki right next to your own home-grown content with no distinction. There is no copy-on-write mechanism and no easy way to extract a list of pages you actually care about. I ended up combining some globbing telling me which pages had at least 2 revisions with a manual scan of page names to fish out my data from the sea of default MoinMoin pages. In hindsight, the smarter thing would have been to install a blank MoinMoin wiki, extract the set of page names, and then do a difference with the set of page names of my existing wiki, but my approach of just ls */revisions/00000002 worked almost as well, missing only a handful of pages, which a manual scan caught.

Sadly, gitit is not provided as an out of the box Ubuntu x64 package, which I REALLY prefer not to mess with. But alas, as with most webby stuff, I have to accept my fate and deal with manual installation, configuration, maintenance, and upgrade. Other than that fact, gitit has been working well enough so far. I had to write my own init script for it, but that is just annoying as opposed to a serious drawback.

The cool part about this migration was I was able to for the most part script it. I wrote a python script that would take the VERY limited set of MoinMoin syntax that I use (primarily headers, lists, and links) and convert it to markdown. Sadly its seems MoinMoin is one of the few markup formats that pandoc cannot use as input. :-( I also coded my script to start with revision 00000001 (used by MoinMoin) and commit each distinct MoinMoin page revision into git sequentially. Now in my migrated gitit repo, I have the full MoinMoin edit history for every page, which is pretty sweet.

Unit tests are like Ewoks

2010-12-02T03:24:43.000Z

Unit tests are like Ewoks. They are your friends. They will help you. They will help you fight formidable enemies. They are cute and cuddly and adorable.

Managing complexity and chaos

2010-11-23T04:02:23.000Z

I read somewhere (sorry I've forgotten where by now) that as an engineering, our #1 omnipresent mandate is "manage complexity". I interpret "manage" to mean "reduce" or "prevent" or "minimize" or "mitigate" in this context. When I read this, it resonated very strongly with me.

I was recently asked for feedback on one of the managers I work with and I thought in many was, in the manager role, the mantra is "manage chaos". I think that fits nicely. When everything is clear, well-understood, prioritized, under control, the team hums along efficiently. As a team gets overwhelmed or sidetracked or confused, chaos ensues, and it seems to me the role of management to prevent/reduce/minimize/mitigate that chaos.

OS X and CD burning non-smartness

2010-10-19T19:46:59.000Z

So since making the linux to Mac transition, I have this mindset of things on the Mac are supposed to "just work" and be fantastic and marvelous and flawless. This preconception was shattered again when I tried to burn some CDs recently. I downloaded and ISO and tried to burn it with Finder. In Ubuntu, if you try to burn a single .iso file to disc, it prompts you that you probably want to just make a disc from that CD image, and thus does the right thing. Not so on the Mac. It happily spit out a data disc containing a single .iso file. Duh. Then I went to burn a music CD using some .wav files a friend sent. Again, on Ubuntu, if you ask it to burn a CD of just .wav music files, it suggests that you burn an audio CD automatically. Mac, nope. Data CD with .wav files. Facepalm. In case you're curious, yes I did figure out that you need to use Disk Utility on the Mac to burn the .iso file and iTunes for the music CD. I thought this was supposed to be seamless?

jQuery and Ball World

2010-10-14T07:54:43.000Z

So in my first computer science course, one of the earliest labs with Stephen Wong was called Ball World. Professor Wong had set up some fill-in-the-blanks java code for us that brought up a basic frame. We wrote code to add some ball sprites and we had to define their velocity and have code to detect collisions and compute new velocities after balls collided with each other or the boundaries. I still remember the first time you get everything fired up and see two balls collide and bounce off each other realistically. Keep in mind this is somewhere pretty earlier in my first semester ever programming. It was pretty powerful and pretty exciting.

Now here I am with ten years or so professional programming experience. I'm teaching myself jQuery and javascript. Well, I guess I should put "teaching" in quotes because I'm basically just doing it since there's no learning required. I've had about twenty different things I wanted to do so far on my practice project, and each of them involves a 3-word google query like "jquery sort drag", a very tiny snippet of code, averaging about 2 or 3 lines, and it working perfectly the first time. It's amazing. I've got all kinds of HTML5 Ajaxy Web 2.0 goodness going and it works great and it's easy. AND there's a graphical debugger and rapid development tools. It's pretty darn sweet.

Home LAN fails and wins

2010-10-11T04:16:55.000Z

So my trusty Linksys wireless router up and died on me a few months ago. Of course, in a geek's world, lack of Wifi at home is a stop-the-line emergency, so of course I just immediately drove to a local brick and mortar store to get a replacement. I bought a Belkin as it was the cheapest. It seems to not have the ability to remember MAC to IP mappings for any significant length of time, which meant all my laptops and VMs were constantly changing IPs. I even ended up writing a little script to update /etc/issue on my VMs so I could see their current IP without needing to log in. Well, after a while I was finally frustrated with this enough to go see if the DD-WRT replacement firmware was available. It wasn't, so I thought I'd get a different model so I could enjoy the DHCP with static IP mappings goodness. I clicked my handy bookmark for local craigslist and bam, there's a guy in my town selling a Buffalo WHR-G126 for $25 with the latest dd-wrt already installed. w00t! After a conversation over a few emails, calls, and texts, we rendezvoused in town just an hour or so later and completed our transaction.

So now my network is so nice and lovely. All my devices use DHCP, but the router remembers the devices that "live here" and gives them each a static IP address and entry in DNS. Hurray for dd-wrt and craigslist.

In case you might find this useful, here's a script to find the current IP address of a machine and put it into /etc/issue so you can see it on the login screen without actually logging in.

First, edit /etc/issue with a placeholder line (as root) like this: echo "IP Address: " /etc/issue

Second, edit /etc/rc.local and append this little bit of code.

#plyons. Display the IP at the login screen so we can SSH in
#without loggin in on the console
getip() {
    IP=`ifconfig eth0 | egrep " inet addr:" | cut -d : -f 2 | cut -d " " -f 1`
}
getip
if [ -z "${IP}" ]; then
    sleep 10 #Wait for network to initialize
    getip
fi
perl -pi -e "s/IP Address: .*/IP Address: ${IP}/" /etc/issue

Now this has a lot of assumptions (one NIC called eth0, etc) and is in no way a generic solution, but for most VMs, it will probably get the job done as is or with a small tweak.

Cease and desist your password restrictions

2010-09-09T22:45:38.000Z

I just tried to create an account at Bed Bath and Beyond. They said my password could only contain letters and numbers. Many sites impose various restrictions on the password, including

Password must be exactly some length
Password must be greater than some minimum length
Password must be smaller than some maximum length
Password must contain some minimum number of "complex" characters
Password must NOT contain complex characters
Password is actually a 4-digit PIN

What this creates is A) major frustration B) a problem with remembering dozens of totally different passwords and C) inability to have a simple "low importance" password for your dozens of accounts you aren't super concerned about.

Stop this madness. Stop your silly password restrictions. They are incompatible, frustrating, and probably not helping. See the articles below for reference, but there are many others out there if you do a web search.

References

A response to "Handling Bugs in an Agile Context"

2010-09-08T21:07:27.000Z

Handling Bugs in an Agile Context is a blog post I came across via Hacker News this morning. As is often my experience when reading material on Agile from my perspective of an enterprise software developer, the experience was one of frustration and disbelief.

So let me quote the portions of the article I find untenable in a enterprise software realm.

Let’s start with the Product Owner. Not all Agile teams use this term. So where my definition says “Product Owner,” substitute in the title or name of the person who, in your organization, is responsible for defining what the software should do. This person might be a Business Analyst, a Product Manager, or some other Business Stakeholder.

This person is not anyone on the implementation team. Yes, the testers or programmers may have opinions about what’s a bug and what’s not. The implementation team can advise the Product Owner. But the Product Owner decides.

Most of this matches my experience to some degree. Yes we have product owners that are primarily business people. We actually have about two levels of these, we have what we call "Product Managers" who set the more abstract direction for the product overall, and another role called "Functional Architect" that is a mostly technical person that deals with more detailed issues. One way to think about this is the Product Manager decides mostly WHAT the product will do, and the Functional Architect refines that and specifies HOW the product will behave in detail. Note the Functional Architect doesn't define how the IMPLEMENTATION will be done, just the more detailed behavior.

This person is also not the end user or customer. When end users or customers encounter problems in the field, we listen to them. The Product Owner takes their opinions and preferences and needs into account. But the Product Owner is the person who ultimately decides if the customer has found something that violates valid expectations of the behavior of the system.

Yes, that does put a lot of responsibility on the shoulders of the Product Owner, but that’s where the responsibility belongs. Defining what the software should and should not do is a business decision, not a technical decision.

Well, that would be great if it worked that way, but it doesn't. The product owner doesn't have enough technical skill or understanding of the details to make decisions on specific bugs. I'm talking about deep, subtle bugs. Think error handling, file encoding, network performance tuning, etc. If a customer complains that our NFS server should by default be configured for a block size of 32768, I'm sorry but the product owner is just not equipped to make a decision on that. Yes, the technical team could explain the situation to him or her in enough detail for understanding, but the decision would be a "no brainer" dictated by how the team framed the explanation. There are trade-off decisions where the tech lead for the feature needs to make a trade off between two things that are both desirable, like high throughput and low latency. And it's not like we get these once in a blue moon. They happen many times in every sprint. We live them day in and day out, and it's the job of the implementation team to independently make good decisions on them. It's a bit demeaning to the implementation team to suggest that they are mere instruments of the product owner's omniscient will. It's a TEAM. EVERYONE takes into account many factors in making decisions on designs, implementations, and bugs every day. The team needs authority and autonomy to make the set of decisions that it is appropriate for them to own, and decisions need to escalate as appropriate for their scope and impact. In my experience, 95% of all bugs are correctly resolved by the team members without any input from the product owner.

Before we declare a story “Done,” if we find something that would violate the Product Owner’s expectations, we fix it. We don’t argue about it, we don’t debate or triage, we just fix it. This is what it means to have a zero tolerance for bugs. This is how we keep the code base clean and malleable and maintainable. That’s how we avoid accumulating technical debt. We do not tolerate broken windows in our code. And we make sure that there are one or more automated tests that would cover that same case so the problem won’t creep back in. Ever.

Well, that's cute but again, sometimes it's not possible to do that. Here's a real world example. I built a feature that installed some ZIP packages onto some servers. We tested it. It passed and worked. We moved on to the next sprint. Later on, we found out that if the ZIP file's install path contained non-ascii characters, we ran into problems and it failed. OK, so that's a bug. So you are saying "we fix it". You say "We don’t argue about it, we don’t debate or triage, we just fix it". Well, in this case, after several days of me trying to "just fix it", I informed the team that in order to correctly fix this bug I would have to re-implement the entire user story taking a very different approach. This would take several days and since it was a de-facto rewrite, all of the tests would need to be re-run. So how does your advice apply here? It doesn't. We needed to debate and discuss and plan and adjust the backlog and otherwise deal with this reality. I wish I could go to the fire departments that are currently battling the giant wildfire in the next town over and say "Don't debate. Just extinguish it". But it's not a helpful thing to say.

And since we just fix them as we find them, we don’t need a name for these things. We don’t need to prioritize them. We don’t need to track them in a bug tracking system. We just take care of them right away.

Sorry, but this is the delusional rubbish the agile community puts out that makes it OK for me to dismiss you as utterly and hopelessly clueless. Have you really never heard of a bug that is time consuming or difficult to fix? Have you never seen a bug that needs multiple completely different fixes tried before one that really works is identified? Are you accepting applications for citizenship in your universe? It sounds nice.

Usually the motivation for wanting to keep a record of things we won’t fix is to cover our backsides so that when the Product Owner comes back and says “Hey! Why didn’t you catch this?” we can point to the bug database and say “We did too catch it and you said not to fix it. Neener neener neener.” If an Agile team needs to keep CYA records, they have problems that bug tracking won’t fix.

Well, we have a lot of bugs. My product has been around for almost a decade now. There's value in tracking them. For one, it gives management some concrete numbers to understand that they have technical debt, the product has issues, and we're accumulating more. For two, people hit the bugs. They hit them over and over again. It's a waste to have the issue re-triaged, re-explained every time. We track them so there's one place that explains what this bug is, how it is reproduced, why we haven't fixed it, and how you can work around it. Whether we track this in a bug tracker or a wiki or a KB or whatever doesn't seem as important to me as acknowledging the fact that there are issues that might affect users continually for quite a while (maybe for the rest of the life of the product), and not tracking that data makes the problem worse.

Further, there is a high cost to such record keeping.

Arguably. But there is also a cost to just abandoning them and leaving the new team members and customers to encounter them over and over again and refile the same bug. Yes, sometimes these things last a long time and there's no cost effective way to solve them. For example, in early versions of my product we installed Windows OSes via a DOS boot environment. There was no other viable alternative at the time. DOS has crippling network issues that we couldn't solve, so under some circumstances copying a full OS over the network on DOS would result in DOS hanging. We can't fix this. We just waited for Microsoft to announce WinPE, but that took several years. Instead we documented that if you had an Intel NIC and encountered this issue, there was an obscure workaround you had to do. This seems like a perfectly valid use of a long-term bug database to me.

And if we’re not doing things right, we may find out that there are an overwhelming number of the little critters escaping. That’s when we know that we have a real problem with our process.

OR, maybe we have a legacy code base with technical debt. Maybe the folks who wrote that code don't work here anymore and this is the only way we find the problems with it. This doesn't necessarily mean we're not doing Agile correctly.

Stop the bugs at the source instead of trying to corral and manage the little critters.

Thanks. Tell that to my several million lines of legacy code in a half dozen languages that runs on 72 different operating system versions. Done "stopping them at the source" yet? I'll wait. Still not done? Hmm, what's the problem?

Anyway, that concludes my response to this blog post. The general theme I see in the Agile world is the practitioners are mostly working from a mindset of small web development projects with a new code base. Building enterprise software has realities that create real struggles for us, and we're looking for help, but the pundits out there generally dismiss our struggles without really understanding them and acknowledging their reality. And that's what I find so frustrating. The truth is most of these agile folks could probably provide us with real useful insights, but they first need to come to terms that we have some real problems that need to be considered even if they don't fit within their idealized vision for agile utopia.

Sonos pricing insanity continued

2010-09-07T07:17:01.000Z

So I'm a fan of Sonos music systems, but they've always been expensive. Almost prohibitively so. But now I can either spend $349 for an additional controller that controls my sonos and does precisely fuck-all else, or I can buy an iPod Touch on craigslist for $125 and then download the free Sonos controller application, plus do all the other iPod Touch stuff. I'm not sure who Sonos thinks is going to buy this thing anymore other than the utterly rich and utterly clueless.

No one really cares about web accessibility anyway...

2010-09-07T05:12:19.000Z

So if a picture is worth a thousand words, shouldn't it be:


    
        one two three four five six seven....

No squids for me, Bruce

2010-09-05T05:16:30.000Z

Bruce Schneier is a leading author in the technology security and cryptography fields. He is exceedingly rational and pragmatic. I consider his insights on dealing with terrorism to be bar none the best available. His blog often has great insights. However, for reasons I can't understand, he always posts a random post on Fridays about squids. Yes, the sea creature. I don't know why, but it stopped being cute long ago.

To that end, I've created a Yahoo Pipe Feed of the Schneier on Security blog with the Friday Squid Blogging filtered out. Feel free to subscribe and enjoy.

UNIX is broken

2010-09-04T19:40:03.000Z

Reading this blog post just makes me cringe. This is why I am so persistently hesitant to bother with C and low level systems programming. They are broken. They are impossible to get right. If even after numerous decades, the cat program still has bugs in basic system interaction, it's not the programmers, it's the system. Thanks, but no. I'll code in a high level language that lets me focus on delivering functionality to my users, not performing sacrificial rituals to the Gods of K&R, signal processing, and bitmasks.

At work we have certain code bases that no matter how long we tweak them and how many dozens of bugs we fix, they just never reach a point of stability and reliability. You need to just abandon them and rethink the problem from scratch. Come at it with a different approach and a brand new code base and some analysis that actually addresses all of the edge cases.

Interestingly, I think the mobile space (iOS, Android) have potential to finally provide a next generation operating system where applications written by average, normal programmers usually function perfectly. Contrast this to most existing operating systems where most programs can be easily made to crash or misbehave in the course of normal use. We've got to get beyond this notion that in order to make a well-behaved application, the developer needs encyclopedic knowledge of bizzarre archaic minutia of the underlying OS.

Incidentally, that post is written by Slava Pestov, the creator of the jEdit text editor that has been my primary editor for many years. This guy is good.

On flash eating my key browser keyboard commands

2010-08-30T19:48:19.000Z

Dear Universe,

Please manifest a way for me to specify that my key browser keyboard shortcuts will work even when I'm watching a flash video. Specifically I always want "close window", "new tab", and "cycle active window" to work. I hate that after watching a video I have to click on the surrounding non-flash web page to get my keyboard commands working again. Please post a comment on this blog post when you are done manifesting this.

Thanks,

Pete

How to Install Ubuntu 10.04 over a LAN

2010-07-17T04:40:24.000Z

Overview and Requirements

So I've volunteered to help Boulder Community Computers set up some automated Ubuntu OS installations over their local area network. Being as this is a large part of what I do professionally, this is something easy for me to do that should help them be more efficient. Their network layout is going to look something like this I think:

Internet
   ^
   |
Office LAN (private addresses)<-> Various workstations (DHCP from router)
   ^
   |
Multihomed Netboot Server
   ^
   |
OS Build Network (other private addresses)
   ^
   |
   <->Bare metal target machines needing  an OS (DHCP from Netboot Server)

OK, so let's talk about some of the goals/requirements:

I want the build network to be separate from the office network. This will keep things simple, avoid DHCP collisions, prevent users from accidentally provisioning their workstations, and keep the intense network traffic of OS installation off the office LAN
I want to mainly download cached objects from the Netboot server to the targets. I want to avoid downloading the OS over the Internet connection repeatedly.
I don't want a lot of ongoing maintenance tasks
I would like to be able to do a fully automatic hands-off installation
I would also like to be able to boot into a live CD to test compatibility
I would like to be able to do an interactive custom install
I do want internet access from the OS Build Network (mostly for NTP, but it's just handy in general)

So overall, Ubuntu can meet all of these goals easily. Here are the details.

Setting up the Multihomed Network Install Server

So we want an Ubuntu server machine with two network interfaces (multihomed) to act as our network install server. This machine will provide DHCP/PXE, TFTP, NFS, HTTP, and HTTP proxy services to the target machines. It will also act as their default gateway, routing their traffic from the OS Build Network out to the Office LAN and then out onto the Internet and back. Here's how we get this server installed and configured.

Install a basic Ubuntu 10.04 Server amd64 host

I used a VirtualBox VM for this while testing this setup, and we will probably use that in their "production" environment as well. If you use a physical computer, make sure it has 2 network connections. You can select "OpenSSH Server" during the install or install openssh-server later. Once the OS is up and running, we need to configure both of the network interfaces. We'll be using RFC 1918 private addresses for both. In this case we have a 10.0.0.0/8 network as eth0 which is the Office LAN and a 192.168.0.0/16 network as eth1 which is the OS Build Network. The Office LAN is your typical home or office type network where a router is serving private IP addresses over DHCP and providing Internet access. In VirtualBox I set the first interface to be a bridged interface eth0 and the second interface to an internal only interface eth1. To do this we edit /etc/network/interfaces as root as follows.

A quick note about code samples and long lines

Several of the code samples in this article require very long lines. These don't display well on the web, but they are displayed using a CSS overflow horizontal scrollbar. Hopefully they are easy enough to read, copy, and paste.

# This file describes the network interfaces available on your system
# and how to activate them. For more information, see interfaces(5).

# The loopback network interface
auto lo
iface lo inet loopback

# The primary network interface
auto eth0
iface eth0 inet dhcp

# The internal build network for OS installation
auto eth1
iface eth1 inet static
address 192.168.8.1
netmask 255.255.255.0

After that edit, activate the config with (as root) ifdown eth1; ifup eth1. Verify it's working by pinging both of your IP addresses (the 10.X.X.X one and the 192.168.X.X one).

Install and Configure PXE Boot Services

Install dnsmasq via apt-get install dnsmasq. Then as root do a mkdir /var/lib/tftpboot. Edit /etc/dnsmasq.conf to look as follows.

interface=eth1
dhcp-range=192.168.8.100,192.168.8.254,12h
dhcp-boot=pxelinux.0
enable-tftp
tftp-root=/var/lib/tftpboot/

Restart it with service dnsmasq restart

The next big step is to make the Debian Installer available as a PXE boot image. We're basically following this article here, except we're doing Ubuntu Lucid 10.04 instead of Feisty. Do the following.

cd /var/lib/tftpboot
sudo wget http://archive.ubuntu.com/ubuntu/dists/lucid/main/installer-i386/current/images/netboot/netboot.tar.gz
sudo tar -xzvf netboot.tar.gz
sudo rm netboot.tar.gz

OK, at this point we should be ready to boot our first test target machine into the Debian Installer network boot service OS. PXE boot a machine on the OS Build Network (again, I used a VirtualBox VM). You may have to fit a key like F12 to instruct the machine to boot from a network card. I have to hit F12 then "l" to boot from the LAN on my VirtualBox VM. If all is well, you should see a lovely Ubuntu installer menu like this (you won't have the menu items containing "10.04" yet, but we'll add them later).

So now we can use this to install Ubuntu onto the target. Well, almost. We don't have Internet access from our OS Build Network working yet, so let's get that going.

Setting up Routing to the Internet

We want to allow our target machines to connect to the Internet, which means we will configure our Net Boot server as a very basic router and NAT firewall. This is often called Internet Connection Sharing when used for this simple purpose. We're basically following Internet Connection Sharing Setup docs from the Ubuntu community wiki. As root, add net.ipv4.ip_forward=1 to /etc/sysctl.conf. Then run these commands.

sudo iptables -A FORWARD -i eth0 -o eth1 -s 192.168.0.0/16 -m conntrack  --ctstate NEW -j ACCEPT
sudo iptables -A FORWARD -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT
sudo iptables -A POSTROUTING -t nat -j MASQUERADE 
sudo iptables-save | sudo tee /etc/iptables.sav
sudo sh -c "echo 1 > /proc/sys/net/ipv4/ip_forward"

Now add iptables-restore < /etc/iptables.sav to /etc/rc.local as root just above the exit 0 line. This will get us going again when we reboot.

Now you should be able to PXE boot a target, select "Install" from the menu and do an Ubuntu installation. However, it's going to be a normal install over the Internet, which is fine for doing one or two boxes, but since we want to crank these out en masse, we'll want to cache the bits and grab them locally.

Caching the Ubuntu Packages with apt-cacher-ng

I tried both squid and apt-cacher for a solution to locally cache the bulk of the Ubuntu binary package payload, but neither worked very well. apt-cacher-ng seems to do exactly what we want and nothing more with no configuration. install apt-cacher-ng. This "just works" with no setup. After it's installed, if you PXE boot a target, select "Install" and go through the interactive install, when prompted for a proxy, enter http://192.168.8.1:3142. This is the OS Build Network IP of the Net Boot server and the apt-cacher-ng proxy port. Now all your packages will be cached on the OS Boot Server. The first install will download them from the Internet. Subsequent installs will get them from the local cache and thus be much much faster and more efficient.

Hands-off Automatic OS Installation

OK, we now have network based OS installs working, but since we're planning on doing many of these, we want to fully automate this. To do that, we'll use a preseed file to configure the Debian Installer to not ask any questions. Configure a file at /var/lib/tftpboot/bococo.seed with the the content linked here: bococo.seed.

Now we'll add a new item to the PXE menu to trigger a hands-off automated OS install. Edit /var/lib/tftpboot/ubuntu-installer/i386/boot-screens/text.cfg and add this entry.

label install-10.04-hands-off 
        menu label ^Install 10.04 Hands Off
        kernel ubuntu-installer/i386/linux
        append vga=normal initrd=ubuntu-installer/i386/initrd.gz locale=en_US.UTF-8 debian-installer/keymap=us auto hostname=bococo preseed/url=http://192.168.8.1/bococo.seed -- quiet

In order to get that file over HTTP, we'll server it up with the nginx web server. install nginx via apt-get install nginx. Then we can edit /etc/nginx/sites-enabled/default and change location / block to set the new document root to the tftpboot directory so it looks like this.

        location / {
                root   /var/lib/tftpboot;
                index  index.html index.htm;
        }

Restart nginx via service nginx restart. Now we should be able to PXE boot the target and select the "Install 10.04 Hands Off" option and the entire thing should happen automatically.

Network Booting into the Live Image

One final bit of utility functionality we want to provide is a network bootable live Ubuntu image. This will be handy for testing hardware compatibility, performance, etc. First we'll download the primary Ubuntu Desktop CD image and make it available over NFS. Do the following as root.

apt-get install nfs-kernel-server
cd /var/opt
wget 'http://ubuntu.cs.utah.edu/releases/lucid/ubuntu-10.04-desktop-i386.iso'
mkdir /var/lib/tftpboot/ubuntu-10.04-desktop-i386
echo /var/opt/ubuntu-10.04-desktop-i386.iso /var/lib/tftpboot/ubuntu-10.04-desktop-i386 auto ro,loop 0 0 >> /etc/fstab
mount /var/lib/tftpboot/ubuntu-10.04-desktop-i386
echo "/var/lib/tftpboot/ubuntu-10.04-desktop-i386 *(ro,sync,no_subtree_check)" >> /etc/exports
service nfs-kernel-server restart

Now add another entry to your PXE menu for the live Session. I also like to move the menu default option to the live session, so my final /var/lib/tftpboot/ubuntu-installer/i386/boot-screens/text.cfg looks like this:

default install-10.04-hands-off
label live
        menu label ^Live Session 10.04
        menu default
        kernel ubuntu-10.04-desktop-i386/casper/vmlinuz
        append initrd=ubuntu-10.04-desktop-i386/casper/initrd.lz boot=casper netboot=nfs nfsroot=192.168.8.1:/var/lib/tftpboot/ubuntu-10.04-desktop-i386          -- quiet
label install
        menu label ^Install
        kernel ubuntu-installer/i386/linux
        append vga=normal initrd=ubuntu-installer/i386/initrd.gz -- quiet 
label install-10.04-hands-off
        menu label ^Install 10.04 Hands Off
        kernel ubuntu-installer/i386/linux
        append vga=normal initrd=ubuntu-installer/i386/initrd.gz locale=en_US.UTF-8 debian-installer/keymap=us auto hostname=bococo preseed/url=http://192.168.8.1/bococo.seed -- quiet
label cli
        menu label ^Command-line install
        kernel ubuntu-installer/i386/linux
        append tasks=standard pkgsel/language-pack-patterns= pkgsel/install-language-support=falseac vga=normal initrd=ubuntu-installer/i386/initrd.gz -- quiet

And that's it! We can now netbook into a live session, an interactive install, or a fully automated install.

References

Acknowledgements

Thanks to the authors of the above reference web pages and Nick Flores for collaborating on this.

Windows Server 2008 Setup Annoyances

2010-07-10T07:12:54.000Z

So I do a lot of work with automated unattended intallations of Operating Systems, including Windows. Here's some of my primary complaints about the new Windows setup program in Windows Server 2008.

No good way to validate the unnatend.xml file. Now Microsoft does provide tools to help generate these files, and hopefully any file you generate with those (graphical, Windows-only) tools should be at least well-formed and semantically valid. However, there's no way to do a deeper validation that a given XML file is compatible with a particular target machine.
The unattend.xml encodes the processor architecture all over the place for no reason. I have to do a search and replace of processorArchitecture="x86" with processorArchitecture="amd64" in every component tag. There's pretty much zero information in most unattend.xml files that's CPU architecture specific anyway. This is a real nuisance. I should be able to use the exact same file on x86 and amd64 without issues.
Setup doesn't do the simple but important hardware compatibility validation that would make users' lives easier. For example, neither winpe nor windows setup with complain if the target system has insufficient RAM. WinPE will just behave very oddly and things won't work. There's no checking for sufficient disk space ahead of time or that the disk layout is feasible. There's no checking for suitable network or storage drivers. When you don't have viable storage drivers, you just reboot out of WinPE into a lovely Blue Screen of Death 0x7b stop error. Hurray! Similarly, no one at MS seems to care if you don't have a working NIC driver. The OS has the word "Server" in it. You need a network driver or your OS is in a useless void.
The fact that windows setup reboots into an environment with zero networking and zero third party applications allowed is just a mind-boggling recipe for end user frustration. We have to resort to brute time out calculation to even know whether the Windows install worked or not. We can't provide a good user experience to people looking to do UNATTENDED installs.
Another one in the "we have no idea what unattended means" even though our configuration file is called "unattend.xml" department: STOP PRESENTING GUI DIALOGS. There are many issues that unattend.xml supposedly allows a showGui="never" but in my experience they either have no effect (GUI displays and halts the install anyway) or they just flat out break the install. Microsoft just doesn't "get it" here. Any automated install isn't a fully attended graphical install rejiggered to try not to pop up a GUI. It's an entirely different use case. Get it through your thick skull: NO ONE IS LOOKING AT THE CONSOLE. THESE ARE SERVERS. THE INSTALL IS BEING DRIVER AUTOMATICALLY BY SOFTWARE. NEVER EVER SHOW A GUI.
Same category of cluelessness. Windows setup doesn't return a non-zero exit code on failure. Duh. I'm aware of some other mechanisms like setupcomplete.cmd that MS claims to provide for this, but from what I can tell after several attempts, they simply don't work.
The log files and error messages are just a mess. And I'm not even talking about obscure edge case failures. Simple things like an invalid product key or computer name can create weird mysterious failures and behavior.

Remove your SCM system from your job postings

2010-07-07T04:54:58.000Z

Most job postings will list one or more source code management (SCM) tools in their laundry list of required skills and buzzwords. This seems to both be unnecessary and also to miss the point. As an employer, your concern shouldn't be whether or not your candidate is intimately familiar with subversion, git, perforce, starteam, mercurial, CVS, sourcesafe, or whatever system your code happens to reside within at this point in time. Why not? Because it just doesn't matter and it doesn't help you distinguish good candidates from bad. This in my mind seems akin to asking potential postal employees "Do you have experience driving from the right hand side of a boxy little truck?". It's just not something that's going to be a stumbling block. If you can drive, you'll get the hang of the postal delivery truck soon enough. It's the same with SCM. If you have worked successfully on a few sizeable projects in one or two of them, you'll be fine. I've never heard this story: "Oh yeah, we hired this woman Sandra and she wrote this fantastic code for us, but she just could not figure out how to commit it to subversion, so we had to let her go." It's just not going to be the issue. Well, what is going to be the issue?

In this context, it's more important to see if the candidate understands branch and release organization and management as generic principles. When and how should the code be branched? What steps do you take when it's time to ship a release? When a bug needs to get fixed against an old version of the product, how does that work with SCM? These types of questions are OK sanity checks to spend 2 minutes on in an interview, but they don't need to be on your job posting.

Now, here's what I see missing that seems in my mind much more likely to actually cause you problems down the road. Does your candidate understand software packaging, distribution, installation, and upgrade? Do they understand the principles of package management systems like RPM or DEB, and the complexities around dependency management, upgrade, and rollback? Do they understand how to package and ship (or not) third party libraries? These questions may be more or less relevant based on the deployment model you use (web vs. bundled vs. embedded, etc). However, for lots of projects, they are key and lots of companies are clearly in the dark here.

So the summary: don't bother listing a specific SCM as a requirement in your job posting. Instead, briefly interview your candidates on general SCM methods and principles. And in addition to that, given your deployment model(s) find out how much your candidate knows about packaging and installation.

On Idempotence, intention, and unix commands

2010-05-21T09:42:05.000Z

Idempotence means that running a command or function several times produces the same result as running it only once. This is an very important design principle that is a blessing when used appropriately and a scourge when not used where warranted.

For analogy, imagine you ask a housemate (or butler if that's how you roll) to empty the dishwasher. They dutifully go over there, open the dishwasher door, and find it's already empty. How do they react? Do they come back to you shouting in confusion "You fool! How can I empty the dishwasher if there's nothing in it! Oh woe is me. What am I to do?"? Or do they just think to themselves "score!" and go on a coffee break, leaving you to go about your business trusting that the dishwasher is now empty?

Another analogy is from the military's notion of "management by intent" wherein a commander might order his troops to "have camp fully operational by noon" as opposed to dictating specific tactics that must be taken in order to achieve the intended outcome. This way, the troops can rely on their own abilities to achieve the intent and are empowered to respond to changing or unexpected circumstances independently.

Now, when it comes to computer programs, UNIX has a mixed bag of utilities that understand this and some that don't.

mkdir /tmp/2;echo $?;mkdir /tmp/2;echo $?
0
mkdir: cannot create directory `/tmp/2': File exists
1

rm /tmp/foo;echo $?;rm /tmp/foo;echo $?
0
rm: cannot remove `/tmp/foo': No such file or directory
1

So the bad examples include mkdir, rmdir, rm, ln, and perhaps kill (debatable). Think about how much simpler using a command line and writing shell scripts would be if these were idempotent and instead of panicking in horror when the user does not know the current state of the filesystem, just allowed the user to describe the desired end state. I would love to have idempotent and recursive by default commands like mkdir -p or rm -rf in combination with a transactional filesystem with built in undo capabilities.

Good idempotent examples include touch, tar, zip, cp, chmod.

So the point about design and usability here is it's good to ask oneself "What is the user's intent here?", and try to do everything in your power to work in concert with that intention. A strong and painful negative example from my career has to do with the fact that the Solaris patchadd program is not idempotent and it doesn't return exit codes according to the user's intent. So when I run patchadd 123456-01, really my intention is "I want this system to be OK with regard to patch 123456-01". patchadd will return a non-zero exit code if the patch is already installed or the patch is not applicable to the server or if a newer revision is already installed. As a user of patchadd, I don't care. It's all success to me, and nor do I want to be bothered with implementation details within patchadd such as not installing a patch if a newer revision is already installed. I think many shell scripts would be a lot smaller and clearer and simpler without always having to wrap mkdir in an if [ ! -d /blah/dir ] clause to avoid spurious error output.

A few other links on this topic:

The Importance of Idempotence (devhawk)
Java Glossary entry on Idempotent. I like this quote "Elevator call buttons are also idempotent, though many people think they are not."
Stack Overflow: What is an idempotent operation?

Bleeding Edge and Rotting Core

2010-03-19T08:53:53.000Z

I just wanted to post some thoughts on the topic of selecting software components with regard to the maturity thereof. I think overall the programmer community is by default gung-ho about the bleeding edge. We like the shiny new toys with the bells and whistles. Once something's been around enough to have its weaknesses well understsood, we find it very frustrating to have to continue to work with it. I'm not going to offer any specific recommendations, just some things to keep in mind. The general gist though is that it takes some hard-earned pragmatism and real production experience to understand the value of using older releases of components.

First, let's define some terms. We're familiar with what is known as the bleeding edge. The new hotness. The stuff straight off the presses instilled with the glimmering light of state of the art knowledge. There's probably always been a lot of this, but there seems to have been a flurry in the past five years of so of interest in ruby, rails, erlang, clojure, scala, dozens of python app and web frameworks, etc. On the other hand, we have the old guard, which I'd like to call the rotting core. Generally we shy away from this, but there are times when it is absolutely the correct choice in certain situations.

So, let's look at some pros and cons.

Bleeding edge pros:

The freshest and (usually) best designs and thinking are made available
Almost always more succinct and expressive
Often more coherent, clean, and consistent
Embodies improvements based on lessons learned from past failings and shortcomings
Development tools and processes are sometimes more productive

Bleeding edge cons:

Development tools are usually immature and inferior
- IDE support is likely to lag behind
- Debugger may lag behind as may remote graphical debugging
- Performance profilers might not be there
Deployment issues may not have been well addressed yet
Updates will come more frequently causing churn
Software has not had as broad testing in production and is therefore likely to have more "surprises". Sometimes these can be showstoppers.
Community size will be smaller
Depth of knowledge in the community will be shallower
Standard library may be undergoing more flux

Rotting core pros:

Stable, known quantity. It may have warts and bugs, but at least we're aware of most of them by now
Development tools generally have solid support including remote graphical debugging, mature performance profilers, etc
Community size will be larger
Community depth of knowledge will be much deeper
Updates are rare and only for occasional major issues or security patches
standard library will be well known and stable

Rotting core cons:

Less exciting to developers. Yesterday's designs and paradigms.
Often tedious compared to the bleeding edge
Support issues. Standard answer may always be "update to the latest version"

And now, let's back this up with some examples and anecdotes. I think when it comes to rotting core technologies, you have both the "oldie but a goodie" category and the "oldie and a baddie" one. Currently my project has a component written against the now ancient Python 1.5.2 runtime, and we have hundreds of thousands of copies of that component installed at customer sites. It is running on something around seventy different OSes. Now, at the time when that component was originally written, this was close to the bleeding edge. We've still not entirely upgraded it because it's an oldie and a goodie. We've patched it a bunch and run it under huge loads and huge scales. We know what it can do, and we know what it can't do. We even had famous python educator Mark Lutz (Programming Python) come in to train us and give us quizzical looks when we explain that half of what he is saying doesn't apply to us since it wasn't available in python 1.5.2. Over the years, I've come to see the merits of this and even though its frustrating, the business reality is that every year that stuff continues to run without issue is bettering the return on the initial R&D investment. It ain't broke, so we're not in a hurry to fix it.

Of course, on the other side, you've got things like Java 1.2, which I also worked with. Python has come a long way since 1.5.2, but really it's still basically the same deal, and the design was good from the start. Java has probably come even farther, but the design was a mess from the beginning and they've since seen the error of their ways and made some great improvements. I would put that one in the "oldie but a baddie" category and do what it takes to upgrade.

I remember chatting with a stranger on a plane after we each noticed that we were both programmers and were both actively programming on the plane. This was a few years ago and Ruby was still pretty much bleeding edge. He looked at me with desperation and asked me if I knew anything about debugging deadlocks, threading issues, and core dumps since his production ruby app was regularly hitting issues and his team was basically at a point where they didn't have the knowledge or tools to solve them, and it was jeopardizing their whole project. Sadly I couldn't offer any help, but I could certainly sympathize.

I also have a friend who used to work at a DNS registry run by someone very much of the "rotting core" philosophy. They ran Solaris 8 and ancient versions of lots of core C/unix utilities (bind et al), and to actually run versions that old took significant effort on their part, but it made sense for that project. They are running a piece of the Internet backbone. It's not bleeding edge stuff. It just needs stability, stability, stability, and those are the tools they needed to meet their business goals.

So next time you join a new project and start to reflexively freak out when they explain their software stack, supress your urge for a minute and get some information about the choices they have made and the reasoning and circumstances that got them where they are. You might be surprised at the difficult but pragmmatic choices that were made and hopefully you can admire and appreciate the character of those who made them.

And finally, think about the value of being able to look across a broad set of available components and correctly determine where components are in a "sweet spot" of their lifecycle, ripe to be chosen and deployed at length. That is a deep wisdom that is a long time coming.

MoinMoin Columns Macro

2010-03-08T00:43:31.000Z

I just updated the "Columns" macro for the MoinMoin wiki. This allows you to lay out a wiki page in two to ten columns. This makes it easier to get lots of info on one page in certain situations and I've used it to great benefit on my personal wiki where I organize my stuff.

Here's the MacroMarket page where my update has been posted for discussion. The original author may not like it, so it might not become the canonical fork, but that's how it goes with open source.

Here's what it looks like with four columns:

Optional Syntax Should Be Illegal

2010-02-14T03:33:31.000Z

Why in the world do some programming languages include optional syntax? To a true type A engineer, this is incomprehensible and unacceptable. For example, in Adobe's ActionScript, statements may optionally be terminated with a semicolon. Usually this is not required, except in a few situations you need it. Evil. The statement that our number one job as software engineers is to manage complexitity really resonates with me, and willy nilly allowing of optional syntax just destroys consistency, predictability, and simplicity for no reason whatsoever. Optional syntax seems to me a bad language design smell that indicates the language authors need to rethink a bit and find something that works always and should be required.

Part of the impetus for this post is my annoyance when my thinking cycles are wasted on unimportant details in a source code file. I'd rather have a strict format so that whenever I am reading or writing in a language, there will be a strong and deep consistency. I don't have to spend time deciding whether I'm going to use some optional syntax or which of the several ways to express the same thing I'm going to use. Similarly, when I come upon someone else's code and it's a mixture of two optional approaches, I feel compelled to go and make it consistent, which is another time waster.

Environment Variables Considered Harmful

2010-02-14T03:30:14.000Z

Many projects reference environment variables at either build time, install time, or run time to handle configuration that can't be made to work across all of the target environments. It is better to use plain text simple configuration files for the reasons that follow. First, let's quickly review common usage of environment variables.

Directory path to supporting tools and libraries (JAVAHOME, LDLIBRARYPATH, CATALINAHOME, etc)
Customization of build time locations (BUILDDIR, OUTPUTDIR, DIST_DIR, etc)
Customization of compiler options and other build time configurations (STATIC_LINK, etc)
Settings that apply OS-wide and to several programs (httpproxy, etc). In theory this would almost make sense. You set your httpproxy environment variable in one place, and any program that makes HTTP requests respects that setting. In practice, these settings are more realistically effective higher up in your desktop environment, and AFAIK in the whole GNU/Linux/UNIX ecosystem, there are only a small handful of cross-program environment variables that are actually used commonly.

So what's the problem with environment variables?

The are ephemeral, nebulous, stored in memory within your shell and process tree
How and where they are set is inconsistent across shells (~/.bash_profile, ~/.zshrc, etc)
The syntax to specify them is needlessly different across different shells (csh vs. bash vs. cmd.exe, etc)
How to fully unset them varies per shell and is often unclear
There is widespread confusion on the distinction between shell variables and environment variables, how to set each, and how each interacts with subprocesses
They are often tied to a user account due to where they are specified above, and can vary between login shell verses non-login shell. They can therefore often vary when a program runs via init compared to run from an interactive root login shell. This can be difficult to detect and troubleshoot
They are rife with major security concerns and a common attack vector

All of these reasons combined mean that in general environment variables are losers in our goal of managing complexity and making simple, easy to use software that is cross platform. So what's the solution? The solution, as it so often is, is simple plain text configuration files. At the end of the day, environment variables end up set in a shell script as KEY=VALUE type pairs, and that's where they belong in a configuration file on the filesystem. How does this make things better?

One consistent place to set your application's configuration
Same syntax regardless of shell, programming language or OS
Files on disk are concrete and reliable. You can email it to someone for help with troubleshooting and be confident about its content

So go forth and configure with simple plain text configuration files. And there will be much rejoicing.

Business hours

2009-09-22T10:38:56.000Z

NOTICE: To all businesses with a single physical location and a web site. You will put your address, phone number, and business hours on your home page. There will not be a "Contact" page. There will not be an "About" page. END OF NOTICE

The wheel of not waiting

2009-09-18T08:01:00.000Z

So flash videos are everywhere now. Generally as they load they show some spinning wheel type graphic. The problem is as an end user, I have no visual differentiation between a video that is loading slowly and a video sitting there waiting for TCP from an overloaded server that is simply never going to work, and certainly not in a timeframe smaller than my attention budget. Show me whether or not you are getting any data, and I might be willing to wait, but if you have me watching your spinner until your TCP connection times out, you are just frustrating me.

How to disable wpautop in WordPress blogs

2009-05-23T19:53:19.000Z

So, when creating a WordPress blog, even if you are editing in HTML mode, WordPress includes a feature called "wpautop" that will replace any pair of line feed characters in your post markup with a

tag. This is helpful I think in general for people who blog mostly paragraphs with some links and images. However, if you blog with more complex markup, this can invalidate your HTML. I run my HTML through the W3C HTML Validator to check it and wpautop can cause validation to fail. I hunted around online for an easy way to disable this and didn't see one, so I made the changes described below.

One thing to keep in mind is that if you HAVE been relying on wpautop and you have not been including your own explicit

tags, disabling wpautop will cause all your paragraphs to run together and thus your layout will be broken. To prepare for this, pre-edit all your posts so they have the paragraph tags and remove extra blank lines from them. You can check how they look in that state since when there are no blank lines wpautop won't do anything. Once they look good like that, you can disable wpautop.

In your WordPress installation, edit the file wp-includes/formatting.php. Search for "function wpautop" and insert the following two lines at the beginning of the function to disable it.

function wpautop($pee, $br = 1) {
        //plyons disabling this. 20090516
        return $pee;

Of course, this change will be undone when you upgrade to a newer WordPress release, so it's just a convenient hack. Once you have your posts with proper paragraph tags and no extra line feeds, wpautop should not change your markup and therefore you shouldn't have a problem when it is re-enabled after a WordPress upgrade.

Maritz: 1 - Very Dissatisfied

2009-05-17T04:27:56.000Z

I bought a new car this fall and a few months later I got a follow-up survey in the mail from Maritz Research. Having a few pieces of feedback to give, such as the orange readout on the Bose sound system being invisible through Sunglasses, I endeavored to fill it out. Holy SAT Test, Batman! The survey is nine jam-packed, small-font pages long. There are 76 officially numbered questions, but many questions involve dozens of individual line-items. See the example below where question 58 asks you to rate 67 individual aspects of the vehicle! Sixty frigging seven! I gave up in frustration long before getting there.

This represents a complete failure to do your job as a market research company. This is their business. Did they exert any effort to make the customer do less work? No. Does the survey include dozens and dozens of line items that completely do not apply to my vehicle because it's not a pick-up truck, and so forth? Yes. Did they select only the really meaningful things for me to rate? No. For example, I am asked to supply my satisfaction level from 5 "Completely Satisfied" to 1 "Very Dissatisfied" on the topic "absence of engine stalling". Give me a break. You need to survey you customers to find out A) your cars stall and B) customers find that unsatisfactory. Please. Do they have any section for free-form comments, unprompted feedback, or even brief descriptions? No. Do they have a special "green traffic light" insert stapled in reading "Your Opinion Counts! Pleas Proceed..."? Yes. Apparently my opinion doesn't count enough for them to create a survey with less complexity than an income tax form. I'm hunting around for the section to fill in my non-farm income.

Please rate your satisfaction with this Maritz Research survey:

5 - Completely Satisfied
4 - Very Satisfied
3 - Satisfied
2 - Somewhat Dissatisfied
1 - Very Dissatisfied

How to zip a directory in python

2009-04-27T10:04:03.000Z

I came across this problem at work and also over on this www.stackoverflow.com thread. You have a directory and you want to recursively zip it up. Simple, right? The equivalent of the unix command "zip myDir.zip myDir". Should be like 5 lines of code? Python even has a built in zipfile module, sweet! Well, as is often the case (see urllib and friends), python's "batteries included" slogan is more like "enough batteries for 36 seconds included". Anyway, it's more like 23 lines of functional code, which is still pretty good, but I would have expected the zipfile module to have this included and not have to use os.walk() to do this.

#!/usr/bin/python
# -*- coding: utf-8 -*-
"""This is a sample function for zipping an entire directory into a zipfile"""

#This seems to work OK creating zip files on both windows and linux. The output
#files seem to extract properly on windows (built-in Compressed Folders feature,
#WinZip, and 7-Zip) and linux. However, empty directories in a zip file appear
#to be a thorny issue. The solution below seems to work but the output of
#"zipinfo" on linux is concerning. Also the directory permissions are not set
#correctly for empty directories in the zip archive. This appears to require
#some more in depth research.

#I got some info from:
#http://www.velocityreviews.com/forums/t318840-add-empty-directory-using-zipfile.html
#http://mail.python.org/pipermail/python-list/2006-January/535240.html
import os
import zipfile

def zipdir(dirPath=None, zipFilePath=None, includeDirInZip=True):
    """Create a zip archive from a directory.
    
    Note that this function is designed to put files in the zip archive with
    either no parent directory or just one parent directory, so it will trim any
    leading directories in the filesystem paths and not include them inside the
    zip archive paths. This is generally the case when you want to just take a
    directory and make it into a zip file that can be extracted in different
    locations. 
    
    Keyword arguments:
    
    dirPath -- string path to the directory to archive. This is the only
    required argument. It can be absolute or relative, but only one or zero
    leading directories will be included in the zip archive.

    zipFilePath -- string path to the output zip file. This can be an absolute
    or relative path. If the zip file already exists, it will be updated. If
    not, it will be created. If you want to replace it from scratch, delete it
    prior to calling this function. (default is computed as dirPath + ".zip")

    includeDirInZip -- boolean indicating whether the top level directory should
    be included in the archive or omitted. (default True)

"""
    if not zipFilePath:
        zipFilePath = dirPath + ".zip"
    if not os.path.isdir(dirPath):
        raise OSError("dirPath argument must point to a directory. "
            "'%s' does not." % dirPath)
    parentDir, dirToZip = os.path.split(dirPath)
    #Little nested function to prepare the proper archive path
    def trimPath(path):
        archivePath = path.replace(parentDir, "", 1)
        if parentDir:
            archivePath = archivePath.replace(os.path.sep, "", 1)
        if not includeDirInZip:
            archivePath = archivePath.replace(dirToZip + os.path.sep, "", 1)
        return os.path.normcase(archivePath)
        
    outFile = zipfile.ZipFile(zipFilePath, "w",
        compression=zipfile.ZIP_DEFLATED)
    for (archiveDirPath, dirNames, fileNames) in os.walk(dirPath):
        for fileName in fileNames:
            filePath = os.path.join(archiveDirPath, fileName)
            outFile.write(filePath, trimPath(filePath))
        #Make sure we get empty directories as well
        if not fileNames and not dirNames:
            zipInfo = zipfile.ZipInfo(trimPath(archiveDirPath) + "/")
            #some web sites suggest doing
            #zipInfo.external_attr = 16
            #or
            #zipInfo.external_attr = 48
            #Here to allow for inserting an empty directory.  Still TBD/TODO.
            outFile.writestr(zipInfo, "")
    outFile.close()

Here's some samples of how you use this:

zipdir("foo") #Just give it a dir and get a .zip file
zipdir("foo", "foo2.zip") #Get a .zip file with a specific file name
zipdir("foo", "foo3nodir.zip", False) #Omit the top level directory
zipdir("../test1/foo", "foo4nopardirs.zip", False) #exclude some leading dirs
zipdir("../test1/foo", "foo5pardir.zip") #Include some leading dirs

Thoughts on Scrum

2009-04-17T08:15:07.000Z

So I had a chance to work in the Scrum methodology for about six months recently. I thought I'd write up some of my thoughts on the experience and the process. Please note that first, this is my first experience working in the Scrum process. I am sure as I use it more these opinions will change. Second, these are strictly my personal thoughts and opinions and in no way represent the experience of my team or anyone else. Third, my project had a specific set of tasks and particulars. On a different project, with more sprints or shorter tasks or other relevant variations, I might have drawn (and may well draw in the future) different conclusions.

Overall, agile and Scrum are a vast improvement over the waterfall model traditionally used at large companies. Some of the specifics benefits:

Far less time wasted doing detailed design and estimates for huge amounts of work that will take a long time to build, or very often might not get built at all
Overall a very steady and metered workload. We did hit a few days of mad dash toward the end of one or two sprints preparing for the demo, but much more stable than the wild variances and unpredictability of waterfall
The focus on build and test automation really does enable better agility in the code and prevent regressions
Having something demoable every month is just all around good for all parties involved. I think perhaps this is the single most important piece of the methodology. Ignore this and I bet a substantial amount of the benefit of Scrum would be lost.
Similar to demoable, the notion of "potentially shippable product" really rings true for me and forces you to deal with the issues that often lurk in the shadows in the waterfall model only to jump out at the last minute: installation, upgrade, documentation, fit and finish, etc.

So the remainder of this will mostly comment on things I found to be problematic or confusing, but I want it to be clear that I am a huge proponent of agile and Scrum and this is not meant to be an argument that the status quo is better. Quite the contrary.

The Backlog and User Stories

I want to opine a bit about this notion of "User Stories". Conceptually, this feels like a perfectly healthy way to constantly remind the team to be focused on the important things that will be valuable to the end users. I think this works best for user interfaces and web development where the ratio of engineering effort to visible, tangible change in the end user interface or experience is high. However, I think in many other areas of software development, such as embedded systems, and in our case complex enterprise software, one user story can generate a boatload of tasks and there end up being a lot of backlog items that are just engineering internal stepping stones to functionality. In our team, we were able to bridge this gap and refer to the work items in the backlog just as general "backlog items", but I still felt a bit guilty adding more raw engineering items to the backlog even though it would be a bit of a stretch to tie them closely to a user story. I think we managed to get beyond this eventually and I don't think we strayed off into the woods of unimportant engineering diversions. However, in many if not most backlog items, we were not able to apply the "As a $USER_ROLE, I want to $ACTION so I can $BENEFIT" user story template.

I was just browsing around www.openagile.com and noticed they call the backlog the "Work Queue". I like this name better because it more easily allows the notion of "anything that requires work" can go in there and I dislike the word "backlog" because it connotes a buildup of work debt that for me has negative motivational effects.

Secondly, I found some problems trying to implement this idea that a user story should fit on a "story card" the size of an index card and be fleshed out through face to face conversation. The software I develop at work tends to have a high complexity level and attention to detail. It's more complex than web development. Our work items often can't be clearly expressed with something straightforward like "As a shopper, I want to see the total cost of the items in my shopping cart in the page header". The problem with discussion is A) our team was in three cities with no more than two people in any single location B) we wanted something in writing that could be referred to over and over again throughout and after the sprint and C) no one could remember the details otherwise. Due to C), if you asked the product owner (1/2 me) to clarify a user story through conversation on sprint day 2, 12, and 16, you were liable to get three variations. Our stories ended up being usually at least a few paragraphs worth of detail plus a smattering of acceptance tests, and I think that is OK and worked better for us.

Also, it turns out that populating and maintaining the product backlog is actually a large amount of work. Keeping the backlog items detailed enough, in the right order, with good acceptance tests takes an awful lot of time. Normally the product owner is not on the scrum team. However, at my company we're not adopting that aspect at this time (for several reasons), and in this project myself and one other scrum team member acted as the product owner role. Basically after working hard to get the sprint review demo up and working, my co-product-owner and I would have about 2 hours Friday afternoon while we were completely fried and 1 hour Monday morning to try to whip the backlog into shape for the next sprint planning meeting Monday morning. In retrospect, we should have allocated about two full days per sprint just for care and feeding of the product backlog. For the early sprints, this number might be larger - like a week, and then as the effort congeals, the amount of time the product owner needs to allocate to backlog care and feeding will probably grow shorter, unless a major change of requirements comes down the pike.

One final point about the backlog or work queue. After several sprints, we ended up having a fairly enormous work queue with hundreds of items. This became really unwieldy to deal with as a flat list ordered by priority. We ended up wanting to leave the items in the sprint ordered by priority, and about a sprint or two's worth of work items ordered by priority in the backlog, but for all the stuff further down the road, it was much easier to group those into folders based on functionality. I guess each product owner's mileage may vary here, but my point is do whatever it is you need to do as a product owner to be able to comprehend and manipulate your work queue with facility.

User Stories and Product Backlog summary and suggestions:

Use the terms "Work Queue" and "Work Item" instead
Don't feel obligated to use the "As an X, I want to Y, so I can Z" template if it is unnatural (But do stick with it when it is appropriate)
Put how ever much detail is needed into your Work Items
Allocate sufficient Product Owner time to keep the work queue healthy
Do what's needed to organize large work queues for easy manipulation by the product owner

Sprint Planning, Estimation, and Time Tracking

In terms of planning and estimating, even with the shorter four week sprints, there are still some difficulties built into the Scrum process in my experience. We used story points, a somewhat abstract relative unit of measurement designed to express relative difficulties between two stories, not necessarily any absolute measure of time, effort, or difficulty. For me personally, this just does not feel natural or come easily. First, I don't think about estimates in relative comparison normally. I don't think task A is easy, and task B is four times as hard as task A. As a general rule, I think most people don't think accurately in terms of multiplicative values. Conceptualizing the idea that 13 story points is 6.5 times more effort than 2 story points doesn't come naturally for me. Given any two stories, I can tell you which one is harder, and I could probably use that basic operation to sort a list of tasks by difficulty, but computing the relative sizes of two along the Fibonacci scale is awkward for me. Secondly, the seemingly arbitrary and perhaps a bit weird use of the Fibonacci sequence just sticks out a bit to me as obtuse. Here's my suggested improvements. When it comes to story points, I really believe you need three and only three values: small, medium, large. Fibonacci gives us 1 2 3 5 8 13 21, but 1s and 2s were not particularly common in our backlog, and when we saw a 13 we generally panicked and broke it into smaller pieces. That's what I suggest. If you have three or four stories that are truly super easy (1 or 2 in Fibonacci story points), just stick them all into one story and call it small. If you have something that seems very hard, like it's going to take one developer half the sprint to do it, break it up into a large and a few mediums. I think the "small, medium, large" terminology has another benefit of being obvious even when discussing with someone non-technical or not familiar with Scrum or Story Points. No explanation is required. Story Points using Fibonacci are subtle enough to require explanation, and perhaps they don't add enough value to justify their subtlety.

OK, so now onto the sprint planning meeting. The default scrum schedule has a single marathon sprint planning meeting at the beginning of each sprint. For our team of five developers with four week sprints, we are looking to estimate somewhere around 600 hours worth of tasks ranging in granularity from 4-30 hours. It's just too much to focus on in a single session (again, my opinion). After two and a half hours of this, I'm bleary-eyed and fried and unable to motivate myself to do the kind of careful thinking required to make the task lists complete and accurate. I start falling into wanting to list the same three tasks for every story: 1. Figure it out 2. Code it 3. Test it. My suggestion is that this be done for one hour a week as many as four times during the sprint if needed (probably 3 of these will get the job done) just to break it up into manageable chunks.

Now, another point on the sprint planning meetings. Our goal was to take user stories that had story points and acceptance criteria and define a full set of tasks and estimates for five developer-months worth of time in a single session, and then try not to have to add/remove/change during the sprint. (Note, the try not to have to add/remove/change is my own understanding, although my editor pointed out that Scrum itself has no such restriction and allows for task adjustments - focusing only on remaining work. However, it seems the goal is to define the tasks to whatever degree possible at the beginning or we wouldn't bother doing it in the first place, but then adjustments are accomodated). In any case, I wonder whether this is a realistic, achievable goal. Or let me say that in my experience at least about 10% churn in the tasks as development progresses (regardless of what overall methodology I am working within) seems to be my limit and I can't get it more accurate than that up front. So if the team is OK with somewhere around 10% task churn mid-sprint, then all is well.

When it comes to estimating in hours an individual task in the backlog, our team was developing a new product from scratch with a team of all highly experienced engineers. So we actually ended up having pretty good interchangeability between developers where several different people could complete a task and generally take about the same amount of time to do it. However, when working on an big existing code base, it can be an order of magnitude difference in the estimate between when someone who has already learned a complex subsystem and implemented a similar change does a task verses someone else doing it for the first time. I think if we want real accuracy we need some way to model this more accurately. I'm not sure how best to do this. Maybe one task for learning curve that gets skipped if it's not needed because the person who completes the task didn't need it? I don't have a good suggestion to improve this yet. Also, along similar lines, in general I always get nervous when one person (or a team) creates estimates for work that will be completed by another person. I think there's just a lot of risk there. Again, I don't have an alternative to propose at this time.

With regard to ordering the work queue items in the queue, and thus determining which ones get into the sprint, I think in general the scrum notion that the product owner does this and it's based on business value to the product owner is reasonable and beneficial for the most part. However, we also tried to adhere to this order for the order in which items were implemented during the sprint, and I think in many cases this didn't work out as well as it could have. Even in the backlog, I feel that a certain amount of the time, a logical or technical dependency will suggest a different backlog item order than strict product owner importance. But for now I'm OK with the product owner importance ordering for the backlog. However, inside the sprint, I think it might work out better to allow the scrum team a certain amount of discretion and control of the order of implementation on any technical or logistical grounds they feel will help. Within a sprint, we bumped up against a fair number of interdependencies where two people would need to heavily edit the same source code file on the same day, or one person was blocking waiting for another person to complete some code needed to build their feature. I can understand that taking the most important item and implementing it last is probably not acceptable, but I think some well-reasoned adjustment should be allowed. Ultimately, I think more stories will get completed due to the efficiency gains. I think a good guideline would be just to discuss these proposed changes to implementation order with the product owner during the daily Scrum call, and based on the status of the burn down chart and how things seem to be going, the product owner may permit or veto a re-ordering.

Sprint Planning, Estimation, and Time Tracking summary and suggestions:

Use small, medium, large instead of Fibonacci story points
Divide the sprint planning across several meetings throughout the sprint, never exceeding the motivation/attention span threshold for this somewhat tedious endeavor
Define criteria under which it is OK for developers to add/remove/change tasks during the sprint
May need some way to estimate tasks differently depending on who implements them
Within a single sprint, team should be allowed to make implementation order changes that will help with efficiency with consent of the product owner

Code Conventions

2009-03-15T03:06:23.000Z

Here's a link to my article on code conventions. Feel free to post comments here.

How to run two wordpress blogs on one web site

2009-03-15T02:57:53.000Z

There is ample detailed information out there on installing wordpress. However, I wanted to just provide a small supplement about setting up two distinct wordpress blogs within a single apache2 web site. The system I am using is Ubuntu Linux 8.10, but other than the package installation, the configuration steps should be the same on other linux distributions.

As a first step, read through the Wordpress Installation Instructions. You will find them to be thorough and clear. The starting point for my setup is that I already had a web site up and running under apache2. I just wanted to add the Wordpress (and underlying MySQL database) setup and have two separate blogs with separate themes and content.

So here's my starting setup:

Ubuntu 8.10 on an amd64 system
apache2 already installed and working
static content for the web site deployed in /var/www/example.com
MySQL and Wordpress are not yet installed

Install wordpress and mysql

First let's install wordpress and mysql. I'll do this on the command line using the apt-get program, but you can use one of the graphical options as well

sudo apt-get install wordpress virtual-mysql-server

You should see a bunch of packages that will get installed and press y to proceed. The MySQL install will prompt you to create a new mysql root account password, so go ahead and do that.

Set up the wordpress databases

OK, so let's say we are going to call our blogs blog1 and blog2. We need to create mysql databases for them. Note that you may see tutorials telling you you can store the data for two separate blogs in one database. While true, two databases is a much cleaner way to go. End users shouldn't be going around making up database table names. So, we are going to use the mysql command line tools to do this. Again, the wordpress docs here are fine and describe graphical alternatives as well. Let's connect to mysql (use the password you created above) as root and create the databases. We'll call them blog1 and blog2 and we'll also set up user accounts inside mysql that wordpress will use to access the databases. Again for simplicity, we'll also call the user accounts blog1 and blog2. Replace "MakeUpPassword" with your own chosen password.

mysql -u root -p
create database blog1;
GRANT ALL PRIVILEGES ON blog1.* TO "blog1"@"localhost" IDENTIFIED BY "MakeUpPassword";
flush privileges;
create database blog2;
GRANT ALL PRIVILEGES ON blog2.* TO "blog2"@"localhost" IDENTIFIED BY "MakeUpPassword";
flush privileges;
quit;

Install wordpress files and configure DB access

OK, when we installed wordpress above, Ubuntu put a copy of the wordpress PHP files into /usr/share/wordpress, so now we're going to make 2 copies, one for each blog, underneath our web site's document root. Note that since these are two blogs in the same web site, we don't want either blog to be the top level home page of the site, so each gets its own separate subdirectory.

sudo mkdir -p /var/www/example.com/blog1 /var/www/example.com/blog2
sudo cp -r /usr/share/wordpress/* /var/www/example.com/blog1
sudo cp -r /usr/share/wordpress/* /var/www/example.com/blog2
sudo chown -R www-data:www-data /var/www/example.com/blog*

Now I should note that Debian/Ubuntu has a customized wordpress configuration. Often, these are well crafted by the experts and will save you a lot of time and hassle if you follow the patterns they suggest. In this case, I don't think their setup exactly matches my goal of two different blogs under one web site, so I am bypassing their pattern and doing my own simple alternative. So what happens is that by default the file /usr/share/wordpress/wp-config.php is a symbolic link to /etc/wordpress/wp-config.php, and when we copied these files, that symlink was copied too, which means if we aren't careful both of our blogs will point at the same database, making them one blog instead of two! So we do the following:

sudo rm /var/www/example.com/blog*/wp-config.php
sudo cp /usr/share/wordpress/wp-config-sample.php /var/www/example.com/blog1/wp-config.php
sudo cp /usr/share/wordpress/wp-config-sample.php /var/www/example.com/blog2/wp-config.php
sudo chown -R www-data:www-data /var/www/example.com/blog*

OK, now each blog has it's own separate copy of wp-config.php. Go ahead and edit those files to point blog1 at the blog1 database and blog2 at the blog2 database using vi or your text editor of choice. When done, they should look as follows:

/var/www/example.com/blog1/wp-config.php

/var/www/example.com/blog2/wp-config.php

Permalinks

OK, now let's make sure we have mod_rewrite enabled for wordpress permalinks, then we restart apache2 to get our new software and config to take effect. We'll also make sure we have a writeable .htaccess file so wordpress can set it up for us.

sudo a2enmod rewrite
sudo apache2ctl restart
sudo touch /var/www/example.com/blog1/.htaccess
sudo touch /var/www/example.com/blog2/.htaccess
sudo chown www-data:www-data /var/www/example.com/blog*/.htaccess
sudo chmod 644 /var/www/example.com/blog*/.htaccess

Almost done, we can now point a web browser to http://localhost/blog1/wp-admin/install.php and fill out that form and launch the wordpress self-install. Repeat the process for blog2 at http://localhost/blog2/wp-admin/install.php. Now, this is going to set up the blog in the database and create the admin user with a default password. If your computer has a mail transport agent running, you should get the email with the default password. If not, you won't, which is what happened to me, so we can just set the admin password to one of our choosing. First, we need to know the MD5 checksum of our chosen password. You can use this online form to get the MD5 of your password, but sending your password to some random web site in clear text is too insecure for me. Therefore, if you have python, you can use this little python one-liner to do it. This will securely prompt you for your password and print out the corresponding MD5. Nothing is sent over the network. You will end up with a 32-character hex string. Use this instead of the For details from wordpress, read resetting your password in the wordpress online docs. The cheat sheet is below.

python -c "import getpass,md5;m=md5.new();m.update(getpass.getpass());print m.hexdigest()"

OK, so let's set this password into mysql so we can start managing our blog

mysql -u root -p
use blog1;
update wp_users set user_pass="useyour32charhexstringmd5here" where user_nicename="admin";
use blog2;
update wp_users set user_pass="useyour32charhexstringmd5here" where user_nicename="admin";
quit;

OK, you are good to go to http://localhost/blog1, log in as admin, and start managing your blog. You can go in and set up custom pretty permalinks in the admin section and when you save, wordpress should be able to write the rewrite settings to the .htaccess file we set up for you. Good luck and happy blogging!

Music subscription and Rhapsody

2009-03-15T02:54:03.000Z

Flat fee music subscription service has changed my life. I wish there was some less dramatic way to put it, but it's the honest truth. I don't remember exactly when, but somewhere in late 2006 probably, after reading this Joel on Software - The infinite music collection article, I was intrigued by the idea and did a little shopping. I know I looked at Napster, Virgin, Yahoo, and Rhapsody at least. I think I initially went with Rhapsody because of the Sonos integration. I experimented for a bit listening on the computer and then I think that Christmas I asked for a basic set of Sonos gear, which was and still is exorbitantly priced.

OK, so first some comments on the whole notion of subscription music service. It's amazing. For someone like me with a voracious appetite for new music cultivated spending long hours every week in the fantastic Oberlin Conservatory Music Library, it was complete drooling brain-fry. It was a bit overwhelming at first. There were numerous aspects of this arrangement that were just awesome. Obviously, the size of the library is number one. Having extremely ecclectic and somewhat obscure taste in music, I was worried that it would be basically a top-40 archive that I would quickly grow bored with. This was not the case. The collection of jazz, classical, and less main stream stuff was really quite good. Now, every CD in my personal library is not available, but the service library as a whole is so infinitely broad and compelling that I don't care that much.

Secondly, there is the complete removal of the financial penalty for exploring. Before I expound on this let me just state that I listen to whole albums in totality in order. Fuck shuffle. Fuck 30 second previews. I usually listen to complete albums start to finish numerous times before I decide how I feel about them. In addition, when checking out a new artist, I actually prefer to listen to all the albums in chronological order. Sometimes I'll scan a "best of" to see if it's worth my time, but usually I go straight for the debut album. For years, I would gather suggestions from peers, and then go plunk down $17 for a CD. No more. Now I can try relentlessly at full speed for a flat fee. This is fantastic and significant. Now I can actually listen to artists I know I don't care for, if only to better understand what it is about them that I don't like.

Third, in the jazz and classical genres, musicians are vastly more prolific compared to their pop/rock peers. Go try to get the complete recordings of Miles Davis. Be sure to bring like $1500. With a flat fee subscription, I can go and listen to LOTS of music by folks I like. One of the first things that got me completely hooked on this was listening to a 10-disc set of Steve Reich: Works 1965-1995. This retails at amazon.com for $99.98. So the trade-off for this ONE collection is own this collection forever or get over six months access to millions of songs online.

Fourth, you can compare recordings to your heart's content. I have a physical copy of the venerable Emerson String Quartet's recording of the six Bela Bartok String Quartets. Currently going for $22 on amazon. It is a cherished record in my collection. However, the chances of me buying another recording of this work are slim to none, even though I would love to hear various interpretations. Rhapsody has no less than ten complete recordings of these works available! When it comes to jazz and you want to learn a new song, Rhapsody is going to give you about thirty recordings to choose from. No better way to truly absorb the song's full meaning from the repertoire by listening to numerous different recordings.

Fifth, there is no personal music library management. I've spent hours and hours labeling my CD collection (over 800 discs), ripping them, fixing the cddb track metadata, transferring stuff to portable devices, re-transfering it when it gets corrupted, etc. Then there's the idea that I have to keep this all backed up. I have so far ripped about 30 GB of music and don't really want to deal with backing that up. With a subscription, all the music is just instantly there. It's all indexed and searchable. The track metadata is correct. I don't have to personally back it all up.

Sixth, it works with portable players in a reasonable way. I think for the next Christmas I asked for the SanDisk Sansa e280R portable MP3 player that is integrated with Rhapsody. You can download tracks to the device and as long as you connect it once a month to verify your subscription is active, you are good to go. It works pretty well and doesn't really add any hassle. I'll be posting another entry soon about the various devices (basically all bad) I have used with Rhapsody and how they work.

So I'm totally hooked on this model of flat fee all you can eat subscription music. I was initially concerned that streaming this in real time would not work, but honestly there have been almost no glitches. Either the service is working or it isn't, but it doesn't do any buffering interrupting the song or anything like that, which would be a complete deal-breaker. To try to listen to this much music on iTunes would cost me hundreds of dollars a month.

So, let's look at some of the cons. There are no linear notes. I do miss that. I also miss detailed personnel listings per track. However, when I really am interested in that, the info is usually available online if I can remember to go look it up. As of now, in order for this to work, it relies on Plays For Sure Digital Right Management (DRM) scheme. Generally I am against DRM, but if it is required in order for the music industry to be willing to make a subscription service available, so be it.

OK, so that is mostly focussing on the subscription model in the abstract. Let's talk about the Rhapsody service in particular. After a few years of using it, overall I'd probably give it about 7 out of 10. It's good, but it has some annoyances. On the plus side, the web based Rhapsody online product, which was formerly a proprietary plug-in that gave me a few hassles on Linux, is now entirely standard flash based app. In general it works great on linux however my new laptop I installed 64 bit Ubuntu and the rhapsody flash app won't log in currently. I'll be contacting support (sigh) to yell at them to make it work.

So some of the annoyances include inability to re-order tracks in your queue/playlist. You can append to the end of it and delete items, but you can't reorder. Again, since I generally listen to whole albums it doesn't bother me much, but sometimes it can be a real pain. Also, their web site used to have a frigging NORMAL LOGIN HTML FORM, thus I could save my credentials in my browser and not be bothered with it. Now the flash app itself prompts for credentials which means I have to re-type my password a few times a day. I asked support to make it go back to the old way but you can guess how that went.

Reliability and bill payment have been good. The Windows application that I need to use to actually transfer music to a portable player is decent. It's better than the flash app, but still a bit cumbersome. It's got a web browser pane embedded in it, which has all the annoyances of a web browser, but not all the screen components and tools to properly manage it. Here they have a proper playlist editor window. Ideally I would be able to transfer to my portable from a linux application, but I have realistic expectations here. It ain't going to happen.

Rhapsody has user and celebrity playlists, both of which are a great way to explore and check out new stuff.

So if you are a big music fan, and still haven't tried a subscription service, give it a shot. I can never go back. Look for more posts soon about my experierces with music player devices over the years.

Announcing Pete's Points

2009-03-15T02:53:24.000Z

I have decided to start a "professional" blog. I use the word "professional" only in the sense of "not personal". I will post about software engineering, computers, consumer electronics, and music technology primarily. You may want to read this description of what this blog is all about. I hope you enjoy it and look forward to any conversations this generates.