As part of evaluating several libraries for specification testing in Ruby (MSpec, RSpec, Bacon), I wanted to benchmark the performance of the library against a simple suite of tests to see if one was particularly slower than any of the others. It wasn’t intended to be very scientific but to at least expose a slow framework, if any.

Each benchmark was performed by creating a suite of specifications based around Bacon’s whirlwind sample (consisting of 5 specs), and executed the suite 10,000 times. This benchmark test was run 5 times in order to weed out any statistical anomolies. Nb: For this analysis, I didn’t benchmark RSpec because it’s not terribly compatible with IronRuby just yet.

The results and code can be found below. In a nutshell, it does seem as though MSpec performs slower than Bacon, but when you consider that over a 50,000 test sample it was only (roughly) 10 seconds slower, the difference is negligible.

	Bacon	MSpec
Run 1	27.642	37.337
Run 2	25.598	37.755
Run 3	25.607	37.424
Run 4	25.317	36.439
Run 5	25.105	36.352

# This is the code for the spec test wrapper. 
# To execute, save the file as "spec_runner.rb" and execute
#    ruby spec_runner.rb
#
#

ITERATIONS = 10000

require 'rubygems'

@old_stdout = $stdout
$stdout = StringIO.new

def milestone(n)
$stdout = @old_stdout
puts ("Reached milestone: ##{n}")
$stdout = StringIO.new
	
end

def time_it(&func)
	start_time = Time.now
	1.upto(ITERATIONS) do |n|
		func.call
		milestone(n) if n % 1000 == 0
	end
	end_time = Time.now
	end_time-start_time
end

bacon_time = time_it do 
	load 'whirlwind_bacon.rb' 
end

mspec_time = time_it do 
	load 'whirlwind_mspec.rb' 
end

$stdout = @old_stdout
puts "bacon time: #{bacon_time}"
puts "mspec time: #{mspec_time}"

# This is the Bacon test file. Save it as "whirlwind_bacon.rb"
#
#

require 'bacon'

describe 'A new array' do
	before do
		@ary = Array.new
	end

	it 'should be empty' do
		@ary.should.be.empty
		@ary < < 1
		@ary.should.include 1
	end

	it 'should have zero size' do
		@ary.size.should.equal 0
		@ary.size.should.be.close 0.1, 0.5
	end

	it 'should raise on trying fetch any index' do
		lambda { @ary.fetch 0 }.
			should.raise(IndexError).
			message.should.match(/out of array/)
	end

	it 'should have an object identity' do
		@ary.should.not.be.same_as Array.new
	end

	palindrome = lambda { |obj| obj == obj.reverse }
	it 'should be a palindrome' do
		@ary.should.be.a palindrome
	end
end

# This is the MSpec test file. Save it as "whirlwind_mspec.rb"
#
#
require 'mspec'

describe 'A new array' do
	before do
		@ary = Array.new
	end

	it 'should be empty' do
		@ary.should be_empty
		@ary < < 1
		@ary.should include(1)
	end

	it 'should have zero size' do
		@ary.size.should.equal 0
		@ary.size.should be_close(0.1, 0.5)
	end

	it 'should raise on trying fetch any index' do
		d = lambda { @ary.fetch 0 }
		d.should raise_error(IndexError, /out of array/)
	end

	it 'should have an object identity' do
		@ary.should !equal(Array.new)
	end

	palindrome = lambda { |obj| obj == obj.reverse }
	it 'should be a palindrome' do
		(palindrome.call @ary).should be_true
	end
end