As part of evaluating several libraries for specification testing in Ruby (MSpec, RSpec, Bacon), I wanted to benchmark the performance of the library against a simple suite of tests to see if one was particularly slower than any of the others. It wasn’t intended to be very scientific but to at least expose a slow framework, if any.
Each benchmark was performed by creating a suite of specifications based around Bacon’s whirlwind sample (consisting of 5 specs), and executed the suite 10,000 times. This benchmark test was run 5 times in order to weed out any statistical anomolies. Nb: For this analysis, I didn’t benchmark RSpec because it’s not terribly compatible with IronRuby just yet.
The results and code can be found below. In a nutshell, it does seem as though MSpec performs slower than Bacon, but when you consider that over a 50,000 test sample it was only (roughly) 10 seconds slower, the difference is negligible.
| |
Bacon |
MSpec |
| Run 1 |
27.642 |
37.337 |
| Run 2 |
25.598 |
37.755 |
| Run 3 |
25.607 |
37.424 |
| Run 4 |
25.317 |
36.439 |
| Run 5 |
25.105 |
36.352 |
# This is the code for the spec test wrapper.
# To execute, save the file as "spec_runner.rb" and execute
# ruby spec_runner.rb
#
#
ITERATIONS = 10000
require 'rubygems'
@old_stdout = $stdout
$stdout = StringIO.new
def milestone(n)
$stdout = @old_stdout
puts ("Reached milestone: ##{n}")
$stdout = StringIO.new
end
def time_it(&func)
start_time = Time.now
1.upto(ITERATIONS) do |n|
func.call
milestone(n) if n % 1000 == 0
end
end_time = Time.now
end_time-start_time
end
bacon_time = time_it do
load 'whirlwind_bacon.rb'
end
mspec_time = time_it do
load 'whirlwind_mspec.rb'
end
$stdout = @old_stdout
puts "bacon time: #{bacon_time}"
puts "mspec time: #{mspec_time}"
# This is the Bacon test file. Save it as "whirlwind_bacon.rb"
#
#
require 'bacon'
describe 'A new array' do
before do
@ary = Array.new
end
it 'should be empty' do
@ary.should.be.empty
@ary < < 1
@ary.should.include 1
end
it 'should have zero size' do
@ary.size.should.equal 0
@ary.size.should.be.close 0.1, 0.5
end
it 'should raise on trying fetch any index' do
lambda { @ary.fetch 0 }.
should.raise(IndexError).
message.should.match(/out of array/)
end
it 'should have an object identity' do
@ary.should.not.be.same_as Array.new
end
palindrome = lambda { |obj| obj == obj.reverse }
it 'should be a palindrome' do
@ary.should.be.a palindrome
end
end
# This is the MSpec test file. Save it as "whirlwind_mspec.rb"
#
#
require 'mspec'
describe 'A new array' do
before do
@ary = Array.new
end
it 'should be empty' do
@ary.should be_empty
@ary < < 1
@ary.should include(1)
end
it 'should have zero size' do
@ary.size.should.equal 0
@ary.size.should be_close(0.1, 0.5)
end
it 'should raise on trying fetch any index' do
d = lambda { @ary.fetch 0 }
d.should raise_error(IndexError, /out of array/)
end
it 'should have an object identity' do
@ary.should !equal(Array.new)
end
palindrome = lambda { |obj| obj == obj.reverse }
it 'should be a palindrome' do
(palindrome.call @ary).should be_true
end
end
