DNS seems to have propagated, so now I can tell you about ...
ironruby.info is a new site I’ve been working on for a week or so; a place to track daily information about how IronRuby is doing. It shows simple performance and RubySpec information, compared to Ruby, and random build/source information. There’s a lot more to do, like tracking working set, showing graphs or historical data, and running traces popular Ruby frameworks to show which specs are “most important” to fix … but it provides some useful information already.
Oh, and the code is on github. stats.rb runs nightly collecting the data, and app.rb is the Sinatra website showing the data. Feel free to send me patches/changes, or fork it to track other implementation :)
http://github.com/jschementi/ironruby-stats
Disclaimer: This information is gathered on a Intel Pentium 4 3.5 ghz machine, so performance numbers will be different if you have a faster processor and/or more processors/cores.
Performance
For startup time, it measures the time it takes to start ir.exe empty.rb from the command-line, where empty is a empty file. IronRuby is currently 10x slower for startup on average, regardless of whether you’re using the interpreter or not (though the interpreter is slightly faster … we’re working on fixing that).
The “100,000 Iterations” is just $a *= 2 100,000 times. Nothing fancy yet. Ideally we’d be running the ruby-benchmark-suite here. IronRuby is only 2.5x slower on this, but the interpreter is way slower. So, for now only use the interpreter for straight-line code ... no loops!
RubySpec
For those who don’t know what RubySpec is:
RubySpec is a project to write a complete, executable specification for the Ruby programming language.
IronRuby has been using this specification for a while now to gauge our progress towards being a Ruby-1.8.6 compliant implementation. They are divided into three sections: Language which tests Ruby itself, Core which tests the libraries which Ruby needs to run, like Array, Hash, etc, and Library which tests the Ruby standard library.
While these are a great judge of a Ruby implementation’s progress, they take a bit of time to run yourself (including setup time), and we never gave detailed reports of our own runs. So now, if you’re ever curious to know how IronRuby is doing, you can just check ironruby.info and get the data immediately. Pretty cool. “But wow, that’s a ton of red on the page.” --- yeah, it looks worse than it is, maybe I should use a range of red? Regardless, let’s explain the results.
Pass rate
The pass rates in the ir.exe and ruby.exe columns are “the number of expectations which passed, divided by the total number of expectations which ruby.exe runs”. So the “all specs pass rate” number is the pure pass rate, regardless of how well/poor ruby.exe does. The “diff” column shows how far away IronRuby is from Ruby for each row.
Overall, IronRuby passes 86% of all the specs. Ruby (on Windows) passes 98%, so that leaves us with 12% of the specs left to get to Ruby compatibility on Windows. Not too shabby. The Core tests have the highest failure rate (-18%), followed by the Language tests (-5%), and the Library tests have a strangely-low failure rate (1.87%) since Ruby is running 482 expectations less than IronRuby … I need to investigate that.
Execution time
The Core and Library specs are between 4-5x slower (Ruby’s Core specs report a WAY too long run time right now, usually they’re around 2 minutes). The Language specs are a whopping 50x slower, which is crazy. I have to investigate why.
Fun facts
There’s other fun facts it collects; build time, binary size, repository size. Not that interesting, but still good to see … I’d like to know if our repository size shoots up dramatically.
Anyway, that’s what ironruby.info gives you today. Hopefully this gives you an idea of where we are (getting close on the specs, and putting a lot of work into performance).