Looking for a free statisitical software?

I have been following a Linkedin discussion for a couple of weeks on what open source analytical software to use for a company with little money. I put up a recommendation for R, but it was really was wild to see the range of suggestions. One was just to use SAS because it really was not that expensive when you considered what you got. Others recommended R, Rattle, Knime, and a whole bunch that I have never even heard of.  Overall a very entertaining discussion, but I an not sure it provided any real benefit for the person who posed the question. I will be the first to admit I have an R basis so my knee jerk reaction to the question was to reply R without fully understanding this guys needs. Therefore, another solution might be superior to R in his particular case.

One person responded with a link to The Impoverished Social Scientist's Guide to Free Statisticial Software and Resources by Professor Micab Altman of Harvard. First what a great title! Second what a fine resource. Yes it is a little dated with a last update in 2008, but I think it is still pretty on target even three years later. So if you are an Impoverished Social Scientist take a look. If you are simply a person wondering what open source tools are available to address you needs this is a good place to start.

All great ideas will be copied

I am not sure when they actually started doing predictive analytics competitions, but in the last year I do not think a day has gone by without me hearing something about Kaggle. While I do not always agree with the structure of some of the contests particularly with the recent Heritage Health Prize contest license, there is no doubt of the impact Kaggle's contests have had on improving models and interest in those models. I can not count the number of Meetups that I have gone to that the presentation was the result of the work the presenter had done on a Kaggle competition. I have also been to a number of meetings where the Kaggle guys themselves have joined in the presentation and subsequent conversations.

In fact, Anthony Goldbloom is presenting at DC R user group tonight May 17. Anthony will also present at the Philadelphia UseR Group on May 26.

Now comes the rush of the me too contests. On Friday I got an email about an Overstock.com contest for the reclab prize for $1,000,000. I like this contest less than the Heritage Health Prize because in addition to the restrictive software license there is a peer review section rather than a scoring system. I really view this as weak copy of what the Kaggle guys have already done rather than an step forward. So rather than waste time on talking about why I think these competitions need to be open in order to achieve good results I want to look at ways I think they can be better in general.

My last two companies have spent countless hours working on not only how to get a good answer, but also an answer in a reasonable amount of time. We do that by doing a lot of code optimization and parallelization. We have had a lot of success.  However,  there usually comes a time were we need to give up a little predictive accuracy to reduce processing time. Given the size of some of these potential data sets and their expected growth it seems logical that some contests should have a computation time element to their scoring system. I have also heard that some contestants have improved their results by tuning or incorporating outside information into their models. While I think this is unfair if the competition specifically prohibits it, I firmly believe these contests also have their value. We have worked on many a model that became a powerful predictor after the addition of outside data or incorporation of expert opinion.

These are just two simple ideas, but I think they and others like them have the potential to improve and expand the reach of these contests. The addition of other elements will attract other types of talent to these competitions (HPC, Factor researchers, forensics, etc.) producing even better results.

Finally at the other end of the spectrum I always thought a Kaggle Contest Newbie Kit would be great thing. This could be basic as pre-loaded R packages like Max Kuhn's Caret with some additions to simplify use. This would lower the barrier to entry and bring the next generation of teams into the game faster so they can contribute real improvements faster. Besides since most people baseline the data before they move onto more complex models this would relieve some of that work and give more time to perfect the final submission.

In Baseball too much data is never enough

A couple of weeks ago I ran across a post for a intern position at TrackMan which uses information of ball flight to improve performance. They have been very successful in golf. In fact I tried one of their units out over the winter. This job post was more interesting to me because it was looking for an analytic intern for baseball. My first reaction was just what baseball needs more data points in a hulking cloud of data.  Bill James and the Sabermatrics guys have already culled and studied the baseball stats to death even throwing out some stats as irrelavent and creating some others that are better predictors of results.

Then I realized the error of my ways. TrackMan is looking to enrich the result data with physical data. So not just if the ball was a strike or hit or even if it was a fast ball or a curve ball, but what was its speed, location and spin at points along its trajectory from mound to plate. This is very cool. In her talk at the NYC Rusers group Amanda Cox presents a heat map of Rivera's pitches crossing the plate versus other pitchers which was a simple piece of the total pitch but explains why Rivera was better in a very clear way (22:00). I believe this has the potential to change the way pitchers pitch and batters hit.

Newbie R

It is generally accepted that R has one of the steepest learning curves of all the statisitical platforms (SAS, SPSS, etc.). For a person new to the R environment or someone trying to get the lay of the land there are many ways to start. There are some online tuturials and books, I used R in a Nutshell myself. However, more needs be done for developing basic skills of new R users. I had seen comments that a good IDE or GUI like  RStudio can flatten out the learning curve. Another option is going to a training class offered by many different companies. There are also the many local area R user groups.

The R User Meetups are a great place to get started with R. Sometimes the presentations at the R meetups are directed more at the accomplished user than the Newbie. The Greater Boston Area useRs group has come up with a great idea on how to help the newer members to the R community.

At every meeting Jeffery Breen has been doing an opening talk that is directed at the new R user.  So far he has done:

Reshaping Data in R
Grouping & Summarizing Data in R
R plus 15 minutes = Hadoop Cluster

These talks and others like them are a critical piece and an important contribution to building a large competent user base of any open source tools. I am hopeful that other groups will give more of these basic talks and post them so that the R community can build a large repository of presentations on how to do the basics in R.