Everything old in Data Science is new again - Graphs

20 years ago when I was still in school we used graphs heavily before we tried to fit a model. The data sets were pretty small, but we used the graphs to help select the model we were going to fit. The process of fitting a model at the time was time consuming and tedious so it was important to minimize the number of inappropriate models that you might try. I even remember having templates of graphs of data which basically said if your data looks like this do this.

As time went on, I continued to use graphs, but I confess I spent less time graphing the data in various ways then I did back then and more time experimenting with a suite of models. The result was that I lost some of the connection to the data that is the art and feel side of data science.

Lately I have been playing with RStudio's shiny app ( along with everyone else). The result has been that it brought back to me the importance of exploring your data graphically and introduced me to a lot of types of graphs that I had never seen or used. Many of these plots provide powerful insight into the structure and the nature of the data that I was working with that I do not believe I could have gotten any other way.

So I thought I might spend the next couple of weeks doing relative short blog posts on graph different types of data set with different types of graphs. I hope this simple review of one of the core elements of data science will be as helpful to you as it has been to me.

For more detail you can always depend on Hadley. His stuff is always great

Figuring out the number of R users in comparison to SAS, SPSS and all

I have wrote about this problem before and have tried to come up with various approaches to come up with a prediction.  I have quoted the article  The Popularity of Data Analysis Software by Robert A. Muechen before. It is a great article, Muechen maintains it as a living document constantly updating its contents. If you have not looked at it in a while it is worth another look.

There are two areas of Muechen's article that I would like to talk about. They are Internet Searches and Job Postings. For R both of these types of measurables are problematic and may under report the number of Internet Searches or Jobs Posted.

A person can effectively search for SAS, SPSS, Strata, but not really for R. The problem is that R is a single character that is heavily used ( Toys R Us, R Kelly, R rated movies, etc). Therefore a number of the searches for R may be for other things or are more likely to have other terms in the search line than SAS, SPSS and all. This results in a disconnect in the search results no matter how you define it. I will say as the number of R users has grown in the last few years this has improved. I ran the Google Adwords for the website at Revolution Computing and this proved to be a challenging problem to crack. It took a lot of thought and refinement that would not be required if R had a more unique name. Another search term to stay away from in Google Adwords is BI. It does not always mean Business Intelligence.

Job searches have a similar issue, but with an added twist. Rarely do job posters list the requirement as solely an "R" programmer, and rarely do people describe their skill as "R" programmer. Again I ran into this problem while trying to find talent for Revolution Computing. Job Posters and Job Seekers often list their "R" skill as "R/S", "R/S+","R/SPlus" or "R/S-Plus" and numberous other permutations of that. Again R's single character name is problematic.

On twitter this has been addressed by using the Hashtag of Rstats instead of R. The difference in ease of access and elimination of confusion has been huge. If R would ever expand its name to something like Rstats it would radically improve the quality of searches for information, and help employers find employees. Just a thought.

What Operating Systems do R users use?

I went to the Greater Boston R meetup last night where the Rstudio guys did a presentation on their IDE. It was well attended, and Josh and JJ did a great job.

At the end of the talk JJ said that the breakdown of Operating Systems for the Rstudio downloads was roughly 60% windows, 30% mac and 10% linux. Frankly I was surprised by those numbers. I always felt that the majority of people who use open source R would run it on an open source Operating System. Rstudio's results would seem to strongly counter that thought. After the talk, I remembered a survey done by KDnuggets in September of 2007 on what Operating system did people use for their analytics work. It also showed about 60% Windows users but 30% linux and 10% Mac. I believe this suggests an increase in the number of people using Macs to do analytics in the last four years.

I am not sure the data from downloads of an IDE or a survey of Operating Systems by KDnuggets is a good indicator of what is the relative popularity of the various Operating Systems being used by the R community. I do believe it does show the importance of supporting Windows which is consistantly shown as the dominate platform among people surveyed and downloads at least Rstudio. It also showed that if you make an IDE that is not supported on the Mac you could be missing 30% of the potential users.

The recent KDnuggets survey on which R interfaces which was interesting. For example, I would not have expect so many people to respond that they use Tinn-R or R commander but they do.  I always enjoyed the KDnuggets surveys.

KDnuggets Operating System survey
KDnuggets R interface poll