Your verification ID is: guDlT7MCuIOFFHSbB3jPFN5QLaQ Big Computing

Sunday, October 5, 2014

An Example of Plotting a Histogram in base R

Doing plots in R is easy, and there are countless types of graphs you can do in base R, Lattice and other packages like ggplot2. One of the most used plot is a simple Histogram. Histogram ared use to plot the number of times a element in a column has a certain value. It is a great way to first visualise a single column of data. In order to use the histogram plot the values must be numeric.

Lets do a simple plot of data
numbers<-c(1,2,2,3,3,3,4,4,4,4,5,5,5,5,5,6,6,6,6,6,6)
hist(numbers)
plot of chunk unnamed-chunk-1 Well, that is a little funky. The histgrom is technically correct, but what has happened here is the program determined the breaks which resulted in the integers 1 and 2 being put together in the same bin. The way to fix this is to add the break term to the Histgram Plot.
hist(numbers,breaks=c(0,1,2,3,4,5,6))
plot of chunk unnamed-chunk-2 The issue here is you really need to know what you have. A better method may be to create a lot of bins (here I use the length of the numbers vector).
hist(numbers,breaks=length(numbers))
plot of chunk unnamed-chunk-3
Typically the historgram is used to determine if the data you have visually matches a certian distribution like normal, t, poisson, gamma, etc. In order to do that we need to add another element to the histogram plot and add a plot of the actual curve of the distrbution to our histogram. In the call for the histogram we need to add the term freq=FALSE. This will convert the frequencies of the Histogram from pure counts to ratios that add up to 1. Lets see what this looks like with a sample of 10,000 random normal values
dist=rnorm(10000)
hist(dist,freq=FALSE)
curve(dnorm,add=TRUE)
plot of chunk unnamed-chunk-4
So that is really it from a functionality standpoint. Everything else is really to dress up the plot with colors and labels etc.
hist(dist,breaks=100,freq=FALSE,col="blue",border="dark blue",main="Histogram",xlab="randomly generated normal data")
curve(dnorm,add=TRUE,col="yellow",lwd=4)
plot of chunk unnamed-chunk-5
With these things you should be able to do histograms in R.

Wednesday, October 1, 2014

Everything old in Data Science is new again - Graphs

20 years ago when I was still in school we used graphs heavily before we tried to fit a model. The data sets were pretty small, but we used the graphs to help select the model we were going to fit. The process of fitting a model at the time was time consuming and tedious so it was important to minimize the number of inappropriate models that you might try. I even remember having templates of graphs of data which basically said if your data looks like this do this.

As time went on, I continued to use graphs, but I confess I spent less time graphing the data in various ways then I did back then and more time experimenting with a suite of models. The result was that I lost some of the connection to the data that is the art and feel side of data science.

Lately I have been playing with RStudio's shiny app ( along with everyone else). The result has been that it brought back to me the importance of exploring your data graphically and introduced me to a lot of types of graphs that I had never seen or used. Many of these plots provide powerful insight into the structure and the nature of the data that I was working with that I do not believe I could have gotten any other way.

So I thought I might spend the next couple of weeks doing relative short blog posts on graph different types of data set with different types of graphs. I hope this simple review of one of the core elements of data science will be as helpful to you as it has been to me.

For more detail you can always depend on Hadley. His stuff is always great