Your verification ID is: guDlT7MCuIOFFHSbB3jPFN5QLaQ Big Computing

Sunday, October 5, 2014

An Example of Plotting a Histogram in base R

Doing plots in R is easy, and there are countless types of graphs you can do in base R, Lattice and other packages like ggplot2. One of the most used plot is a simple Histogram. Histogram ared use to plot the number of times a element in a column has a certain value. It is a great way to first visualise a single column of data. In order to use the histogram plot the values must be numeric.

Lets do a simple plot of data
numbers<-c(1,2,2,3,3,3,4,4,4,4,5,5,5,5,5,6,6,6,6,6,6)
hist(numbers)
plot of chunk unnamed-chunk-1 Well, that is a little funky. The histgrom is technically correct, but what has happened here is the program determined the breaks which resulted in the integers 1 and 2 being put together in the same bin. The way to fix this is to add the break term to the Histgram Plot.
hist(numbers,breaks=c(0,1,2,3,4,5,6))
plot of chunk unnamed-chunk-2 The issue here is you really need to know what you have. A better method may be to create a lot of bins (here I use the length of the numbers vector).
hist(numbers,breaks=length(numbers))
plot of chunk unnamed-chunk-3
Typically the historgram is used to determine if the data you have visually matches a certian distribution like normal, t, poisson, gamma, etc. In order to do that we need to add another element to the histogram plot and add a plot of the actual curve of the distrbution to our histogram. In the call for the histogram we need to add the term freq=FALSE. This will convert the frequencies of the Histogram from pure counts to ratios that add up to 1. Lets see what this looks like with a sample of 10,000 random normal values
dist=rnorm(10000)
hist(dist,freq=FALSE)
curve(dnorm,add=TRUE)
plot of chunk unnamed-chunk-4
So that is really it from a functionality standpoint. Everything else is really to dress up the plot with colors and labels etc.
hist(dist,breaks=100,freq=FALSE,col="blue",border="dark blue",main="Histogram",xlab="randomly generated normal data")
curve(dnorm,add=TRUE,col="yellow",lwd=4)
plot of chunk unnamed-chunk-5
With these things you should be able to do histograms in R.

Wednesday, October 1, 2014

Everything old in Data Science is new again - Graphs

20 years ago when I was still in school we used graphs heavily before we tried to fit a model. The data sets were pretty small, but we used the graphs to help select the model we were going to fit. The process of fitting a model at the time was time consuming and tedious so it was important to minimize the number of inappropriate models that you might try. I even remember having templates of graphs of data which basically said if your data looks like this do this.

As time went on, I continued to use graphs, but I confess I spent less time graphing the data in various ways then I did back then and more time experimenting with a suite of models. The result was that I lost some of the connection to the data that is the art and feel side of data science.

Lately I have been playing with RStudio's shiny app ( along with everyone else). The result has been that it brought back to me the importance of exploring your data graphically and introduced me to a lot of types of graphs that I had never seen or used. Many of these plots provide powerful insight into the structure and the nature of the data that I was working with that I do not believe I could have gotten any other way.

So I thought I might spend the next couple of weeks doing relative short blog posts on graph different types of data set with different types of graphs. I hope this simple review of one of the core elements of data science will be as helpful to you as it has been to me.

For more detail you can always depend on Hadley. His stuff is always great


Monday, June 30, 2014

Introduction to using Random Forests for the Kaggle Titanic Data Set

During the summer a number of the members of the Connecticut R User Group decided to work on a Kaggle competition data set to improve our R programming skills. The first data set we tried was the Titanic data set. This is a fairly simple data set from which we are trying to predict who will survive and who will not. The task for the team was to simply load the data set and the Random Forest package and run the basic model. We completed that task last week. The R code to do that is below:

#load in Package randomForest
install.packages("randomForest")
library(randomForest)
## Read in Titanic Data training and test set
train.data<-read.csv("~/titanic/train(2).csv")
test.data<-read.csv("~/titanic/test.csv")
## Convert data into a simpler dataframe
train<<- data.frame(survived=train.data$Survived,
                    age=train.data$Age,
                    fare=train.data$Fare,
                    pclass=train.data$Pclass,
                    sex=as.integer(factor(train.data$Sex))
                    )
test<<- data.frame( age=test.data$Age,
                    fare=test.data$Fare,
                    pclass=test.data$Pclass,
                    sex=as.integer(factor(test.data$Sex))
                    )
## now we need to get rid of the NAs and make
train$fare[ is.na( train$fare) ]   <- 0
train$age[ is.na( train$age) ]     <- 30
test$fare[ is.na( test$fare) ]   <- 0
test$age[ is.na( test$age) ]     <- 30
labels <- as.factor(train[,1])
train <- train[,-1]
# fit a random forest and make a prediction
rf <- randomForest(train, labels, xtest=test, ntree=100,do.trace=TRUE)
results<<-predictions <- levels(labels)[rf$test$predicted]


As we do more work on the model and try other approached I will update on my blog.

Tuesday, June 10, 2014

Taco Bell's Waffle Taco a novelty that needs to go the way of the pet rock.


This morning I decided to try the Waffle Taco at Taco Bell. What a mistake! It was terrible. A tasteless frozen waffle with rubbery pseudo food inside. Do not ever even bother trying this because you will just regret it like I do. No wonder they use old people in their commericals for this item.Their taste buds are already shot.



Sunday, June 8, 2014

Reading a large number of files into R

I know this is a fairly basic topic, but it is one that caused me problems lately. Normally I only have to read in one data file at a time or I read in a few tables separately.

If I am reading in a single file would do the following

>read.table("file")

 or if it is online

>read.table("url")

If it is a csv file

read.csv("file")

Now the problem arose because I needed to read in 400 files from a directory, but the files were not numerically indexed. So to solve this problem I used the functions list.files and paste.

>names<-list.files("~/directory/")
>complete_names<-paste( "~directory", names, sep="")
>for (i in sep_along(names){
         monitors<-rbind(monitors, read.csv(complete_names[i]))

It was slow, but got the job done.