Independence Day Visualizations and a reminder that it could be worse

So I am getting ready for the Fourth of July like everyone else. Independence Day has a dual meaning for me because not only do I celebrate the birth of our nation on that day, but I also celebrate my Anniversary. So the Fourth of July is a celebration of Liberation and Indentured Servitude on the same day. I try to stay positive. In an effort to do this I hunted up some visualization of the Independence Day ( there were not as many as I expected).

The first is one of two that I found on another blog. It is a fairly standard one of the history of our flag.

The second one is on Fireworks. Who doesn't love Fireworks?

The science and art that goes into Firework to produce those sounds, colors and shapes is just amazing!

On the more humorous side, Josh Sundquist did a Video called "4th of July for Math Nerds" which is pretty funny. I especially like the part where he says we celebrate our country by breaking its laws.

I was searching for more, but I could not find any I liked. Then I came across and map of last years oil spill spread.

I know this is not an uplifting video. However, it does remind me that despite the challenges of 2011, we are in a better place than 2010, and there is reason to celebrate. So have a great weekend and enjoy the Fireworks!

Lighting Talks at Ruser Group Meetups

Recently there has been an increase in the number of lighting talks at Ruser Group Meetups. I first saw this type of talk at the R/Finance 2011 conference in Chicago at UIC. They were without a doubt the best group of talks from that conference. Lighting talks allow groups to cover a wide range of topics in a shallow dive for the listener to further investigate in the future if he or she wants to.

The lighting talks also provide a great format to present some of the more basic topics needed to develop the new R users and remind the more experienced user of good practice. I have become slightly concerned as of late that the R meetups were doing presentations of such a technical level that we were potentially alienating an important group in the R community, the newer user.

I am not sure if there were other people who agreed with me or if it is just a coincidence, but Lighting talks has swept through the Meetup world like a firestorm in the last month. The NYC Predictive Analytics Meetup did a series of Lighting talks as did the NYC Rusers Meetup. I believe they have already posted the presentations on the links that I have provided. The files and videos of presentations posted on these Group Meetup sites are a valuable resource that you should use even if you did not go to the presentation.

On July 13 the Greater Boston Area Ruser Group is doing as series of 5 lighting talks on Predictive Analytics in R, the GoogleViz package, R image cluster and 3D visualizations, RGraphViz and BiomaRt. On July 12 the Los Angeles Ruser Meetup is doing three quick talks on R for Multilevel Modeling, Unit testing and Debugging in R and calling C from R.

Another thing to note is a number of these lighting talks featured a presentation by the guys from RStudio. I think their IDE has quickly become the de facto user interface for R. It has shown itself to be useful to the experienced user, and an invaluable tool to speed up the learning curve for the new user. A friend of mine used RStudio in a class he taught in the spring, and he was astounded by how much faster his students become capable R users compared to when he taught this same class with RStudio in the past. That is a pretty strong endorsement from I guy who I know prefers to program from the command line.

Boston Bruins $156K bar bill data visualization

The Stanley Cup was up on the Martha's Vineyard over the weekend. There was much drinking and celebration to mark its arrival on the small summer party island in Mass., but all that partying palled in comparison to what the Bruins team itself did to a bar at Foxwoods after their victory parade. They ran up a $156K bar bill which included a $100K bottle of champagne! Now that is one high brow drunk feast.

There has been much talk about this crazy bill. Some have even denied that it was really that much. However, there is a picture of the actual bill online. So I think that can put the doubters to rest. Now to get your head around what goes into a $156K bar bill there is nothing better than a visualization. Sixteenwins.com  did just that, and it is a great look at what goes into such a massive bill.



The awesomeness that is the Stanley Cup Champion Bruins bar tab.

I believe this even dwarfs the booze consumed at R/Finance 2011 or the Second Continental Convention. Party on Boston! Sober up Bruins!

Twitter Sentiment Presentations

I have already written about Jeff Gentry's TwitteR package which provides an interface to R. Last night the Boston Predictive Analytics Group gave not one, but three presentations on Twitter sentiment analysis. I have always been a big fan of the Greater Boston R users group meetup which is an unbelievable group with outstanding speakers. It looks like the Boston Predictive Analytics group is following in the same outstanding ways. There are just a huge number of great thinkers and speakers in the Boston area.

Any back to the three talks. One was a presentation by Jeffery Breen on how to use Jeff Gentry's TwitteR Package for R. The second is Shankar Ambaby's python NLTK to look at Twitter among others natural language applications, and finally the now famous twitter flu predictor presentation. All great stuff so take a look at the Boston Predictive Analytics website and go to the next meetup if you are near Boston.

Great Sabermetrics videos with Bill James, cartoons and a Porn Star

I can across three fun videos on sabermetrics. The first is a collage of things including excerpts from a 60 minutes piece on sabermetrics. The tunes are not bad either.

This is actually not a bad introduction in Sabermetrics with spots on some of the luminaries of the field. In fact it is the first video I had ever seen of Bill James. I disagree with Sparky Anderson that Bill James is a "fat little bearded guy who knows nothing about nothing". Although he does have a beard.

The second video is a just totally hilarious. I played it three times and I laughed for the whole thing! I need to find a time to say to someone "because my brain is so big and I am so much better than you". Classic! This cartoon is part of a series and may not really teach you anything, but it is funny.

The only thing that is better than beer and the swedish bikini team is sports and sex. So there is, of course, a video that features Nikki Benz explaining Sabermetrics to a couple of guys preparing for their fantasy baseball draft.

I still like the cartoon better. They had a better script, and they are better actors than the people in Nikki Benz's video

The Art of Beer Making and the Development of Statistics

It is not a commonly known fact that beer making played a major role in the development of Statistics, but it is true. In the last 1800s and early 1900 Arthur Guinness and Sons was a very forward thinking agro-business. which hired some of the most promising students out of universities like Oxford.

In 1899 Guinness hired a young chemist and mathematician for Oxford by the name of William Sealy Gosset. Gosset would be the creator of the t-distribution which is used in Bayesian Data Analysis and the Students t-test. The Student's t-test is the common method for looking at the statistical significance of two sample means and linear regression analysis. Most of his work at Guinness was confidential because Guinness forbade its employee's from publishing any of their work for fear that they would give away trade secrets.  However, Gosset was able to persuade Guinness to permit publishing his work and others under the alias Student to hide his relationship with Guinness arguing that this research was of little use to Guinness's competitors.

Beyond Gosset himself it is amazing the relationships he had with other notable people of his time. All of Gosset's papers were published in Biometrika which was edited by Karl Pearson. Pearson also helped Gosset with the mathematics of the t-distribution. Apparently Pearson was a better chemist and theorist than a mathematician.  Albert Einstein would enlist mathematicians for the same kind of help. Pearson's contributions to statistics are also wide ranging from the Correlation Coefficient to the p-test to PCA. The first book read by Einstein's study group was Pearson's The Grammar of Science.

Gosset was also a friend of R.A. Fischer who in 2010 was named the Greatest Biologist since Darwin. R.A Fischer was not only a biologist but also an important contributor to statistical science. He is the one who recognized the importance of Gosset's work. Fischer created the term "null hypothesis" which rolls off the tongue of every classical statistician to this day. He also is known for the f-distribution, maximum likelihood and  the Fischer-Kolmogorov equation.

It is interesting to note that Gosset was friends with both Pearson and Fischer even though both men had large egos and did not like each other. Maybe back then like today a man like Gosset with a few beers can assuage the ego and befriend even the most unlikely of fellows.

Sabermeterics to predict pitching injury

I just finished The Extra 2% by Jonah Keri on the raise of Sabermetrics at the Tampa Bay Rays, and the resulting World Series that it won them. It is a quick read and a great introduction to the business of Baseball and how MLB teams have incorporated Sabermetrics into their management on recent years.

I met Keri up at the Sabermetric seminar at Harvard which was a fundraiser for the Dana Faber Cancer Institute. I believe they are going to do it again next year, and I strongly encourage anyone interested in Sabermetrics to go. It was my first time talking to people about sports statistics instead of running models and playing with data. Again if it happens again next year go!

In the last year at Bigcomputing we have done a great deal of work on predicting people's health for hospitals and healthcare companies with great success. The are a number of predictive analytics competitions that are trying to develop predictive models for things like predicting if a patient will be hospitalized within the week, month or year. The most well known of these competitions is the Heritage Health Care Prize for $3 Million dollars that is being hosted by Kaggle.com. Obviously with prizes of that size there is real potential to predict things like injury and disease.

Tom Tippett, head analyst of the Boston Red Sox, talked at the seminar about what things the Red Sox look at when they evaluate a player for contract. After this talk, I asked him if there was a predictive component for injury in their sabermetric models for predicting a players future performance. He said their was not. That surprised me, because I thought it could be done with the vast amount of data that is collected on the various players.

In Jonah Keri's book the Rays hired a guy who was able to predict injury of pitchers within a short time frame based on their Pitch F/X data. Josh Kalk published an article in Hardball Times called "The Injury Zone". He was later hired by the Rays where he has continued his work on Pitch F/X data among a mountain of other things. Injury prediction based on the physical and results data openly available in baseball is possible. Maybe even more so with the confidential information the teams have access to like Trackman data and medical and scouting reports. The key is being able to incorporate this into a predictive model with the idea of predicting injury. This type of risk analysis has a lot of value when you are talking about players who make an average of around $4M a year. I also love that Kalk's boss at the Rays was James Click. Click and Kalk should always work together.