Your verification ID is: guDlT7MCuIOFFHSbB3jPFN5QLaQ Big Computing: doRedis: A parallel back end for R/foreach using Redis.

Saturday, April 16, 2011

doRedis: A parallel back end for R/foreach using Redis.

Bryan Lewis recently did a Vignette using RStudio to run some financial data using R/foreach and doRedis. Parallelizing R continues to become more important as users want to run computations on larger and larger quantities of data in reasonable amounts of time particularly on clusters or the cloud. doRedis includes the following features:

  • Support for dynamic pools of parallel workers during running computations.
  • Simple cross-platform parallel computing, including at least Windows, GNU/Linux and OS X.
  • Fault-tolerant 
doRedis Vignette

Redis

2 comments:

  1. Another very promising solution from Zircon Computing http://zircomp.com/content/zNetR. Available on Amazon EC2 , has built in fault tolerance and dynamically adopts to environment changes via auto discovery. we ran financial models on 10 instances 4 cores each and gained almost liner speed up in a range 37 - 39. Was extremely simple to integrate into our app. Zero management , on cloud just provision de-provision instances. We registered @ Zircon web and downloaded zNetR product first which has full set of documentation.

    ReplyDelete
  2. There are many HPC solutions for R that are deployable to the cloud. Some are open source and some are not. Some are more fault tolerant than others. While I have not ever used Zircon Computing's products I am sure they are a good solution for some applications and not so much for others. I was glad that they were using Steve Weston's R/foreach which I believe was a huge contribution to the R package base.

    ReplyDelete