Working with Big Data in R

This year has been crazy in terms of data I use for my analysis. Frequently the traditional methods I use in R would fail to allocate enough memory for the task at hand. Luckily, R has great support for such tasks. I will note down a few package names that have served me well lately for future reference.

ffbase – Working with flatfiles, without storing them in memory
biglm – Doing analysis on such ff
foreach – to parallelize loops
parallel – ^
doMC – ^ alternative to parallel
glmmML – simplified random intercepts binary models. Faster than lme4 but still not fast enough for my purposes…

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.

Create a website or blog at

Up ↑

%d bloggers like this: