Available "Big data sets" over the internet...

[Atabarezz]

I'm opening this topic for everyone to list some Big data* sets available over the net.

• Feel free to list competion/datathon data sets
• Results of web scraping
• Social media data
• Anything bigger than 1 mio records (beyond excel and access)

Best

Altan

* Big data is data that is usually with sizes beyond the ability of commonly used software tools to manage and process within a tolerable elapsed time. A year long credit card transaction history or CDR (Call data record) of a telecoms company for the last 9 months, behavioral credit data of a large financial institution are some examples... 

--------------

[Edited by a Moderator]

We've compiled the responses to this thread into the following Knowledge Base article: Available "Big Data Sets" on the Web

--------------

[Cristian · Accepted solution]

@Atabarezz

This is for you!

Temporal networks with igraph and R (with 20 lines of code!

Regards,

Cristian.

[GarthM · Accepted solution]

Having not fully vetted this list myself I expect there to be a few good resources in there:

http://www.datasciencecentral.com/profiles/blogs/big-data-sets-available-for-free

[raphaelrosati · Accepted solution]

Here are some more....

KD Nuggets is a well respected analytics blog, they have put together a very nice and deep list:

http://www.kdnuggets.com/datasets/index.html

UK Data

https://data.gov.uk/data

Google's PublicData Directory:

http://www.google.com/publicdata/directory

For the Spatial and GIS folks:

http://gisgeography.com/best-free-gis-data-sources-raster-vector/

[Aamir · Accepted solution]

I give to you the mother of big datasets - Reddit. 1.7bn JSON objects; 250GB compressed.

https://www.reddit.com/r/datasets/comments/3bxlg7/i_have_every_publicly_available_reddit_comment

[AndyC]

Bookmarked!!!  Awesome work going on here!

[ategg · Accepted solution]

Loads of really great links from here as well:

https://www.quora.com/Where-can-I-find-large-datasets-open-to-the-public

[pvijendr · Accepted solution]

Here you go!! IMDB Datasets..

http://www.imdb.com/interfaces

A subset of the IMDb plain text data files is available from their FTP sites as follows:

• ftp.fu-berlin.de (Germany)
• ftp.funet.fi (Finland)

Thanks,

PV

[Cristian · Accepted solution]

Twitter 2010 data set http://an.kaist.ac.kr/traces/WWW2010.html

[Atabarezz]

Tweets are removed with the new Terms&Conditions

but the follower data remains and is still usefull...

Thanks

[andrewdatakim · Accepted solution]

One of my favorites are Data.gov where there is tons of public data from all sectors, different size sets and in different formats including API connections.This url,  http://www.data.gov/open-gov/ ,  shows each of the local goverments in the US. They have varying degrees of completion on the local level.