Community Spring Cleaning week is here! Join your fellow Maveryx in digging through your old posts and marking comments on them as solved. Learn more here!

General Discussions

Discuss any topics that are not product-specific here.
SOLVED

Available "Big data sets" over the internet...

Atabarezz
13 - Pulsar

I'm opening this topic for everyone to list some Big data* sets available over the net.

 

  • Feel free to list competion/datathon data sets
  • Results of web scraping
  • Social media data
  • Anything bigger than 1 mio records (beyond excel and access)

 

Best

 

Altan

 

* Big data is data that is usually with sizes beyond the ability of commonly used software tools to manage and process within a tolerable elapsed time. A year long credit card transaction history or CDR (Call data record) of a telecoms company for the last 9 months, behavioral credit data of a large financial institution are some examples... 

 

--------------

[Edited by a Moderator]

We've compiled the responses to this thread into the following Knowledge Base article: Available "Big Data Sets" on the Web

--------------

62 REPLIES 62
Cristian
9 - Comet
GarthM
Alteryx Alumni (Retired)

Having not fully vetted this list myself I expect there to be a few good resources in there:

http://www.datasciencecentral.com/profiles/blogs/big-data-sets-available-for-free

 

raphaelrosati
6 - Meteoroid

Here are some more....

 

KD Nuggets is a well respected analytics blog, they have put together a very nice and deep list:

http://www.kdnuggets.com/datasets/index.html

 

UK Data

https://data.gov.uk/data

 

Google's PublicData Directory:

http://www.google.com/publicdata/directory

 

For the Spatial and GIS folks:

http://gisgeography.com/best-free-gis-data-sources-raster-vector/

Aamir
7 - Meteor

I give to you the mother of big datasets - Reddit. 1.7bn JSON objects; 250GB compressed.

 

https://www.reddit.com/r/datasets/comments/3bxlg7/i_have_every_publicly_available_reddit_comment

AndyC
Alteryx Alumni (Retired)

Bookmarked!!!  Awesome work going on here!

Andy Cooper
Senior Solutions Engineer - EMEA
Alteryx
ategg
5 - Atom

Loads of really great links from here as well:

https://www.quora.com/Where-can-I-find-large-datasets-open-to-the-public

pvijendr
7 - Meteor

Here you go!! IMDB Datasets..

 

http://www.imdb.com/interfaces

 

A subset of the IMDb plain text data files is available from their FTP sites as follows:

 

 

Thanks,

PV

Cristian
9 - Comet
Atabarezz
13 - Pulsar

Tweets are removed with the new Terms&Conditions

but the follower data remains and is still usefull...

 

Thanks

andrewdatakim
12 - Quasar
12 - Quasar

One of my favorites are Data.gov where there is tons of public data from all sectors, different size sets and in different formats including API connections.This url,  http://www.data.gov/open-gov/ ,  shows each of the local goverments in the US. They have varying degrees of completion on the local level.

Labels