General Discussions

Discuss a wide range of topics! Questions about the Alteryx Platform should be directed to the appropriate Product discussion forum.
SOLVED

Available "Big data sets" over the internet...

Atabarezz
13 - Pulsar

I'm opening this topic for everyone to list some Big data* sets available over the net.

 

  • Feel free to list competion/datathon data sets
  • Results of web scraping
  • Social media data
  • Anything bigger than 1 mio records (beyond excel and access)

 

Best

 

Altan

 

* Big data is data that is usually with sizes beyond the ability of commonly used software tools to manage and process within a tolerable elapsed time. A year long credit card transaction history or CDR (Call data record) of a telecoms company for the last 9 months, behavioral credit data of a large financial institution are some examples... 

 

--------------

[Edited by a Moderator]

We've compiled the responses to this thread into the following Knowledge Base article: Available "Big Data Sets" on the Web

--------------

61 REPLIES 61
Cristian
9 - Comet
GarthM
Alteryx Alumni (Retired)

Having not fully vetted this list myself I expect there to be a few good resources in there:

http://www.datasciencecentral.com/profiles/blogs/big-data-sets-available-for-free

 

raphaelrosati
6 - Meteoroid

Here are some more....

 

KD Nuggets is a well respected analytics blog, they have put together a very nice and deep list:

http://www.kdnuggets.com/datasets/index.html

 

UK Data

https://data.gov.uk/data

 

Google's PublicData Directory:

http://www.google.com/publicdata/directory

 

For the Spatial and GIS folks:

http://gisgeography.com/best-free-gis-data-sources-raster-vector/

Aamir
7 - Meteor

I give to you the mother of big datasets - Reddit. 1.7bn JSON objects; 250GB compressed.

 

https://www.reddit.com/r/datasets/comments/3bxlg7/i_have_every_publicly_available_reddit_comment

AndyC
Alteryx Alumni (Retired)

Bookmarked!!!  Awesome work going on here!

Andy Cooper
Senior Solutions Engineer - EMEA
Alteryx
ategg
5 - Atom

Loads of really great links from here as well:

https://www.quora.com/Where-can-I-find-large-datasets-open-to-the-public

pvijendr
7 - Meteor

Here you go!! IMDB Datasets..

 

http://www.imdb.com/interfaces

 

A subset of the IMDb plain text data files is available from their FTP sites as follows:

 

 

Thanks,

PV

Cristian
9 - Comet
Atabarezz
13 - Pulsar

Tweets are removed with the new Terms&Conditions

but the follower data remains and is still usefull...

 

Thanks

andrewdatakim
12 - Quasar
12 - Quasar

One of my favorites are Data.gov where there is tons of public data from all sectors, different size sets and in different formats including API connections.This url,  http://www.data.gov/open-gov/ ,  shows each of the local goverments in the US. They have varying degrees of completion on the local level.