Question
Where can I find available "Big Data Sets" over the internet?
Bigdata is data that is usually with sizes beyond the ability of commonly used software tools to manage and process within a tolerable elapsed time. A year-long credit card transaction history or CDR (Call data record) of a telecoms company for the last 9months, behavioral credit data of a large financial institutionare some examples...
Answer
- Amazon (AWS) has a Large Data Sets Repository
- Data.gov has close to 190k public data sets
-
One of the standard datasets for Hadoop is the Enron email dataset comprising emails between Enron employees during the scandal. It's a great practice dataset for dealing with semi-structured data (file scraping, regexes, parsing, joining, etc.). It's ~400MB (compressed) and available for download athttp://www.cs.cmu.edu/~enron/
-
Collection of audio features and metadata for a million contemporary popular music tracks http://labrosa.ee.columbia.edu/millionsong/. SecondHandSongs dataset -> cover songs musiXmatch dataset -> lyrics Last.fm dataset -> song-level tags and similarity Taste Profile subset -> user data thisismyjam-to-MSD mapping -> more user data tagtraum genre annotations -> genre labels Top MAGD dataset -> more genre labels You can eitherdownload the entire dataset (280 GB) ora subset of 10,000 songs (1.8) for a quick taste.
- GDELT set:http://www.gdeltproject.org/data.html
-
NY City taxi data sets 1.1BN records:http://www.nyc.gov/html/tlc/html/about/trip_record_data.shtmlAnalyzing 1.1 Billion NYC Taxi and Uber Trips, with a Vengeance
An open-source exploration of the city's neighborhoods, nightlife, airport traffic, and more, through the lens of publicly available taxi and Uber data
-
Airline data set 1987-2008:https://github.com/h2oai/h2o-2/wiki/Hacking-Airline-DataSet-with-H2O
-
Google Big Table - hosted by Google:https://cloud.google.com/bigquery/sample-tablesWeather, timeline of actions such as pull requests and comments on GitHub repositories with a nested or flat schema, US births 1969-2008, Shakespeare - number of times each word appears, Wikipedia articles over 300,000,000 million rows.
-
LENDING CLUB:https://www.lendingclub.com/info/download-data.action
- Here is a telecom Italia datasetasa result of a computation over the Call Detail Records (CDRs)generated by the Telecom Italia cellular network over the city of Milano; You may have to sign-in and activate your account but it's totally free...https://dandelion.eu/datagems/SpazioDati/telecom-sms-call-internet-mi/description/
- Data Science Centralhttp://www.datasciencecentral.com/profiles/blogs/big-data-sets-available-for-free
- KD Nuggets is a well respected analytics blog, they have put together a very nice and deeplist:http://www.kdnuggets.com/datasets/index.html
- UK Datahttps://data.gov.uk/data
- Google's Public Data Directory:http://www.google.com/publicdata/directory
- For the Spatial and GIS folks:http://gisgeography.com/best-free-gis-data-sources-raster-vector/
- The mother of big datasets - Reddit. 1.7bn JSON objects; 250GB compressed.https://www.reddit.com/r/datasets/comments/3bxlg7/i_have_every_publicly_available_reddit_comment
- Loads of really great links from hereas well:https://www.quora.com/Where-can-I-find-large-datasets-open-to-the-public
- http://www.imdb.com/interfacesA subset of the IMDb plain text data files is available from their FTP sites as follows:ftp.fu-berlin.de (Germany) ftp.funet.fi (Finland)
- One of my favorites are Data.gov where there is tons of public data from all sectors, different size sets and in different formats including API connections. This url, http://www.data.gov/open-gov/, shows each of the local governmentsin the US. They have varying degrees of completion on the local level.
- https://www.linkedin.com/pulse/need-data-bob-wyman?trk=mp-author-card
- The Government of Canada has an Open Data portal -- http://open.canada.ca/en/open-data -- it takes some digging to find the gems, but there are some.
- There's also some open mapping data at -- http://open.canada.ca/en/open-maps.
- This 3TB+ dataset comprises the largest released source of GitHub activity to date. It contains a full snapshot of the content of more than 2.8 million open-source GitHub repositories including more than 145 million unique commits, over 2 billion different file paths, and the contents of the latest revision for 163 million files, all of which are searchable with regular expressions.https://cloud.google.com/bigquery/public-data/github
- 15.49TB of research data available.http://academictorrents.com/
- Australia, New South WalesOpen data http://data.nsw.gov.au/
- USAFacts: Our Nation, in numbers. Federal, state, and local data from over 70 government sources.
