Available "Big data sets" over the internet...

[Atabarezz]

I'm opening this topic for everyone to list some Big data* sets available over the net.

• Feel free to list competion/datathon data sets
• Results of web scraping
• Social media data
• Anything bigger than 1 mio records (beyond excel and access)

Best

Altan

* Big data is data that is usually with sizes beyond the ability of commonly used software tools to manage and process within a tolerable elapsed time. A year long credit card transaction history or CDR (Call data record) of a telecoms company for the last 9 months, behavioral credit data of a large financial institution are some examples... 

--------------

[Edited by a Moderator]

We've compiled the responses to this thread into the following Knowledge Base article: Available "Big Data Sets" on the Web

--------------

[ChrisTX]

On January 23, 2020 Google released Dataset Search, a free tool for searching 25 million publicly available datasets.

Google blog post: https://www.google.com/amp/s/blog.google/products/search/discovering-millions-datasets-web/amp/

Related post: https://towardsdatascience.com/google-just-published-25-million-free-datasets-d83940e24284

• about half of the datasets in the search results were from for-profit aggregators
• Other popular dataset publishers on the platform include government agencies and research institutions
• US government agencies alone have published over 2 million datasets
• most of the datasets are related to “geosciences, biology, and agriculture.”

Dataset Search: http://g.co/datasetsearch

[mnguyen0803]

Thank you for sharing these. These look great to do some analysis!

[NeilR]

Home - Data Commons... You can build a viz and then click DOWNLOAD DATA in the bottom right corner.