Available "Big data sets" over the internet...

[Atabarezz]

I'm opening this topic for everyone to list some Big data* sets available over the net.

• Feel free to list competion/datathon data sets
• Results of web scraping
• Social media data
• Anything bigger than 1 mio records (beyond excel and access)

Best

Altan

* Big data is data that is usually with sizes beyond the ability of commonly used software tools to manage and process within a tolerable elapsed time. A year long credit card transaction history or CDR (Call data record) of a telecoms company for the last 9 months, behavioral credit data of a large financial institution are some examples... 

--------------

[Edited by a Moderator]

We've compiled the responses to this thread into the following Knowledge Base article: Available "Big Data Sets" on the Web

--------------

[JessHansen]

... and it has a rather ease of access API to get the data into Alteryx

[Atabarezz · Accepted solution]

Search New York Times articles from 1851 to today... Wow!

• retrieving headlines,
• abstracts and
• links to associated multimedia.

You can also search;

• book reviews,
• movie reviews,
• NYC event listings, 
• top stories with images and more.

Here is the link;https://developer.nytimes.com/

Merry Xmass!

Altan @Atabarezz

altan.atabarut@altdata.co

[Cristian · Accepted solution]

http://www.butleranalytics.com/big-data-70-amazing-free-data-sources/

[asabau · Accepted solution]

Here is a good source for Financial data. Some of it is free and nice to play around with.

https://www.quandl.com/

[LordNeilLord · Accepted solution]

Thirty Eight Five have opened up all of their datasets:

https://data.fivethirtyeight.com/

[Cristian · Accepted solution]

https://www.sajari.com/public-data

[GarthM · Accepted solution]

I didn't see this is previous posts so here:

SQUAD Dataset

(Stanford Question Answering Dataset)

apologies if this is a duplicate

[Atabarezz]

Definitely not a duplicate and an excellent source... Thanks

A short brief; Stanford Question Answering Dataset is a reading comprehension dataset, consisting of questions posed by crowdworkers on a set of Wikipedia articles, where the answer to every question is a segment of text, or span, from the corresponding reading passage.

With 100,000+ question-answer pairs on 500+ articles, SQuAD is significantly larger than previous reading comprehension datasets.

Basically you can build your SIRI or better WATSON for Jeopardy out of this...

[Atabarezz · Accepted solution]

Hope you have heard of process mining. It's essentially the same as data mining; to analyze data from different perspectives and summarize it into insights that can be used when making business decisions.

But this time the context are the business processes of an organization. In process mining event logs, data that exists in the information systems of a company are use to visualize and benchmark what is actually happening in the company’s processes and how they are executed in real life.

Almost all IT systems store data in data bases and create logs that can be described in process mining terms as event data.

So below you'll be able to reach out many different event log data and start doing your process mining tasks using @Alteryx

http://data.4tu.nl/repository/collection:event_logs_real

 

above pic represents an inflection point where there is huge processing time... potentially a "bottleneck"

[sschakra]

Hi there - I am not able to access this link:

http://www.gedis-studio.com/online-call-detail-records-cdr-generator.html

I'm looking for a cdr generator and was hoping this would help. Please guide. Many thanks!