We have completed maintenance on DiscoverMagazine.com and action may be required on your account. Learn More

How Data Mining Visualizes Story Lines in the Twittersphere

A vast new dataset reveals the popularity of words and phrases on Twitter and how they change over time.

The Physics arXiv Blog iconThe Physics arXiv Blog
By The Physics arXiv Blog
Aug 13, 2020 4:00 PMAug 13, 2020 6:25 PM
shutterstock 1084540781
(Credit: PopTika/Shutterstock)


Sign up for our email newsletter for the latest science news

One curious side-effect of the work to digitize books and historical texts is the ability to search these databases for words, when they first appeared and how their frequency of use has changed over time.

The Google Books n-gram corpus is a good example (an n-gram is a sequence of n words). Enter a word or phrase and it’ll show you its relative usage frequency since 1800. For example, the word “Frankenstein” first appeared in the late 1810s and has grown in popularity ever since.

By contrast, the phrase “Harry Potter” appeared in the late 1990s, gained quickly in popularity but never overtook Frankenstein — or Dracula, for that matter. That may be something of surprise given the unprecedented global popularity of J.K. Rowling’s teenage wizard.

And therein lies the problem with a database founded on an old-fashioned, paper-based technology. The Google Books corpus records “Harry Potter” once for each novel, article and text in which it appears, not for the millions of times it is printed and sold. There is no way to account for this level of fame or how it leaves others in the shade.

Today that changes, thanks to the work of Thayer Alshaabi at the Computational Story Lab at the University of Vermont and a number of colleagues. This team has created a searchable database of over 100 billion tweets in more than 150 languages containing over a trillion 1-grams, 2-grams and 3-grams. That’s about 10 per cent of all Twitter messages since September 2008.

Data Visualization

The team has also developed a data visualization tool called Storywrangler that reveals the popularity of any words or phrases based on the number of times they have been tweeted and retweeted. The database shows how this popularity waxes and wanes over time.

“In building Storywrangler, our primary goal has been to curate and share a rich, language-based ecology of interconnected n-gram time series derived from Twitter,” say Alshaabi and co.

Storywrangler immediately reveals the “story” associated with a wide range of events, individuals and phenomenon. For example, it shows the annual popularity of words associated with religious festivals such as Christmas and Easter. It tells how phrases associated with new films burst into Twittersphere and then fade away, while TV series tend to live on, at least throughout the series’ lifetime. And it reveals the emergence of politico-social movements such as Brexit, Occupy #MeToo and Black Lives Matter.

The storylines can also be compared with other databases to provide more fine-grained insight and analysis. For example, the popularity of film titles on Twitter can be compared with the film’s takings at the box office; the emergence of words associated with disease can be compared with the number of infections recorded by official sources; and words associated with political unrest can be compared with incidents of civil disobedience.

That’s useful because this kind of analysis provides a new way to study society, potentially with predictive results. Indeed, computer scientists have long suggested that social media can be used to predict the future.

Cultural Significance

These storylines have social and cultural significance too. “Our collective memory lies in our recordings — in our written texts, artworks, photographs, audio and video — and in our retellings and reinterpretations of that which becomes history,” say Alshaabi and colleagues.

Now anyone can study it with Storywrangler. Try it, it’s interesting.

As for Harry Potter, Frankenstein and Dracula, the tale that Storywrangler tells is different from the Google Books n-gram corpus. Harry Potter is significantly more popular than his grim-faced predecessors and always has been on Twitter. In 2011, Harry Potter was the 44th most popular term on Twitter while Dracula has never risen higher than 2653rd. Frankenstein’s best rank is 3560th.

Of course, fame is a fickle friend and an interesting question is whether Harry Potter will fare as well as Frankenstein two hundred years after publication. Storywrangler, or its future equivalent, would certainly be able to help.

Ref: Storywrangler: A massive exploratorium for sociolinguistic, cultural, socioeconomic, and political timelines using Twitter. arxiv.org/abs/2007.12988

1 free article left
Want More? Get unlimited access for as low as $1.99/month

Already a subscriber?

Register or Log In

1 free articleSubscribe
Discover Magazine Logo
Want more?

Keep reading for as low as $1.99!


Already a subscriber?

Register or Log In

More From Discover
Recommendations From Our Store
Shop Now
Stay Curious
Our List

Sign up for our weekly science updates.

To The Magazine

Save up to 40% off the cover price when you subscribe to Discover magazine.

Copyright © 2024 Kalmbach Media Co.