Twitter: Expensive APIs prevent its use for academic research

Twitter was once a mainstay of academic research, a way to take the pulse of the internet. But as new owner Elon Musk has sought to monetize the service, researchers are rushing to replace a once-crucial tool.

Unless Twitter does another about-face soon, this could close the chapter on an entire era of search. “Research using social media data was mostly ‘Twitterology,'” says Gordon Pennycook, associate professor of behavioral sciences at the University of Regina, Italy. “It was the main font that people used.”

Until the Musk acquisition, Twitter’s API, which allows third-party developers to collect data, was considered one of the best on the internet.

It has enabled studies of everything from how people respond to weather disasters to how to stop the spread of misinformation online. The problems they faced are only getting worse, making this type of research more important than ever.

But Twitter decided to end free access to its API in February and launched paid tiers in March. The company said it was “looking for new ways to continue serving” the gym, but even so, it has begun unceremoniously shutting down access to non-paying third-party users.

While the cut has caused problems for many different types of users, including public transport agencies and first responders, academics are among the groups most affected.

Researchers who have relied on Twitter for years told al The limit that he had to stop using it. It’s very expensive to pay for access to its API, which has reportedly skyrocketed to $42,000 a month or more for a business account.

As a result, scientists have lost important information about human behavior. And as they struggle to find new sources, there’s still no clear alternative.

help researchers

Twitter gave researchers a way to observe people’s real reactions rather than having to ask study participants how they think they might react in certain scenarios.

This was central to Pennycook’s research into strategies to avoid fomenting misinformation online, such as by showing people content that asks them to think about accuracy before sharing a link.

Without being able to see what an individual is actually tweeting, researchers like Pennycook can simply ask someone in a survey what kind of content they would share on social media.

“It’s basically hypothetical,” says Pennycook. “For tech companies that would actually be able to implement one of these interventions, they wouldn’t be impressed… We’ve had to do experiments somewhere to show that it really can work in nature.”

Twitter’s cheapest API tier of $100 per month would allow third parties to raise just $10,000 per month. That’s just 0.3% of what they previously had free access to in a single day, according to the letter.

And even its “outrageously expensive” corporate level, the coalition argued, was not enough to conduct some ambitious studies or maintain important instruments.

One such tool is Botometer, a system that ranks the likelihood that a Twitter account is a bot.

While Musk has expressed skepticism about things like disinformation research, he actually publicly used the Botometer to estimate how many bots were on the platform during his bid to get out of the deal he made to buy Twitter. Now, your decision to charge for accessing the API may lead to the closure of that API.

A notice on Botometer’s website states that the tool will likely stop working soon. “We are actively looking for ways to keep this site alive and free for our users, which will involve training a new machine learning model and working with Twitter’s new paid API plans,” he says.

“Please note that even though it is possible to create a new version of the Botometer website, it will have limited functionality and quota compared to the current version due to the limited Twitter API.”

The impending shutdown comes as a major blow to Botometer co-creator Kai-Cheng Yang, a researcher who studies disinformation and bots in social media and who recently received his doctorate in computer science from Indiana University Bloomington.

“My entire PhD, my entire career, is basically based on Twitter data now. It is likely that it will no longer be available in the future,” Yang al said The limit. When asked how he might have to approach his job differently now, he says, “I’ve asked myself that question constantly.”

Other researchers are equally baffled. “The platform has gone from being one of the most transparent and accessible on the planet to rock bottom,” says letter signatory Rebekah Tromble, director of the Institute for Data, Democracy and Politics (IDDP) at George Washington University.

Some of Tromble’s previous work, studying political conversations on Twitter, was funded by the company before it changed its API policies.

“The Twitter API has been absolutely vital to the research I’ve been doing for years,” Tromble says The limit. And, like Yang, he has to adapt to the platform’s new pricing schemes. “I’m just not studying Twitter right now,” she says.

But there aren’t many other options for mass-gathering data from social media. While collecting data from a website without using an API is an option, it’s more tedious work and can be fraught with other risks.

Twitter and other platforms have tried to cut back on some scraping because it can be hard to discern whether it’s done in the public interest or for malicious purposes like phishing.

End of APIs?

Meanwhile, other social media giants have been even more restrictive than Twitter with access to APIs, making it difficult to migrate to a different platform. And the restrictions appear to be getting tighter: Last month, Reddit similarly announced it would begin restricting third-party access to its API.

“I just wonder if this is the beginning of companies becoming less and less willing to have the data-sharing API,” says Hause Lin, a postdoctoral researcher at MIT and the University of Regina who develops ways to prevent the spread of hate speech and misinformation online. “It looks like the landscape is totally changing, so we don’t know where it’s going now.”

There are signs that things could be even worse. Last week, the news reported that Twitter has told some researchers they should scrap the data they’ve already collected through its decahose, which provides a random sample of 10% of all content on the platform, unless they pay for a business account that can run it. thousand (R $ 212.3 thousand) per month. The move is “big data’s equivalent of book burning,” said an anonymous academic who received the press notice. news.

OR The limit he failed to verify that information with Twitter, precisely because the social network routinely replies to journalists’ questions with a poop emoji.

According to the portal, none of the researchers it spoke to received such an alert, and so far it appears to be limited to users who previously paid to use decahose (just a use of the Twitter API that would previously have been free or low cost for academics).

Both Tromble and Yang have used decahose in their work in the past. “Never before has Twitter gone back to researchers and said that now that your contract is up, you have to give up all the data,” Tromble says. “It’s a complete farce. It will devastate a number of really important ongoing research projects.”

Other scholars also tell the The limit that Twitter’s reported effort to have researchers “delete all Twitter data stored and cached on their systems” without the company’s signature would be devastating.

This can prevent students from completing work they have invested years in if they are forced to clear the data before publishing the results. Even if they’ve already published their work, access to the raw data is what allows other researchers to test the strength of the study by being able to replicate it.

“This is very important for transparent science,” Yang says. “This is just personal preference. I would probably go against Twitter policy and still share the data and make it available because I think the science is more important here.”

Twitter has been a great place for digital field experiments in part because it has encouraged people from diverse backgrounds to come together in one place online. This is unlike Facebook or Mastodon, which tend to have more friction between social circles. This centralization sometimes fomented conflict, but was invaluable to academics.

“If the research isn’t as good, we won’t be able to know as much about the world as we used to,” says Pennycook. “And maybe we can figure out a way to close that gap, but we haven’t figured it out yet.”

through The limit

The Twitter post: Expensive API prevents use for academic research first appeared on Olhar Digital.

Source: Olhar Digital