“According To A Study…”: The Art of Verification
Discover cutting-edge techniques to critically analyze modern investigations. Learn how to identify biases and accurately assess data from public sources for well-rounded insights.
We’ve all seen those news stories with wild scientific claims backed by just one study or one so-called expert. It’s easy to get caught up in the excitement, but how do you know if what they’re saying is really true? Is the expert really an expert, and does the study actually say what the headline claims? In this article, I’ll show you some simple ways to check if the science behind the story holds up, so you can spot the difference between real discoveries and just another flashy headline.
This article will teach you how to tell if a scientific claim in the news is legitimate or just another flashy headline. You’ll learn how to use tools like fact-checkers to verify claims, check if someone is really an expert, and research the science behind the claim. Plus, you’ll find out how to spot biased news sources and understand the basics of legitimate scientific research. By the end, you’ll have simple, practical steps to make sure you’re getting the facts, not just the hype.
Fact-Checkers
Fact-checking suspicious claims is easier when someone has already done the work for you. The process of figuring out if a professional has already fact-checked something has become easier with the advent of fact-check search engines. My favorite is Google Fact-Check Explorer.
Using the search bar you can look for fact-checks on any particular claim by searching topics, people, or images. In this case I am using the word “claim” as a catchall for any kind of questionable news, story, or thing you’ve heard.
So if there is a hit, the result will list:
- where the story originated
- a credible source of information that has fact-checked the claim with a link to the fact-check article/post
- a quick summary of the finding (true, false, humor, satire, needs context, etc.)
The search engine will also provide keyword guesses as you type, just like with regular Google. So for example, when I start typing “flat", a dropdown appears.
Interestingly, the tool also checks social media, like in this case my search of “flat earth” yielded an Instagram rumor about an astronaut supposedly admitting the earth was flat. See the following result:
The listing shows that there was a claim stating that a Polish astronaut “Admitted The Earth Is Flat”, and it originated from Instagram. It also identifies the fact-checker was Lead Stories. The claim was rated as Humor (i.e. the guy was joking).
Even if there are no fact-checks about the thing you are researching, you can search the expert’s name in one of the previously mentioned fact-checkers. This is the most important step in verifying the expert. They may have legit credentials, but they could still make ridiculous claims. So you want to look up if there are other fact-checks of this expert’s statements.
Other Fact-Check Tools
There are also other fact-checking sites and search engines to consider. These are useful in case Google fact-check Explorer does not have what you are looking for.
Fake Experts’ Research Background
When it comes to scientific experts, their work is public. You will often find that fake claims were made by real scientists.
You can also research the person making the claim by using Google Scholar.
Google Scholar is a great tool for checking out an expert’s research background.
Below you see a simple search on the name of a scholar (a real expert in this case). In the results it suggests a profile and also shows articles by authors with the name. If I click on either it brings me to their profile.
Here is what the profile of a real expert looks like. Note that Dr. Stark's profile shows that she is affiliated with the University of Oxford and has a verified email address from the university.
For further information about them you can click on the articles tab for all of their articles and the public access tab for any of their work that is publicly accessible.
You can also choose the “cited by” and “co-authors” tabs. The "cited by" tab refers to articles that cited the profile owner's work and "co-authors" are colleagues that co-wrote articles with them.
This adds legitimacy to the expert by showing that they are recognized within the community. In this case we see that our expert's work has been cited over 600 times by her colleagues.
The following list of co-authors is full of established experts that chose to work with our expert.
Real Scientists, Fake Experts
In contrast, watch out for experts that do not have any sort of published research or credentials. However, you will often find that fake claims are made by actual scientists with research backgrounds. Many so-called experts have legitimate degrees and have published research, but that does not mean they are real experts.
There are two notable kinds of real scientists that are fake experts.
The first kind are scientists that do not have expertise in the field that they are talking about. Simply put, a geologist should not claim to be an expert in neurology. You can quickly check someone's area of expertise with their Google Scholar profile.
Remember that the profile lists the person's research topics at the top of the profile. In our example above, we see that the expert researches cognitive neuroscience.
Secondly, there are researchers who are on the fringe or even fake their research. One example is Dr. Judy Mikovits, who made several disproven claims about vaccines causing cancer and alleged that she was sent to jail because of her discoveries. See this fact-check about Dr. Mikovits.
Dr. Mikovits did earn a PhD in Biology from George Washington University and published scientific research. However, she has also had her research papers retracted because her findings could not be replicated.
In this sort of case, you can use Retraction Watch which monitors developments related to retractions and maintains a database of retracted scientific research.
If you use this tool, make sure to type in the person's last name first. See below, when I searched "Judy Mikovits" I got no results.
Instead, I needed to search "Mikovits, Judy"
Mikovits has three retractions identified in the database. Here is one of the hits, which is obviously pretty dense reading material.
License
Now lets return to other means of verifying credentials. Many experts are in jobs that require a license of some sort, such as a medical doctor, which are all typically issued by the state or province government. And job licenses are publicly available.
In addition to verifying their profession, licenses can also reveal if the person was suspended or involved in any problematic issues.
If you are researching anyone where you know their profession, it is worth a quick Google search asking if that profession requires a license in that state, or in general.
If you are researching someone in the United States, the website Career One Stop has a great tool. Career One Stop lists out all of the jobs requiring licenses and allows you to search by job title and state to see what kind of licensing requirements exist. The tool also shows where the public can look up an individual’s license for a specific profession and location.
Verify College Degrees, Enrollments, Dropouts
You can verify if someone attended a university in the U.S. by calling the school to request a student’s “directory information” or by submitting a request through studentclearinghouse.org.
But be aware that the website requires a $30 fee (but you are only charged if they find results).
Lookup The Study
Fake experts and suspicions claims often reference a specific study as evidence. However, this could be a mischaracterization of the study. Therefore, you can lookup the study or article that was referenced.
Google Scholar is also a tool for checking a research paper or study that is referenced by an alleged expert.
If you can’t access the article you can check for other versions that are possibly accessible or there may be a link on the right to a library affiliated.
Above you see how researchers are listed with the article, and you can click on the name to pull up their profile.
You can see how the article is considered by the community in that field by searching articles that cite the original article.
You can look for other studies that are on the same or similar topic be clicking “Related Articles”.
This is useful for seeing if the other articles agree or provide conflicting results. For example, maybe you can find one article saying smoking is good for your lungs, but most other articles and studies on the topic will not agree.
What is Legitimate Scientific Research?
Double-Blind, Peer-Reviewed, and Published in a Scholarly Journal.
Scientific research is generally considered legitimate when it follows a few key steps. First, it should be peer-reviewed, meaning other experts in the field have evaluated the work and found it solid. Next, the study should use a double-blind method, which means neither the participants nor the researchers know who’s in which group, reducing bias. Finally, it should be published in a scholarly journal, which adds another layer of credibility by ensuring the research meets certain standards for quality and accuracy.
For more information on what defines a scholarly journal, you can go to:
Levels of Scientific Evidence
The News Literacy Project explains the different levels of evidence. This is important to understand because misinformation often cites a single study (ignoring the rest of the studies on that topic), one expert’s opinion, or anecdotal evidence. All three of those should not be considered sufficient evidence.
This is also good to know so that when you search for the definitive position (or positions) on a topic. If you simply search google scholar for studies on a topic it is easier to know what you should look for. If it is available, the best evidence is a Meta-Analysis and Systematic Review.
As an example, I did a search in Google Scholar to see if Acupuncture was effective treatment for Arthritis. In the search results note that on the left side you have the option to choose a recent timeframe. The top and bottom results show that they are each “a systematic review”. However, in this case they were both also Meta Analyses. Also note that the top result was from 2008, and the bottom result is from 2018, and therefore more relevant.
Credibility Checkers
Here are two free tools for evaluating credibility. The first, Is This Credible, will evaluate specific articles. The second, Media Bias fact-check, evaluates the news source itself.
- Is this Credible (by Factual) - This tool assesses the credibility of articles using an automated algorithm
It describes itself as “A simple website to get the rating of any news article you find, or search for the most factual news on any topic.”
Factual describes its rating system as follows:
Each article receives a grade between 1-100% based on four metrics: the site quality, the author’s expertise, the quality and diversity of sources, and the tone of the writing. Each of these metrics answers key questions about the information value of the article.
Site quality: Does this site have a history of producing well-sourced, highly-informative articles?
Author’s expertise: Does the author have a track record of writing well-researched, informative articles on the topic? Does the author focus on the topic and hence may have some expertise?
Quality and diversity of sources: How many unique sources and direct quotes were used in the article? What is the site rating of those sources?
Article’s tone: Was the article written in a neutral, non-opinionated tone or was it opinionated with emotional language?
These four metrics combine to give a single percentage grade, which we interpret as the probability of the article being informative. Grades above 75% are regarded as highly likely to be informative while grades below 50% are less likely to be informative.
- Media Bias Fact-Check - This tool produces media bias ratings based on a website’s level and direction of bias, and also identifies questionable and conspiracy sources.
Though it is irritatingly full of ads, you can search a given news source, and the site returns a very detailed and extensive analysis about credibility and bias (with a quick summation visual at the top in case you want a quick answer). The results are from human beings, not an algorithm.
Research Guidance
If you are looking for more in-depth guidance on researching and evaluating a news source, The News Literacy Project, Pierce College, and Mind Tools created resources specifically for this situation.
The News Literacy Project suggests the following steps:
1 - Do a quick search: Conducting a simple search for information about a news source is a key first step in evaluating its credibility.
2 – Look for standards: Reputable news organizations aspire to ethical guidelines and standards, including fairness, accuracy and independence.
3 – Check for transparency: Quality news sources should be transparent, not only about their reporting practices, but also about their ownership and funding.
4 – Examine how errors are handled: Credible news sources are accountable for mistakes and correct them. Do you see evidence that this source corrects or clarifies errors?
5 – Assess news coverage: An important step in vetting sources is taking time to read and assess several news articles.
Wrapping It Up
In today’s fast-paced world of news and headlines, it’s easy to get swept up in bold claims about scientific discoveries. But now you’ve got the tools to dig deeper. Whether it's verifying if someone is truly an expert or checking if a study actually backs up the claim, you can take a few extra steps to get the real story. By using fact-checkers, researching expert credentials, and understanding what legitimate science looks like, you can confidently separate credible information from sensational headlines. Remember, a little investigation goes a long way in making sure you’re getting the truth.