A Citizen Science Peer Reviewer’s Guide

What Is Citizen Science?

Citizen science is the idea that science belongs to everyone—not just professional scientists. Throughout history, many important discoveries have come from curious people asking good questions, collecting observations, and contributing to scientific knowledge. Citizen science expands the scientific process beyond universities and laboratories by inviting the public to participate.

What Is Citizen Science Peer Review?

Citizen science peer review applies that same principle to evaluating scientific research. Instead of limiting review to a small group of experts, citizen reviewers help assess whether a paper's claims are supported by evidence. You do not need to be an expert in a field to ask powerful scientific questions like: What is the evidence? Does the conclusion match the evidence? Are the limitations clearly explained? By participating in peer review, you help make science more transparent, accountable, and trustworthy. Every thoughtful review strengthens the scientific record and helps move knowledge closer to reality. Science is too important to be left only to scientists. The health of a free and informed society depends on citizens who can understand evidence, ask good questions, and participate in the creation and evaluation of knowledge. Citizen science—and citizen peer review—help ensure that science remains transparent, accountable, and connected to the public it serves. You pay for it, you should have access to it. Peer review is also an important skill to have as a citizen for everyday life, especially in an age where information is abundant, misinformation is rampant, disinformation campaigns are common, and citizens must inoculate themselves by moving from “consumers of information” to “organizers of information to make meaning.”

Why Citizen Peer Review Matters More Than Ever

We live in an age of unprecedented information abundance. Every day, people are bombarded with news stories, research findings, expert opinions, political arguments, advertisements, social media posts, podcasts, videos, and AI-generated content. Information arrives faster than any human can reasonably process it. It comes from countless sources, in countless formats, with wildly varying levels of quality and trustworthiness.

Information scientists often describe this challenge using the “four Vs” of information: volume (how much information exists), velocity (how fast it arrives), variety (how many forms it takes), and veracity (how reliable it is). Never before have ordinary citizens faced such a difficult environment for making sense of the world.

The result is often overwhelm, anxiety, confusion, and distrust. Faced with too much information and too little time, many people resort to shortcuts. They trust sources that align with their beliefs, dismiss sources they dislike, follow authority figures without verification, or simply disengage altogether. Yet these strategies leave people vulnerable to misinformation, manipulation, and poor decision-making.

Citizen science peer review offers a practical alternative.

At its core, peer review is not a specialized academic activity. It is a way of thinking. Scientific peer review simply applies a set of fundamental questions to a scientific paper:

• What is the claim?

• What is the evidence?

• Does the evidence support the claim?

• Could someone repeat the process?

• What are the limitations?

These same questions can be applied far beyond science.

When you hear about a jury verdict, you can ask: What evidence led to that conclusion?

When a news story makes a dramatic claim, you can ask: How do they know?

When an influencer posts about a new scientific study, you can ask: What did the study actually find?

When reading a heated discussion on social media, you can ask: What observations support these opinions?

The underlying skill is the same. Scientific peer review is simply the most formal expression of a much broader habit of mind: evaluating claims against evidence.

In this sense, citizen peer review is not only about improving science. It is about developing the capacity to navigate modern life. The skills required to review a scientific paper are the same skills required to evaluate political claims, marketing messages, online arguments, health advice, financial recommendations, and public policy debates.

The scientific mindset is often misunderstood as something reserved for laboratories and professional researchers. In reality, it is a practical survival skill. It helps us distinguish observation from opinion, evidence from assertion, and confidence from certainty. It encourages curiosity instead of cynicism and questioning instead of blind acceptance.

A healthy society depends on citizens who can think this way. Democracies require voters who can evaluate competing claims. Markets require consumers who can assess information critically. Communities require people who can reason together despite disagreement. Science itself depends on a public capable of understanding and participating in the evaluation of evidence.

Citizen science peer review provides a simple way to practice these skills. By reviewing scientific papers, citizens learn habits of inquiry that extend far beyond the scientific literature. They learn to ask better questions, seek stronger evidence, recognize uncertainty, and make more informed judgments.

In an age defined by information overload, these abilities are no longer optional. They are among the most important skills a person can possess.

From the Black Box to the Glass House: The Evolution of Peer Review

For more than a century, science has relied on a system known as pre-publication peer review. Before a paper is published, journal editors send it to a small number of reviewers—typically two or three—who recommend whether it should be accepted, revised, or rejected.

The public is often led to believe that this process is the gold standard of scientific quality control. In reality, the system is deeply flawed.

Many scientists now openly acknowledge that traditional peer review is broken.

The first problem is that it is largely a black box. Most reviews are hidden from public view. Readers do not know who reviewed the paper, what criticisms were raised, what concerns were ignored, what disagreements occurred, or why editors ultimately made their decisions. The scientific community sees the final product but not the reasoning that led to its publication.

The second problem is that “blind review” is often not truly blind.

In many disciplines, reviewers can easily infer who wrote a paper from the topic, methods, citations, writing style, conference presentations, preprints, laboratory affiliations, or prior publications. In highly specialized fields with only a handful of active research groups, anonymity can be largely symbolic. Authors often know who their likely reviewers are, and reviewers often have a good idea who wrote the paper they are evaluating.

The third problem is that many fields have become increasingly opinion-driven. In principle, peer review is supposed to evaluate whether evidence supports claims. In practice, reviewers frequently evaluate whether a paper aligns with their preferred theories, methodologies, assumptions, paradigms, or interpretations. Researchers commonly receive contradictory reviews in which one reviewer recommends acceptance while another recommends rejection of the very same paper. This suggests that reviews are often evaluating more than evidence alone.

A fourth problem is concentration of power. A paper that may eventually influence public policy, education, medicine, technology, or public understanding can be accepted or rejected based on the opinions of only a few anonymous individuals. Scientific knowledge that affects millions of people can be filtered through the judgments of a remarkably small group.

Then there is the economics.

Much of scientific research is publicly funded through taxpayers, governments, universities, and charitable foundations. Scientists conduct the research. Scientists write the papers. Scientists review the papers. Scientists often serve as editors.

Yet many commercial publishers charge substantial subscription fees and paywalls to access that work.

The public pays for the science, pays for the infrastructure that supports the science, and pays the salaries of many of the scientists producing the science—only to be told they must pay again to read the results.

The result is a system in which a small number of publishing companies generate enormous revenues by controlling access to knowledge that was largely produced and reviewed without compensation. To many observers, this arrangement increasingly appears difficult to justify.

These criticisms do not mean traditional peer review has no value. Pre-publication review can identify obvious errors, improve clarity, and filter out some low-quality work. But it was designed for a world of printed journals, limited communication, and restricted participation.

That world no longer exists.

Today, we can publish instantly, communicate globally, and invite feedback from thousands rather than a handful of reviewers.

This is the promise of Post-Publication Peer Review (PPPR).

PPPR treats publication not as the end of review but as the beginning. Instead of relying primarily on two or three anonymous reviewers before publication, scientific claims remain open to examination, criticism, correction, replication, and discussion after publication.

The process becomes transparent rather than hidden.

The reviewers become visible rather than anonymous.

The conversation becomes ongoing rather than one-time.

Most importantly, scientific ideas are judged by the broader community rather than a small gatekeeping group.

The model is similar to how society evaluates almost everything else. Products are reviewed after people use them. Software is evaluated after deployment. Books are discussed after publication. Public policies are assessed after implementation.

There are some pros and cons of PPPR. Pros: Democratizes scientific research and feedback beyond a closed circle of reviewers. It is entirely transparent and allows for continuous correction. Cons: It can expose researchers to nonconstructive criticism “trolls,” or "social media mobs." Although this can be mitigated with things like ORCID# and reviewer rating features. 

Science is increasingly moving in the same direction.

Post-publication peer review does not eliminate expertise. Experts remain essential. Rather, PPPR recognizes that science improves when more people examine evidence, challenge assumptions, identify weaknesses, and propose alternative explanations. And there are new emerging leaders in PPPR such as arXiv, JoST, ScienceOpen, F1000Research, and PubPeer.

This includes professional scientists, practitioners, students, and informed citizens.

The future of science is unlikely to be more secrecy, more gatekeeping, and more black-box decision making.

The future is transparency.

The future is open dialogue.

The future is post-publication peer review.

Citizen science peer review is part of that future.

How to Review a Scientific Paper in 10 Minutes

Science advances when claims are tested by evidence, not by authority, status, or popularity. As a citizen reviewer, your job is simple: evaluate whether the authors did the Love Reality Loop™ — whether they observed something in the real world and connected their claims to that evidence.

Wrong Question	Right Question
“Do I agree with this?”	“Did the authors observe it, or are they just thinking it (i.e., is it merely the author’s opinion)?”

Science begins with observation. Opinion comes later. Peer review assesses the quality of the science and distinguishes it from the author’s opinion. It also keeps your opinion out of it.

The 5 Questions Every Reviewer Should Ask

Your review matters because science belongs to everyone. You do not need a PhD to ask good scientific questions. Your job is not to decide whether you agree with the authors. Your job is to decide whether the paper is supported by evidence. You only need to ask 5 questions

#	Question	Good Sign	Red Flag
1	What is the main claim?	The claim is clear and easy to find.	You can’t tell what the paper is trying to say.
2	What evidence is presented?	The paper provides observations, measurements, experiments, or data.	The paper mostly gives opinions or speculation.
3	Does the evidence support the claim?	The conclusion follows from the evidence.	The conclusion goes beyond what was observed.
4	Could someone repeat it?	The methods are explained clearly.	The methods are vague or missing.
5	Are limitations acknowledged?	Authors explain weaknesses and uncertainty.	Authors act as if their findings are absolute truth.

Or visually:

The questions map visually as follows. Your review essentially is a commentary on the “red” or “green” status of (1) the left and right part-whole systems and (2) the relationship of “fit or alignment” between them.

Or slightly more advanced is doing a similar questioning (red/green status) on each of the standardized elements of a scientific paper and their relationships

Is/Is Not List: Your Job as peer reviewer

Not Your Job	Your Job
❌ Decide whether you like the authors	✅ Identify unclear claims
❌ Decide whether you agree with the conclusion	✅ Identify missing evidence
❌ Judge the work based on politics or personal beliefs	✅ Point out leaps in logic
❌ Reject findings because they make you uncomfortable	✅ Ask for clarification
❌ Criticize the paper for not being the study you would have done	✅ Suggest improvements
❌ Review your opinions	✅ Explain your reasoning
These are opinions.	Science reviews evidence.

Example Reviews

Example	Focus	Review
✅ Good	Evidence	"The evidence generally supports the conclusion, but participant selection is unclear. Please provide additional methodological detail."
❌ Bad	Opinion	"I don't believe this conclusion. The authors should consider a completely different theory."

Before you submit, ask yourself: Did I review the evidence—or did I review my opinion? If you reviewed your opinion, don't submit the review. If you reviewed the evidence, you're doing science.