What about studies?

Potential Problems with Studies (June 1, 2023)

Churches may come across social and demographic studies that can be very helpful in developing church plans and programs.  However, if they haven't perused the actual study report, or trust someone who has, then the study may not be worth the paper it is printed on.  (Or, the bandwidth through which it is downloaded?)

There can be significant problems with studies that limit their accuracy.  First, the methodology used may be unable to scientifically get to the facts needed to support the study’s findings.  Examples of this are using a sample size that is too small or using a study group that is not sufficiently random.  Secondly, the study may not have sufficient methods to counter subconscious bias.  For example, consider drug trials.  Doctors, patients and lab technicians should not know whether or not the study participant is receiving real medicine or a placebo (referred to as a triple-blind approach).  Thirdly, it is a problem if the desired outcome of the study is previously known to the study group or to people who are being studied, resulting in overt or unconscious bias.  This is often inferred by knowing who is funding the study.

So, if you are not in a position to read the full study report and don’t have enough knowledge to assess the methodology used, how can you rely on any of them?  A few approaches come to mind.  

Was the study peer-reviewed?  Many studies, especially in university settings, are reviewed by members in the same field who were not involved in the study.  The study group has to respond to all pertinent questions and concerns expressed, and the study is not considered complete until the peer group approves it.  This is the gold-standard for trustworthiness in studies.  This only breaks down when the number of available peers are few and generally they all work closely together.  In such an environment, there can be a tendency for peers to go easy on each other, knowing that their own work will come up for review before long.

Was the study and its methodology reviewed by an entity that could be significantly impacted by the study?  I saw a case where a think-tank organization did a study comparing private sector wages to those of government employees.  I was very impressed to see a section in the report that described how the methodology and results were reviewed and found acceptable by the Policy Board of the Federal Government.  The Federal Government had no other apparent involvement in the study.

You can consider whether the study results are consistent with previous similar studies, and/or whether an independent group successfully repeated the study and had similar findings.  Finally, you can gain some comfort if the study was performed by a well known, prestigious organization with no known biases.

This is a lot of technical information and it generally will not be reported by those from whom we hear about study results, most notably the news media.  One exception is reporting by the more established professional journals.  If a reported study is important to you, you probably should try to trace it back to a source which honours the importance of describing the study processes used.  Using the Internet, this may not be too difficult.

I want to conclude with one problematic area relating to public reporting on study results. The media often has a desire to report on high-interest studies; often medical in nature.  Even though a news report might indicate that a very small participant group was used in the study, the news reporting will sometimes say that the study is ‘suggestive’ of a particular fact.  ‘Suggestive’ may be interesting, even entertaining, but it basically says that there is insufficient scientific basis for the study results being reported.  User beware.