A recent press release by the National Health and Medical Research Council (NHMRC) stated that where there was evidence to support the use of homeopathy the trials were too small and not well designed. Its kind of damning with faint praise isn’t it?? By the way most of the headlines were a lot less favourable but I read the detail of the report.

So I decided to write about research and try and explain what is contained in the report and perhaps shed a little light on the whole process. I did try and avoid a lot of maths in this discussion but this is going to be a little longer than my usual blog as it covers a fair amount of information.

Research – hierachy and different types

First up there is a hierachy in terms of research evidence in western medicine. At the top of the research hierachy is the randomised controlled trial (RCT) , which is regarded as the gold standard of evidence. Unfortunately it takes a lot of gold to meet the gold standard and it will always be more difficult with natural medicine which can’t really patent its findings to generate the types of revenues that fund this standard of research.

How much of western medicine is based on the gold standard of RCT’s? Surprisingly a lot less than you would think. Back in the 1980’s when the Cochrane Collection was established only 10- 15% of drugs in use had been subject to this research. The rest of the time they relied on long term clinical use (kind of like a lot of natural medicine). That number has at least doubled in part due to the efforts of the Cochrane Collection which is an independent research body promoting the use of evidence based research in clinical decision making.

Before you get to the RCT however there are other forms of research which contribute to the picture , this can include case study reports, in vitro research (cells in a test tube), animal studies and cohort studies, case control studies and clinical use. A case study report is usually a report which can highlight problems with an existing treatment or a way of dealing with a case which yields some new information. Studies on cells in test tubes or animals may be done at the start of research. They are  not considered as reliable as human trials as humans may react differently to animals.

Case control studies try to identify what is different between a group who have developed a condition and those who didn’t. They tend to be more cost effective but not always as reliable as a Cohort study where participants are tracked over a long period of time. Try remembering what you were eating like ten years ago? See? Its really not that easy. A Cohort study that tracks it over time can document what you were eating at different times in your life and see if the group who develop a condition were eating differently or eating more of a particular food group.

How do we design an RCT?

Whilst the RCT is the gold standard its fairly hard to find any RCT that is perfect. Typically an RCT is set up to compare one treatment against another or against placebo. You then recruit a group of people to your trial who need treatment for that condition – therefore you need to make sure they have the type of condition that will respond to your drug. To avoid complications you try to exclude people in the research who have other conditions as well. In the US students are often happy to participate in those trial’s as its a good way to earn some extra money.

So is that group of people you have just recruited to your study the same as the group of people who will ultimately take your drug? Quite often people as they get a little older are on multiple medications and they aren’t as robust as the group of students who did the trial.

So in your analysis you have just looked at the recruitment of participants and already you may not be that happy with the design.

See how easy it is to start to poke holes in the design of any trial and I haven’t even looked at numbers of participants yet?

Size of trial

To determine how many people you need in a trial you need to look at what the size of the treatment effect is and then a calculation is done (called a power calculation) to determine how many participants you need in your trial at the end to be able to see that the treatment effect is actually statistically significant.

This means a bigger group doesn’t add up to better research. It tends to mean that the smaller benefit someone gets from treatment the larger the group you need to show that it was  statistically significant and didn’t just occur by random chance. When newer drugs are being trialled against older drugs you often see only small improvements in the treatment so large scale trials will be needed.

Most natural medicine trials are quite small as funding is usually restricted however there are processes in place to ensure that the size of the group is adequate to measure the treatment effect. The NHMRC was generally negative about the fact that so few of the homeopathic trials had in excess of 500 participants but neglected to explain why that may not be necessary. They completely omitted the explanation of power calculations or whether in fact these indicated that smaller studies were still reliable evidence. Sometimes its not all about size!

Natural Medicine and RCT’s

In the next blog I will look at some of the reasons why natural medicine doesn’t fit well into the world of RCT’s. I was going to cover it here but its already a bit long!! Would love to hear your thoughts so please feel free to comment.