General concepts of statistical trial design and methodology
Types of study
Different types of study designs can be used to answer a clinical research question. They are presented in Figure 1 and explained below.
- Experimental studies
- Experimental studies, commonly referred to as clinical trials, are studies in which a treatment (or procedure) is intentionally introduced by the researcher and a result (or outcome) is observed. The aim of these types of studies is to evaluate the effectiveness of a new treatment (or treatment regimen or procedure) in comparison with the current standard of care. These form the 'arms' of a trial, where Arm A might be the new treatment and Arm B the current standard (control) treatment. The allocation or assignment of individuals to a particular arm (treatment type) of a trial can be randomised. In this case we refer to the clinical trial as a Randomised Controlled Trial (RCT). If the allocation is not randomised, we talk about Non-Randomised Controlled Trials. The conduct of trials is commonly used to advance evidence based medicine through well designed protocols and rigorous evaluation (Vist et al., 2007).
- Randomised Controlled Trial (RCT)
- A Randomised Controlled Trial is a type of experimental study considered as gold standard in a clinical trial. Individuals who meet specific clinical trial inclusion and exclusion criteria, are randomly allocated to receive either the experimental treatment, or the control treatment. These individuals are followed for a specific length of time (e.g. until a certain event occurs or for a precise number of months/years) and then the outcomes of the two groups analysed and compared. ANZMTG encourages investigators to perform this type of trial when possible to avoid all main forms of bias that could be controlled by RCTs (see section "Clinical Trials" for more details).
Example of a randomised controlled trial sponsored by ANZMTG: ANZMTG 01.07 WBRTMel Trial
Whole Brain Radiotherapy Following Local Treatment of Intracranial Metastases of Melanoma (WBRTMel) is a phase III, multicentre study aiming to accrue 200 patients over five years with brain metastases who are randomised to receive whole brain radiotherapy after local treatment of the brain metastases, or they are observed (i.e. they receive no immediate further treatment). The objectives of the study are to determine the effect of adding WBRT to local treatment on distant intracranial control (primary endpoint), quality of life (QoL), performance status, neurocognitive function (NCF) and overall survival.
- Non-Randomised Controlled Trial
- A Non-Randomised Controlled Trial is an experimental study in which participants are assigned to different treatments by a method that is not random. This design is used when the act of random allocation can reduce the effectiveness of the intervention, when it would be unethical to perform random allocation or when it is impractical (cost or convenience factors). However, in non-randomised controlled trials confounders exist and conclusions must take into account these potential biases.
Example of a non-randomised controlled trial sponsored by ANZMTG: ANZMTG 02.14 CombiRT
CombiRT is a phase I/II, multicentre study aiming to accrue up to 30 patients with BRAF V600E/K mutation positive advanced stage melanoma over 12 months. The study will examine the safety and tolerability of combining dabrafenib, trametinib and RT for treatment of extracranial disease in patients with unresectable or metastatic melanoma.
- Observational studies
Observational studies are those in which patients are observed in their natural state (Bland et al., 2009). The researchers simply "observe" a group of patients without actually "doing" anything to the patients. Patients may be monitored overtime (e.g. for a change in the appearance of moles) and significant details recorded, but no intervention is introduced. The experimenter has no control of the control groups, and cannot randomise the allocation of patients. This can create bias, and can also mask cause and effect relationships.
Selecting the appropriate group of controls can be one of the most demanding aspects of an observational study. An important principle is that the distribution of exposure should be the same among observed patients and controls and meet the same inclusion criteria in the study. The clinician may also consider the control group to be an at risk population, with the potential to develop the outcome (Song and Chung, 2010).
Examples and more vocabulary are presented in Mann (2003).
Example of a fictional observational study
There are different types of observational studies (Gay,1998) such as cohort studies, case-control studies, cross sectional studies or ecological studies.
Selecting the right study design to answer a scientific question is dependent upon the nature of the question being asked and is influenced by theoretical and practical considerations such as what literature and data already exists around the subject. Questions can pertain to a variety of central clinical issues such as therapy, diagnosis or prognosis. For a "therapy" clinic issue, RCTs are the most often used. When the clinical issue is the “prognosis”, prospective cohort studies are the preferred ones (see section "Types of research study"). A range of factors such as resources, feasibility and ethical considerations can also influence the choice of study design. More details are given in Bragge (2010), and information about how to choose the right study design is explained in the hierarchy of the Australian National Health and Medical Research Council (Coleman et al., 2008).
Types of research study
Several kinds of research studies exist (Gay, 1998) and the two following types of trial are commonly conducted by ANZMTG:
- Prospective Study
- A prospective study is a cohort study that usually consists in taking a cohort of patients and watching them for the events of interest over a long period. For example, one might follow a cohort of middle-aged women who vary in terms of sun exposure and sunscreen protection, to test the hypothesis that the 20 year incidence rate of skin cancer will be highest among highly exposed women without sun protection, followed by highly exposed women with sun protection and then non exposed women.
- Retrospective Study
Retrospective studies are conceived after patients have already developed the outcomes of interest and data are generated from historical records and from recall. For example, one might follow a cohort of patients who have developed an early stage melanoma and review their family history, sun exposure and sun bed use to ascertain if any of those factors contribute to melanoma development. Most sources of error due to confounding and bias are more common in retrospective studies than in prospective studies.
Some other types of studies exist such as explanatory studies (to make causal inferences about the nature of relationships between risk factors and outcomes), descriptive studies (to describe the distribution of variables in a group), contemporary comparison (to compare two groups experiencing the treatment at the same time) or historical comparison (to compare the same group at different times).
Bias and confounding effects
Bias is the lack of neutrality that leads to the deviation from the truth. Observational studies are more subject than experimental study designs to a number of potential problems that may bias their results. The two main types of bias are selection bias and information bias (Jepsen, 2004).
- Selection bias
- Selection bias occurs when participants are selected for an intervention on the basis of a variable that is associated with outcome (it is more common in case control studies than in cohort studies). For example, if in a trial looking at a new treatment in comparison with a standard treatment, patients with overall better health were chosen to be on the new treatment arm instead of being randomly being allocated between the two arms, there is potential for the new treatment to look more effective than it really is. Randomisation or other similar methods abolishes this selection bias.
- Information bias
- Information bias is erroneous study data due to invalid or imprecise study measures (is the variable measuring what it is supposed to measure?), problem in sensitivity or sensibility (ability to identify patients with and without the disease), etc. Another type of information bias is the interviewer bias. This can occur in case-control studies for example when interviewers conduct interviews differently for cases and controls if they know the case status of the patient. One way to avoid that is to blind interviewers to patient status when possible (blinding process).
Another issue are potential dropouts or missing values. Dropouts can be related to the duration of the study or toxicity of the drug for example and can't be ignored. Treatment duration in clinical trials has an impact on the evaluation of the study drug. An inadequate duration of treatment may not provide an unbiased response rate of the study drug. In practice, more patients are enrolled to account for this potential loss of data so that the desired power of the trial can still be achieved (see Sample size and power calculation of a trial)
There are other issues that can produce a bias in studies. For example, the comparative effect is important and having a control group allows knowing if whether any improvement is due to the drug or just the act of being treated (placebo effect).
- Confounding bias
- The aim of an observational study is to examine the effect of the exposure, but sometimes the apparent effect of the exposure is actually the effect of another characteristic, which is associated with the exposure and with the outcome. This other characteristic is called a confounder (Szklo & Nieto, 2000).
Example of a fictional confounder
Sun exposure is a known risk factor for melanoma and skin cancer the endpoint here. As sun exposure is also associated with the type of work (e.g. working outdoors, trucks drivers, etc), this gives the false impression that type of work and skin cancer are associated.
There are two principal ways to reduce confounding in observational studies: prevention in the design phase by restriction or matching, and adjustment in the statistical analyses by either stratification or multi-variable techniques. The most common stratification factors are age, sex, recruitment center (in the case of multicenter studies) and prognostic factors.
Sometimes in observational studies, and more specifically in case-control studies, cases and controls are matched (Rubin, 1973). For each case, one or more controls are found that have the same values in a set of matching variables. Matching variables typically include age and sex. It is hoped that by matching, any differences between cases and controls are not a result of differences between groups in the matching variables. The main purpose of matching is to control for confounding.
In an RCT, the randomisation process allows the investigator to assume that not only known, but also unknown, potential confounders are distributed evenly among the exposed and the unexposed.
- Altman D.G. (1991) Practical Statistics for Medical Research. Chapman and Hall, London.
- Bland, J. M. et al. (2009) Statistics guide for research grant applicants. St. George's Hospital Medical School, London, 41.
- Bonita R. et al. (2007) Basic epidemiology. 2nd ed. New York: World Health Organisation.
- Bragge, P. (2010) Asking good clinical research questions and choosing the right study design. Injury, 41, S3-S6.
- Breslow N.E. and Day N.E. (1980) Statistical Methods in Cancer Research: Volume I - The analysis of case-control studies. IARC Scientific Publications No. 32, Lyon.
- Breslow N.E. and Day N.E. (1987) Statistical Methods in Cancer Research: Volume II - The design and analysis of cohort studies. IARC Scientific Publications No. 82, Lyon.
- Coleman K. et al. (2008) NHMRC additional levels of evidence and grade for recommendations for developers of guidelines: Stage 2 consultation. NHMRC (National Health and Medical Research Council).
- Gay, J. (1998) Clinical epidemiology & evidence-based medicine glossary: clinical study design and methods terminology. Pullman WA: College of Veterinary Medicine. Washington State University.
- Mann, C. J. (2003) Observational research methods. Research design II: cohort, cross sectional, and case-control studies. Emergency Medicine Journal, 20(1), 54-60.
- Morgenstern H. (2008) Ecological studies. In: Modern Epidemiology. 3rd Edition. Editors: Rothman, Greenland, Lash. Lippincott Williams and Wilkins.
- Pocock, S.J. (1983) Clinical trials: a practical approach. Chichester: John Wiley.
- Rubin, D.B. (1973) "Matching to Remove Bias in Observational Studies". Biometrics, 29(1):159-183.
- Szklo M., Nieto F.J. (2000) Epidemiology: beyond the basics, 1st ed. Gaithersburg, Maryland: Aspen Publishers.
- Vist G.E. et al. (2007) Outcomes of patients who participate in randomised controlled trials compared to similar patients receiving similar interventions who do not participate. Cochrane Database Syst Rev 2007;(2):MR000009.
Clinical trials (Experimental studies)
Inclusion and exclusion criteria
In a clinical trial, the researchers must specify inclusion and exclusion criteria (that can be presented as a list) for participation in the study. For example age, sex, stage of disease, previous treatment history, may all have certain parameters that have to be met for an individual to be deemed suitable for study participation (see Bland et al., 2009).
In clinical trials a bias in patient's response can occur if the patient knows which treatment he is receiving. Furthermore, the clinician can be affected by knowing which treatment a patient is receiving (Day & Altman, 2000). For these reasons it is preferable that neither the patient nor the clinician knows which treatment is allocated to the patient. This is a double blind design. Sometimes, when it is impossible for one party to be unaware of the treatment, the trial is single blind (in most cases it is the patient who is unaware of the treatment).
In the case of a double blind trial where comparing a new treatment with a current treatment is not feasible (because the clinician/patient might be aware of obvious differences between the treatments), a placebo must be introduced. This would be a treatment that looks like and is given in exactly the same way as the experimental treatment. This makes the two treatments indistinguishable and prevents psychological effects whereby a patient's condition improves because he knows he is receiving a particular treatment. For further explanation see Schulz & Grimes (2002).
Randomisation is a method of dividing patients into groups in such a way that the characteristics of the patient do not affect the group to which they are allocated. Each patient is equally likely to be allocated to any of the available groups and any differences between these groups happen by chance (see Altman, 1999a and Altman, 1999b). Randomisation is preferred in clinical trials:
- to be able to conclude that any differences that we observe between the treatment groups are due to differences in the treatments alone and not due to differences between the patients themselves
- to facilitate the concealment of the type of treatment from the researchers and patients to further reduce bias in treatment comparison
- to lead to treatment groups which are random samples of the population sampled and thus makes valid the use of standard statistical tests based on probability
An ideal randomisation procedure would maximise statistical power (generally, equal group sizes maximise statistical power), minimise selection bias (a good randomisation procedure will be unpredictable so that investigators cannot guess the next patient's group assignment based on prior treatment assignments) and minimise confounding. Performing an RCT is beneficial over other trial types because it meets these ideals. Different types of randomisation exist and the most often used are given in the Figure 2.
Illustrations of these randomization methods are presented in Kang et al. (2008) and online randomization software are introduced and discussed in Suresh (2011).
Considering an appropriate design for a trial is important during the initial development of a study protocol because it impacts upon the type of data analysis performed and therefore upon the results. Different trial designs exist and are summarised below (for more explanations, illustrations and examples see Chow & Liu, 2013).
- Parallel design
- Compare treatments A, B and control by dividing patients at random amongst the three groups (each patient receives only one treatment, see figure 3). In an example such as this where more than two groups are considered, One-way ANOVA or Kruskal-Wallis tests can be used to compare groups. If two groups are considered, statistical tests such as t-tests or Mann-Whitney tests can be performed.
- Cross-over design
- Patients receive all treatments, but not necessarily in the same order (randomised). In this trial the patient is his own control which reduces the confounding covariates (see figure 3). There may be a carry over of treatment effect from one period to the next and so it may be necessary to have a 'wash-out' period (no treatment at all) between the two treatments. This kind of design is not suitable if more than 3 treatments have to be compared.
- Sequential design
- Patients are entered into the trial in pairs; one receives treatment A and the other treatment B (allocated at random). The trial continues until either there is a clear benefit of one treatment or it becomes evident that no difference is likely to emerge. Data are analysed after each patient's results are available. The main advantage is that as soon as one treatment is shown to be superior, then the trial is stopped. However some drawbacks have to be taken into account such as drop-outs which can cause difficulties and a constant surveillance which is necessary.
- Factorial design
- Factorial design is used to investigate the effects of 2 or more treatments (factors) by allowing patients to receive combinations of treatments. A treatment could be either a single therapy or a combination of interventions. In this type of design treatments are varied systematically (i.e. some groups receive more than one treatment), and the experimental groups are arranged in a way that permits testing whether or not the treatments interact with one another (Piantadosi, 2013). However it is not always applicable because of drug interaction problems.
- Altman D.G. and Bland J.M. (1999a) Statistics notes: treatment allocation in controlled trials. Why randomise? British Medical Journal, 318(7192):1209.
- Altman D.G. and Bland J.M. (1999b) How to randomise. British Medical Journal, 319(7211),703-704.
- Bland, J. M. et al. (2009) Statistics guide for research grant applicants. St. George's Hospital Medical School, London, 41.
- Chow, S.C. and Liu, J.P. (2013) Design and analysis of clinical trials: concepts and methodologies (Vol. 979). John Wiley & Sons.
- Day, S.J. and Altman, D.G. (2000) Blinding in clinical trials and other studies. British Medical Journal, 321(7259), 504.
- Kang, M., Ragan, B.G. and Park, J.H. (2008) Issues in outcomes research: an overview of randomization techniques for clinical trials. Journal of athletic training, 43(2), 215.
- Schulz, K.F. and Grimes, D.A. (2002) Blinding in randomised trials: hiding who got what. The Lancet, 359(9307), 696-700.
- Suresh, K.P. (2011) An overview of randomization techniques: An unbiased assessment of outcome in clinical research. Journal of human reproductive sciences, 4(1), 8.