This is an R Shiny App which facilitates calculations of the value of research proposals in a timely manner. The inputs required in the app represent the minimum needed to understand the consequences of uncertainty and the need for further research. Full details of the approach used and applied examples using these methods are forthcoming. In the meantime click here for further details.
Users unfamiliar with value of informaiton methods are encouraged to read the information in the 'How to estimate research value' tab. This section describes the value of information approach and how it applies to research funding in a resource constrainted health care system.
To carry out an analysis click the 'Inputs' tab
The following section outlines the principles of the assessments required when considering the need for additional evidence and the priority of proposed research. These assessments help inform the two questions which must be answered when considering whether to prioritize and commission research:
Health outcomes can be improved by conducting research or implementing the findings of existing research. In order to understand the value of additional evidence it is necessary to distinguish between these two very different ways to improve health outcomes. It is also necessary to take account of the costs associated with research projects and the fact that minimum changes in outcomes may need to be observed before clinical practice will change.How does conducting research improve health outcomes?
Additional evidence is valuable because it can improve patient outcomes by resolving existing uncertainty about the effectiveness of the interventions available. This helps inform treatment decisions for subsequent patients. For example, based on the balance of existing evidence a clinician might judge that a particular intervention is the most effective option, but there will be a chance that the other alternative interventions are in fact more effective. Therefore, when the existing evidence is uncertain there is a chance that one of the alternative interventions would have improved health outcomes to a greater extent. This means that there are adverse health consequences associated with uncertainty. A judgement about the level of uncertainty can come from a systematic review and meta-analysis, expert elicitation, extrapolation, meta-epidemiological study, or a combination of these sources. The level of uncertainty in the decision arises from the range of plausible values that the outcome can take. This is represented by the confidence interval (CI) or standard error around the estimate. A wide CI implies a large amount of uncertainty.
As an example, consider the evidence on the use of corticosteroids following traumatic brain injury (TBI) before the large definitive trial of CRASH. Before CRASH, a meta-analysis of 19 randomised controlled trials indicated that the effects of corticosteroids on death and disability were unclear. Taking death as an endpoint, we can start to understand the consequences of the uncertainty on mortality. The odds ratio for death was 0.93 in favour of the use of corticosteroids but with 95% CI from 0.71 to 1.18. This uncertainty means that every decision about the use of corticosteroids following TBI may not have been the most effective choice. In this case, there was a 74% chance that corticosteroids were effective and improved survival. However, there was a 26% chance that corticosteroids resulted in excess deaths per annum. This uncertainty can be translated into the consequences for patient outcomes, i.e. number of expected deaths per annum due to uncertainty, by combining the uncertain relative effect with an estimate of the baseline risk (derived from the control arms of the trials or from an alternative source) and multiplying by the incidence of TBI per year.
These expected health consequences can be interpreted as an estimate of the health benefits that could be gained each year if the uncertainty surrounding treatment choice could be resolved, i.e., it provides an expected upper bound on the health benefits of further research which would confirm whether corticosteroids following TBI increase deaths or reduce them. In effect, this is the value of reducing the 95% confidence interval to a single point. These potential expected benefits increase with the size of the patient population whose treatment choice can be informed by additional evidence and the time over which evidence about the effectiveness of these interventions is expected to be useful.How does implementing the findings of existing evidence improve health outcomes?
In addition to funding research, it is also possible to improve health outcomes by ensuring that the treatment option that is expected to be best based on the findings of existing evidence is implemented into clinical practice. In fact, the improvements in health outcomes from implementing the findings of the current evidence base (implementation value) may be greater than the potential improvements in health outcomes through conducting further research.What change in the primary endpoint is required to change practice?
The health benefits of conducting further research will only improve patient outcomes if the findings of the research change clinical practice. Again, it is important to recognise that there are many ways to influence implementation other than by conducting more research. However, concerns about implementation might influence research priority and the design of research. For example, if it is very unlikely that the findings of research will affect clinical practice and other mechanisms are unlikely to be effective at changing practice, then another area of research might be prioritised even though the expected benefits are smaller. Furthermore, if the research must demonstrate highly statistically significant results to be implemented this will influence design, cost and time taken for research to report. In some cases, larger clinical differences in effectiveness may be required before research would have an impact on practice. This will tend to reduce the potential benefits of further research as larger differences are less likely to be observed than smaller ones. The change in the primary endpoint required is called the minimum clinical difference (MCD)Research costs imposed on the health system
Carrying out research consumes resources in the general health system e.g. doctors, nurses, and pharmacists whose time commitments are moved away from general patient care and reallocated to research projects. The general health budget may also bear the costs of administrative staff and equipment that is needed to carry out the research. Another important type of cost that is often borne by the health budget is the acquisition costs of the health technologies under investigation (e.g. drug or device costs). These costs must be taken into account to comprehensively understand the expected health benefits of research projects.
Since research funding bodies have limited resources, not all research proposals can be funded and so the benefits of some research projects must be foregone in order to fund others. Quantitative estimates of the health benefits of research projects are required to compare the benefits of funded research to the foregone benefits of research not funded. These assessments can help to inform which research projects represent “best buys” for the research funder. They allow decision makers to compare value across proposals make explicit judgements about the trade-offs between different outcomes. If the potential health benefits of research are not in generic health outcomes (such as QALYs), implicit extrapolations are required to link changes in primary outcomes to compare health outcomes.
The primary outcome measure or endpoint captures the most important aspects of health outcome in the research.Express results in natural outcomes (e.g. heart attacks avoided) or in QALYs?
The benefits of research can be expressed in either natural outcomes or in Quality Adjusted Life Years (QALYs). Using QALYs requires more inputs but enables a comparison of the health benefits of further research and implementation efforts across diverse clinical areas.Is the outcome a benefit (e.g. cure) or a harm (e.g. heart attack)?
For natural outcomes: the value of additional research is expressed in terms of ‘benefits gained’ or ‘harms avoided’ depending on whether this outcome is a benefit or harm.Name of outcome e.g. heart attack
For natural outcomes: this will be used in reporting results.Do the treatment costs depend on the primary outcome?
For binary outcomes: in some cases treatment costs will depend on whether the primary outcome occurs or not. For example, if a treatment is used to prevent disease progression then it will cease to be used (and its cost will no longer be incurred) if the individual progresses.Number of possible states if the primary outcome does / does not occur (4 maximum)
For binary outcomes: the primary endpoint may be a scale or a composite outcome which is composed of a number of health states. If there are different levels of health and costs associated with these health states then these can be considered explicitly here. For example, if primary outcome is a composite of heart attack and stroke then there are two possible states if the primary outcome occurs and the costs and health consequences associated with these states must be explicitly considered.Conditional on the primary outcome occurring, what is the probability of being in this state?
For binary outcomes: if the primary outcome is composed of distinct health states, different proportions of individuals may be expected to enter these health states. For example, if primary outcome is a composite of heart attack and stroke then conditional on the primary outcome occurring 40% and 60% of these individuals may be expected to experience heart attack and stroke respectively.Patient time horizon / time in this state (years)?
For binary outcomes: if differential survival is considered then represents the expected survival time associated with each state. Otherwise this represents the patient time horizon considered for the decision i.e. how far into the future individual patient outcomes are modelled.What is the health utility associated with this state?
For binary outcomes: this is a number which represents the health related quality of life associated with a state.What are the disease related costs associated with this state?
For binary outcomes: these are the costs associated with a particular disease state, they do not include the costs of the treatment under consideration.What is the health utility associated with the pre-transition health state?
For survival outcomes: this is a number which represents the health related quality of life associated with the pre-transition state.What are the expected monthly disease related costs associated with the pre-transition health state?
For survival outcomes: in the same manner as for health utility changes in the expected survival must be linked to changes in disease related costs. These are the costs associated with a particular disease state, they do not include the costs of the treatment under consideration.By how much is a one unit increase in the primary outcome expected to increase/decrease the health state utility?
For continuous outcomes: the effect of changes from baseline on changes in health related quality of life (utility) will depend on the severity of the disease and range of the outcome measure. 'Mapping' studies which use statistical methods to estimate the effect of a unit change in a natural outcome on utility provide this link.By how much is a one unit increase in the primary outcome expected to increase/decrease monthly disease related costs?
For continuous outcomes: changes in the primary outcome may also be expected to result in changes in disease related costs. These are the costs associated with a particular disease state, they do not include the costs of the treatment under consideration.How long is the treatment effect expected to last? (months)
For continuous outcomes: the scale of the health gains and disease related costs associated with changes in the primary outcome will depend on the expected treatment effect duration. Estimates of treatment effect duration exist for few outcomes so in practice with will require expert opinion to inform this.
There may be a number of relevant treatment options for a given indication. This app currently allows for up to 4 options to be considered.Current level of utilisation (%)
Some estimate of the current level of utilisation of the interventions in clinical practice is required to establish the value of changing practice if the results of new research suggest a change. It can also be used to establish whether there is greater value from encouraging the implementation of what existing evidence suggests is the most effective intervention rather than conducting new research. The utilisation of all treatments must sum to 100%.Choose method of entering baseline probability of outcome
An estimate of event rate with the baseline treatment is required. This is used to obtain an estimate of the absolute effect of the interventions on the primary outcome by applying the relative measure of effect to the baseline risk. There are two options for entering this data. 1) Upper and lower 95% range: this may come from discussion with an expert and/or from a confidence interval reported in the literature. 2) Number of events and number at risk: this may come from an observational study or control arm of an RCTScale for relative effect
An estimate of the relative effectiveness of the intervention is required for the primary outcome, along with an estimate of its uncertainty. This can be expressed with a 95% confidence interval in terms of an odds ratio (binary), relative risk (binary), risk difference (binary), hazard ratio (survival) or mean difference (continuous).Minimum clinical difference (MCD)
Specifying a MCD required to change clinical practice is one way to incorporate differences in costs, adverse events or other considerations which are not captured in the primary endpoint. The MCD for a particular treatment is always defined relative to a current standard of care (baseline treatment). The MCD is defined as the 'the minimum improvement in the primary endpoint which is required for the new treatment to be considered worthwhile'. For example, a larger MCD may be required if the new treatment is more expensive than the current standard of care. How the MCD is expressed differs between binary, continuous and survival endpoints.
For binary outcomes: here Treatment costs are assumed to be the same for all individuals treated; regardless of health outcomes (see below). If treatment costs accrue over multiple years they should be discounted to present value.Treatment costs over patient time horizon if the primary outcome occurs / does not occur
For binary outcomes: in some cases treatment costs will depend on the primary outcome, for example; intensive preventative treatment may be administered continuously until an event occurs (e.g. heart attack). Once the event has occurred the preventative treatment is halted and these treatment costs are no longer incurred.Treatment costs per month
For continuous or survival outcomes: treatment costs incurred each month.Are individuals always treated until progression under this treatment?
For survival outcomes individuals may be treated until progression or there may be a maximum duration of treatment.
The value of research can be calculated for either full research (e.g a randomised controlled trial (RCT)) or feasibility studies. The inputs required for the analysis will depend on the type of study chosen.
For feasibility studies: the motivation is that there is uncertainty about whether a full trial is possible. If the feasibility study shows that the full trial is not possible, the research budget spent on the feasibility study will have no impact on health outcomes. For this reason, the likelihood of a feasibility study leading to the full trial is an important determinant of its value.Expected duration of research
Some assessment of the duration of time for the proposed research to be conducted and for the results of the research to report is required since the value of research declines the longer it takes to report. This might be informed by an assessment of sample size, recruitment rates, or historical experience from conducting similar types of studies. For feasibility studies: estimates of duration for both the feasibility study and the full trial are required.Costs of the research to funder
These costs are the costs of research which are directly borne by the research funder. For feasibility studies: estimates of research funder costs for both the feasibility study and the full trial are required.Costs of the research imposed on health system
These costs are the costs of research which fall on the general health system. Carrying out research consumes valuable resources from the general health care budget e.g. additional treatment costs and health professionals whose time commitments are moved away from general patient care and reallocated to research projects. For feasibility studies: estimates of research health system costs for both the feasibility study and the full trial are required.Length of time for which the new evidence would be valuable
The information generated by new research will not be valuable indefinitely because other changes occur over time. For example, over time new and more effective interventions become available which will eventually make those currently available obsolete. This means that new information about effectiveness is only relevant for a specific amount of time. A judgement about the length of time that the evidence from the proposed research might be valuable is required to estimate the expected benefits over an appropriate time horizon.Discount rate
Discounting should be used to reflect the fact that resources committed today could be invested at a real rate of return to free up more resources in the future.Incidence per annum
An estimate of the number of individuals facing the uncertain choice between alternative interventions is required in order to establish the size of the benefits to the target population.Opportunity cost of health system expenditure
Increasing treatment costs will be associated with health opportunity costs. These are the improvement in health that would have been possible if any additional resources required had, instead, been made available for other health care activities.
No baseline input required for continuous outcomes
Click once and go to the Results tab
Click once, and go to the Results tab. The calculation to reconsider the evidence can take up to 10 minutes to report.
This app was developed using R Shiny by David Glynn supervised by Claire Rothery and Karl Claxton. This research was funded by the UK National Institute for Health Research (NIHR) Health Technology Assessment (HTA) Programme as project number 16/29/01 and through a PhD studentship to David Glynn, funded by the Centre for Health Economics, University of York. The views expressed in this application are those of the authors and not necessarily those of the NIHR HTA Programme. Any errors are the responsibility of the authors.
The source code is available on GitHub at https://github.com/david-glynn.