Health economic models are a potentially useful tool to help health decision makers navigate complexity, but can have significant limitations such as:

Mistakes: Errors, common in even relatively simple health economic models, become both more likely to occur and more difficult to detect as model complexity grows;
Poor transparency: the validity of model analyses can be difficult to adequately ascertain if it is not clear who contributed to the model, what assumptions they made, how model algorithms were implemented, how those algorithms were tested and what data they were applied to;
Contested legitimacy: the value judgments of the model development team (e.g. what types of question are most important for a model to address, what parts of the workings of the system of interest to represent and in what detail, what outcome variables to include, which stakeholders to consult, etc) may differ from those using or affected by model outputs;
Narrow applicability: a model might be too simple to adequately explore some problems and too complex to reliably address others and be hard to transfer beyond a very specific decision context (e.g. within a specific jurisdiction);
Limited interoperability: different approaches to model implementation, dissemination, ownership and reporting makes it more difficult for multiple models to be efficiently and safely combined;
Ease of misuse: in the absence of clear documentation and prominent caveats, a model can easily be applied to decision problems to which it is poorly suited (potentially supporting decisions with serious negative consequences);
Restricted access: a potential overcompensation for fear of model misuse is constructing high barriers to accessing model code and data - thus limiting model testing, reuse and refinement; and
Growing stale: health economic models are rarely updated, meaning they can lose validity with time (e.g. input data becomes less relevant, new better performing algorithms are not incorporated, sudden major changes in environment / epidemiology / policy / service system are not accounted for).

Reponse

To help address these issues, the ready4 software framework aims to support transparent, reusable and updatable model implementations using R.

2 - Concepts

A number of concepts are helpful to understand prior to reading ready4 documentation.

2.1 - Model

A health economic model is a conceptual, mathematical and computational representation of systems relating to human health that can be used to help solve economic problems.

A model is a simplified representation of a system of interest. In the way we use the term, we also mean that a model is:

abstract and general (i.e. largely free of non-modifiable data, including numeric values, that are assumption- or context- specific) and
a tool (i.e. a model can be used to help undertake an analysis, it is not the analysis itself).

If a model is used to help solve economic problems (e.g. those arising from scarcity) relating to health and healthcare it is a health economic model. Many health economic models are developed to inform a decision or set of decisions (e.g. relating to youth mental health policy and system design), in which case they can also be called a decision model.

Ideally, a health economic model should have three inter-related representations - conceptual, mathematical and computational.

Conceptual Model

A conceptual model refers to underlying theory and beliefs about a system of interest that can be described in words and pictures.

Mathematical Model

A mathematical model formalises a conceptual model as a set of equations.

Computational Model

A computational model implements the conceptual and mathematical models of a system of interest as computer code.

Computational models can take a modular approach to implementation.

2.2 - Module

Some computational models are implemented by combining self-contained, reusable components called “modules”.

A modular computational model is one that constructed from multiple self-contained components, called modules. The advantages of developing a modular model include:

each module can be independently re-used in other computational models;
a complex model can be developed iteratively, beginning with a simple representation that is easier to validate and then progressively expanding in scope and functionality through the addition of new modules, to be validated independently and jointly; and
long term and resource intensive model development is more feasible through combining multiple independently managed and financed modelling projects.

To ensure that all ready4 computational model modules can be safely and flexibly combined, each module is created from a template using authoring tools that support standardisation.

2.3 - Modelling project

A ready4 modelling project develops a computational model, adds data and runs analyses.

A ready4 modelling project will use the ready4 software framework to implement the three steps of:

Developing and validating a computational model;
Adding context-specific data to that computational model; and
Applying the computational model to the supplied data to undertake analyses.

The key components of each step are summarised here.

2.4 - Repeatable research

Some core concepts relating to repeatable research have multiple conflicting definitions - this is how we use them.

Although there is widespread support from the scientific community on the importance of repeatable research, the definition of key terms such as reproducibility and replicability can vary across disciplines and methodologies (e.g. the extent to which computational modelling is used). The meanings we intend when using these terms are described below.

Reproduction and Replication

The distinctions we make between reproduction and replication have been guided by the approach outlined in a report by the National Academies of Sciences, Engineering and Medicine. However, we have adapted their definitions slightly as the meanings in that report were framed in terms of study findings / outcomes, whereas our usage relates more to intended objectives when deploying tools.

Our usage

Reproduction

Applying the same analysis code to the same input data with the expectation of generating identical outputs (with the exception of trivial artefacts like datestamps for when analysis reports were produced).

Replication

Applying analysis code used in a study to new input data. The analysis code is reused with only minimal edits that are necessary to account for differences in input data paths and variable names and to study metadata (e.g. investigator names, sample descriptions). The new data can be real or fake, but will include the same structure and concepts / measures as those found in the original study’s dataset. If the new data is a sample from the same population as the original study, then the expectation when undertaking replications is for results across studies to be broadly similar.

Suggested policies

When publishing analysis code we suggest adopting the following rules of thumb:

If the data required to re-run a study analysis are publicly available (or declared by the analysis program itself), publish the code as a reproduction program (e.g. this program for creating a synthetic population).
If the data required to re-run a study analysis are not publicly available, we publish the replication version of the code. The replication version of the code may be configured to ingest a synthetic (fake) representation of the study dataset as with this utility mapping replication program. Details of the (minimal) steps required to revert the replication code to a version that can be used for reproduction purposes are typically embedded within the program itself.

2.5 - Transferability

Some models have the potential to be used in multiple contexts - but will often need adaptation for this to be appropriate.

It is common for discussions of scientific studies to consider the extent to which findings can be generalised (e.g. if a well conducted study concludes with high confidence that an intervention is cost-effective in Australia, is it valid to infer that it is likely to be cost-effective in the United Kingdom?). However, we are more interested in the transferability of computational models (e.g. the extent to which the data-structures and algorithms from a computational model developed for an Australian context can be used to explore similar topics in the United Kingdom). Our usage of the term “transferring” (and by extension “transferability”, “transferable”, “transfers”) reflects this motivation.

Transferring - our meaning

Adapting a computational model, in whole or in part, to extend the types of data and/or research questions to which it can be applied. The new types of data will possess some differences in structure and / or concepts from that to which the computational model had previously been applied and these differences may be why research questions need to be reformulated.

When we use the term transferring, we are typically referring to either (a) authoring or (b) using on of the following:

An analysis program (or sub-routine) that has been adapted from an executable from another study to account for differences in the input data / research question.
Inheriting data-structures and algorithms that selectively re-use, discard and replace elements of a study’s computational model based on an alternative use-case.
(Multi-purpose) function libraries that have been created by decomposing a study’s (single purpose) analysis program.

Examples

The scorz module library was originally developed to provide an R implementation of algorithms in other languages for scoring adolescent AQoL-6D health utility as part of a utility mapping study (which also used the analysis program mentioned above). Examples of all three approaches mentioned in the previous section can be seen by examining the documentation and source code of the scorz library:

Two vignette programs from the scorz library website score different utility instruments. The first program scores adolescent AQoL-6D health utility and acts as a template for the second, which has been modified to score EQ-5D health utility.
Inspecting those example programs shows that one of the key adaptations in the EQ-5D program is to use the ScorzEuroQol5 module instead of the ScorzAqol6Adol module. Both of these modules inherit from ScorzProfile. This arrangement means that all three modules share some features (in terms of both structure and algorithms) but selectively differ (e.g. aspects that are necessarily different for scoring different instruments).
The algorithms attached to each module from the scorz library are principally implemented by functions (the source code for which can be viewed here) that were created when decomposing an early draft of the above mentioned study algorithm. These functions are called by module methods (source code viewable here).

3 - Users

How you use and contribute to ready4 will depend on the type of work you do.

3.1 - Coders

Coders can use ready4 to enhance the impact and re-usability of their algorithms.

If you are a coder, you may be a data scientist or software engineer by training or are perhaps a quantitative researcher who spends a high proportion of their time writing code to undertake their work.

Role

The primary role of coders in ready4 modelling projects is to author modules that implement computational models.

Tools

The ready4 tools of most use to coders are the software framework libraries for authoring modules.

Benefits

ready4 provides an opportunity to write software that matters! Our aim is to help improve population health through empowering decision makers with better models. If you already write code for health-economic modelling projects, the ready4 software framework may help you enhance your impact (facilitating code re-use) and work-efficiency (through partial automation of code development and quality-assurance workflows).

Contributing to ready4

The types of contribution you can make to ready4 include:

3.2 - Modellers

Modellers can use ready4 to leverage the work of other modellers and to implement reproducible modelling analyses.

If you are a modeller, you are responsible for the overall implementation of a modelling study from initial conceptualisation through to analysis and reporting. You are likely to be an economist, epidemiologist or statistician and are probably reasonably comfortable with writing analysis scripts in statistical software (potentially including R), without necessarily being a coding wizard.

Role

The primary role of modellers in ready4 modelling projects is to use modules to undertake analyse as part of modelling projects.

Tools

The ready4 tools of most use to modellers are the software framework libraries for authoring model datasets and analyses and model module libraries for use in computational modelling.

Benefits of using ready4

We hope that ready4 can be of benefit to you by helping you to efficiently build on work by other modellers, to implement more reproducible workflows, and to share your work so that it can be reused.

Contributing to ready4

The types of contribution you can make to ready4 include:

3.3 - Planners

Planners can use ready4 decision aids to generate useful insights.

If you are an planner, you contribute to policy development or service planning that aims to improve population health. You probably value the role of modelling to inform your work, but are likely to rely on others to provide much of the technical expertise to implement computational models.

Role

The primary role of planners in ready4 modelling projects is co-design and use of models to support decision making.

Tools

The ready4 tools of most use to planners are user-interfaces that convert computational models into useful decision aids.

Benefits of using ready4

We hope that ready4 can provide you with transparent reusable and updatable decision support.

Contributing to ready4

The types of contribution you can make to ready4 include:

4 - Stakeholders

In addition to the main types of intended user, a number of other stakeholders can benefit from and contribute to ready4.

4.1 - Community members

Community members can help ensure that ready4 remains accountable for addressing topics of importance to them.

Community members have an important role to play in both the development of the overarching ready4 model and the applicability of ready4 to specific decision problems.

One of the main contributions that community members can make is to provide advice. To date, the advice we have elicited from community members has related to shaping the design and conduct of individual modelling projects. In the medium term, he hope to supplement these opportunities through growing a ready4 support community.

4.2 - Funders

ready4 provides funders with opportunities to improve the quality, breadth and accountability of supports provided to health policymakers and system planners.

There are six main types of funder that can provide cash and/or in kind support to ready4:

Grant making research bodies can support modelling project proposals submitted to their existing funding schemes. These types of funder could also consider making a number of changes to how they work including the assessment weightings and levels of financial support given to the reproducibility, replicability and transferability components of research proposals and initiating targeted calls for proposals to improve the transparency, reuse and maintenance of models to inform policy.
Government departments can support the development of ready4 as part of programs to enhance data analysis and modelling capability in youth mental health by providing support to develop core ready4 infrastructure (e.g. our software maintenance and community development priorities). When commissioning new modelling projects, governments could make providing open access to code and (to the greatest extent feasible, balancing confidentiality considerations) data a requirement of all applicants.
Youth mental health service commissioners can commission data analysis and modelling projects that develop novel decision aids and to apply existing ready4 modules to undertake new analyses.
Philanthropic donors can help accelerate our development and enhance our impact by supporting us to bring our existing in-development software to launch and to further extend the ready4 model.
Corporate sponsors can provide cash, expertise and free product licenses to support both our core open-source infrastructure and individual modelling projects.

4.3 - Researchers

Researchers can use ready4 to enhance the reproducibility, replicability and transferability of their work.

Researchers in multiple discipline enhance prior, current or planned future projects related to how environmental, social and technical systems shape health-economic outcomes by using ready4 to:

label and share non-confidential data (e.g. tables of summary statistics / parameter values) from prior and current work to extend access and impact;
working with modellers to undertake modelling analyses;
working with coders to redevelop study analysis scripts by authoring modules that are transferable to other contexts;
supervising student projects to assist with ready4’s current priorities; and/or
develop research proposals that incorporate a modelling project.

Researchers considering using ready4 should ensure they understand the development status of the tools they wish to use. If the required software is not yet a production release (a process we are working on!) we’d suggest only using it for testing or exploratory work that is not designed to inform decision making. All our software is free and open source so you don’t need to ask our permission to use it. We are however, very happy to discuss ideas for potential collaborations - contact the project lead to arrage a chat.

We also welcome advice from researchers about how we can make ready4 more relevant and useful.

5 - Purpose

The ready4 prototype software framework supports model implementations that meet explicit transparency, reusability and updatability criteria.

The ready4 software framework provides tools to help implement computational health economic models that are transparent, reusable and updatable (TRU).

Transparent

Model code and data are publicly available in online code repositories and data collections. Algorithms are documented and transparently and regularly tested. Model development occurs in the open and invites community participation, with each individual’s contribution publicly identifiable. Analyses are reproducible and replicable.

Reusable

Model modules and datasets originally developed in one modelling project can be independently reused in other projects. As they share a common framework, model modules can be combined in other models and analyses to address multiple topics. Code implementation paradigms facilitate the transfer of model modules for use in other decision contexts.

Updatable

Model code, data and analyses are versioned, with an ongoing program of making new and updated releases. Software is maintained, with opportunities for users and contributors to flag issues, request features and supply code contributions.

The criteria we use for assessing TRU attributes of computational models are described in more detail in this preprint, with a worked example of the application of these criteria provided in this article from PharmacoEconomics.

6 - Implementation

The ready4 prototype software framework is distributed as a collection of framework code libraries that support object-oriented and functional approaches to implementing modular and open source computational models.

6.1 - Paradigms

ready4 software is implemented using a combination of object-oriented and functional programming paradigms.

6.1.1 - Why ready4 is object oriented

ready4 uses an object oriented programming (OOP) paradigm to implement computational models.

This below section renders a vignette article from the ready4 library. You can use the following links to:

view the vignette on the library website (adds useful hyperlinks to code blocks)
view the source file from that article, and;
edit its contents (requires a GitHub account).

library(ready4)

Motivation

The practical utility and ease of use of computational health economic models are in part shaped by the choice of programming paradigm used to develop them. ready4 adopts an object oriented programming (OOP) paradigm which in practice means that the framework principally consists of classes (representations of data structures useful for modelling health systems) and methods (algorithms that can be applied to these data-structures to generate insight useful for policy-making). Adopting an OOP approach is particular useful for helping to make models transparent, reusable and updatable.

Implementation

Modular Computational Models

Two commonly noted features of OOP - encapsulation and inheritance are particularly useful when developing modular computational health economic models.

Encapsulation

Encapsulation allows us to define the data structures (“classes”) used in computational modelling projects in a manner that allows them to be safely combined. For example, assume there are two computational health economic models, one (A) focused on predicting the types and intensity of services used by individuals that present to health services and the other (B) that predicts outcomes for recipients of these services. It may be desirable to develop a new model (C) that combines A and B to model both service use and outcomes. Using encapsulated code allows all of the features and functionality of A to be made available to B in a manner that protects the integrity of A. Specifically, B can only interact with A using the algorithms (“methods”) that have been already defined for A.

Furthermore, if appropriately implemented, methods associated with a class will work with any combination of input values that can be encapsulated by that class - making computational models more transferable. For example, imagine a class (X) that is used to structure summary data relevant to health systems. Methods associated with X (e.g. a method to derive an unmet need statistic) can then applied to two instances of X - one containing data relevant to the Australian context and one with data from the UK context.

Inheritence

The examples highlighted in the previous section have some potential limitations. What if the developers of A didn’t define methods that would allow B to interact with it in the desired way? Or what if there are a number of differences between the Australian and UK system that need to be accounted for when transferring a method from the former to the latter? These types of issues can be addressed by another key feature of OOP - inheritance. Inheritance allows for a “child” class to be created from a “parent” class. By default, the “child” inherits all of the features of the “parent” including all methods associated with the “parent” class. Importantly however, alternative or additional features can also be specified for the “child” to allow it to implement different methods where necessary. For example, when developing our new computational model C we could create a number of new classes that are children of the classes defined in A. We can then define any additional/alternative methods for these classes that overcome any integration issues between the classes and methods of A and B. In this way, we can enjoy the best of both worlds - leveraging all relevant algorithms from A and B (as there is no need to re-invent the wheel), while ensuring that we transparently develop the additional code required for C. This approach also ensures that the respective contributions of the (potentially different) authorship teams behind A, B and C is clearer.

Similarly, inheritance would allow re-use of much of the code from a model of the Australian health system when exploring similar topics within the UK context, while making it straightforward to develop additional code that addresses relevant divergence in features between the two jurisdictions. In practical terms, this would mean developing two child classes of X - class Y for use with Australian data and class Z for use in the UK system. All methods that are not specific to a particular jurisdiction are defined for X and inherited by Y and Z. Methods that are only appropriate for use in the Australian context are defined for Y, while UK specific methods are defined for Z.

Transparent Computational Models

To make modelling analysis programs more readily understood, the ready4 package provides a simple and consistent syntax. Such simplified approaches are facilitated by two other commonly noted features of OOP - polymorphism and abstraction.

Polymorphism

Polymorphism allows for similar concepts to be represented using consistent syntax. The same top level code can therefore be generalised to multiple model implementations, making algorithms simpler to understand and easier to re-use.

Returning to a previous example, the exact same command (e.g. a call to the method exhibit) can be applied to both Y (used for Australian data) and Z (used for UK data). However, the algorithm implemented by that command can vary based on the class that each method is applied to (ie a different algorithm is applied when the data is specified as being from the UK compared to being specified as Australian).

Abstraction

The simplicity enabled by polymorphism is enhanced by Abstraction, which basically means that only the briefest and easiest to comprehend parts of the code are exposed by default to potential users. Once an instance of a model module template class is created, the entire program to ingest model data, analyse it and produce a scientific summary can be represented in a few brief lines of code, readily comprehensible to non-coders. When using open source languages, the elegance and simplicity of abstraction does not restrict the ability of more technically minded users exploring the detailed workings of the underpinning code.

6.1.2 - The role of functional programming in ready4 development

ready4 uses functional programming to maximise the re-usability of model algorithms.

Although the object-oriented programming (OOP) approach ready4 implements has many advantages, it can also have some limitations. Some of these limitations have been colorfully highlighted by a popular quote attributed to Joe Armstrong:

“The problem with object-oriented languages is they’ve got all this implicit environment that they carry around with them. You wanted a banana but what you got was a gorilla holding the banana and the entire jungle.”

In practical terms, this means that if not carefully planned, using OOP can create barriers to code-reuse as algorithms come bundled with artefacts of no/low relevance to many potential users. To help maximise the accessibility and re-usability of ready4 algorithms, these algorithms are primarily written using the functional programming paradigm. Only once an algorithm has been implemented using functions are they then linked to a data-structure by means of a calling method. The typical development workflow for a ready4 computational modelling project might therefore look something the following three step process:

A modelling study algorithm is initially implemented as a program.
To help transfer the methods used in the study algorithm, it is decomposed into functions, which are bundled as a code library (or libraries). The program is updated to use the newly authored functions.
A model module is authored from a template to define a data-structure. A method (or methods) that call the functions authored in the previous step is attached to the module(s) using the ready4 framework’s syntax. The new module is added to the previously created code library and the program is again updated so that the algorithm is now implemented by supplying data to the ready4 module and then calling the desired method(s).

Modellers using ready4 for the most part will only use ready4 modules and will rarely interact directly with the functions that implement module methods. However, these functions are potentially of significant usefulness to coders authoring new algorithms. All ready4 functions are created with minimal, but consistent documentation with the aid of tools from the ready4fun library.