This is the multi-page printable view of this section. Click here to print.

Return to the regular view of this page.

Documentation

1 - Overview

How ready4 aims to improve the transparency, reusability and updatability of health economic models.

What is ready4?

ready4 is a prototype software framework for implementing transparent, reusable and updatable health economic models in R. The project is open source and is led by researchers at Monash University.

What is a software framework?

A software framework is a shared technology that can be used by multiple teams to collaboratively author software.

What is a computational model?

A computational model is a simplified representation of a system of interest that is implemented in computer code.

What makes a ready4 model modular?

The paradigms we use for computational model development allow individual model components (modules) to be independently reused (in other models) and safely and flexibly combined (to model more extensive systems).

Why is that useful?

We hope that ready4 will help to share data, improve the replicability and transferability of modelling analyses and generate valuable insights for health policymakers and system planners.

Who is it for?

ready4 is designed to be primarily used by coders and modellers working collaboratively on computational modelling projects in which other stakeholders such as funders and researchers make essential contributions.

What is it being used for?

Currently, ready4 is being applied to develop and apply a modular model of youth mental health.

Can I use it?

ready4 is publicly available and free for you to assess (to verify and validate), apply (to use and/or author health economic model modules) and to derive your own derivative works from (to leverage and enhance the work of others) under liberal terms of use.

Why is it a prototype?

Currently the ready4 software suite is only available in the form of development releases. That means ready4 may require more development, documentation and testing before it could be confidently used for scientific purposes other than the specific studies to which the ready4 development team have already applied it.

Why is it called ready4?

ready4 is short-hand for “readyforwhatsnext”, the title of the research project that catalysed the framework and a reference to how computational models can improve the preparedness of decision makers and system planners.

Where should I go next?

We’d recommend reading the documentation in the order in which sections appear in the table of contents (so go next to Examples, then to Getting started and so on). A scientific manuscript is also available.

2 - Examples

See how ready4 has been applied to model real world decision problems in youth mental health.

An scientific summary of the ready4 prototype software framework is available as a pre-print manuscript.

Currently we are using ready4 to implement readyforwhatsnext - a modular and open source economic model of youth mental health.

3 - Getting started

What you need to know to start using ready4.

3.1 - Concepts

A number of concepts are helpful to understand prior to using ready4.

3.1.1 - Model

A health economic model is a conceptual, mathematical and computational representation of systems relating to human health that can be used to help solve economic problems.

A model is a simplified representation of a system of interest. In the way we use the term, we also mean that a model is:

  • abstract and general (i.e. largely free of non-modifiable data, including numeric values, that are assumption- or context- specific) and
  • a tool (i.e. a model can be used to help undertake an analysis, it is not the analysis itself).

If a model is used to help solve economic problems (e.g. those arising from scarcity) relating to health and healthcare it is a health economic model. Many health economic models are developed to inform a decision or set of decisions (e.g. relating to youth mental health policy and system design), in which case they can also be called a decision model.

Ideally, a health economic model should have three inter-related representations - conceptual, mathematical and computational.

Conceptual Model

A conceptual model refers to underlying theory and beliefs about a system of interest that can be described in words and pictures.

Mathematical Model

A mathematical model formalises a conceptual model as a set of equations.

Computational Model

A computational model implements the conceptual and mathematical models of a system of interest as computer code.

Computational models can take a modular approach to implementation.

3.1.2 - Module

Some computational models are implemented by combining self-contained, reusable components called “modules”.

A modular computational model is one that constructed from multiple self-contained components, called modules. The advantages of developing a modular model include:

To ensure that all ready4 computational model modules can be safely and flexibly combined, each module is created from a template using authoring tools that support standardisation.

3.1.3 - Modelling project

A ready4 modelling project develops a computational model, adds data and runs analyses.

As a complex, collaborative and long-term undertaking, it is not feasible for ready4 to be financed by a single funder or progressed as a single project. Instead, our mode of development is via multiple independent modelling projects, each with their own project governance and funding.

A ready4 modelling project will use the ready4 software framework to implement the three steps of:

  • Developing and validating a computational model;

  • Adding context-specific data to that computational model; and

  • Applying the computational model to the supplied data to undertake analyses.

The key components of each step are summarised here.

3.1.4 - Reproducible research

Some core concepts relating to reproducible research have multiple conflicting definitions - this is how we use them.

Although there is widespread support from the scientific community on the importance of reproducible research, the definition of key terms such as reproducibility and replicability can vary across disciplines and methodologies (e.g. the extent to which computational modelling is used). In some cases, entirely different terms (e.g. repeatability) are preferred. The meanings we intend when using these terms are described below.

Reproduction and Replication

The distinctions we make between reproduction and replication have been guided by the approach outlined in a report by the National Academies of Sciences, Engineering and Medicine. However, we have adapted their definitions slightly as the meanings in that report were framed in terms of study findings / outcomes, whereas our usage relates more to intended objectives when deploying tools.

Meanings

Reproduction

Applying the same analysis code to the same input data with the expectation of generating identical outputs (with the exception of trivial artefacts like datestamps for when analysis reports were produced).

Replication

Applying analysis code used in a study to new input data. The analysis code is reused with only minimal edits that are necessary to account for differences in input data paths and variable names and to study metadata (e.g. investigator names, sample descriptions). The new data can be real or fake, but will include the same structure and concepts / measures as those found in the original study’s dataset. If the new data is a sample from the same population as the original study, then the expectation when undertaking replications is for results across studies to be broadly similar.

Examples

Examples of both reproduction and replication code are available. When publishing analysis code we try to adopt (there are exceptions) the following rules of thumb:

  1. If the data required to re-run a study analysis are publicly available (or declared by the analysis program itself), then we publish the code as a reproduction program (e.g. this program for creating a synthetic population).

  2. If the data required to re-run a study analysis are not publicly available, we publish the replication version of the code. The replication version of the code may be configured to ingest a synthetic (fake) representation of the study dataset as with this utility mapping replication program. Details of the (minimal) steps required to revert the replication code to a version that can be used for reproduction purposes are typically embedded within the program itself.

3.1.5 - Transferability

Some models have the potential to be used in multiple contexts - but will often need adaptation for this to be appropriate.

It is common for discussions of scientific studies to consider the extent to which findings can be generalised (e.g. if a well conducted study concludes with high confidence that an intervention is cost-effective in Australia, is it valid to infer that it is likely to be cost-effective in the United Kingdom?). However, we are more interested in the transferability of computational models (e.g. the extent to which the data-structures and algorithms from a computational model developed for an Australian context can be used to explore similar topics in the United Kingdom). Our usage of the term “transferring” (and by extension “transferability”, “transferable”, “transfers”) reflects this motivation.

Transferring - our meaning

Adapting a computational model, in whole or in part, to extend the types of data and/or research questions to which it can be applied. The new types of data will possess some differences in structure and / or concepts from that to which the computational model had previously been applied and these differences may be why research questions need to be reformulated.

When we use the term transferring, we are typically referring to either (a) authoring or (b) using on of the following:

  1. An analysis program (or sub-routine) that has been adapted from an executable from another study to account for differences in the input data / research question.

  2. Inheriting data-structures and algorithms that selectively re-use, discard and replace elements of a study’s computational model based on an alternative use-case.

  3. (Multi-purpose) function libraries that have been created by decomposing a study’s (single purpose) analysis program.

Examples

The scorz module library was originally developed to provide an R implementation of algorithms in other languages for scoring adolescent AQoL-6D health utility as part of a utility mapping study (which also used the analysis program mentioned above). Examples of all three approaches mentioned in the previous section can be seen by examining the documentation and source code of the scorz library:

  1. Two vignette programs from the scorz library website score different utility instruments. The first program scores adolescent AQoL-6D health utility and acts as a template for the second, which has been modified to score EQ-5D health utility.

  2. Inspecting those example programs shows that one of the key adaptations in the EQ-5D program is to use the ScorzEuroQol5 module instead of the ScorzAqol6Adol module. Both of these modules inherit from ScorzProfile. This arrangement means that all three modules share some features (in terms of both structure and algorithms) but selectively differ (e.g. aspects that are necessarily different for scoring different instruments).

  3. The algorithms attached to each module from the scorz library are principally implemented by functions (the source code for which can be viewed here) that were created when decomposing an early draft of the above mentioned study algorithm. These functions are called by module methods (source code viewable here).

3.2 - Motivation

To be transparent, reusable and updatable, ready4 is being implemented as a modular and open source computational model.

Problem

Health economic models are a potentially useful tool to help health decision makers navigate complexity, but can have significant limitations such as:

  • Mistakes: Errors, common in even relatively simple health economic models, become both more likely to occur and more difficult to detect as model complexity grows;

  • Poor transparency: the validity of model analyses can be difficult to adequately ascertain if it is not clear who contributed to the model, what assumptions they made, how model algorithms were implemented, how those algorithms were tested and what data they were applied to;

  • Contested legitimacy: the value judgments of the model development team (e.g. what types of question are most important for a model to address, what parts of the workings of the system of interest to represent and in what detail, what outcome variables to include, which stakeholders to consult, etc) may differ from those using or affected by model outputs;

  • Narrow applicability: a model might be too simple to adequately explore some problems and too complex to reliably address others and be hard to transfer beyond a very specific decision context (e.g. within a specific jurisdiction);

  • Limited interoperability: different approaches to model implementation, dissemination, ownership and reporting makes it more difficult for multiple models to be efficiently and safely combined;

  • Ease of misuse: in the absence of clear documentation and prominent caveats, a model can easily be applied to decision problems to which it is poorly suited (potentially supporting decisions with serious negative consequences);

  • Restricted access: a potential overcompensation for fear of model misuse is constructing high barriers to accessing model code and data - thus limiting model testing, reuse and refinement; and

  • Growing stale: health economic models are rarely updated, meaning they can lose validity with time (e.g. input data becomes less relevant, new better performing algorithms are not incorporated, sudden major changes in environment / epidemiology / policy / service system are not accounted for).

Reponse

To help address these issues, the ready4 software framework aims to support transparent, reusable and updatable model implementations using R.

3.3 - Users

How you use and contribute to ready4 will depend on the type of work you do.

3.3.1 - Coders

Coders can use ready4 to enhance the impact and re-usability of their algorithms.

If you are a coder, you may be a data scientist or software engineer by training or are perhaps a quantitative researcher who spends a high proportion of their time writing code to undertake their work.

Role

The primary role of coders in ready4 modelling projects is to author modules that implement computational models.

Tools

The ready4 tools of most use to coders are the software framework libraries for authoring modules.

Benefits

ready4 provides an opportunity to write software that matters! Our aim is to help improve population health through empowering decision makers with better models. If you already write code for health-economic modelling projects, the ready4 software framework may help you enhance your impact (facilitating code re-use) and work-efficiency (through partial automation of code development and quality-assurance workflows).

Contributing to ready4

The types of contribution you can make to ready4 include:

3.3.2 - Modellers

Modellers can use ready4 to leverage the work of other modellers and to implement reproducible modelling analyses.

If you are a modeller, you are responsible for the overall implementation of a modelling study from initial conceptualisation through to analysis and reporting. You are likely to be an economist, epidemiologist or statistician and are probably reasonably comfortable with writing analysis scripts in statistical software (potentially including R), without necessarily being a coding wizard.

Role

The primary role of modellers in ready4 modelling projects is to use modules to undertake analyse as part of modelling projects.

Tools

The ready4 tools of most use to modellers are the software framework libraries for authoring model datasets and analyses and model module libraries for use in computational modelling.

Benefits of using ready4

We hope that ready4 can be of benefit to you by helping you to efficiently build on work by other modellers, to implement more reproducible workflows, and to share your work so that it can be reused.

Contributing to ready4

The types of contribution you can make to ready4 include:

3.3.3 - Planners

Planners can use ready4 decision aids to generate useful insights.

If you are an planner, you contribute to policy development or service planning that aims to improve population health. You probably value the role of modelling to inform your work, but are likely to rely on others to provide much of the technical expertise to implement computational models.

Role

The primary role of planners in ready4 modelling projects is co-design and use of models to support decision making.

Tools

The ready4 tools of most use to planners are user-interfaces that convert computational models into useful decision aids.

Benefits of using ready4

We hope that ready4 can provide you with transparent reusable and updatable decision support.

Contributing to ready4

The types of contribution you can make to ready4 include:

3.4 - Stakeholders

In addition to the main types of intended user, a number of other stakeholders can benefit from and contribute to ready4.

3.4.1 - Community members

Community members can help ensure that ready4 remains accountable for addressing topics of importance to them.

Community members have an important role to play in both the development of the overarching ready4 model and the applicability of ready4 to specific decision problems.

One of the main contributions that community members can make is to provide advice. To date, the advice we have elicited from community members has related to shaping the design and conduct of individual modelling projects. In the medium term, he hope to supplement these opportunities through growing a ready4 support community.

3.4.2 - Funders

ready4 provides funders with opportunities to improve the quality, breadth and accountability of supports provided to health policymakers and system planners.

There are six main types of funder that can provide cash and/or in kind support to ready4:

  1. Grant making research bodies can support modelling project proposals submitted to their existing funding schemes. These types of funder could also consider making a number of changes to how they work including the assessment weightings and levels of financial support given to the reproducibility, replicability and transferability components of research proposals and initiating targeted calls for proposals to improve the transparency, reuse and maintenance of models to inform policy.

  2. Government departments can support the development of ready4 as part of programs to enhance data analysis and modelling capability in youth mental health by providing support to develop core ready4 infrastructure (e.g. our software maintenance and community development priorities). When commissioning new modelling projects, governments could make providing open access to code and (to the greatest extent feasible, balancing confidentiality considerations) data a requirement of all applicants.

  3. Youth mental health service commissioners can commission data analysis and modelling projects that develop novel decision aids and to apply existing ready4 modules to undertake new analyses.

  4. Philanthropic donors can help accelerate our development and enhance our impact by supporting us to bring our existing in-development software to launch and to further extend the ready4 model.

  5. Corporate sponsors can provide cash, expertise and free product licenses to support both our core open-source infrastructure and individual modelling projects.

3.4.3 - Researchers

Researchers can use ready4 to enhance the reproducibility, replicability and transferability of their work.

Researchers in multiple discipline enhance prior, current or planned future projects related to how environmental, social and technical systems shape health-economic outcomes by using ready4 to:

Researchers considering using ready4 should ensure they understand the development status of the tools they wish to use. If the required software is not yet a production release (a process we are working on!) we’d suggest only using it for testing or exploratory work that is not designed to inform decision making. All our software is free and open source so you don’t need to ask our permission to use it. We are however, very happy to discuss ideas for potential collaborations - contact the project lead to arrage a chat.

We also welcome advice from researchers about how we can make ready4 more relevant and useful.

3.5 - Purpose

The ready4 prototype software framework supports computational health economic model implementation that meet explicit transparency, reusability and updatability criteria.

Transparent, reusable and updatable (TRU) computational health economic models are signifiers of ethical modelling practice. The ready4 software framework provides tools to help implement TRU models.

Transparent

Model code and data are publicly available in online code repositories and data collections. Algorithms are documented and transparently and regularly tested. Model development occurs in the open and invites community participation, with each individual’s contribution publicly identifiable. Analyses are reproducible and replicable.

Reusable

Model modules and datasets originally developed in one modelling project can be independently reused in other projects. As they share a common framework, model modules can be combined in other models and analyses to address multiple topics. Code implementation paradigms facilitate the transfer of model modules for use in other decision contexts.

Updatable

Model code, data and analyses are versioned, with an ongoing program of making new and updated releases. Software is maintained, with opportunities for users and contributors to flag issues, request features and supply code contributions.

3.6 - Implementation

The ready4 software framework is distributed as a collection of framework code libraries that support object-oriented and functional approaches to implementing modular and open source computational models.

3.6.1 - Paradigms

ready4 software is implemented using a combination of object-oriented and functional programming paradigms.

3.6.1.1 - Why ready4 is object oriented

ready4 uses an object oriented programming (OOP) paradigm to implement computational models.

This below section renders a vignette article from the ready4 library. You can use the following links to:

Motivation

The practical utility and ease of use of computational health economic models are in part shaped by the choice of programming paradigm used to develop them. ready4 adopts an object oriented programming (OOP) paradigm which in practice means that the framework principally consists of classes (representations of data structures useful for modelling health systems) and methods (algorithms that can be applied to these data-structures to generate insight useful for policy-making). Adopting an OOP approach is particular useful for helping to make models transparent, reusable and updatable.

Implementation

Modular Computational Models

Two commonly noted features of OOP - encapsulation and inheritance are particularly useful when developing modular computational health economic models.

Encapsulation

Encapsulation allows us to define the data structures (“classes”) used in computational modelling projects in a manner that allows them to be safely combined. For example, assume there are two computational health economic models, one (A) focused on predicting the types and intensity of services used by individuals that present to health services and the other (B) that predicts outcomes for recipients of these services. It may be desirable to develop a new model (C) that combines A and B to model both service use and outcomes. Using encapsulated code allows all of the features and functionality of A to be made available to B in a manner that protects the integrity of A. Specifically, B can only interact with A using the algorithms (“methods”) that have been already defined for A.

Furthermore, if appropriately implemented, methods associated with a class will work with any combination of input values that can be encapsulated by that class - making computational models more transferable. For example, imagine a class (X) that is used to structure summary data relevant to health systems. Methods associated with X (e.g. a method to derive an unmet need statistic) can then applied to two instances of X - one containing data relevant to the Australian context and one with data from the UK context.

Inheritence

The examples highlighted in the previous section have some potential limitations. What if the developers of A didn’t define methods that would allow B to interact with it in the desired way? Or what if there are a number of differences between the Australian and UK system that need to be accounted for when transferring a method from the former to the latter? These types of issues can be addressed by another key feature of OOP - inheritance. Inheritance allows for a “child” class to be created from a “parent” class. By default, the “child” inherits all of the features of the “parent” including all methods associated with the “parent” class. Importantly however, alternative or additional features can also be specified for the “child” to allow it to implement different methods where necessary. For example, when developing our new computational model C we could create a number of new classes that are children of the classes defined in A. We can then define any additional/alternative methods for these classes that overcome any integration issues between the classes and methods of A and B. In this way, we can enjoy the best of both worlds - leveraging all relevant algorithms from A and B (as there is no need to re-invent the wheel), while ensuring that we transparently develop the additional code required for C. This approach also ensures that the respective contributions of the (potentially different) authorship teams behind A, B and C is clearer.

Similarly, inheritance would allow re-use of much of the code from a model of the Australian health system when exploring similar topics within the UK context, while making it straightforward to develop additional code that addresses relevant divergence in features between the two jurisdictions. In practical terms, this would mean developing two child classes of X - class Y for use with Australian data and class Z for use in the UK system. All methods that are not specific to a particular jurisdiction are defined for X and inherited by Y and Z. Methods that are only appropriate for use in the Australian context are defined for Y, while UK specific methods are defined for Z.

Transparent Computational Models

To make modelling analysis programs more readily understood, the ready4 package provides a simple and consistent syntax. Such simplified approaches are facilitated by two other commonly noted features of OOP - polymorphism and abstraction.

Polymorphism

Polymorphism allows for similar concepts to be represented using consistent syntax. The same top level code can therefore be generalised to multiple model implementations, making algorithms simpler to understand and easier to re-use.

Returning to a previous example, the exact same command (e.g. a call to the method exhibit) can be applied to both Y (used for Australian data) and Z (used for UK data). However, the algorithm implemented by that command can vary based on the class that each method is applied to (ie a different algorithm is applied when the data is specified as being from the UK compared to being specified as Australian).

Abstraction

The simplicity enabled by polymorphism is enhanced by Abstraction, which basically means that only the briefest and easiest to comprehend parts of the code are exposed by default to potential users. Once an instance of a model module template class is created, the entire program to ingest model data, analyse it and produce a scientific summary can be represented in a few brief lines of code, readily comprehensible to non-coders. When using open source languages, the elegance and simplicity of abstraction does not restrict the ability of more technically minded users exploring the detailed workings of the underpinning code.

3.6.1.2 - The role of functional programming in ready4 development

ready4 uses functional programming to maximise the re-usability of model algorithms.

Although the object-oriented programming (OOP) approach ready4 implements has many advantages, it can also have some limitations. Some of these limitations have been colorfully highlighted by a popular quote attributed to Joe Armstrong:

“The problem with object-oriented languages is they’ve got all this implicit environment that they carry around with them. You wanted a banana but what you got was a gorilla holding the banana and the entire jungle.”

In practical terms, this means that if not carefully planned, using OOP can create barriers to code-reuse as algorithms come bundled with artefacts of no/low relevance to many potential users. To help maximise the accessibility and re-usability of ready4 algorithms, these algorithms are primarily written using the functional programming paradigm. Only once an algorithm has been implemented using functions are they then linked to a data-structure by means of a calling method. The typical development workflow for a ready4 computational modelling project might therefore look something the following three step process:

  1. A modelling study algorithm is implemented as a program.

  2. To help transfer the methods used in the study algorithm, it is decomposed into functions, which are bundled as a code library (or libraries). The program is updated to use the newly authored functions.

  3. A ready4 module is authored to define a data-structure along with a method (or methods) that call the functions to implement the transferable version of the study algorithm. The new module is added to the previously created code library and the program is again updated so that the algorithm is now implemented by supplying data to the ready4 module and then calling the desired method(s).

Modellers using ready4 for the most part will only use ready4 modules and will rarely interact directly with the functions that implement module methods. However, these functions are potentially of significant usefulness to coders authoring new algorithms. A helpful way of exploring currently available functions is to use the ready4 dependencies app. All ready4 functions are created with minimal, but consistent documentation with the aid of tools from the ready4fun library.

3.6.2 - Modularity

ready4 supports a modular approach to computational model development.

This below section renders a vignette article from the ready4 library. You can use the following links to:

Motivation

A potentially attractive approach to modelling complex health systems is to begin with a relatively simple computational model and to progressively extend its scope and sophistication. Such an approach could be described as “modular” if it is possible to readily combine multiple discrete modelling projects (potentially developed by different modelling teams) that each independently describe distinct aspects of the system being modelled. The ready4 package provides foundational elements of a software framework to support the development of modular and open-source computational health economic models using R.

Implementation

Modular model development is enabled by the encapsulation and inheritance features of Object Oriented Programming (OOP). Specifically, ready4 uses two of R’s systems for implementing OOP - S3 and S4. An in-depth explanation of R’s different class system is beyond the scope of this article, but is explored in Hadley Wickham’s Advanced R handbook. However, it is useful to know some very high level information about S3 and S4 classes:

  • S4 classes are frequently said to be “formal”, “strict” or “rigorous”. The elements of an S4 class are called slots and the type of data that each slot is allowed to contain is specified in the class definition. An S4 class can be comprised of slots that contain different types of data (e.g. a slot that contains a character vector and another slot that contains tabular data).

  • S3 classes are often described as “simple”, “informal” and “flexible”. S3 objects attach an attribute label to base type objects (e.g. a character vector, a data.frame, a list), which in turn is used to work out what methods should be applied to the class.

ready4 Model Modules

As we use the term, a “model module” is comprised of both a data-structure (or “class”) and algorithms (or “methods”) that are associated with that data-structure. A model module can be used to model a discrete component of a health economic system. Model modules can be created from a template - the ready4 package’s Ready4Module class.

We can create an object (X) from the Ready4Module template using the following command.

X <- Ready4Module()

However, if we inspect X we can see it is of limited use as it contains no data other than an empty element called dissemination_1L_chr.

str(X)
#> Formal class 'Ready4Module' [package "ready4"] with 1 slot
#>   ..@ dissemination_1L_chr: chr NA

The Ready4Module class is therefore not intended to be called directly. Instead, the purpose of Ready4Module is to be the parent class of other templates for creating model modules. Ready4Module and all of its child-classes (ie all model module templates) are “S4” classes.

ready4 Concept

Module

An instance of Ready4Module (or classes that inherit from Ready4Module) and its associated methods.

ready4 Model Sub-modules

In ready4, S3 classes are principally used to help define the structural properties of slots (array elements) of model modules and the methods that can be applied to these slots. S3 classes created for these purposes are called sub-modules.

ready4 Concept

Sub-Module

An instance of an informal (S3) class and its associated methods that describes, validates and applies algorithms to a slot of a Module.

Module and Sub-module Methods

All methods associated with modules and sub-modules adopt a common syntax. However, the algorithms implemented by each command in that syntax will vary depending on which module it is applied to. A limited number of methods are defined for the Ready4Module template which by default are inherited by all other module templates. Currently, the only methods defined for Ready4Module are slot-methods and these can be itemised using the get_methods function.

get_methods()
#>  [1] "authorSlot"        "characterizeSlot"  "depictSlot"        "enhanceSlot"       "exhibitSlot"       "ingestSlot"       
#>  [7] "investigateSlot"   "manufactureSlot"   "metamorphoseSlot"  "procureSlot"       "prognosticateSlot" "ratifySlot"       
#> [13] "reckonSlot"        "renewSlot"         "shareSlot"

Developers using the extended ready4 software framework can use our module authoring tools to add new methods or overwrite inherited default methods.

3.6.3 - Syntax

ready4 modules use a simple and consistent syntax.

This below section renders a vignette article from the ready4 library. You can use the following links to:

Motivation

Transparency is one of the underpinning principles of ethical modelling practice. One way to improve the transparency of computational health economic models (such as those we are developing in youth mental health) is to ensure that the programs implementing model analyses can be meaningfully inspected by readers with different levels of technical expertise. Even non-technical readers should be able to follow the high-level logic implemented by model algorithms. If multiple analysis programs are written using a common simplified syntax then reviewers of those programs need to contend with relatively fewer new concepts.

Implementation

ready4 provides a simple syntax that can be consistently applied to attach algorithms (methods) to model modules. It does so by taking advantage of the abstraction and polymorphism features of Object Oriented Programing and R’s use of generic functions. Generic functions don’t implement algorithms themselves - their most salient features are a name and a high level description of the type of task that any method associated with that generic should perform. Whenever a developer creates a method for classes that use R’s S4 and S3 systems (the types used for model modules and sub-modules), they can associate that method with the generic that has a description that is the best match for the algorithm being implemented.

Available generics

A table that summarises the latest ready4 syntax can be ingested from a periodically updated database using get_methods_tb.

Core generics

ready4 includes a number of core generic functions which describe the main types of method to be implemented by model modules. Notably, the ready4 package does not associate methods with any of these core generics. Instead, methods are associated with these generics in R packages of model modules. A HTML table of ready4’s core generics and examples of methods associated with each generic can be displayed using the print_methods function, using the return_1L_chr = "core" argument.

print_methods(x,
              return_1L_chr = "core",
              scroll_width_1L_chr = "100%") 
Method Purpose Examples
author Author and save files 5 , 6, 7 , 16 , 17
characterize Characterize data by generating (tabular) descriptive statistics
depict Depict (plot) features of a dataset 13, 14 , 15
enhance Enhance a dataset by adding new elements
exhibit Exhibit features of a dataset by printing them to the R console 2 , 5 , 6, 13 , 14 , 15 , 16 , 17 , 19 , 20
ingest Ingest data 1 , 2 , 3 , 5 , 6, 13 , 14 , 15 , 16 , 17
investigate Investigate solutions to an inverse problem 16 , 17
manufacture Manufacture a new object
metamorphose Metamorphose data from one model module (or sub-module) instance to an instance of a different model module or sub-module 17
procure Procure items from a dataset 17
prognosticate Prognosticate (make predictions) by solving a forward problem
ratify Ratify that a dataset meets validity criteria 13, 17 , 19 , 20
reckon Reckon (calculate) a value
renew Renew values in a dataset 1 , 2 , 5 , 13, 14 , 15 , 16 , 17 , 19 , 20
share Share data via an online repository 1 , 13, 14 , 15 , 16

Slot generics and methods

Each of the “core” generics also has a “slot” version, for use when applying a core method to a specified slot of a class. The ready4 package associated methods for each of these “slot” generics for the Ready4Module template. Two of these “slot” methods can also be used for additional purposes:

  • procureSlot is a “getter” method - its default behaviour is to return the value of a specified slot. If the argument use_procure_mthd_1L_lgl = T is included in the method call, procureSlot will instead apply the procure method to a specified slot.

  • renewSlot is a “setter” method - if any value other than “use_renew_mthd” (the default) is passed to the new_val_xx argument, that value will be assigned to the specified slot.

A HTML table of the slot generics bundled with ready4 can be displayed using the print_methods function, using the return_1L_chr = "slot" argument.

print_methods(x,
              return_1L_chr = "slot",
              scroll_width_1L_chr = "100%")
Method Purpose Examples
authorSlot Apply the author method to a model module slot
characterizeSlot Apply the characterize method to a model module slot
depictSlot Apply the depict method to a model module slot
enhanceSlot Apply the enhance method to a model module slot
exhibitSlot Apply the exhibit method to a model module slot 4, 17 , 19 , 20
ingestSlot Apply the ingest method to a model module slot
investigateSlot Apply the investigate method to a model module slot
manufactureSlot Apply the manufacture method to a model module slot
metamorphoseSlot Apply the metamorphose method to a model module slot
procureSlot Procure (get) data from a slot 4, 8 , 14 , 16 , 17
prognosticateSlot Apply the prognosticate method to a model module slot
ratifySlot Apply the ratify method to a model module slot
reckonSlot Apply the reckon method to a model module slot
renewSlot Apply the renew method to a model module slot 4, 8 , 14 , 17 , 19 , 20
shareSlot Apply the share method to a model module slot

Extended author generics

Finally, there are a small number of other generics that are more general extensions of the core functions. Currently, these extended generics are all variants on the author generics, with each one specifying the type of output to be authored by the method. The ready4 package does not include methods for any of these extended generics. A HTML table of the extended generics bundled with ready4 can be displayed using the print_methods function, using the return_1L_chr = "extended" argument.

print_methods(x,
              exclude_mthds_for_chr = "Ready4Module",
              return_1L_chr = "extended",
              scroll_width_1L_chr = "100%")
Method Purpose Examples
authorClasses Author and document classes
authorData Author and document datasets 4, 17
authorFunctions Author and document functions
authorReport Author and save a report 4

4 - Software

ready4 is a suite of software, with each included item performing a distinctive role.

4.1 - Code repositories

ready4 software is freely available from multiple open access repositories.

Currently:

We have yet to issue production releases of any of our R packages, but once these are finalised we will be submitting them to the Comprehensive R Archive Network.

4.2 - System requirements

What you need in order to be able to use ready4 software on your machine.

Currently, all ready4 software is written in R (for libraries), R Markdown (for programs and sub-routines) and JavaScript (for the user interface component of Shiny applications). Therefore:

  • to use ready4 libraries and programs / subroutines you must have an up to date version of R installed on your machine and it is recommended that you install the RStudio IDE; and

  • the requirements for using ready4 user interfaces depend on whether you are running a version we have deployed to the web (in which case you just need a supported browser) or whether you are running the app on your local machine (in which case you will need both R and RStudio).

4.3 - Release statuses

Whether and how you should use a specific version of ready4 software depends in part on its release status.

4.3.1 - Unreleased code

Some work in progress code has yet to be publicly released or fornmally acknowledged as part of the ready4 suite.

Currently, a new ready4 software project initiated by the ready4 core team will by default be made public as a pre-release version in the ready4 GitHub organisation. However, there are some important exceptions. Principally, these exceptions relate to code that we authored in the initial phase of ready4’s development to which some or all of the following apply:

  • the code is highly unstable because it has not been (fully) updated to account to major changes implemented in core dependencies;
  • the code uses outdated naming conventions and is potentially confusing when used in conjunction with other elements of the ready4 suite; and/or
  • the code repository has yet to be cleansed of artefacts that are not yet appropriate for public dissemination (e.g. renders of draft scientific manuscripts).

Depending on which of the above issues apply to a code repository, that repository will either be:

  • a private repository (not accessible to anyone outside the core development team); or
  • a public repository stored in a location other than the ready4 GitHub organisation.

4.3.2 - Development releases

Development releases provide the most comprehensive and up to date public record of a ready4 project’s source code but may be poorly documented and tested.

A complete record of all publicly available versions of a ready4 software project’s code over its entire development history (including the most up to date version) is stored in the ready4 GitHub organisation. We refer to these comprehensive publicly available source code resources as “development releases” (even though these records will include versions of our code that we have not formally labelled as “releases”).

Public access to development releases allows individuals to install, test and preview code in advance of production versions being released. Development releases also provide transparency as to who contributed what to a software project and when these contributions were made. Accessing the latest development version of the code is particularly useful to people who wish to contribute bug fixes or new features to our code.

Limitations of development releases include the likelihood that some or all of this code may be inadequately documented or tested. In peer reviewed publications, it is generally considered preferable to avoid citing the copies of code stored in GitHub repositories as these repositories are impermanent (they can be moved, renamed or deleted at any time).

4.3.3 - Production releases

Production releases are the versions of software intended for end-users.

Production releases of our code are intended for end-users and signal that they have undergone a number of quality assurance checks and have some supporting documentation.

We have yet to make any production releases of ready4 software, but plan to do this in 2024. Production releases of ready4 R packages will be submitted to CRAN. Unless and until a software item is submitted to a production code repository like CRAN, the recommended platform from which to install our software is that software’s GitHub repository.

4.3.4 - Archived releases

Archived releases are permanent, uniquely identified records of key project milestones.

Software items that we have formally issued as “releases” are archived as permanent, uniquely identified (with DOI) and citable records within the ready4 Zenodo community. Archived releases of ready4 software are useful as they are snapshots of a project at key milestones in its development (e.g. at the time an analysis was undertaken). As these milestones are purposely selected, archived releases are more likely to have undergone some testing and documenting prior to being released than code not selected for release.

A limitation of archived code libraries is that a greater knowledge of R is required to appropriately install R packages from Zenodo compared to the simpler installation process for the versions of code libraries stored on either GitHub or CRAN.

4.4 - Code libraries

The ready4 framework is distributed as six R libraries.

4.4.1 - Finding framework libraries

There are two types of framework libraries - a foundational library and libraries of authoring tools.

The two types of framework library are:

  • - the foundational ready4 module and syntax; and

  • - tools to implement standardised, semi-automated workflows for authoring and documenting computational models.

Currently available framework libraries are summarised below.



Type Package Purpose Documentation Code Examples
Author, Ingest, Label and Share Health Economic Model Datasets Citation , Website , Manual - Short (PDF) , Manual - Full (PDF) Dev , Archive 1, 2, 3
Author Health Economic Analysis Programs and Reporting Templates Citation , Website , Manual - Short (PDF) , Manual - Full (PDF) Dev , Archive 4
Author and Document Functions that Implement Transferable Health Citation , Website , Manual - Short (PDF) , Manual - Full (PDF) Dev , Archive 5
Author and Document Classes that Make Health Economic Models Citation , Website , Manual - Short (PDF) , Manual - Full (PDF) Dev, Archive 6
Author R Packages of Health Economic Model Modules Citation , Website , Manual - Short (PDF) , Manual - Full (PDF) Dev , Archive 7
Implement Transparent, Reusable and Updatable Computational Health Citation , Website , Manual - Short (PDF) , Manual - Full (PDF) Dev , Archive 8 , 9 , 10, 11

4.4.2 - Code library documentation

Each ready4 code library is supported by a standardised set of documentation resources.

All ready4 code libraries have:

All ready4 code libraries include interactive help. Once you have installed and loaded a library, you can view its contents by using the command library(help="PACKAGE_NAME") (for example, library(help="ready4")).

Note that the manuals and files used by the interactive help are currently all automatically authored by tools from our ready4fun package and are therefore quite basic (and in some cases use clumsy English). In the future we hope to augment this machine generated documentation with human-authored documentation.

Most ready4 libraries (the exceptions are those at very early stages of development) have one or more vignette articles that provide examples of how to use it. These are available from the “Articles” section of each library’s website.

4.4.3 - Dependencies

Search for ready4 library and function dependencies using our interactive app.

As an open-source project, ready4 depends on the software created and shared by others. Using the DependenciesGraphs R package, we have created the Shiny app below to:

  • explore the inter-dependencies between ready4 libraries;
  • highlight how our software depends on other R packages;
  • itemise the contents each ready4 library;
  • display function help files; and
  • map function inter-dependencies across multiple ready4 libraries.

To use the app, choose one of the two potential pathways:

  • For Pathway 1, start at Step 1 (choose the libraries you wish to profile from the drop down menu and click on the Go button), before proceeding to Step 2 (click on one library that you wish to view the contents of), then Step 3 (click on the view functions button) and finally Step 4 (click on the function for which you would like to view documentation);
  • For Pathway 2, start at Step 1 (choose libraries from the drop down menu), then Step 2 (click on the Find functions button), then Step 3 (select functions from the drop down menu) and finally Step 4 (click on the Make graph button).

Note, as the app is displayed on this page via an iFrame, it may be difficult to view on a phone. If so, you can try the following link: https://orygen.shinyapps.io/dependencies/

4.4.4 - Installation and set-up

Important information to review before installing and using our software

ready4 libraries are currently only available as development releases, so you will need to use a tool like devtools to assist with installing ready4 R packages directly from our GitHub organisation. If you do not have devtools on your machine you can install it with the following command.

utils::install.packages("devtools")

4.4.4.1 - Installing the ready4 framework foundation library

The ready4 framework foundation is the first ready4 library you should install.

Before you install

If you plan to use ready4 for any purpose, you will need to install the ready4 foundation library.

However, please note that the ready4 library is not yet available as a production release. You should therefore understand the limitations of using ready4 software development releases before you make the decision to install this software.

As all software in the ready4 suite depends on the ready4 library, so in most cases you do not need to install this library directly (it will come bundled with whatever other ready4 suite software you install).

If you can run the following command without producing an error message, then you already have it.

find.package("ready4")

Installation

You can install the ready4 library directly from its GitHub repository.

devtools::install_github("ready4-dev/ready4")

Try it out!

Before you apply ready4 tools to your own project, you should make sure you can run some or all of the example code included in the package vignettes.

4.4.4.2 - Installing authoring tools

Depending on how you plan to use ready4, you may need to install some or all of its authoring tools.

4.4.4.2.1 - Installing tools for authoring computational models

Instructions for installing the ready4class, ready4fun and ready4pack libraries.

Before you install

If you are a coder planning on using ready4 to author computational models, then you may wish to install the ready4class, ready4fun and ready4pack libraries.

However, please note that none of these libraries are yet available as a production release. You should therefore understand the limitations of using ready4 software development releases before you make the decision to install this software. Although we use these authoring tools intensively to help us write highly standardised model modules, we feel that these tools are only likely to be helpful to others once much more comprehensive documentation and training resources become available. Without this training and support, these packages are unlikely to appear to be very user-friendly. Furthermore, the initial burden of complying with house-style, file-naming and directory structure requirements of these packages is only likely to be worthwhile if you plan on developing multiple ready4 module libraries. If you still think these tools could be useful to you, consider contacting us first to discuss what additional information may be most helpful to you.

Installation

As ready4class and ready4fun are bundled as dependencies of ready4pack, you can install all three from our GitHub organisation using one command.

devtools::install_github("ready4-dev/ready4pack")

Configuration

To use these computational model authoring tools, you will need to have set-up and appropriately configured your own accounts in:

  • GitHub (you will need write permissions to a GitHub organisation and to then enable GitHub actions and GitHub pages support for the repositories you create in that organisation);
  • Zenodo (you will need to have linked each GitHub repository used for your ready4 projects to your Zenodo account); and
  • Codecov (linked to your GitHub organisation).

The machine onto which you install ready4pack will also need to be securely storing your GitHub credentials (i.e. the value for the GITHUB_PAT token).

Try it out!

It should be noted that the development workflow supported by our computational model authoring tools is not yet well documented. We don’t recommend undertaking R package development with these tools until this has been rectified. However, if you still want to try these tools out, the best place to start is review the examples in the ready4class, ready4fun and ready4pack vignettes.

4.4.4.2.2 - Installing tools for authoring and managing model datasets

Instructions for installing the ready4use library.

Before you install

If you are a coder or modeller planning to create, share and access model datasets with ready4, then you will need the ready4use library.

However, please note that ready4use is not yet available as a production release. You should therefore understand the limitations of using ready4 software development releases before you make the decision to install this software.

You may already have ready4use installed on your machine (e.g. if you have previously installed other ready4 framework and module libraries that include ready4use as a dependency). If you can run the following command without producing an error message, then you already have it.

find.package("ready4use")

Installation

You can install ready4use directly from its GitHub repository.

devtools::install_github("ready4-dev/ready4use")

Configuration

If one of your intended uses of ready4use is to share outputs in online datasets, you will need to have set up an account on a Dataverse installation (we recommend using the Harvard Dataverse). Some of the key terms and concepts relating to using a Dataverse installation in conjunction with ready4use are described in this tutorial.

You need to ensure that you have write permissions to any Dataverse Datasets that you plan to use to post files to. Furthermore, the machine on which you install ready4use should also securely store your Dataverse account credentials (specifically, values for the DATAVERSE_KEY and DATAVERSE_SERVER tokens). Details of how to do this are described in documentation for the dataverse R package, an important third party dependency package for ready4use.

Try it out

You should now be able to run the example code included in the package vignettes. To run all of this code you will need to replace the details of the Dataverse Dataset to which files are being written to those of your own Dataverse Dataset.

4.4.4.2.3 - Installing tools for authoring reproducible analyses

Instructions for installing the ready4show library.

Before you install

If you are a coder or modeller planning to implement a reproducible analysis with ready4, you will need to install the ready4show library.

However, please note that ready4show is not yet available as a production release. You should therefore understand the limitations of using ready4 software development releases before you make the decision to install this software.

If you have installed other ready4 libraries, then ready4show may have already been installed as a dependency. If you can run the following command without producing an error message, then you already have it.

find.package("ready4show")

Installation

The ready4show library can be installed directly from its GitHub repository.

devtools::install_github("ready4-dev/ready4show")

Try it out!

Before you apply ready4show tools to your own project, you should make sure you can run some or all of the example code included in the package vignettes.

4.4.4.3 - Installing ready4 computational model modules

To implement a modelling analysis with ready4 you need to install computational model modules.

Before you install

If you plan on using existing ready4 modules for a modelling project, you can review currently available module libraries, to identify which libraries are relevant to your project.

However, please note that no ready4 module library is yet available as a [production release](/docs/software/status/production-releases/. You should therefore understand the limitations of using ready4 software development releases before you make the decision to install this software.

Installation

The command to install each ready4 module takes the following format.

devtools::install_github("ready4-dev/PACKAGE_NAME")

For example, if you are planning to predict health utility using some of the mapping algorithms that we have previously developed, you can install the youthu library with the following command.

devtools::install_github("ready4-dev/youthu")

Configuration

A small number of ready4 modules require that you configure some of the dependencies installed with them before they can be used. In particular:

  • if you are using modules from the TTU package to undertake a utility mapping study, you will need to have both installed and configured the cmdstanr R package as per the instructions on that package’s documentation website; and

  • if you are using the mychoice package to undertake a discrete choice experiment study and are using a Mac, you need to ensure that you have a Fortran compiler installed. Some relevant advice on this: https://mac.r-project.org/tools/ .

Try it out!

Before you apply ready4 modules to your own project, you should make sure you can run some or all of the example code included in relevant library vignette articles. The package website URL takes the form of https://ready4-dev.github.io/PACKAGE_NAME/articles/ (e.g. the vignettes for the youthvars package are available at https://ready4-dev.github.io/youthvars/articles/).

4.5 - Terms of use

ready4 is distributed without warranties under open source licenses - we just ask you to appropriately cite it.

4.5.1 - Open source licensing

ready4 is freely available to all under copy-left licensing arrangements.

To help ensure the models we develop are as transparent as possible and to make their algorithms as useful to others as possible, all ready4 software is free and open-source. You are encouraged to make as widespread use of our software, including the creation of derivative works, as you see fit, so long as it is consistent with each item’s license. Our software is typically licensed under GPL-3, a copy-left open-source licensing regime.

4.5.2 - Citing ready4

If you find ready4 useful, please cite it appropriately - it is easy to do!

To make it easier to cite our software, each software item bundle includes a CITATION.cff file. Inclusion of this file means that the repositories storing our software can generate appropriate citations in the format of most relevance to you.

Currently:

  • Zenodo provides a free text field under the heading “Cite as” which enables you to generate a wide range of citation manager and journal specific citation outputs. There is also an “Export” tool that will generate citation metadata in multiple output formats;
  • OpenAire Explore has a “Cite this software” button that allows you to generate a citation in multiple journal formats or to download BibTeX or RIS files;
  • Github repositories have a “Cite this repository” button that can generate both BibTeX and APA output as well as link to the Citation.cff file.

Additionally, we have included a CITATION file in each of our R libraries so that you can generate a citation from within an R session using the citation function (for example: citation("ready4").

4.5.3 - Disclaimer

ready4 is distributed without any warranties.

All ready4 software is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

Furthermore, no ready4 software is yet sufficiently well documented and tested to be given a production release. All ready4 software should therefore viewed as experimental development releases.

5 - Tutorials

ready4 software framework libraries provide tools for authoring and sharing model modules, datasets and analyses.

5.1 - Author and share model modules

Tools from the ready4class, ready4 fun and ready4pack R libraries streamline and standardise the authoring of ready4 modules.

5.1.1 - Authoring model data structures

The ready4class R package supports partially automated and standardised workflows for defining the data structures to be used in computational models.

This below section renders a vignette article from the ready4class library. You can use the following links to:

Motivation

The ready4 framework uses object oriented programming (OOP) to implement modular approaches to computational models of mental health systems. That means that a standardised approach to developing modules (S4 classes) and sub-modules (S3 classes) is required. ready4class provides the tools to implement this workflow.

Workflow

Prototyes, constructor and manifest

The main classes exported as part of ready4class are readyclass_manifest and ready4class_constructor. ready4class_pt_lup is a tibble based ready4 sub-module, which contains metadata on the prototypes of classes that can be used as sub-components of ready4 modules and sub-modules (for example a tibble based class can be used as a slot in an S4 class). When authoring ready4 R packages, you will create a ready4class_pt_lup instance and store it in an online repository that you have write permissions to. As you create new ready4 modules and sub-modules using ready4class tools, your ready4class_pt_lup object will be updated so that these classes can be made available to any future modules or sub-modules that you author. The ready4class_pt_lup sub-module recently used in workflows for authoring ready4 modules is reproduced below.

x <- ready4use::Ready4useRepos(gh_repo_1L_chr = "ready4-dev/ready4",
                               gh_tag_1L_chr = "Documentation_0.0") %>%
  ingest(fls_to_ingest_chr = "prototype_lup",
         metadata_1L_lgl = F) 
x %>%
  exhibit(scroll_box_args_ls = list(width = "100%"))
Class Prototypes Lookup Table
Class Value Namespace Function Default Is Old Class
TTUSynopsis TTU::TTUSynopsis() TTU TTUSynopsis FALSE
TTUReports TTU::TTUReports() TTU TTUReports FALSE
TTUProject TTU::TTUProject() TTU TTUProject FALSE
AusLookup aus::AusLookup() aus AusLookup FALSE
AusOrygen aus::AusOrygen() aus AusOrygen FALSE
AusHeadspace aus::AusHeadspace() aus AusHeadspace FALSE
AusProjections aus::AusProjections() aus AusProjections FALSE
AusTasmania aus::AusTasmania() aus AusTasmania FALSE
AusACT aus::AusACT() aus AusACT FALSE
character NA_character\_ base NA_character\_ FALSE
data.frame data.frame() base data.frame() FALSE
integer NA_integer\_ base NA_integer\_ FALSE
list list(list()) base list list() FALSE
logical NA base NA FALSE
numeric NA_real\_ base NA_real\_ FALSE
POSIXt .POSIXct(NA_character\_) base .POSIXct NA_character\_ FALSE
CostlySource costly::CostlySource() costly CostlySource FALSE
CostlySeed costly::CostlySeed() costly CostlySeed FALSE
CostlyStandards costly::CostlyStandards() costly CostlyStandards FALSE
CostlyCorrespondences costly::CostlyCorrespondences() costly CostlyCorrespondences FALSE
CostlyCountries costly::CostlyCountries() costly CostlyCountries FALSE
CostlyCurrencies costly::CostlyCurrencies() costly CostlyCurrencies FALSE
dfidx dfidx::dfidx(dfidx()) dfidx dfidx dfidx() FALSE
Ready4Module Ready4Module() ready4 Ready4Module FALSE
Ready4Private Ready4Private() ready4 Ready4Private FALSE
Ready4Public Ready4Public() ready4 Ready4Public FALSE
ready4class_constructor ready4class::ready4class_constructor() ready4class ready4class_constructor TRUE
ready4class_pt_lup ready4class::ready4class_pt_lup() ready4class ready4class_pt_lup TRUE
ready4class_manifest ready4class::ready4class_manifest() ready4class ready4class_manifest TRUE
ready4fun_badges ready4fun_badges() ready4fun ready4fun_badges TRUE
ready4fun_abbreviations ready4fun_abbreviations() ready4fun ready4fun_abbreviations TRUE
ready4fun_objects ready4fun_objects() ready4fun ready4fun_objects TRUE
ready4fun_functions ready4fun_functions() ready4fun ready4fun_functions TRUE
ready4fun_executor ready4fun_executor() ready4fun ready4fun_executor TRUE
ready4fun_description ready4fun_description() ready4fun ready4fun_description TRUE
ready4fun_metadata_a ready4fun_metadata_a() ready4fun ready4fun_metadata_a TRUE
ready4fun_metadata_b ready4fun_metadata_b() ready4fun ready4fun_metadata_b TRUE
ready4fun_manifest ready4fun_manifest() ready4fun ready4fun_manifest TRUE
ready4fun_dataset ready4fun_dataset() ready4fun ready4fun_dataset TRUE
ready4pack_manifest ready4pack::ready4pack_manifest() ready4pack ready4pack_manifest TRUE
ready4show_authors ready4show::ready4show_authors() ready4show ready4show_authors TRUE
ready4show_institutes ready4show::ready4show_institutes() ready4show ready4show_institutes TRUE
ready4show_correspondences ready4show::ready4show_correspondences() ready4show ready4show_correspondences TRUE
Ready4showPaths ready4show::Ready4showPaths() ready4show Ready4showPaths FALSE
Ready4showSynopsis ready4show::Ready4showSynopsis() ready4show Ready4showSynopsis FALSE
ready4use_distributions ready4use::ready4use_distributions() ready4use ready4use_distributions TRUE
ready4use_dataverses ready4use::ready4use_dataverses() ready4use ready4use_dataverses TRUE
ready4use_imports ready4use::ready4use_imports() ready4use ready4use_imports TRUE
ready4use_mapes ready4use::ready4use_mapes() ready4use ready4use_mapes TRUE
ready4use_dictionary ready4use::ready4use_dictionary() ready4use ready4use_dictionary TRUE
Ready4useFiles ready4use::Ready4useFiles() ready4use Ready4useFiles FALSE
Ready4useRaw ready4use::Ready4useRaw() ready4use Ready4useRaw FALSE
Ready4useProcessed ready4use::Ready4useProcessed() ready4use Ready4useProcessed FALSE
Ready4useArguments ready4use::Ready4useArguments() ready4use Ready4useArguments FALSE
Ready4useDyad ready4use::Ready4useDyad() ready4use Ready4useDyad FALSE
Ready4useIngest ready4use::Ready4useIngest() ready4use Ready4useIngest FALSE
Ready4useRepos ready4use::Ready4useRepos() ready4use Ready4useRepos FALSE
Ready4usePointer ready4use::Ready4usePointer() ready4use Ready4usePointer FALSE
Ready4useRecord ready4use::Ready4useRecord() ready4use Ready4useRecord FALSE
ScorzProfile scorz::ScorzProfile() scorz ScorzProfile FALSE
ScorzAqol6 scorz::ScorzAqol6() scorz ScorzAqol6 FALSE
ScorzAqol6Adol scorz::ScorzAqol6Adol() scorz ScorzAqol6Adol FALSE
ScorzAqol6Adult scorz::ScorzAqol6Adult() scorz ScorzAqol6Adult FALSE
ScorzEuroQol5 scorz::ScorzEuroQol5() scorz ScorzEuroQol5 FALSE
sf sf::st_sf(sf::st_sfc()) sf st_sf sf::st_sfc() FALSE
specific_models specific::specific_models() specific specific_models TRUE
specific_predictors specific::specific_predictors() specific specific_predictors TRUE
SpecificParameters specific::SpecificParameters() specific SpecificParameters FALSE
SpecificPrivate specific::SpecificPrivate() specific SpecificPrivate FALSE
SpecificShareable specific::SpecificShareable() specific SpecificShareable FALSE
SpecificResults specific::SpecificResults() specific SpecificResults FALSE
SpecificProject specific::SpecificProject() specific SpecificProject FALSE
SpecificInitiator specific::SpecificInitiator() specific SpecificInitiator FALSE
SpecificModels specific::SpecificModels() specific SpecificModels FALSE
SpecificPredictors specific::SpecificPredictors() specific SpecificPredictors FALSE
SpecificFixed specific::SpecificFixed() specific SpecificFixed FALSE
SpecificMixed specific::SpecificMixed() specific SpecificMixed FALSE
SpecificConverter specific::SpecificConverter() specific SpecificConverter FALSE
SpecificSynopsis specific::SpecificSynopsis() specific SpecificSynopsis FALSE
tbl_df tibble::tibble() tibble tibble FALSE
vicinity_parameters vicinity::vicinity_parameters() vicinity vicinity_parameters TRUE
vicinity_values vicinity::vicinity_values() vicinity vicinity_values TRUE
vicinity_abbreviations vicinity::vicinity_abbreviations() vicinity vicinity_abbreviations TRUE
vicinity_processed vicinity::vicinity_processed() vicinity vicinity_processed TRUE
vicinity_raw vicinity::vicinity_raw() vicinity vicinity_raw TRUE
vicinity_resolutions vicinity::vicinity_resolutions() vicinity vicinity_resolutions TRUE
vicinity_points vicinity::vicinity_points() vicinity vicinity_points TRUE
vicinity_templates vicinity::vicinity_templates() vicinity vicinity_templates TRUE
vicinity_identifiers vicinity::vicinity_identifiers() vicinity vicinity_identifiers TRUE
vicinity_mapes vicinity::vicinity_mapes() vicinity vicinity_mapes TRUE
VicinityLookup vicinity::VicinityLookup() vicinity VicinityLookup FALSE
VicinityMacro vicinity::VicinityMacro() vicinity VicinityMacro FALSE
VicinityMesoRegion vicinity::VicinityMesoRegion() vicinity VicinityMesoRegion FALSE
VicinityMesoArea vicinity::VicinityMesoArea() vicinity VicinityMesoArea FALSE
VicinityMicro vicinity::VicinityMicro() vicinity VicinityMicro FALSE
VicinityProfile vicinity::VicinityProfile() vicinity VicinityProfile FALSE
VicinitySpaceTime vicinity::VicinitySpaceTime() vicinity VicinitySpaceTime FALSE
VicinityArguments vicinity::VicinityArguments() vicinity VicinityArguments FALSE
VicinityLocal vicinity::VicinityLocal() vicinity VicinityLocal FALSE
VicinityLocalRaw vicinity::VicinityLocalRaw() vicinity VicinityLocalRaw FALSE
VicinityLocalProcessed vicinity::VicinityLocalProcessed() vicinity VicinityLocalProcessed FALSE
youthvars_aqol6d_adol youthvars::youthvars_aqol6d_adol() youthvars youthvars_aqol6d_adol TRUE
youthvars_chu9d_adolaus youthvars::youthvars_chu9d_adolaus() youthvars youthvars_chu9d_adolaus TRUE
youthvars_phq9 youthvars::youthvars_phq9() youthvars youthvars_phq9 TRUE
youthvars_bads youthvars::youthvars_bads() youthvars youthvars_bads TRUE
youthvars_gad7 youthvars::youthvars_gad7() youthvars youthvars_gad7 TRUE
youthvars_oasis youthvars::youthvars_oasis() youthvars youthvars_oasis TRUE
youthvars_scared youthvars::youthvars_scared() youthvars youthvars_scared TRUE
youthvars_k6 youthvars::youthvars_k6() youthvars youthvars_k6 TRUE
youthvars_k6_aus youthvars::youthvars_k6_aus() youthvars youthvars_k6_aus TRUE
youthvars_k10 youthvars::youthvars_k10() youthvars youthvars_k10 TRUE
youthvars_k10_aus youthvars::youthvars_k10_aus() youthvars youthvars_k10_aus TRUE
youthvars_sofas youthvars::youthvars_sofas() youthvars youthvars_sofas TRUE
YouthvarsDescriptives youthvars::YouthvarsDescriptives() youthvars YouthvarsDescriptives FALSE
YouthvarsProfile youthvars::YouthvarsProfile() youthvars YouthvarsProfile FALSE
YouthvarsSeries youthvars::YouthvarsSeries() youthvars YouthvarsSeries FALSE

ready4class_constructor is another tibble based ready4 sub-module that summarises the desired features of the ready4 modules and sub-modules that you are authoring. An instance of ready4class_constructor is combined with a ready4fun_manifest sub-module to create a ready4class_manifest sub-module. Instances of ready4class_constructor are most efficiently created using the make_pt_ready4class_constructor function.

Typical use

The most important method included in ready4class is the author method for the ready4class_manifest sub-module, that enhances the author method defined for the ready4fun_manifest so that consistently documented R package classes are also generated.

## Not run
author(y)

Examples

ready4class sub-modules and methods are not intended for independent use, but instead should be deployed as part of ready4pack R package authoring workflow.

Future documentation

It should be noted that some ready4class methods require files of a standardised format to be saved in specific sub-directories of the package data-raw directory. Detailed instructions on how to prepare these files are not yet available, but will be outlined in documentation to be released in 2022.

5.1.2 - Authoring model algorithms

The ready4fun R package supports standardised approaches to code authoring that facilitate partial automation of the documenting of model algorithms.

This below section renders a vignette article from the ready4fun library. You can use the following links to:

Motivation

The ready4 framework uses an object-oriented programming (OOP) approach to implement computational health economic models. One motivation for using OOP is the concept of “abstraction” - making things as simple as possible for end-users of ready4 modules by exposing the minimal amount of code required to implement each method.

However, some users of the ready4 modules will want to “look under the hood” and examine the code that implements module algorithms in much more detail. Reasons to do so include to:

  • gain detailed insight into how methods are implemented;
  • test individual sub-components of methods as part of code verification and model validation checks;
  • re-use sub-components of existing methods when authoring new methods.

To help facilitate achieving these objectives, methods associated with ready4 modules can be de-composed into functions that can be used independent of ready4 modules. However, these functions need to be documented and will be easier to comprehend if they adopt a consistent house style (e.g. naming conventions). ready4fun provides workflow tools (classes, methods, functions and datasets) to achieve these goals.

ready4fun function authoring taxonomies, abbreviations and workflow

The ready4fun package uses taxonomy and abbreviation datasets to ensure standardised function code style and documentation.

Function names begin with a meaningful verb

Consistent with a naming convention popular in the R development community, all functions authored with the ready4 framework need to begin with a verb. Furthermore, the choice of verb is meaningful - it communicates something about the type of task a function implements. For example, all functions beginning with the word “fit” will fit a model of a specified type to a dataset. The definitions of all meaningful verbs used in functions authored for a ready4 framework model implementation can be retrieved using get_fn_types(), which by default returns a dataset instance of the ready4fun_functions submodule.

x <- get_fn_types(gh_repo_1L_chr = "ready4-dev/ready4")
class(x)
#> [1] "ready4fun_functions" "tbl_df"              "tbl"                 "data.frame"
exhibit(x,select_int = 1:2, scroll_box_args_ls = list(width = "100%"))
Meaningful verbs
Verb Description
Add Updates an object by adding new values to new or empty fields.
Assert Validates that an object conforms to required condition(s). If the object does not meet all required conditions, program execution will be stopped and an error message provided.
Bind Binds two objects together to create a composite object.
Calculate Performs a numeric calculation.
Close Closes specified connections.
Fit Fits a model of a specified type to a dataset.
Force Checks if a specified local or global environmental condition is met and if not, updates the specified environment to comply with the condition.
Format Modifies the format of an object.
Get Extracts data from an object.
Import Reads a data object in its native format and converts it to an R object.
Impute Imputes data.
Knit Knits an RMD or Rmarkdown file.
Launch Launches an R Shiny app.
Make Creates a new R object.
Plot Plots data.
Predict Applies a model to make predictions.
Print Prints output to console.
Randomise Randomly samples from data.
Read Reads an R script into memory.
Remove Edits an object, removing a specified element or elements.
Rename Renames elements of an object based on a pre-specified schema.
Reorder Reorders an object to conform to a pre-specified schema.
Replace Edits an object, replacing a specified element with another specified element.
Reset Edits an object, overwriting the current version with a default version.
Rowbind Performs custom rowbind operations on table objects.
Scramble Randomly reorders an object.
Transform Edits an object in such a way that core object attributes - e.g. shape, dimensions, elements, type - are altered.
Unload Performs a custom detaching of a package from the search path.
Update Edits an object, while preserving core object attributes.
Validate Validates that an object conforms to required criteria.
Write Writes a file to a specified local directory.

Function inputs and outputs have meaningful suffices

The type of input (arguments) required and output (return) produced by a function can be efficiently communicated by using meaningful suffices. For example all objects ending in “_chr” are character vectors and all objects ending in “_int” are integer vectors. Definitions of all meaningful suffices used in functions authored for a ready4 framework model implementation can be retrieved using get_obj_types(), which by default returns a dataset instance of the ready4fun_objects submodule.

y <- get_obj_types(gh_repo_1L_chr = "ready4-dev/ready4")
class(y)
#> [1] "ready4fun_objects" "tbl_df"            "tbl"               "data.frame"
exhibit(y, select_int = 1:2, scroll_box_args_ls = list(width = "100%"))
Meaningful suffices
Suffix Description
arr array
chr character
dbl double
df data.frame
dtm date
env environment
fct factor
fn function
int integer
lgl logical
ls list
lup lookup table
mat matrix
mdl model
plt plot
prsn person
r3 ready4 submodule
r4 ready4 module
rgx regular expression
s3 S3
s4 S4
sf simple features object
tb tibble

Consistent use of abbreviations

Further information about the purpose of a function and the nature of its inputs and outputs can be encoded by using naming conventions that make consistent use of abbreviations. A master table of the abbreviations used in a ready4 framework model implementation can be retrieved using get_abbrs(), which by default returns a dataset instance of the ready4fun_abbreviations submodule.

z <- get_abbrs(gh_repo_1L_chr = "ready4-dev/ready4")
class(z)
#> [1] "ready4fun_abbreviations" "tbl_df"                  "tbl"                     "data.frame"
exhibit(z %>% head(50), select_int = 1:2, scroll_box_args_ls = list(width = "100%"))
Abbreviations
Abbreviation Description
... additional arguments
1L length one
1L_chr character vector of length one
1L_chr_ls list of character vectors of length one
1L_dbl double vector of length one
1L_dbl_ls list of double vectors of length one
1L_dtm date vector of length one
1L_dtm_ls list of date vectors of length one
1L_fct factor vector of length one
1L_fct_ls list of factor vectors of length one
1L_int integer vector of length one
1L_int_ls list of integer vectors of length one
1L_lgl logical vector of length one
1L_lgl_ls list of logical vectors of length one
1L_rgx regular expression vector of length one
1L_rgx_ls list of regular expression vectors of length one
1Ls length ones
8d 8 Dimension
8ds 8 Dimensions
abbr abbreviation
abbrs abbreviations
abs absolute
abss absolutes
addl additional
addls additionals
adol adolescent
adol6d Assessment of Quality of Life Six Dimension (Adolescent version)
adols adolescents
alg algorithm
algs algorithms
altv alternative
anlys analysis
anlyss analyses
aqol Assessment of Quality of Life
aqol6d Assessment of Quality of Life Six Dimension
aqol6dU Assessment of Quality of Life Six Dimension Health Utility
arg argument
args arguments
arr array
arr_ls list of arrays
arr_r3 ready4 submodule extension of array
arr_r3_ls list of ready4 submodule extension of arrays
artl article
artls articles
att attribute
atts attributes
aus Australia
AusACT Meta data for processing ACT population projections.
AusHeadspace Meta data for constructing Headspace Centre geometries.
AusLookup Lookup tables for Australian geometry and spatial attribute data.

The ready4fun_abbreviations submodule is searchable. It is therefore possible to see if an abbreviation has been defined for an existing word or phrase…

procure(z,"template")
#> # A tibble: 2 × 3
#>   short_name_chr long_name_chr plural_lgl
#>   <chr>          <chr>         <lgl>     
#> 1 tmpl           template      FALSE     
#> 2 tmpls          templates     TRUE

…and to look-up the meaning of an abbreviation…

procure(z,"org",type_1L_chr = "extension")
#> # A tibble: 2 × 3
#>   short_name_chr long_name_chr plural_lgl
#>   <chr>          <chr>         <lgl>     
#> 1 org            organisation  FALSE     
#> 2 orgs           organisations TRUE

…or whether a potential abbreviation has already been defined.

procure(z,"org", type_1L_chr = "extension", what_1L_chr = "string")
#> [1] "org"  "orgs"

Workflow

Manifest

The main class exported as part of ready4fun is the ready4 sub-module ready4fun_manifest which is used to specify metadata (including details of the repository in which the fn_types_lup, seed_obj_lup_tb and abbreviations_lup objects are stored) for the functions being authored and the R package that will contain them.

Typical Usage

A ready4fun_manifest object is most efficiently created with the aid of the make_pkg_desc_ls and make_manifest functions rather than a direct call to the ready4fun_manifest() function.

## Not run
x <- ready4fun::make_pkg_desc_ls(
  pkg_title_1L_chr = "Your Package Title",
  pkg_desc_1L_chr = "Your Package Description.",
  authors_prsn = c(
    utils::person("Author 1 Name",
      role = c("aut", "cre")
    ),
    utils::person("Author 2 Name", role = c("cph"))
  ),
  urls_chr = c(
    "Package website url",
    "Package source code url",
    "Project website"
  )
) %>%
  ready4fun::make_manifest(
    copyright_holders_chr = "Organisation name",
    custom_dmt_ls = ready4fun::make_custom_dmt_ls(user_manual_fns_chr = c("Functions to be included in main user manual are itemised here")),
    dev_pkgs_chr = c("Any development package dependencies go here"),
    path_to_pkg_logo_1L_chr = "Local path to package logo goes here",
    piggyback_to_1L_chr = "GitHub Release Repository to which supporting files will be uploaded",
    ready4_type_1L_chr = "authoring",
    zenodo_badge_1L_chr = "DOI badge details go here"
  )

The main method defined for ready4fun_manifest is author which, assuming the raw undocumented function files are saved in the appropriate directories, will author an R package in which all functions are consistently documented.

## Not run
author(x)

Examples

The ready4fun_manifest sub-module and its methods along with the make_pkg_desc_ls and make_manifestfunctions are designed to be used as part of the ready4pack R package authoring workflow. That vignette includes links to two examples of where the ready4pack workflow has been used to author R package. To illustrate how readyfun tools used as part of that workflow are used to document functions, we are just going to focus on the program used to create the ready4show package.

That program makes use of ready4fun tools that read all undocumented package functions, performs automated checks to ensure that these functions appropriately use the taxonomies and abbreviations mentioned previously (prompting authors to make specific amendments if they do not) and then rewrites these functions to the package R directory, appending tags (with the aid of the sinew package) that will generate meaningful documentation.

For example, one of the functions to be documented is the knit_from_tmpl, which is transformed to a version with tags. The tags added to all functions are then used to generate the package documentation, including the package manual. Two versions of the ready4show package manual are generated - a slimmed down version for end-users and a more detailed inventory of contents intended for developers.

Future documentation

Detailed guidance for how to apply ready4fun workflow tools has yet to be prepared but is planned for 2024.

5.1.3 - Dissemating citable, documented and quality assured model module libraries

ready4 supports tools to streamline the testing, description and distribution of computational model modules.

This below section renders a vignette article from the ready4pack library. You can use the following links to:

ready4pack is a toolkit for bundling collections of modules for computational health economic models authored with the ready4 framework as R packages that are:

  • Citable (with a Zenodo generated DOI and an algorithm generated CITATION file);
  • Community-minded (applying deprecation conventions supported by lifecycle);
  • Documented (applying a function self-documenting algorithm that extends sinew, deploying a GitHub pages hosted and pkgdown generated website and authoring PDF manuals stored in a GitHub Release via piggyback);
  • Internally consistent implementing automated checks to ensure consistency in naming conventions, etc;
  • Licensed (via a usethis generated GPL-3 license);
  • Quality assured (using continuous integration via GitHub actions and R-CMD-Check); and
  • Versioned (applying usethis version increments).

ready4pack extends ready4 framework tools for authoring module algorithms (ready4fun) and data structures (ready4class) and wraps functions from a number of third party R development workflow tools (such as devtools). ready4pack integrates these tools in a common workflow, while adding tools for authoring and documenting datasets to be shipped with model module R packages.

A combination of the ready4_pack_manifest class and author method are used to implement this workflow. This workflow has been used to author all public versions of the ready4 R packages available in the ready4 github repository.

Workflow

Manifest

The main class exported as part of ready4pack is readypack_manifest list based ready4 sub-module, that extends the ready4fun_manifest and ready4class_manifest sub-modules.

Typical usage

readypack_manifest sub-module is most efficiently created with the aid of the make_pt_ready4pack_manifest function and combines instances of the ready4fun_manifest and ready4class_constructor sub-modules.

x <- make_pt_ready4pack_manifest(ready4fun::ready4fun_manifest(),
                                 constructor_r3 = ready4class::ready4class_constructor()) %>%
  ready4pack_manifest()

The main method defined for readypack_manifest is author which extends the author method for ready4class_manifest to author a consistently documented R package.

## Not run
author(x)

Examples

Workflow example one

The program to author and document the ready4show package is relatively simple and authors:

Workflow example two

The program to author and document the youthvars package is a bit more complex as it includes syntax to create package datasets. In addition to the package datasets, the algorithm creates content corresponding to the previous example, specifically:

Future documentation

A more detailed guide to using ready4pack will be created in 2023.

5.2 - Ingest, label and share model data

The ready4use R package provides tools for supplying data to youth mental health computational models.

5.2.1 - Ingest data from an open access repository

A tutorial from the Acumen website about using ready4 to search and retrieve data from the Australian Mental Health Systems Models Dataverse.

This below section renders a R Markdown program from the Acumen website. You can use the following links to:

1. Objectives

On completion of this tutorial you should be able to:

  • Understand basic concepts relating to the Australian Mental Health Systems Models Dataverse Collection; and

  • Have the ability to search for, download and ingest files contained in Dataverse Datasets that are linked to by the Australian Mental Health Systems Models Dataverse Collection using two alternative approaches;

    • Using a web based interface; and
    • Using R commands.

2. Prerequisites

You can complete most of this tutorial without any specialist skills or software other than having a web-browser connected to the Internet. However, if you wish to try running the R code for finding and downloading files described in the last part of the tutorial, then you must have R (and ideally RStudio as well) installed on your machine. Instructions for how to install this software are available at https://rstudio-education.github.io/hopr/starting.html .

3. Concepts

Before searching for or retrieving data from the Australian Mental Health Systems Models Dataverse Collection, the following concepts are useful to understand:

  • The Dataverse Project is “an open source web application to share, preserve, cite, explore, and analyze research data.” It is developed at Harvard’s Institute for Quantitative Social Science (IQSS). More information about the project is available on the Dataverse Project’s website.

  • There are many Dataverse Installations around the world (85 at the time of writing this tutorial). Each Dataverse Installation is an instance of an organisation installing the Dataverse Project’s software on its own servers to create and manage online data repositories. At the time of writing there is one Australian Dataverse Installation listed on the Dataverse Project’s website, which is the Australian Data Archive.

  • The Harvard Dataverse is a Dataverse Installation that is managed by Harvard University, that is open to researchers from all disciplines from anywhere in the world. More details are available from its website.

  • A Dataverse Collection (frequently and confusingly also referred to as simply a “Dataverse”) is a part of a Dataverse Installation that a user can set up to host multiple “Dataverse Datasets” (see next bullet point). Dataverse Collections typically share common attributes (for example, are in the same topic area or produced by the same group(s) of researchers) and can be branded to a limited degree. Dataverse Collections will also contain descriptive metadata about the purpose and ownership of the collection.

  • A Dataverse Dataset is a uniquely identified collection of files (some of which, again confusingly, can be tabular data files of the type that researchers typically refer to as “datasets”) within a Dataverse Collection. Each Dataverse Dataset will have a name, a Digital Object Identifier, a version number, citation information and details of the licensing/terms of use that apply to its contents.

  • A Linked Dataverse Dataset is a Dataverse Dataset that appears in a Dataverse Collection’s list of contents without actually being in that Dataverse Collection (it is hosted in another Dataverse Collection and is potentially owned and controlled by another user).

  • The Australian Mental Health Systems Models Dataverse Collection (which we will refer to as “our Dataverse Collection”) is a Dataverse Collection of Linked Dataverse Datasets within the Harvard Dataverse. We established our Dataverse Collection in the Harvard Dataverse because of the robustness and flexibility that this service provides. A factor in our choice of the Harvard Dataverse was that the aim of our Dataverse Collection is to promote easy access to non-confidential data relevant to modelling Australian mental health policy and service planning topics. The non-confidential nature of the data means that the additional administrative requirements that some other Dataverse Installations place on users were potentially unnecessary for our specific purposes. As a collection of Linked Dataverse Datasets, our Dataverse Collection can be used by modelling groups as both a centralised location to find relevant data and as an additional promotion / distribution channel to share Dataverse Datasets from their own Harvard Dataverse Collections without surrendering any control over the management of their data (they continue to curate their Dataverse Dataset and can modify Dataverse Dataset contents, metadata and terms of use as they see fit).

3. Search and download dataset files

There are multiple options for searching and downloading files contained in our Dataverse Collection. This tutorial will discuss just two - one based on using a web browser and the other based on using R commands. For details on other options, it is recommended to consult the Harvard Dataverse user guide and (for more technical readers) api guide.

3.1. Web browser approach

Searching and retriving data from our Dataverse Collection via a web-browser is very simple, and this methods is suitable for low volume requests (i.e. occasional use) where reproducibility is not important.

To find and download data using your web browser, implement the following steps:

  • Go to our Dataverse Collection at https://dataverse.harvard.edu/dataverse/openmind

  • Search for the Linked Dataverse Dataset most of interest to you by using the tools provided on the landing page.

  • Click on the link to your selected Dataverse Dataset. Note that by doing so you will leave our Dataverse Collection and be taken to the Dataverse Collection controlled by the Dataverse Dataset’s owner.

  • (Optional) - Click on the “Metadata”, “Terms” and “Versions” tabs or (if available) the Related Publication links to discover more about the dataset. When you are done, click on the “Files” tab to review the files contained in the Dataverse Dataset.

  • Select the files that you wish to download using the checkboxes and click on the “Download” button.

  • When prompted, review any terms of use you are presented with and either accept them or cancel the download as you feel appropriate.

More detail on some of the above steps is available in the following section of the Harvard Dataverse user guide: https://guides.dataverse.org/en/latest/user/find-use-data.html#finding-data

3.2 Using R commands

Some limitations of relying purely on a web-browser are that it is a purely manual approach that can become inefficient for large number of data requests and which is not reproducible (thereby limiting transparency about the specific data items / versions used in an analysis). It can therefore be desirable to explore alternatives that are based on programming commands. Programmatic approaches have the advantage of being more readily incorporated into automated and reproducible workflows.

There are a range of software tools in different languages that can be used to programmatically search and retrieve files in Dataverse Collections. More information on these resources on a dedicated page within the Dataverse Project’s documentation.

One of these tools is dataverse - “the R Client for Dataverse Repositories”. The dataverse R package has a range of functions that are very helpful for general tasks relating to the search and retrieval of files contained in Dataverse Datasets. These functions are not the focus of this tutorial, but you can read more about them on the [packages documentation website]((https://iqss.github.io/dataverse-client-r/).

The remainder of this tutorial is focused on the use of another R package called ready4use which created by Orygen to help manage open-source data for use in mental health models. The ready4use R package extends the dataverse R package and one of its applications is to ingest R objects stored in Dataverse Datasets in the “.Rds” file format directly into an R Session’s working memory. More information about ready4use is available on its documentation website.

3.2.1 Install and load required R packages

As ready4use is still a development package, you may need to first install the devtools package to help install it. The following commands entered in your R console will do this.

utils::install.packages("devtools")
devtools::install_github("ready4-dev/ready4use")

We now load the ready4use package and the ready4 framework for youth mental health modelling that it depends on. The ready4 framework will have been automatically installed along with ready4use.

3.2.2 Specify repository details

The next step is to create a Ready4useRepos object, which in this example we will call X, that contains the details of the Dataverse Dataset from which we wish to retrieve R objects. We need to supply three pieces of information to Ready4useRepos. Two of these items of information will be the same for any data item retrieved from our Dataverse Collection and are the Dataverse Collection identifier (which for us is “openmind”) and the server on which the containing Dataverse Installation is hosted (in our case “dataverse.harvard.edu”). The one item of information that will vary based on your requirements is the name / identifier (DOI) of the Dataverse Dataset from which we wish to retrieve data. In this example we are using the DOI for the “Synthetic (fake) youth mental health datasets and data dictionaries” Dataverse Dataset.

X <- Ready4useRepos(dv_nm_1L_chr = "openmind",
                    dv_server_1L_chr = "dataverse.harvard.edu",
                    dv_ds_nm_1L_chr = "https://doi.org/10.7910/DVN/HJXYKQ")

Having supplied the details of where the data is stored we can now ingest the data we are interested in. We can either ingest all R object in the selected Dataverse Dataset or just objects that we specify. By default R objects are ingested along with their metadata, but we can choose not to ingest the metadata.

3.2.3 Ingest all R objects from a Dataverse Dataset along with its metadata

To ingest all R objects in the dataset, we enter the following command.

Y <- ingest(X)

We can now create separate list objects for the ingested data and its metadata.

data_ls <- procureSlot(Y,"b_Ready4useIngest@objects_ls")
meta_ls <- procureSlot(Y,"a_Ready4usePointer@b_Ready4useRepos@dv_ds_metadata_ls$ds_ls")

We can itemise the data objects we have ingested with the following command.

names(data_ls)
#> [1] "ymh_clinical_dict_r3" "eq5d_ds_tb"           "eq5d_ds_dict"         "ymh_clinical_tb"

We can also see what metadata fields we have ingested.

names(meta_ls)
#>  [1] "id"                  "datasetId"           "datasetPersistentId" "storageIdentifier"   "versionNumber"       "versionMinorNumber"  "versionState"        "lastUpdateTime"     
#>  [9] "releaseTime"         "createTime"          "publicationDate"     "citationDate"        "termsOfUse"          "fileAccessRequest"   "metadataBlocks"      "files"

There can be a lot of useful information contained in this metadata list object. For example, we can retrieve descriptive information about the Dataverse Dataset from which we have ingested data.

meta_ls$metadataBlocks$citation$fields$value[[7]]$dsDescriptionValue$value
#> [1] "The datasets in this collection are entirely fake. They were developed principally to demonstrate the workings of a number of utility scoring and mapping algorithms. However, they may be of more general use to others. In some limited cases, some of the included files could be used in exploratory simulation based analyses. However, you should read the metadata descriptors for each file to inform yourself of the validity and limitations of each fake dataset. To open the RDS format files included in this dataset, the R package ready4use needs to be installed (see https://ready4-dev.github.io/ready4use/ ). It is also recommended that you install the youthvars package ( https://ready4-dev.github.io/youthvars/) as this provides useful tools for inspecting and validating each dataset."

The metadata also contains descriptive information on each file in the Dataverse Dataset.

meta_ls$files$description[5]
#> [1] "Annotated program to generate fake clinical youth mental health population for use in demonstrating AQoL-6D utility mapping study algorithms"

3.2.4 Ingest all R objects from a Dataverse Dataset without metadata

If we wished to ingest only the R objects without metadata, we could have simply run the following command.

data_2_ls <- ingest(X,
                    metadata_1L_lgl = F)

We can see that this ingest is identical to that made using the previous method.

identical(data_ls, data_2_ls)
#> [1] TRUE

3.2.5 Ingest selected R objects

If we only want to ingest one specific object, we can supply its name.

ymh_clinical_tb <- ingest(X,
                          fls_to_ingest_chr = c("ymh_clinical_tb"),
                          metadata_1L_lgl = F)

The output from an object specific call to the ingest method is the requested object.

ymh_clinical_tb %>%
  head()
#> # A tibble: 6 × 43
#>   fkClientID    round    d_interview_date d_age d_gender d_sex_birth_s d_sexual_ori_s d_ATSI d_country_bir_s d_english_home d_english_native d_studying_working  d_relation_s s_centre
#>   <chr>         <fct>    <date>           <int> <chr>    <chr>         <fct>          <chr>  <chr>           <chr>          <chr>            <chr>               <chr>        <chr>   
#> 1 Participant_1 Baseline 2020-03-22          14 Male     Male          Heterosexual   No     Australia       Yes            Yes              Not studying or wo… In a relati… Southpo…
#> 2 Participant_2 Baseline 2020-06-15          19 Female   Female        Heterosexual   Yes    Other           No             No               Studying only       In a relati… Regiona…
#> 3 Participant_3 Baseline 2020-08-20          21 Female   Female        Other          NA     NA              NA             NA               Studying only       Not in a re… Canberra
#> 4 Participant_4 Baseline 2020-05-23          12 Female   Female        Heterosexual   Yes    Other           No             No               Not studying or wo… In a relati… Southpo…
#> 5 Participant_5 Baseline 2020-04-05          19 Male     Male          Heterosexual   Yes    Other           No             No               Not studying or wo… Not in a re… Southpo…
#> 6 Participant_6 Baseline 2020-06-09          19 Male     Male          Heterosexual   Yes    Other           No             No               Studying only       In a relati… Regiona…
#> # ℹ 29 more variables: c_p_diag_s <chr>, c_clinical_staging_s <chr>, k6_total <int>, phq9_total <int>, bads_total <int>, gad7_total <int>, oasis_total <int>, scared_total <int>,
#> #   c_sofas <int>, aqol6d_q1 <int>, aqol6d_q2 <int>, aqol6d_q3 <int>, aqol6d_q4 <int>, aqol6d_q5 <int>, aqol6d_q6 <int>, aqol6d_q7 <int>, aqol6d_q8 <int>, aqol6d_q9 <int>,
#> #   aqol6d_q10 <int>, aqol6d_q11 <int>, aqol6d_q12 <int>, aqol6d_q13 <int>, aqol6d_q14 <int>, aqol6d_q15 <int>, aqol6d_q16 <int>, aqol6d_q17 <int>, aqol6d_q18 <int>,
#> #   aqol6d_q19 <int>, aqol6d_q20 <int>

We can also request to ingest multiple specified objects from a Dataverse Dataset.

data_3_ls <- ingest(X,
                    fls_to_ingest_chr = c("ymh_clinical_tb","ymh_clinical_dict_r3"),
                    metadata_1L_lgl = F)

This last request produces a list of ingested objects.

names(data_3_ls)
#> [1] "ymh_clinical_dict_r3" "ymh_clinical_tb"

5.2.2 - Add a data dictionary to a dataset

Pairing a dataset with its dictionary makes it easier to interpret. This tutorial describes how a module from the ready4use R package can help you to pair a dataset and its dictionary.

This below section renders a vignette article from the ready4use library. You can use the following links to:

Note: This vignette is illustrated with fake data. The dataset explored in this example should not be used to inform decision-making.

ready4use includes a number of tools for labeling health economic model data and forms part of the ready4 framework.

Create a dataset-dictionary pair

A data dictionary contains useful metadata about a dataset. To illustrate this point, we can ingest examples of a toy (fake) dataset and its data-dictionary.

objects_ls <- Ready4useRepos(dv_nm_1L_chr = "fakes",
                    dv_ds_nm_1L_chr = "https://doi.org/10.7910/DVN/HJXYKQ",
                    dv_server_1L_chr = "dataverse.harvard.edu") %>%
  ingest(metadata_1L_lgl = F)

Importantly (and a requirement for subsequent steps), the data dictionary we ingest is a ready4use_dictionary object.

class(objects_ls$eq5d_ds_dict)
#> [1] "ready4use_dictionary" "ready4_dictionary"    "tbl_df"               "tbl"                  "data.frame"

We can now pair the data dictionary with its dataset in a new object X, a Ready4useDyad.

X <- Ready4useDyad(ds_tb = objects_ls$eq5d_ds_tb,
                   dictionary_r3 = objects_ls$eq5d_ds_dict)

Inspect data

We can inspect X by printing selected information about it to console using the exhibit method. If we only wish to see the first or last few records, we can pass “head” or “tail” to the display_1L_chr argument.

 exhibit(X,
         display_1L_chr = "head",
         scroll_box_args_ls = list(width = "100%"))
Dataset
uid Timepoint data_collection_dtm d_age Gender d_sex_birth_s d_sexual_ori_s d_relation_s d_ATSI CALD Region d_studying_working eq5dq_MO eq5dq_SC eq5dq_UA eq5dq_PD eq5dq_AD K10_int Psych_well_int
1 BL 2019-10-22 14 Male Male Heterosexual In a relationship No No Metro Not studying or working 1 1 1 1 2 11 87
2 BL 2019-10-17 19 Female Female Heterosexual In a relationship Yes Yes Regional Studying only 1 2 1 1 1 14 65
2 FUP 2020-02-14 19 Female Female Heterosexual In a relationship Yes Yes Regional Studying only 3 1 1 1 1 10 71
3 BL 2020-02-15 21 Female Female Other Not in a relationship NA NA Metro Studying only 1 1 3 1 1 13 74
3 FUP 2020-06-14 21 Female Female Other Not in a relationship NA NA Metro Studying only 1 1 2 1 1 10 64
4 BL 2019-12-14 12 Female Female Heterosexual In a relationship Yes Yes Metro Not studying or working 1 1 1 3 1 18 40

The dataset may be more meaningful if variables are labelled using the descriptive information from the data dictionary. This can be accomplished using the renew method.

X <- renew(X,
           type_1L_chr = "label")
exhibit(X,
        display_1L_chr = "head",
         scroll_box_args_ls = list(width = "100%"))
Dataset
Unique identifier Data collection round Date of data collection Age Gender (grouped) Sex at birth Sexual orientation Relationship status Aboriginal or Torres Strait Islander Culturally And Linguistically Diverse Region of residence (metropolitan or regional) Education and employment status EQ5D - Mobility domain score EQ5D - Self-Care domain score EQ5D - Usual Activities domain score EQ5D - Pain / Discomfort domain score EQ5D - Anxiety / Depression domain score Kessler Psychological Distress - 10 Item Total Score Overall Wellbeing Measure (Winefield et al. 2012)
1 BL 2019-10-22 14 Male Male Heterosexual In a relationship No No Metro Not studying or working 1 1 1 1 2 11 87
2 BL 2019-10-17 19 Female Female Heterosexual In a relationship Yes Yes Regional Studying only 1 2 1 1 1 14 65
2 FUP 2020-02-14 19 Female Female Heterosexual In a relationship Yes Yes Regional Studying only 3 1 1 1 1 10 71
3 BL 2020-02-15 21 Female Female Other Not in a relationship NA NA Metro Studying only 1 1 3 1 1 13 74
3 FUP 2020-06-14 21 Female Female Other Not in a relationship NA NA Metro Studying only 1 1 2 1 1 10 64
4 BL 2019-12-14 12 Female Female Heterosexual In a relationship Yes Yes Metro Not studying or working 1 1 1 3 1 18 40

To remove dataset labels, use the renew method with “unlabel” passed to the type_1L_chr argument.

X <- renew(X,
           type_1L_chr = "unlabel")

By default, the exhibit method will print the dataset part of the Ready4useDyad instance. To inspect the data dictionary, pass “dict” to the type_1L_chr argument.

exhibit(X,
        display_1L_chr = "head",
        type_1L_chr = "dict",
        scroll_box_args_ls = list(width = "100%"))
Data Dictionary
Variable Category Description Class
CALD demographic Culturally And Linguistically Diverse factor
d_age demographic age integer
d_ATSI demographic Aboriginal or Torres Strait Islander character
d_relation_s demographic relationship status character
d_sex_birth_s demographic sex at birth character
d_sexual_ori_s demographic sexual orientation factor

5.2.3 - Share data via online repositories

The retrieval and dissemination of data from online data repositories is an essential enabler of open source modelling. This tutorial describes how a module from the ready4use R package can help you to manage this process.

This below section renders a vignette article from the ready4use library. You can use the following links to:

Note: This vignette is illustrated with fake data. The dataset explored in this example should not be used to inform decision-making.

ready4use includes a number of tools for sharing health economic model data that forms part of the ready4 framework.

Identify data to be shared

To illustrate how to share data using ready4use classes and methods, we will first need some data to publish. In this example, we are going to share X, a Ready4useDyad (a data structure explained in another vignette) that we can create using data ingested from an online repository.

objects_ls <- ingest(Ready4useRepos(dv_nm_1L_chr = "fakes",
                                    dv_ds_nm_1L_chr = "https://doi.org/10.7910/DVN/HJXYKQ",
                                    dv_server_1L_chr = "dataverse.harvard.edu",
                                    gh_repo_1L_chr = "ready4-dev/ready4",
                                    gh_tag_1L_chr = "Documentation_0.0"),
                     fls_to_ingest_chr = c("ymh_clinical_tb","ymh_clinical_dict_r3"),
                     metadata_1L_lgl = F)
X <- Ready4useDyad(ds_tb = objects_ls$ymh_clinical_tb,
                   dictionary_r3 = objects_ls$ymh_clinical_dict_r3) %>%
  renew()

Share data

We now specify where we plan to publish X in Y, a Ready4useRepos object (described in another vignette). Note, you must have write permissions to the repositories you specify in this step. The values entered in this example (the https://doi.org/10.7910/DVN/W95KED dataset from the fakes dataverse will not work for you).

Y <- Ready4useRepos(dv_nm_1L_chr = "fakes", # Replace with values for a dataverse & dataset for which
                    dv_ds_nm_1L_chr = "https://doi.org/10.7910/DVN/W95KED", #  you have write permissions.
                    dv_server_1L_chr = "dataverse.harvard.edu")

We can now upload X to our preferred data repository using the share method. By default, if more than one data repository was specified in Y, then the dataverse repository will be preferred when sharing. We can overwrite this default by passing either “prefer_gh” or “all” values to the type_1L_chr argument. The Ready4useDyad object is now available for download at https://doi.org/10.7910/DVN/W95KED.

Y <- share(Y,
           obj_to_share_xx = X,
           fl_nm_1L_chr = "ymh_clinical_dyad_r4",
           description_1L_chr = "An example of a Ready4useDyad - a dataset (clinical youth mental health, AQoL-6D) and data dictionary pair. Note this example uses fake data.")

5.3 - Author and share reproducible analyses

Tools from the ready4show R package support authoring of programs and subroutines to implement and report analyses with ready4.

5.3.1 - Authoring scientific manuscripts

Tools from the ready4show R package support authoring of scientific summaries of analyses with ready4.

This below section renders a vignette article from the ready4show library. You can use the following links to:

Motivation

Open science workflows should ideally span an unbroken chain between data-ingest to production of a scientific summary such as a manuscript. Such extensive workflows provide an explicit means of linking all content in a scientific summary with the analysis that it reports.

Implementation

ready4show includes a number of classes and methods that help integrate manuscript authoring into a reproducible workflow. These tools are part of the ready4 framework for transparent, reusable and updatable health economic models.

Load required libraries

We first begin by loading the libraries we will require to implement this workflow.

By default, methods in the ready4show package will request your consent before writing files to your machine. This is the safest option. However, as there are many files that need to be written locally for this program to execute, you can overwrite this default by supplying the value “Y” to methods with a consent_1L_chr argument.

consent_1L_chr <- "" # Default value - asks for consent prior to writing each file.

Create a synopsis of the manuscript to be authored

To start with we create X, an instance of Ready4showSynopsis, a ready4 module (S4 class). We can use X to record metadata about the manuscript to be authored (including details about the study being summarised and the title and format of the intended output).

X <- Ready4showSynopsis(background_1L_chr = "Our study is entirely fictional.",
                        coi_1L_chr = "None declared.",
                        conclusion_1L_chr = "These fake results are not interesting.",
                        digits_int = 3L,
                        ethics_1L_chr = "The study was reviewed and granted approval by Awesome University's Human Research Ethics Committee (1111111.1).",
                        funding_1L_chr = "The study was funded by Generous Benefactor.",
                        interval_chr = "three months",
                        keywords_chr = c("entirely","fake","do", "not","cite"),
                        outp_formats_chr = "PDF",
                        sample_desc_1L_chr = "The study sample is fake data that pretends to be young people aged 12 to 25 years who attended Australian primary care services for mental health related needs between November 2019 to August 2020.",
                        title_1L_chr = "A hypothetical study using fake data")

Add authorship details

Authorship details can be added to slots of X that contain ready4show_authors and ready4show_instututes ready4 sub-modules.

As we can see from the below call to exhibitSlot, X was created with no authorship information.

exhibitSlot(X,
            "authors_r3",
            scroll_box_args_ls = list(width = "100%")) 
First-name Middle-name Last-name Title Qualifications Institutes Sequence Position Corresponding Email Joint-first

We can add details on each author by repeated calls to the renewSlot method.

X <- renewSlot(X,
          "authors_r3",
          first_nm_chr = "Alejandra",
          middle_nm_chr = "Rocio",
          last_nm_chr = "Scienceace",
          title_chr = "Dr",
          qualifications_chr = "MD, PhD",
          institute_chr = "Institute_A, Institute_B",
          sequence_int = 1,
          is_corresponding_lgl = T,
          email_chr = "fake_email@fake_institute.com") %>%
  renewSlot("authors_r3",
            first_nm_chr = "Fionn",
            middle_nm_chr = "Seamus",
            last_nm_chr = "Researchchamp",
            title_chr = "Prof",
            qualifications_chr = "MSc, PhD",
            institute_chr = "Institute_C, Institute_B",
            sequence_int = 2,
            email_chr = "fake_email@unreal_institute.com") 

The updated authorship table can now be inspected.

X %>%
  exhibitSlot("authors_r3",
              scroll_box_args_ls = list(width = "100%")) 
First-name Middle-name Last-name Title Qualifications Institutes Sequence Position Corresponding Email Joint-first
Alejandra Rocio Scienceace Dr MD, PhD Institute_A, Institute_B 1 TRUE \_institute.com NA
Fionn Seamus Researchchamp Prof MSc, PhD Institute_C, Institute_B 2 NA \_institute.com NA

We now need to add additional information for each author institute.

X <- renewSlot(X,
          "institutes_r3",
          short_name_chr = "Institute_A", 
          long_name_chr = "Awesome University, Shanghai") %>%
  renewSlot("institutes_r3",
            short_name_chr = "Institute_B", 
            long_name_chr = "August Institution, London") %>%
  renewSlot("institutes_r3",
            new_val_xx = "use_renew_mthd",
            short_name_chr = "Institute_C", 
            long_name_chr = "Highly Ranked Uni, Montreal")

The updated institutes table can now be inspected.

X %>%
  exhibitSlot("institutes_r3",
              scroll_box_args_ls = list(width = "100%")) 
Reference Name
Institute_A Awesome University, Shanghai
Institute_B August Institution, London
Institute_C Highly Ranked Uni, Montreal

Add correspondences

We can also add a look-up table about any changes we wish to make from the analysis code of how names of variables / parameters are presented in the manuscript text.

X <- renewSlot(X,
               "correspondences_r3",
               old_nms_chr = c("PHQ9", "GAD7"),
               new_nms_chr = c("PHQ-9", "GAD-7"))

These edits can now be inspected with a call to exhibitSlot.

X %>%
  exhibitSlot("correspondences_r3",
              scroll_box_args_ls = list(width = "100%")) # Add Exhibit Method
Old name New name
PHQ9 PHQ-9
GAD7 GAD-7

Specify output directory

We now update X with details of the directory to which we wish to write the manuscript we are authoring and all its supporting files.

X <- renewSlot(X,
               "a_Ready4showPaths@outp_data_dir_1L_chr",
               new_val_xx = tempdir())

Create dataset of literate programming files

Our next step is to copy a dataset of files that can implement a literate program to generate our manuscript. If you have a template you wish to work with, you can specify its local path using the a_Ready4showPaths@mkdn_source_dir_1L_chr slot of the X. Skip this step if you wish to use the default markdown dataset, which leverages popular rmarkdown toolkits such as bookdown and rticles.

## Not run
# procureSlot(X,
#             "a_Ready4showPaths@mkdn_source_dir_1L_chr",
#             new_val_xx  = "PATH TO MARKDOWN DATASET")

We create the dataset copy with the authorData method.

authorData(X, consent_1L_chr = consent_1L_chr)

Having created a local copy of the template literate program files dataset, it is now possible to manually edit the markdown files to author the manuscript. However, in this example we are skipping this step and will continue to use the unedited template in conjunction with the metadata we have specified in X. We combine the two to author a manuscript using the authorReport method.

authorReport(X, consent_1L_chr = consent_1L_chr)

If we wish, we can now ammend X and then rerun the authorReport method to generate Word and HTML versions of the manuscript.

renewSlot(X,
          "outp_formats_chr",
          new_val_xx = "Word") %>%
  authorReport(consent_1L_chr = consent_1L_chr)
renewSlot(X,
          "outp_formats_chr",
          new_val_xx = "HTML") %>%
  authorReport(consent_1L_chr = consent_1L_chr)

The outputed files are as follows:

6 - Model

The ready4 software framework is currently being used to develop a modular economic model of youth mental health.

6.1 - Model modules

A ready4 computational model is implemented as a set of modules, all of which are authored with the ready4 software framework. These modules can be re-used and combined to create other computational models.

6.1.1 - Finding modules and sub-modules

How to find individual ready4 modules and sub-modules.

This below section renders a vignette article from the ready4 library. You can use the following links to:

Motivation

When considering whether to use model modules, model developers and users will often want to see examples of each module’s application The ready4 package includes tools to allow a modelling project’s maintainers to perform automated searches for vignette examples of model modules and to output tabular summaries of these modules that are suitable for inclusion on documentation website pages.

Implementation

Details of how to search for themed collections of modules is described in another article. A table itemising individual model modules and sub-modules authored with ready4 can be generated using make_modules_tb. This function scrapes relevant data from the websites of module libraries that have been developed within a specified project’s GitHub organisation and may take a couple of minutes to execute. In this example, we are going to examine modules from a youth mental health model.

modules_tb <-  make_modules_tb(gh_repo_1L_chr = "ready4-dev/ready4")

A slightly quicker method to achieve a similar result is to use the get_modules_tb function. This function retrieves an archived version (and therefore potentially less up to date) of the modules summary table.

# Not run
# modules_tb <- get_modules_tb(gh_repo_1L_chr = "ready4-dev/ready4")

The modules_tb object itemises both ready4 modules (which always use R’s “S4” class type) and sub-modules (“S3” class type). To display a HTML summary of just ready4 modules, you can use the print_modules function.

print_modules(modules_tb, what_1L_chr = "S4")
Class Description Examples
AusACT Meta data for processing ACT population projections
AusHeadspace Meta data for constructing Headspace Centre geometries
AusLookup Lookup tables for Australian geometry and spatial attribute data
AusOrygen Meta data for constructing OYH Specialist Mental Health Catchment geometries
AusProjections Meta data for constructing custom Australian population projections boundary
AusTasmania Meta data for processing Tasmanian population projections
CostlyCorrespondences Collection of input, standards definition and results datasets for projects to generate standardised costing datasets 1, 2
CostlyCountries Collection of input, standards definition and results datasets for projects to generate standardised country data for use in costing datasets 1, 2
CostlyCurrencies Collection of input, standards definition and results datasets for projects to generate standardised currency data for use in costing datasets 2
CostlySeed Original (non-standardised) dataset (and metadata) 1, 2
CostlySource Input dataset (and metadata) for generating standardised costing datasets
CostlyStandards Dataset (and metadata) defining the allowable values of specified variables 1, 2
ScorzAqol6 A dataset and metadata to support implementation of an AQoL-6D scoring algorithm
ScorzAqol6Adol A dataset and metadata to support implementation of a scoring algorithm for the adolescent version of AQoL-6D 3
ScorzAqol6Adult A dataset and metadata to support implementation of a scoring algorithm for the adult version of AQoL-6D
ScorzEuroQol5 A dataset and metadata to support implementation of an EQ-5D scoring algorithm 4
ScorzProfile A dataset to be scored, its associated metadata and details of the scoring instrument
SpecificConverter Container for seed objects used for creating SpecificModels modules 5
SpecificFixed Modelling project dataset, input parameters and complete fixed models results
SpecificInitiator Modelling project dataset, input parameters and empty results placeholder
SpecificMixed Modelling project dataset, input parameters and complete mixed models results
SpecificModels Modelling project dataset, input parameters and model comparison results
SpecificParameters Input parameters that specify candidate models to be explored
SpecificPredictors Modelling project dataset, input parameters and predictor comparison results
SpecificPrivate Analysis outputs not intended for public dissemination
SpecificProject Modelling project dataset, parameters and results
SpecificResults Analysis results
SpecificShareable Analysis outputs intended for public dissemination
SpecificSynopsis Input, Output and Authorship Data For Generating Reports
TTUProject Input And Output Data For Undertaking and Reporting Utility Mapping Studies 6
TTUReports Metadata to produce utility mapping study reports
TTUSynopsis Input, Output and Authorship Data For Generating Utility Mapping Study Reports
VicinityArguments Function arguments for constructing a spatial object
VicinityLocal Object defining data to be saved in local directory
VicinityLocalProcessed Object defining data to be saved in local directory in a processed (R) format
VicinityLocalRaw Object defining data to be saved in local directory in a raw (unprocessed) format
VicinityLookup Look up tables for spatiotemporal data
VicinityMacro Macro level context
VicinityMesoArea Meso level context - area
VicinityMesoRegion Meso level context - region
VicinityMicro Micro level context
VicinityProfile Information to create a profiled area object
VicinitySpaceTime Spatiotemporal environment
YouthvarsDescriptives Metadata about descriptive statistics to be generated
YouthvarsProfile A dataset and its associated dictionary, descriptive statistics and metadata 8
YouthvarsSeries A longitudinal dataset and its associated dictionary, descriptive statistics and metadata 8

You can use the same function to display only ready4 sub-modules.

print_modules(modules_tb, what_1L_chr = "S3")
Class Description Examples
specific_models Candidate models lookup table
specific_predictors Candidate predictors lookup table
vicinity_abbreviations ready4 S3 class for tibble object lookup table for spatial data abbreviations
vicinity_identifiers ready4 S3 class for tibble object lookup table of unique feature identifiers used for different spatial objects
vicinity_mapes ready4 S3 class for tibble object that stores spatial simulation parameters relating to Mean Absolute Prediction Errors
vicinity_parameters ready4 S3 class for tibble object that stores simulation structural parameters relating to the spatial environment
vicinity_points ready4 S3 class for tibble object lookup table of the longitude and latitude cordinates of sites of services / homes
vicinity_processed ready4 S3 class for tibble object lookup table of meta-data for spatial data packs (imported and pre-processed data)
vicinity_raw ready4 S3 class for tibble object lookup table of metadata about raw (un-processed) spatial data to import
vicinity_resolutions ready4 S3 class for tibble object lookup table of the relative resolutions of different spatial objects
vicinity_templates ready4 S3 class for tibble object lookup table for base file used in creation of certain spatial objects
vicinity_values ready4 S3 class for tibble object that stores simulation parameter values for each iteration
youthvars_aqol6d_adol youthvars ready4 sub-module (S3 class) for Assessment of Quality of Life Six Dimension Health Utility - Adolescent Version (AQoL6d Adolescent) 7
youthvars_bads youthvars ready4 sub-module (S3 class) for Behavioural Activation for Depression Scale (BADS) scores 7
youthvars_chu9d_adolaus youthvars ready4 sub-module (S3 class) for Child Health Utility Nine Dimension Health Utility - Australian Adolescent Scoring (CHU-9D Australian Adolescent) 7
youthvars_gad7 youthvars ready4 sub-module (S3 class) for Generalised Anxiety Disorder Scale (GAD-7) scores 7
youthvars_k10 youthvars ready4 sub-module (S3 class) for Kessler Psychological Distress Scale (K10) - US Scoring System scores 7
youthvars_k10_aus youthvars ready4 sub-module (S3 class) for Kessler Psychological Distress Scale (K10) - Australian Scoring System scores 7
youthvars_k6 youthvars ready4 sub-module (S3 class) for Kessler Psychological Distress Scale (K6) - US Scoring System scores 7
youthvars_k6_aus youthvars ready4 sub-module (S3 class)for Kessler Psychological Distress Scale (K6) - Australian Scoring System scores 7
youthvars_oasis youthvars ready4 sub-module (S3 class) for Overall Anxiety Severity and Impairment Scale (OASIS) scores 7
youthvars_phq9 youthvars ready4 sub-module (S3 class) for Patient Health Questionnaire (PHQ-9) scores 7
youthvars_scared youthvars ready4 sub-module (S3 class) for Screen for Child Anxiety Related Disorders (SCARED) scores 7
youthvars_sofas youthvars ready4 sub-module (S3 class) for Social and Occupational Functioning Assessment Scale (SOFAS) 7

6.1.2 - Using ready4 modules

ready4 modules can be be used to model the people, places, platforms and programs that shape young people’s mental health.

6.1.2.1 - Modules for modelling people

Modules to model the characteristics, relationships, behaviours, risk factors and outcomes of young people and individuals who interact with young people are collectively referred to as the “Spring To Life” model. The currently available modules listed here will be supplemented by additional unreleased work in progress.

6.1.2.1.1 - Add metadata to datasets of individual human records

Appending appropriate metadata to datasets of individual unit records can facilitate partial automation of some modelling tasks. This tutorial describes how a module from the youthvars R package can help you to add metadata to a youth mental health dataset so that it can be more readily used by other ready4 modules.

This below section renders a vignette article from the youthvars library. You can use the following links to:

Note: This vignette is illustrated with fake data. The dataset explored in this example should not be used to inform decision-making.

Youthvars provides two ready4 framework modules - YouthvarsProfile and YouthvarsSeries that form part of the ready4 economic model of youth mental health. The ready4 modules in youthvars extend the Ready4useDyad module and can be used to help describe key structural properties of youth mental health datasets.

Ingest data

To start we ingest X, a Ready4useDyad (dataset and data dictionary pair) that we can download from a remote repository.

X <- ready4use::Ready4useRepos(dv_nm_1L_chr = "fakes",
                               dv_ds_nm_1L_chr = "https://doi.org/10.7910/DVN/W95KED",
                               dv_server_1L_chr = "dataverse.harvard.edu") %>%
  ingest(fls_to_ingest_chr = "ymh_clinical_dyad_r4",
         metadata_1L_lgl = F)

Add metadata

If a dataset is cross-sectional or we wish to treat it as if it were (i.e., where data collection rounds are ignored) we can create Y, an instance of the YouthvarsProfile module, to add minimal metadata (the name of the unique identifier variable).

Y <- YouthvarsProfile(a_Ready4useDyad = X, id_var_nm_1L_chr = "fkClientID")

If the temporal dimension of the dataset is important, it may be therefore preferable to instead transform X into a YouthvarsSeries module instance. YouthvarsSeries objects contain all of the fields of YouthvarsProfile objects, but also include additional fields that are specific for longitudinal datasets (e.g. timepoint_var_nm_1L_chr and timepoint_vals_chr that respectively specify the data-collection timepoint variable name and values and participation_var_1L_chr that specifies the desired name of a yet to be created variable that will summarise the data-collection timepoints for which each unit record supplied data).

Z <- YouthvarsSeries(a_Ready4useDyad = X,
                     id_var_nm_1L_chr = "fkClientID",
                     participation_var_1L_chr = "participation",
                     timepoint_vals_chr = c("Baseline","Follow-up"),
                     timepoint_var_nm_1L_chr = "round")

YouthvarsProfile methods

Inspect data

We can now specify the variables that we would like to prepare descriptive statistics for by using the renew method. The variables to be profiled are specified in the profile_chr argument, the number of decimal digits (default = 3) of numeric values in the summary tables to be generated can be specified with nbr_of_digits_1L_int.

Y <- renew(Y, nbr_of_digits_1L_int = 2L, profile_chr = c("d_age","d_sexual_ori_s","d_studying_working"))

We can now view the descriptive statistics we created in the previous step.

Y %>%
  exhibit(profile_idx_int = 1L, scroll_box_args_ls = list(width = "100%"))

We can also plot the distributions of selected variables in our dataset.

depict(Y, var_nms_chr = c("c_sofas"), labels_chr = c("SOFAS"))
SOFAS total scores

SOFAS total scores

YouthvarsSeries methods

Validate data

To explore longitudinal data we need to first use the ratify method to ensure that Z has been appropriately configured for methods examining datasets reporting measures at two timepoints.

Z <- ratify(Z,
            type_1L_chr = "two_timepoints")

Inspect data

We can now specify the variables that we would like to prepare descriptive statistics for using the renew method. The variables to be profiled are specified in arguments beginning with “compare_”. Use compare_ptcpn_chr to compare variables based on whether cases reported data at one or both timepoints and compare_by_time_chr to compare the summary statistics of variables by timepoints, e.g at baseline and follow-up. If you wish these comparisons to report p values, then use the compare_ptcpn_with_test_chr and compare_by_time_with_test_chr arguments.

Z <- renew(Z,
           compare_by_time_chr = c("d_age","d_sexual_ori_s","d_studying_working"),
           compare_by_time_with_test_chr = c("k6_total", "phq9_total", "bads_total"),
           compare_ptcpn_with_test_chr = c("k6_total", "phq9_total", "bads_total")) 

The tables generated in the preceding step can be inspected using the exhibit method.

Z %>%
  exhibit(profile_idx_int = 1L,
          scroll_box_args_ls = list(width = "100%"))
Z %>%
  exhibit(profile_idx_int = 2L,
          scroll_box_args_ls = list(width = "100%"))
Z %>%
  exhibit(profile_idx_int = 3L,
          scroll_box_args_ls = list(width = "100%"))

The depict method can create plots, comparing numeric variables by timepoint.

depict(Z,
       type_1L_chr = "by_time",
       var_nms_chr = c("c_sofas"),
       label_fill_1L_chr = "Time",#
       labels_chr = c("SOFAS"),#
       y_label_1L_chr = "")
SOFAS total scores by data collection round

SOFAS total scores by data collection round

Share data

If and only if the dataset you are working with is appropriate for public dissemination (e.g. is synthetic data), you can use the following workflow for sharing it. We can share the dataset we created for this example using the share method, specifying the repository to which we wish to publish the dataset (and for which we have write permissions) in a (Ready4useRepos object).

A <- Ready4useRepos(gh_repo_1L_chr = "ready4-dev/youthvars", # Replace with your repository 
                          gh_tag_1L_chr = "Documentation_0.0"), # (need write permissions).
A <- share(A,
           obj_to_share_xx = Z,
           fl_nm_1L_chr = "ymh_YouthvarsSeries")

Z is now available for download as the file ymh_YouthvarsSeries.RDS from the “Documentation_0.0” release of the youthvars package.

6.1.2.1.2 - Validate variable total scores

Vector based classes can be used to help validate variable values. This tutorial describes how to do that with sub-module classes exported as part of the youthvars R package.

This below section renders a vignette article from the youthvars library. You can use the following links to:

Variable classes and data integrity

The youthvars package includes a number of ready4 framework sub-module classes that form part of the ready4 economic model of youth mental health. The primary use of youthvars sub-modules is to quality assure the variables used in model input and output datasets by:

  1. facilitating automated data integrity checks that verify no impermissible values (e.g. utility scores greater than one) are present in source data, transformed data or results; and
  2. implementing rules-based automated selection and application of appropriate methods for each dataset variable.

Included sub-module classes

The initial set of sub-module classes included in the youthvars package are one class for Assessment of Quality of Life (Adolescent) health utility and one for each of the predictors used in the utility prediction algorithms included in the related youthu package.

Assessment of Quality of Life Six Dimension (Adolescent) Health Utility

The youthvars_aqol6d_adol class is defined for numeric vectors with a minimum value of 0.03 and maximum value of 1.0.

youthvars_aqol6d_adol(0.4)
#> [1] 0.4
#> attr(,"class")
#> [1] "youthvars_aqol6d_adol" "numeric"
youthvars_aqol6d_adol(c(0.03,0.2,1))
#> [1] 0.03 0.20 1.00
#> attr(,"class")
#> [1] "youthvars_aqol6d_adol" "numeric"

Non numeric objects and values outside these ranges will produce errors.

youthvars_aqol6d_adol("0.5")
#> Error in make_new_youthvars_aqol6d_adol(x): is.numeric(x) is not TRUE
youthvars_aqol6d_adol(-0.1)
#> Error: All non-missing values in valid youthvars_aqol6d_adol object must be greater than or equal to 0.03.
youthvars_aqol6d_adol(1.2)
#> Error: All non-missing values in valid youthvars_aqol6d_adol object must be less than or equal to 1.

Child Health Utility Nine Dimension - Australian Adolescent Scoring

The youthvars_chu9d_adolaus class is defined for numeric vectors with a minimum value of -0.2118 and maximum value of 1.0.

youthvars_chu9d_adolaus(0.4)
#> [1] 0.4
#> attr(,"class")
#> [1] "youthvars_chu9d_adolaus" "numeric"
youthvars_chu9d_adolaus(c(0.03,0.2,1))
#> [1] 0.03 0.20 1.00
#> attr(,"class")
#> [1] "youthvars_chu9d_adolaus" "numeric"

Non numeric objects and values outside these ranges will produce errors.

youthvars_chu9d_adolaus("0.5")
#> Error in make_new_youthvars_chu9d_adolaus(x): is.numeric(x) is not TRUE
youthvars_chu9d_adolaus(-0.3)
#> Error: All non-missing values in valid youthvars_chu9d_adolaus object must be greater than or equal to -0.2118.
youthvars_chu9d_adolaus(1.2)
#> Error: All non-missing values in valid youthvars_chu9d_adolaus object must be less than or equal to 1.

Behavioural Activation for Depression Scale (BADS)

The youthvars_bads class is defined for integer vectors with a minimum value of 0 and maximum value of 150.

youthvars_bads(143L)
#> [1] 143
#> attr(,"class")
#> [1] "youthvars_bads" "integer"
youthvars_bads(as.integer(c(1,15,150)))
#> [1]   1  15 150
#> attr(,"class")
#> [1] "youthvars_bads" "integer"

Non-integers and values outside these ranges will produce errors.

youthvars_bads(22.5)
#> Error in make_new_youthvars_bads(x): is.integer(x) is not TRUE
youthvars_bads(-1L)
#> Error: All non-missing values in valid youthvars_bads object must be greater than or equal to 0.
youthvars_bads(160L)
#> Error: All non-missing values in valid youthvars_bads object must be less than or equal to 150.

Generalised Anxiety Disorder Scale (GAD-7)

The youthvars_gad7 class is defined for integer vectors with a minimum value of 0 and a maximum value of 21.

youthvars_gad7(15L)
#> [1] 15
#> attr(,"class")
#> [1] "youthvars_gad7" "integer"
youthvars_gad7(as.integer(c(0,14,21)))
#> [1]  0 14 21
#> attr(,"class")
#> [1] "youthvars_gad7" "integer"

Non-integers and values outside these ranges will produce errors.

youthvars_gad7(14.6)
#> Error in make_new_youthvars_gad7(x): is.integer(x) is not TRUE
youthvars_gad7(-1L)
#> Error: All non-missing values in valid youthvars_gad7 object must be greater than or equal to 0.
youthvars_gad7(22L)
#> Error: All non-missing values in valid youthvars_gad7 object must be less than or equal to 21.

Kessler Psychological Distress Scale (K6) - Australian Scoring System

The youthvars_k6_aus class is defined for integer vectors with a minimum value of 6 and a maximum value of 30.

youthvars_k6_aus(21L)
#> [1] 21
#> attr(,"class")
#> [1] "youthvars_k6_aus" "integer"
youthvars_k6_aus(as.integer(c(6,13,25)))
#> [1]  6 13 25
#> attr(,"class")
#> [1] "youthvars_k6_aus" "integer"

Non-integers and values outside these ranges will produce errors.

youthvars_k6_aus(11.2)
#> Error in make_new_youthvars_k6_aus(x): is.integer(x) is not TRUE
youthvars_k6_aus(1L)
#> Error: All non-missing values in valid youthvars_k6_aus object must be greater than or equal to 6.
youthvars_k6_aus(31L)
#> Error: All non-missing values in valid youthvars_k6_aus object must be less than or equal to 30.

Kessler Psychological Distress Scale (K6) - US Scoring System

The youthvars_k6 class is defined for integer vectors with a minimum value of 0 and a maximum value of 24.

youthvars_k6(21L)
#> [1] 21
#> attr(,"class")
#> [1] "youthvars_k6" "integer"
youthvars_k6(as.integer(c(0,13,24)))
#> [1]  0 13 24
#> attr(,"class")
#> [1] "youthvars_k6" "integer"

Non-integers and values outside these ranges will produce errors.

youthvars_k6(11.2)
#> Error in make_new_youthvars_k6(x): is.integer(x) is not TRUE
youthvars_k6(-1L)
#> Error: All non-missing values in valid youthvars_k6 object must be greater than or equal to 0.
youthvars_k6(25L)
#> Error: All non-missing values in valid youthvars_k6 object must be less than or equal to 24.

Kessler Psychological Distress Scale (K10) - Australian Scoring System

The youthvars_k10_aus class is defined for integer vectors with a minimum value of 10 and a maximum value of 50.

youthvars_k10_aus(21L)
#> [1] 21
#> attr(,"class")
#> [1] "youthvars_k10_aus" "integer"
youthvars_k10_aus(as.integer(c(13,25,41)))
#> [1] 13 25 41
#> attr(,"class")
#> [1] "youthvars_k10_aus" "integer"

Non-integers and values outside these ranges will produce errors.

youthvars_k10_aus(11.2)
#> Error in make_new_youthvars_k10_aus(x): is.integer(x) is not TRUE
youthvars_k10_aus(9L)
#> Error: All non-missing values in valid youthvars_k10_aus object must be greater than or equal to 10.
youthvars_k10_aus(51L)
#> Error: All non-missing values in valid youthvars_k10_aus object must be less than or equal to 50.

Kessler Psychological Distress Scale (K10) - US Scoring System

The youthvars_k10 class is defined for integer vectors with a minimum value of 0 and a maximum value of 40.

youthvars_k10(21L)
#> [1] 21
#> attr(,"class")
#> [1] "youthvars_k10" "integer"
youthvars_k10(as.integer(c(0,13,34)))
#> [1]  0 13 34
#> attr(,"class")
#> [1] "youthvars_k10" "integer"

Non-integers and values outside these ranges will produce errors.

youthvars_k10(11.2)
#> Error in make_new_youthvars_k10(x): is.integer(x) is not TRUE
youthvars_k10(-1L)
#> Error: All non-missing values in valid youthvars_k10 object must be greater than or equal to 0.
youthvars_k10(41L)
#> Error: All non-missing values in valid youthvars_k10 object must be less than or equal to 40.

Overall Anxiety Severity and Impairment Scale (OASIS)

The youthvars_oasis class is defined for integer vectors with a minimum value of 0 and a maximum value of 20.

youthvars_oasis(15L)
#> [1] 15
#> attr(,"class")
#> [1] "youthvars_oasis" "integer"
youthvars_oasis(as.integer(c(0,12,20)))
#> [1]  0 12 20
#> attr(,"class")
#> [1] "youthvars_oasis" "integer"

Non-integers and values outside these ranges will produce errors.

youthvars_oasis(14.2)
#> Error in make_new_youthvars_oasis(x): is.integer(x) is not TRUE
youthvars_oasis(-1L)
#> Error: All non-missing values in valid youthvars_oasis object must be greater than or equal to 0.
youthvars_oasis(21L)
#> Error: All non-missing values in valid youthvars_oasis object must be less than or equal to 20.

Patient Health Questionnaire (PHQ-9)

The youthvars_phq9 class is defined for integer vectors with a minimum value of 0 and a maximum value of 27.

youthvars_phq9(11L)
#> [1] 11
#> attr(,"class")
#> [1] "youthvars_phq9" "integer"
youthvars_phq9(as.integer(c(0,13,27)))
#> [1]  0 13 27
#> attr(,"class")
#> [1] "youthvars_phq9" "integer"

Non-integers and values outside these ranges will produce errors.

youthvars_phq9(15.2)
#> Error in make_new_youthvars_phq9(x): is.integer(x) is not TRUE
youthvars_phq9(-1L)
#> Error: All non-missing values in valid youthvars_phq9 object must be greater than or equal to 0.
youthvars_phq9(28L)
#> Error: All non-missing values in valid youthvars_phq9 object must be less than or equal to 27.

The youthvars_scared class is defined for integer vectors with a minimum value of 0 and a maximum value of 82.

youthvars_scared(77L)
#> [1] 77
#> attr(,"class")
#> [1] "youthvars_scared" "integer"
youthvars_scared(as.integer(c(0,42,82)))
#> [1]  0 42 82
#> attr(,"class")
#> [1] "youthvars_scared" "integer"

Non-integers and values outside these ranges will produce errors.

youthvars_scared(33.2)
#> Error in make_new_youthvars_scared(x): is.integer(x) is not TRUE
youthvars_scared(-1L)
#> Error: All non-missing values in valid youthvars_scared object must be greater than or equal to 0.
youthvars_scared(83)
#> Error in make_new_youthvars_scared(x): is.integer(x) is not TRUE

Social and Occupational Functioning Assessment Scale (SOFAS)

The youthvars_sofas class is defined for integer vectors with a minimum value of 0 and a maximum value of 100.

youthvars_sofas(44L)
#> [1] 44
#> attr(,"class")
#> [1] "youthvars_sofas" "integer"
youthvars_sofas(as.integer(c(0,23,89)))
#> [1]  0 23 89
#> attr(,"class")
#> [1] "youthvars_sofas" "integer"

Non-integers and values outside these ranges will produce errors.

youthvars_sofas(73.2)
#> Error in make_new_youthvars_sofas(x): is.integer(x) is not TRUE
youthvars_sofas(-1L)
#> Error: All non-missing values in valid youthvars_sofas object must be greater than or equal to 0.
youthvars_sofas(103L)
#> Error: All non-missing values in valid youthvars_sofas object must be less than or equal to 100.

6.1.2.1.3 - Score health utility

Using modules from the scorz R package, individual responses to a multi-attribute utility instrument survey can be converted into health utility total scores. This tutorial describes how to do for adolescent AQoL-6D health utility.

This below section renders a vignette article from the scorz library. You can use the following links to:

Note: This vignette is illustrated with fake data. The dataset explored in this example should not be used to inform decision-making. Some of the methods illustrated in this AQoL-6D vignette can also be used to score other health utility instruments - see a vignette about scoring EQ-5D.

AQoL-6D scoring

To derive a health utility score from the raw responses to a multi-attribute utility instrument it is necessary to implement a scoring algorithm. Scoring algorithms for the Assessment of Quality of Life Six Dimension (AQoL-6D) are publicly available in SPSS format (https://www.aqol.com.au/index.php/scoring-algorithms).

However, to include scoring algorithms in reproducible research workflows, it is desirable to have these algorithms available in open science languages such as R. The scorz package includes ready4 framework model modules of the ready4 youth mental health economic model that provide R implementations of the adult and adolescent versions of the AQoL-6D scoring algorithms.

Ingest data

To begin, we ingest an unscored dataset as an instance of the Ready4useDyad from the ready4use package. In this case we download our data from a remote repository.

X <- ready4use::Ready4useRepos(dv_nm_1L_chr = "fakes",
                               dv_ds_nm_1L_chr = "https://doi.org/10.7910/DVN/W95KED",
                               dv_server_1L_chr = "dataverse.harvard.edu") %>%
  ingest(fls_to_ingest_chr = "ymh_clinical_dyad_r4",
         metadata_1L_lgl = F) 

To make the ingested dataset easier to interpret, we can add labels from the dictionary.

X <- X %>%
  renew(type_1L_chr = "label")

We can now inspect our ingested dataset using the exhibit method.

exhibit(X,
        display_1L_chr = "head",
         scroll_box_args_ls = list(width = "100%"))
Dataset
Unique client identifier Round of data collection Date of data collection Age Gender Sex at birth Sexual orientation Aboriginal or Torres Strait Islander Country Of birth Speaks English at home Native English speaker Education and employment status Relationship status Service centre name Primary diagnosis Clinical stage Kessler Psychological Distress Scale (6 Dimension) Patient Health Questionnaire Behavioural Activation for Depression Scale Generalised Anxiety Disorder Scale Overall Anxiety Severity and Impairment Scale Screen for Child Anxiety Related Disorders Social and Occupational Functioning Assessment Scale Assessment of Quality of Life (6 Dimension) question 1 Assessment of Quality of Life (6 Dimension) question 2 Assessment of Quality of Life (6 Dimension) question 3 Assessment of Quality of Life (6 Dimension) question 4 Assessment of Quality of Life (6 Dimension) question 5 Assessment of Quality of Life (6 Dimension) question 6 Assessment of Quality of Life (6 Dimension) question 7 Assessment of Quality of Life (6 Dimension) question 8 Assessment of Quality of Life (6 Dimension) question 9 Assessment of Quality of Life (6 Dimension) question 10 Assessment of Quality of Life (6 Dimension) question 11 Assessment of Quality of Life (6 Dimension) question 12 Assessment of Quality of Life (6 Dimension) question 13 Assessment of Quality of Life (6 Dimension) question 14 Assessment of Quality of Life (6 Dimension) question 15 Assessment of Quality of Life (6 Dimension) question 16 Assessment of Quality of Life (6 Dimension) question 17 Assessment of Quality of Life (6 Dimension) question 18 Assessment of Quality of Life (6 Dimension) question 19 Assessment of Quality of Life (6 Dimension) question 20
Participant_1 Baseline 2020-03-22 14 Male Male Heterosexual No Australia Yes Yes Not studying or working In a relationship Southport Other 0-1a 8 7 96 6 6 28 69 2 3 1 2 3 1 1 2 4 3 3 4 2 4 2 2 2 2 2 1
Participant_2 Baseline 2020-06-15 19 Female Female Heterosexual Yes Other No No Studying only In a relationship Regional Centre Anxiety 0-1a 13 13 63 12 12 41 58 3 3 1 1 3 2 1 3 2 4 4 3 4 3 1 2 2 2 1 1
Participant_3 Baseline 2020-08-20 21 Female Female Other NA NA NA NA Studying only Not in a relationship Canberra Anxiety 1b 12 17 72 16 12 43 72 2 3 2 5 1 1 1 2 4 5 2 4 2 2 2 1 1 1 1 1
Participant_4 Baseline 2020-05-23 12 Female Female Heterosexual Yes Other No No Not studying or working In a relationship Southport Depression and Anxiety 2-4 17 17 75 12 10 51 88 1 2 1 1 3 3 1 4 4 3 3 3 4 2 1 1 2 1 3 1
Participant_5 Baseline 2020-04-05 19 Male Male Heterosexual Yes Other No No Not studying or working Not in a relationship Southport Depression and Anxiety 0-1a 12 22 82 14 14 51 67 2 2 1 3 5 1 1 1 1 5 4 4 3 2 1 2 1 3 2 3
Participant_6 Baseline 2020-06-09 19 Male Male Heterosexual Yes Other No No Studying only In a relationship Regional Centre Anxiety 1b 11 8 105 8 3 46 60 1 2 2 1 2 2 4 1 3 3 4 3 4 2 1 2 1 2 1 1

We now add meta-data that identifies our dataset as being longitudinal using the YouthvarsSeries module of the youthvars package.

X <- youthvars::YouthvarsSeries(a_Ready4useDyad = X,
                                id_var_nm_1L_chr = "fkClientID",
                                timepoint_var_nm_1L_chr = "round",
                                timepoint_vals_chr = levels(X@ds_tb$round))

We now use the data and meta-data we have created in the previous steps to create an instance of the ScorzAqol6Adol class. This class is specifically designed to facilitate scoring of the adolescent version of the AQoL-6D instrument.

Y <- ScorzAqol6Adol(a_YouthvarsProfile = X)

By default, instances of the ScorzAqol6Adol class are created with a slot specifying a value for the prefix for AQoL-6D questionnaire item responses.

procureSlot(Y,
            slot_nm_1L_chr = "itm_prefix_1L_chr")
#> [1] "aqol6d_q"

If this default value needs to be updated to match the prefix used in your dataset, use the renewSlot method.

# Not run
# Y <- renewSlot(Y, slot_nm_1L_chr = "itm_prefix_1L_chr", new_val_xx = "new_prefix")

Calculating scores

To calculate AQoL 6D adolescent utility scores, use the renew method.

Y <- renew(Y)

Viewing the updated dataset

We can inspect our updated dataset using the exhibit method. We can see that the updated dataset now has additional variables that include the intermediate and final calculations for AQoL-6D adolescent utility scores.

exhibit(Y,
        display_1L_chr = "head",
         scroll_box_args_ls = list(width = "100%"))
Dataset
Unique client identifier Round of data collection Date of data collection Age Gender Sex at birth Sexual orientation Aboriginal or Torres Strait Islander Country Of birth Speaks English at home Native English speaker Education and employment status Relationship status Service centre name Primary diagnosis Clinical stage Kessler Psychological Distress Scale (6 Dimension) Patient Health Questionnaire Behavioural Activation for Depression Scale Generalised Anxiety Disorder Scale Overall Anxiety Severity and Impairment Scale Screen for Child Anxiety Related Disorders Social and Occupational Functioning Assessment Scale Assessment of Quality of Life (6 Dimension) question 1 Assessment of Quality of Life (6 Dimension) question 2 Assessment of Quality of Life (6 Dimension) question 3 Assessment of Quality of Life (6 Dimension) question 4 Assessment of Quality of Life (6 Dimension) question 5 Assessment of Quality of Life (6 Dimension) question 6 Assessment of Quality of Life (6 Dimension) question 7 Assessment of Quality of Life (6 Dimension) question 8 Assessment of Quality of Life (6 Dimension) question 9 Assessment of Quality of Life (6 Dimension) question 10 Assessment of Quality of Life (6 Dimension) question 11 Assessment of Quality of Life (6 Dimension) question 12 Assessment of Quality of Life (6 Dimension) question 13 Assessment of Quality of Life (6 Dimension) question 14 Assessment of Quality of Life (6 Dimension) question 15 Assessment of Quality of Life (6 Dimension) question 16 Assessment of Quality of Life (6 Dimension) question 17 Assessment of Quality of Life (6 Dimension) question 18 Assessment of Quality of Life (6 Dimension) question 19 Assessment of Quality of Life (6 Dimension) question 20 Assessment of Quality of Life (6 Dimension) item disvalue1 Assessment of Quality of Life (6 Dimension) item disvalue2 Assessment of Quality of Life (6 Dimension) item disvalue3 Assessment of Quality of Life (6 Dimension) item disvalue4 Assessment of Quality of Life (6 Dimension) item disvalue5 Assessment of Quality of Life (6 Dimension) item disvalue6 Assessment of Quality of Life (6 Dimension) item disvalue7 Assessment of Quality of Life (6 Dimension) item disvalue8 Assessment of Quality of Life (6 Dimension) item disvalue9 Assessment of Quality of Life (6 Dimension) item disvalue10 Assessment of Quality of Life (6 Dimension) item disvalue11 Assessment of Quality of Life (6 Dimension) item disvalue12 Assessment of Quality of Life (6 Dimension) item disvalue13 Assessment of Quality of Life (6 Dimension) item disvalue14 Assessment of Quality of Life (6 Dimension) item disvalue15 Assessment of Quality of Life (6 Dimension) item disvalue16 Assessment of Quality of Life (6 Dimension) item disvalue17 Assessment of Quality of Life (6 Dimension) item disvalue18 Assessment of Quality of Life (6 Dimension) item disvalue19 Assessment of Quality of Life (6 Dimension) item disvalue20 Disvalue Score for Dimension 1 - Independent Living Disvalue Score for Dimension 2 - Relationships Disvalue Score for Dimension 3 - Mental Health Disvalue Score for Dimension 4 - Coping Disvalue Score for Dimension 5 - Pain Disvalue Score for Dimension 6 - Senses Adult Score Dimension 1 - Independent Living Adult Score Dimension 2 - Relationships Adult Score Dimension 3 - Mental Health Adult Score Dimension 4 - Coping Adult Score Dimension 5 - Pain Adult Score Dimension 6 - Senses Overall score on a 0-1 disvalue scale Overall score on a life-death disutility scale AQoL-6D Adolescent Disutility Score (Untransformed) AQoL-6D Adolescent Disutility Score (Transformed) Instrument utility score Instrument utility score rotated AQOL-6D (weighted total) AQOL-6D (unweighted total)
Participant_1 Baseline 2020-03-22 14 Male Male Heterosexual No Australia Yes Yes Not studying or working In a relationship Southport Other 0-1a 8 7 96 6 6 28 69 2 3 1 2 3 1 1 2 4 3 3 4 2 4 2 2 2 2 2 1 0.073 0.240 0.000 0.040 0.461 0.000 0.000 0.133 0.824 0.330 0.368 0.722 0.055 0.826 0.133 0.2 0.072 0.033 0.024 0.000 0.19334101 0.2964368 0.7312060 0.7708396 0.2619285 0.03009428 0.8066590 0.7035632 0.2687940 0.2291604 0.7380715 0.9699057 0.6436897 0.7286568 0.55838936 0.55838936 0.4416106 0.5078265 0.5698492 46
Participant_10 Baseline 2020-08-05 15 Female Female Other Yes Other No No Studying and working Not in a relationship Canberra Other 0-1a 11 17 34 13 15 38 60 1 2 2 3 5 1 3 3 4 4 3 4 3 3 1 2 2 3 2 1 0.000 0.033 0.041 0.297 1.000 0.000 0.648 0.392 0.824 0.784 0.368 0.722 0.382 0.423 0.000 0.2 0.072 0.223 0.024 0.000 0.27064870 0.7770111 0.8683514 0.6579841 0.1935407 0.13938313 0.7293513 0.2229889 0.1316486 0.3420159 0.8064593 0.8606169 0.7541542 0.8537026 0.74739738 0.74739738 0.2526026 0.3413671 0.3916050 52
Participant_10 Follow-up 2020-11-07 15 Female Female Other Yes Other No No Not studying or working Not in a relationship Regional Centre Depression 1b 7 17 95 14 10 48 64 2 3 2 1 2 2 2 2 2 3 3 5 3 2 3 1 2 2 3 2 0.073 0.240 0.041 0.000 0.074 0.193 0.197 0.133 0.142 0.330 0.368 1.000 0.382 0.057 0.642 0.0 0.072 0.033 0.205 0.187 0.18835933 0.2602305 0.5155772 0.5858738 0.4342728 0.21476953 0.8116407 0.7397695 0.4844228 0.4141262 0.5657272 0.7852305 0.6473112 0.7327563 0.56418597 0.56418597 0.4358140 0.5027214 0.5645345 47
Participant_100 Baseline 2020-07-19 25 Female Female Other Yes Other No No Working only In a relationship Canberra Depression and Anxiety 0-1a 7 0 120 3 0 21 76 1 1 1 1 2 1 2 2 2 2 2 2 5 3 2 1 3 1 1 1 0.000 0.000 0.000 0.000 0.074 0.000 0.197 0.133 0.142 0.097 0.064 0.056 1.000 0.423 0.133 0.0 0.338 0.000 0.000 0.000 0.00000000 0.1433888 0.2505682 0.7769222 0.2866694 0.00000000 1.0000000 0.8566112 0.7494318 0.2230778 0.7133306 1.0000000 0.4558633 0.5160373 0.29587849 0.29587849 0.7041215 0.7390198 0.7978085 36
Participant_1000 Baseline 2020-09-06 16 Male Male Heterosexual Yes Other No No Not studying or working Not in a relationship Canberra Anxiety 0-1a 0 0 128 0 0 0 71 2 1 1 1 1 2 1 2 1 2 2 1 2 3 1 1 1 2 1 1 0.073 0.000 0.000 0.000 0.000 0.193 0.000 0.133 0.000 0.097 0.064 0.000 0.055 0.423 0.000 0.0 0.000 0.033 0.000 0.000 0.02813508 0.1346642 0.1819574 0.3514811 0.0000000 0.01916297 0.9718649 0.8653358 0.8180426 0.6485189 1.0000000 0.9808370 0.2379252 0.2693314 0.08939064 0.08939064 0.9106094 0.9208737 0.9511345 29
Participant_1000 Follow-up 2020-12-20 16 Male Male Heterosexual Yes Other No No Not studying or working Not in a relationship Southport Anxiety 1b 5 0 117 5 1 14 71 2 2 1 1 1 1 2 1 3 1 2 3 2 2 1 1 1 1 2 1 0.073 0.033 0.000 0.000 0.000 0.000 0.197 0.000 0.392 0.000 0.064 0.338 0.055 0.057 0.000 0.0 0.000 0.000 0.024 0.000 0.04719190 0.1002056 0.2658587 0.2080310 0.0000000 0.01111253 0.9528081 0.8997944 0.7341413 0.7919690 1.0000000 0.9888875 0.2228889 0.2523102 0.07926885 0.07926885 0.9207312 0.9297879 0.9576133 31

Creating summary plots

To create plots, we use the depict method.

We can create a list of summary plots by timepoint for all individual items.

plot_ls <- depict(Y, type_1L_chr = "item_by_time")

We can then select a desired item’s summary plot by using its index number.

plot_ls[[1]]
AQoL-6D Item 1 scores by data-collection round

AQoL-6D Item 1 scores by data-collection round

Alternatively, we can generate individual plots by passing the item index number to the var_idcs_int argument of depict.

depict(Y, type_1L_chr = "item_by_time", var_idcs_int = 2L)
AQoL-6D Item 2 scores by data-collection round

AQoL-6D Item 2 scores by data-collection round

We can also plot domain scores by time.

depict(Y, type_1L_chr = "domain_by_time", var_idcs_int = 1L)
AQoL-6D Independet Living Domain weighted scores by data-collection round

AQoL-6D Independet Living Domain weighted scores by data-collection round

Total AQoL-6D scores can also be plotted using the same approach, where var_idcs_int = 1L is used to plot the weighted total distribution and var_idcs_int = 2L is used for plotting the unweighted total.

depict(Y, type_1L_chr = "total_by_time", var_idcs_int = 1L)
AQoL-6D item total weighted scores by data-collection round

AQoL-6D item total weighted scores by data-collection round

Composite plots can be generated as well, though these are not currently optimised to reliably produce quality plots suitable for publication.

depict(Y, type_1L_chr = "comp_item_by_time")
AQoL-6D item responses by data-collection round

AQoL-6D item responses by data-collection round

depict(Y, type_1L_chr = "comp_domain_by_time")
AQoL-6D weighted domain scores by data-collection round

AQoL-6D weighted domain scores by data-collection round

Share output

We can now publicly share our scored dataset and its associated metadata, using Ready4useRepos and its share method as described in a vignette from the ready4use package.

Z <- ready4use::Ready4useRepos(gh_repo_1L_chr = "ready4-dev/scorz", # Replace with details of your repo.
                               gh_tag_1L_chr = "Documentation_0.0") # You must have write permissions.
Z <- share(Z,
           obj_to_share_xx = Y,
           fl_nm_1L_chr = "ymh_ScorzAqol6Adol")

Y is now available for download as the file ymh_ScorzAqol6Adol.RDS from the “Documentation_0.0” release of the scorz package.

6.1.2.1.4 - Explore candidate utility mapping models

Using modules from the specific R package, it is possible to undertake an exploratory utility mapping analysis. This tutorial illustrates a hypotehtical example of exploring how to map to EQ-5D health utility.

This below section renders a vignette article from the specific library. You can use the following links to:

Note: This vignette uses fake data - it is for illustrative purposes only and should not be used to inform decision making. The specific package includes ready4 framework model modules that form part of the ready4 youth mental health economic model. Currently, these modules are not optimised to be used directly, but are instead intended for use in other model modules. For example, the TTU package includes modules that extend specific modules to help implement utility mapping studies. However, to illustrate the main features of specific modules this vignette demonstrates how specific modules could be used independently. In practice, workflow illustrated in this article would probably need to be performed iteratively in order to identify the optimal model types, predictors and covariates and to update default values to ensure model convergence.

By default, modules in the specific package will request your consent before writing files to your machine. This is the safest option. However, as there are many files that need to be written locally for this program to execute, you can overwrite this default by supplying the value “Y” to methods with a consent_1L_chr argument.

consent_1L_chr <- "" # Default value - asks for consent prior to writing each file.

Import data

We start by ingesting our data. As this example uses EQ-5D data, we import a ScorzEuroQol5 ready4 framework module (created using the steps described in this vignette from the scorz pacakge) into a SpecificConverter Module and then apply the metamorphose method to convert it into a SpecificModel module.

X <- SpecificConverter(a_ScorzProfile = ready4use::Ready4useRepos(gh_repo_1L_chr = "ready4-dev/scorz", 
                                                                  gh_tag_1L_chr = "Documentation_0.0") %>%
                         ingest(fls_to_ingest_chr = "ymh_ScorzEuroQol5",  metadata_1L_lgl = F)) %>% 
  metamorphose() 
class(X)
#> [1] "SpecificModels"
#> attr(,"package")
#> [1] "specific"

Inspect data

The dataset we are using has a total of 1786 records at two timepoints on 1068 study participants. The first six records are reproduced below.

Dataset
Unique identifier Data collection round Date of data collection Age Gender (grouped) Sex at birth Sexual orientation Relationship status Aboriginal or Torres Strait Islander Culturally And Linguistically Diverse Region of residence (metropolitan or regional) Education and employment status EQ5D - Mobility domain score EQ5D - Self-Care domain score EQ5D - Usual Activities domain score EQ5D - Pain / Discomfort domain score EQ5D - Anxiety / Depression domain score Kessler Psychological Distress - 10 Item Total Score Overall Wellbeing Measure (Winefield et al. 2012) EuroQol (EQ-5D) - (weighted total) EuroQol (EQ-5D) - (unweighted total)
1 BL 2019-10-22 14 Male Male Heterosexual In a relationship No No Metro Not studying or working 1 1 1 1 2 11 87 0.879 6
2 BL 2019-10-17 19 Female Female Heterosexual In a relationship Yes Yes Regional Studying only 1 2 1 1 1 14 65 0.846 6
2 FUP 2020-02-14 19 Female Female Heterosexual In a relationship Yes Yes Regional Studying only 3 1 1 1 1 10 71 0.850 7
3 BL 2020-02-15 21 Female Female Other Not in a relationship NA NA Metro Studying only 1 1 3 1 1 13 74 0.883 7
3 FUP 2020-06-14 21 Female Female Other Not in a relationship NA NA Metro Studying only 1 1 2 1 1 10 64 0.906 6
4 BL 2019-12-14 12 Female Female Heterosexual In a relationship Yes Yes Metro Not studying or working 1 1 1 3 1 18 40 0.796 7

To source dataset of X is contained in the a_YouthvarsProfile slot and is a YouthvarsSeries module. For more information about methods that can be used to explore this dataset, read this vignette from the youthvars package.

Specify parameters

In preparation for exploring our dataset, we need to declare a set of model parameters in a b_SpecificParameters slot of X. This can be done in one step, or in sequential steps. In this example, we will proceed sequentially.

Dependent variable

The dependent variable (total EQ-5D utility score) has already been specified when we imported the data from the ScorzEuroQol5 module.

procureSlot(X, "b_SpecificParameters@depnt_var_nm_1L_chr")
#> [1] "eq5d_total_w"

We can now add details of the allowable range of dependent variable values.

X <- renewSlot(X, "b_SpecificParameters@depnt_var_min_max_dbl", c(-1,1))

Candidate predictors

We can now specify the names of candidate predictor variables.

X <- renewSlot(X, "b_SpecificParameters@candidate_predrs_chr", c("K10_int","Psych_well_int")) 

We next add meta-data about each candidate predictor variable in the form of a specific_predictors object.

X <- renewSlot(X, "b_SpecificParameters@predictors_lup", class_chr = "integer", class_fn_chr = c("youthvars::youthvars_k10_aus","as.integer"), covariate_lgl = F, increment_dbl = 1,
               long_name_chr = c("Kessler Psychological Distress - 10 Item Total Score", "Overall Wellbeing Measure (Winefield et al. 2012)"), max_val_dbl = c(50,90), min_val_dbl = c(10,18), mdl_scaling_dbl = 0.01,
               short_name_chr = c("K10_int","Psych_well_int"))

The specific_predictors object that we have added to X can be inspected using the exhibitSlot method.

exhibitSlot(X, "b_SpecificParameters@predictors_lup", scroll_box_args_ls = list(width = "100%"))
Variable Description Minimum Maximum Class Increment Function Scaling Covariate
K10_int Kessler Psychological Distress - 10 Item Total Score 10 50 integer 1 youthvars::youthvars_k10_aus 0.01 FALSE
Psych_well_int Overall Wellbeing Measure (Winefield et al. 2012) 18 90 integer 1 as.integer 0.01 FALSE

Covariates

We also specify the covariates that we aim to explore in conjunction with each candidate predictor.

X <- renewSlot(X, "b_SpecificParameters@candidate_covars_chr", c("d_sex_birth_s", "d_age",  "d_sexual_ori_s", "d_studying_working"))

Descriptive variables

We also specify variables that we will use for generating descriptive statistics about the dataset.

X <- renewSlot(X,"b_SpecificParameters@descv_var_nms_chr", c("d_age","Gender","d_relation_s", "d_sexual_ori_s", "Region", "d_studying_working")) 

Temporal variables

The name of the dataset variable for data collection timepoint and all of its unique values were imported when converting the ScorzEuroQol5 module.

procureSlot(X,"a_YouthvarsProfile@timepoint_var_nm_1L_chr")
#> [1] "Timepoint"
procureSlot(X,"a_YouthvarsProfile@timepoint_vals_chr")
#> [1] "BL"  "FUP"

However, we also need to specify the name of the variable that contains the datestamp for each dataset record.

X <- renewSlot(X, "b_SpecificParameters@msrmnt_date_var_nm_1L_chr", "data_collection_dtm")

Candidate models

X was created with a default set of candidate models, stored as a specific_models sub-module, which can be inspected using the exhibitSlot method.

exhibitSlot(X, "b_SpecificParameters@candidate_mdls_lup", scroll_box_args_ls = list(width = "100%"))
Model types lookup table
Reference Name Control Familty Function Start Predict Transformation Binomial Acronym (Fixed) Acronymy (Mixed) Type (Mixed) With
OLS_NTF Ordinary Least Squares (no transformation) NA NA lm NA NA NTF FALSE OLS LMM linear mixed model no transformation
OLS_LOG Ordinary Least Squares (log transformation) NA NA lm NA NA LOG FALSE OLS LMM linear mixed model log transformation
OLS_LOGIT Ordinary Least Squares (logit transformation) NA NA lm NA NA LOGIT FALSE OLS LMM linear mixed model logit transformation
OLS_LOGLOG Ordinary Least Squares (log log transformation) NA NA lm NA NA LOGLOG FALSE OLS LMM linear mixed model log log transformation
OLS_CLL Ordinary Least Squares (complementary log log transformation) NA NA lm NA NA CLL FALSE OLS LMM linear mixed model complementary log log transformation
GLM_GSN_LOG Generalised Linear Model with Gaussian distribution and log link NA gaussian(log) glm -0.1,-0.1 response NTF FALSE GLM GLMM generalised linear mixed model Gaussian distribution and log link
BET_LGT Beta Regression Model with Binomial distribution and logit link betareg::betareg.control NA betareg::betareg -0.5,-0.1,3 response NTF FALSE GLM GLMM generalised linear mixed model Binomial distribution and logit link
BET_CLL Beta Regression Model with Binomial distribution and complementary log log link betareg::betareg.control NA betareg::betareg -0.5,-0.1,3 response NTF FALSE GLM GLMM generalised linear mixed model Binomial distribution and complementary log log link

We can choose to select just a subset of these to explore using the renewSlot method. As this is an illustrative example, we have restricted the models we will explore to just four types, passing the relevant row numbers to the slice_indcs_int argument.

X <- renewSlot(X, "b_SpecificParameters@candidate_mdls_lup", slice_indcs_int = c(1L,5L,7L,8L))

Other parameters

Depending on the type of analysis we plan on undertaking, we can also specify parameters such as the number of folds to use in cross validation, the maximum number of model runs to allow and a seed to ensure reproducibility of results. In this case we are going to use the default values generated when we first created X.

procureSlot(X, "b_SpecificParameters@folds_1L_int")
#> [1] 10
procureSlot(X, "b_SpecificParameters@max_mdl_runs_1L_int")
#> [1] 300
procureSlot(X, "b_SpecificParameters@seed_1L_int")
#> [1] 1234

Model testing

Before we start to use the data stored in X to undertake modelling, we must first validate that it contains all necessary (and internally consistent) data by using the ratify method. The call to ratify will update any variable names that are likely to cause problems when generating reports (e.g. through inclusion of characters like “_” in the variable name that can cause problems when rendering LaTeX documents).

X <- ratify(X)

Set-up workspace

We add details of the directory to which we will write all output. In this example we create a temporary directory (tempdir()), but in practice this would be an existing directory on your local machine.

X <- renewSlot(X, "paths_chr", tempdir())

It can be useful to save fake data (useful for demonstrating the generalisability and replicability of an analysis) and real data (required for write-up and reproducibility) is distinctly labelled directories. By default, X is created with a flag to save all output in a sub-directory “Real”. As we are using fake data, we can override this value.

X <- renewSlot(X, "b_SpecificParameters@fake_1L_lgl", T)

We can now write a number of sub-directories to our specified output directory.

X <- author(X, what_1L_chr = "workspace", consent_1L_chr = consent_1L_chr)
#> New directories created:
#> C:\Users\mham0053\AppData\Local\Temp\RtmpQBtivk/Fake
#> C:\Users\mham0053\AppData\Local\Temp\RtmpQBtivk/Fake/Markdown
#> C:\Users\mham0053\AppData\Local\Temp\RtmpQBtivk/Fake/Output
#> C:\Users\mham0053\AppData\Local\Temp\RtmpQBtivk/Fake/Reports
#> C:\Users\mham0053\AppData\Local\Temp\RtmpQBtivk/Fake/Output/_Descriptives
#> C:\Users\mham0053\AppData\Local\Temp\RtmpQBtivk/Fake/Output/H_Dataverse

Descriptives

The first set of outputs we write to our output directories is a set of descriptive tables and plots.

X <- author(X, consent_1L_chr = consent_1L_chr, digits_1L_int = 3L,  what_1L_chr = "descriptives")

Model comparisons

The investigate method can now be used to compare the candidate models we have specified earlier. In so doing it will transform X into a SpecificPredictors object.

X <- investigate(X, consent_1L_chr = consent_1L_chr, depnt_var_max_val_1L_dbl = 0.99, session_ls = sessionInfo())
class(X)
#> [1] "SpecificPredictors"
#> attr(,"package")
#> [1] "specific"

The investigate method will write each model to be tested to a new sub-directory of our output directory.

The investigate method also outputs a table summarising the performance of each of the candidate models.

exhibit(X, what_1L_chr = "mdl_cmprsn", type_1L_chr = "results") 
Comparison of candidate models using highest correlated predictor

Training model fit (averaged over 10 folds)

Testing model fit (averaged over 10 folds)

Model R-Squared RMSE MAE R-Squared RMSE MAE
Beta Regression Model with Binomial distribution and logit link 0.4318533 0.0742448 0.0587307 0.4128497 0.0741236 0.0587733
Beta Regression Model with Binomial distribution and complementary log log link 0.4174181 0.0751836 0.0593447 0.3996947 0.0750880 0.0594047
Ordinary Least Squares (no transformation) 0.4106104 0.0756222 0.0596955 0.3933147 0.0755461 0.0597672
Ordinary Least Squares (complementary log log transformation) 0.4105040 0.0756284 0.0597793 0.3913360 0.0755268 0.0598295

We can now identify the highest performing model in each category of candidate model based on the testing R2 statistic.

procure(X, what_1L_chr = "prefd_mdls") 
#> [1] "BET_LGT" "OLS_NTF"

We can override these automated selections and instead incorporate other considerations (possibly based on judgments informed by visual inspection of the plots and the desirability of constraining predictions to a maximum value of one). We do this in the following command, specifying new preferred model types, in descending order of preference.

X <- renew(X, new_val_xx = c("BET_LGT", "OLS_CLL"), type_1L_chr = "results", what_1L_chr = "prefd_mdls")

Use most preferred model to compare all candidate predictors

We can now compare all of our candidate predictors (with and without candidate covariates) using the most preferred model type.

X <- investigate(X, consent_1L_chr = consent_1L_chr)
class(X)
#> [1] "SpecificFixed"
#> attr(,"package")
#> [1] "specific"

Now, we compare the performance of single predictor models of our preferred model type (in our case, a Beta Regression Model with Binomial distribution and logit link) for each candidate predictor. The last call to the investigate saved the tested models along with model plots in a sub-directory of our output directory. These results are also viewable as a table.

exhibit(X, scroll_box_args_ls = list(width = "100%"), type_1L_chr = "results", what_1L_chr = "predr_cmprsn")
Comparison of all candidate predictors using preferred model
predr_chr %IncMSE IncNodePurity
K10 0.0066197 3.888246
Psychwell 0.0011094 2.342784

The most recent call to the investigate method also saved single predictor R model objects (one for each candidate predictors) along with the two plots for each model in a sub-directory of our output directory. The performance of each single predictor model can also be summarised in a table.

exhibit(X, type_1L_chr = "results", what_1L_chr = "fxd_sngl_cmprsn")
Preferred single predictor model performance by candidate predictor

Training model fit (averaged over 10 folds)

Testing model fit (averaged over 10 folds)

Model R-Squared RMSE MAE R-Squared RMSE MAE
K10 0.4318533 0.0742448 0.0587307 0.4128497 0.0741236 0.0587733
Psychwell 0.1507472 0.0907813 0.0699606 0.1341090 0.0909203 0.0700686

Updated versions of each of the models in the previous step (this time with covariates added) are saved to a new subdirectory of the output directory and we can summarise the performance of each of the updated models, along with all signficant model terms, in a table.

exhibit(X, scroll_box_args_ls = list(width = "100%"), type_1L_chr = "results", what_1L_chr = "fxd_full_cmprsn")

We can now identify which, if any, of the candidate covariates we previously specified are significant predictors in any of the models.

procure(X, type_1L_chr = "results", what_1L_chr = "signt_covars")
#> [1] NA

We can override the covariates to select, potentially because we want to select only covariates that are significant for all or most of the models. However, in the below example we have opted not to do so and continue to use no covariates as selected by the algorithm in the previous step.

# X <- renew(X, new_val_xx = c("COVARIATE OF YOUR CHOICE", "ANOTHER COVARIATE"), type_1L_chr = "results", what_1L_chr = "prefd_covars")

Test preferred model with preferred covariates for each candidate predictor

We now conclude our model testing by rerunning the previous step, except confining our covariates to those we prefer.

X <- investigate(X, consent_1L_chr = consent_1L_chr)
class(X)
#> [1] "SpecificMixed"
#> attr(,"package")
#> [1] "specific"

The previous call to the write_mdls_with_covars_cmprsn function saves the tested models along with two plots for each model in the “E_Predrs_W_Covars_Sngl_Mdl_Cmprsn” sub-directory of “Output”.

Apply preferred model types and predictors to longitudinal data

The next main step is to use the preferred model types and covariates identified from the preceding analysis of cross-sectional data in longitudinal analysis.

Longitudinal mixed modelling

Prior to undertaking longitudinal mixed modelling, we need to check the appropriateness of the default values for modelling parameters that are stored in X. These include the number of model iterations, and any custom control parameters and priors (by default, empty lists).

procureSlot(X, "b_SpecificParameters@iters_1L_int")
#> [1] 4000

In many cases there will be no need to specify any custom control parameters or priors and using the defaults may speed up execution.

procureSlot(X, "b_SpecificParameters@control_ls")
#> [[1]]
#> list()
procureSlot(X,"b_SpecificParameters@prior_ls")
#> [[1]]
#> list()

However, in this example using the default control parameters would result in warning messages suggesting a change to the adapt_delta control value (default = 0.8). Modifying the adapt_delta control parameter value can address this issue.

X <- renewSlot(X, "b_SpecificParameters@control_ls", new_val_xx = list(adapt_delta = 0.99))
X <- investigate(X, consent_1L_chr = consent_1L_chr)
class(X)
#> [1] "SpecificMixed"
#> attr(,"package")
#> [1] "specific"

The last call to investigate function wrote the models it tests to a sub-directory of the output directory along with plots for each model.

Create shareable outputs

The model objects created by the preceding analysis are not suitable for sharing as they contain duplicates of the source dataset. To create model objects that can be shared (where dataset copies are replaced with fake data) use the authorData method.

X <- authorData(X, consent_1L_chr = consent_1L_chr)

Purge dataset copies

For the purposes of efficient computation, multiple objects containing copies of the source dataset were saved to our output directory during the analysis process. We therefore need to delete all of these copies by supplying “purge_write” to the type_1L_chr argument of the author method.

X <- author(X, consent_1L_chr = consent_1L_chr, type_1L_chr = "purge_write")

A copy of the module X is available for download as the file eq5d_ttu_SpecificMixed.RDS from the “Documentation_0.0” release of the specific package.

6.1.2.1.5 - Implement a utility mapping study

Using modules from the TTU R package, it is possible to implement a fully reproducible utility mapping study. This tutorial illustrates the main steps using a hypothetical AQoL-6D utility mapping study.

This below section renders a vignette article from the TTU library. You can use the following links to:

Note: This vignette uses fake data - it is for illustrative purposes only and should not be used to inform decision making. This vignette outlines the workflow for developing utility mapping models using longitudinal data. The workflow for developing utility mapping models is broadly similar, with some minor modifications. An example of developing models using cross-sectional data is available at https://doi.org/10.5281/zenodo.8098595 .

Motivation

Health services do not typically collect health utility data from their clients, which makes it more difficult to place an economic values on outcomes attained in these services. One strategy for addressing this gap is to use data from similar samples of patients that contain both health utility and the types of outcome measures that are collected in clinical services. The TTU package provides a toolkit for conducting and reporting a utility mapping (or Transfer to Utility) study.

Implementation

The TTU package contains modules of the ready4 youth mental health economic model that combine and extend model modules for:

  • labeling, validating and summarising youth mental health datasets (from the youthvars package);
  • scoring health utility (from the scorz package);
  • specifying and testing statistical models (from the specific package);
  • generating reproducible analysis reports (from the ready4show package); and
  • sharing data via online data repositories (from the ready4use package).

Additionally, TTU relies on two RMarkdown programs:

Outputs generated by the TTU package are designed to be compatible with health economic models developed with the ready4 framework).

Workflow

Background and citation

The following workflow illustrates (using fake data) the same steps we used in a real world study, a summary of which is available at https://doi.org/10.1101/2021.07.07.21260129). Citation information for that study is:

@article {Hamilton2021.07.07.21260129,
    author = {Hamilton, Matthew P and Gao, Caroline X and Filia, Kate M and Menssink, Jana M and Sharmin, Sonia and Telford, Nic and Herrman, Helen and Hickie, Ian B and Mihalopoulos, Cathrine and Rickwood, Debra J and McGorry, Patrick D and Cotton, Sue M},
    title = {Predicting Quality Adjusted Life Years in young people attending primary mental health services},
    elocation-id = {2021.07.07.21260129},
    year = {2021},
    doi = {10.1101/2021.07.07.21260129},
    publisher = {Cold Spring Harbor Laboratory Press},
    URL = {https://www.medrxiv.org/content/early/2021/07/12/2021.07.07.21260129},
    eprint = {https://www.medrxiv.org/content/early/2021/07/12/2021.07.07.21260129.full.pdf},
    journal = {medRxiv}
}

The program applied in that study, which this workflow closely resembles is available at https://doi.org/10.5281/zenodo.6116077 and can be cited as follows:

@software{hamilton_matthew_2022_6212704,
  author       = {Hamilton, Matthew and
                  Gao, Caroline},
  title        = {{Complete study program to reproduce all steps from 
                   data ingest through to results dissemination for a
                   study to map mental health measures to AQoL-6D
                   health utility}},
  month        = feb,
  year         = 2022,
  note         = {{Matthew Hamilton and Caroline Gao  (2022). 
                   Complete study program to reproduce all steps from
                   data ingest through to results dissemination for a
                   study to map mental health measures to AQoL-6D
                   health utility. Zenodo.
                   https://doi.org/10.5281/zenodo.6116077. Version
                   0.0.9.3}},
  publisher    = {Zenodo},
  version      = {0.0.9.3},
  doi          = {10.5281/zenodo.6212704},
  url          = {https://doi.org/10.5281/zenodo.6212704}
}

Load required packages

We begin by loading our required packages.

By default, methods associated with TTU modules will request your consent before writing files to your machine. This is the safest option. However, as there are many files that need to be written locally for this program to execute, you can overwrite this default by supplying the value “Y” to methods with a consent_1L_chr argument.

consent_1L_chr <- "" # Default value - asks for consent prior to writing each file.

Add dataset metadata

We use the Ready4useDyad and Ready4useRepos modules to retrieve and ingest and to then pair a dataset and its data dictionary.

A <- Ready4useDyad(ds_tb = Ready4useRepos(dv_nm_1L_chr = "fakes", dv_ds_nm_1L_chr = "https://doi.org/10.7910/DVN/HJXYKQ", dv_server_1L_chr = "dataverse.harvard.edu") %>%
                     ingest(fls_to_ingest_chr = c("ymh_clinical_tb"), metadata_1L_lgl = F) %>% youthvars::transform_raw_ds_for_analysis(),
                   dictionary_r3 = Ready4useRepos(dv_nm_1L_chr = "TTU", dv_ds_nm_1L_chr = "https://doi.org/10.7910/DVN/DKDIB0", dv_server_1L_chr = "dataverse.harvard.edu") %>%
                     ingest(fls_to_ingest_chr = c("dictionary_r3"), metadata_1L_lgl = F)) %>%
  renew(type_1L_chr = "label")

We use the YouthvarsSeries module to supply metadata about our longitudinal dataset vignette.

A <- YouthvarsSeries(a_Ready4useDyad = A, id_var_nm_1L_chr = "fkClientID", timepoint_var_nm_1L_chr = "round",
                     timepoint_vals_chr = levels(procureSlot(A, "ds_tb")$round))

Score health utility

We next use the ScorzAqol6Adol module to score adolescent AQoL-6D health utility.

A <- TTUProject(a_ScorzProfile = ScorzAqol6Adol(a_YouthvarsProfile = A))
A <- renew(A, what_1L_chr = "utility") 
#> Joining with `by = join_by(fkClientID, match_var_chr)`

Evaluate candidate models

Over the next few steps we will use modules from the specific package to specify and assess a number of candidate utility mapping models.

Specify modelling parameters

We begin by specifying the parameters we will use in our modelling project. The initial step is to ensure the fields in A for storing parameter values are internally consistent with the data we have entered in the previous steps.

A <- renew(A, what_1L_chr = "parameters")

We next ingest a lookup table of metadata about the variables we plan to explore as candidate predictors. In this case, we are sourcing the lookup table from an online data repository.

A <- renew(A, "use_renew_mthd", fl_nm_1L_chr = "predictors_r3", type_1L_chr = "predictors_lup", 
           y_Ready4useRepos = Ready4useRepos(dv_nm_1L_chr = "TTU", dv_ds_nm_1L_chr = "https://doi.org/10.7910/DVN/DKDIB0", 
                                             dv_server_1L_chr = "dataverse.harvard.edu"),
           what_1L_chr = "parameters")

We can inspect the metadata on candidate predictors that we have just ingested.

exhibit(A, scroll_box_args_ls = list(width = "100%"))

We add additional metadata about variables in our dataset that will be used in exploratory modelling.

A <- renew(A, c(0.03,1), type_1L_chr = "range", what_1L_chr = "parameters") %>%
  renew(c("BADS","GAD7", "K6", "OASIS", "PHQ9", "SCARED"),
        type_1L_chr = "predictors_vars", what_1L_chr = "parameters") %>%
  renew(c("d_sex_birth_s", "d_age",  "d_sexual_ori_s", "d_studying_working", "c_p_diag_s", "c_clinical_staging_s", "SOFAS"),     
        type_1L_chr = "covariates", what_1L_chr = "parameters") %>%
  renew(c("d_age","Gender","d_relation_s", "d_sexual_ori_s" ,"Region", "d_studying_working", "c_p_diag_s", "c_clinical_staging_s","SOFAS"), 
        type_1L_chr = "descriptives", what_1L_chr = "parameters") %>%
  renew("d_interview_date", type_1L_chr = "temporal", what_1L_chr = "parameters")

We record that the data we are working with is fake (this step can be skipped if working with real data).

A <- renew(A, T, type_1L_chr = "is_fake", what_1L_chr = "parameters")

We update A for internal consistency with the values we have previously supplied and create a local workspace to which output files will be written.

A <- renew(A, consent_1L_chr = consent_1L_chr, paths_chr = tempdir(), what_1L_chr = "project")

We now generate tables and charts that describe our dataset. These are saved in a sub-directory of our output data directory, and are available for download. One of the plots is also reproduced here.

A <- author(A, consent_1L_chr = consent_1L_chr, digits_1L_int = 3L, what_1L_chr = "descriptives")

We next compare the performance of different model types. We perform this step using the investigate method. This is the first of several times that we use this method. Each time the method is called A is updated to that the next time the method is called, a different algorithm will be used. The sequence of calls to investigate is therefore important (it should be in the same order as outlined in this example and you should not attempt to repeat a call to investigate to redo a prior step).

A <- investigate(A, consent_1L_chr = consent_1L_chr, depnt_var_max_val_1L_dbl = 0.9999, session_ls = sessionInfo())

The outputs of the previous command are saved into a sub-directory of our output directory. An example of this output is available for download). Once we inspect this output, we can then specify the preferred model types to use from this point onwards.

A <- renew(A, c("GLM_GSN_LOG", "OLS_CLL"), type_1L_chr = "models", what_1L_chr = "results")

Next we assess multiple versions of our preferred model type - one single predictor model for each of our candidate predictors and the same models with candidate covariates added.

A <- investigate(A, consent_1L_chr = consent_1L_chr)

The previous step saved output into a sub-directory of our output directory. Example output is available for download: (single predictor comparisons) and multivariate model comparisons. After reviewing this output, we can specify the covariates we wish to add to the models we will assess from this point forward.

A <- renew(A, "SOFAS", type_1L_chr = "covariates", what_1L_chr = "results")

We can now assess the multivariate models.

A <- investigate(A, consent_1L_chr = consent_1L_chr)

As a result of the previous step, more model objects and plot files have been saved to a sub-directory of our output directory. Examples of this output are available for download here and here. Once we inspect this output we can reformulate the models we finalised in the previous step so that they are suitable for modelling longitudinal change. For our primary analysis, we use a mixed model formulation of the models that we previously selected. A series of large model files are written to the local output data directory.

A <- investigate(A, consent_1L_chr = consent_1L_chr)

For our secondary analyses, we specify alternative combinations of predictors and covariates.

A <- investigate(A, consent_1L_chr = consent_1L_chr,
                 scndry_anlys_params_ls = make_scndry_anlys_params(candidate_predrs_chr = c("SOFAS"),
                                                                   candidate_covar_nms_chr = c("d_sex_birth_s", "d_age", "d_sexual_ori_s", "d_studying_working"),
                                                                   prefd_covars_chr = NA_character_) %>%
                   make_scndry_anlys_params(candidate_predrs_chr = c("SCARED","OASIS","GAD7"),
                                            candidate_covar_nms_chr = c("PHQ9", "SOFAS", "d_sex_birth_s", "d_age", "d_sexual_ori_s", "d_studying_working"),
                                            prefd_covars_chr = "PHQ9"))

Report findings

Create shareable models

The model objects created and saved in our working directory by the preceding steps are not suitable for public dissemination. They are both too large in file size and, more importantly, include copies of our source dataset. We can overcome these limitations by creating shareable versions of the models. Two types of shareable version are created - copies of the original model objects in which fake data overwrites the original source data and summary tables of model coefficients.

A <- author(A, consent_1L_chr = consent_1L_chr, what_1L_chr = "models")

Specify study reporting metadata

We update A so that we can begin use it to render and share reports.

A <- renew(A, what_1L_chr = "reporting")

We add metadata relevant to the reports that we will be generating to these fields. Note that the data we supply to the Ready4useRepos object below must relate to a repository to which we have write permissions (otherwise subsequent steps will fail).

A <- renew(A, ready4show::authors_tb, type_1L_chr = "authors", what_1L_chr = "reporting") %>%
  renew(ready4show::institutes_tb, type_1L_chr = "institutes", what_1L_chr = "reporting") %>%
  renew(c(3L,3L), type_1L_chr = "digits", what_1L_chr = "reporting") %>%
  renew(c("PDF","PDF"), type_1L_chr = "formats", what_1L_chr = "reporting") %>%
  renew("A hypothetical utility mapping study using fake data", type_1L_chr = "title", what_1L_chr = "reporting") %>%
  renew(renew(ready4show_correspondences(), old_nms_chr = c("PHQ9", "GAD7"), new_nms_chr = c("PHQ-9", "GAD-7")), type_1L_chr = "changes", what_1L_chr = "reporting") %>%
  renew(Ready4useRepos(dv_nm_1L_chr = "fakes", dv_ds_nm_1L_chr = "https://doi.org/10.7910/DVN/D74QMP", dv_server_1L_chr = "dataverse.harvard.edu"), type_1L_chr = "repos", what_1L_chr = "reporting") 

Author model catalogues

We download a program for generating a catalogue of models and use it to summarising the models created under each study analysis (one primary and two secondary). The catalogues are saved locally.

A <- author(A, consent_1L_chr = consent_1L_chr, download_tmpl_1L_lgl = T, what_1L_chr = "catalogue")

Author manuscript

We add some content about the manuscript we wish to author.

A <- renew(A, "Quality Adjusted Life Years (QALYs) are often used in economic evaluations, yet utility weights for deriving them are rarely directly measured in mental health services.", 
           type_1L_chr = "background", what_1L_chr = "reporting") %>%
  renew("None declared", type_1L_chr = "conflicts", what_1L_chr = "reporting") %>%
  renew("Nothing should be concluded from this study as it is purely hypothetical.", type_1L_chr = "conclusion", what_1L_chr = "reporting") %>%
  renew("The study was reviewed and granted approval by no-one." , type_1L_chr = "ethics", what_1L_chr = "reporting") %>%
  renew("The study was funded by no-one.", type_1L_chr = "funding", what_1L_chr = "reporting") %>%
  renew("three months", type_1L_chr = "interval", what_1L_chr = "reporting") %>%
  renew(c("anxiety", "AQoL","depression", "psychological distress", "QALYs", "utility mapping"), type_1L_chr = "keywords", what_1L_chr = "reporting") %>%
  renew("The study sample is fake data.", type_1L_chr = "sample", what_1L_chr = "reporting") 

We create a brief summary of results that can be interpreted by the program that authors the manuscript.

A <- renew(A, c("AQoL-6D", "Adolescent AQoL Six Dimension"), type_1L_chr = "naming", what_1L_chr = "reporting")
A <- renew(A, "use_renew_mthd", type_1L_chr = "abstract", what_1L_chr = "reporting")

We create and save the plots that will be used in the manuscript.

A <- author(A, consent_1L_chr = consent_1L_chr, what_1L_chr = "plots")

We download a program for generating a template manuscript and run it to author a first draft of the manuscript.

A <- author(A, consent_1L_chr = consent_1L_chr, download_tmpl_1L_lgl = T, what_1L_chr = "manuscript")

We can copy the RMarkdown files that created the template manuscript to a new directory (called “Manuscript_Submission”) so that we can then manually edit those files to produce a manuscript that we can submit for publication.

A <- author(A, consent_1L_chr = consent_1L_chr, type_1L_chr = "copy", what_1L_chr = "manuscript")

At this point in the workflow, additional steps are required to adapt / author the manuscript that will be submitted for publication. However, in this example we are going to skip that step and keep working with the unedited template manuscript. If we had a finalised manuscript authoring program stored online, we could now specify the repository from which the program can be retrieved.

# Not run
# A <- renew(A, c("URL of GitHub repository with", "Program version number"), type_1L_chr = "template-manuscript", what_1L_chr = "reporting")

We can now configure the output to be generated by the manuscript authoring program. The below commands will specify a Microsoft Word format manuscript and a PDF technical appendix. Unlike the template manuscript, the figures and tables will be positioned after (and not within) the main body of the manuscript. Note that the Word version of the manuscript generated by these values will require some minor formatting edits (principally to the display of tables and numbering of sections).

A <- renew(A, F, type_1L_chr = "figures-body", what_1L_chr = "reporting") %>%
  renew(F, type_1L_chr = "tables-body", what_1L_chr = "reporting") %>%
  renew(c("Word","PDF"), type_1L_chr = "formats", what_1L_chr = "reporting")

Once any edits to the RMarkdown files for creating the submission manuscript have been finalised, we can run the following command to author the manuscript. If we are using a custom manuscript authoring program downloaded from an online repository the download_tmpl_1L_lgl argument will need to be set to T.

A <- author(A, consent_1L_chr = consent_1L_chr, download_tmpl_1L_lgl = F, type_1L_chr="submission", what_1L_chr = "manuscript")

We can now generate the Supplementary Information for the submission manuscript.

A <- author(A, consent_1L_chr = consent_1L_chr, supplement_fl_nm_1L_chr = "TA_PDF", type_1L_chr="submission", what_1L_chr = "supplement")

Share outputs

We can now share non-confidential elements (ie no copies of individual records) of the outputs that we have created via our study online repository. To run this step you will need write permissions to the online repository. In the below step we are sharing model catalogues, details of the utility instrument, the shareable mapping models (designed to be used in conjunction with the youthu package), our manuscript files and our supplementary information. In most real world studies the manuscript would not be shared via an online repository - the what_chr argument would need to be ammended to reflect this.

A <- share(A, types_chr = c("auto", "submission"), what_chr = c("catalogue", "instrument" ,"manuscript", "models", "supplement"))

The dataset we created in the previous step is viewable here: https://dataverse.harvard.edu/dataset.xhtml?persistentId=doi:10.7910/DVN/D74QMP

Tidy workspace

The preceding steps saved multiple objects (mostly R model objects) that have embedded within them copies of the source dataset. To protect the confidentiality of these records we can now purge all such copies from our output data directory.

A <- author(A, what_1L_chr = "purge")

6.1.2.1.6 - Find and deploy utility mapping models

Using tools (soon to be formalised into ready4 modules) from the youthu R package, it is possible to find and deploy relevant utility mapping algorithms. This tutorial illustrates the main steps for predicting AQoL-6D utility from psychological and functional measures collected on clinical samples of young people.

This below section renders a vignette article from the youthu library. You can use the following links to:

This vignette outlines a workflow for:

  • Searching, selecting and retrieving transfer to utility models;
  • Preparing a prediction dataset for use with a selected transfer to utility model; and
  • Applying the selected transfer to utility model to a prediction dataset to predict Quality Adjusted Life Years (QALYs).

The practical value of implementing such a workflow is discussed in the economic analysis vignette and a scientific manuscript. Note, this example uses fake data - it should should not be used to inform decision making.

Search, select and retrieve transfer to utility models

To identify datasets that contain transfer to utility models compatible with youthu (ie those developped with the TTU package), you can use the get_ttu_dv_dss function. The function searches specified dataverses (in the below example, the TTU dataverse) for datasets containing output from the TTU package.

ttu_dv_dss_tb <- get_ttu_dv_dss("TTU")

The ttu_dv_dss_tb table summarises some pertinent details about each dataset containing TTU models found by the preceding command. These details include a link to any scientific summary (the “Article” column) associated with a dataset.

Transfer to Utility Datasets
ID Utility Predictors Article
1 aqol6dtotalw BADS total score , GAD7 total score , K6 total score , OASIS total score , PHQ9 total score , SCARED total score, SOFAS total score

To identify models that predict a specified type of health utility from one or more of a specified subset of predictors, use:

mdls_lup <- get_mdls_lup(ttu_dv_dss_tb = ttu_dv_dss_tb,
                         utility_type_chr = "AQoL-6D",
                         mdl_predrs_in_ds_chr = c("PHQ9 total score",
                                                  "SOFAS total score"))

The preceding command will produce a lookup table with information that includes the catalogue names of models, the predictors used in each model and the analysis that generated each one.

Selected elements from Models Look-Up Table
Catalogue reference Predictors Analysis
PHQ9_1_GLM_GSN_LOG PHQ9 Primary Analysis
PHQ9_1_OLS_CLL PHQ9 Primary Analysis
PHQ9_SOFAS_1_GLM_GSN_LOG PHQ9 , SOFAS Primary Analysis
PHQ9_SOFAS_1_OLS_CLL PHQ9 , SOFAS Primary Analysis
OASIS_SOFAS_1_GLM_GSN_LOG OASIS, SOFAS Primary Analysis
OASIS_SOFAS_1_OLS_CLL OASIS, SOFAS Primary Analysis
BADS_SOFAS_1_GLM_GSN_LOG BADS , SOFAS Primary Analysis
BADS_SOFAS_1_OLS_CLL BADS , SOFAS Primary Analysis
K6_SOFAS_1_GLM_GSN_LOG K6 , SOFAS Primary Analysis
K6_SOFAS_1_OLS_CLL K6 , SOFAS Primary Analysis
SCARED_SOFAS_1_GLM_GSN_LOG SCARED, SOFAS Primary Analysis
SCARED_SOFAS_1_OLS_CLL SCARED, SOFAS Primary Analysis
GAD7_SOFAS_1_GLM_GSN_LOG GAD7 , SOFAS Primary Analysis
GAD7_SOFAS_1_OLS_CLL GAD7 , SOFAS Primary Analysis
SOFAS_1_GLM_GSN_LOG SOFAS Secondary Analysis A
SOFAS_1_OLS_CLL SOFAS Secondary Analysis A
OASIS_PHQ9_1_GLM_GSN_LOG OASIS, PHQ9 Secondary Analysis B
OASIS_PHQ9_1_OLS_CLL OASIS, PHQ9 Secondary Analysis B
GAD7_PHQ9_1_GLM_GSN_LOG GAD7, PHQ9 Secondary Analysis B
GAD7_PHQ9_1_OLS_CLL GAD7, PHQ9 Secondary Analysis B
SCARED_PHQ9_1_GLM_GSN_LOG SCARED, PHQ9 Secondary Analysis B
SCARED_PHQ9_1_OLS_CLL SCARED, PHQ9 Secondary Analysis B

To review the summary information about the predictive performance of a specific model, use:

get_dv_mdl_smrys(mdls_lup,
                 mdl_nms_chr = "PHQ9_SOFAS_1_OLS_CLL")
#> $PHQ9_SOFAS_1_OLS_CLL
#>        Parameter Estimate    SE          95% CI
#> 1 SD (Intercept)    0.348 0.017   0.312 , 0.382
#> 2      Intercept    0.428 0.129   0.174 , 0.686
#> 3  PHQ9 baseline   -9.115 0.249 -9.601 , -8.618
#> 4    PHQ9 change   -7.331 0.339 -8.007 , -6.665
#> 5 SOFAS baseline    0.960 0.172   0.616 , 1.292
#> 6   SOFAS change    1.146 0.235   0.674 , 1.607
#> 7             R2    0.767 0.012   0.743 , 0.788
#> 8           RMSE    0.925 0.004   0.922 , 0.928
#> 9          Sigma    0.406 0.012   0.384 , 0.429

More information about a selected model can be found in the online model catalogue, the link to which can be obtained with the following command:

get_mdl_ctlg_url(mdls_lup,
                 mdl_nm_1L_chr = "PHQ9_SOFAS_1_OLS_CLL")

[1] “https://dataverse.harvard.edu/api/access/datafile/6484935

Prepare a prediction dataset for use with a selected transfer to utility model

Import data

You can now import and inspect the dataset you plan on using for prediction. In the below example we use fake data.

data_tb <- make_fake_ds_one()
Illustrative example of a prediction dataset
UID Timepoint Date PHQ_total SOFAS_total
Participant_1 Baseline 2022-05-22 7 69
Participant_10 Baseline 2022-04-07 17 60
Participant_10 Follow-up 2022-06-22 17 64
Participant_100 Baseline 2022-07-29 0 76
Participant_1000 Baseline 2022-02-10 0 71
Participant_1000 Follow-up 2022-05-05 0 71

Confirm dataset can be used as a prediction dataset

The prediction dataset must contain variables that correspond to all the predictors of the model you intend to apply. The allowable range and required class of each predictor variable are described in the min_val_dbl, max_val_dbl and class_chr columns of the model predictors lookup table, which can be accessed with a call to the get_predictors_lup function.

predictors_lup <- get_predictors_lup(mdls_lup = mdls_lup,
                                     mdl_nm_1L_chr = "PHQ9_SOFAS_1_OLS_CLL")
Model predictors lookup table
short_name_chr long_name_chr min_val_dbl max_val_dbl class_chr increment_dbl class_fn_chr mdl_scaling_dbl covariate_lgl
PHQ9 PHQ9 total score 0 27 integer 1 youthvars::youthvars_phq9 0.01 FALSE
SOFAS SOFAS total score 0 100 integer 1 youthvars::youthvars_sofas 0.01 TRUE

The prediction dataset must also include both a unique client identifier variable and a measurement time-point identifier variable (which must be a factor with two levels). The dataset also needs to be in long format (ie where measures at different time-points for the same individual are stacked on top of each other in separate rows). We can confirm these conditions hold by creating a dataset metadata object using the make_predn_metadata_ls function. In creating the metadata object, the function checks that the dataset can be used in conjunction with the model specified at the mdl_nm_1L_chr argument. If the prediction dataset uses different variable names for the predictors to those specified in the predictors_lup lookup table, a named vector detailing the correspondence between the two sets of variable names needs to be passed to the predr_vars_nms_chr argument. Finally, if you wish to specify a preferred variable name to use for the predicted utility values when applying the model, you can do this by passing this name to the utl_var_nm_1L_chr argument.

predn_ds_ls <- make_predn_metadata_ls(data_tb,
                                      id_var_nm_1L_chr = "UID",
                                      msrmnt_date_var_nm_1L_chr = "Date",
                                      predr_vars_nms_chr = c(PHQ9 = "PHQ_total",SOFAS = "SOFAS_total"),
                                      round_var_nm_1L_chr = "Timepoint",
                                      round_bl_val_1L_chr = "Baseline",
                                      utl_var_nm_1L_chr = "AQoL6D_HU",
                                      mdls_lup = mdls_lup,
                                      mdl_nm_1L_chr = "PHQ9_SOFAS_1_OLS_CLL")

Apply the selected transfer to utility model to a prediction dataset to predict Quality Adjusted Life Years (QALYs)

Predict health utility at baseline and follow-up timepoints

To generate utility predictions we use the add_utl_predn function. The function needs to be supplied with the prediction dataset (the value passed to argument data_tb) and the validated prediction metadata object we created in the previous step.

data_tb <- add_utl_predn(data_tb,
                         predn_ds_ls = predn_ds_ls)
#> Joining with `by = join_by(UID, Timepoint)`

By default the add_utl_predn function samples model parameter values based on a table of model coefficients when making predictions and constrains predictions to an allowed range. You can override these defaults by adding additional arguments new_data_is_1L_chr = "Predicted" (which uses mean parameter values), force_min_max_1L_lgl = F (removes range constraint) and (if the source dataset makes available downloadable model objects) make_from_tbl_1L_lgl = F. These settings will produce different predictions. It is strongly recommended that you consult the model catalogue (see above) to understand how such decisions may affect the validity of the predicted values that will be generated.

Prediction dataset with predicted utilities
UID Timepoint Date PHQ_total SOFAS_total AQoL6D_HU
Participant_1 Baseline 2022-05-22 7 69 0.9080468
Participant_10 Baseline 2022-04-07 17 60 0.5533808
Participant_10 Follow-up 2022-06-22 17 64 0.4006010
Participant_100 Baseline 2022-07-29 0 76 0.6809903
Participant_1000 Baseline 2022-02-10 0 71 0.9877882
Participant_1000 Follow-up 2022-05-05 0 71 0.9602037

Our health utility predictions are now available for use and are summarised below.

summary(data_tb$AQoL6D_HU)
#>    Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
#> 0.06646 0.42781 0.63403 0.62335 0.83351 1.00000

Calculate QALYs

The last step is to calculate Quality Adjusted Life Years, using a method assuming a linear rate of change between timepoints.

data_tb <- data_tb %>% add_qalys_to_ds(predn_ds_ls = predn_ds_ls,
                                       include_predrs_1L_lgl = F,
                                       reshape_1L_lgl = F)
Prediction dataset with QALYs
UID Timepoint Date PHQ_total SOFAS_total AQoL6D_HU AQoL6D_HU_change_dbl duration_prd qalys_dbl
Participant_1 Baseline 2022-05-22 7 69 0.9080468 0.0000000 0S 0.0000000
Participant_10 Baseline 2022-04-07 17 60 0.5533808 0.0000000 0S 0.0000000
Participant_10 Follow-up 2022-06-22 17 64 0.4006010 -0.1527798 76d 0H 0M 0S 0.0992507
Participant_100 Baseline 2022-07-29 0 76 0.6809903 0.0000000 0S 0.0000000
Participant_1000 Baseline 2022-02-10 0 71 0.9877882 0.0000000 0S 0.0000000
Participant_1000 Follow-up 2022-05-05 0 71 0.9602037 -0.0275845 84d 0H 0M 0S 0.2239991

6.1.2.1.7 - Use utility mapping algorithms to help implement cost-utility analyses

Using tools (soon to be formalised into ready4 framework modules) from the youthu R package, it is possible to use utility mapping algorithms to help implement cost-utility analyses. This tutorial illustrates the main steps for doing so using psychological and functional measures collected on clinical samples of young people.

This below section renders a vignette article from the youthu library. You can use the following links to:

This vignette illustrates the rationale for and practical decision-making utility of youthu’s QALYs prediction workflow. Note, this example is illustrated with fake data and should not be used to inform decision-making.

Motivation

The main motivation behind the youthu package is to extend the types of economic analysis that can be undertaken with both single group (e.g. pilot study, health service records) and matched groups (e.g. trial) longitudinal datasets that do not include measures of health utility. This article focuses on its application to matched group datasets.

Example dataset

First, we must first import our data. In this example we will use a fake dataset.

ds_tb <- make_fake_ds_two()
#> Joining with `by = join_by(fkClientID, study_arm_chr)`

Our dataset includes 268 matched comparisons, with each comparison containing baseline and follow-up records for one intervention arm participant and one control arm participant. The first few records are as follows.

First few records from input dataset
fkClientID round date_psx duration_prd PHQ9 SOFAS costs_dbl study_arm_chr match_idx_int
Participant_20 Baseline 2023-03-15 0S 16 41 301.1868 Intervention 1
Participant_593 Baseline 2023-01-20 0S 19 43 259.3190 Control 1
Participant_593 Follow-up 2023-07-14 175d 0H 0M 0S 16 65 1290.4220 Control 1
Participant_20 Follow-up 2023-09-09 178d 0H 0M 0S 15 74 1787.4242 Intervention 1
Participant_259 Baseline 2023-05-10 0S 19 39 311.0018 Control 2
Participant_962 Baseline 2023-06-22 0S 10 45 276.2181 Intervention 2

This dataset contains features that make it possible to use in conjunction with youthu’s economic analysis functions. These requirements are described in the vignette about finding and using models compatible models to predict QALYs;

The dataset also contains a cost variable, which is a requirement for most, though not all, of the economic analyses that can be undertaken with youthu.

Limitations of datasets without measures of health utility

A notable omission from the dataset is any measure of utility. This omission means that, in the absence of using mapping algorithms such as those included with youthu, the most feasible types of economic evaluation to apply to this dataset would likely be cost-consequence analysis (where a synopsis of the differences in a range of measures are presented alongside cost differences) and cost-effectiveness analysis (where a summary statistic - the incremental cost-effectiveness ratio or ICER - is calculated by dividing differences in costs by differences in a single outcome measure).

These types of economic analyses can be relatively simple to interpret if either the intervention or control arm is simultaneously cheaper and more effective across all included outcome measures. However, these conditions don’t hold in our sample data.

summary((ds_tb %>% dplyr::filter(study_arm_chr == "Control" & round == "Baseline"))[5:6])
#>       PHQ9          SOFAS      
#>  Min.   : 0.0   Min.   :39.00  
#>  1st Qu.: 7.0   1st Qu.:60.00  
#>  Median :12.0   Median :66.00  
#>  Mean   :10.9   Mean   :66.13  
#>  3rd Qu.:15.0   3rd Qu.:72.00  
#>  Max.   :19.0   Max.   :89.00
summary((ds_tb %>% dplyr::filter(study_arm_chr == "Control" & round == "Follow-up"))[5:7])
#>       PHQ9            SOFAS         costs_dbl     
#>  Min.   : 0.000   Min.   :39.00   Min.   : 889.9  
#>  1st Qu.: 4.000   1st Qu.:64.00   1st Qu.:1321.1  
#>  Median : 8.000   Median :71.00   Median :1486.7  
#>  Mean   : 8.493   Mean   :70.65   Mean   :1489.0  
#>  3rd Qu.:13.000   3rd Qu.:77.00   3rd Qu.:1627.0  
#>  Max.   :27.000   Max.   :98.00   Max.   :2216.5
summary((ds_tb %>% dplyr::filter(study_arm_chr == "Intervention" & round == "Baseline"))[5:6])
#>       PHQ9           SOFAS      
#>  Min.   : 0.00   Min.   :36.00  
#>  1st Qu.: 7.00   1st Qu.:61.00  
#>  Median :11.00   Median :67.00  
#>  Mean   :10.81   Mean   :66.74  
#>  3rd Qu.:15.00   3rd Qu.:72.25  
#>  Max.   :19.00   Max.   :88.00
summary((ds_tb %>% dplyr::filter(study_arm_chr == "Intervention" & round == "Follow-up"))[5:7])
#>       PHQ9            SOFAS      costs_dbl     
#>  Min.   : 0.000   Min.   :40   Min.   : 923.4  
#>  1st Qu.: 2.000   1st Qu.:60   1st Qu.:1625.6  
#>  Median : 6.500   Median :68   Median :1777.3  
#>  Mean   : 6.851   Mean   :68   Mean   :1807.8  
#>  3rd Qu.:11.000   3rd Qu.:77   3rd Qu.:1996.0  
#>  Max.   :25.000   Max.   :93   Max.   :2872.7

The pattern of results summarised above create some significant barriers to meaningfully interpreting economic evaluations that are based on cost-consequence or cost-effectiveness analysis:

  • A cost-effectiveness analysis in which change in PHQ-9 was the benefit measure would be difficult to interpret as the Intervention arm is both more effective and more costly, which begs the question is it worth paying the extra dollars for this improvement? Also - would a judgment of cost-effectiveness remain the same if the study had measured a slightly different incremental benefit or recorded change over a longer or shorter time horizon? It is likely that there is no commonly used value for money benchmark for improvements measured in PHQ-9, nor is there any time weighting associated with the measure. Furthermore, if the potential funding for the intervention is from a budget that is allocated to non-depressive illnesses (e.g. physical health), results from a cost-effectiveness analysis using PHQ-9 as its benefit measure are not readily comparable with economic evaluations of interventions from other illness groups using different benefit measures that are potentially competing for the same scarce funding.

  • A cost consequence analyses that summarised the differences in costs with the differences in changes in PHQ-9 and SOFAS score would be difficult to interpret because while the intervention is more effective than control for improvements measured on PHQ-9 (where lower scores are better), the control group is superior if benefits are based on functioning improvements as measured by SOFAS scores (where higher scores are better). The lack of any formal weighting for how to trade off clinical symptoms and functioning means that interpretation of this analysis will be highly subjective and likely to change across potential decision makers.

These types of short-comings can be significantly addressed by undertaking cost-utility analyses (CUAs) as:

  • they use a measure of benefit - the Quality Adjusted Life Year (QALY) - that captures multiple domains of health, weighted by time and population preferences in a single index measure that can be applied across health conditions;
  • there are published benchmark willingness to pay values for QALYs that are routinely used by decision makers in many countries to make ICER statistics readily interpretable in the context of health budget allocation.

The rest of this article demonstrates how youthu functions can be used to undertake CUA based analyses on the type of data we have just profiled.

Using youthu in a cost-utility analysis workflow

Predict adolescent AQoL-6D health utility

Our first step is to identify which youthu models we will use to predict adolescent AQoL-6D and apply these models to our data. This step was explained in more detail in another vignette article about finding and using transfer to utility models, so will be dealt with briefly here.

First we make sure that our dataset can be used as a prediction dataset in conjunction with the model we intend using.

predn_ds_ls <- make_predn_metadata_ls(ds_tb,
                                      cmprsn_groups_chr = c("Intervention", "Control"),
                                      cmprsn_var_nm_1L_chr = "study_arm_chr",
                                      costs_var_nm_1L_chr = "costs_dbl",
                                      id_var_nm_1L_chr = "fkClientID",
                                      msrmnt_date_var_nm_1L_chr = "date_psx",
                                      round_var_nm_1L_chr = "round",
                                      round_bl_val_1L_chr = "Baseline",
                                      utl_var_nm_1L_chr = "AQoL6D_HU",
                                      mdls_lup = get_mdls_lup(utility_type_chr = "AQoL-6D",
                                                              mdl_predrs_in_ds_chr = c("PHQ9 total score",
                                                                                       "SOFAS total score"),
                                                              ttu_dv_nms_chr = "TTU"),
                                      mdl_nm_1L_chr =  "PHQ9_SOFAS_1_OLS_CLL")

We now use our preferred model to predict health utility from the measures in our dataset.

ds_tb <- add_utl_predn(ds_tb,
                       predn_ds_ls = predn_ds_ls) %>%
  dplyr::select(fkClientID, round, study_arm_chr, date_psx, duration_prd, dplyr::everything())
#> Joining with `by = join_by(fkClientID, round)`

Calculate QALYs

Next we combine the health utility data with the interval between measurement data to calculate QALYs and add them to the dataset.

ds_tb  <- ds_tb %>% add_qalys_to_ds(predn_ds_ls = predn_ds_ls,
                                    include_predrs_1L_lgl = T,
                                    reshape_1L_lgl = T)
First few records from updated dataset with QALYs
fkClientID study_arm_chr match_idx_int date_psx_Baseline date_psx_Follow-up duration_prd_Baseline duration_prd_Follow-up costs_dbl_Baseline costs_dbl_Follow-up PHQ9_Baseline PHQ9_Follow-up SOFAS_Baseline SOFAS_Follow-up AQoL6D_HU_Baseline AQoL6D_HU_Follow-up PHQ9_change_dbl_Baseline PHQ9_change_dbl_Follow-up SOFAS_change_dbl_Baseline SOFAS_change_dbl_Follow-up AQoL6D_HU_change_dbl_Baseline AQoL6D_HU_change_dbl_Follow-up qalys_dbl_Baseline qalys_dbl_Follow-up
Participant_10 Control 243 2022-12-29 2023-06-24 0S 177d 0H 0M 0S 647.9386 1696.235 8 10 61 64 0.7597988 0.6079774 0 2 0 3 0 -0.1518214 0 0.3314119
Participant_1000 Control 191 2023-02-24 2023-08-27 0S 184d 0H 0M 0S 428.9205 1619.037 4 2 63 82 0.8459579 0.7688131 0 -2 0 19 0 -0.0771448 0 0.4067322
Participant_1001 Intervention 230 2023-01-19 2023-07-17 0S 179d 0H 0M 0S 429.3703 1844.219 10 14 59 72 0.6138300 0.8607305 0 4 0 13 0 0.2469005 0 0.3613228
Participant_1003 Intervention 115 2023-02-17 2023-08-18 0S 182d 0H 0M 0S 395.1637 1537.365 9 0 71 81 0.5808015 0.9315788 0 -9 0 10 0 0.3507773 0 0.3768011
Participant_1005 Intervention 183 2023-05-21 2023-11-23 0S 186d 0H 0M 0S 402.9910 1826.511 17 0 78 88 0.5460607 0.9593811 0 -17 0 10 0 0.4133204 0 0.3833158
Participant_1006 Intervention 219 2023-06-16 2023-12-12 0S 179d 0H 0M 0S 534.2285 2401.478 9 14 75 73 0.7239490 0.5885972 0 5 0 -2 0 -0.1353518 0 0.3216232

Analyse results

Now we can run the main economic analysis. This is implemented by the make_hlth_ec_smry function, which first bootstraps the dataset (implemented by the boot function from the boot package) before passing the mean values for costs and QALYs from each bootstrap sample to with bcea function of the BCEA package to calculate a range of health economic statistics. For this example we pass a value of 50,000 for the willingness to pay parameter, as this is the dollar amount commonly used in Australia as a benchmark for the value of a QALY.

Note, for this illustrative example we only request 1000 bootstrap iterations - in practice this number may be higher.

he_smry_ls <- ds_tb %>% make_hlth_ec_smry(predn_ds_ls = predn_ds_ls,
                                                 wtp_dbl = 50000,
                                                 bootstrap_iters_1L_int = 1000L)
#> Warning: There was 1 warning in `dplyr::summarise()`.
#>  In argument: `dplyr::across(.fns = mean)`.
#> Caused by warning:
#> ! Using `across()` without supplying `.cols` was deprecated in dplyr 1.1.0.
#>  Please supply `.cols` instead.

As part of the output of the make_hlth_ec_smry function is a BCEA object, we can use the BCEA package to produce a number of graphical summaries of economic results. One of the most important is the production of a cost-effectiveness plane. This plot highlights that, with an ICER of $-98,145.56, less than half of the bootstrapped iteration incremental cost and QALY pairs fall within the zone of cost-effectiveness (green). In fact, at the cost-effectiveness threshold we supplied, the results suggest there is a 8% probability that the intervention is cost-effective.

library(ggplot2)
BCEA::ceplane.plot(he_smry_ls$ce_res_ls, wtp =50000,    
                   area_color = "green",
                    graph = "ggplot2",
          theme = ggplot2::theme_light())
#> Warning: Duplicated aesthetics after name standardisation: colour

6.1.2.1.8 - Develop choice models

Using tools (soon to be formalised into ready4 framework modules) from the mychoice R package, it is possible to develop choice models from responses to a discrete choice experiment survey.

This below section renders a vignette article from the mychoice library. You can use the following links to:

library(mychoice)
#> The legacy packages maptools, rgdal, and rgeos, underpinning the sp package,
#> which was just loaded, will retire in October 2023.
#> Please refer to R-spatial evolution reports for details, especially
#> https://r-spatial.org/r/2023/05/15/evolution4.html.
#> It may be desirable to make the sf package available;
#> package maintainers should consider adding sf to Suggests:.
#> The sp package is now running under evolution status 2
#>      (status 2 uses the sf package in place of rgdal)

The tools in mychoice are designed to make it easier to develop and use choice models with ready4 - an open source health economic model of the systems shaping mental health and wellbeing in young people.

This development version of the mychoice package has been made available as part of the process of testing and documenting the package.

Currently there are no vignettes available. However, examples of the application of mychoice functions to a real world discrete choice experiment are in programs available at https://doi.org/10.5281/zenodo.6626256 (design of a discrete choice experiment survey) and https://doi.org/10.5281/zenodo.7223286 (analysis of discrete choice experiment survey responses). PDF versions of each program, along with the artefacts produced by each are available in the online dataset at https://doi.org/10.7910/DVN/VGPIPS.

6.1.2.2 - Modules for modelling places

Modules for spatio-temporal modelling of the environments that shape young people’s mental health are collectively referred to as the “Springtides” model. Two module libraries are currently available - vicinity and aus, though both are highly preliminary and without any vignette articles to demonstrate their use. An app built using a combination of these libraries and unreleased work in progress module libraries is available for illustration purposes.

6.1.2.3 - Modules for modelling platforms

Modules that model the processes, eligibility requirements, staffing and configurations of youth service platforms are collectively referred to as the “First Bounce” model. No platforms modules are yet available - see details on unreleased work in progress.

6.1.2.4 - Modules for modelling programs

Modules for modelling the efficacy, cost-effectiveness and budget impact of youth mental health programs (e.g. interventions for prevention, treatment and wellbeing) are collectively referred to as the “On Target” model. Some initial modules from the costly library are available. There is also even more preliminary work in progress.

6.1.2.4.1 - Standardise Variable Values With Fuzzy Logic And Correspondence Tables

Costing health economic datasets is an activity that can involve repeated use of lookup tables. This tutorial describes how a module from the costly R package can help you to use a combination of fuzzy logic and correspondence tables to standardise variable values and thus facilitate partial automation of costing algorithms.

This below section renders a vignette article from the costly library. You can use the following links to:

In brief

The steps described and explained in this vignette can also be (more succinctly) accomplished with the following code.

X <- CostlyCountries() 
X <- renew(X, type_1L_chr = "default") 
X <- renew(X, "jw", type_1L_chr = "slot", what_1L_chr = "logic") 
X <- renew(X, T, type_1L_chr = "slot", what_1L_chr = "force")
X <- ratify(X) 

Create project

We begin by creating X, an instance of the CostlyCorrespondences module.

Supply seed dataset

We begin by creating a CostlySeed module instance that includes a dataset containing our variable of interest (in this case, countries). The dataset needs to be paired with a dataset dictionary using the Ready4useDyad module from the ready4use R library. You can supply a custom standards dataset (a tibble), dictionary (a ready4use_dictionary) and the concept represented by our variable of interest using a command of the following format.

# Not run
# A <- CostlySeed(Ready4useDyad_r4 = Ready4useDyad(ds_tb = tibble::tibble(), dictionary_r3 = ready4use_dictionary()), include_chr = c("Country"), label_1L_chr = "Country")

The add_default_country_seed function will perform the previous step using values that pair the world.cities dataset of the maps R library with an appropriate dictionary and specifies countries as the concept we will be standardising.

We can now inspect the first few records from our labelled seed dataset.

renewSlot(A, "Ready4useDyad_r4", type_1L_chr = "label") %>%
exhibitSlot("Ready4useDyad_r4", display_1L_chr = "head", scroll_box_args_ls = list(width = "100%"))
Dataset
City name Country name Population size Latitude coordinate Longitude coordinate Is the nation's capital city
'Abasan al-Jadidah Palestine 5629 31.31 34.34 0
'Abasan al-Kabirah Palestine 18999 31.32 34.35 0
'Abdul Hakim Pakistan 47788 30.55 72.11 0
'Abdullah-as-Salam Kuwait 21817 29.36 47.98 0
'Abud Palestine 2456 32.03 35.07 0
'Abwein Palestine 3434 32.03 35.20 0

We can also inspect the data dictionary contained in A.

exhibitSlot(A, "Ready4useDyad_r4", type_1L_chr = "dict", scroll_box_args_ls = list(width = "100%"))
Data Dictionary
Variable Category Description Class
name City City name character
country.etc Country Country name character
pop Population Population size integer
lat Latitude Latitude coordinate numeric
long Longitude Longitude coordinate numeric
capital Capital Is the nation's capital city integer

We now specify the dictionary category that corresponds to the variable we wish to standardise (“Country”). We need to use the same category name to label the results objects that we generate in subsequent steps.

A@include_chr <- A@label_1L_chr <- "Country"

We now add A to X.

X <- renew(X, A, what_1L_chr = "seed")

Specify standards

We next must specify a dataset that includes the complete list of allowable variable values.

This workflow for this step is similar to that for specifying standards, except that instead of a CostlySeed module we use a CostlyStandards module.

# Not run
# Y <- CostlyStandards(Ready4useDyad_r4 = Ready4useDyad(ds_tb = tibble::tibble(), dictionary_r3 = ready4use_dictionary()))

In many cases using the ISO_3166_1 dataset from the ISOcodes library will be the optimal choice for the standardised form of country names. We can use the add_country_standards function to pair this dataset with its dictionary and create B, a CostlyStandards module instance.

We can inspect the first few cases of the labelled version of the dataset in B.

renewSlot(B, "Ready4useDyad_r4", type_1L_chr = "label") %>% 
  exhibitSlot("Ready4useDyad_r4", display_1L_chr = "head", scroll_box_args_ls = list(width = "100%"))
Dataset
Alpabetical country code (two letters) Alpabetical country code (three letters) Numeric country code Country name Country name (official) Country name (common alternative)
AW ABW 533 Aruba NA NA
AF AFG 004 Afghanistan Islamic Republic of Afghanistan NA
AO AGO 024 Angola Republic of Angola NA
AI AIA 660 Anguilla NA NA
AX ALA 248 Åland Islands NA NA
AL ALB 008 Albania Republic of Albania NA

We can also inspect the data dictionary contained in B.

exhibitSlot(B, "Ready4useDyad_r4", type_1L_chr = "dict", scroll_box_args_ls = list(width = "100%"))
Data Dictionary
Variable Category Description Class
Alpha_2 A2 Alpabetical country code (two letters) character
Alpha_3 A3 Alpabetical country code (three letters) character
Numeric N Numeric country code character
Name Country Country name character
Official_name Official Country name (official) character
Common_name Common Country name (common alternative) character

We can now specifying both the concept (from the “Category” column of the data dictionary) that specifies allowable values for our target variable and all concepts we plan to use for fuzzy logic matching (described below).

B@label_1L_chr <- "Country"
B@include_chr <- c("Country", "Official","Common","A3","A2")

We now add B to X.

X <- renew(X, B, what_1L_chr = "standards")

Compare variable of interest values from seed and standards dataset.

To identify any disparities between the variable of interest in our seed and standards datasets we can use the ratify method. Supplying the value “identity” ensures that the output will differ from input only in the slot reserved for results.

X <- ratify(X, new_val_xx = "identity")

We can now identify the values from our seed dataset variable of interest that were not in our standard values.

X@results_ls$Country_Output_Validation$Invalid_Values

We can also identify standard values that were not present in the seed dataset variable of interest.

X@results_ls$Country_Output_Validation$Absent_Values
#>  [1] "Åland Islands"                                "Antarctica"                                  
#>  [3] "Bolivia, Plurinational State of"              "Bonaire, Sint Eustatius and Saba"            
#>  [5] "Bouvet Island"                                "British Indian Ocean Territory"              
#>  [7] "Brunei Darussalam"                            "Cabo Verde"                                  
#>  [9] "Christmas Island"                             "Cocos (Keeling) Islands"                     
#> [11] "Congo, The Democratic Republic of the"        "Côte d'Ivoire"                               
#> [13] "Curaçao"                                      "Czechia"                                     
#> [15] "Eswatini"                                     "Falkland Islands (Malvinas)"                 
#> [17] "French Southern Territories"                  "Guernsey"                                    
#> [19] "Heard Island and McDonald Islands"            "Holy See (Vatican City State)"               
#> [21] "Hong Kong"                                    "Iran, Islamic Republic of"                   
#> [23] "Korea, Democratic People's Republic of"       "Korea, Republic of"                          
#> [25] "Lao People's Democratic Republic"             "Macao"                                       
#> [27] "Micronesia, Federated States of"              "Moldova, Republic of"                        
#> [29] "Palestine, State of"                          "Réunion"                                     
#> [31] "Russian Federation"                           "Saint Barthélemy"                            
#> [33] "Saint Helena, Ascension and Tristan da Cunha" "Saint Martin (French part)"                  
#> [35] "Saint Vincent and the Grenadines"             "Sint Maarten (Dutch part)"                   
#> [37] "South Georgia and the South Sandwich Islands" "Syrian Arab Republic"                        
#> [39] "Taiwan, Province of China"                    "Tanzania, United Republic of"                
#> [41] "Timor-Leste"                                  "Turks and Caicos Islands"                    
#> [43] "United Kingdom"                               "United States"                               
#> [45] "United States Minor Outlying Islands"         "Venezuela, Bolivarian Republic of"           
#> [47] "Viet Nam"                                     "Virgin Islands, British"                     
#> [49] "Virgin Islands, U.S."

Standardise variable values

We can explore the extent to which we can use fuzzy logic to reconcile some of these discrepancies. To identify the types of fuzzy logic algorithms we could use, run the following command to explore the relevant part of the documentation from the stringdist library.

# Not run
# help("stringdist-metrics", package=stringdist)

In this case, we have chosen the Jaro, or Jaro-Winkler distance method (“jw”).

X <- renew(X, "jw", type_1L_chr = "slot", what_1L_chr = "logic") 
X <- ratify(X, new_val_xx = NULL)

This method will replace every previously invalid seed dataset variable value with the best available match identified by the selected fuzzy logic algorithm.

X@results_ls$Country_Output_Validation$Invalid_Values
#> character(0)

However, some of the replacements will be spurious as can be seen by inspecting the record of the replacements made.

X@results_ls$Country_Output_Correspondences
#> # A tibble: 41 × 2
#>    old_nms_chr               new_nms_chr                          
#>    <chr>                     <chr>                                
#>  1 Azores                    Timor-Leste                          
#>  2 Bolivia                   Bolivia, Plurinational State of      
#>  3 British Virgin Islands    Virgin Islands, British              
#>  4 Brunei                    Brunei Darussalam                    
#>  5 Canary Islands            Åland Islands                        
#>  6 Cape Verde                Cabo Verde                           
#>  7 Congo Democratic Republic Congo, The Democratic Republic of the
#>  8 Czech Republic            Czechia                              
#>  9 East Timor                Eswatini                             
#> 10 Easter Island             Christmas Island                     
#> # ℹ 31 more rows

For each of the incorrect correspondences, we will need to manually specify correct values. We can do this using the ready4show_correspondences sub-module.

# Not run
# a <- ready4show::renew.ready4show_correspondences(ready4show::ready4show_correspondences(), 
#         old_nms_chr = c("old_name_1", "old_name_2", "etc...."), new_nms_chr = c("new_name_1", "new_name_2", "etc...."))

The make_country_correspondences can be used as a shortcut for creating the alternative correspondences for this specific example.

We can inspect the values of this correspondence table.

exhibit(a, scroll_box_args_ls = list(width = "100%"))
Old name New name
Azores Portugal
Canary Islands Spain
Easter Island Chile
East Timor Timor-Leste
Ivory Coast Côte d'Ivoire
Kosovo Kosovo
Madeira Portugal
Netherlands Antilles Bonaire, Sint Eustatius and Saba
Sicily Italy
Vatican City Holy See (Vatican City State)

When the ratify method was used to apply the fuzzy logic algorithm in a previous step, X was modified so that this logic is by default switched off for future calls to ratify. If we had created a new correspondence table that specified replacements for all invalid values, this would not be a problem. However, in this example we are only specifying correspondences where the fuzzy logic algorithm failed, so we need to again supply our desired fuzzy logic value.

X <- renew(X, "jw", type_1L_chr = "slot", what_1L_chr = "logic") 

We now rerun our ratify method (which in this example will combine fuzzy logic with lookups from the manually created correspondences table).

X <- ratify(X, new_val_xx = a)

We once again inspect results.

Our correspondences table looks better.

X@results_ls$Country_Output_Correspondences
#> # A tibble: 41 × 2
#>    old_nms_chr               new_nms_chr                          
#>    <chr>                     <chr>                                
#>  1 Azores                    Portugal                             
#>  2 Bolivia                   Bolivia, Plurinational State of      
#>  3 British Virgin Islands    Virgin Islands, British              
#>  4 Brunei                    Brunei Darussalam                    
#>  5 Canary Islands            Spain                                
#>  6 Cape Verde                Cabo Verde                           
#>  7 Congo Democratic Republic Congo, The Democratic Republic of the
#>  8 Czech Republic            Czechia                              
#>  9 East Timor                Timor-Leste                          
#> 10 Easter Island             Chile                                
#> # ℹ 31 more rows

There is still a value that is not included in our standards.

X@results_ls$Country_Output_Validation$Invalid_Values
#> [1] "Kosovo"

We can rerun the ratify method to force the removal of any record that is not included in our standards dataset.

X <- renew(X, T, type_1L_chr = "slot", what_1L_chr = "force") 
X <- ratify(X, new_val_xx = "identity")

No invalid values remain.

X@results_ls$Country_Output_Validation$Invalid_Values
#> character(0)

However, there are also a some values from our standards dataset that are not represented in the results dataset values.

X@results_ls$Country_Output_Validation$Absent_Values
#>  [1] "Åland Islands"                                "Antarctica"                                  
#>  [3] "Bouvet Island"                                "British Indian Ocean Territory"              
#>  [5] "Christmas Island"                             "Cocos (Keeling) Islands"                     
#>  [7] "Curaçao"                                      "French Southern Territories"                 
#>  [9] "Heard Island and McDonald Islands"            "Hong Kong"                                   
#> [11] "Macao"                                        "Sint Maarten (Dutch part)"                   
#> [13] "South Georgia and the South Sandwich Islands" "United States Minor Outlying Islands"

Whether this is a problem or not depends on the intended purposes of the standardised dataset we are creating. We could choose to rerun the previous steps after making edits to either or both of the standards dataset (e.g. we could delete any superfluous, outdated or incorrect records or use an entirely new standards dataset) and seed dataset (e.g. adding new records or recategorising existing records so that there are corresponding values for every missing standard value). In this case we are going to assume that the above missing values are not a cause for concern for the valid use of our updated dataset for it intended purposes. We can now create a new object Y, using our results dataset’s Ready4useDyad module instance.

Y <- X@results_ls$Country_Output_Lookup

We can inspect the records for cases corresponding to capital cities from our new dataset.

renewSlot(Y,"ds_tb",Y@ds_tb %>% dplyr::filter(capital==1)) %>%
  renew(type_1L_chr = "label") %>%
  exhibit(scroll_box_args_ls = list(width = "100%"))
Dataset
City name Country name Population size Latitude coordinate Longitude coordinate Is the nation's capital city
'Amman Jordan 1303197 31.95 35.93 1
Abu Dhabi United Arab Emirates 619316 24.48 54.37 1
Abuja Nigeria 178462 9.18 7.17 1
Accra Ghana 2029143 5.56 -0.20 1
Adamstown Pitcairn 51 -25.05 -130.10 1
Addis Abeba Ethiopia 2823167 9.03 38.74 1
Agana Guam 1041 13.47 144.75 1
Algiers Algeria 2029936 36.77 3.04 1
Alofi Niue 627 -19.05 -169.92 1
Amsterdam Netherlands 744159 52.37 4.89 1
Andorra la Vella Andorra 20314 42.51 1.51 1
Ankara Turkey 3579706 39.93 32.85 1
Antananarivo Madagascar 1463754 -18.89 47.51 1
Apia Samoa 40805 -13.83 -171.76 1
Asgabat Turkmenistan 823013 37.95 58.38 1
Asmara Eritrea 578860 15.33 38.94 1
Astana Kazakhstan 351343 51.17 71.47 1
Asuncion Paraguay 507574 -25.30 -57.63 1
Athens Greece 725049 37.98 23.73 1
Avarua Cook Islands 13645 -21.20 -159.76 1
Baghdad Iraq 5753612 33.33 44.44 1
Bairiki Kiribati 45982 1.33 172.99 1
Baku Azerbaijan 1118725 40.39 49.86 1
Bamako Mali 1342519 12.65 -7.99 1
Bandar Seri Begawan Brunei Darussalam 67077 4.93 114.95 1
Bangkok Thailand 4935988 13.73 100.50 1
Bangui Central African Republic 547668 4.36 18.56 1
Banjul Gambia 34388 13.46 -16.60 1
Basse-Terre Guadeloupe 11298 16.00 -61.72 1
Basseterre Saint Kitts and Nevis 12883 17.31 -62.73 1
Bayrut Lebanon 1273440 33.88 35.50 1
Beijing China 7602069 39.93 116.40 1
Belgrade Serbia 1113589 44.83 20.50 1
Belmopan Belize 14590 17.25 -88.79 1
Berlin Germany 3378275 52.52 13.38 1
Bern Switzerland 120596 46.95 7.44 1
Biskek Kyrgyzstan 915625 42.87 74.57 1
Bissau Guinea-Bissau 404119 11.87 -15.60 1
Bogota Colombia 7235084 4.63 -74.09 1
Brasilia Brazil 2260541 -15.78 -47.91 1
Bratislava Slovakia 422452 48.16 17.13 1
Brazzaville Congo 1326975 -4.25 15.26 1
Bridgetown Barbados 98725 13.11 -59.61 1
Brussels Belgium 1031925 50.83 4.33 1
Bucharest Romania 1862930 44.44 26.10 1
Budapest Hungary 1700019 47.51 19.08 1
Buenos Aires Argentina 11595183 -34.61 -58.37 1
Bujumbura Burundi 336561 -3.37 29.35 1
Cairo Egypt 7836243 30.06 31.25 1
Canberra Australia 324736 -35.31 149.13 1
Caracas Venezuela, Bolivarian Republic of 1808937 10.54 -66.93 1
Castries Saint Lucia 12904 14.03 -60.98 1
Cayenne French Guiana 62926 4.92 -52.34 1
Charlotte Amalie Virgin Islands, U.S. 10415 18.35 -64.94 1
Chisinau Moldova, Republic of 623671 47.03 28.83 1
Cockburn Town Turks and Caicos Islands 174 21.46 -71.14 1
Colombo Sri Lanka 649496 6.93 79.85 1
Conakry Guinea 1970382 9.55 -13.67 1
Copenhagen Denmark 1091978 55.68 12.57 1
Dakar Senegal 2406598 14.72 -17.48 1
Damascus Syrian Arab Republic 1580909 33.50 36.32 1
Dhaka Bangladesh 6724976 23.70 90.39 1
Dili Timor-Leste 163305 -8.57 125.58 1
Dodoma Tanzania, United Republic of 188150 -6.17 35.74 1
Doha Qatar 351381 25.30 51.51 1
Douglas Isle of Man 25621 54.15 -4.48 1
Dublin Ireland 1030431 53.33 -6.25 1
Dushanbe Tajikistan 538456 38.57 68.78 1
Dzaoudzi Mayotte 14558 -12.77 45.25 1
Fakaofo Tokelau 267 -9.38 -171.22 1
Fort-de-France Martinique 89233 14.60 -61.08 1
Freetown Sierra Leone 818709 8.49 -13.24 1
Gaborone Botswana 214412 -24.65 25.91 1
George Town Cayman Islands 30570 19.28 -81.39 1
Georgetown Guyana 236878 6.79 -58.16 1
Gibraltar Gibraltar 26404 36.14 -5.35 1
Guatemala Guatemala 1010253 14.63 -90.55 1
Ha Noi Viet Nam 1452055 21.03 105.84 1
Hamilton Bermuda 889 32.30 -64.79 1
Harare Zimbabwe 1575127 -17.82 31.05 1
Havanna Cuba 2163132 23.13 -82.39 1
Helsinki Finland 558341 60.17 24.94 1
Honiara Solomon Islands 57410 -9.43 159.91 1
Islamabad Pakistan 794431 33.72 73.06 1
Jakarta Indonesia 8556798 -6.18 106.83 1
Jamestown Saint Helena, Ascension and Tristan da Cunha 603 -15.92 -5.71 1
Jerusalem Israel 731731 31.78 35.22 1
Jibuti Djibouti 633884 11.56 43.15 1
Kabul Afghanistan 3120963 34.53 69.17 1
Kampala Uganda 1403619 0.32 32.58 1
Kathmandu Nepal 822930 27.71 85.31 1
Khartoum Sudan 2090001 15.58 32.52 1
Kiev Ukraine 2491404 50.43 30.52 1
Kigali Rwanda 800003 -1.94 30.06 1
Kingston Jamaica 585300 17.99 -76.80 1
Kingston Norfolk Island 890 -29.03 168.05 1
Kingstown Saint Vincent and the Grenadines 18160 13.16 -61.23 1
Kinshasa Congo, The Democratic Republic of the 8096254 -4.31 15.32 1
Koror Palau 11458 7.35 134.51 1
Kuala Lumpur Malaysia 1482359 3.16 101.71 1
Libreville Gabon 591356 0.39 9.45 1
Lilongwe Malawi 683477 -13.97 33.80 1
Lima Peru 7857121 -12.07 -77.05 1
Lisbon Portugal 508209 38.72 -9.14 1
Ljubljana Slovenia 254188 46.06 14.51 1
Lome Togo 737751 6.17 1.35 1
London United Kingdom 7489022 51.52 -0.10 1
Longyearbyen Svalbard and Jan Mayen 1263 78.21 15.61 1
Luanda Angola 2875277 -8.82 13.24 1
Lusaka Zambia 1306577 -15.42 28.29 1
Luxemburg Luxembourg 76380 49.62 6.12 1
Madrid Spain 3146804 40.42 -3.71 1
Malabo Equatorial Guinea 161409 3.74 8.79 1
Male Maldives 87154 4.17 73.50 1
Managua Nicaragua 990417 12.15 -86.27 1
Manama Bahrain 147894 26.21 50.58 1
Manila Philippines 10546511 14.62 120.97 1
Maputo Mozambique 1220167 -25.95 32.57 1
Maseru Lesotho 116268 -29.31 27.49 1
Mata'utu Wallis and Futuna 1310 -13.28 -176.13 1
Mbabane Eswatini 78740 -26.32 31.14 1
Mexico City Mexico 8659409 19.43 -99.14 1
Minsk Belarus 1747482 53.91 27.55 1
Mogadishu Somalia 2723378 2.05 45.33 1
Monaco-Ville Monaco 975 43.74 7.42 1
Monrovia Liberia 954458 6.31 -10.80 1
Montevideo Uruguay 1271664 -34.87 -56.17 1
Moroni Comoros 43704 -11.74 43.23 1
Moscow Russian Federation 10472629 55.75 37.62 1
Muscat Oman 24122 23.61 58.54 1
N'Djamena Chad 737281 12.11 15.05 1
Nairobi Kenya 2864667 -1.29 36.82 1
Nassau Bahamas 231519 25.06 -77.33 1
Ni Dilli India 321883 28.60 77.22 1
Niamey Niger 801297 13.52 2.12 1
Nicosia Cyprus 202488 35.16 33.38 1
Nicosia Cyprus 42372 35.18 33.37 1
Nouakchott Mauritania 731242 18.09 -15.98 1
Noumea New Caledonia 94751 -22.27 166.44 1
Nuku'alofa Tonga 23733 -21.14 -175.22 1
Nuuk Greenland 15243 64.18 -51.73 1
Oranjestad Aruba 30710 12.53 -70.03 1
Oslo Norway 821445 59.91 10.75 1
Ottawa Canada 885542 45.42 -75.71 1
Ouagadougou Burkina Faso 1119775 12.37 -1.53 1
Pago Pago American Samoa 4180 -14.24 -170.72