Co-development of the educational content of a proposed educational mobile health application prototype on oral cancer using Delphi technique

Study design

This study adopted a Delphi study design, and it was conducted and reported in accordance with the Guidance on Conducting and REporting DElphi Studies (CREDES) framework²⁸. This Delphi study comprised four phases. The first phase involved the development of an oral cancer educational content through literature review while the second, third, and fourth phases involved expert consultations on the developed educational content (Fig. 1).

The flow of the phases of the Delphi study.

Scientific committee

It is crucial to have a scientific committee who oversee the conduct of a Delphi study²⁸. The scientific committee are experts in the research topic area, and they play a crucial role in ensuring the study is conducted with rigour^26,28. In this study, the authors played the role of the study’s scientific committee²⁶. This study’s scientific committee consists of two dental public health scientists/practitioners (KKK, AAS, LAN), one oral pathologist (AOA), one oral and maxillofacial surgeon (RDJ), and one biomedical scientist in oral oncology (YAJ). The scientific committee members were chosen based on their academic portfolio, expertise, and experience in oral cancer research, practice, and advocacy (all members have at least three years of experience in oral cancer research, practice, or advocacy, and they have published extensively on oral cancer). None of the scientific committee members was a participant in this study.

Subject matter experts

The subject matter experts engaged in this study were individuals who met the following eligibility criteria:

1. 1.

   Have at least one academic/specialist qualification in oral and maxillofacial surgery, oral medicine, oral pathology, dental public health, public health, general medicine, and general dentistry.

2. 2.

   Have at least two years of experience in their area of specialization.

3. 3.

   Are willing to participate in the Delphi study.

A sample size of sixty subject matter experts was considered sufficient for this study, as this sample size is well above the average range of sample size (6 to 50) for most Delphi studies^25,26.

These subject members were recruited through two international oral cancer research organisations which has memberships across different continents (including Africa, Asia, Europe, and North America). These organisations are namely: (i) Consortium for Head and Neck Cancer in Africa (formerly called the International Head and Neck Cancer Working Group)²⁹, and (ii) the Asia-Pacific Oral Cancer Network³⁰.

The scientific committee of this study contacted one executive in each these two organisations requesting them to invite eligible experts in their networks to participate in this study. Only those experts that met all the eligibility criteria were engaged in this study as subject matter experts.

Study instruments

Two study instruments were used in this Delphi study, and both were semi-structured questionnaire. The first instrument (Supplementary file 1) was used in the second phase of this study, and it obtained the following information from the subject matter experts:

1. i.

   Sociodemographic characteristics (including country location, age, gender, area of specialisation, and years of practice in specialisation area).

2. ii.

   Grading of the relevance the content of each section in the educational content using a Likert scale of 1 to 4 (1 – Not relevant; 2 – Somewhat relevant; 3 – Quite relevant; 4 – Highly relevant).

3. iii.

   Comments on each section of the shared educational content.

The second instrument (Supplementary file 2) was used in the third phase of this study, and it obtained the following information from the subject matter experts:

1. i.

   Grading of their satisfaction on the content of each section in the revised educational content using a Likert scale of 1 to 4 (1 – Very dissatisfied; 2 – Dissatisfied; 3 – Satisfied; 4 – Very satisfied).

2. ii.

   Comments on each section of the shared educational content.

The second instrument did not obtain information on the sociodemographic characteristics of the subject matter experts engaged in the third phase because only those that participated in the first round of expert consultation (i.e. Phase 2 of this study) was invited to participate in this study phase.

Phase 1: development of the first version of the educational content

This study phase involved the development and production of the first version of an educational content on oral cancer through literature review. The literature used to develop this educational content was obtained through searches of selected databases (PubMed, SCOPUS, ScienceDirect, and Google Scholar) and websites of selected national and international health organisations (including World Health Organization (WHO), National Health Service [UK], Centers for Disease Control and Prevention, American Cancer Society, National Center for Chronic Disease Prevention and Health Promotion, and Cancer Research UK). The literature used for the content development were recent reviews, published between 2013 and 2024, and were obtained using multiple combinations of the following search terms: ‘oral cancer’, ‘oral squamous cell carcinoma’, ‘epidemiology’, ‘burden’, ‘risk factors’, ‘clinical features’, ‘symptoms’, ‘signs’, ‘diagnosis’, ‘treatment’, ‘management’, ‘prognosis’, ‘support’, and ‘organisation’. From the search, the most informative, recent, and relevant literature were chosen and used to develop the content. Furthermore, in the content development process, the recommendations for developing an informative, educational and communication materials for laypersons, by Thorseth³¹, were adopted as a guide, since the target population for the content are lay populations.

Phases 2–4: expert consultation on the educational content

Phases 2 to 4 of this study involved consulting the participating subject matter experts to obtain their feedback on the education content. All consultation processes were done via email correspondence.

Phase 2

This study phase (Phase 2) was the first round of consultation on the first version of the educational content. In this phase, sixty consenting subject matter experts were sent the first version of the educational content and a questionnaire (Supplementary file 1) to obtain qualitative and quantitative feedback concerning the relevance and quality of each section of the educational content. Based on the feedback, the first version of educational content was revised, producing the second version.

Phase 3

This study phase (Phase 3) was the second round of consultation on the second version of the educational content. In this phase, only those subject matter experts that participated in Phase 2 of this study were invited. They were sent the second version of the educational content together with a questionnaire (Supplementary file 2) and a document containing the feedback on the rebuttal and actions taken by the scientific committee concerning each comment provided by the subject matter experts in the first round of Delphi consultation and were asked to review the new version of the content and provide their feedback on their level of satisfaction and additional comments, if any, concerning the revisions made, using the questionnaire provided. Based on the feedback obtained, the second version of educational content was revised, producing the third version.

Phase 4

This study phase (Phase 4) was the third round of consultation on the third version of the educational content. In this study phase, only those subject matter experts that provided additional comments (i.e. qualitative comments) were contacted. These experts were sent the third version of the educational content, and a document containing the feedback on the rebuttal and actions taken by the scientific committee concerning each comment they have provided in the previous study phase (Phase 3). In this phase, one or more cycles of email correspondence was done between the subject matter experts and the scientific committee, and all subject matter experts contacted finally agreed with the actions/rebuttal of the scientific committee. Based on the inputs obtained from the subject matter experts, the third version of the educational content was revised to produce the fourth version of the content.

Data analysis

Both quantitative and qualitative data were obtained in this Delphi study. However, only the quantitative data were analysed. The qualitative data obtained were review comments on the educational content. In line with the study objectives, the qualitative data obtained do not require data analysis; however, they were used to inform the rounds of revisions done on the educational content that is under development.

The Statistical Package for Social Sciences (SPSS) version 28 software was used for the quantitative data analysis. In line with the study objectives, only descriptive statistics were done to determine the frequency distribution of all variables, and patterns of some selected variables using measures of central tendency (mean, mode, and median) and dispersion (range, and standard deviation).

Definition of consensus

Consensus was determined using the mean of the responses obtained via Likert scales in the first and second rounds of expert consultations. The Likert scale used in the first round obtained feedback on the level of relevance of each section of the educational content while the one used in the second round obtained feedback on the level of satisfaction of the subject matter experts.

Consensus was considered to be achieved for each section of the educational content only when a mean Likert score of 3 or above was obtained from the responding subject matter experts during the first round of Delphi consultation, and this established the content validity of each section of the content. Furthermore, only those sections with a mean score of 3 or above were retained in the educational content³².

Ethical considerations

This study was done in strict compliance with the 1964 Helsinki Declaration on health research involving human subjects. Ethical clearance to conduct this study was approved by the School of Health and Life Science Research Sub-Committee of Teesside University (Ref: 2024 Mar 20233 Kanmodi). Participation in this study was completely voluntary and confidential.