Tool for assessment of attitudes for applying a new approach in the decision-making process for reimbursement list in Bosnia and Herzegovina

Abstract

The healthcare structure in Bosnia and Herzegovina is decentralized, thus facing patients with unequal access to healthcare services especially for medicines. We attempted to develop a tool for assessment of the stakeholders’ opinion, and with the further research we will propose the model that can bring equality in Bosnia and Herzegovina. The developed tool will examine the attitudes and opinions for introducing new methods in the decision-making process during the listing of medicines for the reimbursement list in Bosnia and Herzegovina. An update of the country’s legislation can be presented based on the research results. The tool was developed using the Delphi method. The experts who were included in the Delphi panel are qualified for rating and discussing questions. The questionnaire was validated on a 5-point Likert scale, and additional comments or clarification are optional. Introductory interviews were held face to face with each expert individually; after that the panel was anonymous. After 3 rounds of Delphi, the created tool was checked through a pilot study. The developed tool was categorized into three groups based on the KAP survey (Knowledge, Attitudes, Perceptions). All questions were verified in pilot and with the results for Cronbach’s alpha 0.96. This shows sufficient reliability of the created questionnaire, and it can be administered to a larger group of respondents, which has been planned for further research. The findings provide reliable information useful for planning the country’s legislation updates and planning for the introduction of a new approach in the decision-making process.

Keywords:

• Delphi method
• decision-making
• reimbursement list
• pharmacoeconomic
• healthcare

Introduction

Key characteristics of the healthcare systems in Bosnia and Herzegovina are that it is complex, decentralized, fragmented and with low efficiency [1]. Bosnia and Herzegovina is defined as a state with two entities (Republic of Srpska comprising of approximately 1 million residents and Federation of Bosnia and Hercegovina comprising approximately 2.1 million residents) each with a high degree of autonomy. There is also a self-governing Brcko District of Bosnia and Herzegovina (comprising of approximately 80,000 residents). The entity of Federation of Bosnia and Herzegovina (hereinafter FBIH) is further divided into 10 cantons with their own governments each having joint jurisdiction of healthcare with FBIH. All administrative units meaning State, Entity, Cantonal and District have their own governance structures and jurisdictions often overlapping with each other. This also reflects the complexity and non-effectiveness in reimbursement policies and reimbursement lists creation on all administrative levels.

This kind of constitutional division of jurisdictions results in an extremely complex institutional structure, whereas the healthcare policy decision-making process is devolved to the entity/district level, and the two entities and the district each have distinct Laws on Health Care and on Health Insurance. Furthermore, in FBIH, there is a shared jurisdiction in healthcare between FBIH (entity) and 10 cantons. This results in a total of 14 ministries of health and 13 health insurance funds, of which 10 are health insurance funds of cantons in FBIH, 2 are entity ones (FBIH and RS) and 1 is in Brcko District, all having more or less the same jurisdictions systematically overlapping. This structure often results in, not only different contents of reimbursement lists, but often in completely different model approach in the creation of reimbursement lists [2–4].

The reimbursement list in the FBIH contains only a list of INNs. Listing a new INN on the reimbursement list requires administrative data, and updating should be performed annually, but this is not the case in practice. The cantonal reimbursement lists of medicines are created based on a list from FBIH. Cantons choose independently the trade names of a drug for listing. The criterion for listing the brands on the cantonal list is that medicines need to be registered and marketed in the territory. The updating of the cantonal lists is not regulated, so it can be done annually or not, depending on each canton’s decision.

Contrary to entity FBIH, the health system in entity RS is fully centralized. A reimbursement list in the RS is determined by the Health Insurance Fund. Afterwards, the entire determined list of drugs becomes available in the whole territory of RS. Updates are available during the year when the registration has finished. Recent legislation update requires submission of budget impact analysis and cost-effectiveness analysis within application for listing the new INN or extension of indication. There are no clear guidelines for the above-mentioned requirement.

In BD, the Department of Health is in charge of making a proposal of a reimbursement list that contains only INNs [5].

Prime concern for medicines are safety and efficacy but economic evaluation enables efficient decision for allocation healthcare resources [6]. Economic evaluation provides information about value for money of medicines [7–10]. Choosing adequate method for economic evaluation should consider components, comparator, outcome, internal and external validity [11] and decision-making context. Based on William F., the social perspective should be always discussed in ­evaluative reports [12]. Evidence is visible thought publications that analyzed cost-utility, cost-effectiveness methods and quality of life [8–10,13]. Pharmacoeconomic data can support decisions for inclusion or exclusion of medicines and support national guidelines [14] with attention to analytical methods used for economic analyses [15].

One of the detected issues in the B&H healthcare system is the introduction of new methodologies and legislation updates such as health technology assessment. The incompatibility of the process during the evaluation and decision-making process pertaining to medicines is a significant issue [16–18]. Stakeholders’ opinions are an important part of legislation updates and harmonization on every level in B&H.

The purpose of this research was to develop a questionnaire for assessment of the attitudes for applying a new approach in the decision-making process that will be distributed to the stakeholders (pharmaceutical industries and decision makers). The developed tool will be used to assess the attitudes for applying a new approach in decision-making process for new medicines on the reimbursement list.

Methods

Ethics statement

All work was conducted with the formal approval of the Board of Ethics of the University of Sarajevo, Faculty of Pharmacy for experimental tests for preparation of projects, scientific research papers and final papers (No. 0101-5474/22 from 18-Oct-2022).

Study design

The questionnaire was validated using the Delphi method. The Delphi methodology commences with the identification of experts, proceeds through the design of questionnaires, data collection (including the number of rounds), and data analysis, culminating in data interpretation. Although there were different methods that could be used to avoid the limitations of the Delphi study, we previously published а systematic literature research on this topic where we assessed the appropriateness of using the Delphi in our study [19].

The study lasted for two months (from 01.09.2023 to 01.11.2023). The criteria for the study were based on recommendations from Diamond et al. [20], and the score was achieved.

In this research, the Delphi method was used to assess and generate expert opinions for the proposed questionnaire. The questionnaire was developed by the research group utilizing a literature review, official government guidelines and recommendations, and the known situation in Bosnia and Herzegovina. The questionnaire comprised of 17 questions, which were categorized into three groups based on the KAP survey (Knowledge, Attitudes, Perceptions) [21]. All procedures were executed in the local language, and translations have been conducted specifically for the purpose of this paper. The potential risk of the Delphi is that it should not be utilized as a decision-making tool, but rather as a decision-analysis tool [22]. Following this potential risk, Delphi was created and conducted for evaluation and assessment of the given statement/questions. The research team decided for final formulation of the questions based on the received comments.

Expert identification as one of the challenging parts was supported with data from literature considering five aspects: (i) Size of the panel, (ii) level of expertise, (iii) level of heterogeneity, (iv) level of interest and (v) access to panel [23]. The representatives invited to Delphi are from the academic community involved in the research field and experts from the health authority (not involved in the decision-making process, but involved in the creation of the legislation). Considering the sample size of the final research, which included respondents from the pharmaceutical industry and healthcare institutions in the country, we deemed it justified to select five experts for the Delphi method. Delphi was organized as an anonymous panel [24]. During the introductory interviews, the primary researcher informed the experts of the purpose of the Delphi method and the methodology for its execution. As presented, the research objectives were also divided into two distinct parts. It was conducting a Delphi and pilot research with smaller groups of respondents. The participants were also apprised that all procedures were to be conducted anonymously. Communication and rounds were conducted via email. Originally, the Delphi method was planned to have at least three rounds. The questions were validated on a 5-point Likert scale, and additional comments or clarification are optional. Considering the lack of clear guidelines for consensus [20], it was decided that the percentage of consensus would be 85%. If consensus was not achieved after the third round, the research team would discontinue the Delphi process and conduct a re-evaluation of the questionnaire based on the expert opinions and comments received and conclude the survey with the appropriate justification.

Exclusion criteria were a cut-off value (consensus below 85%) and expert comment. If the experts failed to achieve consensus of 85% for a particular question, and if they failed to comment on it or propose modifications, it would be excluded from the questionnaire [25].

Pilot study

The questionnaire that was created was validated in a pilot study. We conducted a survey on this topic due to the impact of the country status on the size of the samples. It is worth mentioning that B&H is a relatively small country, with a population of approximately 3 million. There are 84 registered Marketing Authorization Holders (MAH) representing the pharmaceutical industry and healthcare institutions. These individuals are deemed decision makers in certain points for medicines, and there are currently 12 institutions involved. During the creation of this questionnaire, we considered the level of knowledge and capacity of both the MAH and the healthcare institution. It was decided to distribute the questionnaire to 30 respondents (20 from the pharmaceutical industries and 10 from the healthcare authority). The pilot study was performed with approximately 30 participants.

The respondents were selected from the pharma industry decision makers (national health institutions). After the pilot, a thorough evaluation of the marketing authorizations holders in Bosnia and Herzegovina, as well as the national health institutions involved in the decision-making process for medicines, was conducted.

Results

The questionnaire contained 17 questions, which were categorized into three groups based on the KAPA (Knowledge, Attitudes, Perceptions) model. The first group consisted of knowledge-based questions, such as: What is PE study? Have you ever used it in the decision-making process? The second group of questions focused on the perspective of PE and the data that PE should provide to decision makers. This tool was created to assess the attitudes toward the methods and main principles. The third group of questions examined the perceptions about the country’s status and individual opinion about the quality of the PE legislation in the country.

First round

The questionnaire was sent to all experts for assessment at the same time. The respondents were prompted to evaluate each question using a 5-point Likert scale, which categorized it as Very Important, Important, Moderately Important, Slightly Important and Unimportant. The initial round lasted the longest, with reminders being sent to experts. The first round of assessment was conducted, with results presented in Table 1. For 9 questions, consensus was achieved with a percentage of 88% or more. For questions where consensus was achieved, the standard deviation was between 0.447 and 0.894.

Table 1. Delphi process results.

	1st round			2nd round			3rd round
	Std. Deviation	% consensus	Cronbach’s Alpha	Std. Deviation	% consensus	Cronbach’s Alpha	Std. Deviation	% consensus	Cronbach’s Alpha
Question 1	0.548	88	0.672
Question 2	0.548	92
Question 3	0.548	88
Question 4	1.304	84		1.095	76	0.632
Question 5	1.517	72		0.548	92
Question 6	0.548	92
Question 7	0.707	80		0.837	84		–	92	–
Question 8	1.342	68
Question 9	0.548	88
Question 10	1.140	72
Question 11		80
Question 12	0.447	96
Question 13	1.304	76
Question 14	0.894	92
Question 15		100
Question 16	0.548	92
Question 17	1.095	76

In the initial round, consensus was not achieved for 7 statements out of a total of 17. Cronbach’s Alpha value was 0.672, which indicates a re-evaluation and adaptation of the statements. It was done according to the comments received from experts. On the other hand, for the statements that did not receive any comments or a consensus, as per the methodology established, the items were excluded. Additional comments were received regarding questions 4, 5 and 7 and corrections were made accordingly.

Second round

The second round had three questions that were modified based on the feedback received. The responses were received within a few days. Table 1 shows the results for three amended questions, with a consensus for one question (Table 1) having a standard deviation of 0.548. Two questions without a consensus were evaluated. One of them was amended based on the received comment and was sent back for assessment in the third round. The second one was excluded based on the above-mentioned exclusion criteria.

Third round

During Round 3, the remaining questions were evaluated, and consensus was reached with 92%.

The questionnaire was looked at again by researchers, and after they agreed, the pilot was started.

Pilot study

It was performed with smaller samples. The questionnaire was sent to 30 respondents (around 20 respondents from pharmaceutical industries and 10 respondents from healthcare institutions). The survey was anonymous, and the responses did not show respondents’ personal data.

Twelve responses were received, which was sufficient for validation. The coefficient alpha, commonly referred to as Cronbach’s alpha, was calculated from the responses, and it was determined to have a value of 0.96. This shows sufficient reliability of the developed questionnaire, and it could be administered to larger groups of respondents as it has been planned for further research.

Discussion

Considering the fact that Bosnia and Herzegovina lacks centralized or harmonized regulation in the healthcare sector, this study aimed to develop a fundamental questionnaire to explore the level of knowledge and willingness to adopt novel methods. Eleven reimbursement lists are valid for different parts of B&H (cantons and entities). They represent a major issue for financing and equal healthcare for all patients in B&H. The participants in this Delphi study were familiar with this complex and unharmonized situation, and they provided their insights and validated the questionnaire appropriately.

The feedback we received during the initial round was highly encouraging, as we received very thoughtful comments. Unfortunately, not all the experts were willing to provide comments, which can be justified by their working position and habits. However, comments were not required. Delphi was created with the mandatory point being statement assessment by a 5-point Likert scale, not commenting on the proposed statements. The rationale for expert evaluation was also elucidated by Cronbach’s coefficient, which was 0.672 for the initial round, indicating the necessity for correction and reformulation of certain statements. Cronbach’s alpha was an additional criterion because we assessed the percentage of consensus as the main parameter, as it was set up in the methodology.

Through groups of questions, consensus was achieved for the first group, basic questions, in the first round without any additional comments or corrections. In the third group of questions, there were disagreements only about two questions. The first excluded question was related to the needs of external experts for conducting the PE study. The panelists’ approach was encouraging, as they were confident that the local experts were adequate to support the idea of incorporating novel methods into the decision-making process. The second question from this group that was excluded concerned the availability of space or additional comments to include ideas or proposals, defined as ‘free space’ in the questionnaire. The experts did not consider these questions important. Finally, in the initial round, agreement was reached for the first and third groups of questions.

The second group of questions had the most disagreements among the panelists. A detailed process flow is explained below.

The consensus was not achieved for question number 4, and the comment for it is clearly stated.

Which product should undergo a pharmacoeconomic study?

1. For each medicine, both generic and innovative, that was included in the reimbursement list during revision

2. Only for new medicines when applying for the first time for the reimbursement list.

3. For the medicines specified by the authority

4. Other

One of the experts assessed that the question was essential, but also with comments that it should be reformulated, and added also that PE should be requested when the applicant does not agree with the price determined by local legislation for price calculation. The legislation mandates the implementation of a referral pricing system for all registered products in B&H. Another expert responded that for option c, they recommend formulation according to the EU guidelines. The question was amended according to these comments, but it was still not able to reach consensus in further rounds and was excluded from the final questionnaire. Some experts did not find this question relevant for the questioning of the pharmaceutical industries and healthcare institutions. The research team deliberated on the possibility of eliminating this inquiry as it was deemed relevant. However, despite the new proposal of questions and correction as suggested by the experts, consensus was not achieved. Therefore, the ultimate decision was to adhere to the methodology and exclude this question from the final questionnaire.

Question number 5:

What should be the purpose of the PE study?

1. cost limitation

2. value for money

3. total budget

4. patient well-being.”

The question was also below the cut-off value, but received the following comment: ‘The proposed choice should be redefined to explain, e.g. Define usage of medicines, availability for a defined group of patients, and sustainability of medicine financing.’ It would be advisable to refrain from presenting PE as a limitation tool, instead presenting it as a rational resource allocation. This was an interesting and useful observation. Considering the sensitivity of the matter and the reimbursement list users, it is imperative to concentrate on the accurate interpretation of our objective, which is to ensure equal healthcare and equal availability of medicines. This can only be achieved through the utilization of appropriate methodologies in the decision-making process. It is also imperative to prevent any potential misinterpretation. The question was amended and stated as follows:

“The main purpose of PE study should be.

1. Defining the scope of the usage of medicines, rationalization of prescribing.

2. Ensure the availability of medicines for certain groups of patients within the approved indication.

3. Sustainability of financing the health system.”

In the second round, the amended question achieved consensus and was listed in the final questionnaire.

The second-group question that was discussed was question number 7.

The comparator in PE studies should be

1. Applicants choice

2. Defined by authority

3. Defined by legislative

4. Other

This statement was considered crucial, but also gave a comment: ‘Please explain or add standard therapy currently on the reimbursement list.’ A second expert suggested including ‘EU guidelines.’ The comments were useful and, based on them, this statement was amended and sent to the second round.

Experts received formulation: ‘Determining the comparator in PE studies should be:

1. applicant’s choice

2. current used therapy, medicines already on reimbursement list

3. EU guidelines.’

Experts from the second round received 3 statements for assessment according to the amendments mentioned above. The main purpose of the PE study was agreed upon, but two other statements did not achieve consensus. A question related to the determination of the comparator was received (question number 7). As it was pointed out, the used comparator is applicable only for the current therapy, and reimbursed medicines, so points b) and c) are insignificant. The research team assessed that the formulation of the questions needed to be different. Based on that, the research team came to the conclusion for the question to state: ‘Comparators used in PE study should be medicines currently listed in reimbursement list? Yes/No’. The third round of the formulation of the statement achieved 92% consensus.

After three rounds, we received a list of the agreed-upon inquiries (Supplementary Appendix). The questionnaire was analyzed one more time by the research team and was checked before the pilot study. A questionnaire for the pilot study was dispatched to a total of 30 respondents. The responses that were provided were sufficient for statistical purposes. The alpha value of 0.96 for the pilot indicates a high level of clear comprehension of questions, thereby indicating the success of the Delphi study.

The initial round produced a significant number of comments and disagreement regarding question number 4 (For which product should a pharmacoeconomic study be conducted?). We considered this statement relevant for further research, and we excluded it based on the methodology set up after the second round. The other example was with question number 7. The panelist asked for a re-formulation, which led to an entirely different statement after two rounds. Firstly, it was ‘Determining the comparator in PE studies should be:…’ and finally ‘Comparators used in PE study should be medicines currently listed in reimbursement list? Yes/No’. The research team did not agree completely because we thought that giving more options for answering would provide us with more information and possibilities than just selecting Yes or No.

We received 12 responses from 30 respondents. The response rate was lower than expected, as the objective was to investigate the opinions of ­stakeholders regarding the implementation of novel methods and approaches in the decision-making process.

The questionnaire will provide us with information about the attitudes of pharmaceutical industries and decision-makers. Both groups of stakeholders will provide us with an understanding of what to expect. The aforementioned group of inquiries provides us with valuable insights regarding the potential degree of method complexity, the primary viewpoint, and the potential income. The third group of questions will provide us with exact information about stakeholders’ opinion on the current situation of legislation and possible solutions.

In addition to the collected data from the questionnaire, a proposal for a legislation update should consider literature recommendations; and the expected perspective will be valuable for methodology development [26].

Even though the legislation has been updated partially (only in RS cost-effectiveness and budget impact analysis were introduced), implementation moves very slowly. There are no additional guidelines; only the part of legislation that requires “…(5) analysis on Fund’s budget, (6) cost-effectiveness analysis (pharmacoeconomic analysis)…” [27].

Novel research for the reimbursement challenges recommends “managed entry agreements, fair pricing, and joint health technology assessment” [28]. Based on this, further research should be focused on the pricing system in B&H and joint health technology assessment. Because of the similarity in culture and language, we also consider the possibility of applying the developed questionnaire to other countries in the region with cultural adaptation.

Limitations

It is widely acknowledged that the Delphi method possesses a multitude of limitations [29]. Initially, the reproducibility of the outcomes and the ­potential for bias are of utmost importance [30]. Reproducibility is also conditioned by the choice of panelists and experts, but since the study is locally specific, our criteria cover local specific heterogeneity.

Conclusions

For development or modification of any tool for assessment, it is crucial to include all factors such as local specifics, level of development of the country and social status. These factors require inclusion, and the Delphi study provides this despite its limitations. The questionnaire was developed with a high level of reliability and is ready for further investigation. The pilot study confirmed the findings, and the second phase of the research, which will encompass the entire pharmaceutical industry and healthcare institutions in Bosnia and Herzegovina, should reaffirm them. Based on the data obtained here, it is feasible to consider a proposal for an update of the local legislation. The final result of the study will be the development of harmonized guidelines in B&H, resulting in the implementation of equal criteria for medicines across the country and, most importantly, equal availability of medicines for patients. The questionnaire can also be applicable to other countries in the region, with minor modifications.

Authors contributions

S.B., conception and design of the work, supervision, acquisition, analysis, interpretation of data, origial draft preparation; B.V, acquisition, analysis of data, writing; V. M., conception and design of the work, critical review for important intellectual content. All authors have read and approved the final version of the paper.

Acknowledgments

The authors thank all the panelists who took part in the Delphi rating process and the individuals who contributed to the study.

Disclosure statement

No potential conflict of interest was reported by the author(s).

Data availability statement

The data that support the findings of this study are available from the corresponding author, SB, upon reasonable request.

Additional information

Funding

This research was funded by the Ministry of Science, Technological Development and Innovation, Republic of Serbia through Grant Agreement with University of Belgrade-Faculty of Pharmacy No: [451-03-47/2023-01/200161].

Publication of study results was not contingent on the sponsor’s approval or censorship of the manuscript.

The funder had no role in the design and conducting of the study; collection, management, analysis and interpretation of the data; preparation, review, or approval of the manuscript; and decision to submit the manuscript for publication.