COLOSS B-RAP Expert Evaluation of Beekeeping Advice From ChatGPT, Part 1

The advanced language model ChatGPT is capable of understanding and generating human-like text. It can be integrated into various services, ranging from customer support to educational platforms, providing personalized assistance, information and guidance. For straightforward, low-complexity medical quest­ions, ChatGPT has been shown to have potential as an AI-assisted decision support tool in medicine (Harskamp & De Clercq, 2024). In apiculture, hive management is an important factor in maintaining healthy and productive honey bee colonies (Sperandio et al., 2019; Steinhauer et al., 2021). Artificial intelligence-based linguistic models could provide an easy-to-access advisory service in countries where no advisory services are available or to relieve advisors. At a workshop of the COLOSS core project B-RAP (Fabricius Kristiansen et al., 2022) held in Olomouc, Czechia, in February 2024, we, therefore, tested the ability of ChatGPT3.5 to deal with some common questions in beekeeping. The question formulation always included rough information on location and date and formulated the beekeeping-related problem as a question allowing an open answer. The panel of 13 experts present (researchers, beekeeping advisors, veterinarians), many of them beekeepers themselves, evaluated the answers.

For this article, we consider two questions which we asked ChatGPT3.5 regarding the serious honey bee disease American Foulbrood (=AFB), which is a notifiable disease in the whole of the European Economic Area (Chauzat et al., 2013). The questions differed in level of detail - the first one asked about general symptoms, while the second one asked for advice in a certain situation (AFB outbreak). The questions, (unaltered) ChatGPT output and experts’ comments are shown in Tables 1 and 2.

Table 1. ChatGPT output and experts’ comments on the question: ‘What are the symptoms of American foulbrood?’; Rating: C = correct and valuable information, sometimes minor details are missing; U = correct, but useless information for the question, M = correct, but important details are missing, I = Incorrect information.

ChatGPT3.5 output	Experts’ rating	Experts’ comments (additionally checked with WOAH, 2023)
American foulbrood (AFB) is a highly contagious and destructive bacterial disease that affects honey bee larvae. Some symptoms of American foulbrood include:	C
Discolored larvae: Infected larvae may appear darkened or discolored, often turning from pearly white to brown or black as they decay.	C
Ropy or sticky texture: Infected larvae may exhibit a "ropy" or sticky texture. When a toothpick is inserted into the decaying larvae, it often stretches out in a characteristic manner due to its stickiness.	C
Foul odor: A distinctive foul odor resembling that of rotting meat or sour, fermenting odor may be present around the affected hive.	C	better: in the affected hive
Sunken cappings: The cappings of infected brood cells may appear sunken or punctured, often indicating the larvae within have died and decomposed.	C	addition: cappings appear moist and darkened
Spotty brood pattern: Infected colonies typically exhibit a spotty brood pattern, with irregular or patchy distribution of healthy and infected brood cells.	M	spotty brood pattern only seen in severely infected colonies, single standalone capped cells are missing as symptom
Weak or dying colonies: Colonies affected by AFB may weaken over time, with reduced population numbers, decreased foraging activity, and ultimately, colony death if left untreated.	U
Presence of scale: In advanced stages, the remains of infected larvae may dry out and form hard, scale-like structures at the bottom of brood cells.	M	Scales are not removable (differentiation to scales of European foulbrood and Sacbrood)
If you suspect American foulbrood in your honey bee colonies, it’s crucial to consult with a qualified beekeeping expert or apiarist for accurate diagnosis and appropriate treatment options. AFB is a serious threat to bee colonies and can spread rapidly if not managed effectively.	C

Table 2. ChatGPT output and experts’ comments on the question: ‘What measures should be implemented in a beekeeping operation in Norway affected by American foulbrood?’ Rating: C = correct and valuable information, sometimes minor details are missing; U = correct and useless information for the question, M = correct, but important details are missing, I = Incorrect information.

ChatGPT3.5 output	Experts’ rating	Experts’ comments
American foulbrood (AFB) is a serious bacterial disease affecting honey bee larvae, and it can devastate bee colonies if not managed properly. Here are some measures that could be implemented in a beekeeping operation in Norway affected by AFB:	C
Isolation and Quarantine: Isolate-affected hives to prevent the spread of the disease to healthy colonies. Implement strict quarantine measures to prevent movement of infected equipment or bees.	I	Isolation is not possible, because in Norway it is not allowed to move colonies out of an area with an acute AFB case. Information is contradictory (quarantine vs. prevent movement).
Inspection and Diagnosis: Regularly inspect all hives for signs of AFB. Train beekeepers to recognize symptoms such as sunken, perforated, or discolored brood cappings, foul odor, and spotty brood patterns.	U	preventive measure
Hygienic Practices: Emphasize strict hygiene practices among beekeepers, including washing hands and tools thoroughly between hive inspections to prevent cross-contamination.	U	Preventive measure
Destroy Infected Material: Promptly and safely destroy any equipment or comb that shows signs of AFB. This includes burning infected frames and thoroughly sterilizing hive components.	M	Contradictory information (destroy - sterilize). Information is missing on how to deal with colonies themselves (e.g. destroy, shook swarm…).
Antibiotic Treatment: In some cases, antibiotics may be used to treat AFB-infected hives. However, this should be done under the guidance of a veterinarian or experienced beekeeping expert, as improper use of antibiotics can lead to antibiotic-­resistant strains of bacteria.	I	In Norway, the use of antibiotics in beekeeping is completely banned. Generally, there is a ban on the use of antibiotics in the EU beekeeping sector and there are no MRLs for antibiotics in honey bee products. There are no antibiotics registered for use as veterinary medical products for honey bees in the EU.
Requeen: Requeening the hive with a queen from a reputable breeder can help strengthen the colony’s resistance to diseases like AFB.	I	That’s not correct for this situation. New resistant queens may be a preventive measure, but do not help in an outbreak situation
Monitor Hive Strength: Monitor the strength and health of colonies regularly. Weak colonies are more susceptible to diseases, including AFB.	U	Preventive measure
Education and Training: Provide education and training for beekeepers on AFB prevention, detection and management. Encourage beekeepers to attend workshops, seminars, or courses on bee health.	U	Preventive measure
Record Keeping: Maintain detailed records of hive inspections, treatments, and any AFB incidents. This information can help track the spread of the disease and evaluate the effectiveness of management strategies	U + I	Preventive measure, in addition treatments (antibiotics) is not allowed in Norway
Collaboration and Reporting: Collaborate with local beekeeping associations, veterinary authorities, and research institutions to share information and resources for AFB management. Report any suspected or confirmed cases of AFB to the relevant authorities to help prevent further spread of the disease.	M	Very weakly phrased for one of the most important points. Information needs to refer to regulatory compliances (which it did in a similar question). This means that the information given in the answer is not reliable.
By implementing these measures, beekeepers can help mitigate the impact of American foulbrood on their operations and protect the health of their honey bee colonies.	C

Our small challenge on AFB shows: ChatGPT3.5 in its current state is a good teacher for standard beekeeping knowledge (e.g. content of a beginner’s course), but its use should be avoided for critical or specific situations. The results of question 1 show the strength of ChatGPT. It summarizes common knowledge about an important bee disease at the level of detail that can be found in textbooks for general beekeeping. All information given is correct although some important details are missing, according to the experts’ panel (Table 1). Also, the level of useless information is low. However, when being asked a detailed question on a critical situation (Table 2), the answer given is a mixture of insufficient information, useless clutter and incorrect information. This might, again according to the experts’ panel, even be dangerous for the health of the honey bees in the region (pathogen spread through transport or application of antibiotics). Inexperienced beekeepers may be confused at best and ill-advised at worst. Furthermore, the answers are not in compliance with Norway’s legislation and may lead to legal problems for the beekeeper.

It is evident that ChatGPT does not differentiate between important and nice-to-have information, as an advisor would do (Figure 1). It also does not structure information in a logical way, which would help the inquirer to put the acquired knowledge into action (e.g. step-by-step instruction).

Nonetheless, we see a possibility of ChatGPT (or a similar advanced language model) to act as powerful tools in an advisory service. They could provide beekeepers with personalized and summarized information for their current needs, in the language and level of detail, which they need. Let’s imagine a new beekeeping advisor tool, named ChatBEE being usable at both European and national levels, which has previously been trained with public domain resources of high quality, like those being produced within various (transnational) honey bee research efforts (e.g. FAO et al., 2021; BPRACTICES Consortium, 2020), research associations (e.g. the COLOSS BEEBOOK, see Carreck et al., 2022) or by the national beekeeping structures (e.g. reference laboratories for bee health, beekeeping advisor centers). Such basic information would allow ChatBEE to rank information for quality (e.g. preferring official sources or resources from research institutions over other internet sources). Additionally, the inquirers can do their own quality checks as ChatBEE gives the sources of the presented information. It is sensitive to regional differences in its answers (e.g. not advising for antibiotics in Europe) and highlights contradictory information (e.g. antibiotics not allowed in each country). It is also transparent about its ‘knowledge gaps’, giving short high-quality answers rather than filling the answer up with unnecessary information.

In Part two of the COLOSS B-Rap AI challenge, the focus will be on practical beekeeping and the importance of interaction in beekeeping advisory services.

Figure 1. AI-generated pictures from Bing Image Creator showing how beekeeping advising is depicted by an AI. It is based on the following prompts: ‘a photo of a beekeeping advisor giving advice to a group of beekeepers in Europe’.

Disclosure Statement of funding

No potential conflict of interest was reported by the author(s).