Construction of a virtual reality platform for computer-aided navigation Lingnan bone setting technique

ABSTRACT

To establish a standard Traditional Chinese medicine (TCM) bone setting technique, standardize the operation and inherit the TCM bone setting technique. This project was based on the interactive tracking of bone setting techniques with a dedicated position tracker, the motion tracking of bone setting techniques based on RGBD (Red Green Blue Depth) cameras, the digital analysis of bone setting techniques, and the design of the virtual reality platform for bone setting techniques. These key technical researches were combined to construct an interactive bone setting technique. The virtual simulation system can reproduce the implementation process of the expert’s bone setting technique. The user can observe the implementation of the manipulative technique from multiple angles; through human-computer interaction, the whole process of implementation of the bone setting technique can be simulated, and the movement and reduction of the affected bone can be observed at the same time. It can be used as a teaching and training system for assisting bone setting techniques. Students can use the system to carry out repeated self-training, and can instantly compare with the standard techniques of the expert database, breaking the traditional teaching mode of ‘expected and unspeakable’ and avoid directly using patients. Therefore, this research makes it possible to reduce teaching costs, reduce risks, improve teaching quality, and make up for the lack of teaching conditions. It is very positive for the inheritance of the traditional Chinese ‘intangible culture’ of bone setting techniques, and to promote the digitalization and standardization of bone setting techniques.

HIGHLIGHTS

• Using computer technology to digitally record bone-setting manipulations.

• Construct a virtual simulation system for interactive bone-setting manipulation.

• Promote the digitization and standardization of bone-setting techniques.

KEYWORDS:

• TCM orthopedics
• computer aided
• digitization
• virtual reality platform
• bone-setting

Introduction

The TCM bone setting technique is characterized by ‘no surgery, fast recovery, and low cost,’ which treats bone injuries such as fractures and joint dislocations, and is very popular among patients [1]. However, many hospitals and even TCM hospitals have basically abandoned this therapy. The formal orthopedics is only passed on in less than ten hospitals of the Northern and Southern schools. In these hospitals, only the elderly and retired doctors are familiar with Chinese orthopedics, and other doctors use western medical methods such as surgery to treat bone injuries. TCM orthopedic therapy is experiencing embarrassment, struggling, and is on the verge of being lost [2,3].

One of the reasons for this situation is that the traditional orthopedic teaching adopts the traditional Chinese doctor’s apprenticeship, one-to-one teaching method, and directly performs manual manipulation exercises on the fracture patients, which increases the patient’s pain and suffers from incorrect operations. The patient brings new injuries, and at the same time makes the students feel nervous and flustered by visiting the patient, the learning cycle is long and the effect is poor. On the other hand, for the storage and dissemination of bone setting techniques, the traditional methods are recorded through text and pictures, but they cannot vividly show the entire process of bone setting techniques; it is saved by making animations, and the animation produced each time cannot be used in the next time. Repeated use has caused a very serious waste of resources; recording the implementation process of the bone setting technique through video recording often fails to record the implementation of the technique from multiple angles, unable to reflect the condition of the bone in the treatment process, and the audience cannot interact with it [4,5]. Experience the problem of bone setting process. The third reason is that the bone-setting technique of TCM has not yet formed a standard and objective operating specification, which has greatly affected the inheritance of the experience of famous doctors [6,7].

The orthopedic technique of Foshan TCM Hospital originated from the Li’s Traumatology Department in the Qing Dynasty. It has been passed down for five generations and is one of the important representatives of the Lingnan Orthopedics School. He has accumulated rich experience in the treatment of bone and joint injuries and intractable diseases in orthopedics and traumatology. With its unique diagnosis and treatment style and excellent curative effect, it has a long-standing reputation in China, Hong Kong, Macao, Taiwan and Southeast Asia. Professor Chen Zhiwei is an outstanding representative of the fourth generation of Li’s Traumatology, with superb bone setting techniques.

The project ‘Research and Realization of the Key Technology of Virtual Simulation of TCM Bone Setting’ plans to use computer three-dimensional virtual simulation technology to record the Bone Setting in a digital way. Through the research of the key technologies in this project, the interactive bone setting virtual simulation system built with these technologies can reproduce the expert’s implementation process of the bone setting method, and users can observe the implementation of the method from multiple angles; through human-computer interaction, The whole process of simulating bone setting techniques can be used to observe the movement and reduction of the affected bone at the same time. It can be used as a teaching and training system for assisting bone setting techniques. Students can use the system to perform repeated self-training and compare with the standard techniques in the expert library. To break the traditional teaching mode of ‘expected but unspeakable,’ avoid the shortcomings of directly using patients for training, and improve the efficiency and quality of teaching. At the same time, through the research of this project, explore the standardization of bone setting techniques.

We speculate that through the development of this project, it can reduce teaching costs, reduce risks, improve teaching quality, and make up for the lack of teaching conditions. Therefore, we expect to digitize and standardize bone-setting manipulations through computer technology, so as to promote the inheritance of traditional Chinese medicine bone-setting manipulations.

Method

Research on the interaction between virtual human hand and human hand

Geometric modeling of virtual human hand

The geometric model of the virtual human hand is required to be realistic, mainly to build the shape and appearance of the human hand. The model is not realistic and lacks realism; the model is too complex, which will affect the real-time interaction between the data glove and the virtual hand. The shape of the virtual hand is determined according to the real shape of the human hand. The size, body shape and characteristics of the human hand determine the complexity of the shape of the virtual hand. The focus of this project is on the interaction technology between human and virtual hands. Therefore, the virtual human hand model is simplified, that is, the shape of the virtual human hand only includes the wrist, palm, fingers and joints, excluding other elements such as handprints. The study was approved by the Foshan Hospital of Traditional Chinese Medicine [Approval No. KY[2021]136]. Informed consent was obtained from all individuals included in this study.

In this project, the geometric modeling of the virtual human hand adopts the method of combining the skeleton model and the volume model to keep the skeleton model and appearance consistent. The construction of the geometric model is first completed through the 3dsMax modeling tool, and then the 3ds model is called through OpenGL to redraw, avoiding the excessive modeling workload caused by the pure use of OpenGL for modeling, and at the same time, real-time control of the model can be achieved [8]. Taking into account the real-time requirements of the virtual hand to interact with the data glove, we choose to draw the virtual human hand using a triangular mesh (the same method is used for modeling the human body and skeletal skin). Compared with the parametric surface method, although the fidelity is worse, the real-time performance is better. In the study of the appearance of the virtual hand, the characteristics of the shape, material, and illumination of the virtual hand are set according to common situations.

Dynamic control of the virtual human hand

After the geometric model of the virtual human hand is established, its motion control relationship needs to be established according to the movement characteristics of the human hand, so that the spatial data and position data input by the position tracking system can be used to change the motion of the virtual human hand in real time. We regard the human hand as a multi-bar and multi-hinge structure. The joints of different parts of the joint chain realize different levels of movement, and have the transfer characteristics from top to bottom, which constitute the hand model, palm model, finger model, and joint model. Hierarchical relationships and association relationships. Joints of the same level have the same or similar characteristics, and joints of different levels have a hierarchical attachment relationship between upper and lower levels, forming horizontal and vertical correlation and irrelevance. The relevance is mainly reflected in the relevance of the graphic coordinate system. The origin of the coordinate system of the virtual human hand is set at the fulcrum of wrist movement, and the joints of the fingers are set as the coordinate origin of the finger knuckle model. The relationship between the global coordinate system and the local coordinate system of the virtual human hand. The movement of the virtual human hand includes movement, rotation, swing and other movements. These movements are constrained by various joints. The motion of the virtual human hand is composed of the motion of the virtual human hand itself as a whole and the gesture motion compounded by the simple motion of multiple joints. The final compound motion can be seen as a simple movement, rotation, swing, etc., which are superimposed several times. We realize the control of the virtual human hand through a series of coordinate transformations. In order to transfer the gesture information collected by the data glove to the virtual human hand, we established a one-to-one mapping relationship between the sensor of the data glove and the joints of the virtual hand [9].

Modeling research method of fracture end and skin

Modeling of virtual scene We mainly focus on the construction of physical model of fractured bone segment and skin. For the construction of the geometric model of the typical fracture end, we use CT, MRI and other image data to perform 3D reconstruction on the virtual reality work platform, and then import the model into the 3dsMax tool for further processing to obtain a model that can be processed by OpenGL. We simplified the skin, muscle, fat and other tissues and considered them to be the same soft tissue. We mainly used the face model and the body model for geometric modeling. When observing the appearance of the skin, the face model was used for modeling; The cross-section animation display of the end-of-side re-correction simulation process adopts volume model modeling. We adopt the mass-spring model for the physical model of soft tissue [10]. Think of soft tissue as a mass damping system.

Collision detection research methods

At present, the collision detection algorithms for the two models can be roughly divided into two categories: space decomposition method and hierarchical bounding box method [11,12]. The space decomposition method divides the entire virtual space into small cells of equal volume. Only the geometric objects occupying the same cell or adjacent cells are tested for intersection. The spatial decomposition method is usually suitable for geometric models with a relatively uniform distribution in a sparse environment. The core idea of the hierarchical bounding box method is to use the volume-level bounding box method. The core idea is to use a slightly larger bounding box with simple geometric characteristics to approximate complex geometric objects, so that only objects with overlapping bounding boxes are further intersected. test. The hierarchical bounding box rule is more widely applied to collision detection in complex environments. Using the hierarchical bounding box method to detect the collision between the virtual human hand and the virtual fractured bone will achieve better results.

Deformation calculation research methods

At present, the methods for deformation calculation include the principle of elasticity and the finite element method. Due to the principle of elasticity, the problem of soft tissue deformation is to solve the mass point displacement when the total energy equation of the system is the minimum under the action of external force. As for the solution of such a mathematical equation, it is very difficult to find its solution due to the irregularities of geometry, skin characteristics and external loads. So, we consider using the finite method. The finite element model is composed of interconnected sub-domains (units), and its model gives a piecewise (sub-domain) approximate solution of the basic equation, because the continuum (or object) can be divided into various shapes and sizes of different sizes The unit (sub-domain), so it can adapt to complex geometric shapes, complex material properties and complex boundary conditions. In order to show the process of deformation, we add mass and damping to the model, regard the process of deformation as a dynamic linear elastic finite element system, and solve the deformation through numerical calculations.

Establishing the database of standard manipulative techniques for bone setting

For the same type of fractures, 30 patients were selected, and the experts performed manual reduction to obtain the movement data of the hands. The data was fitted with statistical methods to generate a standard movement trajectory, and the experts calculated multiple times The fluctuation range of the technique. For different maneuvers and fracture sites, the maneuver process is divided into multiple stages according to the key force points, and the expert action direction of the key force points is abstracted, the maneuver database model is established and the relevant maneuver data is stored in the database.

Results

This study constructed a virtual simulation system of interactive bone-setting manipulation, which could reproduce the implementation process of expert bone-setting manipulation, and could be used as a teaching and training system to assist bone-setting manipulation. The system could promote the digitization and standardization of bone-setting manipulations.

Interactive tracking of bone setting techniques based on a dedicated position tracker

The entire process of bone setting virtual training involves three aspects: users, hardware devices and training systems. In the virtual training environment, the device management module establishes a connection with the 5DT data glove and the trakSTAR position tracker to obtain the user’s hand movement information in real time. The information data is transferred to the virtual training module after transformation, and then the virtual training module completes the hand movement Changes and realize the movement of the hands in the virtual scene. After the bone setting operation is over, the training module intercepts the manipulative movement trajectory and outputs it to the judging module. After the evaluation module completes the evaluation of the bone setting technique, it returns the result to the training module of the system, and sends the evaluation result to the scene.

The system uses 5DT Data Glove Ultra 14 series data gloves to capture hand movements. 5DT Data Glove is a product specially created for professionals in the field of motion capture and animation production. It has the characteristics of comfort, high refresh rate, cross-platform, and high data quality. The data glove captures the hand gestures in real time by measuring the curvature of each joint. There are 14 sensors built into the glove to measure the curvature of 14 positions, as shown in Figure 1a. Among them, the joint curvature of each finger is described by two sensors, and the degree of expansion between adjacent fingers is measured by one sensor. The data collected by each sensor is a floating-point number between 0 and 1. In the actual use process, these data need to be converted into the actual rotation angle of the joint.

Figure 1. Hand movement information collection system. (a) the position of the joint point of the data glove. B, 5DT Data Glove’s data collection hand interface. C,trakSTAR position tracker. D, the data collection icon interface of the 5DT data glove. E, bone setting technique record and playback. 3D trajectory acquisition and playback software.

Ascension 3D Guidance position tracker (trakSTAR) is an actual high-precision electromagnetic tracker specially designed for applications that require short-distance motion tracking. It consists of an electronic unit, a transmitter and multiple sensors, as shown in Figure 1c. The position tracker has the characteristics of fast, dynamic tracking, anti-metal interference, micro-sensor, and high precision, and is suitable for but not limited to the medical field.

It can track multiple sensors within the working range (1 meter) of the transmitter and obtain their position and direction information in real time. Wherein, the position information includes x coordinate, y coordinate, and z coordinate, which are the components of each coordinate axis of the local coordinate system formed by the transmitter as the origin, and the unit is inch. The heading information is composed of azimuth, pitch and tilt angles.

When collecting data, judge whether the data glove needs to be calibrated according to the calibration settings. When collecting initial data, the 14-sensor system of the 5DT data glove generates real-time data of each sensor cyclically, and outputs it as a CSV format file, which is then converted according to a certain algorithm and processing, the output is the angle data actually needed. (Figure 1b,d) Three-dimensional trajectory acquisition and playback software are used in the process of trajectory entry, as shown in Figure 1e. This software is a trajectory acquisition tool specially designed for the trakSTAR position tracker to acquire bone setting techniques. The three-dimensional trajectory acquisition tool supports real-time tracking of multiple sensors, and can dynamically display the motion trajectory in the software, and supports offline modification of trajectory data. The trajectory data collected by the software will be stored in the set file directory in the format of a text file. The file stores up to four motion track information, including the acquisition time (hour: minute: second: millisecond) of each track point in the motion track and the corresponding position information (x coordinate, y coordinate, z coordinate) and orientation information (Azimuth angle, pitch angle, tilt angle).

Bone setting motion tracking based on RGBD camera

The movement track of the doctor’s wrist is tracked and calibrated by binding a Bluetooth sensor on the doctor’s wrist, and the rotation of the wrist is recorded. Through the complementary correction of the two devices, the purpose of accurately simulating the doctor’s hand movements is achieved. The data flow of this system is shown in Figure 2. There are four data source methods, one of which is to read from a data file that has been saved locally. Take, the other three are from different devices, first from the finger movement information of the 5DT glove. Secondly, it comes from another subsystem model control system under the TCM bone setting simulation system. It is responsible for obtaining the rotation information of the doctor’s bone node from the Kinect device and controlling the character model in the virtual scene, and calculates the position information of the hand model and transmits it to this system. Finally, the hand posture information obtained by the hand sensor. After these data are summarized in the system, they are cached using the trajectory data structure, and after relevant processing such as smoothing and cropping, they are saved in a local file.

Figure 2. Schematic diagram of data flow.

Digital analysis of bone setting techniques

The spatio-temporal data of experts’ techniques are collected digitally and merged into the database. It can record the results of multiple implementations of the same technique by different experts. The actions of the orthopedic surgeon in the virtual environment are collected in real time through computer software and specific time and space data are archived in the database, which can record the results of multiple implementations of the same technique by different surgeons. As a result, accurate comparative analysis and research can be conducted on the technique of the surgeon. The digital analysis of the virtualized results of the bone setting technique, that is, the identification and statistical analysis of the trajectory of the technique, includes the following contents.

The complex motion trajectory can be seen as a combination of multiple simple trajectory segments. Any four points on the curve have three relationships in space: collinear, coplanar but not collinear, and different planes. We define four basic shapes according to common manipulative actions: straight line, plane curve, left-handed space curve, right-handed space curve (Figure 3a).

Figure 3. Digital analysis of bone setting techniques. (a) the basic type of motion trajectory. (b) the direction of movement of the track deviating from the plane. (c) the result of trajectory segmentation, the red line is a straight-line type, the green is a plane curve, the blue is a left-handed space curve, and the black is a right-handed space curve. (d) Based on DTW-based trajectory segment matching, the black path represents the optimal matching result, where the left side refers to the structure matching of the motion trajectory, and the right side refers to the trajectory point matching between the trajectory segments. (e), Manipulation characterization and recognition. F, match technique.

These four basic types include the change information of the various dimensions of the motion trajectory, and can completely represent a motion trajectory. All three-dimensional motion trajectories can be divided into sub-trajectory segments of these four basic shapes, and these four types of trajectories are used. Segments are modeled and represented.

The motion trajectory is presented in the form of a space curve in three-dimensional space. A space curve can describe its spatial shape through curvature and torsion. The former is used to describe the degree of curvature of the curve, and the latter is used to describe the degree of torsion of the curve. These two geometric attributes can also be used to describe the spatial shape of the trajectory. We can obtain the relationship between curvature, torsion and basic shape, so the basic shape trajectory segment can be divided by curvature and torsion. It has also been found in practice that the curvature and torsion are unstable and sensitive to changes in the trajectory scale. This topic further proposes a segmentation method based on the direction of motion, which uses the angular change of the trajectory direction to replace the curvature and torsion: the angle of the trajectory movement vector from the plane is used to represent the torsion, and the angular change value of the trajectory movement vector is used to represent the curvature, and finally clear the shorter trajectory segment (Figure 3b). The trajectory segmentation method based on the direction of motion is the same as the segmentation method based on curvature torsion, which is the process of traversing the markers of the trajectory points. We first traverse each trajectory point, calculate the angle of change of their movement direction, and mark them as categories. The trajectory points where the marker type changes are determined as segmentation points, and finally the motion trajectory is divided into several trajectory segments according to these segmentation points (Figure 3c).

After the motion trajectory is divided, it can be regarded as a sequence of trajectory segments composed of a set of basic shapes. The trajectory segment needs to be further described in order to be suitable for expression and matching. Based on the dynamic time warping algorithm (Dynamic Time Warping, DTW) trajectory segment matching method, we realize the alignment between trajectory segments. The main purpose of DTW is to find an optimal matching path from two trajectory segment sequences, the cumulative value of the distance of all elements of the path is the smallest, and the elements on the path represent the distance between the two matched trajectory segments. Find an optimal matching path from the two trajectory segment sequences, the cumulative value of the distance of all elements of the path is the smallest, and the elements on the path represent the distance between the two matched trajectory segments. The smaller the total distance of the optimal matching path, the more similar the two trajectories, with the same structure, and the more similar trajectory segments (Figure 3d). Finally, based on the above method, the real maneuver trajectory recorded by the expert is required to characterize and match as shown in Figures 3e,f.

Action matching verification based on expert techniques

The above method is applied to the track of expert technique recorded by the system to verify the effect of technique matching. Preparation stage: data collection of bone setting, invited a number of experts in the field of bone injury to collect data; establishing a standard bone setting data set, and classifying maneuvers. Manipulative action evaluation: Calculate the average distance between the manipulative data of each level and the standard data as the basis for action evaluation.

Because we have accurate three-dimensional spatial data of the key observation points of the operator’s hand, we can perform quantitative decomposition and analysis of expert techniques in different dimensions. Figure 4a projected from three planes and analyzed the shape of the expert maneuver; Figure 4b observed the action parameters in the three dimensions, and we can see the moments when the action changes in different directions. Figure 4c characterizes the speed change when the expert performs the technique. When the movement is smooth, the movement speed is relatively high, while the movement direction changes greatly, and the movement speed is relatively slow. Figure 4d shows the geometric characteristics of the trajectory curve when the orthopedic expert performs the technique with curvature and torsion respectively. From Figure 4e, it can be seen that the curvature and torsion fluctuate greatly when there is a bending action. Based on the curvature and torsion, the three-dimensional trajectory curve of the action of the osteopathy expert can be segmented and detected (Figure 4f). It can digitally judge the spatial change of the analysis technique. Compare the expert bone setting maneuver with each level template, calculate and evaluate their similarity distance.

Figure 4. Action matching verification based on expert techniques. (a) Three plane projections and analysis of the form of expert maneuvers. (b) Observe the action parameters in three dimensions. (c) the speed change when the expert uses the technique. (d) shows the geometric characteristics of the trajectory curve when the bone orthopedic expert performs the technique with curvature and torsion. E, the fluctuation of curvature and torsion. F, expert method segmentation and recognition. (g) matching template. H, the trajectory of the technique used in the test.

The design of the virtual reality platform for bone-setting techniques

The functions of the entire system are summarized as recording mode, movie mode, learning mode, and auxiliary modules. The recording mode is responsible for the recording and saving of the bone-setting technique data, and reproduces the Chinese process in the virtual scene in real time while recording. In this process, it is necessary to handle the process of equipment access, equipment calibration, equipment data acquisition, real-time reproduction of the real scene in the virtual scene, cutting of recorded data, and writing of recorded data to files. The movie mode is responsible for reading the existing bone setting technique data and reproducing this process in the virtual scene. The main process is to read file data and control the character model in the virtual scene. Of course, in order to facilitate students’ learning, it is necessary to add playback progress control here. Because there is a comparison process in the learning mode, there are two input sources, one of which is the reference input source and the other is the comparison input source. It is necessary to control the movement of the character model in the scene according to these two input sources in the same virtual scene. In this way, students can clearly see the difference between the two-input data bone-setting techniques, which is convenient for learning. Therefore, the learning module has the following process: reading existing file data as a reference input source, real-time recording or reading existing data as a reference input source, and controlling the movement of the character model in the virtual scene respectively according to these two input sources. The auxiliary module is common in the business scenarios of the above three modules, and mainly includes the following content, scene roaming, used to control the perspective change in the virtual scene. Character model motion trajectory drawing, used to draw the motion trajectory of the character model in the virtual scene. Parameter chart, used to draw some parameter charts calculated from bone setting data, such as speed-time curve chart. So far, we have clearly expressed some requirements for this simulation system. The scene and motion effect diagram of the system is shown in Figure 5. The scene of the system is inside a small hospital, which is created and completed in 3D modeling software.

Figure 5. The scene and motion effect diagram of the design system of the virtual reality platform of the bone setting technique. A, system scenario. B, system operation diagram.

Discussion

Chinese medicine bone therapy is experiencing embarrassment, struggling, and is on the verge of being lost [13–15]. In order to establish a standard TCM bone setting technique, standardize the operation and inherit the TCM bone setting technique, this project is based on the interactive tracking of the bone setting technique with a dedicated position tracker, the movement tracking of the bone setting technique based on the RGBD camera, the digital analysis of the bone setting technique, and the virtual reality of the bone setting technique. The platform design and the constructed interactive bone setting virtual simulation system can reproduce the expert’s bone setting method implementation process. Users can observe the method’s implementation from multiple angles; through human-computer interaction, the entire process of simulating the bone setting method is simulated simultaneously. It can observe the movement and reduction of the affected bone. It can be used as a teaching and training system for assisting bone setting techniques. Students can use the system for repeated self-training. Through this research, it can reduce teaching costs, reduce risks, improve teaching quality, and make up for teaching conditions. The shortcomings of TCM have a very positive significance for the inheritance of the traditional Chinese ‘intangible culture’ and the promotion of the digitization and standardization of bone setting techniques.

Virtual reality technology is a key research field of computer science, involving computer graphics, multimedia technology, artificial intelligence, human-machine interface technology, sensor technology, and highly parallel real-time computing technology [16]. It also includes many key technologies such as human behavior research. Virtual reality can give users a more realistic experience, and it provides great convenience for exploring the macroscopic and microscopic worlds and for various reasons that are not convenient for directly observing the law of movement of things. It is this advantage that makes virtual reality technology gain wide attention in the medical field, and has been applied to medical teaching, disease diagnosis, surgery simulation, rehabilitation medicine, telemedicine. The Karlsruhe Research Center in Germany successfully developed the Karlsruhe Endoscopic Surgery Trainer, a virtual laparoscopic surgery training system in 1997 [17]. They developed the powerful 3D simulation software KISMET as the core software of the training system, which can complete real-time physics simulation, kinematics simulation and fast 3D graphics rendering [18,19]. The training system uses a spring vibrator model to simulate the deformation of soft tissues such as the uterus, and connects the nodes on both sides of the object by adding parent nodes, so that the deformation of the surface model shows body behavior, which meets the requirements of real-time and fidelity. The virtual surgery system developed by the Boston Dynamics Research Center can not only produce very realistic medical three-dimensional images of blood vessels, heart, and brain, but also has precise force feedback effects [20–23]. Surgery students can use the system’s human-computer interaction equipment to train virtual surgery. When using the system, surgical trainers can use surgical tools such as ‘virtual forceps’ and ‘virtual suture needles’ to perform ‘virtual blood vessel’ suture operations. Through polarized glasses and stereoscopic display device, the trainer can get the experience of vascular suture operation in a real operation environment. The use of virtual reality technology has been reported in the teaching and training of TCM acupuncture and moxibustion, the virtual teaching system of wrist and ankle acupuncture, and bone and joint surgery [24–26]. From the current situation at home and abroad, the application of virtual reality technology to TCM bone setting provides interactive virtual simulation of bone setting Research on teaching has not yet started [27,28]. This project builds an interactive bone setting virtual simulation system based on the bone setting method interactive tracking of the dedicated position tracker, the bone setting method motion tracking based on the RGBD camera, the bone setting method digital analysis and the design of the bone setting method virtual reality platform. The system can record the bone setting techniques digitally, and can reproduce the implementation process of the expert bone setting techniques.

The recording and playback of the bone setting technique is the core issue of this research. Considering the cost and promotion convenience of a dedicated location tracker, we explored the use of general-purpose visual equipment to carry out a complete bone setting technique capture program. Based on the current common RGBD depth camera Kinect’s human bone motion tracking technology, it realizes the recording of bone setting techniques. Only using Kinect to record the bone setting technique has the aforementioned inaccuracy problem. The first is that Kinect itself only supports the collection of information such as the depth position of one or two people. When there is a third assisting doctor, the system will not work properly. Second, the trajectory of the hand of the bone setting technique is important, but a series of ‘skills’ or skills such as the turning of the wrist is also the key to the success of the treatment, and the 5DT glove can only collect information on the change in the degree of bending of the finger, and Kinect The same will ignore the rotation of the wrist, so if you record the movement trajectory of the hand but cannot record the simulation of the rotation of the wrist or the details of the hand, the treatment that can be completed is also greatly limited. In this study, a Bluetooth sensor was bound to the doctor’s wrist to track and calibrate the movement trajectory of the doctor’s wrist, and to record the rotation of the wrist. Through the complementary correction of the two devices, the purpose of accurately simulating the doctor’s hand movements is achieved. There are four ways to source data in this system, one of which is to read from a data file that has been saved locally, and the other three are from different devices. The first is from the finger movement information of the 5DT glove. Secondly, it comes from another subsystem model control system under the TCM bone setting simulation system. It is responsible for obtaining the rotation information of the doctor’s bone node from the Kinect device and controlling the character model in the virtual scene, and calculates the position information of the hand model and transmits it to this system. Finally, the hand posture information obtained by the hand sensor. After these data are summarized in the system, they are cached using the trajectory data structure, and after relevant processing such as smoothing and cropping, they are saved in a local file. The advantage of this system is that there is enough dimensional information to track the movement of the hand, and the repeated information can be used to restrain and correct the information. This study also has certain limitations. The establishment of the model requires more input from famous Chinese medicine practitioners, and more actual cases for verification and adjustment. In addition, prognostic assessment and follow-up are lacking. This is our future research direction and content to be optimized.

Conclusion

This study successfully constructed a virtual reality platform for computer-aided navigation Lingnan bone setting technique. Through this research, it is very positive to promote the digitalization and standardization of bone-setting techniques.

Author contributions

RJH is responsible for the guarantor of integrity of the entire study, study concepts & design, literature research, manuscript preparation; ZQX is responsible for the study concepts, manuscript editing & review; HGG is responsible for the study concepts, study design, manuscript editing & review; JZ and KYY are responsible for the definition of intellectual content; SWP is responsible for the experimental studies, data analysis and statistical analysis; HBY is responsible for the clinical studies, experimental studies, data analysis and statistical analysis; MYF is responsible for the clinical studies; LLH is responsible for the clinical studies; YZZ is responsible for the clinical studies; YJ is responsible for the clinical studies; GZ is responsible for the literature research, data acquisition; HLZ and CHC are responsible for the data acquisition. All authors read and approved the final manuscript.

Ethics approval and consent to participate

The study was approved by the Foshan Hospital of Traditional Chinese Medicine [Approval No. KY[2021]136]. Informed consent was obtained from all individuals included in this study.

Availability of data and material

All data generated or analyzed during this study are included in this. Further enquiries can be directed to the corresponding author.

Disclosure statement

No potential conflict of interest was reported by the authors.

Additional information

Funding

This study was funded by the Foshan Competitive Support Talent Project (2021) [Fo Tutong (2021) No. 52].