Assessing navigational map reading competencies with the location-based GeoGame “GeoGami”

Abstract

School curricula as well as scientific literature emphasize the need of understanding how children navigate with maps. To date, however, there is no empirically validated assessment tool for analyzing navigational map reading competencies that can be used in map-based navigation tasks in the real-world. This paper fills this gap by presenting GeoGami NMR Assessment, a test for measuring children’s navigational map reading (NMR) competencies for educational purpose. Our target group are primary school children aged six to ten. The assessment uses configural and directional map information to assess the learning progress of individual children regarding their spatial navigational performance. While we evaluated our GeoGami NMR Assessment for two specific locations, our study design, our script scoring participants’ performances as well as the assessment can easily be adapted by researchers and teachers for their own locations. Our item analysis evaluates difficulties of individual tasks. We validate our assessment and demonstrate, that GeoGami NMR Assessment is suitable to distinguish strong and weak navigational map readers.

Keywords:

• Navigational map reading
• navigational map reading assessment
• location-based GeoGame
• navigational map reading test

Introduction

Orientation and navigation with maps play an important role in people’s everyday life as well as in professional contexts (Hemmer et al. 2008). Both adults and children differ in their way and success in using the maps. For instance, literature reveals differences in navigation strategies of women and men (Dabbs et al. 1998) or in the map-based orientation among girls and boys or younger and older children (Hemmer et al. 2013). Considerable developmental changes in the use of maps were observed to take place at the age of eight to twelve (Hemmer et al. 2013; Liben & Myers 2007). However, the performance in using maps is not fixed to gender or age: Hemmer et al. (2013) studied more than 300 third to fifth graders and concluded that orientating with maps can be fostered via educational training. The results show a positive correlation between the children’s experiences in using maps and their success in orientating with them. These outcomes underline the significance of maps in national school curricula and educational guidelines for primary or secondary schools from all over the world (e.g.: Germany: DGfG 2020; Hong Kong: CDC-HKEAA 2017; UK: Department of Education (2013); USA: NAEP 2020). Scientific literature agrees with these curricula in the relevance of understanding and using maps for geoscience in school education (Liben & Myers 2007; Jones & Lambert 2017).

However, research still lacks an assessment for investigating children’s competencies in navigating with maps that is applicable at different outdoor locations. Existing studies do not fill this gap: For instance, Hemmer et al. (2013) investigated influencing factors on children’s map-based orientation competence with a factor-analytically evaluated test, but did not create a test with a set-up replicable at other settings. Lobben (2007) created a real-world map navigation exercise with an indoor set-up for validating navigational map reading ability paper pencil tests with adult participants, but did not validate the real-world exercise itself, nor developed a test especially for children or real-world environments. This paper presents a real-world assessment for measuring children’s competencies in using maps for orientation that is empirically evaluated and applicable at different locations.

Ishikawa and Kastens (2005) accentuate that methods in geoscience education should be chosen according to the individual abilities of each student. This raises the question of which learning resource can meet such individual needs. One answer can be found in geoinformation (GI) technologies: especially location-based tools bear the potential to automatically detect the user’s map interaction and movement in space as well as to react individually to it. Authors indicate the need of exploring this kind of technology from an educational perspective (Baker et al., 2015). The demand is supported by studies emphasizing people’s preference to use digital tools instead of paper maps for navigation (Hergan & Umek 2017; Hurst & Clough 2013; Speake & Axon 2012) indicating that GI technologies found their way into people’s lives. However, there is an evidence for negative short- and long-term effects of GI tools like navigation aid systems on people’s spatial knowledge, wayfinding, and orientation (Ishikawa 2019; Field et al. 2011; Münzer et al. 2006). The user’s passivity when following navigation instructions is one explanation for these effects (Ishikawa 2019). Since the tools investigated in these studies were not adjusted for educational purpose, we suspect that GI technologies can also have positive effects through implementations that actively address spatial thinking. This paper presents an assessment for investigating orientation with maps which is based on an educationally adjusted GI tool.

Research question and research design

We developed and evaluated an assessment for NMR competencies of primary school children aged six to ten (Figure 1). The assessment is created for educational purposes. This paper investigates to what extent it is possible to assess spatial NMR competencies. We define “assess” as the pedagogical evaluation of children’s NMR performances. In a pilot study, our assessment was empirically evaluated for two exemplary locations by estimating its reliability and validating it against teacher judgements. The implementation of our NMR assessment makes use of our mobile app GeoGami, which provides an avenue for implementing NMR assessment tests.

Figure 1. Design framework for the GeoGami NMR Assessment construction. The assessment development and evaluation are closely interlinked and are to be understood as iterative processes. The intention of the assessment determined the decisions on methods for the development and validation of the instrument.

Additionally, this paper offers a detailed description of our assessment design and provides open access to GeoGami (Bistron et al. 2021) as well as to an evaluation script (Bistron & Schwering 2021b) making the assessment also applicable at other locations. By running the evaluation script with their own data, the readers have access to the NMR scores for their participants and a validation of the assessment for the new location. This paper builds upon the preprint (Bistron & Schwering, 2021a).

Development of the NMR assessment

Theoretical construct “navigational map reading”

In the scientific literature, there are different terms defining map reading for orientation or navigation.

Hemmer et al. (2013) coin the term map-based (spatial) orientation (MBO) competence which refers to spatial transformation between map and reality. According to Ishikawa and Kastens (2005), three main understandings are involved in a map-reality transformation: The representational correspondence (understanding the meaning of the symbols), the configurational correspondence (finding the relation between a specific object and its representation on the map) and the directional correspondence (understanding the alignment of the map). MBO is defined as a competence which is a broadly defined term in educational contexts: on the one hand, competence or competency refers to the input (people’s skills required to achieve a certain performance) and, on the other hand, it describes the output (result of training) in the sense of a “competent” performance (Hoffmann 1999). Hemmer et al. (2013) formulate a theoretical construct for the MBO competence by dividing it into the ability to locate objects from the map in the real world and vice versa, the abilities in taking correct turns, in pointing in directions correctly, in estimating distances correctly as well as in aligning the map according to the view direction. Some of these abilities ask for map-to-real-world transformations, others for real-world-to-map transformations. The theoretical construct was revised factor analytically in a study with more than 300 participating children that were asked to navigate through a city center by following a route which was marked on a map. The results showed that tasks related to the above-mentioned abilities loaded onto one factor defined as the MBO competence.

Lobben (2007) uses the term navigational map reading (NMR) to describe navigating with a map. Her definition of NMR is close to the construct of the MBO competence by Hemmer et al. (2013) and the term competency defined by Hoffmann (1999): associated with NMR tasks, Lobben (2007) defines different spatial abilities referring to people’s aptitude (e.g. innate brain function) and achievements (through training and experience) in making spatial relations between objects in geographic space. In a study with 45 adults, she proved the predictive power of three of these abilities: self-location showed the highest predictive power followed by route memory and map rotation. The predictive power of these abilities was determined by correlating the participants’ performance in corresponding paper pencil tests with their results in a real-world NMR performance test. Self-location was studied by presenting participants a map and photos of different locations from the map. Using the photos, the subjects determined locations and view directions on the map. Route memory was investigated by showing the participants routes on maps and asking them to remember these routes by ordering verbal route descriptions. The map rotation ability was tested by presenting the participants two maps simultaneously and asking them to determine whether the second map was flipped or rotated. The real-world NMR performance test itself included self-location and navigation tasks. It took place in a hallway network of a large building. The task performance was evaluated by considering the number of manual map rotations and hesitancies, the number and duration of stops, as well as the amount of time spent studying the map.

In our research approach (c.f. Figure 1, “theoretical construct”), we will use the term navigational map reading (NMR) competencies in analogy with the definition of Lobben (2007). In accordance to Hemmer et al. (2013) we use the term competencies instead of abilities since we are interested in children’s trainable performances in NMR tasks.

Technical realization with the location-based geogame GeoGami

We created a mobile app called GeoGami (Bistron et al. 2021, Bartoschek et al., 2018, Bistron et al. 2022) for testing and training navigational map reading competences. For motivating children to participate in our studies and play our app, we implemented components of location-based game mechanics. GeoGami is running on GPS enabled mobile devices (e.g. tablets and smartphones). The app allows the user to create map-based rallies to implement NMR trainings or NMR assessment for a particular location. GeoGami automatically collects data on the participants’ movements in space (GPS based trajectory, speed, and cardinal view direction) and on their interaction with the tablet (tapping on the map, panning, and zooming), as well as timestamps (in particular the beginning and the end of each task) and GPS inaccuracy data and stores this in an encrypted data base. Additionally to the spatial information, the app records the performance on the thematic tasks.

The digital map used in GeoGami is based on standard OpenStreetMap material. Streets and paths are shown as colored lines, areas of grass, forests or water are colored in natural colors, houses are shown as polygons in their shape from top view, and objects like trees, statues, benches, bridges, or trash cans are represented with a symbol on the map.

We created tests for two locations for our NMR assessment using GeoGami. GeoGami displays the map of the test area, places the tasks, and automatically records participants’ movement and map interactions.

Task development

Based on our above defined understanding of Navigational Map Reading competence, we derive tasks to assess their NMR competence (c.f. steps in Figure 1). Our GeoGami NMR assessment consists of four NMR task types which we classify into two groups: Location Tasks (12 tasks in total) and Direction Tasks (8 tasks in total) (Figure 2). Location Tasks include self-location tasks (LOC) that ask the user to mark their own location by tapping on the map and navigation-to-a-flag tasks (LNV) that ask to navigate to a location indicated by a flag on the map. Direction Tasks include mark-view-direction tasks (DM) that ask the user to mark their own view direction by tapping on the map as well as adopt-view-direction tasks (DA) that ask to change their own view direction through turning by adopting a direction that is presented on the map.

Figure 2. Self-location (LOC), navigation-to-a-flag (LNV), mark-view-direction (DM) and adopt-view-direction (DA) tasks in the GeoGami NMR assessment (location A) as displayed on the tablet screen. The hand is representing the participant’s tapping on the tablet screen © GeoGami.

We chose tasks that represent the main activities of NMR situations following already existing tests and assessments. In line with the real-world tests of Lobben (2007) and Hemmer et al. (2013), our assessment consists of location and navigation tasks (LOC and LNV tasks). Additionally, we included direction-pointing tasks (Hemmer et al. 2013) in a way that is perceptible for mobile devices: instead of pointing with a finger, we ask the participants to turn the body and device in a certain view direction (DA tasks). The existing above-mentioned test instruments include not only configurational and directional aspects of NMR, but also representational transformations such as symbol or label reading. Symbol reading is often not the crux in navigating with a map (Lobben 2004). Label-reading must not be the crux to avoid that reading skills influence the NMR performance. Our assessment also represents the configurational and directional correspondences between map and real-world as mentioned by Ishikawa and Kastens (2005): LNV and DA tasks ask for map to real-world transformations and LOC tasks for real-world to the map transformations. We equipped the assessment with DM tasks in order to also include Direction Tasks with real-world-to-map transformations.

In total, there are six LOC, six LNV, four DM, and four DA tasks. These tasks are presented in pairs (twins) of two similar tasks having the same task difficulty. The tasks of each task twin are (i) of the same task type, (ii) located at comparable locations (i.e. the locations are comparable with respect to the surrounding objects and the path structures) and (iii) oriented in the same direction (i.e. the main view direction during or toward the tasks is the same). Having two test halves of the same difficulty allows us to divide the assessment into two parts for conducting reliability estimations and for using the test halves as pre- and post-tests.

In order to differentiate between strong and weak performances, tests and assessments require different task difficulties (Moosbrugger & Kelava 2020). Therefore, the task twins in our assessment are created in order to differ by the angle between map alignment (which is fixed to north) and view direction during or toward the task (which is north, south, or west/east). The difficulty of the tasks correlates with the angle between map alignment and view direction. The most difficult tasks and thus the worst performances can be observed for 180° angles (Presson 1982). These findings might be explained with different strategies used by the children depending on the angle: for instance, reading 180° rotated maps often lead to a “left-right-reversal” strategy (Aretz & Wickens, 1992) (“objects on my right side are on the left side of the map”) or, angles larger than 90°, to mentally rotate the map. For angles smaller than 90°, people tend to mentally manipulate the own perspective (Kozhevnikov & Hegarty 2001).

In order to avoid participants changing the task difficulty by rotating the map, we did not allow participants to rotate the device. Lobben (2007) interprets manual map rotations as a criterion for a weaker NMR performance.

The tasks in our NMR assessment are placed in a sequence with Transition Tasks leading from one task to the next one. These tasks are not scored, but give all participants the same initial location and orientation regardless if they solved the previous task correctly or not. Moreover, they separate the task locations in order to reduce knowledge transfer from one task to the next task. The transition tasks preceding a LOC task is an arrow-navigation task leading the children to a new location without showing them the map. The transition task preceding a LNV is an arrow-navigation task but children see their initial location on the map which disappears when they start the LNV task. The transition tasks preceding DM and DA are arrow-navigation tasks, in the case of DM followed by a task turning the children to a certain view direction.

All tasks are given in an easy and short written form or read out loud by the app. The test area is marked with a yellow line on the map. It is allowed to zoom and pan the map. At the beginning of each task, the app zooms out to the whole test area. At the end of each task it zooms into the task area (to the location- or direction-marker given by the task or set by the children) and asks the users to verify their given answer for a second time. This procedure ensures the same zoom level for all participants when solving tasks and prevents accidental responses. Location- and direction-marker are disabled during the assessment and no automatic feedback on the accuracy of the task solution is given. Instead GeoGami plays a sound after solving a task and shows the message “answer saved”.

The GeoGami NMR assessment is created for park areas to minimize the influence of urban structures on spatial NMR: the test area needs a branched network of paths and different repetitive objects for orientation (e.g. benches, trash cans, and trees) without street names and house numbers. These characteristics prevent orientation just by reading street names and house numbers or by recognizing unique landmarks. Instead, they trigger NMR that asks for the combination of several spatial hints for orientation and navigation.

The assessment starts with a tutorial about the tablet usage, task types, and symbol meanings to ensure a common knowledge for all participants with regard to the representational correspondence of real world and map. The total duration of the assessment (without tutorial) is around 30 to 45 minutes. We did control for time, however none of the students reached the time limit of 60 minutes.

The following data is recorded by GeoGami for each task: (i) time duration until a task was solved, (ii) the performance while solving the task such as amount of zooming, panning and (iii) the error: for the navigation tasks the app recorded the distance error of the actual location to the intended location. For the direction tasks the app recorded the direction error as the angle difference of the actual to the intended viewing direction.

NMR competency score development

The existing NMR test instruments described above are based on direct observations. While current research reveals the great potential of GI technologies for automatically collecting and analyzing spatial trajectory data (Mazimpaka & Timpf 2016), in the studies on navigational map reading competences mentioned in the related work section, the collected data consist of notes taken by external observers. The potential of GI technologies and its data for assessments has not yet been exploited in the context of map reading. This paper aims to close this gap.

For the computation of a NMR competency score development (c.f. steps in Figure 1), we wrote an evaluation script that calculates an NMR competency score based on the participants performance data (Bistron & Schwering 2021b). From this data, we derive the children’s accuracy and the process in solving NMR tasks (Table 1). The accuracy in solving a task is defined by the deviation (in m or in degrees) from the correct solution. The process data gives additional information on the way a user solved a task. We assume that the number of pan and zoom interactions with the map (for LOC tasks), the length of the route walked (for LNV tasks), as well as the sum of turning movements (for DA tasks) grow with the uncertainty in solving these tasks. These assumptions follow the work of Hergan and Umek (2017) and Lobben (2007) who define similar data for evaluating the NMR performance of their participants, e.g. the number of going “one crossroad too far”, taking wrong turns, stops, spins around the own axis, hesitations, map-rotations, as well as the time in solving a task.

Table 1. Accuracy and process data on each task type in the GeoGami Navigational Map Reading assessment.

		Accuracy data	Process data
Location Tasks	Self-location	Distance (in m) between correct and user’s marked location.^1	Number of pan and zoom interactions with the map.
	Navigation-to-a-flag	Distance (in m) between correct and user’s location.^1	Route length (in m) walked during the task.
Direction Tasks	Mark-view-direction	Deviation (in °) between correct and user’s marked view direction.	No process data.
	Adopt-view-direction	Deviation (in °) between correct and user’s view direction.	Sum of turning movements (in °) during the task.

¹ The distances were corrected by the GPS inaccuracy (in m): We subtracted GPS inaccuracy values from the distances and replaced negative results by 0. For each task and participant, the GPS inaccuracy value was detected when the participant confirmed the task answer by tapping an OK-button (mean: 4.08 for location A and 4.14 for location B).

Using the accuracy data, we score one point for a correct solution. One additional point is obtained, if the process of giving the correct answer has the following characteristics: depending on the task type, the task has to be solved with small numbers of pan and zoom events, short route lengths (in comparison to the shortest correct solution of all participants) or a small sum of all turning-angles. The resulting score is the sum of all achieved points. Thus, the score scale ranges from 0 to 20 points for considering only the accuracy data and 0 to 40 for considering the accuracy and process data. When focusing on Location Tasks only, the scale is defined from 0 to 12 points for considering only the accuracy data and 0 to 24 for considering the accuracy and process data.

We define thresholds (Table 2) for assigning distances and route lengths, angles, as well as numbers of pan and zoom events to the different point categories (two, one, or zero points), i.e. for defining a “correct” and “incorrect” or “confident” and “unconfident” solution. These thresholds result from data collected in test-runs in which we simulated different correct and incorrect answers with the help of children and adults. They were verified in the assessment validation described below.

Table 2. Thresholds defining different point categories for defining correct/incorrect confident/inconfident solutions in the GeoGami NMR assessment.¹

		Thresholds for accuracy data	Thresholds for process data
Location Tasks	Self-location	$d_{\mathit{LOC}}\mathrm{ }=\mathrm{ }10\mathrm{ }m$	$n_{\mathit{pan}}=\mathrm{ }25$ $n_{\mathit{zoom}}\mathrm{ }=\mathrm{ }10$
	Navigation-to-a-flag	$d_{\mathit{LNV}}\mathrm{ }=\mathrm{ }5\mathrm{ }m$	${l_{\mathit{route}}=2 \times l}_{\mathit{route}\_\min }$
Direction Tasks	Mark-view-direction	$d_{DM}\mathrm{ }=\mathrm{ }25°$	No process data.
	Adopt-view-direction	$d_{DA}\mathrm{ }=\mathrm{ }25°$	${\alpha }_{\mathrm{turn}}={\displaystyle \sum }_o^i{t}_i=500°$

¹ $d$ (deviation in m or ° from the correct solution), $n$ (number of events), $l_{\mathit{route}}$ (length in m of the route toward the correct destination), $l_{\mathit{route}\_\min }$ (length in m of the shortest route toward the correct destination taken by a participant), $t_i$ (angle in ° of a turning movement $i$ while solving the task), ${\alpha }_{\mathit{turn}}$ (sum of all $i$ turning angles in ° while solving a task).

The score does not consider absolute deviations (in m or in degrees) from the correct solutions. There are two main reasons: (i) smaller deviations from a correct solution might be induced by bad GPS signals or inaccuracies in marking locations and directions on the map and (ii) wrong answers with larger deviations might not necessarily be worse than wrong answers with smaller ones (e.g.: a child could choose a wrong location in a LOC task in 200 m deviation from the correct solution which shows more similarities to the correct solution than any other solution with a deviation of less meters).

Assessment validation

For the assessment validation (c.f. design of the framework in Figure 1, right part), we conducted a pilot study to evaluate the GeoGami NMR assessment for two different locations.

Study population, setting, and materials

Fifty-one primary school students (29 boys and 22 girls) participated in the study. The mean age was 8.33 (sd: 1.11) years (range 6 to 10). We recruited the participants via letters handed out in local primary schools. The study was conducted as a summer event during the school vacations. We accepted primary school students who lived nearby and thus concluded that they are familiar with the environment of the study area. No further assessment of familiarity or time spent outside (e.g. walking, hiking, riding the bike and thus navigating through the environment) was done. We had no participant dropouts, but technical issues that made the data of one further participant useless. The children were rewarded with a science-toy and a certificate of participation.

The study areas were located in two different parks in North Rhine-Westphalia, Germany, the Citizens’ Park in Senden (location A) and the Wildlife-Park in Coesfeld (location B), each with short distances to walk. 29 participants took part in Senden and 22 in Dülmen.

Each participant had a tablet (with a 10.2” display) in their hand with GeoGami installed. A camera mounted on their head filmed the surroundings from the child’s perspective, the children’s speech and behavior. A screen-video recorded the GeoGami app (incl. digital map, written task instructions, and markers on the map).

Data collection

After the tutorial, the children performed the assessment tasks. The experimenter accompanied them during the assessment. The children were asked to think aloud explaining their task solutions. They were allowed to take breaks and to ask questions about task instructions and symbols, but didn’t received cues about solution strategies or explicit configurational or directional correspondences. All children participated alone without the influence of other participants. The study was done in compliance with the current Covid-19 regulations. The automatically collected data was scored via evaluation script (Bistron & Schwering 2021b).

After the experiment, three experts judged each child’s NMR competencies on a scale from zero to ten based on the audio (think-aloud statements) and video material. The experts were student teachers recruited via job posting that asked for strong navigational skills and experiences in working with young school children. The student teachers were payed for this task and signed a declaration of confidentiality. They received instruction on difficulties children can have in navigating with a map. They had neither access to the GeoGami data nor to the participants’ scores. Instead, they had to recognize correct and incorrect answers just by watching the videos remembering the correct locations from conducting the assessment themselves and by consulting photos of the correct locations and task solutions. They used the definition of NMR competencies described above and were asked to analyze every child in a wholistic way (i.e. without just calculating total scores on the basis of the task solutions). We informed the experts that they were allowed to ignore individual tasks if they thought they were not relevant for evaluating the NMR competencies of the children. The experts made their individual judgments and agreed together on one score for each child by discussing the individual cases.

Data analysis

The assessment was evaluated following the work of Moosbrugger and Kelava (2020). The authors emphasize that a test instrument should be evaluated with respect to its intention. We understand our instrument as assessment for navigational map reading that shall reflect a teacher’s evaluation of students’ NMR competencies. It is supposed to be applicable for educational diagnostics as a basis for the development of NMR trainings. Therefore, it has to distinguish between weak and strong performances and should be designed to be used as a pre- and post-test.

Our assessment tasks shall be able to distinguish between weak and strong navigational map readers and, hence, we investigate the item-total correlations, the item difficulties as well as the score distribution. As a reliability measure we decided to estimate the split-half reliability in order to evaluate the assessment in forms of two parallel test halves that can be used as pre- and post-tests. For estimating the reliability, some preconditions need to be checked in forms of a factor analysis. If the preconditions check is not possible (due to the small sample size, as in the case of this study), Moosbrugger and Kelava (2020) recommend estimating the reliability via Cronbach’s Alpha which gives even reasonably results when the preconditions are not met completely.

Knowing that the assessment validation is affected by pre-defined thresholds for the point categories, we did a sensitivity analysis on different possible threshold combinations and its effect on the resulting assessment validation.

The data and assessment validation are documented in an evaluation script in R (Bistron & Schwering 2021b) to ensure reproducibility and for applying the NMR assessment in other locations.

Analogue to the test of Hemmer et al. (2013), the tasks (items) of our assessment are expected to be unidimensional, but the participant number is not sufficient for conducting a factor analysis (Bühner 2011). Therefore, the assessment validation was done for (i) all tasks (assuming one factor behind all tasks) and for (ii) the Location Tasks and Direction Tasks separately (assuming one factor behind each group of tasks) as well as for (a) the accuracy and process data (assuming one factor behind the data for each task) and (b) the accuracy data only.

Results of the assessment validation

Item analysis and score distribution

We determined the item difficulties $P_i$ of each task (item) $i$. As intended, the individual tasks show different difficulty indices spanning from $P_{\mathit{LNV}4}=33$ (hardest task) to $P_{DA4}=93$ (easiest task) for location A and $P_{\mathit{LNV}3}=23$ (hardest task) to $P_{DA3}=91$ (easiest task) for location B (Table 3). Based on the item difficulty indices and the task types, we built task twins and divided the assessment into two halves.

Table 3. Item difficulties $\mathrm{ }P$ for each task ranging from 23 for the hardest task to 93 for the easiest task. Based on the item difficulty indices and the task types, the GeoGami NMR assessment was divided into test halves with similar tasks (task twins) in each test half.¹

Location	A				B
	Only accuracy data		Accuracy data and process data		Only accuracy data		Accuracy data and process data
Data	Half 1	Half 2	Half 1	Half 2	Half 1	Half 2	Half 1	Half 2
Location Tasks	LOC5 (41)	LOC2 (62)	LOC5 (40)	LOC2 (59)	LOC6 (36)	LOC5 (41)	LOC6 (36)	LOC3 (39)
	LOC6 (69)	LOC3 (66)	LOC6 (67)	LOC3 (59)	LOC2 (55)	LOC3 (45)	LOC4 (50)	LOC5 (39)
	LOC1 (76)	LOC4 (79)	LOC1 (71)	LOC4 (76)	LOC1 (59)	LOC4 (59)	LOC2 (55)	LOC1 (57)
	LNV3 (62)	LNV4 (48)	LNV3 (50)	LNV4 (33)	LNV6 (50)	LNV3 (27)	LNV6 (30)	LNV3 (23)
	LNV5 (62)	LNV6 (69)	LNV5 (52)	LNV6 (67)	LNV4 (55)	LNV1 (59)	LNV1 (48)	LNV4 (48)
	LNV2 (90)	LNV1 (76)	LNV2 (76)	LNV1 (74)	LNV5 (68)	LNV2 (64)	LNV5 (64)	LNV2 (59)
Direction Tasks	DM2 (55)	DM1 (86)	DM2 (55)	DM1 (86)	DM4 (45)	DM2 (55)	DM4 (45)	DM2 (55)
	DM3 (93)	DM4 (86)	DM3 (93)	DM4 (86)	DM3 (82)	DM1 (73)	DM3 (82)	DM1 (73)
	DA1 (72)	DA3 (76)	DA1 (47)	DA3 (72)	DA2 (68)	DA1 (73)	DA1 (61)	DA2 (61)
	DA4 (93)	DA2 (86)	DA4 (84)	DA2 (78)	DA3 (91)	DA4 (86)	DA3 (91)	DA4 (84)

¹ Item difficulties are presented in brackets (in %). The abbreviations stand for self-location tasks (LOC), navigation-to-a-flag tasks (LNV), mark-view-direction tasks (DM), adopt-view-direction tasks (DA).

Within the group of Location Tasks and the group of Direction Tasks, we determined item-total correlations with a part-whole-correction for each task. The majority of the items show a highly positive item-total correlation, but we also observe items with a correlation close to zero or with negative coefficients (Table 4). Therefore, we proceeded the further assessment validation in two different ways: (1.) we evaluated the assessment for all items and (2.) we also did the calculations by excluding tasks with an item-total correlation lower than 0.3 and their task twin partners.

Table 4. Item-total correlations within the group of location tasks and the group of direction tasks of the GeoGami NMR assessment.¹

Location	A				B
	Only accuracy data		Accuracy data and process data		Only accuracy data		Accuracy data and process data
Data	Half 1	Half 2	Half 1	Half 2	Half 1	Half 2	Half 1	Half 2
Location Tasks	LOC5 (0.30)	LOC2 (0.43)	LOC5 (0.29)	LOC2 (0.43)	LOC6 (0.63)	LOC5 (0.82)	LOC6 (0.66)	LOC3 (0.72)
	LOC6 (0.42)	LOC3 (0.39)	LOC6 (0.47)	LOC3 (0.39)	LOC2 (0.47)	LOC3 (0.51)	LOC4 (0.72)	LOC5 (0.88)
	LOC1 (0.61)	LOC4 (0.81)	LOC1 (0.67)	LOC4 (0.71)	LOC1 (0.55)	LOC4 (0.76)	LOC2 (0.39)	LOC1 (0.49)
	LNV3 (0.48)	LNV4 (0.53)	LNV3 (0.41)	LNV4 (0.62)	LNV6 (0.54)	LNV3 (0.74)	LNV6 (0.45)	LNV3 (0.71)
	LNV5 (0.65)	LNV6 (0.66)	LNV5 (0.66)	LNV6 (0.66)	LNV4 (−0.06)	LNV1 (−0.12)	LNV1 (−0.25)	LNV4 (−0.01)
	LNV2 (0.68)	LNV1 (0.14)	LNV2 (0.75)	LNV1 (0.04)	LNV5 (0.16)	LNV2 (0.44)	LNV5 (0.23)	LNV2 (0.47)
Direction Tasks	DM2 (0.30)	DM1 (0.26)	DM2 (0.33)	DM1 (0.31)	DM4 (0.57)	DM2 (0.35)	DM4 (0.52)	DM2 (0.46)
	DM3 (0.46)	DM4 (0.78)	DM3 (0.40)	DM4 (0.76)	DM3 (0.15)	DM1 (−0.17)	DM3 (0.18)	DM1 (−0.14)
	DA1 (0.63)	DA3 (0.83)	DA1 (0.58)	DA3 (0.82)	DA2 (0.15)	DA1 (0.60)	DA1 (0.67)	DA2 (0.15)
	DA4 (0.53)	DA2 (0.15)	DA4 (0.59)	DA2 (0.37)	DA3 (0.64)	DA4 (0.61)	DA3 (0.57)	DA4 (0.53)
All Tasks	LOC5 (0.32)	LOC2 (0.44)	LOC5 (0.32)	LOC2 (0.43)	LOC6 (0.63)	LOC5 (0.81)	LOC6 (0.69)	LOC3 (0.62)
	LOC6 (0.41)	LOC3 (0.45)	LOC6 (0.45)	LOC3 (0.46)	LOC2 (0.38)	LOC3 (0.44)	LOC4 (0.74)	LOC5 (0.87)
	LOC1 (0.67)	LOC4 (0.77)	LOC1 (0.67)	LOC4 (0.68)	LOC1 (0.55)	LOC4 (0.78)	LOC2 (0.35)	LOC1 (0.53)
	LNV3 (0.54)	LNV4 (0.64)	LNV3 (0.46)	LNV4 (0.66)	LNV6 (0.60)	LNV3 (0.66)	LNV6 (0.48)	LNV3 (0.63)
	LNV5 (0.55)	LNV6 (0.61)	LNV5 (0.61)	LNV6 (0.61)	LNV4 (0.05)	LNV1 (−0.09)	LNV1 (−0.25)	LNV4 (0.08)
	LNV2 (0.58)	LNV1 (0.05)	LNV2 (0.65)	LNV1 (0.02)	LNV5 (0.23)	LNV2 (0.52)	LNV5 (0.25)	LNV2 (0.55)
	DM2 (0.46)	DM1 (0.38)	DM2 (0.50)	DM1 (0.37)	DM4 (0.12)	DM2 (0.48)	DM4 (0.15)	DM2 (0.47)
	DM3 (0.45)	DM4 (0.45)	DM3 (0.41)	DM4 (0.41)	DM3 (0.12)	DM1 (−0.07)	DM3 (0.15)	DM1 (−0.08)
	DA1 (0.46)	DA3 (0.70)	DA1 (0.49)	DA3 (0.68)	DA2 (0.42)	DA1 (0.54)	DA1 (0.64)	DA2 (0.42)
	DA4 (0.46)	DA2 (0.28)	DA4 (0.60)	DA2 (0.39)	DA3 (0.37)	DA4 (0.51)	DA3 (0.32)	DA4 (0.46)

¹ Item-total correlations lower than 0.3 are marked in grey. The abbreviations stand for self-location tasks (LOC), navigation-to-a-flag tasks (LNV), mark-view-direction tasks (DM), adopt-view-direction tasks (DA).

The NMR competency scores span most of the intended score scale from very low to high scores (Table 5). The interquartile range for the score including all tasks and all data (accuracy and process data) is ${\delta }_A=10$ (25% of total scale) with a median of 28 and ${\delta }_B$=13.75 (34% of total scale) with a median of 20.5. For the Location Tasks it is ${\delta }_A=8$ (33% of total scale) with a median of 16 and ${\delta }_B$=8.5 (35% of total scale) with a median of 10, describing a wider distribution of the score for Location Tasks than for all tasks, especially for location A. The skewness of the score distribution is negative for location A and positive for location B. For location A the median is always slightly above the mean. When considering accuracy and process data for location B, the opposite is the case.

Table 5. NMR competency score distribution in our GeoGami NMR assessment study.

	Location	A		B
	Data	Accuracy data	Accuracy data and process data	Accuracy data	Accuracy data and process data
Location Tasks	Range	2–12	3–24	1–11	2–20
	Mode	9	17	7	5
	Mean (Stand. deviation)	8.00 (2.99)	14.45 (5.69)	6.18 (2.91)	10.91 (5.44)
	Median	9	16	6.5	10
	Quantiles	Q1 = 6, Q3 = 10	Q1 = 10, Q3 = 18	Q1 = 4, Q3 = 8.75	Q1 = 7.25, Q3 = 15.75
	Skewness	−0.75	−0.55	−0.01	0.27
excl. tasks with low item-total correlation	Range		0–16		0–15
	Mode		11		4
	Mean (Stand. deviation)		9.48 (4.49)		6.55 (4.86)
	Median		11		5
	Quantiles		Q1 = 6, Q3 = 12		Q1 = 2.5, Q3 = 10.5
	Skewness		−0.58		0.37
All tasks	Range	4–20	6–40	6–19	11–36
	Mode	15	20	13	33
	Mean (Stand. deviation)	14.48 (4.28)	26.48 (8.44)	11.91 (4.09)	21.95 (7.85)
	Median	16	28	12.5	20.5
	Quantiles	Q1 = 12, Q3 = 17	Q1 = 22, Q3 = 32	Q1 = 8.25, Q3 = 15.5	Q1 = 15.5, Q3 = 29.25
	Skewness	−0.95	−0.77	0.12	0.35
excl. tasks with low item-total correlation	Range		3–32		2–21
	Mode		22		21
	Mean (Stand. deviation)		21.52 (7.31)		12.05 (6.33)
	Median		23		11
	Quantiles		Q1 = 18, Q3 = 27		Q1 = 7.25, Q3 = 18.5
	Skewness		−0.92		0.20

Estimating reliability and determining validity

We estimated Cronbach’s Alpha for the Location and Direction Tasks separately (assuming one factor behind each group of tasks) as well as for all tasks (assuming one factor behind all tasks) (Table 3). For Location Tasks, we observed values of 0.78 and higher for location A and of 0.71 and higher for location B. For Direction Tasks, we observed values of 0.66 and higher for location A and low values of 0.42 and lower for location B. For all tasks, we found values of 0.84 and higher for location A and 0.77 and higher for location B.

Additionally, we decided to estimate the split-half reliability (Table 2). We correlated the scorings of the halves via Spearman’s $\rho$ (with $p < 0.001$) and estimated the split-half reliability with the Spearman-Brown formula. For both locations, the split-half reliability values are higher than 0.82 and, in most cases, higher than the corresponding Cronbach’s Alphas (Table 6). The split-half reliability is higher when including all data (accuracy and process data) instead of calculating with accuracy data only.

Table 6. The split-half reliability¹ for both locations of the GeoGami NMR assessment, exceeding the Cronbach’s Alpha in most of the cases.

Location	A				B
Data	Accuracy data		Accuracy data and process data		Accuracy data		Accuracy data and process data
	Cronb. Alpha	Split-half	Cronb. Alpha	Split-half	Cronb. Alpha	Split-half	Cronb. Alpha	Split-half
Location Tasks	0.78	0.83	0.78	0.87	0.71	0.82	0.71	0.86
excl. tasks with low item-total correlation	0.80		0.80	0.82	0.72		0.72	0.92
Direction Tasks	0.66		0.70		0.35		0.37
excl. tasks with low item-total correlation	0.70		0.70		0.35		0.42
All tasks	0.84	0.89	0.85	0.91	0.77	0.83	0.78	0.87
excl. tasks with low item-total correlation	0.85		0.85	0.84	0.80		0.80	0.93

¹ $p\mathrm{ }<\mathrm{ }0.001$.

We correlated the score of our assessment with the scorings of the expert judgements via Spearman’s $\rho$. The determined correlation coefficients range from 0.82 to 0.92 ($p < 0.001$). They are at highest when including process data and at lowest when dropping tasks with an item-total correlation lower than 0.3 (Table 7).

Table 7. Correlations coefficients for NMR competency score of the GeoGami NMR assessment and scoring of the expert judgements.¹ The correlations are higher when process data is included.

Location	A		B
Data	Accuracy data	Accuracy data and process data	Accuracy data	Accuracy data and process data
Location Tasks	0.86	0.90	0.92	0.92
excl. tasks with low item-total correlation		0.85		0.84
All tasks	0.90	0.91	0.88	0.89
excl. tasks with low item-total correlation		0.89		0.82

¹ $p\mathrm{ }<\mathrm{ }0.001$.

Moreover, we did a sensitivity analysis on different possible threshold combinations and its effect on the resulting assessment validation. In Table 8 we present five examples showing increases and decreases in reliability and validity, but with consistently high values for reliability and validity. However, changes in the thresholds can also deteriorate the assessment: details on the complete sensitivity analysis can be found in the corresponding evaluation script (Bistron & Schwering 2021b) that allows the readers to manipulate and analyze the thresholds on their own.

Table 8. Sensitivity analysis: thresholds and their effect on the GeoGami NMR assessment validation.¹ The five examples in the table show small increases and decreases in reliability and validity, but overall a consistently high value for reliability and validity.

Example	Thresholds				Reliability value for all tasks (incl. accuracy and process data)		Validity coefficient for all tasks (incl. accuracy and process data)
	Accuracy data		Process data		Loc. A	Loc. B	Loc. A	Loc. B
Initial	$d_{\mathit{LOC}}\mathrm{ }=\mathrm{ }10\mathrm{ }m$ $d_{\mathit{LNV}}\mathrm{ }=\mathrm{ }5\mathrm{ }m$	$d_{DM}\mathrm{ }=\mathrm{ }25°$ $d_{DA}\mathrm{ }=\mathrm{ }25°$	$n_{\mathit{pan}}=\mathrm{ }25$ $n_{\mathit{zoom}}\mathrm{ }=\mathrm{ }10$	${l_{\mathit{route}}=2\times l}_{\mathit{route}\_\min }$ ${\alpha }_{\mathit{turn}}=500°$	0.91	0.87	0.91	0.89
1	${\mathit{d}}_{\mathit{LOC}}\mathrm{ }=\mathrm{ }15$ $d_{\mathit{LNV}}\mathrm{ }=\mathrm{ }5\mathrm{ }m$	$d_{DM}\mathrm{ }=\mathrm{ }25°$ $d_{DA}\mathrm{ }=\mathrm{ }25°$	$n_{\mathit{pan}}=\mathrm{ }25$ $n_{\mathit{zoom}}\mathrm{ }=\mathrm{ }10$	${l_{\mathit{route}}=2\times l}_{\mathit{route}\_\min }$ ${\alpha }_{\mathit{turn}}=500°$	0.90 ⇓	0.82 ⇓	0.91	0.91 ⇑
2	$d_{\mathit{LOC}}\mathrm{ }=\mathrm{ }10\mathrm{ }m$ ${\mathit{d}}_{\mathit{LNV}}\mathrm{ }=\mathrm{ }10\mathrm{ }\mathit{m}$	$d_{DM}\mathrm{ }=\mathrm{ }25°$ $d_{DA}\mathrm{ }=\mathrm{ }25°$	$n_{\mathit{pan}}=\mathrm{ }25$ $n_{\mathit{zoom}}\mathrm{ }=\mathrm{ }10$	${l_{\mathit{route}}=2\times l}_{\mathit{route}\_\min }$ ${\alpha }_{\mathit{turn}}=500°$	0.91	0.89 ⇑	0.93 ⇑	0.89
3	$d_{\mathit{LOC}}\mathrm{ }=\mathrm{ }10\mathrm{ }m$ $d_{\mathit{LNV}}\mathrm{ }=\mathrm{ }5\mathrm{ }m$	${\mathit{d}}_{\mathit{DM}}\mathrm{ }=\mathrm{ }30°$ ${\mathit{d}}_{\mathit{DA}}\mathrm{ }=\mathrm{ }30°$	$n_{\mathit{pan}}=\mathrm{ }25$ $n_{\mathit{zoom}}\mathrm{ }=\mathrm{ }10$	${l_{\mathit{route}}=2\times l}_{\mathit{route}\_\min }$ ${\alpha }_{\mathit{turn}}=500°$	0.90 ⇓	0.79 ⇓	0.92 ⇑	0.90 ⇑
4	$d_{\mathit{LOC}}\mathrm{ }=\mathrm{ }10\mathrm{ }m$ $d_{\mathit{LNV}}\mathrm{ }=\mathrm{ }5\mathrm{ }m$	$d_{DM}\mathrm{ }=\mathrm{ }25°$ $d_{DA}\mathrm{ }=\mathrm{ }25°$	${\mathit{n}}_{\mathit{pan}}=\mathrm{ }30$ ${\mathit{n}}_{\mathit{zoom}}\mathrm{ }=\mathrm{ }20$	${l_{\mathit{route}}=2\times l}_{\mathit{route}\_\min }$ ${\alpha }_{\mathit{turn}}=500°$	0.93 ⇑	0.88 ⇑	0.91	0.88 ⇓
5	$d_{\mathit{LOC}}\mathrm{ }=\mathrm{ }10\mathrm{ }m$ $d_{\mathit{LNV}}\mathrm{ }=\mathrm{ }5\mathrm{ }m$	$d_{DM}\mathrm{ }=\mathrm{ }25°$ $d_{DA}\mathrm{ }=\mathrm{ }25°$	$n_{\mathit{pan}}=\mathrm{ }25$ $n_{\mathit{zoom}}\mathrm{ }=\mathrm{ }10$	${{\mathit{l}}_{\mathit{route}}=3\times \mathit{l}}_{\mathit{route}\_\mathit{min}}$ ${\mathit{\alpha }}_{\mathit{turn}}=600°$	0.89 ⇓	0.88 ⇑	0.91	0.88 ⇓

¹ Changes in the thresholds (tab. II) are marked in bold, increases ⇑ and decreases ⇓ in the resulting reliability values and validity coefficients are represented with an arrow. The abbreviations stand for self-location tasks (LOC), navigation-to-a-flag tasks (LNV), mark-view-direction tasks (DM), adopt-view-direction tasks (DA).

Discussion and limitations

Reliability and validity of the assessment

The assessment validation for both locations resulted in satisfactory reliability values as well as adequate validity coefficients. The sensitivity analysis revealed that changes in the thresholds can go along with similar results. However, unidimensionality is precondition for calculating a score or estimating the reliability (Moosbrugger & Kelava 2020). We tried to counter this by calculating the reliability for different groups of tasks. However, this needs to be investigated further with a larger data sample. Moreover, since the reported split-half reliability values are higher than the Cronbach’s Alphas, there is a need for conducting a factor analysis to find explanations.

Measurement range of the assessment

An intermediate assessment difficulty as well as a wide range of item difficulties were found for the GeoGami NMR assessment for both locations A and B. This enables the assessment for both locations to distinguish between strong and weak navigational map readers. Moreover, the assessment includes enough easy tasks to motivate students, but still sufficiently intermediate and harder ones to distinguish performances among stronger navigational map readers. However, additional tasks of intermediate difficulty would improve it for distinguishing between performance differences in even higher resolution.

Possible differences in the results for the locations and individual tasks can be explained by differences in the characteristics of the task locations: whereas the park area of location A showed more path intersections and more objects on a smaller scale, in the park area of location B distances were larger and landmarks were less.

Item quality of the assessment

Several tasks of the assessment show a satisfactory item-total correlation. However, a few show an item-total correlation less than 0.3, especially some Direction Tasks and some LNV-tasks, predominantly for the Location B. These results are also represented in the Cronbach’s Alphas: we observed acceptable Cronbach’s Alphas for both locations, except from the ones for the Direction Tasks of location B that are unsatisfactory low. The low item-total correlations may represent a low internal consistency of the items: low values then would represent only slight or no differences in the task performances of strong and weak navigational map readers and negative values would represent better performances for weak than for strong navigational map readers.

The low item-total correlations for task LNV1 illustrates this interpretation: some children passed the flag destination and came back to it, others passed it and chose a wrong destination for the flag. In both groups, several children recognized their mistake afterwards and explained it to the experimenter: since it was the initial task, they were not familiar with the scale and therefore surprised that the flag was located close to the starting point. This initial-scale-problem could be observed for weak and strong navigational map readers equally which might explain the low item-total correlations. However, low or negative item-total correlations can not only be induced by weak internal consistencies but also by the affiliation of individual tasks to different factors (Moosbrugger & Kelava 2020) which underlines the need for a factor analysis.

Moreover, we saw that Direction Tasks were less suitable to distinguish between strong and weak navigational map readers and show lower reliability values than Location Tasks, especially for the location B. Two possible reasons can be figured out. First, the Direction Tasks were presented in a package of four tasks (one task for each cardinal direction) which made it easy to derive a task solution from former tasks. Task performances of previous tasks may have influenced performances in the following tasks. Second, Direction tasks are possibly also solvable without transforming directions in mind (e.g. just by turning to objects the arrow on the map is pointing at). Performances in such cases does not necessarily say anything about the children’s spatial NMR competencies and, thus, does not necessarily correlate with the performances in other NMR tasks of the assessment.

Usefulnes of process data for the assessment

Including process data for determining the NMR competency score led to slight changes in the score itself, the task difficulties, the item-total correlations, Cronbach’s Alpha, split-half reliability, as well as the validity coefficients. What we could not find is a clear improvement or deterioration of our assessment when including process data. We assume that we need to examine for each task individually whether the inclusion improves the assessment or not. Two examples illustrate this assumption:

1. LNV4_A was solved correctly by nearly half of the participants. During the experiment we witnessed that a lot of children solving this task correctly by trial-and-error. Since the destination of the flag was unique regarding the shape of a nearby object, the children recognized the destination when seeing it in their environment regardless of whether they expected the destination at that location or not – in other words: weak navigational map readers, possibly with a trial-and-error solution, solved this task correctly. This explains the low item-total correlation for this specific task. The only difference between strong and weak navigational map readers then is the length of the route walked (process data): some children took longer routes than others. A longer route possibly goes along with a trial-and-error solution which would explain the increase in the task difficulty and item-total correlation when including process data into the scoring.

2. For LOC4_A the opposite is the case: including process data for this specific task led to a decrease in the task difficulty and in the item-total correlation. Possibly, some weak map readers chose the solution by recognizing the unique spot directly on the map without thinking spatially and without checking their solution against other locations for a second time via panning and zooming. Thus, low numbers for panning and zooming would not represent a confident or elaborated solution.

We assume that the process data mainly describes the uncertainty in solving tasks. However, in order to learn more about this aspect, a factor analysis is needed to show that process data defines no own factor itself and can be used additionally to accuracy data in order to evaluate task performances.

Recommended settings of the assessment for other locations

As mentioned in the introduction, our research aims to develop an NMR competence assessment that is applicable for different outdoor locations. While the GeoGami tool to implement the assessment tasks as well as the script scoring participants performance are location independent there are certain requirements for the study site when adapted by researchers for their own locations. To ensure that other skills such as reading or symbol understanding do not affect the performance, the study site shall not contain place name signs (e.g. street names or signs with landmark names) that can be found on the map and used for orientation instead of the spatial layout. The same holds for unique landmarks that can be identified via symbols on the map. The study site should consist of a certain spatial complexity: A branched network of paths is required to implement orientation tasks toward different cardinal directions. Several different, repetitive objects that are included in the map representation are important to implement orientation tasks based on landmarks.

Conclusion

We developed the GeoGami NMR assessment, an educational tool for assessing NMR competencies of primary school children aged six to ten in a real-world scenario. The assessment makes use of the digital geogame GeoGami (Bistron et al. 2021). It focusses on configurational and directional information and therefore helps teachers to understand the spatial navigational performances of children by excluding confounding factors like text and symbol reading skills.

We evaluated the assessment for two exemplary locations by analyzing the items, estimating the reliability, and validating the NMR competency score against an expert judgment. For these two locations and with the interpretation of the score in the sense of a pedagogical evaluation of NMR performances, the assessment is shown to be reliable and valid, i.e. it is able to quantify children’s NMR competencies with a score that reliably represents teachers’ evaluation of these competencies.

All suggested tasks (Location and Direction Tasks) as well as all kind of data (accuracy and process data) are helpful to learn more about the participants’ NMR competencies. The assessment validation showed that it is sufficient to focus on process data only when it improves the item-total correlation of the individual task. Especially tasks that can be solved by searching for a unique location on the map or in the environment (trial-and-error solution) do not assess NMR competencies as accurate as tasks that can only be solved via spatial thinking. Thus, for evaluating the children’s competencies, not only the accuracy in solving a task but also the process in doing can be important to understand the children’s thinking behind a solution—a finding that can be applied to many learning situations.

Our results are based on assumptions which need to be factor analytically proven with a larger sample size. A factor analysis would also help to decide on dropping individual tasks and on the method to estimate reliability.

Implications for classroom teaching

With our assessment, we are able to distinguish between strong and weak navigational map readers. Since it has no norm-reference, it is not intended to be used for evaluating the performance of an individual against a norm group. Instead, the assessment can reveal the learning progresses of individual children and the differences in children’s learning levels within a group (for a certain location). Teachers can use this knowledge to make instructional and methodical decisions for NMR trainings for a whole class or individual children. Moreover, they can conduct the assessment for evaluating NMR teaching and learning processes by comparing the children’s competencies before and after an intervention (Bistron et al. 2022).

Especially in class situations in which multiple children are supervised by only one teacher, the GeoGami NMR assessment can give a reliable overview on individual performances without the need to accompany each child individually in its navigational process. The finding that a digital tool enables the measurement of NMR competencies indicates that it is worth to also investigate related learning analytics solutions that automatically react on the children’s task solutions and behavior by giving suitable and individual feedback, instructions, and support.

Implications for navigational map reading research and future work

We offer an assessment design that is applicable at different locations: it is implemented in a location-based geogame which allows the reader to create and conduct it for any location. We created an evaluation script (Bistron & Schwering 2021b) that automatically measures and scores the participants’ performances and evaluates the assessment and its items for a new location. Moreover, the evaluation script helps the reader to determine two comparable assessment halves which can be a basis for developing pre- and posttests for learning progress investigations in NMR trainings. Although the assessment can be applicated at different locations, a validity check for each location is recommended to ensure an adequate application of the assessment tasks within the new location: disregard for the above task and task location requirements can lead to undesirable effects on test scores. For example, putting navigation flags next to unique landmarks can lead to good navigational performances for strong and weak navigational map readers since the task can be solved without thinking spatially.

Although the assessment is transferable to different locations, we suggest future work to consider our findings for the development of a location-independent scientific test instrument for measuring NMR competencies. A virtual reality solution might be a suitable approach in order to avoid location-dependent scores and the need for reevaluating the test for new locations. Virtual solutions would also facilitate the creation of “ideal” worlds that do not include unique locations and therefore prevent trial-and-error solutions.

Additional information

Funding

This article has received funding from the programme “Profilbildung 2020,” an initiative of the Ministry of Culture and Science of the State of North Rhine-Westphalia. The sole responsibility for the content of this publication lies with the authors.