Assuring the quality of VGI on land use and land cover: experiences and learnings from the LandSense project

References

• Ahlqvist, O. 2008. “Extending Post-Classification Change Detection Using Semantic Similarity Metrics to Overcome Class Heterogeneity: A Study of 1992 and 2001 US National Land Cover Database Changes.” Remote Sensing of Environment 112 (3): 1226–1241.
   Web of Science ®Google Scholar
• Antoniou, V., J. Morley, and M. Haklay. 2010. “Web 2.0 Geotagged Photos: Assessing the Spatial Dimension of the Phenomenon.” Geomatica 64 (1): 99–110.
   Google Scholar
• Bastin, L., S. Schade, and C. Schill. 2017. ”Data and Metadata Management for Better VGI Reusability.” In Mapping and the Citizen Sensor, edited by G. Foody, L. See, S. Fritz, P. Mooney, C. C. Fonte, A. M. Olteanu-Raimond, and V. Antoniou, 249–272. London: Ubiquity Press.
   Google Scholar
• Brabyn, L., and D. M. Mark. 2011. “Using Viewsheds, GIS, and a Landscape Classification to Tag Landscape Photographs.” Applied Geography 31 (3): 1115–1122. doi:10.1016/j.apgeog.2011.03.003.
   Web of Science ®Google Scholar
• Burrough, P. A., R. McDonnell, R. A. McDonnell, and C. D. Lloyd. 2015. Principles of Geographical Information Systems. Oxford: Oxford University Press.
   Google Scholar
• Capineri, C., M. Haklay, H. Huang, V. Antoniou, J. Kettunen, F. Ostermann, and R. Purves. 2016. European Handbook of Crowdsourced Geographic Information. London: Ubiquity Press.
   Google Scholar
• Card, D. H. 1982. “Using Known Map Category Marginal Frequencies to Improve Estimates of Thematic Map Accuracy.” Photogrammetric Engineering and Remote Sensing 48: 431–439.
   Web of Science ®Google Scholar
• Chesnokova, O., and R. S. Purves. 2018. “From Image Descriptions to Perceived Sounds and Sources in Landscape: Analyzing Aural Experience Through Text.” Applied Geography 93: 103–111. doi:10.1016/j.apgeog.2018.02.014.
   Web of Science ®Google Scholar
• Comber, A., P. Fisher, and R. Wadsworth. 2005. “What is Land Cover?” Environment and Planning B, Planning & Design 32 (2): 199–209. doi:10.1068/b31135.
   Web of Science ®Google Scholar
• Comber, A. J., R. A. Wadsworth, and P. F. Fisher. 2008. “Using Semantics to Clarify the Conceptual Confusion Between Land Cover and Land Use: The Example of ‘Forest’.” Journal of Land Use Science 3 (2–3): 185–198.
   Google Scholar
• Comber, A., L. See, and S. Fritz. 2014. “The Impact of Contributor Confidence, Expertise and Distance on the Crowdsourced Land Cover Data Quality.” GI Forum 2014-Geospatial Innovation for Society 309–313. doi:10.1553/giscience2014s309.
   Google Scholar
• Comber, A., P. Mooney, R. S. Purves, D. Rocchini, and A. Walz. 2016. “Crowdsourcing: It Matters Who the Crowd Are. The Impacts of Between Group Variations in Recording Land Cover.” PLoS One 11 (7): e0158329.
   PubMed Web of Science ®Google Scholar
• Congalton, R. G., and K. Green. 2009. Assessing the Accuracy of Remotely Sensed Data: Principles and Practices. 2nd ed. Boca Raton: CRC Press.
   Google Scholar
• Congalton, R. G., R. G. Oderwald, and R. A. Mead. 2009. “Assessing Landsat Classification Accuracy Using Discrete Multivariate Analysis Statistical Techniques.” Photogrammetric Engineering and Remote Sensing 49 (12): 1671–1678.
   Google Scholar
• Costa, H., G. M. Foody, and D. S. Boyd. 2017. “Using Mixed Objects in the Training of Object-Based Image Classifications.” Remote Sensing of Environment 190: 188–197. doi:10.1016/j.rse.2016.12.017.
   Web of Science ®Google Scholar
• Costa, H., G. M. Foody, and D. S. Boyd. 2018. “Supervised Methods of Image Segmentation Accuracy Assessment in Land Cover Mapping.” Remote Sensing of Environment 205: 338–351. doi:10.1016/j.rse.2017.11.024.
   Web of Science ®Google Scholar
• Danylo, O., I. Moorthy, T. Sturn, L. See, J. C. Laso Bayas, D. Domian, D. Fraisl, C. Giovando, et al. 2018. ”The Picture Pile Tool for Rapid Image Assessment: A Demonstration Using Hurricane Matthew.” ISPRS Annals of Photogrammetry, Remote Sensing and Spatial Information Sciences IV-4: 27–32: 27–32. doi:10.5194/isprs-annals-IV-4-27-2018.
   Google Scholar
• De Marchi, B., A. Ficorilli, and A. Biggeri. 2022. “Research is in the Air in Valle Del Serchio.” Futures 137: 102906. doi:10.1016/j.futures.2022.102906.
   Web of Science ®Google Scholar
• Demetriou, D., M. Campagna, I. Racetin, and M. Konecny. 2017. ”Integrating Spatial Data Infrastructures (SDIs) with Volunteered Geographic Information (VGI) Creating a Global GIS Platform.” In Mapping and the Citizen Sensor, edited by G. Foody, L. See, S. Fritz, P. Mooney, C. C. Fonte, A. M. Olteanu-Raimond, and V. Antoniou, 273–297. London: Ubiquity Press.
   Google Scholar
• Ehrlinger, J., K. Johnson, M. Banner, D. Dunning, and J. Kruger. 2008. “Why the Unskilled are Unaware: Further Explorations of (Absent) Self-Insight Among the Incompetent.” Organizational Behavior and Human Decision Processes 105 (1): 98–121. doi:10.1016/j.obhdp.2007.05.002.
   PubMed Web of Science ®Google Scholar
• Elia, A., S. Balbo, and P. Boccardo. 2018. “A Quality Comparison Between Professional and Crowdsourced Data in Emergency Mapping for Potential Cooperation of the Services.” European Journal of Remote Sensing 51 (1): 572–586. doi:10.1080/22797254.2018.1460567.
   Web of Science ®Google Scholar
• Elwood, S., M. F. Goodchild, and D. Z. Sui. 2012. “Researching Volunteered Geographic Information: Spatial Data, Geographic Research, and New Social Practice.” Annals of the Association of American Geographers 102: 571–590.
   Web of Science ®Google Scholar
• Feick, R., and C. Robertson. 2015. “A Multi-Scale Approach to Exploring Urban Places in Geotagged Photographs.” Computers, Environment and Urban Systems 53: 96–109. doi:10.1016/j.compenvurbsys.2013.11.006.
   Web of Science ®Google Scholar
• Fielding, A. H., and J. F. Bell. 1997. “A Review of Methods for the Assessment of Prediction Errors in Conservation Presence/absence Models.” Environmental Conservation 24: 38–49.
   Web of Science ®Google Scholar
• Finn, J. T. 1993. “Use of the Average Mutual Information Index in Evaluating Classification Error and Consistency.” International Journal of Geographical Information Science 7 (4): 349–366.
   Web of Science ®Google Scholar
• Flanagin, A., and M. Metzer. 2008. “The Credibility of Volunteered Geographic Information.” GeoJournal 72 (3–4): 137–148.
   Google Scholar
• Fleiss, J. L. 1971. “Measuring Nominal Scale Agreement Among Many Raters.” Psychological Bulletin 76: 378–382.
   Web of Science ®Google Scholar
• Fogliaroni, P., F. D’-Antonio, and E. Clementini. 2018. “Data Trustworthiness and User Reputation as Indicators of VGI Quality.” Geo-Spatial Information Science 21 (3): 213–233. doi:10.1080/10095020.2018.1496556.
   Web of Science ®Google Scholar
• Fonte, C. C., L. Bastin, L. See, G. Foody, and F. Lupia. 2015a. “Usability of VGI for Validation of Land Cover Maps.” International Journal of Geographical Information Science 29 (7): 1269–1291. doi:10.1080/13658816.2015.1018266.
   Web of Science ®Google Scholar
• Fonte, C. C., L. Bastin, L. See, G. Foody, and J. Estima 2015b. “Good Practice Guidelines for Assessing VGI Data Quality.” In The 18th AGILE International Conference on Geographic Information Science. Lund, Sweden.
   Google Scholar
• Foody, G. M. 2013. “Ground Reference Data Error and the Mis-Estimation of the Area of Land Cover Change as a Function of Its Abundance.” Remote Sensing Letters 4: 783–792.
   Web of Science ®Google Scholar
• Foody, G. M., L. See, S. Fritz, M. Van der Velde, C. Perger, C. Schill, D. S. Boyd, and A. Comber. 2015. “Accurate Attribute Mapping from Volunteered Geographic Information: Issues of Volunteer Quantity and Quality.” The Cartographic Journal 52 (4): 336–344. doi:10.1080/00087041.2015.1108658.
   Web of Science ®Google Scholar
• Foody, G., L. See, S. Fritz, P. Mooney, C. Fonte, A. M. Olteanu-Raimond, and V. Antoniou, edited by. 2017a. Mapping and the Citizen Sensor 398pp. London: Ubiquity Press. doi:10.5334/bbf.
   Google Scholar
• Foody, G., S. Fritz, C. C. Fonte, L. Bastin, A. M. Olteanu-Raimond, P. Mooney, L. See, et al. 2017b. In Mapping and the Citizen Sensor, In edited by G. Foody, L. See, S. Fritz, P. Mooney, C. C. Fonte, A. M. Olteanu-Raimond, and V. Antoniou, 1–12. London: Ubiquity Press.
   Google Scholar
• Foody, G., L. See, S. Fritz, I. Moorthy, C. Perger, C. Schill, and D. Boyd. 2018. “Increasing the Accuracy of Crowdsourced Information on Land Cover via a Voting Procedure Weighted by Information Inferred from the Contributed Data.” ISPRS International Journal of Geo-Information 7 (3): 80. doi:10.3390/ijgi7030080.
   Web of Science ®Google Scholar
• Foody, G. M. 2020. “Explaining the Unsuitability of the Kappa Coefficient in the Assessment and Comparison of the Accuracy of Thematic Maps Obtained by Image Classification.” Remote Sensing of Environment 239: 111630. doi:10.1016/j.rse.2019.111630.
   Web of Science ®Google Scholar
• Freeman, A. 2017. Essential Docker for ASP. NET Core MVC. Berkeley, CA: Apress.
   Google Scholar
• Garnett, R., and R. Stewart. 2015. “Comparison of GPS Units and Mobile Apple GPS Capabilities in an Urban Landscape.” Cartography and Geographic Information Science 42: 1–8.
   Web of Science ®Google Scholar
• Goodchild, M. F. 2007. “Citizens as Sensors: The World of Volunteered Geography.” GeoJournal 69 (4): 211–221. doi:10.1007/s10708-007-9111-y.
   Google Scholar
• Goodchild, M. F., and L. Li. 2012. “Assuring the Quality of Volunteered Geographic Information.” Spatial Statistics 1: 110–120. doi:10.1016/j.spasta.2012.03.002.
   Web of Science ®Google Scholar
• Gopal, S., and C. Woodcock. 1994. “Theory and Methods for Accuracy Assessment of Thematic Maps Using Fuzzy Sets.” Photogrammetric Engineering and Remote Sensing 60: 181–188.
   Web of Science ®Google Scholar
• Griesbaum, L., S. Marx, and B. Höfle. 2017. “Direct Local Building Inundation Depth Determination in 3-D Point Clouds Generated from User-Generated Flood Images.” Natural Hazards and Earth System Sciences 17 (7): 1191–1201. doi:10.5194/nhess-17-1191-2017.
   Web of Science ®Google Scholar
• Haklay, M., S. Basiouka, V. Antoniou, and A. Ather. 2010. “How Many Volunteers Does It Take to Map an Area Well? the Validity of Linus’ Law to Volunteered Geographic Information.” The Cartographic Journal 47 (4): 315–322. doi:10.1179/000870410X12911304958827.
   Web of Science ®Google Scholar
• Havlik, D., J. Soriano, C. Granell, S. E. Middleton, H. van der Schaaf, A. Berre, and J. Pielorz. 2013. “Future Internet Enablers for VGI Applications.” Proceedings of the 27th Conference on Environmental Informatics (EnviroInfo 2013), Aachen, Hamburg, Germany: Shaker Verlag, pp. 622–630.
   Google Scholar
• Hay, A. M. 1979. “Sampling Designs to Test Land-Use Map Accuracy.” Photogrammetric Engineering and Remote Sensing 45: 529–533.
   Web of Science ®Google Scholar
• Hickling Arthurs Low Corporation. 2012. Volunteered Geographic Information (VGI) Primer, Canadian Geospatial Data Infrastructure Product 21e. Retrieved from: https://geoscan.nrcan.gc.ca/starweb/geoscan/servlet.starweb?path=geoscan/downloade.web&search1=R=291948
   Google Scholar
• Higgins, C.I., J. Williams, D. G. Leibovici, I. Simonis, M. J. Davis, C. Muldoon, P. van Genuchten, G. O’-Hare, and S. Wiemann. 2016. “Citizen Observatory WEB (COBWEB): A Generic Infrastructure Platform to Facilitate the Collection of Citizen Science Data for Environmental Monitoring.” International Journal of Spatial Data Infrastructures Research 11: 20–48.
   Web of Science ®Google Scholar
• Iwao, K., K. Nishida, T. Kinoshita, and Y. Yamagata. 2006. “Validating Land Cover Maps with Degree Confluence Project Information.” Geophysical Research Letters 33 (23): l23404. doi:10.1029/2006GL027768.
   Web of Science ®Google Scholar
• Ke, Y., X. Tang, and F. Jing 2006. “The Design of High-Level Features for Photo Quality Assessment.” 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’06), New York, NY, USA, 2006, 419–426. doi:10.1109/CVPR.2006.303.
   Google Scholar
• Knight, J. F., and R. S. Lunetta. 2003. “An Experimental Assessment of Minimum Mapping Unit Size.” IEEE Transactions on Geoscience and Remote Sensing 41 (9): 2132–2134. doi:10.1109/TGRS.2003.816587.
   Web of Science ®Google Scholar
• Laso Bayas, J. C., M. Lesiv, F. Waldner, A. Schucknecht, M. Duerauer, L. See, S. Fritz, D. Fraisl, et al. 2017. ”A Global Reference Database of Crowdsourced Cropland Data Collected Using the Geo-Wiki Platform.” Scientific Data 4 :e170136. doi:10.1038/sdata.2017.136.
   PubMed Web of Science ®Google Scholar
• Liao, S., A. Jain, and L. Stan. 2014. “A Fast and Accurate Unconstrained Face Detector.” IEEE Transactions on Pattern Analysis and Machine Intelligence 38: 211–223.
   Web of Science ®Google Scholar
• Lo, S. W., J. H. Wu, F. P. Lin, and C. H. Hsu. 2015. “Visual Sensing for Urban Flood Monitoring.” Sensors 15: 20006–20029.
   PubMed Web of Science ®Google Scholar
• Masood, S. Z., G. Shu, A. Dehghan, and E. G. Ortiz. 2017. ”License Plate Detection and Recognition Using Deeply Learned Convolutional Neural Networks.” arXiv preprint arXiv:1703.07330.
   Google Scholar
• Menard, T., J. Miller, M. Mowak, and D. Norris 2011. “Comparing the GPS Capabilities of the Samsung Galaxy S, Motorola Droid X, and the Apple iPhone for Vehicle Tracking Using FreeSim Mobile.” 14th International IEEE Conference of Intelligent Transportation Systems, Washington, DC.
   Google Scholar
• Merkel, D. 2014. “Docker: Lightweight Linux Containers for Consistent Development and Deployment.” Linux Journal 239: 2.
   Google Scholar
• Merry, K., and P. Bettinger. 2019. “Smartphone GPS Accuracy Study in an Urban Environment.” PLoS One 14 (7): e0219890.
   PubMed Web of Science ®Google Scholar
• Meyer, H., and E. Pebesma. 2022. “Machine Learning-Based Global Maps of Ecological Variables and the Challenge of Assessing Them.” Nature Communications 13 (1): 1–4. doi:10.1038/s41467-022-29838-9.
   PubMed Web of Science ®Google Scholar
• Minghini, M., V. Antoniou, C. C. Fonte, J. Estima, A. M. Olteanu-Raimond, L. See, and M. Laakso, et al. 2017. ”The Relevance of Protocols for VGI Collection.” In Mapping and the Citizen Sensor, edited by G. Foody, L. See, S. Fritz, P. Mooney, C. C. Fonte, A. M. Olteanu-Raimond, and V. Antoniou, 223–247. London: Ubiquity Press.
   Google Scholar
• Mobasheri, A., A. Zipf, and L. Francis. 2018. “OpenStreetmap Data Quality Enrichment Through Awareness Raising and Collective Action Tools—experiences from a European Project.” Geo-Spatial Information Science 21 (3): 234–246. doi:10.1080/10095020.2018.1493817.
   Web of Science ®Google Scholar
• Mocnik, F. B., A. Mobasheri, L. Griesbaum, M. Eckle, C. Jacobs, and C. Klonner. 2018. “A Grounding-Based Ontology of Data Quality Measures.” Journal of Spatial Information Science 2018 (16): 1–25.
   Google Scholar
• Mooney, P., M. Minghini, M. Laakso, V. Antoniou, A. M. Olteanu-Raimond, and A. Skopeliti. 2016. “Towards a Protocol for the Collection of VGI Vector Data.” ISPRS International Journal of Geo-Information 5 (11): 217.
   Web of Science ®Google Scholar
• Mooney, P., A. M. Olteanu-Raimond, G. Touya, N. Juul, S. Alvanides, and N. Kerle. 2017. ”Considerations of Privacy, Ethics and Legal Issues in Volunteered Geographic Information.” In Mapping and the Citizen Sensor, edited by G. Foody, L. See, S. Fritz, P. Mooney, C. C. Fonte, A. M. Olteanu-Raimond, and V. Antoniou, 119–135. London: Ubiquity Press.
   Google Scholar
• Moorthy, I., S. Fritz, L. See, and I. McCallum. 2017. “LandSense: A Citizen Observatory and Innovation Marketplace for Land Use and Land Cover Monitoring.” In EGU General Assembly Conference Abstracts, (pp. 8562). Vienna, Austria.
   Google Scholar
• Moorthy, I. 2020. “LandSense: A Citizen Observatory and Innovation Marketplace for Land Use and Land Cover Monitoring.” 18th European Week of Regions and Cities, Zenodo. doi:10.5281/zenodo.4146821
   Google Scholar
• Nada, H., V. A. Sindagi, H. Zhang, and V. M. Patel. 2018. “Pushing the Limits of Unconstrained Face Detection: A Challenge Dataset and Baseline Results.” In 2018 IEEE 9th International Conference on Biometrics Theory, Applications and Systems (BTAS), Redondo Beach, CA, USA: IEEE.
   Google Scholar
• Obe, R., and L. Hsu. 2011. “PostGis in Action.” GEOInformatics 14 (8): 30.
   Google Scholar
• Olofsson, P., G. M. Foody, S. V. Stehman, and C. E. Woodcock. 2013. “Making Better Use of Accuracy Data in Land Change Studies: Estimating Accuracy and Area and Quantifying Uncertainty Using Stratified Estimation.” Remote Sensing of Environment 129: 122–131. doi:10.1016/j.rse.2012.10.031.
   Web of Science ®Google Scholar
• Olofsson, P., G. M. Foody, M. Herold, S. V. Stehman, C. E. Woodcock, and M. A. Wulder. 2014. “Good Practices for Estimating Area and Assessing Accuracy of Land Change.” Remote Sensing of Environment 148: 42–57. doi:10.1016/j.rse.2014.02.015.
   Web of Science ®Google Scholar
• Olteanu Raimond, A. M., G. Hart, G. M. Foody, G. Touya, T. Kellenberger, and D. Demetriou. 2017a. “The Scale of VGI in Map Production: A Perspective on European National Mapping Agencies.” Transactions in GIS 21 (1): 74–90.
   Web of Science ®Google Scholar
• Olteanu-Raimond, A. M., M. Laakso, V. Antoniou, C. C. Fonte, A. Fonseca, M. Grus, J. Harding, T. Kellenberger, M. Minghini, and A. Skopeliti. 2017b. ”VGI in National Mapping Agencies: Experiences and Recommendations.” In Mapping and the Citizen Sensor, edited by G. Foody, L. See, S. Fritz, P. Mooney, C. C. Fonte, A. M. Olteanu-Raimond, and V. Antoniou, 299–326. London: Ubiquity Press.
   Google Scholar
• Olteanu-Raimond, A. M., L. See, M. Schultz, G. Foody, M. Riffler, T. Gasber, L. Jolivet, et al. 2020. “Use of Automated Change Detection and VGI Sources for Identifying and Validating Urban Land Use Change.” Remote Sensing 12 (7): 1186.
   Web of Science ®Google Scholar
• Ostermann, F. O., and C. Granell. 2017. “Advancing Science with VGI: Reproducibility and Replicability of Recent Studies Using VGI.” Transactions in GIS 21 (2): 224–237. doi:10.1111/tgis.12195.
   Web of Science ®Google Scholar
• Pech-Pacheco, J., L. G. Cristobal, J. Chamorro-Martinez, and J. Fernandez-Valdivia. 2000. “Diatom Autofocusing in Brightfield Microscopy: A Comparative Study.” Proceedings 15thInternational Conference on Pattern Recognition Barcelona, Spain. ICPR-2000, 2000, pp. 314–317.
   Google Scholar
• Ploton, P., F. Mortier, M. Réjou-Méchain, N. Barbier, N. Picard, V. Rossi, C. Dormann, et al. 2020. “Spatial Validation Reveals Poor Predictive Performance of Large-Scale Ecological Mapping Models.” Nature Communications 11 (1): 1–11. doi:10.1038/s41467-020-18321-y.
   PubMed Web of Science ®Google Scholar
• Pontius, R. G. 2000. “Quantification Error versus Location Error in Comparison of Categorical Maps.” Photogrammetric Engineering and Remote Sensing 66: 1011–1016.
   Web of Science ®Google Scholar
• Pontius, R. G., Jr, and M. Millones. 2011. “Death to Kappa: Birth of Quantity Disagreement and Allocation Disagreement for Accuracy Assessment.” International Journal of Remote Sensing 32 (15): 4407–4429. doi:10.1080/01431161.2011.552923.
   Web of Science ®Google Scholar
• Salk, C. F., T. Sturn, L. See, S. Fritz, and C. Perger. 2016. “Assessing Quality of Volunteer Crowdsourcing Contributions: Lessons from the Cropland Capture Game.” International Journal of Digital Earth 9 (4): 410–426. doi:10.1080/17538947.2015.1039609.
   Web of Science ®Google Scholar
• Saura, S. 2002. “Effects of Minimum Mapping Unit on Land Cover Data Spatial Configuration and Composition.” International Journal of Remote Sensing 23 (22): 4853–4880. doi:10.1080/01431160110114493.
   Web of Science ®Google Scholar
• Scepan, J. 1999. “Thematic Validation of High-Resolution Global Land-Cover Data Sets.” Photogrammetric Engineering and Remote Sensing 65: 1051–1060.
   Web of Science ®Google Scholar
• See, L., P. Mooney, G. Foody, L. Bastin, A. Comber, J. Estima, S. Fritz, et al. 2016. “Crowdsourcing, Citizen Science or Volunteered Geographic Information? the Current State of Crowdsourced Geographic Information.” ISPRS International Journal of Geo-Information 5 (5): 55.
   Web of Science ®Google Scholar
• See, L., and J. C. L. Bayas, M. Lesiv, D. Schepaschenko, O. Danylo, I. McCallum, M. Dürauer, I. Georgieva, D. Domian, D. Fraisl, and G. Hager. 2022. “Lessons Learned in Developing Reference Data Sets with the Contribution of Citizens: The Geo-Wiki Experience.” Environmental Research Letters 17 (6): 065003.
   Web of Science ®Google Scholar
• Senaratne, H., A. Mobasheri, A. L. Ali, C. Capineri, and M. Haklay. 2017. “A Review of Volunteered Geographic Information Quality Assessment Methods.” International Journal of Geographical Information Science 31 (1): 139–167. doi:10.1080/13658816.2016.1189556.
   Web of Science ®Google Scholar
• Severinsen, J., M. de Roiste, F. Reitsma, and E. Hartato. 2019. “Vgtrust: Measuring Trust for Volunteered Geographic Information.” International Journal of Geographical Information Science 33 (8): 1683–1701. doi:10.1080/13658816.2019.1572893.
   Web of Science ®Google Scholar
• Shima, Y., Y. Nakashima, and M. Yasuda. 2017. “Detecting Orientation of In-Plain Rotated Face Images Based on Category Classification by Deep Learning.” In TENCON 2017–2017 IEEE Region 10 Conference Penang, Malaysia, IEEE, 127–132.
   Google Scholar
• Silva, S. M., and C. R. Jung. 2018. “License Plate Detection and Recognition in Unconstrained Scenarios.” In Proceedings of the European Conference on Computer Vision (ECCV), Munich, Germany, 580–596.
   Google Scholar
• Stehman, S. V., and R. L. Czaplewski. 1998. “Design and Analysis for Thematic Map Accuracy Assessment: Fundamental Principles.” Remote Sensing of Environment 64 (3): 331–344. doi:10.1016/S0034-4257(98)00010-8.
   Web of Science ®Google Scholar
• Stehman, S. V. 1999. “Basic Probability Sampling Designs for Thematic Map Accuracy Assessment.” International Journal of Remote Sensing 20 (12): 2423–2441. doi:10.1080/014311699212100.
   Web of Science ®Google Scholar
• Stehman, S. V. 2009. “Sampling Designs for Accuracy Assessment of Land Cover.” International Journal of Remote Sensing 30: 5243–5272.
   Web of Science ®Google Scholar
• Stehman, S. V., C. C. Fonte, G. M. Foody, and L. See. 2018. “Using Volunteered Geographic Information (VGI) in Design-Based Statistical Inference for Area Estimation and Accuracy Assessment of Land Cover.” Remote Sensing of Environment 212: 47–59. doi:10.1016/j.rse.2018.04.014.
   Web of Science ®Google Scholar
• Stehman, S. V., and G. M. Foody. 2019. “Key Issues in Rigorous Accuracy Assessment of Land Cover Products.” Remote Sensing of Environment 231: 111199. doi:10.1016/j.rse.2019.05.018.
   Web of Science ®Google Scholar
• Szeliski, R. 2010. Computer Vision: Algorithms and Applications. London: Springer Science & Business Media.
   Google Scholar
• Vahidi, H., B. Klinkenberg, and W. Yan. 2018. “Trust as a Proxy Indicator for Intrinsic Quality of Volunteered Geographic Information in Biodiversity Monitoring Programs.” GIScience & Remote Sensing 55 (4): 502–538. doi:10.1080/15481603.2017.1413794.
   Web of Science ®Google Scholar
• Wadembere, I., and P. Ogao. 2010. “Geometry Updating for Geospatial Data Integration.” International Archives of Photogrammetry and Remote Sensing 38: 52–57.
   Google Scholar
• Wadoux, A. M. C., G. B. M. Heuvelink, S. de Bruin, and D. J. Brus. 2021. “Spatial Cross-Validation is Not the Right Way to Evaluate Map Accuracy.” Ecological Modelling 457: 109692.
   Web of Science ®Google Scholar
• Waldner, F., A. Schucknecht, M. Lesiv, J. Gallego, L. See, A. Pérez-Hoyos, R. D’-Andrimont, T. de Maet, et al. 2019. ”Conflation of Expert and Crowd Reference Data to Validate Global Binary Thematic Maps”. Remote Sensing of Environment 221: 235–246. doi:10.1016/j.rse.2018.10.039.
   Web of Science ®Google Scholar
• Wannemacher, K., B. Birli, T. Sturn, R. Stiles, I. Moorthy, L. See, and S. Fritz. 2018. “Using Citizen Science to Help Monitor Urban Landscape Changes and Drive Improvements.” Journal for Geographic Information Science-Gi_forum 1: 336–343.
   Google Scholar
• Wickham, J. D., S. V. Stehman, L. Gass, J. Dewitz, J. A. Fry, and T. G. Wade. 2013. “Accuracy Assessment of NLCD 2006 Land Cover and Impervious Surface.” Remote Sensing of Environment 130: 294–304. doi:10.1016/j.rse.2012.12.001.
   Web of Science ®Google Scholar
• Wolf, L., T. Hassner, and I. Maoz. 2011 June. “Face Recognition in Unconstrained Videos with Matched Background Similarity.” In CVPR 2011, Colorado Springs, CO, USA: IEEE, 529–534.
   Google Scholar
• Wu, F., B. Wu, M. Zhang, H. Zeng, and F. Tian. 2021. “Identification of Crop Type in Crowdsourced Road View Photos with Deep Convolutional Neural Network.” Sensors 21 (4): 1165. doi:10.3390/s21041165.
   PubMed Web of Science ®Google Scholar
• Wulder, M. A., J. C. White, S. Magnussen, and S. McDonald. 2007. “Validation of a Large Area Land Cover Product Using Purpose-Acquired Airborne Video.” Remote Sensing of Environment 106: 480–491.
   Web of Science ®Google Scholar
• Xing, D., S. V. Stehman, G. M. Foody, and B. W. Pengra. 2021. “Comparison of Simple Averaging and Latent Class Modeling to Estimate the Area of Land Cover in the Presence of Reference Data Variability.” Land 10 (1): 35. doi:10.3390/land10010035.
   Web of Science ®Google Scholar
• Xu, G., X. Zhu, D. Fu, J. Dong, and X. Xiao. 2017. “Automatic Land Cover Classification of Geo-Tagged Field Photos by Deep Learning.” Environmental Modelling and Software 91: 127–134. doi:10.1016/j.envsoft.2017.02.004.
   Web of Science ®Google Scholar
• Yang, S., P. Luo, C. C. Loy, and X. Tang. 2016. “Wider Face: A Face Detection Benchmark.” In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, (5525–5533). Las Vegas, NV, USA.
   Google Scholar
• Zandbergen, P. A. 2009. “Accuracy of iPhone Locations: A Comparison of Assisted GPS, WiFi and Cellular Positioning.” Transactions in GIS 13 (S1): 5–25. doi:10.1111/j.1467-9671.2009.01152.x.
   Web of Science ®Google Scholar
• Zielstra, D., and H. H. Hochmair. 2013. “Positional Accuracy Analysis of Flickr and Panoramio Images for Selected World Regions.” Journal of Spatial Science 58 (2): 251–273. doi:10.1080/14498596.2013.801331.
   Web of Science ®Google Scholar