Automatically Finding Actors in Texts: A Performance Review of Multilingual Named Entity Recognition Tools

Figures & data

Table 1. Most commonly used techniques for named entity recognition (based on literature review).

NER	Type	No. articles	Language(s)	Purpose(s)
Dictionary (handcrafted)	keyword based	16	English, Dutch, German, Arabic, Chinese, Multilingual (5+ languages)	Location extraction; actor extraction; organization extraction; social groups extraction
Search Terms*	keyword based	6	English, Dutch, Chinese	deductive sample composition
Stanford CoreNLP	supervised ML	5	English (3), Portuguese (1)	Sentiment associations; location extraction
GDELT* (precoded)	supervised ML	2	English, Multilingual (5+ languages)	sample composition
ICEWS (precoded)	keyword based	2	English	Event detection, conflict measurement
IBM Watson*	supervised ML	2	English	Speaker identification, actor extraction
Newsmap	semi-supervised ML	2	English	Geo classification
Apache OpenNLP	supervised ML	1	English	Event extraction
spaCy	supervised ML	1	English	Actor extraction
NLTK	supervised ML	1	Dutch (custom classifier)	Actor extraction
Frog*	supervised ML	1	Dutch	Document similarity based on NE

Note. “Search Terms” refers to a specific collection of keywords used for data sampling (e.g., using names of specific politicians to query LexisNexis).

*NER tool not evaluated in this review.

Table 2. Comparison of tools according to features and speed.

Tool	Method	Languages	Coreference	Tokens/sec
			resolution	High end	Consumer
ICEWS (Boschee et al., 2015)	dictionary	10		16,145	8,740
JRC-Names (Steinberger et al., 2011)	dictionary	45		2,717	1,555
Apache OpenNLP^a	machine learning	3		19,127	10,165
CoreNLP (Manning et al., 2014)	machine learning	7	✓	2,531	1,224
Nametagger (Straková et al., 2013)	machine learning	2		130,207	85,530
NLTK (Bird et al., 2009)	machine learning	1^b		7,638	4,320
spaCy (Honnibal et al., 2020)	neural network	10	✓	8,032	3,414
spaCy (transformers)	neural network	2		3,840	274
XLM-RoBERTa (Conneau et al., 2019)	neural network	13^c		23,155	478

Note. ^aDetails and authorship see: https://opennlp.apache.org/.

^bThe software package ships with one pre-trained model for English, but can be trained on other languages as well. For example, see Burggraaff and Trilling, 2020.

^cThe neural network was pre-trained with texts from over 100 languages, we evaluate its performance in 13 languages.

Table 3. Selected corpora for NER performance evaluation.

Corpus	Released	Languages	Sentences	Tokens
CoNLL	2002	Dutch, English, German, Spanish	12,967	223,713
AnCora	2008	Catalan, Spanish	5,075	161,462
HAREM	2008	Portuguese	342	12,941
AQMAR	2012	Arabic	404	11,171
OntoNotes	2013	Arabic, Chinese, English	21,873	425,244
SoNaR	2013	Dutch	8,993	149,817
CNEC 2.0	2014	Czech	899	20,146
GermEval	2014	German	5,100	96,499
Europeana*	2016	Dutch, French, German	1,840	71,170
Emerging Entities*	2017	English	1,287	23,394
WikiANN**	2017	13/282	325,000	2,850,511
HIPE*	2020	English, French, German	138	88,225
FiNER	2020	Finnish	83	46,363
KIND	2020	Italian	2,088	56,519
NYTK-NerKor	2021	Hungarian	6,752	103,025

Note. Numbers for sentences and tokens only counts the validation splits. For full descriptions of each corpus refer to section B and Table B1 in the Appendix.

*Noisy corpus. **Silver-standard with noisy annotations.

Figure 1. Performance of XLM-RoBERTa with increasing size of training data for fine-tuning. One training sample is one sentence.

Figure 2. Precision, recall, and F₁ score for each NER tool grouped by languages.

Figure 3. Precision, recall, and F₁ score for each NER tool grouped by corpora.

Table 4. Number of passed challenges per tool.

	CoreNLP	ICEWS	JRC-Names	nametagger	NLTK	OpenNLP	spaCy	XLM-RoBERTa	Total
English	5	1	3	2	6	2	13	10	15
Spanish	0	0	1			2	1	5	6
German	1	0	2				1	1	2
Dutch		1	2			2	2	12	13

Note. Empty cells indicate that the tool does not support the language.

Boschee, E., Lautenschlager, J., Shellman, S., & Shilliday, A. (2015). ICEWS dictionaries. https://doi.org/10.7910/DVN/28118

 Google Scholar

Steinberger, R., Pouliquen, B., Kabadjov, M., Belyaeva, J., & Van Der Goot, E. (2011). JRC-NAMES: A freely available, highly multilingual named entity resource. International Conference Recent Advances in Natural Language Processing, RANLP, Hissar, Bulgaria (pp. 104–110). https://aclanthology.org/R11-1015

 Google Scholar

Manning, C., Surdeanu, M., Bauer, J., Finkel, J., Bethard, S., & McClosky, D. (2014). The Stanford CoreNLP natural language processing toolkit. Proceedings of 52nd Annual Meeting of the Association for Computational Linguistics: System Demonstrations, (pp. 55–60). https://doi.org/10.3115/v1/P14-5010

 Google Scholar

Straková, J., Straka, M., & Hajič, J. (2013). A new state-of-the-art Czech Named Entity Recognizer. In I. Habernal & V. Matoušek (Eds.), Text, speech, and dialogue (pp. 68–75). Springer. https://doi.org/10.1007/978-3-642-40585-3_10

 Google Scholar

Bird, S., Klein, E., & Loper, E. (2009). Natural language processing with Python: Analyzing text with the natural language Toolkit. O’Reilly Media.

 Google Scholar

Honnibal, M., Montani, I., Van Landeghem, S., & Boyd, A. (2020). spaCy: Industrial-strength natural language processing in Python. https://doi.org/10.5281/zenodo.1212303

 Google Scholar

Conneau, A., Khandelwal, K., Goyal, N., Chaudhary, V., Wenzek, G., Guzmán, F., Grave, E., Ott, M., Zettlemoyer, L., & Stoyanov, V. (2019). Unsupervised cross-lingual representation learning at scale. https://doi.org/10.48550/arXiv.1911.02116

 Google Scholar

Burggraaff, C., & Trilling, D. (2020). Through a different gate: An automated content analysis of how online news and print news differ. Journalism, 21(1), 112–129. https://doi.org/10.1177/1464884917716699

 Web of Science ®Google Scholar