Public policy instruments to promote freight modal shift in Europe: evidence from evaluations

ABSTRACT

A modal shift of freight from road to rail and waterborne transport can contribute to reduced negative externalities (mainly air pollution) from freight transport. The purpose of this paper is to identify modal shift public policy instruments in Europe and analyse their performance based on existing ex post evaluations. This analysis identifies 93 public policy instruments, in which 20 ex post evaluations were found. The evaluations mainly consider subsidies/grants at the national level or regulations and directives at the European Union (EU) level. The results suggest that evaluations of subsidies and grants at the national level most commonly describe a positive performance, while several evaluations at the EU level describe a poor or mixed performance. Well-defined targets and simpler application processes are mentioned in several ex post evaluations as suggestions for improving the performance of modal shift policy instruments in Europe.

KEYWORDS:

• Modal shift
• freight transport
• public policy instruments
• evaluation
• Europe

1. Introduction

A modal shift from road to both rail and waterborne transport,¹ could reduce external costs from freight transport (Bickford et al., 2014; Nealer et al., 2012). Using rail and waterborne transport generally consumes less energy per tonne than road transport, as well as emits fewer greenhouse gas emissions (GHG), and ultimately reduces the number of accidents and congestion on roads. Although those externalities depend on aspects such as load factor, and the energy sources used,² in 2016 CE Delft (2020) estimated the external costs³ for heavy trucks, rail and inland waterways transport (IWT) to be €0.042, 0.013 and 0.019 per tonne-kilometre respectively. Even though the external costs are generally lower for rail freight and IWT than for road, this is not always the case for intermodal transportation (Kaack et al., 2018). For example, Santos et al. (2015) find that, depending on the length of the road haul, internalizing external costs can even disadvantage intermodal transport operations. Thus, a modal shift should be viewed as achieving the objectives of reducing externalities, and not as an objective in itself.

To address the real need to reduce the negative consequences of freight transport, several public actors in Europe have established targets and adopted policy instruments⁴ to promote a modal shift from road to rail and waterborne transport.⁵ However, this ambition has not been achieved at the aggregate level in the European Union (Eurostat, 2020), and several countries are far from meeting their modal shift objective (Pinchasik et al., 2020). As a result, road transport remains the chief hauling mode in Europe, where it represents more than three-quarters of all inland freight movement. Beyond that, a tendency towards modal backshift can be observed over time as rail and waterborne transport lose market share to road transport (Eurostat, 2020; Pinchasik et al., 2020). Such trends indicate that although public policy instruments in Europe have the clear purpose of enhancing the modal shift of freight transport, the achievement of these existing strategies for reducing negative externalities remains unclear. In response, it is important to evaluate the performance of past and present public policy instruments to identify efficient and effective policies with the potential to advance modal shift and thereby reduce the negative externalities from freight transport in Europe.

In general, the methodology used to understand the performance of public policy instruments is based on policy evaluations. Those methods are helpful tools for gathering evidence concerning the assessment of policy instruments that decision-makers can use (European Commission, 2017a). Considering different approaches, previous research has examined the performance attributed to certain policy instruments in achieving the modal shift of freight transport in Europe. While several analyses have prioritised the expected effect (i.e. ex ante simulations) as a means to demonstrate the potential of specific policy instruments (e.g. subsidies and grants) to foster intermodal transport in Europe (Pinchasik et al., 2020; Santos et al., 2015), studies that have evaluated the performance of policy instruments already implemented (i.e. ex post analysis) have revealed divergent trends. For instance, Suárez-Alemán (2016) and Aperte and Baird (2013) have suggested that policy instruments used to promote modal shift to short sea shipping (SSS) have been insufficient or with limited effect at the European Union (EU) level. Other analyses report that direct grants to combined transport⁶ operations, as well as grants to intermodal facilities, encourage the improvement of combined transport within the EU (KombiConsult GmbH et al., 2015).

This paper aims to identify and classify past and present public policy instruments, as well as evaluate their performance, to estimate their effectiveness and efficiency in achieving a modal shift of freight transport. No previous empirical study has involved an original investigation entailing a comprehensive review and categorisation of the evaluation of public policy instruments regarding the modal shift of freight transport, especially not with the expressed purpose of facilitating valuable information to policymakers and improving knowledge of the instruments’ effectiveness and efficiency. Thus, this research sought to partly fill that research gap by answering the following research questions:

RQ1 – What public policy instruments have been implemented in Europe with the aim of achieving a modal shift of freight transport?

RQ2 – Which policy instruments have been evaluated, and which evaluation methods and performance indicators have been applied?

RQ3 – What conclusions can be drawn regarding the effectiveness and efficiency of European public policy instruments in terms of achieving modal shift and reducing negative externalities?

The methodology employed comprised a qualitative analysis following a comprehensive review of existing policy instruments implemented in Europe to identify, classify, and evaluate their performance based on their ex post evaluations. To that purpose, instead of considering the expected effect (i.e. ex ante simulations), this study focused on ex post evaluations, which allowed considering evidence-based judgements in search for causality between policy instruments and any observed changes after a period long enough to allow for such changes to be identified and measured (European Commission, 2017a). This research demonstrates that the performance of policy evaluations can improve understanding of the functioning of current and future policy instruments, especially ones based on lessons learned from past experience (European Commission, 2017a).

The results suggest that the most common public policy instruments applied to achieve modal shift in Europe are subsidies and grants implemented at the national level and mostly targeting rail freight transport. However, through the identification of 93 policy instruments, only 20 ex post evaluations were detected. That discrepancy suggests a lack of assessment, probably due to limited reliable data and/or common guidelines for evaluations. Moreover, in terms of performance, this study found that several evaluations of EU-policy instruments describe a poor or a mixed performance of the policy instruments, while the performance of subsidies and grants at the national level are often considered positive by the evaluations. In general, it seems to be a more positive performance of policy instruments promoting a modal shift to rail than waterborne transport.

This research contributes to evaluations of the effectiveness and efficiency of public policy instruments in achieving the modal shift of freight transport by implementing a unique classification and evaluation of policy instruments and their performance. As a result, policymakers can enhance their understanding of the existing policies and use that understanding as a basis for implementing future public policy instruments in the EU.

The remainder of the paper is structured as follows: Section 2 outlines a literature review on policy evaluation features and policy instruments, Section 3 describes the methodology applied in the study and the delimitations, Section 4 explains the results of the study, and Section 5 summarises the main conclusions and discusses what they imply for policy.

2. Literature review on policy evaluations and of policy instruments in promoting modal shift in Europe

This research evaluates the performance of past and present policy instruments to identify efficient and effective policies with the potential to advance modal shift and reduce the negative externalities from freight transport in Europe. To begin, this section presents a description of the main features to consider in analysing the performance of policy evaluations and a review of studies conducted to assess the effectiveness and efficiency of public policy instruments already implemented in promoting modal shift in Europe.

2.1. Characteristics of policy evaluations

This section considers the main features of policy evaluations, such as their purpose, main performance, methodology, and actor responsible for the evaluation.

First, in the case of purpose, the performance of policy evaluations can improve understandings of how current and future policy instruments function based on lessons learned from past experience (European Commission, 2017a). According to the European Commission (2017a), an ex post evaluation should be an evidence-based judgement seeking for causality between a policy instrument and any observed changes as a result of the policy. They also highlight the importance for evaluations to be of high quality and following principles concerning comprehensiveness, proportionality, independence, and objectivity. According to the OECD/DAC Network on Development Evaluation (2019), policy instruments (called “interventions”) should be evaluated according to the following criteria:

• “Relevance: is the intervention doing the right things?”

• “Coherence: how well does the intervention fit?”

• “Effectiveness: is the intervention achieving its objectives?”

• “Efficiency: how well are resources being used?”

• “Impact: what difference does the intervention make?”

• “Sustainability: will the benefits last?”

Second, in the case of performance, the European Commission’s (2017a) “Better Regulation Guidelines” also provides guideline for performing policy evaluations. Their evaluation criteria, largely overlapping with the criteria of the OECD/DAC Network on Development Evaluation (2019), and include effectiveness, efficiency, relevance (given current needs), coherence (given other policy instruments), and EU added value.

The European Commission (2017a) has emphasised that even if several types of reports and activities cover some of the abovementioned questions and qualities, not all of them include all of the necessary elements to qualify as evaluations. To control for that limitation, in this paper, we included both evaluations that meet the European Commission’s (2017a) standards as well as evaluations that do not.

Third, in the case of method, despite the existence of several sets of guidelines and criteria for performing evaluations, challenges in evaluating policy instruments persist in practice. For instance, Crabb and Leroy (2012) have suggested that tasks such as finding relevant and reliable data and isolating changes that have occurred during the period for other reasons can became problematic for evaluators. Along similar lines, Huitema et al. (2011) and Christie (2003) found a gap between evaluation theory and how ex post evaluations are performed in practice. Harmelink et al. (2008) studied 20 policy instruments and their ex post evaluations and determined that policy instruments concerning energy often lack quantified targets and clear timeframes and that monitoring information is not collected on a regular basis. They also found that policy evaluations often have different characteristics and large variation in their methodologies, which makes comparing the results of evaluations difficult. Added to that, there is a variation in the quality of different evaluations. As a case in point, Haug et al. (2010) have shown that evaluations of climate policy performed in the EU have often not been systematic, which makes evidence-based policy and decision-making difficult.

Finally, in the case of the actor responsible for the evaluation, Colin et al. (2021) have added that even if the same types of methodologies are used, evaluations performed by private consultancy firms often generate rosier descriptions of policy performance than evaluations performed by others. They have proposed that that tendency can be explained by the fact that the public authorities evaluated have incentives to choose consultancy firms from which they expect more positive results, including continued financing and, in turn, a competitive advantage over other evaluators.

2.2. Evaluations of policy instruments in promoting modal shift in Europe

Policy instruments targeting modal shift have been investigated from several perspectives, all to facilitate freight’s modal shift from road to rail and/or waterborne transport. This study is focused on identifying past and present public policy instruments implemented to achieve such a modal shift in Europe and to evaluate their performance. In that regard, various efforts have been made to assess the performance of policy instruments already implemented (i.e. ex post evaluations) in promoting modal shift in Europe.

Concerning policies focused on promoting modal shift in general, KombiConsult GmbH et al. (2015) investigated several policy instruments promoting combined transport within the EU. In accordance with Aperte and Baird (2013), they argue that direct grants to combined transport operations as well as grants to intermodal facilities have the potential to promote combined transport. However, they also highlight the various downsides of direct grants. For example, transport operations by rail or water risk being shifted back to road when the grant or subsidy expires. Therefore, such policy instruments have to be permanent as a means to eliminate the risk of modal backshift. Furthermore, direct grants may also create distorted competition. In terms of cargo transport, Pittman et al. (2020) investigated policy instruments for the modal shift in transporting grain in Poland, the Czech Republic, Slovakia, and Ukraine and found that policy instruments focused directly on promoting infrastructure may be necessary to advance the modal shift to rail.

By comparison, concerning policies focused on enhancing modes of transport, Suárez-Alemán (2016) investigated how EU policy has contributed to shifting freight to SSS. He found that maritime transport has not been properly promoted and that the EU is not yet on the right path to meet the objectives stated in its “Transport White Paper” from 2011. He thus argues that policy instruments for modal shift in the form of outright grants to companies that shift their mode of transport lack incentives to promote efficiency in SSS. Furthermore, looking into EU investments, he found that little attention has been paid to efficiency in ports, which may be problematic given how crucial ports are in intermodal transport chains. Similarly, to Suárez-Alemán (2016), Aperte and Baird (2013) investigated policy instruments geared towards promoting SSS through Motorways of the Sea (MoS). They found that while policy instruments have achieved some success at the EU level, other instruments at the national level had been effective in promoting a modal shift to MoS, including Italy’s Ecobonus scheme, which is paid in the form of a subsidy (i.e. tariff rebate) to the users of maritime transport. They add that some of the Ecobonus scheme’s success may depend on the simplicity of the programme and its user-friendly approach. In any case, the policy could be further improved by issuing payments more frequently.

Despite the literature reviewed above, no previous study has examined the evaluation methods and performance indicators of all different public policy instruments implemented in Europe as a means to gauge their effectiveness and efficiency in enhancing modal shift in Europe.

3. Research strategy

This section presents an empirical analysis aimed at identifying and classifying past and present public policy instruments, as well as their evaluation methods and performance indicators, to investigate their effectiveness and efficiency in promoting a modal shift from road to rail and waterborne transport in Europe. To address the stated research questions, this study developed a research approach to proceed in three major stages. First, a comprehensive list of policy instruments was compiled via an extensive survey of various sources in order to identify and categorise past and present public policy instruments. Second, an exploratory review and classification of the ex post evaluations related to each policy instruments were conducted. Last, an analysis of the ex post evaluations considering the effectiveness and efficiency of the policy instruments was performed.

3.1. First stage: review process for identifying and categorising policy instruments

To identify as many past and present policy instruments for modal shift as possible, searches included both white and grey literature.⁷ There is currently no database, webpage, or other source that comprises information regarding all policy instruments for modal shift in Europe. Furthermore, because most information regarding those policy instruments can be found in grey literature, which is typically difficult to access via databases such as Scopus and Web of Science, it is impossible to apply the same type of search techniques as when conducting systematic literature reviews that include white literature only. For that reason, several search techniques were applied: (1) examining existing policy databases, primarily the European Commission’s (2021b) database for state aid cases and the OECD (2021) database on policy for the environment; (2) searching for policy instruments using Google and Google Scholar and search terms including “policy instrument”, “freight”, “modal shift”, “multimodal transport”, and “intermodal” and more detailed searches considering programmes, transport modes, and countries’ name (as an example: “subsidy, rail, and Italy”, respectively) and (3) implementing snowball techniques, in which citations of included items were followed to identify additional sources of data until data saturation was reached.

Information regarding policy instruments was retrieved from several different sources, including academic studies, websites of governmental institutions, and reports published by public organisations. The most recent searches for policy instruments were conducted in April 2021.

Following an extensive review, all relevant policy instruments were included in a database⁸ based on the following requirements:

• The policy had to be implemented by a public actor in Europe;

• The policy had to target freight’s modal shift from road to rail and/or waterborne transport, as well as clearly focus on reducing freight transport by road (e.g. internalising external costs); and

• The policy had to be implemented or in force after year 2000.

In the database, the policy instruments were sorted and framed based on several features, including the geographical level of the instrument (i.e. regional, national, or local), targeted transport mode (promoting rail, SSS, IWT, and/or discouraging road transport), and the nature of the instrument, through policy category and sub-category of the policy instrument (see Table 1). To that end, the primary- and sub-categories of policy instruments were mainly based on the Swedish Environmental Protection Agency (2021), where policy instruments are classified according to if they are market based (using prices and other market mechanisms) or non-market based (e.g. regulations, informative policies etc.), and on Swedish Environmental Protection Agency (2012) where policies are classified as economic, administrative, informative and research. Using the Swedish Environmental Protection Agency’s (2012; 2021) classification serves as a source for detailed analysis of policy instruments.

Table 1. Primary categories and sub-categories of policy instruments.

Administrative	Economic	Information
Agreement	Fee	Advising
Infrastructure planning	Funding of infrastructure	Development research
Inspection	Grant
Legislation	Subsidy
Limit	Tax
	Tax deduction
	Toll/vignette

3.2. Second stage: examination process for identifying and classifying ex post evaluations according to their characteristics

For each policy instrument included in the database, searches were conducted to identify ex post evaluations of the policy. The searches were conducted in Google and Google Scholar and included the search terms “[name of the specific policy instrument]” in combination with the words “evaluation”, “impact”, and “assessment” in each search engine. Some evaluations had already been identified when the instruments were searched for.

All types of studies and documents (i.e. grey and white literature) attempting to evaluate the performance of an already implemented policy instrument were included, not only ones that meet the European Commission’s (2017a) standards for classifying as an evaluation. The most recent searches were conducted in April 2021.

Based on the identified evaluation papers, this study has analysed and categorised them according to the quality criteria presented in Section 2.1. Those criteria included the following information:

• Actor performing the evaluation;

• Purpose of the evaluation;

• Performance criteria considered by the evaluation (i.e. relevance, coherence, effectiveness, efficiency, and EU added value); and

• Methodology used to evaluate performance.

Some policy instruments had several targets, not only freight’s modal shift. Therefore, this research distinguishes performance criteria considered by the evaluation as a whole and then, from criteria for evaluating the modal shift performance in particular.

3.3. Third stage: criteria to evaluate policy performance based on effectiveness and efficiency

A policy instrument’s performance can be described according to several criteria, including relevance, coherence, effectiveness, efficiency, impact, sustainability and EU-added value (European Commission, 2017a; OECD/DAC Network on Development Evaluation, 2019). In this study, the focus is on identifying the effectiveness and efficiency of the policy instruments. Therefore, this information was qualitatively summarised in the database for each of the identified evaluations.

The definitions of effectiveness and efficiency formulated by the European Commission (2017a) and the OECD/DAC Network on Development Evaluation (2019) were used as guidelines when gathering information from the evaluations. On the one hand, the performance criterion of effectiveness indicates the progress made towards reaching the policy instrument’s objectives. It includes examining the quantitative and qualitative effects of the policy instrument, as well as looking for evidence of why, whether, and how the observed changes are linked to it (European Commission, 2017a). On the other hand, according to the European Commission (2017a), the performance criterion efficiency requires investigating the costs and benefits of the policy, as well as how they accrue to different stakeholders. The OECD/DAC Network on Development Evaluation (2019) defines the efficiency criterion as “a measure of how economically resources/inputs (funds, expertise, time, etc.) are converted to results”.

Overall, this study primarily considered policy instruments with freight’s modal shift (and resulting reduced externalities) as their primary target. Furthermore, policy instruments with a clear purpose of reducing road freight transportation and can lead to a modal shift, including the internalisation of road transport’s external costs or the funding of rail infrastructure, were also included as long as the instrument considers modal shift to be a clear target. However, when the target is undefined or unclear, to avoid including misleading instruments, in this analysis, we excluded policy instruments that have not targeted a modal shift but that may, in fact, indirectly contribute to such a shift or other effects (e.g. road safety). For example, a CO₂ tax on fuel might lead to costs in the road transport sector that exceed the costs of other modes of transport and indirectly lead to improve road safety or modal shift.

Last, the primary objective of this research was to cover all policy instruments for modal shift of freight transport and evaluations in Europe. Thus, this analysis may have been limited by the fact that the database can present some bias due towards non-English policy instruments evaluation have been excluded from the analysis, as well as a bias towards economic and administrative policy instruments at a regional or national level, which are often more well-documented than ones at the local level. However, we do not expect that limitation to affect the significance of the public policy instruments identified.

4. Results

This section presents the major results of our study in light of the three research questions addressed by the analysis. The section first outlines the findings of the identification and classification of past and present public policy instrument for modal shift in Europe. Second, it presents the results of the analysis of the performance of ex post evaluations. Last, the section focuses on the major results concerning the effectiveness and efficiency of the policy instruments in promoting modal shift.

4.1. Identified policy instruments

The search process resulted in the identification of 93 past and present public policy instruments for modal shift,⁹ sorted according to following categories.

First, the policy instruments were analysed in relation to their geographical level of application. In that regard, 27 public policy instruments were identified at the regional level (i.e. implemented by the EU or specific collaborations between countries), 53 were identified at the national level and 13 were identified at the local level.

Second, the policy instruments were sorted according to what modes of transport they aim to promote (see Figure 1). On that count, 35% focus on promoting only railway, 24% on promoting waterborne transport, 22% on promoting the use of both rail and waterborne transport, and 19% on indirectly promoting a modal shift by discouraging road transportation.

Figure 1. Targeted transport mode by public policy instrument (%).

Last, the identified policy instruments were categorised into three primary policy categories according to their nature – administrative, economic, and informative – as well as 14 sub-categories (i.e. agreement, infrastructure planning, inspection, legislation, limits, fees, funding of infrastructure, grant, subsidy, taxes, tax deductions, toll/vignette, advising, and development research), as presented in Table 2. The categorisation of policy instruments according to policy groups and sub-groups was based on Swedish Environmental Protection Agency (2012; 2021). Sorted according to primary categories, most of the identified policy instruments were economic (70%), followed by administrative (21%), and informative (9%). The most common economic sub-categories were grants and subsidies including the EU’s Marco Polo Programs, Great Britain’s Mode Shift Revenue Support and Waterborne Freight Grant, and the Ecobonus systems implemented in, for example, Italy and Sweden. The most common administrative sub-category was legislation, specifically EU directives and specific regulations. Last, development research was the most common sub-category of informative policy, including the Shift2Rail Joint Undertaking.

Table 2. Primary categories and sub-categories of public policy instruments (%).

Administrative	21%	Economic	70%	Information	9%
Legislation	12%	Grant	35%	Development research	6%
Infrastructure planning	5%	Subsidy	19%	Advising	2%
Limit	2%	Toll/vignette	8%
Agreement	1%	Funding of infrastructure	4%
Inspection	1%	Fee	1%
		Tax	1%
		Tax deduction	1%

4.2. Evaluation of the performance of ex post evaluation studies

Based on the public policy instruments identified and included in the database, searches were conducted to identify ex post evaluations of the policies. The process resulted in the identification of publicly available evaluations for 20 of the 93 policy instruments for modal shift.¹⁰ Table 3 presents the policy instruments with ex post evaluations considered in our research. Because some instruments have been evaluated more than once and because some evaluations have considered multiple instruments, the number of evaluations is not the same as the number of policy instruments evaluated. As shown in Table 3, approximately half (i.e. 11) of the evaluated policy instruments have been implemented at EU level and the other half at the national level (i.e. 8) or local level (i.e. 1). Only economic policy instruments have been covered by evaluations at the national and local levels, as all those evaluations assess either subsidies or grants. However, the evaluated policy instruments at the EU level have covered all three primary categories: economic, administrative, and informative. Despite a comprehensive review of 93 studies, we have found only publicly accessible ex post evaluations for 20 of the policy instruments.

Table 3. - Evaluated policy instruments.

Name of the Public Policy Instrument	Region / country	Promotion of transport mode	Primary category (sub-category)	References regarding policy performance
Connecting Europe Facility (CEF)	European Union	Rail and Water	Economic (funding of infrastructure)	(European Commission, 2018)
Directive 1992/62 and 2011/76/EU – Eurovignette	European Union	Road discourage	Administrative (legislation)	(European Commission, 2013)
Directive 92/106/EEC – Combined Transport of goods between Member States	European Union	Rail and Water	Administrative (legislation)	(European Commission, 2016a, 2016b)
EU Regulation 561/2006 – Rest periods on rolling/floating roads and social legislation relating to road transport	European Union	Rail and Water	Administrative (legislation)	(Windisch et al., 2016)
EU Regulation 913/2010 – European rail network for competitive freight	European Union	Rail	Administrative (legislation)	(European Commission, 2016c)
European Shortsea Network – (Evaluation for the Norwegian Short Sea Promotion Centre)	European Union	Water	Information (development research)	(Askildsen, 2005)
Marco Polo I and II	European Union	Rail and Water	Economic (grant)	(Europe Economics, 2011; European Court of Auditors., 2013; Innovation and Networks Executive Agency (INEA), 2020)
Motorways of the Sea	European Union	Water	Economic (grant)	(ICF et al., 2017)
NAIADES – Navigation and Inland Waterway Action and Development in Europe	European Union	Water	Administrative (infrastructure planning)	(European Commission, 2011a; European Court of Auditors, 2015)
National Aid – “The Mode Shift Revenue Support – MSRS”	Great Britain	Rail and Water	Economic (grant)	(Department for Transport, 2020, 2014).
National Aid – “The Waterborne Freight Grant Scheme”	Great Britain	Water	Economic (grant)	(Department for Transport, 2014, 2020; European Commission, 2020a)
National Aid – Freight Facilities Grant – FFG	Great Britain	Rail and Water	Economic (grant)	(Woodburn, 2007)
Shift2Rail	European Union	Rail	Information (development research)	(Fontanel et al., 2017)
State aid to transfer goods to rail – the Province of Emilia Romagna	Italy (Emilia Romagna Region)	Rail	Economic (subsidy)	(European Commission, 2014a, 2019a).
State aid to transfer goods from road to rail “Ferrobonus”	Italy	Rail	Economic (subsidy)	(European Commission, 2020b, 2016d, 2011b)
State Aid – to transfer goods from road to rail “Nuovo Ferrobonus”	Italy	Rail	Economic (subsidy)	(European Commission, 2019b; Marzano et al., 2018)
State Aid – to transfer goods from road to water “Ecobonus”	Italy	Water	Economic (subsidy)	(European Commission, 2012; RAM, 2019; Tsamboulas et al., 2015)
State Aid – to transfer goods from road to rail “Miljökompensation”	Sweden	Rail	Economic (grant)	(Swedish Transport Administration, 2020)
State Aid – Financial support for rail operations	Austria	Rail	Economic (grant)	(European Commission, 2017b)
Trans European Transport Network (TEN-T)	European Union	Rail and Water	Economic (funding of infrastructure)	(European Commission, 2020c; Steer Davis Gleave, 2011)

To investigate the identified policy evaluations, we analysed them according to a quality criterion encompassing four factors: actors, purpose, performance of the evaluation and methodology.

4.2.1 Actors performing the evaluations

At the EU level, the evaluations have been performed by evaluators such as the European Court of Auditors, different consultancy firms, and/or expert groups. Types of evaluators have also varied at the national and local levels. For example, the Swedish Transport Administration is both the administrating organisation and the evaluator of the scheme on environmental compensation for rail freight transport in Sweden (“Miljökompensation för järnväg”); private consultancy firms have evaluated the Mode Shift Revenue Support and the Waterborne Freight Grant in Great Britain; and an independent researcher has evaluated the Freights Facilities Grant in Great Britain. In cases in which the original evaluation report could not be found, information regarding policy performance was gathered from the European Commission’s decision letters regarding the prolongation of the policy instrument. Unfortunately, those decision letters sometimes lack information regarding the evaluating actor(s).

While the limitation of the data on the sample may have imposed some bias in our study, the potential relationship between evaluators and described policy performance suggest a mix of positive and negative policy performance by all types of evaluators. Thus, we can neither confirm nor reject Colin et al.’s (2021) findings, which suggest that evaluations performed by private consultancy firms often generate a more positive description of policy performance than evaluations performed by other types of evaluators.

4.2.2 Purpose of the evaluations

It is important to understand why some policy instruments are evaluated and others not, for the results from the evaluations can paint a biased picture of the effectiveness of freight policy instruments for modal shift. Several purposes are mentioned in the evaluations, ranging from legal requirements and the prolongation of policy instruments to understanding performance and providing recommendations for further improvement.

Article 318 of the Treaty on the Functioning of the European Union (TFEU) includes a commitment to evaluation, which may explain why EU policy instruments have been evaluated to a greater extent than other policy instruments (European Commission, 2017a). Furthermore, several guidelines and frameworks circulate within the EU, including Better Regulation Guidelines and the Regulatory Fitness and Performance (REFIT) programme. In several cases, the legal framework of a policy instrument specifies that the instrument should be evaluated after a certain amount of time. Cases in point include the Connecting Europe Facility (CEF), EU Regulation 913/2010 regarding a European rail network for competitive freight, and the Shift2Rail Joint Undertaking.

Because grants and subsidies at the national level are classified as state aid, the member states need permission from the European Commission to implement and continue such programmes (European Commission, 2014b). Therefore, the evaluations of such policy instruments have in most cases been performed when the various member states have applied for permission by the European Commission to prolong the aid scheme. However, other purposes are also mentioned in the evaluations, including evaluating the performance of the policy, suggesting improvements, and revising grant or subsidy levels.

In this study, we observed that the evaluations performed to prolong subsidies or grants at the national or local level generally describe the overall positive performance of policy instruments. There are several possible explanations for that result. It could simply reflect that subsidies and grants are effective in achieving freight’s modal shift and reducing negative externalities, as suggested by Pinchasik et al. (2020) and Santos et al. (2015). However, it could also be a result of member states’ desire to prolong policy instruments only if they already believe that the instruments are effective or will be effective in the future. Furthermore, if the likelihood of being allowed to prolong the policy instrument is higher with a positive evaluation, then it could provide incentives to describe a more positive performance, which highlights the importance of independent evaluators. Thus, evaluations of policy instruments that have been prolonged may demonstrate better performance and successful results than evaluations with another resolve. However, it would also be reasonable to evaluate inefficient policy instruments, for incentives would then exist to evaluate whether the policy can be improved or should be discontinued.

4.2.3 Performance criteria considered by the evaluation

The evaluations at the EU level have most commonly followed the “Better Regulation Guidelines” (the European Commission, 2017a) when it comes to performance criteria evaluated (i.e. relevance, coherence, effectiveness, efficiency, and EU added value). However, those criteria can be interpreted differently in the evaluations depending on, for example, the type of policy instrument, the type of evaluation, available data, and when the evaluation was performed. For example, in the evaluations of Shift2Rail and CEF, management “efficiency” is discussed under the performance criterion of efficiency, not the resulting costs and benefits for the society generated by the policy instrument. In several evaluations, the amount of funding allocated to different actions is discussed under the performance criterion of “effectiveness”, but there is a lack of discussion regarding whether the funds have contributed to achieving policy targets. Modal shift, along with associated reductions in external costs, is not the chief objective of several EU policy instruments. Therefore, some of the evaluations discuss the different performance criteria thoroughly in relation to the specific policy instrument’s overarching objectives but only briefly discuss the performance criteria in terms of modal shift.

Policy evaluations at the national and local levels vary slightly more in the performance criteria evaluated. All evaluations address effectiveness to some extent, and most also discuss the relevance of the policy instrument. Of the nine policy evaluations at the national or local level, six estimate the policy’s costs and/or benefits for society. Coherence with other policy instruments is not commonly discussed in the evaluations aside from coherence with the EU internal market, which is a requirement for policy instruments classified as state aid.

4.2.4 Methodologies of evaluating performance¹¹

To the extent that evaluations exist, they differ in methodology, quality, and performance criteria evaluated, which confirms the findings from, for example, Cooksy and Caracelli (2005), Harmelink et al. (2008) and Huitema et al. (2011). Of the 20 evaluations, 10 used both qualitative and quantitative methods, six used only qualitative methods, and four used only quantitative methods. Within each of those categories, a wide range of more specific methodologies have been applied. For example, qualitative methods of evaluation range from analysing policy and strategy documents to targeted stakeholder consultations and on-the-spot audits. By comparison, the quantitative approaches include, for example, the analysis of trends in freight traffic over time, comparisons of expected and achieved modal shift, and cost–benefit analysis.

According to Crabb and Leroy (2012) and the European Commission (2017a), difficulties for evaluators relate to measuring causality between a policy instrument and observed changes, as well as finding relevant and reliable data. For instance, most EU evaluations that have lacked quantitative analysis explain why effectiveness and efficiency have been estimated only qualitatively and not quantitatively.

4.3. Effectiveness and efficiency of policy instruments for modal shift

In this section, the most important findings regarding objectives and targets, effectiveness, and efficiency of the evaluated policy instruments are discussed.¹²

4.3.1 Objectives/targets

Although well-defined objectives and targets are important for assessing the effectiveness and efficiency of a policy, determining performance based solely on targeted achievement can be problematic. A good example is the series of evaluations of the Marco Polo Programmes (2003–2013), which provided grants to achieve a modal shift to rail and waterborne transport (European Court of Auditors., 2013). In terms of target fulfilment, both Marco Polo programmes resulted in a modal shift far below expected levels. Such underachievement is central to discussions regarding effectiveness in two evaluations of the Marco Polo Programmes (Europe Economics, 2011; European Court of Auditors, 2013). However, the European Commission has argued that the objectives were ambitious and that determining the effectiveness based on target fulfilment may have led to the underestimation of the programme’s effectiveness (European Court of Auditors, 2013). It has also been argued that the performance of the Marco Polo Programmes should be viewed in the light of the economic crisis that they coincided with. In a third evaluation, the INEA (2020) estimated the environmental benefits, including air quality, noise, climate change, accidents, and congestion, of Marco Polo II as being €2.9–€3.1 for every euro spent and the elimination of 3.5 billion tonnes of CO₂ emissions. Thus, the described performance of the Marco Polo programmes varies depending on the focus of the evaluations.

4.3.2 Effectiveness

For several of the policy instruments identified, modal shift is considered to be an objective in itself, not a means of achieving reduced external costs. Furthermore, several of the evaluations focus exclusively on the modal shift achieved and do not evaluate the effect on externalities. That limitation could be problematic, because external costs are not necessarily lower for waterborne transport than for road transport (Svindland & Hjelle, 2019).

The performance criterion of effectiveness should be analysed in terms of the progress made towards reaching the policy instrument’s objectives. However, in accordance with Harmelink et al. (2008), we found a lack of well-defined objectives and measurable targets for the policy instruments identified. Furthermore, there is a lack of quantitative evaluations that assess the results achieved in light of the objectives and well-defined targets of the policy, as in several EU evaluations of for example the Trans-European Transport Network (TEN-T), Connecting Europe Facility (CEF), Navigation and Inland Waterway Action and Development in Europe (NAIADES), among others.

4.3.3. Efficiency

The performance criterion efficiency requires investigating the costs and benefits of the policy, as well as how they accrue to different stakeholders (European Commission, 2017a).

Several evaluations of EU policy instruments describe poor or mixed performance in terms of achieving modal shift and reaching the desired outputs.¹³ However, some of those policy instruments do not have modal shift as a primary target and focus on other objectives instead.

Other evaluations argue that different enforcements in the member states have led to inefficiency.¹⁴ For example, the Eurovignette Directive is implemented with different charging systems, technologies, and price signals across the EU, which imposes unnecessarily high administrative costs on hauliers (European Commission, 2013).

The evaluations of the EU funding programmes CEF and TEN-T argue that the policy instruments effectively contribute to modal shift by directing funding to, for example, rail and IWT. However, Steer Davies Gleave (2011) has discussed how a lack of TEN-T investment in projects focusing on multimodality has caused several projects to not meet their full potential due to a lack of investment in other parts of the transport system. Furthermore, in the evaluation of the Shift2Rail Joint Undertaking, it is argued that when all rail research is organised by the rail sector, most focus falls to rail and less to multimodal solutions and innovation.

Last, most evaluations for subsidies and grants at the national and local levels describe the positive performance of the policy instruments, especially ones favouring rail transport over waterborne transport. For example, evaluations of subsidy and grant systems favouring rail freight in Austria and Great Britain estimate benefit-to-cost ratios of between 3.39:1 and 4.27:1 (i.e. including reduced negative externalities from road transport) and have both been effective in achieving freight’s modal shift. Moreover, the evaluations of different grants/subsidies in Italy show that the policy instruments have increased the transport of freight by rail. One exception is the Swedish aid scheme for rail transport, which is paid retroactively to operators that perform or organise freight transport services on the Swedish railway network. According to the evaluation, the policy has more accurately prevented a modal backshift from rail to road than promoted any real modal shift to rail. The policy instrument is criticised in the evaluation for lacking continuity, predictability, and a long-term perspective, as well as for including all freight transport on rail, which has resulted in about 22% of the total funds in 2018 and 2019 going to LKAB, a mining company, for the transport of ore, an industry in which rail is already the dominant mode of transport.

A lack of applications is described as an important problem for several policy instruments, especially for waterborne transport, at both the national level (e.g. the Mode Shift Revenue Support for bulk and waterways and the Waterborne Freight Grant in Great Britain) and at the EU level (e.g the Marco Polo Programmes, Motorways of the Sea, and NAIADES). Several evaluations have found that the lack of applications partly derives from long, complicated processes and heavy administrative procedure of applying for financial support. The evaluation of the Mode Shift Revenue Support in Great Britain mentions that the application process for grants to rail services (i.e. intermodal) is easier than the process for applying to a grant for waterborne transport (Department for Transport, 2014). That difference is explained by the intermodal rail grant’s being standardised, whereas the waterborne grant level is decided case by case, which can partly explain the low number of applications. However, the low number of applications might also arguably reflect difficulties in moving freight by inland waterways in Great Britain.

5. Discussion

This section discusses the main implications of the results, considering the identification of policy instruments, their evaluation performance, and their effectiveness and efficiency.

This research shows that national governments tend to be more proactive in applying public policy instruments to enhance the modal shift of freight transport in Europe. Thus, member states, mainly via economic instruments such as State aid, facilitate the execution of these initiatives through subsidies like the Ecobonus systems implemented in Sweden or grants like the Waterborne Freight Grant Scheme implemented in Great Britain. The results also indicate that the most common policy instrument implemented in Europe is focused on promoting the use of a modal shift to one specific mode, primarily rail infrastructure, followed by inland waterborne transport. As already mentioned in previous studies (Bickford et al., 2014; Nealer et al., 2012), shifting freight transport to rail and waterborne transport decreases negative externalities from road transport.

It is interesting to observe that this result is connected with the assessment of evaluation studies. Economic policy instruments have been evaluated at the national and local levels, as such evaluations assess either subsidies or grants. Most of these evaluations describe the positive performance of the policy instruments, especially ones favouring rail transport over waterborne transport.

To that end, this study identifies 20 publicly available evaluations of the 93 policy instruments considered in this research. This unbalance between policy instruments and evaluations may suggest that public institutions pursue a clear objective to shift freight transport from road to rail or waterborne transport but are less familiar with estimating the effect of such instruments after implementation. In fact, the performance indicators suggest interesting results.

First, in the case of evaluators, our results show a mix of performance evaluation studies with a wide range of assessors (including consultancy firms, public authorities, the European Court of Auditors, expert groups, and independent researchers). For that reason, interpreting how evaluating actors may influence evaluation methods and/or results remains unclear. This conclusion differs from the result suggested by Colin et al. (2021), where private evaluators were more favourable to positive policy performance compared to evaluations conducted by others.

Second, in the case of purpose, the evaluations are very limited, focusing on a range of legal requirements or, as suggested by Pinchasik et al. (2020) and Santos et al. (2015), used as a decision tool for prolonged effective policies.

Third, regarding performance criteria, policy evaluations follow the framework developed by the “Better Regulation Guidelines” (European Commission, 2017a). However, in practice, the differences between policy instruments, such as the available data, the time of the evaluation, and the type of policy and evaluation, make their comparison and performance assessment very difficult. The variation in policy evaluations and the difficulty of comparing their results suggest that the ways in which different performance criteria should be measured and interpreted in the evaluations needs clarification in the European Commission’s Better Regulation Guidelines in order to further harmonise policy evaluations. Moreover, common guidelines for evaluations at the national and local levels would be desirable to facilitate comparisons of policy instruments between countries.

Finally, this study suggests that the most common methodology used in evaluation studies is a mix between qualitative (e.g. interviews, surveys, case studies, questionnaires, and stakeholder consultations) and quantitative methods (e.g. data analysis, estimation analysis, and cost–benefit calculation). Several evaluations mention difficulties with finding relevant and reliable data, as well as with measuring causality between the policy instruments and observed changes. Our results suggest that one way to overcome the problem is to design policy instruments that facilitate evaluation – for example, by requiring firms that receive funding to collect and present data.

Last, some other implications of the results are related to the effectiveness and efficiency of these policy instruments in achieving a modal shift of freight transport.

In the case of effectiveness, i.e. the progress made towards reaching the policy instrument’s objectives, our analysis suggests a lack of well-defined objectives and measurable targets for the policy instruments identified, which creates difficulties in achieving objectives and evaluating the policy instruments’ performance. This result is in line with Harmelink et al. (2008). Furthermore, for several policy instruments, modal shift is considered to be an objective in itself, not a means to achieve reduced external costs from freight transport. That understanding is problematic because modal shift does not automatically result in reduced externalities. Thus, when formulating policy targets, it is important to treat modal shift as a means to reach the ultimate objective of reducing external costs, including GHG emissions. Moreover, the objectives for the policy instruments are often broad and general, which will complicate the necessary action of drastically reducing GHG emissions over the coming years, which makes knowledge of the effectiveness of policy instruments exceedingly valuable if climate targets are to be reached. The lack of quantitative evaluations for the policy instruments may well follow from the lack of quantitative targets to start with, which would further strengthen the argument for setting measurable targets from the start of the process. Thus, it is important that objectives and targets for policy instruments are well-defined and formulated in ways that facilitate evaluation as well as the achievement of targets.

Regarding efficiency, i.e. the analysis of the costs and benefits of the policy, as well as how they accrue to different stakeholders (the European Commission, 2017a). From a regional perspective, EU policy instruments with different enforcements in the member states, for example, the Eurovignette Directive with different charging systems, technologies, and price signals across the EU, have led to inefficiency in their implementation. Consequently, this imposes unnecessary high administrative costs on hauliers (European Commission, 2013). At the national and local levels, most evaluations for subsidies and grants highlight the policy instruments’ positive performance, especially policy instruments favouring rail transport over waterborne transport.

Last, approximately half of the policy instruments evaluated have been subsidies or grants at the national level, whereas the other half have been administered at EU level. On the one hand, most of the evaluations of subsidies and grants at national/local levels have been performed when applying for permission from the European Commission to prolong state aid measures. Those evaluations show overall positive policy performance, which may have several explanations, including that the policies are effective in achieving modal shift. Otherwise, it may also reflect the fact that member states apply only for prolongation if the policy instrument is considered to be effective. On the other hand, the large share of EU policy instruments evaluated might be due to the commitment to evaluation formulated in Article 318 of the Treaty on the Functioning of the European Union and on other evaluation guidelines and frameworks in the EU (e.g. the REFIT programme).

Finally, commonly mentioned factors of the underachievement of policy instruments are problems related to the outreach of the policy, the lack of applications, long and complicated application processes, and a high administrative burden for the companies applying for financial support. Thus, a focus on better outreach and simpler application processes could be one way forward to improving policy instruments for modal shift in Europe.

6. Conclusions

The objective of our study was to identify and classify past and present public policy instruments, as well as to review their performance, and to estimate their effectiveness and efficiency in achieving modal shift of freight transport in Europe. To that purpose, our research identified three main outcomes.

First, we aimed to identify what policy instruments have been implemented in Europe with the aim to achieve modal shift. Through a comprehensive literature review, we identified 93 public policy instruments targeting modal shift in Europe. Most of those identified instruments are subsidies or grants implemented at the national level that target a specific mode of transport, most often rail.

Second, we analysed which policy instruments have been evaluated and which evaluation methods and performance indicators that have been applied. Searches in both grey and white literature resulted in the identification of ex post evaluations for 20 of the 93 identified policy instruments. Most of those evaluations consider economic policy instruments, and specifically subsidies and grants at the national or local level.

Finally, we have some conclusions regarding the effectiveness and efficiency of European policy instruments in terms of achieving modal shift and reducing negative externalities. Several evaluations of EU policy instruments describe the poor or mixed performance of the policy instruments, while the performance of subsidies and grants at the national level are often considered to be positive in the evaluations. In general, there seems to be a more positive performance of policy instruments promoting modal shift to rail than to waterborne transport.

This study aims to contribute to the literature by providing valuable information to policymakers and improving knowledge on implementing public policy instruments to achieve the modal shift to freight transportation in Europe. In this regard, well-defined targets and simpler application processes seem to be good recommendations for improving the performance of modal shift policy instruments in Europe.

The limitations of our analysis include the scarcity of data on ex post evaluation studies and the potential bias resulting from the limited evaluations available in English. Further research could fruitfully examine the influence of additional public policy evaluations, such as ex ante studies, to expand the analysis and evaluate their success by comparing the expected effect (ex ante) with the outcome of the implemented policy (ex post). Similarly, future studies should examine the dynamic relationship between evidence-based policy and decision-makers in greater depth in order to facilitate comparisons and improve policy performance.

Acknowledgements

The authors would like to thank all project members of the MOdal Shift for an Environmental Lift (MOSEL) project for their great efforts in making this publication possible. This paper is based on a chapter of the dissertation of Johanna Takman: “Reducing greenhouse gas emissions – Examples from the freight transport sector: Essays on economic growth, public policy instruments, and renewable energy”. Published by Chalmers University of Technology, October 2021.

Disclosure statement

No potential conflict of interest was reported by the author(s).

Additional information

Funding

This work was supported by Naturvårdsverlet (Grant number: 802-0104-17).

Notes

1 Short Sea Shipping and Inland Waterways.

2 For example, Svindland and Hjelle (2019) found that a relatively high-capacity utilisation is needed for maritime transport to be more CO₂ efficient than road transport.

3 Including accidents, air pollution, climate, noise, congestion, and habitat damage (CE Delft, 2020).

4 Public policy instruments refer to political tools employed to correct for market failures and to reach societal objectives, such as a modal shift to reduce the negative externalities from road freight transports.

5 Several such policies and targets are driven by the European Commission’s “Transport White Paper”, which specifies a modal shift of 30% for long-distance road freight transport (i.e., >300 km) by 2030 and one of more than 50% by 2050 (European Commission, 2021a; Pinchasik et al., 2020).

6 Combined transport refers to intermodal transports where the road leg of the transport is as short as possible (European Commission, 2016a).

7 Grey literature refers to unpublished research and publicly available open-source information, usually available outside of traditional publishing and distribution channels. It may for example include government reports, newsletters, working papers etc.

8 The database is available from the authors upon request.

9 A full list of the identified policy instruments is available in the supplementary material.

10 A full list of the evaluated policy instruments and further characteristics is available in the Supplementary Material.

11 Table 2 in the Supplementary Material provides detailed information on the methodology applied.

12 Further details regarding the policy instruments and their evaluations are available in the Supplementary Material (Table 3).

13 Some examples are the Marco Polo Programmes, Motorways of the Sea, NAIADES, TEN-T and the Eurovignette Directive

14 Such as the Eurovignette Directive, Combined Transport Directive, EU Regulation 561/2006 regarding rest periods on rolling/floating roads.