ISSUE 3/2021

An integrated application of AHP and PROMETHEE in decision making for landscape management

Article by Milena D. Lakićević, Keith M. Reynolds, Beata J. Gawryszewska | 07.10.2021 - 16:30

Schlüsselbegriffe: multikriterielle Entscheidungsanalyse, Managementplan, Programmiersprache R, Nationalpark

Abstract

This paper presents an example of an integrated application of two commonly used multi-criteria decision-making methods in landscape management, AHP and PROMETHEE, in order to select one of four management alternatives as the preferred alternative for management of the Fruska Gora National park in Serbia. AHP was used to derive the weights on five decision criteria, while PROMETHEE was used to assess the performance of four alternative management plans with respect to the five decision criteria. The five decision criteria were „promoting biodiversity”, „wilderness”, „tourism”, „education”, and „use of resources”. The four management alternatives included:

1) maintain current management practices (business as usual),

2) develop eco-tourism,

3) protect natural ecosystems, and 

4) provide sustained use of natural resources. 

After weighting the decision criteria and evaluating the performance of the alternatives against them, the second alternative (develop eco-tourism) was selected as providing the best performance across all criteria. The proposed decision-making scheme is especially useful with qualitative criteria and can be applied to diverse problems in landscape management.

Zusammenfassung

Diese Publikation präsentiert ein Beispiel für eine integrierte Anwendung zweier häufig verwendeter multikriterieller Entscheidungsfindungsmethoden im Landschaftsmanagement, AHP und PROMETHEE. Das Ziel ist eine von vier Managementalternativen als bevorzugte Alternative für das Management des Fruska Gora Nationalparks in Serbien zu identifizieren. AHP wurde verwendet, um die Gewichtungen der fünf Entscheidungskriterien abzuleiten, während PROMETHEE benutzt wurde, um die Performanz der vier Managementalternativen hinsichtlich der fünf Entscheidungskriterien einzuschätzen. Bei den fünf Entscheidungskriterien handelte es sich um „Förderung von Biodiversität”, „Wildnis”, „Tourismus”, „Bildung” und „Ressourcennutzung”. Die vier Managementalternativen umfassen:

1) die derzeitigen Managementpraktiken zu behalten („business as usual“), 

2) Ökotourismus entwickeln, 

3) natürliche Ökosysteme schützen und 

4) nachhaltige Ressourcennutzung. 

Nach Gewichtung der Entscheidungskriterien und Bewertung der Managementalternativen anhand der Entscheidungskriterien wurde die zweite Alternative (Ökotourismus entwickeln) ausgewählt, da sie die beste Leistung über alle Kriterien lieferte. Das hier vorgeschlagene Entscheidungsfindungschema erscheint nützlich für qualitative Kriterien und kann für verschiedene Probleme im Landschaftsmanagement angewendet werden.

1. Introduction

Multi-criteria decision analysis (MCDA) is widely used in landscape management and planning (Lakicevic et al., 2018). The basic methods developed for MCDA have a decade-long history, but new tools and variants on the original methods continue to emerge up to the present. A recent review reports over 100 distinct MCDA tools and methods (Tomashevskii and Tomashevskii, 2019). However, the diversity of available methods also poses challenges for potential users to select a particular tool and method that fits the particular problem of the user. Different MCDA alternative tools and methods can yield different results (Wątróbski et al., 2019), so selecting the most appropriate one for a particular application can be challenging. In decision science research, there has been the tendency to combine applications of different multi-criteria analysis methods in the same decision-making problem to improve the flexibility of the decision process and tailor it more closely to the preferences of decision makers (Prakash and Barua, 2016; Hanne, 2001). For example, the A’WOT model (Kajanus et al., 2004) combines the AHP method and SWOT analysis, Kaya and Kahraman (2011) proposed a scheme for combining the AHP and VIKOR in forestry decision-making, and Srdjevic et al. (2013) proposed combining AHP with the Consensus Convergence Model (Regan et al., 2006) in urban landscape management. The main idea behind integrating two or more methods is to reduce shortcomings in each of the methods in cases when they are being applied individually (Lakicevic et al., 2014). While combining different MCDA methods seems to be a positive development, it also further challenges users of MCDA systems, because now they are not only confronted by basic choices among multiple alternative methods, but now also combinations of methods.

As a contribution to the literature on hybrid MCDA methods (Kangas et al., 2008), this paper presents an example of combining the AHP and PROMETHEE methods in a landscape management problem. Saaty’s AHP method (Saaty, 1992; 1994) has been very popular since the 1980s. In the classical application of AHP, the pair-wise comparison process extends to deriving the weights on the alternatives. However, when pair-wise comparisons are performed at the alternative level of the AHP model, there is the possibility of a rank-reversal phenomenon, in which the addition or subtraction of alternatives considered results in a re-ordering of the relative AHP scores, and this phenomenon was the subject of intense academic debates for some years (see, for example, Wang and Elhag, 2006). In part as a result of the rank-reversal controversy, it has been common since the mid-1990s to combine AHP methods for deriving weights on criteria with other methods for deriving weights on alternatives. For example, Reynolds (2001) has described combining the AHP approach with the Simple Multi-Attribute Rating Technique – SMART (Edwards, 1977; Edwards and Newman, 1982). In SMART, ratings on alternatives are not derived by pair-wise comparisons among the alternatives, but are instead derived by absolute utility functions, which thereby avoid the rank-reversal problem (Reynolds, 2001). Combining the AHP and PROMETHEE methods may provide another interesting alternative hybrid that scientists may want to consider in future work. Like the AHP-SMART hybrid, the AHP-PROMETHEE hybrid avoids the knotty problem of potential rank-reversal in the classical AHP method by direct input of user preferences of alternatives with respect to criteria. On the other hand, the PROMETHEE method also considers the relative performance of alternatives at the final rankings of alternatives, somewhat analogous to pair-wise comparisons among alternatives in the classical AHP.

In this study, the decision-making problem is to select the most suitable management plan for the Fruska gora national park in Serbia, taking into account several, sometimes conflicting, criteria. All calculations necessary for this research have been performed using the R program, so one of the goals of the paper is to demonstrate use of R and R packages in tasks related to landscape management and planning. 

2. Materials and methods

Lakicevic_Fig1.jpg

Figure 1: Location of Fruska gora national park. / Abbildung 1: Lage des Fruska Gora Nationalparks.

2.1 Study area

Our study area is the Fruska gora national park in the northwest region of Serbia (Figure 1). The national park occupies 25,393 ha and is mainly covered by forest ecosystems (90% of the territory). The most dominant species are Quercus robur L., Quercus virgiliana (Ten.) Ten., Quercus cerris L., which represent remnants of native flora of the area, along with the invasive species Tilia argentea Desf., whose spread threatens the natural forest communities. Aside from value of this region regarding biodiversity and wilderness conservation, the national park provides diverse tourist and educational functions.

Lakicevic_Fig2.jpg

Figure 2: Proposed decision-making framework. / Abbildung 2: Vorgeschlagenes Schema für die Entscheidungsfindung.

The proposed scheme of decision-making (Figure 2) depicts application of the AHP method for evaluating criteria and application of the PROMETHEE method for evaluating alternatives. The final results are obtained by aggregating the AHP weights of criteria and weights of alternatives by standard PROMETHEE procedures. The AHP evaluation of criteria was performed in a group context, and therefore one step includes use of the technique for aggregating individual priorities of criteria – AIP (Forman and Peniwati, 1998). This step is only applicable when making decisions in a group context, and therefore is presented with a dashed outline (Figure 2). The next two subsections explain additional details about the AHP and PROMETEE methods used in the paper.

Lakicevic_Tab1.jpg

Table 1: Saaty’s scale of relative importance. / Tabelle 1: Saatys Skala der relativen Bedeutung.

2.3 Analytic Hierarchy Process – AHP

The AHP method is based on decomposing a decision-making problem into a hierarchy (Saaty, 1992), with the simplest version consisting of three levels: a goal, a set of criteria, and a set of alternatives. In traditional application of the AHP, criteria are first pair-wise compared to each other with respect to satisfying the goal using Saaty’s scale of relative importance (Table 1), and then alternatives are similarly pair-wise compared to each other with respect to each criterion using the same importance scale. In the present study, however, the AHP method is only used to derive weights on the decision criteria, whereas weights on alternatives are derived with the PROMETHEE method. 

When applying the AHP method, a decision maker fills in the upper right triangle of the comparison matrix with the numerical values from Table 1. From the comparison matrix, weights (priorities) of the criteria are derived by the eigenvector method (Saaty, 2003), solving the eigenvalue-eigenvector equation (Eq. 1):

in which: A is a comparison matrix (e.g., Figure 4), λ is the principal eigenvalue of the matrix, and ω is a vector that gives the priority of each criterion. 

In this research, the evaluation was performed in a group context, with four decision makers involved. Calculation of the priority vector, representing the importance of each criterion, for each decision maker was calculated in R, using the package “ahp.” The final group priority vector for criteria was derived by the AIP method described in the next section. 

2.3.1 Aggregation of Individual Priorities

Individual AHP evaluations can be aggregated using the AIP method (Forman and Peniwati, 1998) following Equation 2: 

in which AGi is the final priority, αk is the weight of the kth decision maker, and ai(k) is the individual priority provided by the kth decision maker. In this research, all decision makers were assigned an equal weight (α=0.25).

2.4 PROMETHEE

The PROMETHEE method allows comparison between alternatives using preference functions. A preference function Fi(a, b) represents the intensity of the preference of alternative a over b for a criterion Ci. There are six types of preference functions commonly used in the PROMETHEE method, and are referred to as usual, U shape, V shape, level, V shape with indifference, and Gaussian preference function (Brans et al., 1986). Selection of a certain preference function depends on the type of decision-making problem. In this research, we used the so-called V-shape function, which is suitable when having the set of alternatives assessed on an ordinal scale (Čupić and Suknović, 2010). The alternatives in this paper are assessed using a 1-5 scale in which 1 is the lowest preference value and 5 is the highest value. The V-shape function is given by the formula: 

in which: fi(a) is the preference value of alternative a with respect to criterion I, fi(b) is the preference value of alternative b with respect to the same criterion I, q is a preference threshold (value specifying that one alternative is strictly preferred over the other) and Fi(a, b) is the value of a preference function. Once the value of the preference function has been obtained, the preference index π(a, b) is calculated as:

in which ωi is the weight of criterion i in Eq 1. To obtain the ranking of alternatives, it is necessary to calculate the positive outranking flow (ф+) and the negative outranking flow (ф-) as presented in formulas (5) and (6):

These values represent the degree of dominance of alternative a over the others (ф+), and the degree to which alternative a is dominated by the others (ф-). The final net flow relative to alternative i is calculated as:

The final ranking of alternatives is based on their calculated net flows.

Lakicevic_Tab2.jpg

Table 2: Alternative management plans for the Fruska gora national park. / Tabelle 2: Alternative Managementpläne für den Fruska Gora Nationalpark.

2.5 Management plans and evaluation criteria

In this paper, four alternative management plans for the Fruska gora national park are considered. The plans are based on a previously published study (Lakićević, 2013) and a brief explanation is provided here (Table 2).

The four alternative management plans were evaluated in our proposed multi-criteria decision model (MCDM) framework (Figure 2), taking into account criteria adopted from the IUCN guidelines for managing protected areas (IUCN, 2008) and were defined as biodiversity preservation (C1), wilderness protection (C2), tourism (C3), education (C4) and sustainable use of natural resources (C5).

Lakicevic_Fig3.jpg

Figure 3: Multi-criteria decision model structure for the Fruska gora national park. / Abbildung 3: Struktur des multikriteriellen Entscheidungmodells für den Fruska Gora Nationalpark.

2.6 Structure of the decision problem

The structure of the MCDM problem (Figure 3) has, as its goal, to select the most preferred management plan from among the four proposed plans (Table 2), taking into account the five criteria.

Evaluation of the relative importance of criteria was performed individually by four decision makers with a background in ecology, using the AHP pair-wise comparison process. One of the four decision makers, who worked on developing the alternative management plans for the national park, evaluated the management plans using the PROMETHEE method. For the calculations related to application of the AHP method, we used the “ahp” R package (Glur, 2018), and for the PROMETHEE calculations we used the “promethee” R package (Ishizaka et al., 2018). More detailed explanation on creating input files and performing analysis in the “ahp” R package can be found in a manual (Glur, 2018) or, alternatively, in the paper (Lakićević et al., 2020). A manual for the “promethee” R package is provided in Resce et al. (2019).

3. Results

3.1 AHP evaluation of criteria weights

The four decision makers evaluated the set of five criteria within the AHP framework, and the results of their individual evaluations are presented as four pair-wise comparison matrices (Figure 4).

Lakicevic_Fig4.jpg

Figure 4: Pair-wise comparisons among criteria for each of the four decision makers (DM1 to DM4). / Abbildung 4: Paarweise Vergleiche zwischen den Kriterien für jeden der vier Entscheidungsträger (DM1 bis DM4).

Lakicevic_Tab3.jpg

Table 3: Relative weight of criteria. / Tabelle 3: Relative Gewichtung der Kriterien.

The weight vector, ω, from solution of Eq. 1, is shown for each decision maker (Table 3), giving the relative weight of each criterion for each decision maker. The table presents the relative weight of criteria for each decision maker separately, as well as the aggregated results from applying the AIP method (Eq. 2).

When weights were aggregated over the four decision makers using the AIP aggregation method, the most important criterion was C1 – biodiversity preservation (AIP = 0.263). The aggregated weights were input data for the final decision step.

Lakicevic_Tab4.jpg

Table 4: PROMETHEE evaluation of management plans. / Tabelle 4: PROMETHEE Bewertung der Managementpläne.

3.2 PROMETHEE evaluation of criteria weights

One decision maker evaluated the set of alternatives with respect to each criterion (Table 4). For each criterion, the V-shape function has been applied, and it was defined that each criterion should be maximized. As the procedure for this type of function requires, the decision maker stated the preference threshold (q) for each analyzed criterion. 

Given the aggregated weight vector (Table 3) and preferences (Table 4), and applying Eqs. 3-7, we obtained the net flows (ф), as well as positive and negative outranking flows (ф+ and ф-), and the final ranking of management actions (Table 5).

Lakicevic_Tab5.jpg

Table 5: Phi values and final rank of management plans. /  Tabelle 5: Phi Werte und endgültiger Rang der Managementpläne.

After accounting for the aggregated relative weights from the AHP and AIP process (Table 3) and the preferences in PROMETHEE (Table 4), our final result (Table 5) indicates that the highest ranked alternative is development of eco-tourism and introducing new tourist facilities in well-preserved natural areas (A2). The final value of net flow for this management plan is 0.207, and this alternative has the highest value for the positive outranking flows.

4. Discussion

In this study, we have demonstrated how a collection of R packages are easily invoked in sequence to produce a hybrid MCDM that incorporates a combination of AHP, AIP, and PROMETHEE methods to recommend a management plan for Fruska gora national park. In our research, four decision makers participated in evaluation of criteria set, but for practical application, the methods are simply a prototype that could be extended to a larger group if desired. However, sometimes the small group of DMs are really those that need to be involved, and have the most knowledge about the problem (Lakicevic et al., 2014). There are several papers suggesting the inclusion of well-informed experts, in contrast to large numbers of poorly-informed respondents (Ananda and Herath, 2003; Srdjevic et al., 2019; Beck and Storopoli, 2021).

Aggregation of decision makers’ judgments can be done in a variety of ways. Besides aggregation of individual priorities (AIP), presented in this research, there is also a procedure for aggregating individual judgments, i.e., elements at the same position in a pairwise comparison matrix, by calculating their geometric mean (Aczél and Saaty, 1983). However, aggregation of individual judgements is usually performed in cases when a group is expected to act as a unit (Forman and Peniwati, 1998; Srdjevic et al., 2013), for example for decision makers coming from the same organization. In our research, we have used aggregation of individual priorities, because the decision makers all had a background in ecology but were representatives of different institutions, and therefore were not expected to act as a unit. It should be noted that both procedures (aggregation of individual judgments and aggregation of individual priorities give similar results), so the selection of the procedure does not affect the final results crucially.

The procedure described in this paper can be repeated in different landscape decision-making problems, when these are structured as a three-level hierarchy with goal, criteria and alternatives, and future research should test it on a different example/case study area. Even though it can be applied for any type of a three-level decision problem, it is especially useful when a decision maker performs quantitative assessment of sets of alternatives. 

Calculations needed for this decision-making framework were supported by appropriate R packages, and these were the “ahp” and “promethee”. The packages offer a clear and fast calculation procedure that is supported by high-quality graphical outputs of results. The proposed framework does not require any other R packages for calculation, but it should be noted that R provides support for other MCDM methods. One example is the “mcdm” package that implements RIM, TOPSIS (with two normalization procedures), VIKOR, Multi-MOORA and WASPAS methods (Blanca, 2016).

5. Conclusions

Combining different multi-criteria methods is very popular in recent research. This paper follows this direction, and describes a procedure of integrated application of the AHP and PROMETHEE methods. Application of the PROMETHEE method implies that importance of criteria is known, and does not support the process of criteria importance evaluation and that was the reason to include the AHP method in the first phase of the proposed decision-making scheme. Even though the AHP method could be used for the entire decision-making process, our proposal is to apply the PROMETHEE method for the second phase – assessment of alternatives. The advantage of PROMETHEE, in comparison to the AHP, is in the number of alternatives (here, management plans) that can be conveniently evaluated in a particular decision-making problem. Therefore, as a general recommendation, we have proposed to use the PROMETHEE method for the assessment of alternatives. The proposed scheme of decision making is shown on a case study example of the national park Fruska gora in Serbia. For calculation purposes, we have used the R language. Even though multi-criteria methods were mainly developed in the 1970s, they are still commonly applied nowadays, and the number of the R packages recently developed to support their calculation is a good indicator of their continuing importance in contemporary research.

As we have noted, the R MCDM libraries include a variety of methods. This study has demonstrated the application of one novel MCDM hybrid, but others are certainly possible and worth exploration in the continuing effort to improve on the flexibility of MCDM solutions in decision science. We illustrated application of our hybrid MCDM method in the specific context of an environmental management problem, but our hybrid method is broadly applicable to many other problems that can benefit from the application of decision science. 

Finally, this study reports the results of a research prototype for a novel MCDM model. As such, we only used a small group of decision makers highly familiar with the small study area used in this work. However, acknowledging the well-recongized importance of broad public involvement in successful, contemporary environmental decision making, we close by emphasizing that practical application of the methods presented here to operational decisions should strive for broad public involvement to ensure a successful outcome with communities of interested stakeholders.

Acknowledgements

Part of this research was conducted in the USDA Forest Service in Corvallis, Oregon, USA, during a six month visit of the 1st author, sponsored by the Fulbright program of the U.S. Department of State. Part of this research has been supported by the Ministry of Education, Science and Technological Development of the Republic of Serbia, Contract No. 451-03-9/2021-14/200117.

End notes

The use of trade or firm names in this publication is for reader information and does not imply endorsement by the U.S. Department of Agriculture of any product or service.

References

Aczél, J., Saaty, T.L. 1983. Procedures for synthesizing ratio judgements. Journal of Mathematical Psychology, 27(1), 93-102. doi: 10.1016/0022-2496(83)90028-7 

Ananda, J, Herath, G. 2003. The use of Analytic Hierarchy Process to incorporate stakeholder preferences into regional forest planning. Forest Policy and Economics, 5(1), 13-26. doi: 10.1016/S1389-9341(02)00043-6

Beck, D., Storopoli, J. 2021. Cities through the lens of Stakeholder Theory: A literature review. Cities, 118, 103377. doi: 10.1016/j.cities.2021.103377 

Blanca, A.C.M. 2016. MCDM: Multi-Criteria Decision Making Methods for Crisp Data. R package version 1.2. https://CRAN.R-project.org/package=MCDM. 

Brans, J.P., Vincke, P., Mareschal, B. 1986. How to select and how to rank projects – the PROMETHEE method. European Journal of Operational Research, 24, 228-238. doi: 10.1016/0377-2217(86)90044-5

Čupić, M., Suknović, M. 2010. Decision-making. University of Belgrade, Faculty of Organizational Sciences, Belgrade. In Serbian. 

Edwards, W. 1977. How to use multi-attribute utility measurements for social decision making. IEEE Transactions on Systems, Man, and Cybernetics, 7, 326-340. doi: 10.1109/TSMC.1977.4309720

Edwards, W., Newman, J.R. 1982. Multi-attribute Evaluation. Sage, Beverly Hills, CA.

Forman, E., Peniwati, K. 1998. Aggregating individual judgments and priorities with the analytic hierarchy process. European Journal of Operational Research, 108, 165-169. doi: 10.1016/S0377-2217(97)00244-0

Glur C. 2018. Package 'ahp'. https://cran.r-project.org/web/packages/ahp/ahp.pdf 

Hanne, T. 2001. On the Combination of MCDM Methods. In: Intelligent Strategies for Meta Multiple Criteria Decision Making. International Series in Operations Research & Management Science, vol 33. Springer, Boston, MA. doi: 10.1007/978-1-4615-1595-1_4

Ishizaka, A., Resce, G. Mareschal, B. 2018. Visual management of performance with PROMETHEE productivity analysis. Soft Computing, 22, 7325-7338. doi: 10.1007/s00500-017-2884-0 

IUCN 2008. Guidelines for protected areas management categories, part II. The Management Categories. IUCN, Gland.

Kajanus, M., Kangas, J., Kurttila, M. 2004. The use of value focused thinking and the A’WOT hybrid method in tourism management. Tourism Management, 25, 499-506. doi: 10.1016/S0261-5177(03)00120-1 

Kangas, A., Kangas, J., Kurttila, M. 2008. Decision support for forest management. Springer Netherlands. 

Kaya, T., Kahraman, C. 2011. Fuzzy multiple criteria forestry decision making based on an integrated VIKOR and AHP approach. Expert Systems with Applications, 386, 7326-7333. doi: 10.1016/j.eswa.2010.12.003 

Lakićević, M.D. 2013. Application of the Analytic hierarchy process AHP. and participatory decision-making in management of the National Park “Fruška gora”, PhD thesis, University of Belgrade, Faculty of Forestry, Belgrade. In Serbian.

Lakićević M., Srdjevic B., Marto M. 2020. The package “ahp” of R program for evaluating landscape management plans. Annals of Agronomy, 44(1), 19-25. 

Lakićević, M., Srdjevic, Z., Srdjevic, B., Zlatic, M. 2014. Decision making in urban forestry by using approval voting and multicriteria approval method case study: Zvezdarska forest, Belgrade, Serbia. Urban Forestry & Urban Greening, 131, 114-120. doi: 10.1016/j.ufug.2013.11.001 

Lakićević, M., Srdjevic, B., Velichkov, I. 2018. Combining AHP and Smarter in Forestry Decision Making. Baltic Forestry, 241, 42-49.

Prakash, C., Barua, M.K. 2016. A combined MCDM approach for evaluation and selection of third-party reverse logistics partner for Indian electronics industry. Sustainable Production and Consumption, 7, 66-78. doi: 10.1016/j.spc.2016.04.001 

Regan, H.M., Colyvan, M., Markovchick-Nicholls, L. 2006. A formal model for consensus and negotiation in environmental management. Journal of Environmental Management, 802, 167–176. doi: 10.1016/j.jenvman.2005.09.004 

Resce, G., Tasiou, M., Vidoli, F. 2019. Package 'PROMETHEE', https://cran.r-project.org/web/packages/PROMETHEE/PROMETHEE.pdf 

Reynolds, K.M. 2001. Prioritizing salmon habitat restoration with the AHP, SMART, and uncertain data. In: The analytic hierarchy process in natural resource and environmental decision making. Schmoldt, D.L., Kangas, J., Mendoza, G., and Pesonen, M. eds., Kluwer Academic Publishers, London. doi: 10.1007/978-94-015-9799-9 

Saaty, T.L. 1980. The analytic hierarchy process. McGraw Hill, New York.

Saaty, T.L. 1992. Multicriteria Decision Making: The Analytic Hierarchy Process, RWS Publications, Pittsburgh, PA.

Saaty, T.L. 1994. Fundamentals of Decision Making and Priority Theory with the Analytic Hierarchy Process. RWS Publications, Pittsburgh, PA. doi: 10.1007/978-94-015-9799-9_2 

Saaty, T.L. 2003. Decision-making with the AHP: Why is the principal eigenvector necessary. European Journal of Operational Research, 1451, 85-91. doi: 10.1016/S0377-2217(02)00227-8 

Srdjevic, B., Srdjevic, Z., Lakićević, M. 2019. Urban greening and provisioning of ecosystem services within hesitant decision making framework. Urban Forestry and Urban Greening, 43, 126371. doi: 10.1016/j.ufug.2019.126371 

Srdjevic, Z., Lakićević, M., Srdjevic, B. 2013. Approach of decision making based on the analytic hierarchy process for urban landscape management. Environmental Management, 513, 777-785. doi: 10.1007/s00267-012-9990-7

Tomashevskii, I., Tomashevskii, D. 2019. A nonheuristic multicriteria decision-making method with verifiable accuracy and reliability. Journal of the Operational Research Society, 72(1), 78-92. doi: 10.1080/01605682.2019.1650621 

Wang, Y.M., Elhag, T.M.S. 2006. An approach to avoiding rank reversal in AHP. Decision Support Systems, 423, 1474-1480. doi: 10.1016/j.dss.2005.12.002 

Wątróbski, J., Jankowski, J., Ziemba, P., Karczmarczyk, A., Zioło, M. 2019. Generalised framework for multi-criteria method selection. Omega, 86, 107-124. doi: 10.1016/j.omega.2018.07.004