My part is bigger than yours - assessment within a group of peers using the pairwise comparisons method

Abstract

A project (e.g. writing a collaborative research paper) is often a group effort. At the end, each contributor identifies his or her contribution, often verbally. The reward, however, is quite often financial in nature. This leads to the question of what (percentage) share in the creation of the paper is due to individual authors. Different authors may have various opinions on the matter, and, even worse, their opinions may have different relevance. In this paper, we present a simple models that allows aggregation of experts’ opinions linking the priority of his preference directly to the assessment made by other experts. In this approach, the greater the contribution of a given expert, the greater the importance of his opinion. The presented method can be considered as an attempt to find consensus among a group of peers involved in the same project. Hence, its applications may go beyond the proposed study example of writing a scientific paper.

1 Introduction

From time to time, we observe behind-the-scenes discussions about which author in a scientific paper is the most relevant. The answers can vary. First, last, correspondence, first three authors, it doesn’t matter, etc. At its core, it is a question about the value of the contribution, the effort, the effort that an author has made to the paper. Very often, scientific journals try to help answer this question by making it possible to determine the workload of a given author by describing descriptively what that author did. Such a description makes it possible, to some extent, to form an opinion about the significance of an author’s contribution.
However, there are times when such a description is insufficient and a precise, percentage determination of "contribution to the work" for each author separately is necessary. This is the case when a certain amount of money is awarded for the creation of a work to be distributed among the authors. In such a situation, quite often the "distribution of percentages" is done on a dictatorial basis, with the corresponding author playing the role of dictator. In a situation where the authority of all authors is comparable, or worse, where the corresponding author is a less recognized researcher than some other researchers on the author list, misunderstandings and irritations can easily arise. On the one hand, the correspondent (or lead) author should have the best knowledge of the other authors’ contributions to the work. On the other hand, other authors of greater esteem can easily question his or her decisions.
A way out of this somewhat uncomfortable situation may be to adopt arbitrary regulations for the distribution of the award. E.g. dividing the prize equally between all participants, or setting rigid proportions between the prize for the first and subsequent authors. These types of solutions, unless they arouse personal animosity between the authors, leave a certain feeling of injustice. The question therefore arises as to whether a dictatorial approach on the one hand and the establishment of a rigid framework for the distribution of the prize on the other can be avoided. It seems to us that with a team of peers this is possible.

In this paper we propose an approach based on aggregating the opinions of individual authors/experts using prioritization of opinions. Since team members who were more involved in the project usually have a better understanding of the project and the contribution of other team members to the work done we propose that prioritization should be related to the size of this contribution. In this way, we avoid a situation where those whose participation is small or marginal have the same share in the decision as those whose participation is significant. The latter, by virtue of their significant and large contribution, are given greater priority in determining the distribution of the award.
In doing so, we assume that the work took place in a team of peers i.e. that there is no obvious dictator whose opinion everyone naturally aligns with, and, that there is a real opportunity for the contribution of each team member to be assessed by others. The absence of a dictator results in the need for team decisions on the distribution of the prize. The possibility of mutual evaluation, on the other hand, enables group decision-making. This is the case, for example, when a team of researchers of similar experience and reputation work together on a project having from time to time stand-up meetings where the progress of the work is discussed.
The problem outlined above does not only apply to the creation of scientific articles. A similar situation occurs with the distribution of awards for various other team achievements, such as grants, organizational projects or software projects in IT companies.

The solution proposed in this paper is based on the aggregation procedure used in group decision-making (GDM) process as used in the pairwise comparison method. The basics of classical aggregations method are described by Saaty and Aczel [1], Saaty [34] and later by Forman and Peniwati [17]. The problem of aggregating expert opinion for the pairwise comparison method has been studied many times. Aggregation procedures have been adapted for models using fuzzy sets [10], hesitant probabilistic fuzzy linguistic set [36] or intervals [39]. Among GDM studies, consensus building methods have a special place. In [29] a method is proposed that in parallel considering the inconsistency of expert judgements and acceptable consensus. Gong et al. [19] considers the problem of consensus in social networks, where an additional element that can help or hinder consensus building is the actions of the moderator. The proposed maximum fairness consensus model uses decision-maker fairness assessment inspired by social comparison and Gini coefficient. Large-scale group decision-making (LSGDM) is the subject of another paper [27]. Liang et al. propose the framework for LSGDM including clustering, discovering and managing manipulative behavior and consensus reaching. Zhao et al. [41] pay attention to individual participation and satisfaction in the decision-making process. They propose a new consensus model based on optimization of utility aggregated using additive Choquet integral. The issue of consensus, often understood as the minimization of the difference between the preferences of decision-makers and the collective opinion, is described in more detail in [13].

The pairwise comparison of alternatives approach has also resulted in a number of studies in the area of group decision-making. Traditionally, there are two opinion aggregation procedures in the Analytic Hierarchy Process (AHP) [33] method; aggregation of individual judgments (AIJ) and aggregation of individual priorities (AIP) [1, 35, 17]. We will present the latter in more detail in Section 2.2.2. Both procedures are commonly accepted and used for preference aggregation given in the form of pairwise comparisons (PC) matrices (see Sec. 2.1), whereby AIJ is recommended when experts have previously agreed on a common assessment framework [17]. Ramanathan and Ganesh [32] argue in favor of AIP as it meets the Pareto principle when using eigenvector prioritization method (EVM) [33], whereas AIJ does not. Both methods are equivalent when using the geometric mean method (GMM) (or logarithmic least square method (LLSM)) for the ranking calculation [4]. In addition to the two mentioned, other methods of aggregating preferences have emerged. Group euclidean distance (GED) minimization method has been proposed by Blagojevic et al. [5]. The process of progressively modifying the preference matrix of individual experts so as to minimize the defined geometric consensual consistency index (GCCI) lies at the heart of the idea from Dong et al. [14]. Altuzarra et al. proposed Bayesian prioritization (BPP) allowing the aggregation procedure to be extended to preferences (utilities, priorities) between the alternatives that are given as probability distributions [2]. Escobar and Moreno-Jimenez consider [16] a new aggregation method called Aggregation of Individual Preference Structures (AIPS). The aggregation of more diverse preferential information allows phenomena such as judgement uncertainty, problem vision or interdependence to be taken into account. Lin and Kou [28] focus on the Bayesian revision method allowing for PC matrices improvement. Lin et al. [29] propose aggregation of the nearest consistent matrices with the additional adjustment connected with the acceptable consensus. Kulakowski et al. [26] introduce an aggregation method equipped with heuristically adjusted priorities preventing manipulative behavior by some decision-makers.

2 Preliminaries

2.1 Pairwise comparisons

Many methods including AHP, HRE, MACBETH, BWM, outranking methods like e.g. PROMETHEE [25, 20] and various hybrid solutions use pairwise comparisons (PC) as a source of information about decision makers’ preferences. The approach has its extensions to cover different data representations including intervals [21], fuzzy numbers [23] or grey numbers [15]. The best-known method based on pairwise comparisons of alternatives is AHP. It is also the most criticized, although the criticism is often constructive and leads to many improvements and extensions of the original proposal. Of the more important areas of improvement, it is worth mentioning: methods for calculating the vector of weights [22] including calculating rankings for incomplete matrices [6, 7, 24], methods for measuring inconsistencies [8] and others. An interesting critical analysis of the AHP method can be found in Munier and Hontoria [30].

In the PC method, experts, also known as decision-makers, compare alternatives in pairs. Let $A=\{a_{1},\ldots,a_{n}\}$ be finite set of alternatives and $E=\{e_{1},\ldots,e_{k}\}$ be a set of experts. Each expert compares the alternatives with each other to form a set of comparisons represented in the form of a square matrix $C_{q}=\{c_{ijq}\in\mathbb{R}_{+}:i,j=1,\ldots,n\}$ where $1\leq q\leq k$, and $q$ indicates the number of expert. A single $c_{ijq}$ denotes the relative importance of the i-th alternative compared to the j-th alternative according to the experts opinion $e_{q}$. For the sake of legibility of notation, whenever possible we will omit the expert index $q$ by writing the PC matrix as $C=[c_{ij}]$. PC matrix is used to synthesize a vector of weights. Let us define the function resulting from this calculation.

Definition 1.

Let $A$ be a set of alternatives. The priority function for $A$ is the mapping $w:A\rightarrow\mathbb{R}_{+}$ assigning a real and positive number to each alternative.

The priority vector for $A$ resulting from $C_{q}$ takes the form:

$$w_{q}=\left[w_{q}(a_{1}),\ldots,w_{q}(a_{n})\right]^{T}.$$ (1)

Many methods have been described in the literature to calculate the value of priority vector $w$. The first and so far quite popular is the eigenvalue method (EVM). This approach was proposed by Saaty in his seminal paper [33], and consists in calculating a principal eigenvector, and then normalizing it so that the sum of entries is one. Another method is based on calculating the geometric means of the matrix rows and taking the result of the i-th row as the priority of the i-th alternative [12]. The result, thus, obtained is also subject to normalization at the end of the calculation. In addition to the two mentioned, there are quite a few other methods for calculating the priority vector. A comparison of the performance of the six popular priority deriving methods can be found in [37]. Another overview of prioritization methods can be found in [9].

2.2 Group decision-making

2.2.1 Aggregation procedures

When more than one expert participates in the decision-making process, their opinions need to be aggregated. As with a single PC matrix, the end result should be a vector of weights assigning real numbers to each alternative. Aggregation of expert opinions can be done both at the level of individual comparisons and at the level of individual weight vectors. An example of the first approach can be the AIJ (aggregating of individual judgments) [17] procedure, according to which each comparison of the i-th and j-th alternatives by each of $k$ experts is averaged and then a vector of weights is calculated for the matrix composed of such averaged values. In addition to AIJ, there are a number of optimization methods that directly use comparison results from different experts. The previously mentioned model based on minimization of group Euclidean distance (GED) values [5] can serve as an example. A different approach is the AIP procedure according to which priority vectors for individual experts are calculated first and only then are the results aggregated. Compared to AIJ, the AIP approach has several significant advantages. The second does not impose the need for a group of experts to act as a unit, leaving them more freedom to make individual judgments [17]. It can be easily implemented in the situation of having incomplete data from different experts. In such a case, it is enough to use appropriate prioritization methods for incomplete PC matrices [6, 24] to calculate individual weight vectors. The weight vectors obtained in this way may then be subjected to aggregation. In addition, AIP satisfies the Pareto principle with an arithmetic or geometric mean [17].
Aggregation procedures allow each expert’s opinion to be assigned different strengths (priorities). Thus, experts who are considered less competent can have less influence on the final opinion, while experts who are more recognized can have a greater impact on the final outcome. In the next two sections, we will look at two variants of the AIP method that allow for assigning different priorities to individual experts.

2.2.2 Multiplicative aggregation of individual priorities

The most popular AIP approach is based on the use of a weighted geometric mean. Let the vector of weights calculated from the pairwise comparison matrix $C_{q}$ provided by the q-th expert be denoted as follows:

$$w_{q}=\left(\begin{array}[]{c}w_{q}(a_{1})\\ w_{q}(a_{2})\\ \vdots\\ w_{q}(a_{n})\end{array}\right),\,\,\,\text{for}\,\,q=1,\ldots,k.$$ (2)

Then, let $W$ be a matrix of weights vectors so that

$$W=\left(w_{1},\ldots,w_{k}\right),$$

i.e.

$$W=\left(\begin{array}[]{ccccc}w_{1}(a_{1})&w_{2}(a_{1})&\cdots&\cdots&w_{k}(a_{1})\\ w_{1}(a_{2})&w_{2}(a_{2})&\cdots&\cdots&w_{k}(a_{2})\\ w_{1}(a_{3})&\vdots&\ddots&\vdots&\vdots\\ \vdots&\vdots&\vdots&\ddots&\vdots\\ w_{1}(a_{n})&w_{2}(a_{n})&\cdots&\cdots&w_{k}(a_{n})\end{array}\right),$$ (3)

and, let $p$ be the vector of priorities for individual experts i.e.

$$p=\left(\begin{array}[]{c}p_{1}\\ p_{2}\\ \vdots\\ p_{k}\end{array}\right),$$ (4)

so that $\sum_{i=1}^{k}p_{i}=1$. Then

$$\text{GAIP}(W,p)=\left(\begin{array}[]{c}\prod_{i=1}^{k}w_{i}^{p_{i}}(a_{1})\\ \prod_{i=1}^{k}w_{i}^{p_{i}}(a_{2})\\ \vdots\\ \vdots\\ \prod_{i=1}^{k}w_{i}^{p_{i}}(a_{n})\end{array}\right)$$ (5)

denotes AIP procedure using a weighted geometric mean, where $W$ is the matrix of priority vectors derived by individual experts and $p$ is the vector of priorities of those experts.

2.2.3 Additive profile aggregation

An alternative way to aggregate the resulting weight vectors is to use a weighted arithmetic mean. Let priority vector provided by q-th expert be $w_{q}$ (2), matrix of priority vectors be given as $W$ (3) and the vector of expert priorities is $p$ (4) so that so that $\sum_{i=1}^{k}p_{i}=1$. Then

$$\text{AAIP}(W,p)=\left(\begin{array}[]{c}\sum_{i=1}^{k}p_{i}w_{i}(a_{1})\\ \sum_{i=1}^{k}p_{i}w_{i}(a_{2})\\ \vdots\\ \vdots\\ \sum_{i=1}^{k}p_{i}w_{i}(a_{n})\end{array}\right)$$ (6)

denotes AIP procedure using a weighted arithmetic mean.

2.3 Constrained global optimization

Ranking methods very often use optimization methods to determine the best vector of weights corresponding to the alternatives under consideration. If the formulated optimization problem is linear in nature then the Simplex algorithm can be used and the problem is expressed using linear programming [38]. The approach has a rich literature and many extensions. Linear programming underlies many other methods and solutions. An example of a method that grew out of the idea of linear programming is Data Envelopment Analysis (DEA) [11]. When the problem is nonlinear then, depending on the properties of the function under analysis (e.g. convexity), other optimization techniques can be used. For the purpose of testing the solution proposed in this paper we propose the three most popular, well established, heuristic constrained global optimization techniques: Nelder-Mead method, Differential Evolution and Simulated Annealing. The goal of each of the three algorithms is to find the minimum of some function $f:\mathbb{R}^{n}\rightarrow\mathbb{R}$. Finding the minimum is equivalent to proposing the desired vector of weights.

The first of these, the Nelder-Mead method [31], is a direct search method i.e. in its search it does not take into account information, for example, about the direction of the decreasing gradient. For function $f$, in every iteration the algorithm keeps $n+1$ points $x_{1},x_{2},\ldots,x_{n+1}$ forming a polytope. The points are ordered so that $f(x_{1})\leq f(x_{2})\leq\ldots\leq f(x_{n+1})$. Then, the centroid of the $n$ worst points is generated, and the next point is generated by reflecting the worst point through the centroid. Depending on the quality of the generated solution, the procedure can iteratively continue (other points may be selected, or other points removed) or end if the obtained solution is good enough.

The second method, Differential Evolution [40, p. 187], minimizes a function using the genetic programming paradigm. The algorithm works with a population of $m$ candidate solutions, denoted as $\{x_{1},x_{2},\ldots,x_{j},\ldots,x_{m}\}$, where usually $m$ is much larger than the number of variables (denoted by $n$). During each iteration, a new population of $m$ points is generated. For each j-th new point, three random points, $x_{u},x_{v}$, and $x_{w}$, are chosen from the current population. The new point $x_{s}$ is calculated following the formula $x_{s}=x_{w}+s(x_{u}-x_{v})$, with $s$ being the scaling factor. Subsequently, a new candidate $x_{\textit{new}}$ is formed by selecting the i-th coordinate from $x_{s}$ with probability $\rho$, or selecting the i-th coordinate from $x_{j}$. If the objective function value at $x_{\textit{new}}$ is less than that at $x_{j}$, then $x_{\textit{new}}$ replaces $x_{j}$ in the population.

The third used method, Simulated Annealing [18, p. 5] is a straightforward stochastic optimization technique inspired by the annealing process in metallurgy. This process involves heating a metal to a high temperature and then gradually cooling it, allowing its atomic structure to settle into a lower energy state, making the metal stronger. In optimization terms, annealing helps the solution evade local minima and search for a better, potentially global minimum. During each iteration of the algorithm, a new point $x_{\textit{new}}$ is generated within the vicinity of the current point $x$, with the radius of this neighborhood shrinking over time. The algorithm keeps track of the best point encountered so far, $x_{\text{best}}$.

If the objective function value at $x_{\text{new}}$ is better than at $x_{\text{best}}$, $x_{\text{new}}$ replaces both $x_{\text{best}}$ and $x$. If not, $x_{\text{new}}$ can still replace $x$ with a probability $e^{b(i,\Delta f,f_{0})}$, where $b$ depends on the current iteration number $i$, the change in the objective function value $\Delta f$, and the objective function value from the previous iteration $f_{0}$. Similar to the Random Search method, Simulated Annealing uses multiple starting points to enhance the search for an optimum solution. For each starting point, the process continues until either the maximum number of iterations is reached, the solution converges, or the algorithm remains at the same point for the specified number of iterations.

The popularity of these three methods translates into their availability in various packages and computing environments including R, Mathworks Matlab™ or Wolfram Mathematica™ and others.

3 Peer ranking aggregation

3.1 Problem description

The immediate inspiration to take up the topic of aggregation of opinions in a group of peers was the introduction of changes in the regulations for rewarding employees of a certain university, according to which the reward depends on the level of contribution of each author of a paper. Of course, this level must be numerical (percentage) and allow for an unambiguous distribution of money between the parties involved. It must also be based on some compromise because all authors must sign a declaration in which they agree to the assigned share. In practice, most teams deal with this problem by selecting one person who, acting as "dictator", proposes the distribution of "percentages" i.e. funds for the prize. Depending on the custom, team members may (or may not) negotiate the proposed share. In the case of teams working on a scientific article, the role of dictator is played by the correspondent author, first author or simply the head of the research group.

From the point of view of the university’s financial management, the advantage of the model with "one person in charge" is the relative simplicity of the solution. The entire burden of responsibility for the proper allocation of funds is transferred to the person designated for this purpose. The problem, of course, is most often the lack of a methodology to make an informed decision on the distribution of the award, which boils down to adopting the simplest decision-making model, i.e. dictatorship. The obvious disadvantage of the dictatorial approach is the arbitrariness of the decisions made. The decision-maker likes some members of the team more and others less, which can directly translate into the evaluation of their work. He also often overestimates his own participation in the project. On the other hand, the decision-maker is usually the person most involved in the project and, what goes hand in hand, the best informed as to the participation and nature of others’ contributions.

3.2 Model

The above observations led us to propose a peer review model in a group of peers based on pairwise comparison of alternatives. Group members have a dual role. They are simultaneously the object of comparison, i.e. the alternatives $A=\{a_{1},\ldots,a_{n}\}$ in the PC method, and the experts providing their judgments i.e. $E=\{e_{1},\ldots,e_{n}\}$. Each of the experts, i.e. team members, provides comparisons in the form of a PC matrix $C=[c_{ij}]$, where $c_{ij}\in\mathbb{R}_{+}$ in which he assesses the relative level of contribution of the other members of the group. Based on the collected ratings $C_{1}$,…,$C_{n}$, vectors of weights $w_{1},\ldots,w_{n}$ for individual experts are created. The calculated vectors are then subjected to aggregation, resulting in the formation of a final priority vector

$$w=\left(\begin{array}[]{c}w(a_{1})\\ \vdots\\ \vdots\\ w(a_{n})\end{array}\right)$$

reflecting the share of each evaluated person in the award. However, in line with the observation that those more involved in a project are often more competent to assess the engagement of others, the priorities given to expert opinions during aggregation may not be identical. They must reflect this regularity. Let $p:A\rightarrow\mathbb{R}$ be an expert priority function such that for two experts $e_{i}$ and $e_{j}$ if their final evaluation share meets $w(a_{i})\geq w(a_{j})$ then also their priorities during the aggregation process meet $p(a_{i})\geq p(a_{j})$. This regularity comes out of the observation that the one who contributed more to the success than the other has more right than the latter to decide on the distribution of the reward. In the adopted solution, we adopted the simplest possible function $p$ i.e. $p(a_{i})=w(a_{i})$. It grants the right to decide to a team member directly proportional to his level of involvement. In many cases, it also seems to be the most equitable. Below, following [17], we have proposed two aggregation models to support the proposed solution: multiplicative and additive.

3.3 Multiplicative profile aggregation

When proceeding to aggregate the results, we assume that each team member $e_{q}\in E$ provided a PC matrix $C_{q}$ based on which the ranking vector $w_{q}=\left(w_{q}(a_{1}),\ldots,w_{q}(a_{n})\right)^{T}$ was calculated. These vectors form a matrix $W$ of the form:

$$W=\left(\begin{array}[]{ccccc}w_{1}(a_{1})&w_{2}(a_{1})&\cdots&\cdots&w_{n}(a_{1})\\ w_{1}(a_{2})&w_{2}(a_{2})&\cdots&\cdots&w_{n}(a_{2})\\ w_{1}(a_{3})&\vdots&\ddots&\vdots&\vdots\\ \vdots&\vdots&\vdots&\ddots&\vdots\\ w_{1}(a_{n})&w_{2}(a_{n})&\cdots&\cdots&w_{n}(a_{n})\end{array}\right).$$

Denoting vector of experts’ priorities as $p=\left(p(a_{1}),\ldots,p(a_{n})\right)^{T}$ we get an extra condition that the final solution must satisfy:

$$\text{GAIP}(W,p)=\left(\begin{array}[]{c}\prod_{i=1}^{k}w_{i}^{p(a_{i})}(a_{1})\\ \prod_{i=1}^{k}w_{i}^{p(a_{i})}(a_{2})\\ \vdots\\ \vdots\\ \prod_{i=1}^{k}w_{i}^{p(a_{i})}(a_{n})\end{array}\right)=\left(\begin{array}[]{c}p(a_{1})\\ p(a_{2})\\ \vdots\\ \vdots\\ p(a_{n})\end{array}\right).$$ (7)

For the purposes of the calculation, we will also want $p(a_{i})>0$ for $i=1,\ldots,n$ and $\sum_{i=1}^{n}p(a_{i})=1$. The introduction of (7) leads directly to the formulation of a nonlinear optimization problem in the form:

$$\min g(W,p)\,\,\,\,\text{s.t.}$$

$$g(W,p)=\sum_{i=1}^{n}\left(\prod_{j=1}^{n}w_{j}^{p(a_{j})}(a_{i})-y_{i}\right)^{2}$$ (8)

$$p(a_{i})=\frac{y_{i}}{\sum_{j=1}^{n}y_{j}}\,\,\,\text{and}\,\,p(a_{i})>0\,\,\text{for}\,\,i=1,\ldots,n.$$

The above (8) can be solved using constrained global optimization methods (Section 2.3).

One can also attempt to find the final vector of weights using a direct iterative algorithm (listing 3.3) bearing in mind, however, that it may not always converge. The iterative algorithm follows a simple scheme. It takes as input a matrix $W$ and three parameters: the maximum number of iterations $\gamma$, the maximal acceptable difference of consecutive results $\delta$ and maximal permissible error $\epsilon$. Then, in each subsequent iteration, a new vector of weights $w^{\prime\prime}$ is calculated using GAIP where the previously calculated weights vector $w^{\prime}$ is used as the priorities of the individual experts (line: 6). When using the GAIP procedure for the first time (line: 2), the experts’ priority values may be equal. The algorithm ends when the maximum number of iterations allowed is reached (line: 5), or the difference between successive calculated vectors becomes less than $\delta$ (line: 7). In the first case (line: 14) the algorithm did not find a solution. In the second case, however, the algorithm selected a vector that can be a solution to the problem (7). In the second case, although the vector $w^{\prime\prime}$ is calculated, we need to check whether the calculation is accurate enough. For this purpose, the value $g(W,w^{\prime\prime})$ of the criterion function is calculated (8) and compared against the maximum permissible error $\epsilon$ (line: 8).

1:function DIA($W$, $\gamma$, $\delta$, $\epsilon$)

2:     $w^{\prime}\leftarrow\text{GAIP}(W,(1/n,\ldots,1/n)^{T})$

3:     $w^{\prime}\leftarrow\text{normalize}(w^{\prime})$

4:     $\text{i}\leftarrow 1$

5:     for i $\leq$ $\gamma$ do

6:         $w^{\prime\prime}\leftarrow GAIP(W,w^{\prime})$

7:         if  $\left\|w^{\prime\prime}-w^{\prime}\right\|<\delta$ then

8:              if  $g(W,p)\leq\epsilon$ then

9:                  return solution not precise enough

10:              end if

11:              return $w^{\prime\prime}$

12:         end if

13:     end for

14:     return solution not found

15:end function

Direct Iterative Algorithm (DIA)

If the algorithm DIA returns the correct value of the vector of weights, this result can be taken as the correct solution. If the calculated result is too inaccurate you can increase the number of iterations or the delta parameter and try to run the algorithm again. However, if the calculation does not converge despite increasing the parameters $\gamma$, $\delta$ and $\epsilon$ it is necessary to use more advanced optimization methods (Section: 2.3).

3.4 Additive profile aggregation

The aggregation model outlined above (Section 3.3) in which experts’ priority is linked to their ranking score naturally has an additive version. Thus, the solution must satisfy an additional condition of the form:

$$\text{AAIP}(W,p)=\left(\begin{array}[]{c}\sum_{i=1}^{k}p(a_{i})w_{i}(a_{1})\\ \sum_{i=1}^{k}p(a_{i})w_{i}(a_{2})\\ \vdots\\ \vdots\\ \sum_{i=1}^{k}p(a_{i})w_{i}(a_{n})\end{array}\right)=\left(\begin{array}[]{c}p(a_{1})\\ p(a_{2})\\ \vdots\\ \vdots\\ p(a_{n})\end{array}\right).$$ (9)

As before, we also assume that $p(a_{i})>0$ for $i=1,\ldots,n$ and $\sum_{i=1}^{n}p(a_{i})=1$. The optimization problem induced by condition (9) gets the form:

$$\min h(W,p)\,\,\,\,\text{s.t.}$$

$$h(W,p)=\sum_{i=1}^{n}\left(\sum_{j=1}^{n}p(a_{j})w_{j}(a_{i})-y_{i}\right)^{2}$$ (10)

$$p(a_{i})=\frac{y_{i}}{\sum_{j=1}^{n}y_{j}}\,\,\,\text{and}\,\,p(a_{i})>0\,\,\text{for}\,\,i=1,\ldots,n.$$

DIA algorithm itself undergoes a cosmetic change of replacing GAIP with AAIP in the lines 2 and 6.

3.5 Pareto principle

Theorem 2.

Both modified aggregation methods GAIP and AAIP satisfy the Pareto principle [3] i.e. if for any two alternatives $i,j$ all experts prefer the i-th alternative over the j-th alternative i.e. $w_{k}(a_{i})>w_{k}(a_{j})$ for $k=1,\ldots,n$ then this relationship holds in the aggregated vector of weights i.e. $w(a_{i})>w(a_{j})$.

Proof.

Let GAIP and AAIP be as defined in (5) and (6) correspondingly. Then

$$||GAIP(W,p)||=$$

$$=\sqrt{(w_{1}^{p_{1}}(a_{1})\cdots w_{n}^{p_{n}}(a_{1}))^{2}+\ldots+(w_{1}^{p_{1}}(a_{n})\cdots w_{n}^{p_{n}}(a_{n}))^{2}}\leq$$

$$\leq\sqrt{(p_{1}w_{1}(a_{1})+\ldots+p_{n}w_{n}(a_{1}))^{2}+\ldots+(p_{1}w_{1}(a_{n})+\ldots+p_{n}w_{n}(a_{n}))^{2}}=$$

$$=||AAIP(W,p)||=$$

$$=\sqrt{\langle p,w_{1}\rangle^{2}+\ldots\langle p,w_{n}\rangle^{2}}\leq\sqrt{||p||^{2}||w_{1}||^{2}+\ldots+||p||^{2}||w_{n}||^{2}}=$$

$$=||p||\sqrt{||w_{1}||^{2}+\ldots+||w_{n}||^{2}}\leq||p||\sqrt{n}\leq\sqrt{n}.$$

Obviously, the mapping $AAIP(W,\cdot)$ is linear, so for each $p,\hat{p}\in\mathbb{R}^{n}$

$$AAIP(W,p)-AAIP(W,\hat{p})=AAIP(W,p-\hat{p})\leq\sqrt{n}||p-\hat{p}||.$$

∎

Notice that for $p=(\frac{1}{n},\ldots,\frac{1}{n})^{T}$ the norm

$$||p||=\frac{1}{\sqrt{n}},$$

so

$$||GAIP(W,p)||\leq||AAIP(W,p)||\leq 1.$$

On the other hand, if for each $i\in\{1,\ldots,n\}$ we have

$$w_{1}(a_{i})=w_{2}(a_{i})=\ldots=w_{n}(a_{i}),$$

then for each $j\in\{1,\ldots,n\}$ and each $p$ it follows that

$$GAIP(W,p)=AAIP(W,p)=(w_{j}(a_{1}),\ldots,w_{j}(a_{n}))^{T}.$$

Now fix $i,j\in\{1,\ldots,n\}$. Both mappings satisfy the Pareto principle, i.e. for each $p\in\mathbb{R}^{n}$ and for each $k\in\{1,\ldots,n\}$

$$w_{k}(a_{i})\leq w_{k}(a_{i})\Rightarrow\left(w_{k}^{p_{k}}(a_{i})\leq w_{k}^{p_{k}}(a_{j})\textnormal{ and }p_{k}w_{k}(a_{i})\leq p_{k}w_{k}(a_{j})\right),$$

which implies that

$$w_{1}^{p_{1}}(a_{i})\cdot w_{2}^{p_{2}}(a_{i})\cdots w_{n}^{p_{n}}(a_{i})\leq w_{1}^{p_{1}}(a_{j})\cdot w_{2}^{p_{2}}(a_{j})\cdots w_{n}^{p_{n}}(a_{j})$$

and

$$p_{1}w_{1}(a_{i})+p_{2}w_{2}(a_{i})+\cdots+p_{n}w_{n}(a_{i})\leq p_{1}w_{1}(a_{j})+p_{2}w_{2}(a_{j})+\cdots+p_{n}w_{n}(a_{j}).$$

This means that if all the experts scored the $i$-th one equally or lower than the $j$-th one, then the resulting ranking of the $i$-th expert cannot be higher than the ranking of the $j$-th one.

4 Summary

In the work presented here, we proposed two new priority aggregation models for group decision-making in the pairwise comparison method, in which experts’ priorities are linked to the aggregation score. Both models satisfy the Pareto principle, according to which, for two alternatives, $a_{i}$ and $a_{j}$, if each group member prefers $a_{i}$ to $a_{j}$, then the group must also prefer $a_{i}$ to $a_{j}$.

These models correspond to the situation in which the experts being evaluated are the object of evaluation. A study example is determining the distribution of a team achievement award. Such an achievement could be the preparation and publication of a scientific article. However, the presented approach can have many other applications, such as the distribution of bonuses among team members for achieving milestones in an IT project.

We have also indicated methods for calculating the solution to the problem described by such models. Very often, such a solution can be calculated using an iterative process. When this fails, global optimization methods such as the Nelder-Mead method and others come to the rescue. In future research, we will focus on the properties of the proposed models. In particular, we intend to research the safety and robustness of such models against manipulative behavior.