sustainability-13-12121-v2.pdf

sustainability
Article
An AHP-Based Procedure for Model Selection for
Eco-Efficiency Assessment
Maria Julia Xavier Belem 1 , Milton Vieira Junior 1,*, Giovanni Mummolo 2 and Francesco Facchini 3,*

Citation: Xavier Belem, M.J.; Vieira
Junior, M.; Mummolo, G.; Facchini, F.
An AHP-Based Procedure for Model
Selection for Eco-Efficiency
Assessment. Sustainability 2021, 13,
12121. https://doi.org/10.3390/
su132112121
Academic Editor: Andrea Trianni
Received: 22 September 2021
Accepted: 29 October 2021
Published: 2 November 2021
Publisher’s Note: MDPI stays neutral
with regard to jurisdictional claims in
published maps and institutional affil-
iations.
Copyright: © 2021 by the authors.
Licensee MDPI, Basel, Switzerland.
This article is an open access article
distributed under the terms and
conditions of the Creative Commons
Attribution (CC BY) license (https://
creativecommons.org/licenses/by/
4.0/).
1 Production Engineering Graduate Program, Methodist University of Piracicaba, Piracicaba 13400-390, Brazil;
mjuliaxb@gmail.com
2 Ionian Department in Juridical and Economic Systems of the Mediterranean: Society, Environment, Cultures,
University of Bari, 74100 Taranto, Italy; giovanni.mummolo@poliba.it
3 Department of Mechanics, Mathematics and Management, Polytechnic University of Bari, 70125 Bari, Italy
* Correspondence: milton.junior3@unimep.br (M.V.J.); francesco.facchini@poliba.it (F.F.);
Tel.: +55-19-997820096 (M.V.J.); +39-080-596-3612 (F.F.)
Abstract: Modelling eco-efficiency is becoming a wide field of research shown by a large body of sci-
entific literature on the subjects concerned. This paper, after performing a systematic literature review
(SLR) to identify existing models for process eco-efficiency assessments, provides a methodology,
based on the analytic hierarchy process (AHP) method, for choosing the eco-efficiency assessment
model to be used for a given application (process, product, or service). For the SLR, papers from
the databases Scopus, Web of Science and Science Direct were used. Forty articles were considered
for this study, using as the main selection criterion articles that present an eco-efficiency assessment
model, since the purpose was to survey the types of existing models that are used to assess processes,
products or services for eco-efficiency. With the systematic review carried out, it was possible to
identify the types of models that exist and how they are used in different sectors, always aiming to
identify if what was analyzed is eco-efficient and what points need to be improved. The proposed
AHP-based methodology was applied to a numerical model to outline how to apply the methodology.
The approach was easy to use and effective in identifying the proper eco-efficiency model.
Keywords: eco-efficiency; model selection; AHP method
1. Introduction
In manufacturing processes, one of the main factors to be analyzed is the energy
issue, i.e., the energy lost in the form of heat, the emissions into the environment and in
relation to the energy costs spent in the process. This causes companies to face pressure
from government and society to implement sustainable strategies in their business. To
address the economic and environmental challenges, sustainable approaches such as energy
efficiency in manufacturing processes are employed to reduce the impacts of wasted energy
and emissions on the environment [1].
Given this scenario, from the perspective of sustainability, the concept of eco-efficiency
appears to evaluate the environmental and economic conditions of processes, products and
services, through the survey and application of indicators that identify the relationship
between the results of the processes and sustainable and economic factors.
Eco-efficiency is defined as a relationship between production, expressed by economic
values and the ecological impact caused by material consumption or waste generation in the
activities involved in the manufacturing operation. It also promotes the idea of minimizing
the costs of operating goods and services, maximizing efficiency, while optimizing the use
of necessary resources, reducing waste and environmental pollution [2].
An eco-efficiency methodology known and applied in studies in the scientific literature
was developed in 1996 by BASF Corporation to assess the economic and environmental
impacts of chemicals, processes and products in their life cycle [3]. This methodology has
Sustainability 2021, 13, 12121. https://doi.org/10.3390/su132112121 https://www.mdpi.com/journal/sustainability

Sustainability 2021, 13, 12121 2 of 21
been adapted and follows the ISO 14040 and ISO 14044 standards for life cycle analysis
(LCA) [4] and ISO 14045 for Eco-efficiency assessment [5]. BASF’s methodology can be
used for sustainable decision making at all levels of the consumption chain, from industry
to consumer [6].
Lozano and Lozano [7] point out that among the sustainable methodologies, eco-
efficiency stands out for providing an explicit and effective evaluation criterion with the
use of LCA-based indicators. For environmental impact assessment, six categories are
considered: raw material consumption; energy consumption; emissions to air, water and
waste; toxicity potential; risk/potential for misuse; and land use. The data and calculations
for the first three categories are based on ISO 14040 and 14041 and the others by consulting
databases containing these data, for example [4].
There are different tools and methods that can be combined to evaluate the eco-
efficiency of products, processes and services; these are established for comparison, with
respect to life cycle data, environmental impact assessment and process costs, i.e., such
analyses provide only comparative information, without an absolute value. Existing
analyses allow the results to be normalized to enable comparison of indicators, providing
guidance for the optimal choice of product or process that should be developed from a
long-term perspective.
Moreover, eco-efficiency can be used as a decision-making tool, providing the best
choice between transformation processes by combining scientific and technical issues with
economic issues [7] and can be applied in various sectors, in a specific product, in business,
and in industrial processes. Multi-criteria decision analysis (MCDA) can help to make
the best decision by evaluating, prioritizing and selecting among several alternatives and
distinct criteria that must be considered, in a coherent and transparent way for the stake-
holders. The MDCA can be applied following the following phases: define the objective,
choose the criteria to measure the objective, specify the alternatives, assign weights to the
criteria, apply a mathematical algorithm to rank the criteria and/or choose an alternative
and, finally, make the decision or indicate which planning should be followed [8]. There are
different MCDA methods, which can be listed as Advanced MCDA: Multi-Attribute Utility
Theory (MAUT), probabilistic multi-criteria acceptability analysis (ProMAA), Fuzzy-MAVT
(Multi-Attribute Value Theory); Fuzzy-PROMETHEE (Preference Ranking Organization
Method for Enrichment Evaluations); and Basic MCDA methods: MAVT, TOPSIS (Tech-
nique for Order Preference by Similarity to Ideal Solution), PROMETHEE, AHP [8]. The
method should be selected considering what is most appropriate for the study according
to the characteristics of the analyzed problem, which can be in relation to data availability
and problem complexity [9].
The development of indicators to compose eco-efficiency is a fundamental strategy to
evaluate and monitor the process in a simple but systematic and consistent way, enabling a
better understanding of the problem, facilitating decision making and leading towards qual-
ity of life and preservation of natural resources [10]. Moreover, eco-efficiency evaluation
goes far beyond being introduced as a business concept, and can also be used at regional,
national and global levels through evaluation in cities, regions, and countries [11–13].
Thus, studies that present practices, indicators, and methods for evaluating eco-
efficiency can contribute to sustainable development in industries, companies, projects and
processes. The indicators are employed to measure the economic and environmental data
involved in the process and present eco-efficiency results to be discussed and analyzed
by the company. Thus, the objectives of this paper are to: (i) identify existing models
for processing eco-efficiency evaluation through a systematic literature review (SLR); (ii)
provide a procedure, based on the analytic hierarchy process (AHP) method, for choosing
the eco-efficiency evaluation model to be used for a given application (process, product,
or service). AHP approach is an MCDA method that decomposes a complex multi-factor
problem into a hierarchical structure and assigns to each level the related attribute [14].
Consistent with this approach, the AHP method solves complex problems identifying
outcomes and developing weights or priorities. A bibliometric study of MCDA fields

reveals that AHP has been adopted in more scientific publications than other MCDA
methods [15]. According to the scientific literature, the AHP method has been highly
acclaimed [16] by a large number of academics for three main reasons: it can be quickly
implemented considering a small number of parameters only [15]; it is supported by
user-friendly and commercially software packages [17]; and it ensures high reliability
avoiding the ‘rank reversals problem’ by structuring the decision problem before the
decision analysis [18]. All these aspects fit with the nature of the issue addressed in this
research work.
This paper is structured as follows: Section 2 presents a brief description of eco-
efficiency, concepts and applications; Section 3 describes the methodology used to achieve
the objectives of the paper; Section 4 presents the analysis of the eco-efficiency evaluation
models obtained through SLR, presenting a bibliometric analysis of the articles, followed by
a discussion of the eco-efficiency evaluation models; finally, Section 5 presents a suggested
procedure for choosing the eco-efficiency model, based on the AHP method.
2. Eco-Efficiency
The term eco-efficiency was first proposed in 1990 by Schaltegger and Sturm. However,
it was in 1991 that the World Business Council for Sustainable Development (WBCSD) was
responsible for introducing one of the main concepts of eco-efficiency to the world. The
term was defined as the delivery of economically competitive goods and services that meet
the needs of society in terms of quality of life and that the entire process of manufacturing
and availability for consumption, that is, throughout its life cycle, should have the least
possible impact on the environment [19].
Eco-efficiency is presented as a combination of the economic and environmental
performance of a given system, assessed using a widely applied and easily interpreted
framework that can be employed in a wide variety of business sectors, while providing
a common set of indicators. Thus, eco-efficiency in the form of Equation (1) is presented
as [20]:
Eco-efficiency =
Product or service value
Environmental impact
(1)
This concept of eco-efficiency is widely applied to obtain a form that combines envi-
ronmental and economic performance, for the most diverse product systems, processes
and/or companies. It can be understood that the term ‘value’ present in the eco-efficiency
equation can also refer to the related cost in manufacturing up to replacement of the prod-
uct. The definition of eco-efficiency is given by the ratio between the (added) value of what
was manufactured/generated (GDP, income, high quality goods and services offered, etc.)
and the (added) impacts of the product or service [21]. For Huppes and Ishikawa [16],
eco-efficiency is an instrument to analyze sustainability, in a way that indicates an empirical
relationship of economic activities (costs involved) with environmental value (impacts to
the environment). Still, it can be pointed out that eco-efficiency presents a role of encourag-
ing business opportunities, allowing companies to become responsible in relation to the
environment, while presenting profits [19]. Thus, industrial organizations are expected
to produce goods and services by using the least number of resources and generating the
least amount of waste and other pollutants [20].
The WBCSD identified seven characteristics that can be considered a guide for com-
panies to become more eco-efficient: reduce material intensity/quantity; reduce energy
intensity; increase recyclability; maximize the use of renewable resources; increase product
durability; and increase service intensity [19,22].
In the business context, eco-efficiency can be applied at the product, process, corporate
or industry level [23,24] and can also be assessed in relation to a specific service [21].
However, the application of this term goes far beyond being introduced as a business
concept and can also be used at regional, national and global levels, through the evaluation
of the eco-efficiency of cities, regions and countries [11,13,25].

Koskela and Vehmas verified in their work that eco-efficiency can be seen both as
an environmental performance indicator and as a business strategy for sustainable devel-
opment. In their study, the authors pointed out that the definitions and applications of
eco-efficiency are associated with some aspects, such as: high production but with lower
use of natural resources (produce more with less); produce more added value with lower
environmental impact (ratio between economic value and environmental influence); use
eco-efficiency as a management strategy (offer companies the possibility to get a new
vision of their processes and increase their ability to innovate); a guidance to improve the
eco-efficiency of an organization [26].
A wide range of scientific literature is available on eco-efficiency techniques, instru-
ments and models. A systematic review is carried out in the next section.
3. Systematic Literature Review (SLR) Methodology and Procedures
For the development of the work, theoretical research was conducted based on a
systematic literature review (SLR), to substantiate and level the knowledge about the
subjects involved in the proposal. The SLR process involves three main phases: planning
the review, conducting the review and reporting the review [27,28]. In each phase there are
specific steps that are further detailed in this section.
In Planning the Review there are two steps to be performed: formulating the research
problem and developing and validating the review protocol. Thus, the research question
was specified and the review protocol was developed.
In the Conducting the Review phase there are five steps: searching the literature,
screening for inclusion, assessing quality, extracting data, analyzing and synthesizing data.
To search the literature, the databases Scopus, Web of Science, Science Direct were
used and the following keywords were used: “Eco-efficiency”, “Assessment” and “Model”.
In the screening phase for inclusion, the articles were screened excluding the articles
according to the exclusion criteria established according to the research question. Initially,
the titles and abstracts were read to select the articles. Table 1 presents the research protocol,
containing the aspects used for searching and selecting articles for the RSL.
Table 1. Research protocol.
Aspects for Article Selection
Database Scopus, Web of Science, Science Direct
Search strings used “Eco-efficiency” AND “Assessment” AND “Model”
Research fields Article title, abstract and keyword
Language English
Publication date range No limit was set for the date range
Exclusion Criteria
- It is not in the scope of the research (in title, abstract);
- Duplicate articles;
- Conference papers or book chapter
- It does not present development and/or application of
eco-efficiency evaluation models.
Subsequently, in the step of assessing the quality of the selected texts, the texts were
read in full and it was considered whether the articles provide clear and objective informa-
tion about eco-efficiency assessment models.
Thus, primary studies were identified and selected and extracted (extract step) and
the data referring to the theme were analyzed and synthesized (analyze step), according to
the criteria established in Table 1.
The last phase, the Review Report, is composed of the step of reporting the findings
to describe in a systematized way the information obtained from the literature review.
The results obtained from the literature search screening and quality assessment are
presented in the flow chart shown in Figure 1.

Figure 1. Literature search and evaluation for inclusion of articles for SLR.
Initially, with searches in the three databases, as shown in Figure 1, 592 articles
were found. Articles were exported to the StArt® (State of the Art through Systematic
Review) software, a tool that provides support to carry out the systematic review process,
facilitating the classification of articles to select which ones will be used for SLR [29]. Out
of the 592 initial articles, 200 articles were excluded for duplication. By reading of the title
and abstracts, 131 articles out of the 392 articles were selected as most aligned with the
objective of the present research. Finally, 40 articles were selected as having a complete an
eco- efficiency evaluation model and related application to evaluate processes, products, or
services for eco-efficiency.
4. Analysis of Eco-Efficiency Assessment Models
The term eco-efficiency has gained greater visibility through the definition established
by the WBCSD in 1991, aimed directly at business orientation. Since then, studies aimed
at assessing eco-efficiency have gained space in organizations and in the scientific com-
munity. Eco-efficiency has been analyzed in different applications, with concepts focused
on analyses of production and consumption, with the relationship between ‘value’ and
‘environmental impact’. Eco-efficiency models began to be developed and applied to
understand how a product or service could be more or less eco-efficient [29].
The selected articles were analyzed in terms of the application area of the eco-efficiency
assessment models, as shown in Figure 2. Nine areas were identified, with eco-efficiency
assessment of cities and regions being the focus of study in 23% of the total articles
analyzed, followed by construction and manufacturing sectors each of them with about
18% of the articles.
In the following section, the analysis and discussion of the articles surveyed that
resulted from the systematic literature review is shown.

Figure 2. Publications by industry sector.
Discussion of Eco-Efficiency Assessment Models
The evaluation of the eco-efficiency (EE) of products, services or processes is carried
out through different approaches, methods and models offering an appropriate way to
measure eco-efficiency, with measurable indicators and indexes.
According to Yang and Zhang [30], the assessment of eco-efficiency can be performed
through the ratio between the economic value of the goods or services that are produced and
the environmental impacts they have. It is required that the numerator and denominator are
normalized so that a meaningful ratio of compared quantities is obtained. Still, according to
the authors, other approaches can be used to evaluate EE, incorporating life cycle analysis
(LCA) for environmental assessment, for example and even using data envelopment
analysis (DEA), incorporating different inputs and outputs in different dimensions, this
being the approach most often adopted by some researchers [31,32].
Based on the systematic literature review conducted in this work, it was observed
that eco-efficiency is evaluated in different areas and industrial sectors to achieve minimal
environmental impact in relation to the economic gain obtained.
In analyses conducted in the field of electrical and electronic equipment, Barba-
Gutierrez, Adenso-Diaz and Lozano [33] presented an evaluation of EE to present ref-
erence products adopted for improving eco-efficiency in the production of electrical and
electronic equipment. These analyses were performed using an approach based on DEA,
which consists of solving a simple linear programming model to help support decision
making, that considers as output the average retail price of the products, and as input
the LCA data for the environmental impact. DEA is used to measure the efficiency of a
set of decision-making units (DMUs) when there are multiple inputs and outputs in the
production process. In this context, both Kondo and Nakamura [34] and Lozano et al. [23]
showed EE evaluation methods that allow the identification of better strategies for the use
of electrical and electronic products.
EE assessment has also been studied in the construction sector, aiming to analyze the
eco-efficiency of building materials [35–39] and of processes involved in this industrial
sector [40,41]. In the studies conducted in [35] and [36], eco-efficiency evaluation models
based on a DEA framework were presented to evaluate EE of building materials. For this,
the concepts of life cycle cost (LCC) and life cycle assessment (LCA) were used to calculate
the eco-efficiency index. Furthermore, DEA was used in the modeling, considering input
data acquired by LCA and output as data obtained by LCC. The approach supports decision
making by comparing the eco-efficiency of building materials.
Perera et al. [39] calculated eco-efficiency by means of the ratio between LCC and LCA
to select which material is more eco-efficient to be used in construction. In this case, the
economic score, referring to the EE equation, was identified based on life cycle cost (LCC),
while the environmental score was obtained by using life cycle assessment (LCA).
The LCC is the entire cost (initial and future) of the product, from purchase, installation,
maintenance, repair and replacement. The approach that uses the LCC to represent the
economic value added in the numerator of the eco-efficiency equation is commonly used, as

it is possible to estimate all the costs that are related to the environmental effects of the life
cycle, that is, to obtain an economic value for the entire life cycle. As for the denominator,
the LCA is used to obtain data from the environmental effect categories.
Hamid and Shafiq [37] used the ratio of economic score (cost of material and labor
used) and environmental score (data of the amount of carbon emission during the material
manufacturing) to define the eco-efficiency index of reinforced concrete structures, in the
decision-making process for selecting the best sustainable design for reinforced concrete
structure. For Kim et al. [38], to evaluate eco-efficiency in the performance and use of
concrete, it is necessary to also consider the durability of the material used in reinforced
concrete structures, in addition to the environmental load (EC is the environmental cost)
and manufacturing costs (MC is the manufacturing cost and SL is the service life of
concrete).
Studies have showed that evaluating the EE for selecting the most eco-efficient material
requires understanding the entire context involved in manufacturing, from the material
resources involved, which generate environmental impacts, to the costs of transformation
(e.g., the labor involved, logistics, etc.). Hu and Liu [40] explored data of the Australian
construction industry between the years 1990 and 2013 to feed DEA models so as to identify
in a quantitative way the eco-efficiency of Australian industries and point out key measures
to improve EE, such as the implementation of advanced and eco-efficient construction
technologies.
Eco-efficiency analyses applied in construction can both facilitate the circulation of
information to the different stakeholders that are involved in this industry and help in
the identification and procurement of environmentally and economically efficient mate-
rials [41]. Eco-efficiency can be considered an important factor to assess sustainability
and the application and interpretation of the proposed models are able to show a path to
be followed that integrates economic growth, resource conservation and environmental
protection [40].
For systems involving water resources, EE evaluation was performed by [42–45].
The studies aimed to evaluate systems that use water as the main resource, analyzing
efficiency, economic performance, pollutants, among other factors necessary to obtain an
eco-efficiency index of the analyzed water system.
Liu et al. [42] analyzed the eco-efficiency of water systems in regions located in China
by combining the input–output index obtained by the rough set theory (RST) and DEA to
analyze the eco-efficiency of water systems.
Mehmeti et al. [43] estimated a set of proposed EE indicators to identify the best
alternatives for improving irrigation system eco-efficiency.
In [44,45], the eco-efficiency of wastewater treatment plants (WWTPs) was evaluated.
A WWTP is defined as a production facility that uses material and energy resources to
remove pollutants from wastewater and discharge the water that has been treated into the
environment [45]. With the evaluation performed by Molinos-Senante et al. [44], using a
Weighted Russell Directional Distance Model (WRDDM), it was possible to identify the
variables with a direct impact on EE, making it possible to focus on them to improve the
eco-efficiency of the system. In the EE evaluation presented by Gómez et al. [45], data
envelopment analysis (DEA) was used, which allowed for the integration of desirable or
undesirable outputs and inputs.
Sanjuan et al. [46] state that different models are proposed to measure eco-efficiency,
with the main difference between these being the weighting system used to aggregate
environmental outputs.
Approaches using LCA are often used to evaluate EE, as shown in the studies of [47–50].
Life cycle assessment of a product or service consists of a methodology that evaluates the
environmental burdens associated with a product, process, or activity, identifying energy and
materials used and waste released into the environment. LCA encompasses all the different
stages of the life cycle of goods: extraction and processing of raw materials, manufacturing,
transportation, distribution, use, reuse, maintenance, recycling and final disposal [51,52].

In [47], an eco-efficiency modeling method for cattle feedlot systems was presented. In
the eco-efficiency assessment model proposed, producers become aware of the feed costs
and other expenses involved in keeping the animals, as well as in relation to the impacts
on the environment, with enteric, air and water emissions.
LCA was also used to survey the total environmental impact (considering the cate-
gories of human health damage, ecosystem quality, climate change and resources) to obtain
a single indicator of eco-efficiency in the manufacture of light fixtures [50]. Eco-efficiency
was calculated as the ratio of total fixture cost (TFC) to total environmental impact (TEI:
human health, ecosystem quality, climate change and resources).
In other studies, LCA was combined with LCC to obtain standardized EE indicators
and be able to assess which product (standard and redesigned disposable diapers) was
more eco-efficient [48]. LCA, focusing on carbon footprint, was used in conjunction with
DEA to assess eco-efficiency in raspberry production in Chile [49].
In research aimed at evaluating the EE of cities, considering the common definition of
eco-efficiency (ratio between economic value and environmental pressures), DEA-based
models (linear programming analysis) have been presented to evaluate the EE of cities in
China [11,53,54]. A city’s eco-efficiency is in line with its level of economic development,
with continuous cleaner production, circular economy, energy conservation moves and
emissions reduction [53]. By using the DEA model, it was possible to compare samples
and observe changing trends in use of resources, environmental impact and economic
performance over the years, finding that cities with higher eco-efficiencies tend to show
considerable technological progress [30]. Furthermore, more eco-efficient cities focus on
cleaner production while developing the region’s economy [11].
Another study applied a super-slack-based measure (Super-SBM) model, which,
unlike DEA, considers undesirable production indicators, to assess the eco-efficiency and
its influential factors of Chinese cities [12]. In Wang et al. [13] the eco-efficiency of some
European countries was evaluated, as they consume a large share of global energy annually.
DEA and slacks-based measure (SBM) were used to evaluate the eco-efficiency of decision-
making units (DMUs) and the Malmquist Productivity Index (MPI) to analyze the efficiency
change, technological change and total productivity change for DMUs.
Masternak-Janus and Rybaczewska-Błażejowska [25] presented DEA for eco-efficiency
evaluation of Polish regions to promote sustainable transformation of the regions. In
another study, the authors adopted the same economic and environmental factors to assess
regional eco-efficiency by using a combined application of LCA and DEA, complementing
this with the input-oriented BCC model proposed by Banker, Charnes and Cooper’s [55,56]
so as to compare regions and countries with respect to their economic and environmental
performance.
Ecoefficiency evaluation methods have also been used in coal mine production, con-
sidering earnings before interest and taxes (EBIT) as an economic indicator [57], to measure
of eco-efficiency of world-class mining companies, using an enhanced optimization model
based on a directional distance function (DDF) [58,59].
In agricultural activities, EE assessment has been conducted to quantify the potential
environmental impacts through LCA combined with economic benefits using DEA, in
this way calculating eco-efficiency scores of an agricultural production [60]. In Cheng
et al. [61], the EE of agricultural activity in China was evaluated using the DEA-CCR
method developed by Charnes, Cooper and Rhodes [62].
In this same vein, to evaluate EE in agricultural production, Masuda [63] considered
eco-efficiency in his study as agricultural revenue divided by potential global warming.
In Song and Chen [64], a method combining water footprint (WF) analysis and stochastic
frontier approach (SFA) was developed to estimate the eco-efficiency of grain production
in China. The WF analysis was employed to quantify the resource consumption and
environmental impacts associated with grain production and the SFA model quantified the
capital present.

In industrial sectors, an eco-efficiency model assessment was provided by Egilmez
et al. [65]. Here, economic input–output life cycle assessment (EIO-LCA) and DEA were
integrated to analyze the eco-efficiency of manufacturing in the United States.
In Zheng and Peng [66], the eco-efficiency of energy-intensive industries, chains and
industry clusters was evaluated. Data from the chemical industry, building materials
industry, metallurgical industry and thermal energy industry from 2004 to 2015 were
analyzed. DEA was used, by means of a classical input-oriented CCR model, at given
outputs, focusing more attention on inputs.
Finally, studies that presented models to evaluate EE in industrial processes were
reviewed. Ng et al. [67] analyzed the eco-efficiency of technologies used to bond copper to
copper. Low temperature copper to copper bonding (LTCCB) technology was compared
with the conventional method (CM), using the cost of production (C) and the carbon
footprint (CFP) to assess eco-efficiency. A graphical representation was adopted to present
the eco-efficiency analysis performed for both processes.
In Leme Jr. et al. [68], eco-efficiency indicators were used to measure the performance
of a production system. To analyze the eco-efficiency of a machining center, a Lean-Green
model combining single minute exchange of die (SMED) with carbon footprint (CF) was
proposed. The eco-efficiency evaluation was performed by dividing the product value by
the environmental impact, aiming to reduce waste of time, resources and greenhouse gas
emissions.
In another study that evaluated the eco-efficiency of a manufacturing process, Lucato
et al. [24] presented a conceptual proposal in which indicators were combined to obtain a
single eco-efficiency index for a piece of equipment, making it possible to later expand and
evaluate eco-efficiency for a production area. The authors proposed a model that allows
evaluating the eco-efficiency of the process through a radar chart.
Based on the analysis and discussion above, we concluded that eco-efficiency can
be evaluated in different industrial sectors, in addition to evaluating products, processes
and services by applying an appropriate model consistent with the aim of the EE analysis.
To this end, in the following section, a procedure for choosing an eco-efficiency model is
proposed; the procedure is based on the AHP (analytic hierarchy process) decision-making
process.
In Table 2, the 40 selected articles discussed above are classified according to the
application sector, field of application (Product/Process/Service) and the type of model
used to assess EE.
Table 2. Articles surveyed based on the systematic literature review.
Application Sector Product/Process/Service Author/Year Model for EE Evaluation
Electrical and electronic
equipment
Product
Barba-Gutierrez; Adenso-Diaz;
Lozano (2009) [33]
DEA
Product Lozano et al. (2011) [23] DEA
Process/Service Kondo and Nakamura (2006) [34] WIO-LP
Civil construction
Product Tatari and Kucukvar (2011) [35] DEA
Product Tatari and Kucukvar (2012) [36] DEA
Product/Service Hu and Liu (2017) [40] DEA
Product Hamid and Shafiq (2016) [37]
Economic
Score/Environmental Score
Product Kim et al. (2016) [38] (EC + MC) /SL
Product Perera et al. (2017) [39]
Economic
Product/Service Belucio et al. (2021) [41] DEA

Table 2. Cont.
Application Sector Product/Process/Service Author/Year Model for EE Evaluation
Water resources and water
treatment
Process/Service Gómez et al. (2018) [45] DEA
Process/Service Molinos-Senante et al. (2016) [44] DEA
Food industry Process Sanjuan et al. (2011) [46]
Economic
Farming
Product/Process Hengen et al. (2016) [47]
Economic
Process/Service Rebolledo-Leiva et al. (2019) [49] DEA
Process Zhong et al. (2020) [60] DEA
Process/Service Cheng et al. (2012) [61] DEA
Service Song; Chen (2019) [64] SFA
Process/Service Masuda (2016) [63] LP-linear programming
Thermoelectric and thermal
power plants
Process/Service Guo et al. (2017) [69] DEA-SBM
Sustainability of cities and
regions
Product/Process/Service Xu et al. (2017) [11] DEA
Product/Process/Service Zhou et al. (2018) [12] DEA
Product/Process/Service Wang et al. (2020) [13] DEA
Product/Process/Service Xu and Chen (2015) [70] DEA
Product/Process/Service Yang and Zhang (2018) [30] DEA
Product/Process/Service Yin et al. (2014) [53] DEA
Product/Process/Service He et al. (2016) [71] DEA
Product/Process/Service Zhang (2020) [54] DEA
Product/Process/Service
Masternak-Janus and
Rybaczewska-Błażejowska (2016)
[25]
DEA
Product/Process/Service
Rybaczewska-Błażejowska and
Masternak-Janus (2018) [56]
DEA
Mining
Process Czaplicka-Kolarz et al. (2015) [57] EBIT/LCA
Service Oliveira et al. (2017) [58] DEA
Service Oliveira et al. (2015) [59] DEA
Manufacturing
Process Ng et al. (2014) [67] C/CFP
Process Egilmez et al. (2013) [65] EIO-LCA + DEA
Process/Service Zheng and Peng (2019) [66] DEA
Process Lucato et al. (2013) [24]
Product value/Ecological
influence
Product/Process Mendoza et al. (2019) [48]
Economic
Product/Process Vukelic et al. (2019) [50] TFC/TEI
Process Leme Jr. et al. (2018) [68]
Lean-Green Model:
SMED/CF
5. AHP-Based Procedure for Selecting an Appropriate Eco-Efficiency Model
As observed in the previous section, a wide variety of EE models are available in the
scientific literature. The choice of a proper model depends on several factors including the
area of application and the aim of the assessment. Consistently, it needs a systematic and
simple approach to address the choice of an appropriate EE model. To this end, the authors
propose a simplified procedure for choosing the eco-efficiency evaluation model to ensure
that the evaluation and its result are consistent with context and purpose. To this end, the
AHP (analytic hierarchy process) method proposed by Saaty [72] was used. It consists of
a hierarchical analysis method to support multicriteria decision making. Considered as
one of the main mathematical models used to support decision making, the AHP method
is used in situations to prioritize criteria and select alternatives, obtaining as a result a
hierarchical logic.

The AHP method is widely employed in decision making when involving sustainable
issues due to its comprehensibility in theory and simplicity in application, which makes
it the most popular comprehensive method [73]. Five main steps are included to solve
the multi-objective decision-making problem with the AHP algorithm. The first consists
of problem definition. The second consists of dividing the decision-making solution into
three levels: the target, criteria and program. The matrix on the criteria selected is thus
identified by pairwise comparison of the weights by means of eigenvector method (third
step). Afterwards, the single-level sequencing allowing estimation of the corresponding
weights is defined (fourth step). Finally, the general hierarchy ranking calculates the total
weights of each alternative [74].
In [75], the authors point out that the AHP method offers advantages when compared
to other classical decision-making methods, such as the possibility to decompose the
decision problem hierarchically and use subjective and verbal expressions to determine
the relative importance of the established criteria [76]. Furthermore, Jamwal et al. 2021
identified as strengths of AHP: ease of being adapted to the objectives, suitable and flexible
to support decision, can be applied in numerous areas and can measure consistency based
on expert opinion and provide a simple and flexible model to analyze problems [77].
Thus, AHP is a widely employed method due to its flexibility [78] and is considered a
simple and flexible technique when it is necessary to analyze criteria quantitatively and
qualitatively [77].
Wang and Yang analyzed three multicriteria decision making methods (AHP, SMART
and FM) for theoretical validity, predictive and perceived performance. As results, the
AHP was indicated as the preferred one in terms of performance and was considered easier
to apply.
For this study, the AHP method was considered due to its approach to the problem,
being possible to analyze the criteria (EE indicators) in a qualitative and in a quantitative
way. This is considered an adequate point, given the simplicity of using the AHP.
5.1. Fundamentals of Analytic Hierarchy Process—AHP
Mu and M. Pereyra-Rojas [79] summarize in six steps how a decision analysis should
be conducted by using the analytic hierarchy process (AHP):
i. Developing a decision model: decision is structured into a hierarchy of objectives,
criteria and alternatives;
ii. Derive the priorities (weights) for the established criteria: in this step, the importance
of the criteria is made by a pair comparison of criteria to derive their weights. The
consistency of the judgments provided by experts is quantified so as to review the
judgments and ensure that a reasonable level of consistency with respect to propor-
tionality and transitivity is met;
iii. Derive the local priorities (preferences) for the alternatives: at this point, the derivation
of the priorities or the alternatives in relation to each criterion separately is done (this
process is like the previous step, since the comparison of the alternatives in pairs with
respect to each criterion should be performed). Then, consistency is checked and
adjusted when necessary;
iv. Derive the general priorities (this is the model’s synthesis): in this step, all priorities
of the alternatives obtained must be combined as a weighted sum, considering the
weight of each criterion and establishing the general priorities of the alternatives. By
this analysis, the alternative that presents the highest overall priority is considered as
the best choice;
v. Sensitivity analysis: a study is conducted to analyze how changes in the weights of
the criteria can affect the result. The sensitivity analysis allows us to understand the
reason that led to the results obtained;
vi. Final decision making: decision-making is based on the results of the model synthesis
and the sensitivity analysis.

Based on the steps described above, it can be observed that this process does not
determine the decision to be made; in fact, the results should be interpreted as a project
of preferences and alternatives that are based on the level of importance obtained for the
defined criteria, in which the comparative judgments of the decision maker are considered.
This means that the AHP method allows determining the alternative that is most consistent
with the criteria and the level of importance that are given by the individual who is
conducting the decision-making process [72,79,80].
5.2. Applying AHP for Selecting the Appropriate Eco-Efficiency Model
The survey of EE models proposed in Section 4 allows identifying a set of models
for each of the nine production sectors: Electrical and electro-electronic equipment; Civil
construction; Water resources and water treatment; Food industry; Farming; Thermoelectric
and thermal power plants; Sustainability of cities and regions; Mining; and Manufacturing.
The survey addresses the first choice of the set of models: for example, an analyst
interested in evaluating the EE performance of a manufacturing process will initially focus
on the set of corresponding models classified as the ‘Manufacturing’ group in Table 2.
This directs them to a smaller number of model options, to be later analyzed by the
AHP decision making method, according to the established criteria, and thus define which
model is the most appropriate to be applied in his scenario.
Applying AHP requires carrying out a survey of all the eco-efficiency indicators (EE)
(environmental and economic) considered in each model of the sector concerned.
After these initial analyses, an AHP can be performed, according to the six steps
described earlier in Section 5.1 [79].
The first step is defining the hierarchy of the decision making process (Figure 3): At the
first level the objective of the decision is set as “Choose an eco-efficiency evaluation model”;
the second level of the hierarchy pertains the most suitable criteria identified to select the
EE model, i.e., the EE indicators belonging to the sector concerned; finally, the third level of
the hierarchy is located the models among which selecting the most appropriate for the
case under investigation.
Figure 3. Structure of the AHP method for choosing the EE model.
In the second step, judgments concerning the relative importance between each couple
of criteria are provided and priorities (normalized weights) calculated. It is worth noting
that the aim is to choose an EE evaluation model that is optimal for the scenario under
investigation. In the case of the example of the ‘Manufacturing’ sector, EE indicators to
be considered in the AHP were identified in article classified in Table 2; indicators are
summarized in Table 3.

Table 3. Alternatives and criteria to be evaluated in the AHP for the Manufacturing sector.
Author/Year
Model for EE Evaluation
(Alternatives)
Composition of EE Indicators (Criteria)
Ng et al. (2014) [67] A1 C/CFP
C is the cost of production
CFP is the carbon footprint
Egilmez et al. (2013) [65] A2 EIO-LCA + DEA (CCR)
EIO is economic input–output (total economic output)
GHG emissions, energy use, water use, hazardous waste
generation and toxic emissions)
Zheng and Peng (2019) [66] A3 DEA (CCR)
Resource efficiency (comprehensive energy consumption,
unit labor output, unit product capital input)
environmental efficiency (gas emission and waste)
Leme Jr. et al. (2018) [68] A4
Lean-Green Model:
SMED/CF
SMED is single minute exchange of die
CF is the carbon footprint
Lucato et al. (2013) [24] A5
Product value/ Ecological
influence
number of pieces
consumed energy
material resources of equipment consumed
Mendoza et al. (2019) [48] A6
Economic
raw material cost, manufacturing cost, transportation cost
and waste management cost)
Element abiotic depletion potential (ADP), fossil fuel
abiotic depletion potential (ADP), acidification potential
(AP), eutrophication potential (EP), freshwater aquatic
ecotoxicity potential (FAETP), global warming potential
(GWP), human toxicity potential (HTP), marine aquatic
ecotoxicity potential (MAETP), ozone depletion potential
(ODP), photochemical oxidant creation potential (POCP)
and terrestrial ecotoxicity potential (TETP)
Vukelic et al. (2019) [50] A7 TFC/TEI
TFC is the total fixture cost
TEI is the total environmental impact: human health,
ecosystem quality, climate change and resources
Once the initial survey of indicators is performed, the pairwise judgments of the
criteria (indicators extracted from the models) is carried out, according to weights, from 1
to 9, of the Saaty scale (see Table 4). Experts provide a pairwise comparison, Cij, of each
couple of criteria Ci vs. Cj, i, j = 1, n, i j. In each case of i j the reciprocal value
Cji = 1/Cij is set to guarantee consistency between the couple of criteria compared. Finally,
in case of i = j, Cii = 1, i = 1, n.
Table 4. Saaty’s pairwise comparison scale.
Verbal Judgment Numeric Value
Extremely important
9
8
Very strongly more important
7
6
Strongly more important
5
4
Moderately more important
3
2
Equally important 1
Source: [79].
In Table 5, an example with four criteria, i.e., four EE indicators, is considered. EE
indicator C2 is considered by experts as ‘very strongly more important’ than EE indicator
C1 according to the scale of Table 4. Consistently, C21 = 7, C12 = 1/7. The same line of
reasoning applies for each couple of criteria (EE indicators) considered.

Table 5. Example of pairwise judgments: four criteria (EE indicators) adopted.
Indicators
Criteria C1 C2 C3 C4
Weight
Ci
Normalized
Weight
Cin
Indicators
C1 1 1/7 1/9 1 2.25 0.05
C2 7 1 1/3 7 15.33 0.37
C3 9 3 1 9 22.00 0.53
C4 1 1/7 1/9 1 2.25 0.05
Total 41.84
The normalized weight Ci of the i-th criterium can be obtained as:
Cin =
∑n
j=1 Cij
∑n
i=1 ∑n
j=1 Cij
(2)
In the example developed, the normalized weight of each criterium is shown in the
last column.
Next, the consistency ratio (CR) of the judgments is calculated as in Equation (3) [79]:
CR =
CI
RI
(3)
i.e., as the ratio between the level of coherence (CI–Consistency Index) provided in the
judgements (level of importance attributed by pairwise comparison) over the random
index (RI) that would be achieved in a limit case where the judgements were provided by
a pure random judgement generation; RI only depends on the order of the comparison
matrix (Table 5) and can be obtained by numerical simulation where random judgements
in the scale of Saaty are generated.
The coherence index (CI) can be calculated as [79]:
CI =
(λmax − n)
(n − 1)
(4)
where
n is the order of the comparison matrix;
λmax is the maximum eigenvalue of the comparison matrix as described in Mu and
Pereyra–Rojas [79], which can be determined by averaging on the n criteria the ratio of
vectorial product of the i-th vector of the judgement matrix and the vector of normalized
weights (Table 5) over the i-th component of the vector of normalized weights as:
λmax =
Average
{i=1,n}
(
Di ∗ Cin

Cin
)
(5)
where
Di is the i-th row vector of the judgment matrix;
Cin is the vector of normalized weights;
Cin is the i-th component of the vector of the normalized weights.
CR has the purpose of analyzing whether the judgment was made in a way that
ensures a reasonable level of consistency in terms of proportionality and transitivity [79].
To calculate the RI value, according to Saaty, for a square matrix of order 4 is equal to
0.90. Therefore, the researcher should use the tabulated RI value according to the order of
his matrix. Following the example presented in Table 5, calculations were made to analyze
consistency (CR–Consistency Ratio), using Equations (3)–(5). Results are in Table 6.

Table 6. Criteria consistency analysis (EE indicators).
Ci λmax =
n
(Di*Cin)
Cin
o
λmax (Mean) IR CI CR
C1 4.06 4.16 0.90 0.053 0.058
C2 3.54
C3 4.94
C4 4.06
The acceptance of the coherence of judgement expressed in Table 5 is checked by
verifying that CR is lower than an acceptable limit value (usually set equal to 0.10). If it is
higher, a new trial should be performed. In the case example of Table 6, it has a consistency
of 0.058 0.10 which means that the judgment performed is coherent.
In the third step, judgments must be made for the alternatives: considering the same
example cited above, for this step the seven models for EE evaluation (ranked A1 to A7,
Table 3) are the seven alternatives that should be judged, following the same procedure
described in step ii. The judgments should be based on preferences for each EE indicator
(criterion) for each model. Thus, the seven alternatives for each criterion (EE indicator)
must be judged separately, that is, seven matrices must be built.
The weights for the alternatives must be placed in relation to each criterion and then
the weights must be normalized. The consistency of each matrix can be checked, by using
Equations (3)–(5) where the judgment matrix presented (Table 7) refers to the judgments of
the alternative models.
Table 7. Pairwise judgment of alternatives (EE models) and consistency (CR) analysis for criterion (C1).
Alternatives
C1 A1 A2 A3 A4
Weight
Ca
Normalized
Weight
Ci,an
λmax =

(Di*Ci,an)
Ci,an
λmax
(Mean)
IR CI CR
A1 1 1/5 1/2 1/7 1.84 0.06 4.13 4.12 0.90 0.040 0.045
A2 5 1 2 1 9.00 0.30 4.51
A3 2 1/2 1 1/6 3.67 0.12 3.91
A4 7 1 6 1 15.00 0.51 3.93
Tot 29.51
As an example, we considered analyzing four alternatives (A1 to A4). The couples
of alternatives were subject to pairwise comparison with reference to each of the four
criteria (C1 to C4) (previously analyzed in Table 5): the decision maker must consider how
important each criterion is for each alternative. In the example developed, when judging
the preference of the first two alternatives (A1 and A2) in relation to criterion C1, it is
evaluated that alternative A2 is strongly important (scale 5) over alternative A1, then A2 in
relation to alternative A1 is weighted with value 5 on the Saaty scale; consistently, A1 in
relation to A2 weighted by the reciprocal value 1/5. The complete comparison matrix of
the four alternatives (the EE models) evaluated each other with reference to criterion C1 is
in Table 7.
Tables 8–10 show the comparison matrix of the four alternatives compared each other
with reference to the criteria C2, C3 and C4, respectively.

Table 8. Pairwise judgment of alternatives and consistency analysis for criterion (C2).
Alternatives
C1 A1 A2 A3 A4
Weight
Ca
Normalized
Weight
Ci,an
λmax =

(Di*Ci,an)
Ci,an
λmax
(Mean)
IR CI CR
A1 1 1/4 5 2 8.25 0.22 3.67 4.10 0.90 0.033 0.037
A2 4 1 9 9 23.00 0.63 4.56
A3 1/5 1/9 1 1/2 1.81 0.05 4.32
A4 1/2 1/9 2 1 3.61 0.10 3.85
Tot 36.67
Alternatives
C1 A1 A2 A3 A4
Weight
Ca
Normalized
Weight
Ci,an
λmax =

(Di*Ci,an)
Ci,an
λmax
(Mean)
IR CI CR
A1 1 1/6 1/2 1/2 2.17 0.09 3.96 4.02 0.90 0.008 0.009
A2 6 1 2 3 12.00 0.52 3.92
A3 2 1/2 1 1 4.50 0.20 4.26
A4 2 1/3 1 1 4.33 0.19 3.96
Tot 23.00
Alternatives
C1 A1 A2 A3 A4
Weight
Ca
Normalized
Weight
Ci,an
λmax =

(Di*Ci,an)
Ci,an
λmax
(mean)
IR CI CR
A1 1 1/6 1/4 1/4 1.67 0.07 4.10 4.18 0.90 0.059 0.066
A2 6 1 1/2 1 8.50 0.34 3.47
A3 4 2 1 1 8.00 0.32 4.83
A4 4 1 1 1 7.00 0.28 4.31
Tot 25.17
In the fourth step, the decision maker analyzes the general priority of the alternatives
(global performance) for the choice of the most adequate EE model. After performing
the judgments, checking consistency, making the weighted total with the weights and
determining the normalized weights for the criteria and alternatives, it is possible to
obtain the overall priorities, defined in this study as overall performance, followed by
the overall ranking (Table 11). To obtain the results to be filled in Table 11 we adopted
the normalized weights of the criteria (Cin: Table 5) and alternatives (Ci,: Tables 7–10).
The global performance is determined by the total sum of the corresponding row. The
alternative with the highest overall performance will have the best score.
Table 11. Overall performance of the alternatives for each criterion analyzed in the example.
C1 C2 C3 C4
Overall
Performance
Classification
A1 0.003 0.082 0.050 0.004 0.139 4
A2 0.016 0.230 0.274 0.018 0.539 1
A3 0.007 0.018 0.103 0.017 0.145 3
A4 0.027 0.036 0.099 0.015 0.178 2

In the numerical example, the value of 0.003 (A1 × C1) was obtained by multiplying
the 0.05 (Cin: Table 5) by 0.06 (Ci,an: Table 7). The same approach was followed to complete
the final matrix in Table 11.
The overall performance was obtained by summing up the values of each row.
The result obtained allows the decision maker to identify the alternative A2 as the
best choice, i.e., the most appropriate eco-efficiency evaluation model to be applied.
Fifth step consists of sensitivity analysis of the choice of alternative (EE model).
To perform a sensitivity analysis, the decision maker changes criteria weights (by
changing the judgments, i.e., changing the scale weight assigned to each criterion) and
seeing how they change the overall performance of the alternatives.
The last step consists of the choice of the EE assessment model: After analyzing the
result of the overall priority (step iii) and sensitivity (step iv), the final decision is made
and the EE assessment model to evaluate the scenario (e.g., a given production process) is
obtained. In the view of the proposed procedure for selecting the ideal EE model, the flow
chart in Figure 4 contains all the steps to be followed sequentially.
Figure 4. Steps in the procedure to choose the EE evaluation model.
Due to the diverse EE evaluation models existing in the literature, it is understood
that the analyses proposed in this article can facilitate the understanding of how the choice
of a model that is more aligned with the area to be explored by the decision maker should
be made. AHP is an established decision-making method, clear and with well-defined
steps to be followed. The judgments and calculations that should be made in each step
of the AHP can be seen in greater depth in [79–81], since the purpose of this work was
to show how the structure should be outlined, in relation to the objective of the decision
(choice of EE model), the criteria (EE indicators) and the alternatives (EE models).
6. Conclusions
Eco-efficiency is an approach used to evaluate environmental and economic aspects
in an integrated manner, through the identification of indicators capable of providing
sufficient data to be considered when using an evaluation model.
By the systematic review performed, it was possible to identify the types of models
available in the scientific literature and how they are applied in different industrial sectors.
It was also possible to identify which eco-efficiency indicators are used in the models

and which are necessary for analysis according to the area to be explored to evaluate the
eco-efficiency of a product or process. EE evaluations require, among other things, the iden-
tification of material and energy involved in the processes which generate environmental
impacts as well as their costs.
Greater environmental concerns and pressures from governments and society mean
industries are seeking to demonstrate that their processes increasingly cause less impact
on the environment. Eco-efficiency can verify such impacts, encompassing not only the
environmental impact, but also the cost involved in the process.
Finally, the proposal of a procedure for choosing the EE evaluation model was carried
out with the objective of facilitating the understanding of how the decision-making process
should be carried out for the selection of an EE model that guarantees an adequate analysis,
according to the area being evaluated. The AHP method can provide an exploration in a
systematic way, with judgments that come closest to a convenient choice. This procedure is
considered an affordable alternative for conducting analysis, which can be used in many
different areas to evaluate eco-efficiency. It is pointed out that studies such as this one can
contribute to researchers continued exploration of sustainability-related issues necessary
for the development of a conscious and industrially developed society.
Author Contributions: Conceptualization, M.J.X.B.; methodology, M.V.J.; formal analysis, G.M.;
investigation, M.J.X.B.; writing—original draft preparation, M.J.X.B., M.V.J., G.M., F.F.; writing—
review and editing, M.J.X.B., F.F.; visualization, F.F.; supervision, M.V.J. All authors have read and
agreed to the published version of the manuscript.
Funding: This research was funded by CAPES—Brazilian Coordination for the Improvement of
Higher Education Personnel, Financing Code 001.
Institutional Review Board Statement: Not applicable.
Informed Consent Statement: Not applicable.
Data Availability Statement: Data presented in this study are available upon request to the corre-
sponding authors.
Acknowledgments: The authors appreciate the financial support from CAPES, Brazil—Financing
Code 001.
Conflicts of Interest: The authors declare no conflict of interest.
References
1. Taheri, K.; Gadow, R.; Killinger, A. Exergy Analysis as a Developed Concept of Energy Efficiency Optimized Processes: The Case
of Thermal Spray Processes. Procedia CIRP 2014, 17, 511–516. [CrossRef]
2. Bidoki, S.M.; Wittlinger, R.; Alamdar, A.A.; Burger, J. Eco-efficiency analysis of textile coating materials. J. Iran. Chem. Soc. 2006, 3,
351–359. [CrossRef]
3. Saling, P.; Kicherer, A.; Dittrich-Krämer, B.; Wittlinger, R.; Zombik, W.; Schmidt, I.; Schrott, W.; Schmidt, S. Eco-efficiency analysis
by basf: The method. Int. J. Life Cycle Assess. 2002, 7, 203–218. [CrossRef]
4. Uhlman, B.W.; Saling, P. Measuring and communicating sustainability through Eco-efficiency analysis. Chem. Eng. Prog. 2010,
106, 12.
5. Bradlee, C.A.; Saling, P.; Uhlman, B. Submission for NSF Protocol P352 Validation and Verification of Eco-efficiency Analyses,
Part A. BASF’s Eco-Effic. Anal. Methodol. 2009, 1, 1–38.
6. Wall-Markowski, C.A.; Kicherer, A.; Wittlinger, R. Eco-efficiency: Inside BASF and beyond. Manag. Environ. Qual. An Int. J. 2005,
16, 2.
7. Lozano, F.J.; Lozano, R. Assessing the potential sustainability benefits of agricultural residues: Biomass conversion to syngas for
energy generation or to chemicals production. J. Clean. Prod. 2018, 172, 4162–4169. [CrossRef]
8. Suter, G.W. Multi-Criteria Decision Analysis: Environmental Applications and Case Studies; CRC Press: Boca Raton, FL, USA, 2011.
9. Talukder, B.; Blay-Palmer, A.; Hipel, K.W.; VanLoon, G.W. Elimination Method of Multi-Criteria Decision Analysis (MCDA): A
Simple Methodological Approach for Assessing Agricultural Sustainability. Sustainability 2017, 9, 287. [CrossRef]
10. Van Caneghem, J.; Block, C.; Cramm, P.; Mortier, R.; Vandecasteele, C. Improving eco-efficiency in the steel industry: The
ArcelorMittal Gent case. J. Clean. Prod. 2010, 18, 807–814. [CrossRef]
11. Xu, T.; Gao, P.; Yu, Q.; Fang, D. An Improved Eco-Efficiency Analysis Framework Based on Slacks-Based Measure Method.
Sustainability 2017, 9, 952. [CrossRef]

12. Zhou, C.; Shi, C.; Wang, S.; Zhang, G. Estimation of eco-efficiency and its influencing factors in Guangdong province based on
Super-SBM and panel regression models. Ecol. Indic. 2018, 86, 67–80. [CrossRef]
13. Wang, C.-N.; Hsu, H.-P.; Wang, Y.-H.; Nguyen, T.-T. Eco-Efficiency Assessment for Some European Countries Using Slacks-Based
Measure Data Envelopment Analysis. Appl. Sci. 2020, 10, 1760. [CrossRef]
14. Mokarram, M.; Mokarram, M.J.; Gitizadeh, M.; Niknam, T.; Aghaei, J. A novel optimal placing of solar farms utilizing multi-
criteria decision-making (MCDA) and feature selection. J. Clean. Prod. 2020, 261, 121098. [CrossRef]
15. Sarraf, R.; McGuire, M.P. Integration and comparison of multi-criteria decision making methods in safe route planner. Expert Syst.
Appl. 2020, 154, 113399. [CrossRef]
16. Oliveira, M.; Fontes, D.; Pereira, T. Multicriteria Decision Making: A Case Study in the Automobile Industry. Ann. Manag. Sci.
2014, 3, 109–128. [CrossRef]
17. Ferrari, S.; Zagarella, F.; Caputo, P.; Bonomolo, M. Assessment of tools for urban energy planning. Energy 2019, 176, 544–551.
[CrossRef]
18. Vakilipour, S.; Sadeghi-Niaraki, A.; Ghodousi, M.; Choi, S.-M. Comparison between Multi-Criteria Decision-Making Methods
and Evaluating the Quality of Life at Different Spatial Levels. Sustainability 2021, 13, 4067. [CrossRef]
19. Council for Sustainable Development (WBCSD). Measuring eco-efficiency: A guide to reporting company performance WBCSD.
2000, pp. 1–39. Available online: https://www.gdrc.org/sustbiz/measuring.pdf (accessed on 30 September 2021).
20. WBCSD. Introduction to the learning module WBCSD. 2006, pp. 1–231. Available online: https://docs.wbcsd.org/2006/08/
EfficiencyLearningModule.pdf (accessed on 30 September 2021).
21. Zhang, B.; Bi, J.; Fan, Z.; Yuan, Z.; Ge, J. Eco-efficiency analysis of industrial system in China: A data envelopment analysis
approach. Ecol. Econ. 2008, 68, 306–316. [CrossRef]
22. Diebold, W.; Schmidheiny, S. Changing Course: A Global Business Perspective on Development and the Environment. Foreign
Aff. 1992, 71, 202. [CrossRef]
23. Lozano, S.; Adenso-Diaz, B.; Barba-Gutiérrez, Y. Russell non-radial eco-efficiency measure and scale elasticity of a sample of
electric/electronic products. J. Frankl. Inst. 2011, 348, 1605–1614. [CrossRef]
24. Lucato, W.C.; Júnior, M.V.; Santos, J.C.D.S. Measuring the ecoefficiency of a manufacturing process: A conceptual proposal.
Manag. Environ. Qual. Int. J. 2013, 24, 755–770. [CrossRef]
25. Masternak-Janus, A.; Rybaczewska-Błażejowska, M. Comprehensive Regional Eco-Efficiency Analysis Based on Data Envelop-
ment Analysis: The Case of Polish Regions. J. Ind. Ecol. 2016, 21, 180–190. [CrossRef]
26. Koskela, M.; Vehmas, J. Defining Eco-efficiency: A Case Study on the Finnish Forest Industry. Bus. Strat. Environ. 2012, 21,
27. Kitchenham, B. Guidelines for performing systematic literature reviews in software engineering. Tech. Rep. EBSE 2007, 1, 1–65.
28. Brereton, P.; Kitchenham, B.; Budgen, D.; Turner, M.; Khalil, M. Lessons from applying the systematic literature review process
within the software engineering domain. J. Syst. Softw. 2006, 80, 571–583. [CrossRef]
29. Fabbri, S.; Silva, C.; Hernandes, E.M.; Octaviano, F.; Di Thommazo, A.; Belgamo, A. Improvements in the StArt tool to better
support the systematic review process. ACM Int. Conf. Proceeding Ser. 2016, 21, 5. [CrossRef]
30. Yang, L.; Zhang, X. Assessing regional eco-efficiency from the perspective of resource, environmental and economic performance
in China: A bootstrapping approach in global data envelopment analysis. J. Clean. Prod. 2018, 173, 100–111. [CrossRef]
31. Kuosmanen, T.; Kortelainen, M. Measuring Eco-efficiency of Production with Data Envelopment Analysis. J. Ind. Ecol. 2005, 9,
59–72. [CrossRef]
32. Lorenzo-Toja, Y.; Vázquez-Rowe, I.; Chenel, S.; Marín, D.; Moreira, M.T.; Feijoo, G. Eco-efficiency analysis of Spanish WWTPs
using the LCA + DEA method. Water Res. 2015, 68, 651–666. [CrossRef]
33. Gutiérrez, Y.B.; Adenso-Díaz, B.; Lozano, S. Eco-Efficiency of Electric and Electronic Appliances: A Data Envelopment Analysis
(DEA). Environ. Model. Assess. 2008, 14, 439–447. [CrossRef]
34. Kondo, Y.; Nakamura, S. Waste input–output linear programming model with its application to eco-efficiency analysis. Econ.
Syst. Res. 2005, 17, 393–408. [CrossRef]
35. Tatari, O.; Kucukvar, M. Evaluating Eco-Efficiency of Construction Materials: A Frontier Approach. Congr. Comput. Civ. Eng. Proc.
2011, 736–743. [CrossRef]
36. Tatari, O.; Kucukvar, M. Eco-Efficiency of Construction Materials: Data Envelopment Analysis. J. Constr. Eng. Manag. 2012, 138,
37. Hamid, M.S.A.; Shafiq, N. Eco-efficiency index model for reinforced concrete structural design: Malaysia case study. ARPN J. Eng.
Appl. Sci. 2016, 11, 8.
38. Kim, T.; Tae, S.; Chae, C.U.; Lee, K. Proposal for the Evaluation of Eco-Efficient Concrete. Sustainability 2016, 8, 705. [CrossRef]
39. Perera, P.; Hewage, K.; Alarti, M.S.; Sadiq, R. Eco-efficiency analysis of recycled material for residential construction: A case
study of Okanagan (BC). In Proceedings of the 6th CSCE-CRC International Construction Specialty Conference 2017-Held as Part
of the Canadian Society for Civil Engineering Annual Conference and General Meeting 2017, Vancouver, BC, Canada, 31 May–3
June 2017; Volume 1.
40. Hu, X.; Liu, C. Slacks-based data envelopment analysis for eco-efficiency assessment in the Australian construction industry.
Constr. Manag. Econ. 2017, 35, 11–12.

41. Belucio, M.; Rodrigues, C.; Antunes, C.H.; Freire, F.; Dias, L.C. Eco-efficiency in early design decisions: A multimethodology
approach. J. Clean. Prod. 2020, 283, 124630. [CrossRef]
42. Liu, Y.; Sun, C.; Xu, S. Eco-Efficiency Assessment of Water Systems in China. Water Resour. Manag. 2013, 27, 4927–4939. [CrossRef]
43. Mehmeti, A.; Todorovic, M.; Scardigno, A. Assessing the eco-efficiency improvements of Sinistra Ofanto irrigation scheme. J.
Clean. Prod. 2016, 138, 208–216. [CrossRef]
44. Molinos-Senante, M.; Gémar, G.; Gómez, T.; Caballero, R.; Sala-Garrido, R. Eco-efficiency assessment of wastewater treatment
plants using a weighted Russell directional distance model. J. Clean. Prod. 2016, 137, 1066–1075. [CrossRef]
45. Gómez, T.; Gémar, G.; Molinos-Senante, M.; Sala-Garrido, R.; Caballero, R. Measuring the eco-efficiency of wastewater treatment
plants under data uncertainty. J. Environ. Manag. 2018, 226, 484–492. [CrossRef]
46. Sanjuan, N.; Ribal, J.; Clemente, G.; Fenollosa, M.L. Measuring and Improving Eco-efficiency Using Data Envelopment Analysis:
A Case Study of Mahón-Menorca Cheese. J. Ind. Ecol. 2011, 15, 4.
47. Hengen, T.J.; Sieverding, H.L.; Cole, N.A.; Ham, J.M.; Stone, J.J. Eco-Efficiency Model for Evaluating Feedlot Rations in the Great
Plains, United States. J. Environ. Qual. 2016, 45, 4.
48. Mendoza, J.M.F.; D’Aponte, F.; Gualtieri, D.; Azapagic, A. Disposable baby diapers: Life cycle costs, eco-efficiency and circular
economy. J. Clean. Prod. 2019, 211, 455–467.
49. Rebolledo-Leiva, R.; Angulo-Meza, L.; Iriarte, A.; González-Araya, M.C.; Vásquez-Ibarra, L. Comparing two CF+DEA methods
for assessing eco-efficiency from theoretical and practical points of view. Sci. Total. Environ. 2018, 659, 1266–1282. [CrossRef]
50. Vukelic, D.; Agarski, B.; Budak, I.; Simunovic, G.; Buchmeister, B.; Jakovljevic, Z.; Tadic, B. Eco-Design of Fixtures Based on Life
Cycle and Cost Assessment. Int. J. Simul. Model. 2019, 18, 72–85. [CrossRef]
51. ISO. 14040: Environmental management–life cycle assessment—Principles and framework. Int. Organ. Stand. 2006, 3, 28.
52. Finnveden, G.; Hauschild, M.Z.; Ekvall, T.; Guinée, J.; Heijungs, R.; Hellweg, S.; Koehler, A.; Pennington, D.; Suh, S. Recent
developments in Life Cycle Assessment. J. Environ. Manag. 2009, 91, 1–21. [CrossRef]
53. Yin, K.; Wang, R.; An, Q.; Yao, L.; Liang, J. Using eco-efficiency as an indicator for sustainable urban development: A case study
of Chinese provincial capital cities. Ecol. Indic. 2014, 36, 665–671. [CrossRef]
54. Zhang, X. Estimation of eco-efficiency and identification of its influencing factors in China’s Yangtze River Delta urban agglomer-
ations. Growth Chang. 2020, 51, 2.
55. Banker, R.D.; Charnes, A.; Cooper, W.W. Some Models for Estimating Technical and Scale Inefficiencies in Data Envelopment
Analysis. Manag. Sci. 1984, 30, 9.
56. Rybaczewska-Błażejowska, M.; Masternak-Janus, A. Eco-efficiency assessment of Polish regions: Joint application of life cycle
assessment and data envelopment analysis. J. Clean. Prod. 2018, 172, 1180–1192. [CrossRef]
57. Czaplicka-Kolarz, K.; Burchart-Korol, D.; Turek, M.; Borkowski, W. Model of Eco-Efficiency Assessment of Mining Production
Processes. Arch. Min. Sci. 2015, 60, 2.
58. Oliveira, R.; Camanho, A.; Zanella, A. Eco-efficiency assessment at firm level: An application to the mining sector. In Operations
Research and Big Data: IO2015-XVII Congress of Portuguese Association of Operational Research (APDIO); Springer: Berlin/Heidelberg,
Germany, 2015; Volume 15.
59. Oliveira, R.; Camanho, A.; Zanella, A. Expanded eco-efficiency assessment of large mining firms. J. Clean. Prod. 2017, 142,
2364–2373. [CrossRef]
60. Zhong, F.; Jiang, D.; Zhao, Q.; Guo, A.; Ullah, A.; Yang, X.; Cheng, Q.; Zhang, Y.; Ding, X. Eco-efficiency of oasis seed maize
production in an arid region, Northwest China. J. Clean. Prod. 2020, 268, 122220.
61. Cheng, C.Y.; Ren, J.M.; Wang, R.S.; Liu, F. Eco-Efficiency Assessment of Farming Activity in China. Adv. Mater. Res. 2011, 361-363,
1776–1779. [CrossRef]
62. Charnes, A.; Cooper, W.W.; Rhodes, E. Measuring the efficiency of decision making units. Eur. J. Oper. Res. 1978, 2, 429–444.
[CrossRef]
63. Masuda, K. Optimization Model for Mitigating Global Warming at the Farm Scale: An Application to Japanese Rice Farms.
Sustainability 2016, 8, 593. [CrossRef]
64. Song, J.; Chen, X. Eco-efficiency of grain production in China based on water footprints: A stochastic frontier approach. J. Clean.
Prod. 2019, 236. (accessed on 30 September 2021). [CrossRef]
65. Gumus, S.; Egilmez, G.; Kucukvar, M.; Park, Y.S. Integrating expert weighting and multi-criteria decision making into eco-
efficiency analysis: The case of US manufacturing. J. Oper. Res. Soc. 2016, 67, 616–628. [CrossRef]
66. Zheng, J.; Peng, X. Does an Ecological Industry Chain Improve the Eco-Efficiency of an Industrial Cluster? Based on Empirical
Study of an Energy-Intensive Industrial Cluster in China. Sustainability 2019, 11, 1651. [CrossRef]
67. Ng, R.; Nai, S.; Chan, H.L.I.; Shi, C.W.P.; Song, B. Comparative Eco-efficiency Analyses of Copper to Copper Bonding Technologies.
Procedia CIRP 2014, 15, 96–104. [CrossRef]
68. Leme, R.D.; Nunes, A.O.; Costa, L.B.M.; Silva, D.A.L. Creating value with less impact: Lean, green and eco-efficiency in a
metalworking industry towards a cleaner production. J. Clean. Prod. 2018, 196, 517–534. [CrossRef]
69. Guo, Y.; Liu, W.; Tian, J.; He, R.; Chen, L. Eco-efficiency assessment of coal-fired combined heat and power plants in Chinese
eco-industrial parks. J. Clean. Prod. 2017, 168, 963–972. [CrossRef]
70. Xu, T.; Chen, Y. Eco-efficiency assessment of Wuhan based on data envelopment analysis approach. In Proceedings of the 2015
27th Chinese Control and Decision Conference CCDC, Qingdao, China, 23–25 May 2015.

71. He, J.; Wan, Y.; Feng, L.; Ai, J.; Wang, Y. An integrated data envelopment analysis and emergy-based ecological footprint
methodology in evaluating sustainable development, a case study of Jiangsu Province, China. Ecol. Indic. 2016, 70, 23–34.
72. Karayalcin, I.I. The analytic hierarchy process: Planning, priority setting, resource allocation. Eur. J. Oper. Res. 1982, 9, 1.
73. Wang, J.J.; Jing, Y.Y.; Zhang, C.F.; Zhao, J.H. Review on multi-criteria decision analysis aid in sustainable energy decision-making.
Renew. Sustain. Energy Rev. 2009, 13, 9.
74. Ge, Y. Research on green suppliers’ evaluation based on AHP genetic algorithm. In Proceedings of the 2009 International
Conference on Signal Processing Systems, Singapore, 15–17 May 2009.
75. Sadok, W.; Angevin, F.; Bergez, J.; Bockstaller, C.; Colomb, B.; Guichard, L.; Reau, R.; Doré, T. Ex ante assessment of the
sustainability of alternative cropping systems: Implications for using multi-criteria decision-aid methods. A review. Agron.
Sustain. Dev. 2008, 28, 163–174. [CrossRef]
76. Macharis, C.; Springael, J.; de Brucker, K.; Verbeke, A. PROMETHEE and AHP: The design of operational synergies in multicriteria
analysis. Eur. J. Oper. Res. 2004, 153, 2.
77. Jamwal, A.; Agrawal, R.; Sharma, M.; Kumar, V. Review on multi-criteria decision analysis in sustainable manufacturing decision
making. Int. J. Sustain. Eng. 2020, 14, 202–225. [CrossRef]
78. Kurttila, M.; Pesonen, M.; Kangas, J.; Kajanus, M. Utilizing the analytic hierarchy process (AHP) in SWOT analysis—A hybrid
method and its application to a forest-certification case. For. Policy Econ. 2000, 1, 41–52. [CrossRef]
79. Mu, E.; Pereyra-Rojas, M. Practical Decision Making using Super Decisions v3. 2018. In An Introduction to the Analytic Hierarchy
Process; Springer: Berlin/Heidelberg, Germany. [CrossRef]
80. Saaty, T.; Vargas, L. Models, Methods, Concepts Applications of the Analytic Hierarchy Process; Springer: Boston, MA, USA, 2012.
81. Labib, A. Introduction to the Analytic Hierarchy Process. In Learning from Failures; Elsevier: Amsterdam, The Netherlands, 2014;
pp. 33–44. [CrossRef]

sustainability-13-12121-v2.pdf

Related slideshows

More Related Content

sustainability-13-12121-v2.pdf