APRESENTAÇÃO BALAS TESE (1) (1).PDF

Alexandre Bentes

Jorge Carneiro

Herbert Kimura

Jorge Ferreira da Silva

Multidimensional assessment of performance of organizational units: Integrating BSC and AHP methodologies

1

Multidimensional assessment ofperformance of organizational units: Integrating BSC and AHP methodologies

in a Brazilian telecom company

Alexandre Bentes, PUC-Rio, Brazil

Jorge Carneiro, PUC-Rio, Brazil

Herbert Kimura, Mackenzie, Brazil

Jorge Ferreira da Silva, PUC-Rio, Brazil

IAG Business School,

Pontifical Catholic University of Rio de Janeiro

March 24 – 26, 2010

Barcelona, Spain

Alexandre Bentes

Jorge Carneiro

Herbert Kimura



2

�Appropriate assessment of organizational performance is essential for firms

�The multidimensional nature of organizational performance poses challenges to researchers and practitioners

�The performance of functional areas within an organization is an instance of the concept of organizational performance in general

�Several studies have

� used BSC to assess the performance of firms and of functional units within organizations

� used AHP to rank decision alternatives

�But few have used both � This is what is new and contributive in our study

Introduction

Alexandre Bentes

Jorge Carneiro

Herbert Kimura



3

�Verify whether an integrated approach – that combines the multidimensional perspective of the Balanced Scorecard (BSC) with the framework for multi-criteria ranking of decision alternatives provided by the Analytic Hierarchy Process (AHP) – can be useful for the assessment of the (relative) performance of functional units within an organization.

�Extend the external validity of both the BSC and the AHP frameworks by applying those frameworks to another industry (telecommunications) and another setting (Brazilian environment and managers) not often researched

Objectives of the study

Multiple perspectives of organizational performance

Multi-criteria ranking of decision alternatives

Multidimensional assessment and ranking of organizational performance

Alexandre Bentes

Jorge Carneiro

Herbert Kimura



4

�In this study the overall performance of three functional areas of the financial department of a Brazilian Telecommunications company was rank-ordered by taking into consideration:

� the relative degree of importance of (four) distinct perspectives of organizational performance,

� the relative degree of importance of performance indicators within each perspective, and

� the (relative) performance of the (three) functional areas in each performance indicator

Objectives of the study (cont.)

Alexandre Bentes

Jorge Carneiro

Herbert Kimura



5

�Organizational performance and the Balanced Scorecard (BSC)

�Comparison and decision-making based on multiple criteria

�The Analytic Hierarchy Process (AHP)

Theoretical Framework

Alexandre Bentes

Jorge Carneiro

Herbert Kimura



6

Organizational performance and the Balanced Scorecard (BSC)

�Organizational performance is a complex and multi-faceted phenomenon (Cameron, 1986; Chakravarthy, 1986; Venkatraman and Ramanujam, 1986)

�Kaplan and Norton (1996) proposed that organizational performance be simultaneously assessed from distinct, but complementary, perspectives, which were originally detailed as:

� Financial

� Customer

� Internal Business Processes

� Innovation and Learning


Financial

Vision and

Strategy

Innovation and

Learning

Internal Business

ProcessesCustomer

Alexandre Bentes

Jorge Carneiro

Herbert Kimura



7

Organizational performance and the Balanced Scorecard (BSC) (cont.)

�While BSC provides a balance (Kaplan and Norton, 1992)

� between short-term and long-term objectives,

� between financial and non-financial measures,

� between lagging and leading indicators, and

� between internal and external performance perspectives”

�BSC approaches, nonetheless, also present some challenges:

� Overload of information

� Necessity to reach some synthesized judgment


Alexandre Bentes

Jorge Carneiro

Herbert Kimura



8

Comparison and decision-making based on multiple criteria

�Some decision-making problems involve the choice among alternatives which must take into account multiple criteria which may present some degree of inconsistency with one another

� The values (performance) achieved in some criteria may suggest a distinct choice (decision) from the values in some other criteria

� Improvement of results under a given criterion may be obtainable only at the expense of another

� This may be particularly true in the assessment of organizational performance and the choice of the best performing unit (or the rank-ordering of the units under evaluation)


Alexandre Bentes

Jorge Carneiro

Herbert Kimura



9

The Analytic Hierarchy Process (AHP)

�A multi-criteria decision-making tool developed by Saaty (1980)

�Decomposition, comparative judgment, and synthesis

�Decomposition of a complex problem into a multilevel hierarchic structure of objectives, criteria and sub-criteria, and alternatives

�Series of pair-wise comparisons of judgments (of child items below a parent node, i.e., the relative contributions of the child items to the parent node)


Objective

Criteria

Sub-criteria

Alternatives

Alexandre Bentes

Jorge Carneiro

Herbert Kimura



10

The Analytic Hierarchy Process (AHP) (cont.)

�Synthesis in an eigenvector (priority vector)

� Judgments express the relative impact / importance of the elements in the hierarchy and are translated into numbers, whichindicate the relative importance (weight, priority) of an element over another in the determination of their respective higher hierarchical level

�Final weights of any given lowest element are calculated from a weighted (by degree of importance) average over all levels of the hierarchy and synthesize the multiple judgments (often not mutually consistent)

�Rank-ordering of alternatives

�Overall importance represents the relative (percent) contribution of a given indicator with respect to the global goal


Alexandre Bentes

Jorge Carneiro

Herbert Kimura



11

The Analytic Hierarchy Process (AHP) (cont.)

�Given the particularities of the present study, AHP has some advantages over other decision-making tools:

� Synthesis of multiple viewpoints (criteria) into a single unified result

� A consistency index can be calculated to assess the transitivityin judgments given by the paired comparisons

� Involvement of decision-makers who have to explicitly reveal their preferences

� Communication among decision-makers tends to lead to better decisions (reached through a process of conflict resolution until

some consensus or, at least, agreement is reached)


Alexandre Bentes

Jorge Carneiro

Herbert Kimura



12

�Units of analysis: three functional areas of the financial department of a Brazilian telecom company – Fraud unit, Collection unit, and Revenue Assurance unit

�Data collection procedures

�Data collected in 2009

� Interviewees and participants

� three senior mangers (one for each unit; all with long experience in the company),

� three supervisors (one for each unit),

� nine senior analysts (three for each unit), and

� the financial operations director.

� � The diversity in hierarchical levels was deemed important in order to reach a more consensual and “department-wide” opinion about performance that would somehow better represent the variety of views in the financial department.

Methods and data

Alexandre Bentes

Jorge Carneiro

Herbert Kimura



13

�Iterative and interactive process of data collection

� For the selection of dimensions and measures of organizational (functional) performance

− four dimensions and nine performance indicators chosen

� For the definition of the weights (degree of importance) of eachperformance dimension and of each indicator (within each dimension)

− Paired comparisons: How much more (less) important is this dimension / indicator as compared to the other one?

�Five rounds of discussion until some agreement was reached

�Role played by the investigator

�Convergent validity check

Methods and data

Alexandre Bentes

Jorge Carneiro

Herbert Kimura



14

Identify the best

performing unitobjective

criteriaFinancial

perspective

Ind A Ind B Ind C

Fraud unit

Customer

perspective

Innovation and

learning perspective

Internal business

process perspective

Ind D Ind E Ind F Ind G Ind H Ind I

Collection unit

Revenue

assurance unit

Fraud unit

Collection unit

Revenue

assurance unit

Fraud unit

Collection unit

Revenue

assurance unit

Fraud unit

Collection unit

Revenue

assurance unit

sub-criteria

alternatives

Methods and data

�The AHP tree

Alexandre Bentes

Jorge Carneiro

Herbert Kimura



15

Training and capability-building activitiesI

Improvement in employees’ capabilitiesHInnovation and

Learning

Number of new projects fully delivered in

the yearG

Number of improvement projects

implemented in the yearF

Internal Business

Processes

Satisfaction level of external clientsE

Satisfaction level of internal clientsDCustomer

Decrease in operating costsC

ROI of implemented projectsB

Recovered value or avoided lossA

Financial

DefinitionIndicatorBSC Perspective

Methods and data

�Indicators selected for the present study

Alexandre Bentes

Jorge Carneiro

Herbert Kimura



16

Financial Ind A Ind B Ind C

Ind A 1 4 7

Ind B 1/4 1 3

Ind C 1/7 1/3 1

0.71

0.21

0.08

Customer Ind D Ind E

Ind D 1 1/7

Ind E 7 1

0.12

0.88

Processes Ind F Ind G

Ind F 1 3

Ind G 1/3 1

0.75

0.25

Innovation

and LearningInd H Ind I

Ind H 1 5

Ind I 1/5 1

0.83

0.17

CR = 0.03

CR = 0.00

CR = 0.00

CR = 0.00

Application of the Analytic Hierarchy Process

�Relative importance,

�Normalized weights, and

�Consistency ratios

at the performance indicators (sub-criteria) level

Alexandre Bentes

Jorge Carneiro

Herbert Kimura



17

BSC

perspectivesFinance Customer Processes

Innovation

and Learning

Finance 1 5 4 5

Customer 1/5 1 3 3

Processes 1/4 1/3 1 2

Innovation

and Learning1/5 1/3 1/2 1

0.59

0.21

0.12

0.08

CR 0.09


�Relative importance,

�Normalized weights, and

�Consistency ratios

at the BSC perspectives (criteria) level

Alexandre Bentes

Jorge Carneiro

Herbert Kimura



18

0.71

0.21

0.08

x 0.59

0.42

0.12

0.05

Indicators A,B and C

=0.12

0.88

x 0.21 0.03

0.18

Indicators D and E

=

local

weights

global

weights

local

weights

global

weights

0.75

0.25

x 0.12 0.09

0.03

Indicators F and G

=

local

weights

global

weights

0.83

0.17

x 0.08 0.07

0.01

Indicators H and I

=

local

weights

global

weights


�From local weights to global weights of each indicator

weight of

parent

node

weight of

parent

node

weight of

parent

node

weight of

parent

node

Alexandre Bentes

Jorge Carneiro

Herbert Kimura



19

Ind. A Fraud CollectionRevenue

assurance

Fraud 1 1/7 1/2

Collection 7 1 5

Revenue

assurance2 1/5 1

0.09

0.74

0.17

Ind. B Fraud CollectionRevenue

assurance

Fraud1 4 3

Collection1/4 1 1/3

Revenue

assurance

1/3 3 1

0.61

0.12

0.27

Ind. C Fraud CollectionRevenue

assurance

Fraud1 1/4 1/2

Collection4 1 4

Revenue

assurance

2 1/4 1

0.13

0.66

0.21

CR = 0.01

CR = 0.06

CR = 0.05


�Relative degree of success in (performance indicators of) the financial perspective by each functional unit

Alexandre Bentes

Jorge Carneiro

Herbert Kimura



20

Ind D Fraud CollectionRevenue

assurance

Fraud1 1/2 3

Collection5 1 7

Revenue

assurance

1/3 1/7 1

0.19

0.73

0.08

Ind E Fraud CollectionRevenue

assurance

Fraud1 1/4 2

Collection4 1 3

Revenue

assurance

1/2 1/3 1

0.22

0.63

0.15

CR = 0.06 CR = 0.09


�Relative degree of success in (performance indicators of) the customer perspective by each functional unit

Alexandre Bentes

Jorge Carneiro

Herbert Kimura



21

Ind. F Fraud CollectionRevenue

assurance

Fraud1 1/2 1/5

Collection2 1 1/4

Revenue

assurance

5 4 1

0.12

0.20

0.68

Ind. G Fraud CollectionRevenue

assurance

Fraud1 1/2 1/6

Collection2 1 1/3

Revenue

assurance

6 3 1

0.11

0.22

0.67

CR = 0.02 CR = 0.00


�Relative degree of success in (performance indicators of) the internal business process perspective by each functional unit

Alexandre Bentes

Jorge Carneiro

Herbert Kimura



22

Ind. H Fraud CollectionRevenue

assurance

Fraud1 1/5 1/3

Collection5 1 3

Revenue

assurance

3 1/3 1

0.10

0.64

0.26

Ind. I Fraud CollectionRevenue

assurance

Fraud1 1/4 1

Collection4 1 2

Revenue

assurance

1 1/2 1

0.18

0.58

0.23

CR = 0.03 CR = 0.05


�Relative degree of success in (performance indicators of) the innovation and learning perspective by each functional unit

Alexandre Bentes

Jorge Carneiro

Herbert Kimura



23

Fraud 0.09 0.61 0.13

Collection 0.74 0.12 0.66

Revenue

assurance0.17 0.27 0.21

Ind. A 0.42

Ind. B 0.12

Ind. C 0.05

x

Fraud 0.13

Collection 0.35

Revenue

assurance0.11

vectorial product

Ind. A Ind. B Ind. C

0.59

=

Fraud0.19 0.22

Collection0.73 0.63

Revenue

assurance

0.08 0.15

Ind. D 0.03

Ind. E 0.18

Fraud0.05

Collection0.14

Revenue

assurance

0.02

Ind. D Ind. E

0.21

x =

vectorial product

Finance

Customer

Application of the Analytic Hierarchy Process�Partial contribution of each functional unit to the overall

performance objective

Alexandre Bentes

Jorge Carneiro

Herbert Kimura



24

Fraud0.12 0.11

Collection0.20 0.22

Revenue

assurance

0.68 0.67

Ind. F 0.09

Ind. G 0.03

Fraud0.01

Collection0.03

Revenue

assurance

0.08

Ind. F Ind. G

0.12

x

vectorial product

=

Fraud0.10 0.18

Collection0.64 0.58

Revenue

assurance

0.26 0.23

Ind. H 0.07

Ind. I 0.01

Fraud0.01

Collection0.05

Revenue

assurance

0.02

Ind. H Ind. I

0.08

Processes

Innovation

and Learning

x =

vectorial product

Application of the Analytic Hierarchy Process�Partial contribution of each functional unit to the overall performance objective (cont.)

Alexandre Bentes

Jorge Carneiro

Herbert Kimura



25

0.240.240.020.080.020.11Revenue assurance

unit

1.001.000.080.120.210.59Total

Finance CustomersInternal Business

Processes

Innovation and

LearningTotalTotal

Fraud unit 0.13 0.05 0.01 0.01 0.190.19

Collection unit 0.35 0.14 0.03 0.05 0.570.57


�Final results for assessment of the best performing functional unit

Alexandre Bentes

Jorge Carneiro

Herbert Kimura



26

�Results indicate that the Collection unit performed better than the other units in three perspectives – finance, customer, and innovation and learning – and had the highest contribution (57%) to final objective.

�The Revenue Assurance unit performed better in the internal business processes perspective, but its contribution (23%) to the final decision regarding the (overall) best performing unit is not too different from the contribution of the Fraud unit (20%).

Discussion

Alexandre Bentes

Jorge Carneiro

Herbert Kimura



27

�BSC explicitly incorporates several perspectives (besides the usual financial viewpoint) to organizational performance assessment

�Evaluating performance requires some agreement on the relative importance of performance perspectives and performance indicators in the way to reach some aggregated (synthesized) measure or ranking

�AHP handles multiple dimensions and measures with distinct degrees of importance and translates the overall result into a unified metric, explicitly accounting for incongruence among objectives and mutually inconsistent evaluations

�AHP circumvents the pitfalls of having managers use a simpler, ad hoc, weighing approach to make sense of the multiplicity of performance measures from a balance scorecard

�AHP provides more than an ordinal ranking; it also informs aboutthe magnitude of the difference between alternatives, that is, how much more (less) a functional area is performing better (worse) than another

Conclusions

Alexandre Bentes

Jorge Carneiro

Herbert Kimura



28

�AHP provides insights to managers as to which functional areas are

more likely to contribute to overall success and which deserve more

attention

� but all in relative basis;

� note that if all areas improve, some will still be (relative) underperformers…)

� � so, both a relative and an absolute assessment should be conducted

�Results of the AHP + BSC can assist managers in deciding about:

� bonus distribution, incentive schemes

� identification (not covered in this study) of the possible reasons for poor (or good) performance

Conclusions

Alexandre Bentes

Jorge Carneiro

Herbert Kimura



29

Academic and managerial implications:

�Such an integrated approach (between BSC and AHP) was shown to have convergent validity with overall assessments by a seniordirector while providing much more fine-grained information for future managerial actions.

�The interactive and iterative process employed in this study hasthe additional advantage of leading managers to apprehend the diverse perspectives of performance assessment and to understand the possible trade-offs thereof.

Conclusion

Alexandre Bentes

Jorge Carneiro

Herbert Kimura



30

Questions and commentsare very welcome!

Alexandre Bentes ([email protected])

Jorge Carneiro ([email protected])

Herbert Kimura ([email protected])

Jorge Ferreira da Silva ([email protected])

Documents

APRESENTAÇÃO BALAS TESE (1) (1).PDF