Re-discovering Measurement
5. Measurement Criteria
5.1 Principal Source of Criteria
(91) The criteria analysis in this paper builds on the analysis of measurement theory, information
theory, and accounting theory included in the FEE paper referred to in Section 3, of which an extract is available from the
NPI website as indicated by Appendix B. Although the FEE paper had different objectives than this one, it provides
a highly relevant overview
and synthesis of fundamental theoretical concepts.
(92) One can think of information, measurement, and accounting theory as being related as follows:
(93) The FEE paper includes an extensive analysis of, and ultimately a synthesis of criteria emerging
from information, measurement and accounting theory.
(94) In order to adapt the analysis in the FEE paper to profiling measurement approaches, some relevant
distinctions must be made.
(95) The first is to differentiate between criteria for assessing measures, and criteria for assessing
measurement systems or approaches. A measurement system may incorporate various types of measures: necessarily, then,
the criteria used to profile a measurement approach or system will be somewhat more general, or operate at a higher level,
than the criteria that would be applied to individual specific measures.
(96) A second distinction is to differentiate between criteria that would be applied to assess a specific
instance of a measure or measurement approach, and criteria that would be applied to assess the measurement approach as an
approach. In this paper, we will refer to the former as "evaluation criteria", and the latter as "profiling criteria".
(97) In an assurance engagement, an auditor applies "evaluation criteria" to assess the specific
subject matter of the engagement. The auditor provides an opinion on a specific instance of financial statements -- for
example, the financial statements of XYZ Corporation for the year ended December 31, 2004 -- not on the general class of
financial statements as a measurement approach.
(98) By contrast, the focus of this paper is to suggest criteria that could be applied to profile a
measurement approach as an approach, in comparison with other approaches.
(99) To illustrate further, the issue that concerns us here is not whether a particular instance of a
measurement approach demonstrates a certain "bias" (to cite one criterion of interest), but rather whether all instances
of a specific measurement approach are biased because of something inherent in the approach itself.
(100) As a result of these two distinctions, the approach followed here is not to attempt to incorporate
all of the criteria described in the FEE paper, but rather to select from and adapt those criteria that could be usefully
applied to profile measurement approaches, as well as to suggest additional criteria that may be relevant.
(101) Building on this background material, this section sets out to answer the questions:
- What is the minimum set of "profiling criteria" that can be used to authoritatively describe the similarities
and differences between one measurement approach and another?
- How can such criteria be organized into an overall model that can be applied to generate a "profile" for a
measurement approach in relation to the criteria.
5.2 Context
(102) To begin with, there are three contextual overarching criteria that should be
considered at the outset in classifying a measurement approach:
- Who are the users?
- Why do they want to use the measurement approach: in other words, what purposes does it serve?
- What does the approach actually measure?
Let us consider each in more detail.
Who are the "users" of the measurement approach?
(103) Users can potentially be classified first by differentiating among those who are internal
to the organization (e.g., employees), versus those who are outside the organization (e.g., external stakeholders),
versus those who sit at the interface between external and internal (e.g., Board of Directors).
(104) Internal users can subdivided into:
- Operational staff;
- Operational management;
- Executive management.
(105) External users can be subdivided into:
- Those who have an existing relationship with the organization as:
- Owners or members;
- Customers or the equivalent;
- Suppliers or business partners.
- Those who are affected by the organization's actions (members of society);
- Those who may use measurement information in the process of decision whether to enter into a
relationship with the organization in future.
(106) A further differentiation can be made among:
- Users who simply are the recipients of measurement information;
- Users who provide key inputs;
- Users who plan and management the administration of a measurement approach.
What are the purposes of the measurement approach: in other words, why do users use it, or why
should the want to use it?
(107) The purposes of a measurement approach may classified by determining:
- Whether the measurement approach plays a role in the day-to-day operations or transactions of the
organization, or whether its function is related more to management;
- Whether the measurement approach influences decisions, and if so, what sort of decisions:
- Decisions of internal users (management decisions) versus external users (e.g., investment decisions)
- In the case of management decisions, one can differentiate among day-to-day routine decisions, versus routine
decisions that are made routinely, versus strategic decisions that are made infrequently;
- One can also differentiate among decisions that are normally made by operational staff, operational management,
or executive management.
- Whether the emphasis is on quantifying value or quantifying performance.
What does the measurement approach measure: in other words, what are the objects and
properties that are the "subject matter" of the measurement approach?
(108) The subject matter of a measurement approach can be classified with respect to:
- Whether the principal objects / properties being measured relate to:
- Financial transactions of the organization;
- Things owned by the organized (such as tangible and some intangible assets);
- Processes taking place inside or outside the organization;
- People and entities that are internal to the organization, and/or their relationships to the organization;
- Value streams in which the organization participates;
- People and entities that are external to the organization and/or their relationships to the organization;
- Attributes of the organization itself;
- Transactions of external parties that relate to the organization (such as those that influence stock price);
- Events that relate to the organization.
- Whether the units of measure are primarily financial or non-financial;
- Whether the scale of the measurement is primarily numeric or non-numeric;
- Whether the timeframe to which the measures relate is primarily the past or the future.
5.3 Utility of a Measurement Approach
(109) With the foregoing criteria defining the overall context at a high level, let us move on to
criteria that can help analyse in more detail the "utility" of a measurement approach, which broadly relates to what
it can be used to measure.
(110) The "Utility" criteria are derived from information, measurement, and accounting criteria, as
summarized in FEE paper extract, related to such matters as:
- User information needs
- Valuable information
- Relevance / decision-usefulness (making a difference in a decision)
- User Benefits
Does the main focus of the approach involve quantification related to value,
quantification related to performance, or both?
(111) For example a Balanced Scorecard provides insights into performance, but
normally does not measure value. Traditional financial statements measure both value and performance,
based, however, principally on financial transactions.
To the extent an approach quantifies value, what are the principal parameters of the approach?
(112) What are the objects that are or can be measured using this approach:
- Financial transactions of the organization;
- Assets or the organization, tangible and intangible;
- Entities or units of the organization;
- Value streams in which the organization participates;
- Relationships between people or entities and the organization.
- Transactions of external parties that relate to the organization;
(113) What is or are the principal perspective(s) from which value is quantified?
- People or entities internal to the organization;
- The organization itself;
- The owners of the organization;
- The market in general;
- Specify potential buyers or sellers;
- Other external stakeholders.
(114) What are the attributes of the underlying mathematical system used to quantify value?
- Systemic or non-systemic;
- Standards-based or ad-hoc;
- What value concepts are relied upon: for example:
- Historical cost accounting;
- Market comparables as used in valuation;
- Discounting of potential future cash flows;
- Strategic value based on market or competitive analysis.
To the extent an approach quantifies performance, what are the principal parameters of the approach?
(115) What are the objects / properties that are or can be measured using this approach:
- Processes internal to the organization;
- People internal to the organization;
- Entities internal to the organization;
- The organization itself;
- External processes in which the organization participates;
- External relationships of the organization.
(116) What is or are the principal reference point(s) in relation to which performance is quantified?
- Previous performance;
- Performance goals or targets;
- Performance of peer organizations or competitors;
- Performance related to stakeholder expectations;
- Performance related to defined standards;
(117) What are the attributes of the underlying mathematical system used to quantify performance?
- Systemic or non-systemic;
- Standards-based or ad-hoc;
- Absolute or relative;
- Measurement scale.
Comparability: To what extent is it possible to use measures generated by the approach to do value or
performance comparisons?
(118) What types of comparisons does the approach enable?
- Internal comparisons
- Prior periods
- Targets
- External comparisons with peers or competitors
- Benchmarking
Extensibility: To what extent can the measurement approach be adapted for broader purposes?
(119) Extensibility can be assessed based on the following factors:
- Does the approach have the potential to serve broader purposes than at present? How much development
would be required to support this?
- Could the approach be adapted to measure different objects / properties than at present?
- What are the existing inherent limitations of the approach, and to what extent could these be overcome through
further development?"
- Does the approach have an existing or potential role in external corporate reporting?
- Is there an existing or potential linkage to emerging standards such as XBRL?
5.4 Validity of a Measurement Approach
(120) The validity of a measurement approach is another key area for comparisons among
different approaches. For purposes of this analysis, key aspects of validity that build on the criteria s
ummarized in Appendix B include:
- Construct, content, and criterion-related validity
- Reliability
Validity in use: From a user perspective, what is the relative validity of the value or
performance measures related to the objects / properties being measured?
(121) At a pragmatic level, a user of a measurement approach may make a
number of intuitive "judgement calls" with respect to whether a measurement approach is sufficiently valid
for measuring specific categories of objects / properties.
(122) For instance, a user might intuitively perceive that a measurement approach offers
sufficient validity for valuing tangible but not intangible assets; or is sufficiently valid
to measure the performance of business entities but not valid if applied to business processes.
Reliability in use: To what extent are users prepared to trust the results of
the measurement approach sufficiently as to make decisions based on the measures?
(123) A user may also make a judgement call with respect to the types of decisions
he or she is prepared to make based on the measurement approach, also taking into account the extent of
corroborating evidence that is available.
Technical validity
(124) At a technical level, there are three key aspects of validity for any measure:
- Does a measure accurately reflect the underlying concepts of theory (if any) on which it is based? (referred to as Òconstruct validityÓ).
- Does a measure actually measure what it purports to measure? (referred to as Òcontent validityÓ)
- Does a measure provide a satisfactory amount of feedback information (looking backward), or predicative information (looking ahead), and what is the Òshelf-lifeÓ of the measure: in other words, for how long is the information still valid or useful. (referred to as Òcriterion-related validity).
Technical reliability
(125) At a technical reliability has two major aspects:
- Accuracy: does the measure offer a satisfactory level of accuracy, free from systemic error or bias;
- Precision: does the measure offer a satisfactory level of precision, free from random error or bias.
(126) The units of measure and scale are related to both accuracy and precision.
For instance, if the measurement scale is not sufficiently granular, this may limit accuracy, precision, or both.
5.6 Practicality of a Measurement Approach
(127) Building on relevant criteria outlined in the Fee Paper extract,
the practicality of a measurement can be assessed in relation to criteria such as the following:
- Absolute and relative benefits as perceived by users
- Absolute and relative costs
- Short and long-term resource requirements
- Availability of inputs
- Appropriateness of outputs
- Characteristics of users that are essential to make effective use of the measures
- Inherent characteristics of the information on which the measures are based.
