Document(s) 8 of 13

This chapter explores key issues in conducting organizational assessments that respond to the important agendas brought forward by consumers and users. While the chapter provides some general principles and advice for carrying out organizational assessment, it is not intended as a “how to” manual. Rather, the aim is to help the reader understand the considerations needed for effective organizational assessment. An example of an organizational assessment outline is found in Appendix 2.

RATIONALE: WHY DO IT?

Before considering how to approach an organizational assessment, you need to reflect on why and for whom it is being done. Assessments typically are initiated by some sponsor, investor or member of an organization, or by the organization itself.

Some of the main reasons to initiate the assessment are summarized in the accompanying table. Clearly, whoever initiates the assessment shapes the focus. Thus, the agenda is rarely neutral, reflecting instead the needs, interests, values, and preconceptions of the people initiating it. Understanding the overall motivation for initiating an assessment goes a long way toward avoiding problems when implementing it later on.

Given that underdevelopment is to a large extent the result of a constraining institutional framework and non-performing organizations, the ultimate challenge of conducting an organizational assessment and implementing its results is to determine how the intervention can improve the organization’s performance.

THE ASSESSMENT PROCESS

Managing an assessment begins with understanding the motivation for conducting it. It is important to know whether the assessment is motivated from within or outside an organization. Those engaged in the assessment need to determine four points: 1) the central purpose of the assessment; 2) the time and budget; 3) the overall approach; and 4) how to communicate and use the information. These matters are ideally contained in written terms of reference that help clarify and communicate the intentions. The form of those terms will vary for an external assessment versus a self-assessment, but in either case, they are useful in keeping the process and vision of the product on track.

Many assessments suffer from poorly prepared terms of reference that are little more than a compilation of questions from various stakeholders. Such terms of reference reflect inexperience and need to be re-cast before a productive assessment can begin. This can be accomplished by better conceptualizing the work plan that responds to the terms of reference. The process of developing these terms can be a major step forward in any organizational assessment, particularly if the process reflects the engagement of stakeholders and clarification of values, issues and other concerns. Indeed, divergent views on the terms of reference are generally an early sign that stakeholders have fundamental differences in perspectives that will not get any easier later in the assessment process.

An overall management structure for the organizational assessment is the human side of what underlies the terms of reference, and as such, needs careful consideration. For external assessments in developed countries, a stakeholder steering committee generally guides the assessment process. Steering committees, however, are not a

panacea for successful management. They need to have a defined function beyond just receiving the assessment report. The greatest value of these committees can be to clarify stakeholder interests, values and perspectives on the questions, methodologies and sources of data; engage in vetting of the preliminary findings; address political or other problems; and provide a dynamic forum for debate and challenge of the preliminary draft report. As a management mechanism, steering committees must include the key stakeholders; otherwise there is a risk of imperfect management.

The client-reviewer relationship is generally contractual when the head reviewer is external, though it is often an informal relationship for internal assessment. Formality is advantageous in protecting all interests and in preventing the assessment from becoming the tail that wagged the dog. The contract clarifies the effort and cost involved. Both the terms of reference and the contract formalize the relationship, helping to avoid complications that are all too common, given the political nature of all evaluation work.

What should an assessment cost? This is a question that is always asked by clients. The answer is similar to the response given if asked the cost of building the client a house: it depends on what you want. This is not to advocate an unlimited budget—as the architect will attest, this may be more daunting than to have a circumscribed budget. It is possible to do a quick assessment in a week, at minimal cost; on the other hand, it can involve many months and consume hundreds of thousands of dollars. Assessments vary in scope, and organizations vary in their complexity.

How long should an assessment take? Because of the complexity, it is not possible to do a thorough assessment quickly. A rapid assessment that takes a few days can, however, provide an overall impressionistic view and examine certain aspects. A valid and complete assessment takes three to six months; any longer than that and it risks the difficulty of trying to hit a moving target. The assessment needs to

be sufficiently concentrated both to provide for the logistics in a cost-effective way and to convey the seriousness of the endeavor. On the other hand, the time span needs to be long enough to address the core issues fairly. An assessment intended for a major loan for an organization needs to be more robust than one intended to give the new director a sense of the major challenges ahead.

An effective assessment team requires division of labor and coordination mechanisms. Specified roles and areas of responsibility linked to an overall work plan avoid many problems. The team needs to meet for coordination purposes, and it is also helpful to have a continual exchange of such data as interview transcripts, key tables and graphs, and a running list of tentative findings. Every team requires a competent leader skilled at project and process management.

GUIDING THE ASSESSMENT: CHOOSING QUESTIONS

The framework described in previous chapters provides a comprehensive approach to organizational assessment. In reality, however, the assessment is tailored to the needs perceived by the stakeholders who initiate it. Whether in-depth or limited, there are a thousand questions for every organizational assessment. The key is to choose or create those questions that are most important to the organization under review.

The school principal may ask what happened to the many graduates of past years. The director of a company department may wonder if his competitors in other departments are really obtaining the performance they claim. An international development bank may want to know the prospects for success if the power sector is privatized. An international development agency may want to understand capacity needs in a targeted organization (NGO). A nation that contributes to a UN organization wants to know if it is getting its money’s worth.

Lists of questions are easy to generate, but mapping those questions is very difficult, and holds the key to successful assessment. Typically, this part of the assessment process is poorly done, despite it being the most critical element for starting on the right track.

Before addressing specific questions, the issue of hidden agendas and how they relate to organizational assessments needs to be considered. Often, the decision to do some type of review is motivated by perceptions of problems within the organization. Governing boards or funders may suspect poor performance, and they may have prejudged the reasons that generally relate to personnel performance issues. The sponsors may view organizational assessment as a means of obtaining a better under-

standing of the situation without transparently acknowledging the suspected problem. Certainly, an effective organizational assessment can fill in the available data, but it should not be considered a substitute for management or performance reviews.

The organizational assessment framework provides a useful structure to examine the questions that need to be asked. It also indicates the scope of concerns, and provides the reviewers with a reference that ensures that all relevant facets of the assessment were addressed. While the emphasis of an assessment may vary from situation to situation, having a framework provides an overall map that serves as a useful starting point. As noted earlier, generating a large number of questions is not difficult. What is challenging is to reduce the list to the essence. The table above lists generic questions at the most general level for an organizational assessment. That list of

questions becomes much longer when sub-questions are added. What, then, are the most important questions? Essentially, all four aspects of performance must be understood, and then as much of the other dimensions as required to understand how performance is being influenced or could be enhanced. Approaches to questions in each of the four dimensions follow.

Framing Performance Questions

Performance is the paramount theme and should be included in every organizational assessment. The first consideration in analyzing performance in an organization is to understand how people view performance. Unless people are clear and agree on the definition of performance, reviewers and internal stakeholders will disagree on the conclusions of an assessment, because they approach performance from differ-

ing perspectives. There are two requirements: the issues and indicators of performance, and the importance placed on them. Note that there are both issues and indicators, since sometimes what is required is an analysis (issue) that does not reduce to a simplistic indicator. In other cases, indicators give the data needed for analysis.

Importance can be ascertained by reducing the issues or indicators to a small number (one to three per dimension). Often, the realities of available data dictate what indicators can be included, at least the first time an organization is involved with assessment. It is better to concentrate on easily obtainable data and complete the assessment in a timely manner than to spend many months trying to find elusive data. An organizational assessment is just a picture at a point in time. The analysis can, and should, go on in subsequent assessments. If people limit their issues and indicators in this way, it is relatively easy to see what is important. There are situations where diverse stakeholders cannot agree. If the differences cannot be resolved, there is no basis for an assessment that will be endorsed by the different stakeholders, so an inclusive process is out of the question.

Questions that Deal with Capacity

Capacity needs to be understood in terms of its relationship to performance, rather than in response to the wants of people inside the organization. Capacity questions lend themselves to both norm referencing and criterion referencing. Norm referenced approaches compare capacities to benchmarks within similar organizations or industries. This enables reviewers to make comparative judgments once they know the answers to certain questions. For example, what span of control do managers have? What is the ratio of support staff to professionals? What are the cash reserves? How many computers of each type are available? By comparing the answers to averages or best practices, reviewers can make judgments about capacity and its adequacy.

We acknowledge that in many cases, there are no readily available benchmarks. Experienced organizational assessment teams may have access to relevant comparisons that inexperienced teams do not. However, if no benchmarks are available, there must either be an investment in order to collect them, or else the assessment must do without them by using the baseline approach with comparisons over time.

A criterion-referenced approach uses conventions that reflect values for capacity. For example, organizations sometimes refer to standards for such measures as the ratio of support staff to managers, the proportion of staff with stated qualifications, the number of staff who have access to a computer, and so forth. The best developed are standards (such as ISO) that prescribe the necessary requirements for an organization to achieve recognition for meeting the standard.

Motivational Issues and Questions

Assessing motivation is extremely challenging because individuals are complex. Place these people together within groups and organizations and the challenges multiply. Measuring motivation is similar to trying to assess community values— difficult to define, but you know them when you see them. To say that an organization is suffering from malaise is not hard, and may not even be contested; however, to represent this presents difficulties. There are corporate culture instruments that can help, and some of them permit comparisons that position the dimensions of culture relative to norms. It is also often helpful to use qualitative approaches and provide anecdotes, vignettes or quotes to illustrate employee attitudes about their organization.

The crucial consideration in assessing motivation is to understand the types of issues and corresponding data that stakeholders understand. Often, a single event or series of events can have profound effects on the overall motivation of a department, region or the entire organization. For example, one insensitive manager can provoke an entire staff, which in turn has a profound effect on the way work is done and how the organization operates. At the individual level, staff is often personally affected by unfair criticism or, on the other end of the spectrum, by receiving praise.

Asking employees for their impressions of the organization often captures the essence of motivation. Such comments as: “The best place I have ever worked,” or “We are the leader in our field,” suggest a motivation that supports the mission.

Determining What Needs to Be Known about the Environment

We all understand that the environment influences every organization. It exerts expectations on an organization’s ability to achieve its mission, it provides limits on its degrees of freedom, it dictates financial subsidies, and it provides rules of the game that bracket organizational development. Furthermore, the environment can be described in both qualitative and quantitative terms. The challenge for the reviewer is to analyze the extent to which environmental forces positively and negatively impact the organization. While certain environments may make achieving positive performance difficult, it is not hard to identify examples of organizations that prospered despite a challenging environment.

ORGANIZATIONAL ASSESSMENT METHODOLOGY

The organizational assessment approach outlined in the previous chapters sets forth a framework and a set of questions that—with the proper data, analysis and judgments—can lead to a better understanding of the organization and its performance. However, as implied above, the choice of methods used to design the assessment, collect data and select questions raises some series issues.

Basically, organizational assessment follows in the tradition of a methodology known as a “case study.” A case study is a qualitative form of assessment, though it uses both qualitative and quantitative data. Case studies rely on multiple sources of information to gain insight into the organization (Anderson, 1998). In this methodological tradition, the emphasis is on understanding. In other words, in doing an organizational assessment based on the case study approach, the aim is to understand the meaning of a question. There is no a priori answer being tested. The assessment is trying to understand existing capacities and how they affect the performance of the organization under review.

A case study approach requires identifying the sources of information, the instruments to use, and the ways to collect information, as well as analyzing the information. The sections that follow summarize these tasks.

Sources of Data

Six sources of evidence are typically used in conducting case studies: documentation, file data, interviews, site visits, direct observation, and physical artifacts. For most organizational assessment questions, some type of documentation is generally available, including reports, file data, memoranda and previous studies. Interviews are

prime sources of data for assessments. Not only do we interview a range of respondents, we also try to find key informants who have inside knowledge of what is going on. These individuals are critical to enhancing the validity of the conclusions drawn.

Surveys are often used in organizational assessments to gather data from a large number of organizational members. This is particularly important in assessing organizational culture and process issues. Typically, an assessment requires on-site visits for direct observations, which can be very helpful for understanding why things are as suggested by other data sources. Finally, physical artifacts should not be overlooked; some assessors even systematically checks bulletin boards (electronic ones in some organizations) to help understand the organizational culture.

Typical data sources that might be helpful to an organizational assessment include a table of company milestones, that is, dates and events that help in an understanding of the organization, changes in leadership, the introduction of new programs, and construction activities. For organizational structure, it might be helpful to locate present and past organigrams, staff lists, minutes of meetings, policy handbooks, regulations, and perhaps even a diagram of the physical plant. Organizations have lots of data, and the assessor needs to have the experience to choose sources that best answer the key questions.

Data Collection

As in any methodology, assessment requires a work plan that defines what will be done, and how and when it will be done. Organizational assessments tend to be a method of immersion. An important aspect of data collection is the creation of various study databases. Weak assessments generally confuse the data with its reporting, whereas the best ones maintain a separate inventory of data with charts, tables and numbers, some of which are used in the text. Other information is appended, while still other information is not used at all.

Data Analysis

The masses of data potentially available for organizational assessments can present insurmountable problems, unless the assessors know what types of analysis are required. Our assessments provide a categorization system or framework that significantly eases the issues of data analysis. For example, if you were exploring issues

of structure, you would separate out data related to Board governance from data related to operations. In both instances, you might want to explore such issues as the clarity of roles and authority.

It is at the data gathering and analysis stages that important insights emerge. This is usually a good moment for the team to test assumptions and conclusions. It is important to understand that in a case study as used in organizational analysis, the analysis phase takes place as the data are collected. The opportunity to test conclusions in the field is an advantage of this methodology.

SOME KEY ISSUES

Expertise

In general, organizational assessments are complex and require a variety of people with differing expertise to successfully complete them. Ideally, they are carried out by a team with collective skills and a strong leader with a clear vision of the task. Intelligent, well-rounded people with diverse experience and solid research and evaluation skills make the best reviewers, providing the team includes the necessary content expertise representative of the field of the organization under study. Interestingly, one of the most respected institutions for studies on public health, the Institute of Medicine, does not use researchers on its evaluation teams who are expert in the specific area of research being assessed (Stoto, 1997). This contributes to freshness in perspective. One strong advantage of a team, however, is that it can capitalize on the collective strengths, rather than rely on one individual who is always a compromise among stakeholder interests.

There are many sources of analytic and evaluation expertise and many views concerning what type of expertise has value. It depends largely on the purpose of the

organizational assessment and its client. Considerations in choosing reviewers include credibility, expertise and distance from the organization under review. Credibility is crucial, and different stakeholders have differing credibility criteria. Most staff in an organization want to involve respected content experts, presumably because they feel their professional concerns cannot be adequately addressed by someone outside their profession, and because professional peers have the same socialization, and most likely, similar values. They also probably neither understand alternative professions for reviewers—such as a social scientist, economist or evaluator—nor know what they might offer to the task. Even if there is a predilection to relevant professional content, content is not sufficient by itself.

Often the client of an organizational assessment requires evaluation expertise. The evaluation expert has methodological expertise, and may be more technically competent, not to mention more independent and objective in raising certain questions. But he or she is often suspect in terms of credibility due to a lack of professional content expertise.

Whose Perspective? External and Internal Reviewers

In both the analytic and evaluation approaches, the output is a function of your vantage point for viewing the problem, which is the distance between the reviewer and the organization being assessed. The most proximate reviewers are members of the organization whose involvement on an organizational assessment team can either be advantageous or a liability to operations and to objectivity. These people can save a lot of background research by presenting ready information about the organization and its existing data. Involvement of organization members is often helpful for the analytic approach.

However, in accountability evaluations, as Chelimsky and North discovered in their evaluation of the Global Environmental Facility, too much loosely structured involvement of internal stakeholders with vested interests creates conflicts over agendas, methodologies, working relationships and the wording of reports (North, 1997). As a result of the experience, North concluded that it is essential to have an evaluation coordinator acceptable to all parties, clear and unfettered reporting relationships within the team, a common script or report outline for sub-components, and trust and mutual respect among the evaluators and in the evaluators’ relationships with stakeholders.

Another consideration is the need for sufficient distance to avoid “insight fatigue” that is sometimes associated with people who spent many years viewing the

organization from a single perspective. Keep in mind that an insider’s position on issues is generally known, so he or she cannot be neutral or perceived to be objective in making judgments. Furthermore, if an organization shows signs of fracturing through cliques and cabals, an insider may have difficulty getting people from other camps to confide, or at the very least, have difficulty giving their contributions credence. Returning to what skills and knowledge a reviewer requires, the insider needs these in abundance if he or she is to successfully overcome the potential objectivity problems that result from being too close to what is being assessed.

All this considered, the central issue is the extent to which one actually wants an approach that provides an independent and credible organizational assessment that incorporates the highest ethical standards. The purist position as held by Michael Scriven (1997) asserts that with such ethical standards, an evaluator is capable of objectivity, but to be objective, he or she must avoid being too close. Scriven suggests that even such procedures as staff interviews reduce distance and compromise objectivity because the human face of the organization that may be hurt if a negative view is conveyed may influence the assessment. To the critics of this stance who believe that there is no objectivity, he says “the public is less naïve and thinks that regulatory agencies that socialize with those they regulate should be viewed with suspicion, as judges think the same about jurors socializing with defense attorneys” (Scriven, 1997, p. 487).

The role of the head of the organization depends on the person and the context for the assessment. Heads of departments, permanent secretaries or other organization heads often wisely prefer to maintain some distance to ensure they are not seen as overly influencing the process. However, when the process involves more future oriented diagnoses, or initiatives for strategic change, it is preferable that the organization’s leader be involved.

Self-Assessment

Self-assessment is part of what has recently been termed “empowerment evaluation,” which is defined as the use of evaluation concepts, techniques and findings to foster improvement and self-determination (Fetterman, Kaftarian and Wandersman, 1996). This approach embraces an unambiguous value orientation, for it is designed to help people help themselves and improve their organizations and programs using a form of self-evaluation and reflection. Self-assessment embraces organizational development and change questions and is highly applicable when the purpose is organizational development. Involving organizational members provides a direct link to implementation by providing them with data and assessment findings.

One major consideration is the cost-benefit of taking people from their own areas of expertise and involving them in something they need to learn at an opportunity cost to their normal duties. This is a political decision generally made by the head of the organization, who sees the assessment as advancing the organization’s strategic goals. Needless to say, the best results come from involving the most respected and competent staff—the very ones who typically are already making the largest organizational contributions.

Is organizational self-assessment worth the trouble? This is a critical question. On the positive side, this form of reflective practice can deepen stakeholder insight into an organization—its strengths and weaknesses, motivation and performance. But, on the negative side, at the best of times, it takes considerable time and energy. Can an organization afford to involve its personnel in such an extravagance? Can it afford not to? Furthermore, is it an opportune time to stir the emotions of staff about the certain shortcomings that will be raised? Is the organization and its staff prepared to act on the information that will be revealed? Some leaders are confident in their own leadership and welcome new insights. Others are more circumspect. Maybe there are other critical challenges at this juncture. Maybe staff are at odds with the administration; maybe it is best to leave sleeping dogs lie—at least for now.

Engaging in organizational self-assessment raises two important issues. First, is it the best means to achieve organizational development? At times it may be, and at other times not. The organization may need a common task in which to engage people in thinking about the organization. If so, a self-assessment may be an answer. On the other hand, the organization’s staff are not professional reviewers, and they may have only rudimentary skills, resulting in a costly, imperfect and potentially damaging exercise.

The second issue is validity. Because the purpose of self-assessment is organizational development, validity may be less of a concern than with accountability evaluation. However, you often find the assessment attempting to serve both purposes. For example, development assistance donors that fund organizations need accountability, but the organization is interested in self-assessment, so often the agendas end up combined. There are safeguards to validity, such as the use of external reviewers, but it is nevertheless difficult to combine processes with such different purposes. The evaluation approach focuses on how well the organization is performing, whereas the analytic approach may be more interested in how it can improve. One recommendation is to use a two-level structure that first enables the organization to conduct some or all the elements of a self-assessment for its purpose, and then uses this as data for an external assessment (Anderson and Gilsig, 1998).

In deciding whether to engage in self-assessment, it is helpful to have a sense of the context. We have seen instances where employees tried to hijack an organizational assessment for their own ends. We also experienced situations where staff were highly critical of their leader, but were not in a favorable position to communicate their displeasure. In all such contexts, self-assessment should be avoided. It is more advisable to use external teams that are impartial and can deliver the messages without tracing them to particular people on the inside.

The only way a self-assessment can be really useful in organizational development is if the leader fully supports or even leads it. There have been instances, however, where the leader adopted self-assessment merely to be in a position to control the agenda and process. In other words, the leader was inclined to support it as a less risky alternative to external assessment. If there are external reviewers involved, this can work, but it is still only an approximation of a valid organizational assessment. Arnold Love (1991, p. 7) put it well when he wrote:

The manager is a believer in people and programs, a partisan advocate and supporter; by contrast, the evaluator is a doubter who is uncommitted to anything or anyone. Finally, the manager is warm and outgoing, a leader and part of a team, as the evaluator sits in perpetual shade, cool and aloof, a lone wolf detached from the pack.

Qualitative and Quantitative Data

Categorically speaking, data are either quantitative (numeric) or qualitative (non-numeric). Some analysts and evaluators prefer quantitative data. For example, analysts are adept at collecting and interpreting financial data on performance—that is, hard quantitative data. Performance indicators such as return on investment can tell a great deal about an organization, but considered in isolation, such data may fail to capture the underlying upstream reasons for the result. Increased return on investment may be a result of asset depletion or short-term benefits from a management intervention such as borrowing. These indicators, while robust, may have other problems. In government and civil society organizations, it is often difficult to capture their results in economic terms alone. If an NGO’s purpose is to empower civil society, for example, economic results may be the wrong indicator; however, appropriate quantitative indicators are often difficult to agree upon.

Qualitative data incorporates a reviewer’s judgment on its saliency for the assessment. It represents relevance that goes beyond mere counts. There are social reasons for having a university or a hospital that are not reflected in quantifiable measures of performance. Another example is the strength of environment factors, such as the political context. How do you judge the political climate and the effect on the organization under review? Or the cultural values and ethos and their effect on the labor market and, ultimately, the organization? Qualitative data and analysis provides some insight into these concerns.

In the debate over the relative merits of quantitative and qualitative data, the compromise position would seem to incorporate both. While we advocate this position, there are risks, most particularly having a suitable balance with the importance of each data type. It is easy to be in a position of doing neither type of data justice, and being condemned by both quantitative and qualitative adherents. The fact is that people with differing values and orientations have different views about the type of data that they feel is important.

Data Sources

Typical data sources include documents, people and databases. Obviously, documents need to be reviewed based on their authorship, and with an understanding of the original context and purpose for which they were written.

People represent special challenges, either when involved in normative data collection subject to quantitative analysis, or for their qualitative insights. Two of the most common issues are including a proper sample, and ensuring that the data collected are valid or truthful. Sampling is always problematic. Do you include former staff, as well as the present guard? Are critics of an organization identified for inclusion? In some approaches, such as economic analysis, people may be excluded altogether, though possibly at a political cost. Indeed, many donor agencies review their investments without direct contact with the people involved except on a project sample basis.

Turning to the other issue, it is always challenging to understand what people are conveying and to assess the degree of bias in their statements. Instrumentation problems can arise, especially when questionnaires and interviews are conducted in cross-cultural environments. When collecting non-numeric data from people, vetting, pilot-testing and other forms of validation may be required for data collection instruments. One of the costs of the trend toward self-assessment and internal eval-

uation is neglecting the fundamental requirements of the reliability and validity of data collection techniques because people do not understand the importance, or they lack the requisite technical skills. This underscores the advisability of defining indicators and data collection procedures on an ongoing basis, rather than just when a formal organizational assessment takes place.

One of the most difficult people challenges is in knowing how to value data that are distorted by self-interest, be it highly praiseworthy, or unreasonably negative. Missing data are particularly troublesome unless reviewers take the trouble to investigate why people do not respond. People who benefit from an organization’s programs tend to be positive in their assessments. They may have selected the program, and negative comments reflect on their own choice. If they obtained a financial benefit such as an income-support payment or scholarship, they tend to say: “Great, but more money would have been better.” Validity can also be enhanced through triangulation.

Another people issue is how much data are enough within the value-laden environment of an assessment. It matters little if statisticians agree that 100 questionnaires to consumers of an organization’s services are sufficient if the political environment requires hearing from 1,000. Credibility is of as much concern as statistical validity. Thus, if stakeholders are consulted at all, it is imperative that the assessment be seen as inclusive and as providing all stakeholders with a voice.

Databases are another source of information, and we need to understand the sources and collection procedures for the data provided. It is important to check if statistical data are out of date and may be suspect in terms of reliability and validity. Once the data are captured, you need to ensure that new errors do not creep in when different software is used, or when transforming and combining is required. It should be noted that most organizations have a great deal of internally generated data such as financial system data that can be invaluable for assessment. Regrettably, many organizations do not fully understand the link between budget categories and performance, so sometimes these data are not in categories that facilitate efficient analysis. In certain cases, the use of externally generated data, such as industry benchmarking, can be of value to organizational assessment. For example, consulting or accountant fee revenues can be quoted in multiples of daily salary, and reflect the industry norms.

Validity

The principal methodological challenge of organizational assessment is validity. Validity has three principal issues: 1) false assertion of a positive result; 2) failure to

detect a positive result; and 3) asking the wrong questions and contaminating the assessment with organizational or personal bias (Dunn, 1982).

There are many reasons for problems with validity. Organizations are seen as trying their best to do good things, so reviewers “go looking for” rather than just “go looking.” Furthermore, the way questions are framed, upstream perspectives, the way data are collected, and non-response problems build in a bias toward reports of positive performance. The involvement of reviewers close to the organization confounds objectivity. Not-for-profit organizations that do not have a bottom line result in biased judgments in the absence of baselines or benchmarks. It is overly easy to conclude that an organization is doing good things when it is assisting the poor, or building democracy, or housing the homeless—but it is much harder to judge whether it is doing all that it could with its available resources.

Errors also occur when the assessment is restricted in time and resources, such as when reviewers are not able to visit consumers or access other primary data sources. Results may also be missed if by nature they are long term. Organizations that deal with complex effects such as community development may sow the seeds for results that do not occur for a decade. Similarly, educational organizations that may spend years delivering education services cannot be assessed completely until their graduates have years of work experience.

Finally, errors occur because reviewers may be biased or blind to the questions that need to be asked. In multilateral donor-supported ministries, agencies and organizations, this can be a central question. In an evaluation of the Tanzania Railways Project, it was discovered that the railroad was now largely redundant, given the rise of more cost-effective trucking on an improved network of roads. Professional ethics suggest that evaluators are obliged to pose such questions even

if they are not included in the terms of reference, though this may not be the case for team members who are not professional evaluators.

The two ways to counter this validity limitation involve the use of benchmarks or baseline comparisons. Benchmarking enables an organization to compare itself to standards in the industry. This is particularly useful when performance indicators are compared, such as return on investment, cost of haulage per ton-mile, gross margin, or agricultural yield per hectare. Valid measurement techniques can lead to agreement on the data. The interpretation of differences becomes the analytic issue, because the cause of the observed differences needs to be attribute to sources within or outside the organization. The comparisons are only valid when the benchmarks are considered applicable, which is often not the case or even possible across countries that may have different policy and economic environments. It may also not be possible across geographic regions that may have all kinds of different conditions. One type of benchmarking—accreditation—is used in some social sectors such as health and education. Organizations are accredited if they conform to certain standards of capacity, and sometimes performance.

Reviewers who change hats following the assessment and assume the role of facilitators or performance consultants are viewed by some as being in conflict of interest, since they can be perceived as consciously or unconsciously orienting the report to promote their services (Scriven, 1997). Internal reviewers may similarly orient the organization according to their future role within it.

Given all of these validity concerns, the major issue for reviewers is to consider for whose interests they are working. Is it the client? The people who pay for the organization? The organization? The consumers and beneficiaries of the organization? The courtroom may applaud what you say in presenting your case, but to win your case, you need to convince the judge.

THE REPORT: COMMUNICATING THE RESULTS OF THE EXERCISE

The assessment is not complete until it is communicated or reported upon in some tangible form such as a printed report. Not that this is easy, because the printed word has a way of distorting what was intended. Nonetheless, written reports represent important organizational milestones, and serve as baselines for subsequent assessments. They also provide a sense of closure to the process and signify the time to move forward in acting on what was found. Michael Quinn Patton (1990)

advocates a presentation and discussion rather than a formal report, which he feels is too symbolic and paternalistic. But not all reviewers share this view.

Reporting is never easy if you are to preserve integrity and also communicate. There are always issues with style and the need to conform to the client’s expectations. Some agencies and clients want detail, while others want brevity. Some want extensive appendices, while some do not. We were recently involved in producing a report with what we considered to be highly informative graphs, only to learn that one of our clients had difficulty reading graphs and preferred tables. However, despite these challenges, a report says what should be said and hopefully makes it public—at least to those paying for the organizational assessment.

Reports go to audiences that are often diverse. An organizational assessment must serve the needs of the client, and theoretically should go to the client first. However, there are advantages to showing a preliminary draft to the organization, for it gives those directly involved a chance to correct incomplete or incorrect data before it becomes public. It also begins the process of softening the blow that a negative report will create.

Most assessments produce what is known as “report shock.” It is the highly emotional reaction by the leaders of an organization when they first see a critical analysis that represents the organization in a way that is perceptually different from what is imagined. This is not unlike a personnel performance appraisal in which employees view themselves in highly positive terms, and then receive an objective contrary analysis backed up with data. This natural reaction needs to be managed or it can destroy the utility of an assessment report.

It is useful to allow the head of the organization to review the penultimate draft of the report with ample time to immediately meet with the assessment team to review the report and how it is worded. Often, re-wording a few sentences can do wonders for support of the analysis. Experience has taught us to be highly sensitive to the use of negative terms in the report itself. Terms like “not” provoke an understandable human reaction from people who have given a lot to an organization. In

practice, it is beneficial to ask colleagues to review the findings and conclusions, and rate them as positive, negative and neutral. Too many negatives may call for re-wording if the report is to receive a fair hearing.

While the client is the audience that needs to be served, most reports go to other audiences as well. Certainly the organization receives the report, and in many cases comments on it. In some cases, the comments are incorporated as an appendix to the report. The difficulty for the author is to write a report that can be used and understood by different audiences. A good report speaks directly to its primary audience, though the organization should learn from it as well. Sometimes, the terms of reference provide for recommendations to the funder and the organization that are then identified in the report.

CONCLUSIONS

This chapter considered many of the issues that concern people interested in organizational assessment. Some concerns, however, were not addressed. Some concerns are methodological:

• How do you select a particular data collection methodology?
• How do you deal with data analysis when the data are distorted, incomplete or contradictory?
• How do you ensure that the data are sufficient? When do you stop?

These are the concerns of every social science researcher. These tend to be issues that require judgment based on experience or expertise. Researchers deal with them by doing as much as they can in an imperfect world. Those who have never done this type of work cannot be expected to know all the answers, so novices can be greatly assisted by teams of people with more experience.

Another set of concerns centers on the difficulties of organizational change:

• How do you overcome resistance to the assessment process?
• How do you determine which of the areas of organizational improvement are most important?
• How can organizational assessments best support performance?

These concerns require a different expertise and experience as to how to assist organizational development. There is both an art and a science that organizational

development experts can share. While it is not possible to consider this in-depth here, the questions are important and suggest that organizational assessment must be more than a detached and passive research process. Without a dimension of organizational change, an assessment loses much of its rationale.

The best way to address people’s concerns is to engage them in actual experience with the organizations with which they are most familiar. For all the limitations of the organizational assessment process, it does provide insights that were not previously apparent. It does this because of its systematic framework that helps include every important issue, and because of its philosophy of engagement. People who become involved in assessing organizations, even with partial and imperfect data, will benefit from understanding the dynamic of organizational performance.

Document(s) 8 of 13