Note: This is an article in a series celebrating NACADA 30th anniversary. In this series current NACADA members build upon the work done within the 1995 monograph, Advising as a Comprehensive Campus Process, as they highlight the important connections advisors make across campus.

Utilizing Institutional Research in the Assessment of Academic Advising

Rich Robbins

Associate Dean of Arts and Sciences

Bucknell University

What a difference a decade-and-a-half makes! In the 1995 NACADA Monograph Academic Advising as a Comprehensive Campus Process, Creamer and Frost’s chapter on academic advising, institutional research, and assessment introduced to many the idea that academic advising could (and should) be assessed. Today, the importance of assessment in academic advising is evidenced by the increasing number of articles and book chapters on the topic, the popularity of the NACADA Assessment of Advising Institute, and the plethora of information on the NACADA Web site’s Clearinghouse and Commission on Assessment of Academic Advising sites. Although the spotlight in 1994 was more on advisor evaluation and student perception surveys, much of what these authors suggested regarding assessment of advising in general, and the relationship between assessment of academic advising and institutional research more specifically, still rings true today. Further, while the 1995 authors touched upon the importance of student learning as part of the assessment process back then, the assessment of student learning outcomes in higher education, including academic advising, is the focus today. Before discussing the role of institutional research in the assessment of academic advising, a distinction between evaluation and assessment is warranted.

Despite the fact that the terms are often used interchangeably (e.g., Creamer & Scott, 2000; Cuseo, 2008; Lynch, 2000; Troxel, 2008), there are specific distinctions between evaluation and assessment in higher education (Robbins, 2009). Simply put, evaluation is a discrete judgment of value or worth (Creamer & Scott, 2000) typically performed episodically on an individual advisor. The most common form of this is student evaluation of the advising process (Habley, 2004; Macaruso, 2007). Assessment is a continuous, systematic process of collecting, reflecting upon, and utilizing information gathered from multiple data collection techniques (Robbins, 2009), focusing on the improvement of student learning and development (Angelo, 1995; Ewell, 2000; Marchese, 1993; Palomba, 1999; Pellegrino, Chudowsky, & Glaser, 2001). This includes the mastery of learning outcomes by students as a result of advising. Typically conducted at the institutional level or programmatic level, assessment may also be conducted at the advising experience level (Maki, 2004). Purposes of assessment include program effectiveness (for example, is the program meeting its goals and the needs of the students?), program improvement (identification of programmatic shortcomings and strategies to improve the advising program), and program accountability and institutional curiosity (Maki, 2002). All too often, assessment is not performed unless the accountability purpose comes into play with some external entity requesting or requiring assessment data. However, the other purposes are just as important, if not more important, reasons to perform assessment of the academic advising program (Robbins, 2009). Moreover, evaluation may be one of the multiple measures used in assessment, but a single evaluation is not assessment.

This points to the role of institutional research in the assessment of academic advising – existing relevant institutional data can serve as one of the multiple measures of any given outcome for academic advising. For example, if assessment of advising includes data on retention rates, grade point averages, graduation rates, or similar information, these are typically under the purview of institutional assessment. Benchmarking data with peer institutions or programs may also be available, as may be relevant data from any of the national student surveys such as the National Survey of Student Engagement (NSSE) (e.g., Kuh, Cruce, Shoup, Kinzie, & Gonyea, 2008). Based upon these goals of assessment and given their field of expertise and their knowledge of existing institutional data, institutional research staff should be identified as one of the stakeholders in the assessment of academic advising venture, and should be included from the start.

Once the evidence regarding achieving the identified and desired outcomes for academic advising is gathered, the resulting data must be interpreted relative to how they inform the academic advising process, what students learned as a result of academic advising, and even what advisor learning resulted from the advising experience (Robbins, 2009). The interpretation of the data is dependent upon the measurements utilized, and will vary across individual evaluation and assessment processes. Institutional research staff should be included in this analysis and interpretation, and since they have been included as stakeholders from the beginning they will be familiar with the entire process. The interpretation of the resulting data will then be followed by determination of with whom, how, and when the results are to be reported. The outcome data from the assessment of academic advising process may become part of the institutional research database, such that these results can be used to inform other research on campus.

The use of institutional data for program review of academic advising is similarly important. The common components of a program review for academic advising include

•  A historical description of academic advising (both as part of higher education and as part of the institutional history)

•  A self-study (including demographic data, delineation of the services offered and the delivery models used for advising, and outcome assessment)

•  A review of and comparison to peers

•  An external review by recognized experts in the field

The historical information provided tends to be descriptive only, while the latter three involve specific inquiry methodologies and measures. These methodologies and measures may include (but are not restricted to) the use of surveys, benchmarking data, simple advisor: advisee ratios, counts of advisees’ uses of services, qualitative data, quantitative data, and technology utilized to provide academic advising – many of which may already be maintained as institutional data. Student learning outcomes may be the most important of such measures, and the abovementioned methodologies and measures often serve as one of several measures to determine whether the desired student learning outcome has been met. For example, if a desired student learning outcome for academic advising is knowledge of curricular requirements, institutional data such as the number of students declaring a major on time and time to graduation may inform as to whether that specific student learning outcome has been met.

So, why reinvent the wheel? Use existing institutional data that relate to the outcomes for academic advising. Advisors need to become knowledgeable regarding what type of data the institution is already collecting, and to get to know institutional data people and include them as stakeholders in the assessment of the academic advising program. The relationship between assessment of academic advising and institutional research is a reciprocal one, with each informing the other, ideally resulting in more rich and useful information campus-wide.

References

Angelo, T. (1995, November). Reassessing and defining assessment. AAHE Bulletin .

Creamer, E. G., & Scott, D. W. (2000). Assessing individual advisor effectiveness. In V. N. Gordon, W. R. Habley, and Associates, Academic advising: A comprehensive handbook (339-348). San Francisco: Jossey-Bass.

Cuseo, J. (2008). Assessing advisor effectiveness. In V. N. Gordon, W. R. Habley, and T. J. Grites, Academic advising: A comprehensive handbook (2 nd edition) (pp. 369-385). San Francisco: Jossey-Bass.

Ewell, P. (2000). As sessment of learning. Denver, CO: AAHE Assessment Forum.

Kuh, G. D., Cruce, T. M., Shoup, R., Kinzie, J., & Gonyea, R. M. (2008). Unmasking the effects of student engagement on first-year college grades and persistence. Journal of Higher Education, 79 (5), 540-563.

Lynch, M. L. (2000). Assessing the effectiveness of the advising program. In V. N. Gordon, W. R. Habley, and Associates, Academic advising: A comprehensive handbook (pp 324-338). San Francisco: Jossey-Bass.

Maki, P. L. (2004). Assessing for learning: Building a sustainable commitment across the institution. Sterling, VA: Stylus Publishing.

Marchese, T. (1993). AAHE continuous quality improvement project: Profiles of campuses. Braintree, MA: The Assessment Institute.

Palomba, C. A. (1999). Assessment essentials: planning, implementing, and improving assessment in higher education . San Francisco: Jossey-Bass.

Pellegrino, J. W., Chudowsky, N. & Glaser, R. (2001). Knowing what students know. Washington, DC: National Academies Press.

Robbins, R. L. (2009). Evaluation and assessment of career advising. In K. Hughey, D. N. Burton Nelson, J. Damminger, and B. McCalla-Wriggins (Eds.) The Handbook of Career Advising (chapter 12) . San Francisco: Jossey-Bass.

Troxel, W. G. (2008). Assessing the effectiveness of the advising program. In V. N. Gordon, W. R. Habley, and T. J. Grites, Academic advising: A comprehensive handbook (2nd edition) (pp. 386-395). San Francisco: Jossey-Bass.

 Annotated Bibliography

Angelo, T. (November, 1995). “Reassessing and Defining Assessment.” AAHE Bulletin.

Suggests that student learning can be assessed indirectly through correlated measures of teaching, including the assessment of multiple dimensions of learning, use of multiple assessors, and conducting assessment over time. The processes discussed and the importance of student learning as a desired outcome makes this relevant to assessment of academic advising.

Appleby, D. C. (2008). Advising as teaching and learning. In V. N. Gordon, W. R. Habley, and T. J. Grites, Academic advising: A comprehensive handbook (2 nd edition) (pp. 85-102). San Francisco: Jossey-Bass.

Presents academic advising as a form of teaching, emphasizing the shared skills and common competencies involved in both effective teaching and effective academic advising. Discusses the use of an academic syllabus for advising, and emphasizes the importance of identifying and assessing student learning outcomes for academic advising as is done with academic programs.

Astin, A. W. (1991). Assessment for Excellence: The Philosophy and Practice of Assessment and Evaluation in Higher Education . New York: MacMillan.

Discusses the importance of evaluation and assessment in higher education, focusing primarily on evaluation and assessment of academic programs. One of the most important aspects of this work is that the author distinguishes between evaluation and assessment, and provides a discussion of the philosophical underpinnings for conducting evaluation and assessment in higher education.

CAS Standards and Guidelines for Academic Advising Web site www.nacada.ksu.edu/Clearinghouse/Research_Related/CAS.htm

The Council for the Advancement of Standards in Higher Education (CAS) is recognized as a leading authority in the identification of values, principles, and standards of practices for various facets of higher education, including academic advising. CAS offers 16 “relevant and desirable” student learning outcomes for academic advising applicable to every type of higher educational setting, as well as standards and expectations for the academic advising director and the academic advising program. These student learning outcomes (or derivations thereof) may be used as desired outcomes for advising programs, depending on the institutional and programmatic mission, goals and objectives.

Campbell, S. M. (2008). Vision, mission, goals, and programmatic objectives for academic advising programs. In V. N. Gordon, W. R. Habley, and T. J. Grites, Academic advising: A comprehensive handbook (2 nd edition) (pp. 229-243). San Francisco: Jossey-Bass.

An updated chapter following White (2000). Introduces the reader to the concepts of vision, mission, goals, and objectives, as well as the relational nature of such statements and their importance in strategically guiding an effective academic advising program. As it is from these identified aspects of the advising program which student learning outcomes are derived, this chapter provides an important basis for the understanding of the entire process involved in identifying and assessing student learning outcomes for academic advising.

Campbell, S., Nutt, C., Robbins, R., Kirk-Kuwaye, M., & Higa, L. (2005). NACADA guide to assessment in academic advising. Manhattan, KS: National Academic Advising Association.

Provides a theoretical introduction to and a step-by-step process for developing an assessment program specifically for academic advising. Included are examples and templates to walk the reader through the steps of assessment of advising.

Describes the popular methodologies used to evaluate individual advisor performance and effectiveness. Although not a discussion of assessment of academic advising, individual advisor effectiveness can serve as one of multiple forms of measurement in the assessment of student learning outcomes.

An updated chapter following Creamer and Scott (2000). The emphasis here again is on evaluation of advisor effectiveness rather than assessment of student learning outcomes for academic advising, focusing primarily on student perceptions of advisor effectiveness. Although not a discussion of assessment of academic advising, individual advisor effectiveness can serve as one of multiple forms of measurement in the assessment of student learning outcomes.

Provides an overview of the processes involved in assessment of student learning in higher education, with a focus on academic programs. The processes discussed and the importance of student learning as a desired outcome makes this relevant to assessment of academic advising.

Discusses the components of an effective advising program and the differences between process evaluation (assessment) and outcome evaluation (assessment) of advising programs. Includes a discussion of the importance of multiple forms of data measurement in these processes as well as considerations in planning and conducting assessment of advising.

Offers a systematic and collaborative process of assessing for student learning including but going beyond the classroom. Student learning is viewed as a core process of institutional learning, with assessment a necessary activity to improve educational practice and student learning. Included are clearly written definitions and examples of various assessment terms, practices, and resources including worksheets and exercises.

Describes the role of assessment in the Total Quality Management approach to higher education that was popular in the 1980s and early 1990s. Emphasizes the importance of the systematic process of assessment in improving student development and learning, using examples from specific campuses.

This article describes the similarities and differences between assessment and research, emphasizing the progression through which assessment of academic advising can evolve into a research inquiry.

Lists resources primarily for advisor evaluation with some information on assessment of advising services, including examples of evaluative instruments, surveys, and inventories developed and used by institutions of higher education, as well as standardized inventories used for evaluation of advising.

Provides searchable database of thousands of articles and resources about and related to academic advising in higher education. Users can specifically search for resources on evaluation and assessment of advising.

Discusses the importance of a mission statement for academic advising regarding programmatic, institutional, and assessment purposes, and provides an introduction to student learning outcomes for academic advising (including examples), relating student learning outcomes to the advising mission statement.

Examines assessment practices in higher education including developing learning goals and objectives, involving faculty, staff, and students, selecting and designing methods, reporting and using results, and assessing the assessment program. Examples of assessment activities are provided from all types of institutions.

Discusses how expanding knowledge in the fields of human learning and educational measurement can be used to improve assessment of what students know and how well they know it, as well as the methods used to make inferences about student learning. Included are principles for designing and using new kinds of assessments with examples provided.

This straightforward work discusses how the initial purpose for assessment is often based on external requirements for data demonstrating effectiveness, and how assessment processes are often designed with specific accreditation requirements in mind. The emphasis is on assessment of academic programs, but the message is clear and relevant to all areas of assessment in higher education.

Although written primarily in regard to career advising, the concepts and processes included are also used in the assessment of academic advising. This chapter discusses the reasons for conducting assessment of advising, the differences between evaluation and assessment, and detailed concepts, steps, and processes of each. Emphasis is placed on the assessment of student learning outcomes for advising, the mapping of the developmental and learning opportunities for these outcomes, and the use of multiple outcome measures for any single desired outcome as well as acting upon the results of assessment.

Introduces the importance of assessing student learning in areas of higher education beyond the academic program, including student affairs and academic advising. Discusses the importance of identifying student learning outcomes and provides a general discussion of qualitative and quantitative data in outcome assessment of student learning.

Provides a historical review of outcomes assessment in higher education and a general guide to designing, implementing, and evaluating assessment programs. Included are the topics of the role of the political context on assessment practices, factors contributing to the push for assessment, assessment as it has evolved through accrediting agencies, and suggestions for evaluating the effectiveness of the assessment effort at institutions of higher education.

Delineates the various basic elements of an assessment process for academic advising at a somewhat cursory level, discussing in general terms the importance of assessment and general suggested steps in the process.

Provides an overview of program evaluation, the tools to design and perform assessment in student affairs, and how to communicate the results and implement changes based on results. Included are discussions of outcome measures for programs and services, the use of both qualitative and quantitative methods, benchmarking, and professional standards.

Discusses the necessity of mission, goals, and objectives in developing a coherent and purposeful advising program, including the connection between institutional mission and advising mission, the development of relevant goals, and a brief introduction to the importance of these three features of and advising program in the assessment of academic advising.

This extensive and well-written handbook provides eight suggested methods for outcome-based program evaluations, including step-by-step descriptions of these methods, data collection and analysis procedures, and real-life examples of these programs. The authors emphasize the role of theory, examination of previous research, selection of an appropriate evaluation methodology, and the importance of realistic goals in conducting this form of research. Presented are not only the positive aspects of such programs, but also a detailed discussion of the constraints and issues involved, including examples of unexpected or negative results and how to deal with these.