Richard
Light (2001) noted that “good
advising may be the single most underestimated characteristic
of a successful college experience” (p. 81). But how does a campus
community know how their advisors are doing? In
this era when “assessment” is at the forefront of the higher education
community, attempts to develop campus-appropriate instruments
to evaluate advising quality are strongly encouraged. A variety
of factors affect the development of such an instrument and include
such things as: current campus advising model and method of evaluation;
level of constituent support for the development of a new instrument;
diversity and overall demographics of the campus; and methods
available for distribution/collection of the instrument. These
factors lead to a bevy of questions, including “Who is responsible
for the development of the instrument?”, “What questions should
be asked?”, “What format should be used?”, and “How will data
be collected and analyzed?”
Faculty
and staff at North
Central
College
found
that we were dealing with many of these same issues when we determined
that the current advisor evaluation was antiquated, ineffective,
and not meeting the needs of the college. The task of writing
a new, original, online advisor evaluation was charged to the
staff of the Advising
Center
. After
two years and multiple meetings, drafts, feedback sessions, and
focus groups, the new evaluation was introduced during priority
registration in spring, 2006. The following article details the
more pertinent stages of the process, along with five specific
recommendations that might be helpful to advisors and administrators
wishing to undertake a similar challenge on their respective campuses.
North
Central
College
is an
independent, comprehensive college of liberal arts and sciences
with approximately 2200 undergraduate students. There is a faculty-based
advising system with a satellite Advising
Center
. Over
a period of years, there was growing discontent with the advisor
evaluation form that had been in place for decades. As part of
a more general review of the advising system, a faculty committee
issued a report in 2004 suggesting that a new advisor evaluation
be developed. This recommendation was supported by the Dean of
Faculty, Registrar, and faculty governance. Without the support
of these appropriate campus power brokers, the development of
a new evaluation would not have taken place. Recommendation
#1: Become familiar with specific campus processes and engage
appropriate campus power brokers in a discussion of the need for
a new advisor evaluation. Additionally, include an individual
widely respected for his or her academic advising as a major player
on the development team.
In
early 2005, Advising
Center
staff
began work on the new evaluation. Initially, a broad-based survey
of information was conducted, using a variety of sources, but
primarily using the NACADA Clearinghouse of Academic Advising
Resources . This survey of information referenced content,
style, delivery methods, and anything else deemed relevant with
regards to existing advisor evaluations. As summarized by Cuseo
(2002), research has repeatedly demonstrated that students value
advisors who are 1) available/accessible; 2) knowledgeable/helpful;
3) approachable; and 4) able to serve as a mentor. As this information
was conveyed back to the faculty committee providing oversight
to the development of the new advisor evaluation, it was accepted
that the content of the College’s new advisor evaluation should
address these same four areas. Additionally, commercially available
advisor evaluations, such as the Academic Advising Inventory (Winston
and Sandor, 1984) and ACT’s Survey of Academic Advising, were
offered as options. While considered, all parties were in agreement
that the new evaluation should be something original that reflected
the unique nature of North
Central
College
but still
addressed the four core areas mentioned. Recommendation
#2: Review the literature and determine what is already available
while maintaining primary focus on specific campus needs and culture
when developing the content of the evaluation. Use knowledge and
expertise as advising professionals to inform decisions.
Upon
sharing these findings with the faculty committee and as development
of the evaluation began, several key logistical points needed
to be addressed and agreed upon by the staff of the Advising
Center
and the
faculty committee. First, the length of the evaluation was important
– consensus was that it not take more than 5-8 minutes to complete.
Second, there should be both quantitative and qualitative portions.
Third, the instrument should be used not just for evaluative purposes,
but also to educate students and faculty advisors about the kinds
of topics that could be discussed as part of the advising relationship.
The specific wording used in the instrument, particularly in Part
I, reflects this concern. Fourth, the intent was to place the
evaluation online and tie its completion to students’ priority
registration during the spring term although there needed to be
an “opt-out” option for students who wished not to complete the
evaluation. Finally, the evaluation should be anonymous in that
individual student responses would not be conveyed to the advisor,
but rather an overall summary of the results would be shared.
Over the course of three months, six drafts were presented to
the faculty committee before the final version was eventually
accepted and approved by 89% of the full faculty. Recommendation
#3: Keep others informed and solicit input from appropriate parties
as much as possible throughout the process.
Information
Technology Services (ITS) was a major player in this process.
Advising Center staff maintained regular contact with ITS and
continually updated them on the status of the evaluation discussion.
The plan all along was to have the evaluation completed online,
anonymously, by students, yet have enough data collected “behind
the scenes” to forward results to specific advisors as well as
generate campus-wide advising reports. It is beyond the scope
of this article to delve into the technical details of the process,
but, needless to say, this project was a major undertaking by
ITS staff and required a significant commitment from that particular
office. After full faculty approval of the new evaluation, it
was turned over to ITS in November, 2005, with a request to have
it online by April, 2006, in time for priority registration. ITS
developed several different versions (consisting of mostly cosmetic
differences) before completing the final version in February,
2006. This version was then presented to a student focus group
for a test run. After a few minor changes, the evaluation was
complete and ready to be implemented. [Click to view North
Central College’s Advisor Evaluation .] Recommendation
#4:
Become
friends with ITS and maintain regular contact with them throughout
the process. Set hard deadlines for various stages of the process.
The
evaluation was placed online in time for priority registration
in the spring. Students accessed the evaluation by logging in
to their campus computer accounts. In order to achieve as high
a response rate as possible, and with the support of all levels
of administration, a hold was placed on students’ accounts that
prevented them from registering unless they had completed the
advisor evaluation or elected to opt out. Thus, students were
not required to complete the evaluation, but had to at least visit
the evaluation and consciously elect not to complete it (only
6% of the students opted out). Upon either submitting the evaluation
or opting out, the hold was automatically released and the student
was free to register. Alternative measures for capturing graduating
seniors, who were not registering for the upcoming academic year
and thus not impacted by the registration hold, were only partially
successful and need to be revisited.
During
the several week time span when students were completing the evaluation
and registering, regular contact was maintained with ITS to determine
if any problems had developed or if any changes were needed. Very
few issues arose and all parties were quite satisfied with how
the initial roll-out of the new evaluation took place. However,
the one area that did not get enough attention during the development
stage was the reporting of data. ITS collected and maintained
the data in a very raw form, but little attention was paid to
converting that data into user-friendly results until considerably
later in the process. Over a period of several months during the
following academic year, ITS and Advising
Center
staff
developed an appropriate format for the results to be distributed
to each advisor; ideally, this task would have been a more significant
part of the evaluation development. Recommendation
#5: Consider the data reporting format early in the development
process.
In
conclusion, all parties involved in the process are pleased with
the new evaluation, which is a vast improvement of the paper evaluation
previously used. However, by no means is it considered an unchangeable
document; feedback will routinely be gathered and evaluated to
determine if revisions are necessary. Hopefully, the recommendations
included here and summarized below will prove to be beneficial
to other institutions and expedite the development process should
a similar task be undertaken. Please see the resources listed
in the annotated bibliography for opportunities to read more about
advisor assessment and evaluation.
Summary
of recommendations for developing an original advising evaluation
instrument:
- Know campus
processes and keep power brokers informed.
- Consider
existing instruments but remain cognizant of the campus’s culture.
- Solicit input
throughout development.
- Be friends
with ITS, but set deadlines.
- Include discussion
of the format for reporting data early in the process.
References
Council
for the Advancement of Standards ( CAS
). (2005). Academic Advising:
CAS
Standards and Guidelines. Retrieved January
15, 2007 from
http://www.nacada.ksu.edu/Research/Standards.htm
.
Cuseo,
J. (2002, March 19). Advisor evaluation. Policy Center on the
First-Year of College.
Light,
R. (2001). Making the most of college: Students speak their
minds. Cambridge,
MA:
Harvard University
Press.
Winston,
R. B., and Sandor, J. A. (1984). The Academic Advising Inventory.
Athens,
GA:
Student Development Associates.
Annotated
Bibliography
Astin,
A.W. (2002 ). Assessment for excellence: The philosophy and
practice of assessment and evaluation in higher education. Westport
, CT
: American Council of Education
and The Oryx Press.
Banta,
T. R., Jansen, M. J., Black, K. E., & Jackson, J. E. (2002).
Assessing Advising Outcomes. NACADA Journal, 22 (1),
5-14.
Knapper,
C., & Cranton, P. (Eds.). (2001). Fresh approaches to the
evaluation of teaching. New Directions for Teaching and Learning,
No. 88. San Francisco
: Jossey-Bass.
