Research versus Assessment: What's the Difference?
Victoria A. McGillin, Advising Assessment Commissions Member; NACADA Research Committee Past Chair
One frequent question heard from NACADA
members is, "What's the difference between research and assessment?" The
following is an effort to articulate both the overlap, and the
distinctions, between these two.
In our workshop on advising research and grant proposal development,
the NACADA Research Committee discusses the similarities and
differences between research and assessment. The following is a
The goals of experimental research and program assessment
differ significantly. While research focuses on the creation of new
knowledge, testing an experimental hypothesis, or documenting new
knowledge, assessment and evaluation focus on program accountability,
program management, or decision-making and budgeting.
That is, while research is designed to document or
measure a phenomenon not formerly recorded, e.g. applying a new theory
to an advising encounter and documenting how well a model 'explains'
what is going on between advisor and advisee, program assessment
provides information to your campus about whether you are achieving
prescribed goals, expending resources wisely or meeting a documented
While the methods employed in good program assessment and
evaluation may be similar to those used in good research, they need not
be. The range of methods employed in both may range from subjective
field observations through objective questionnaires. If your key
assessment question is how your campus advising compares to national
data on advising (such as the ACT survey), the use of a
nationally-standardized, reliable and valid instrument would be crucial
to answering that question. However, nationally-standardized instruments
may not always 'fit' your campus as they may utilize differently-named
services or institutional structures not present. When an existing
measure just won't do, good research AND good assessment practices call
for the development of a reliable and valid new measure. We must be wary
of developing a 'quick and dirty' measure in an effort to just 'get a
quick answer' to our questions, without taking the time to ensure our
measures are reliable or valid for our own campuses.
One major methodological difference between research and
assessment is that researchers will 'experimentally manipulate a
variable' (for example, randomly assigning students to one model of
orientation or another), while program evaluation tends to be non-random
(we rarely have the luxury of such random manipulation of our
students). At best, assessment just looks at 'natural' differences that
emerge, such as comparing students who chose one orientation event over
Just as experimental research and program assessment
differ in their goals, they also differ in the use of their results.
Research results are expected to be generalizable beyond one's own
campus, with implications for similar institutions or similar
populations. Program assessment results are applicable only to one's own
campus. While both are of great value, research should contribute new
knowledge to the field. When opening the NACADA Journal, you
expect documentation of research that began as an advising question and
culminated with statistically significant research of an advising
method, theory or programmatic intervention that you can apply with some
assurances of success.
Conversely, program assessments are designed to be
site-specific and crucial for campus decision-making. Good program
assessment ensures that you are responsive to the changing (or
unchanging) needs of your populations. You may want to reuse the same
measure each year to document the high level of program success over
time. Your results may be particularly appropriate for the NACADA
Journal's Tool Box section that highlights examples of best advising
practices that link to current research in the field.
Finally, while research should provide possible answers to identified
questions, it should also generate new research questions from the
results. For example, if one's data showed that both male and female
students were more critical of male advisors than female advisors, the
researcher would want to explore this research question further.
Assessment, however, looks for answers. Viewed from an assessment
standpoint, such results might lead to interventions, such as additional
training for male advisors and the desire to assess the effectiveness
of that intervention on one's campus.
Given these differences, it is not surprising that there
are vastly different audiences intended for program assessments, as
compared to research. Assessment results are targeted for the key
decision-makers on your campus. When budgets are cut, new programs
proposed or accreditation rolls around, assessment/evaluation reports
help you make a case for your program. As I am fond of saying, 'Whoever
gets to the table with numbers first, wins.' The ability to produce an
executive summary of key assessment findings (no more than 2 pages)
documents the effectiveness of your work and moves your programs to the
top of the funding lists, ahead of those supported only by anecdotal
In contrast, research is intended for the field of
advising and higher education as a whole. Your results will be read by
many, debated and critiqued, copied and expanded upon to generate even
newer knowledge. While a one-page executive summary submitted to your
dean may get you funding for a new advising initiative, your colleagues
outside your institution look for full documentation of the research
that led you to this question, the literature review of the theory that
guided your process, details on the methods you used, the results
(strengths and weaknesses) of this study, and the conclusions you drew
based upon your research. The 15-20 pages, with bibliography, necessary
for a published article, would only gather dust if submitted as part of a
funding request to most deans or VPs.
Connecting It All
Let me conclude by emphasizing the most crucial point of
connection between assessment and research. Good assessment/evaluation
can be expanded into good research. Good research should lead to even
better assessment procedures. Good assessment makes use of the best
conceptual and theoretical models and the best research measures or
methods. With valid and reliable measures, campus-specific questions may
have national implications. A phenomenon identified on your own campus
may be the cutting edge for an issue of significant importance.
Finally, find significant resources on advising assessment on the Assessment of Advising Commission Web Page.
We urge you to consult with the NACADA Research Committee. They seek cutting-edge proposals. Your assessments may lead to a critical (and fundable) piece of research!
Victoria A. McGillin
From the President
Ruth Darling, NACADA President
After every NACADA National
Conference, I return to my campus with a true sense of belonging to a
profession that has student learning and development as its core value. I
am reassured that I associate with a diverse group of advising
colleagues who approach their life's work with this point of view or
perspective. I am also reassured that there is a professional
association that has as its focus the promotion of academic advising
within higher education along with the professional development of its
members. These beliefs shape how I think about NACADA, my colleagues and
The ideas of perspective/point of view and shared beliefs are an
important piece of a graduate seminar I teach titled 'College Student
Development Theory and Practice.' Throughout the course, the students
and I explore the ideas and concepts underlying various paradigms and
the impact these ideas have on theory, research and practice. To guide
our discussions, I use Guba's (1990) definition of paradigm as - 'an
interpretive framework, a basic set of beliefs that guides action' (p.
17). Together, through the examination of psycho-social/identity
development, cognitive-structural and typology theories, we hope to
arrive at a basic set of beliefs that will guide their personal theories
and practice as entry- level professionals.
NACADA is developing a strategic plan that promotes a
distinct mission and reflects a shared set of beliefs. The plan will be
an 'interpretative framework' that 'guides action' as the leaders and
membership address the work of the association and ultimately the work
we all do in our educational contexts. Within higher education, NACADA's
role is to:
- Champion the educational role of academic advising to enhance student learning and development.
- Affirm the role of academic advising in supporting institutional mission and vitality.
- Address the academic advising needs of higher education.
- Advance the body of knowledge of academic advising.
- Encourage the contributions of all members and promote the involvement of diverse populations.
During this next year, the leadership of the various NACADA Divisions
(regions, commissions/interest groups and committees) as well as the
Board of Directors will be calling on you, the membership, to be
involved in shaping our work, in shaping our goals and, ultimately, in
shaping our shared vision of NACADA. As President, I encourage you to
get in involved with the association. The Executive Office staff, your
regional chairs or the chairs of various commissions, interest groups
and committees can serve as your contacts and information sources as you
seek a meaningful way to connect with your colleagues and with NACADA.
We need your involvement, your knowledge and your skill as we work to
promote academic advising and support the learning and development of
In closing, I hope to see many of you at the regional
meetings this coming spring where we can, once again, connect with
colleagues and explore our 'advising paradigm!' Thank you for the
important work you do each day!
Ruth Darling, President
Guba, E.G. (1990). The alternative paradigm dialog, In E.G. Guba (Ed.), The paradigm dialogue (pp. 17-30). Newbury Park, CA: Sage.
From the Executive Office
Roberta 'Bobbie' Flaherty, NACADA Executive Director
- the much used word within NACADA these days and the most often
confused. As the Association strives to bring greater professional
recognition to advisors, it is exploring a number of ways to recognize
the knowledge and skills that advisors attain and utilize in providing
effective academic advising.
Currently, there are four distinctly different initiatives being
considered that in some way involve a form of the word certificate.
They include a 'participation certificate', an 'advising certificate
program', a 'graduate certificate program in advising', and 'advisor
certification'. Now do we see why folks might be confused?
The participation certificate is simply a NACADA certificate
given for completion of a specific NACADA professional development event
such as the Academic Advising Summer Institute and/or the Advising
Administrators' Institute. It simply denotes that a person participated
in that event and hopefully that they gained some advising knowledge
from that participation. That knowledge, however, is not assessed in any
An advising certificate program is being explored
by the Professional Development Committee as an opportunity for members
to obtain recognition for having participated in a series of
professional development activities that would cover a broad spectrum of
advising information. A 'certificate' might then be awarded to verify
exposure to this broad spectrum of knowledge. Again, this knowledge
would not be assessed or verified in any way.
The Graduate Certificate Program in Academic Advising
is a totally independent Graduate Program offered by Kansas State
University. Participants in this program earn academic credit and upon
completion of the five courses, receive a Graduate Certificate from
Kansas State University verifying completion of the program. Of course,
each course includes knowledge assessment.
Advisor Certification is the subject of the work of a NACADA
Task Force charged with exploring the potential for a 'professional'
certification program and designation for academic advisors. This Task
Force is identifying the knowledge and skills that effective academic
advisors should possess and how this knowledge and these skills can be
assessed to earn the designation as a 'Certified Advisor'. Any or all of
the above programs might serve as leading toward the professional
certification designation through the attainment and assessment of the
knowledge and skills presented by each along with experience and other
learning opportunities. The Executive Office is seeking estimates from
'certifying' entities as to the costs that would be incurred in
assessing the advising competencies as identified by the Task Force and
other expenses to anticipate in the administration of such a program.
That information will be utilized to determine if such a program would
be cost effective and viable for academic advisors.
I hope this helps everyone understand the many uses of
the word 'certificate' and how your association is working to enhance
the recognition you deserve for your continued education and your
expertise. We welcome any suggestions for synonyms!
Roberta 'Bobbie' Flaherty
NACADA Executive Director
NACADA Journal Seeks Editorial Board Members
A. Purpose and Overview
The NACADA Journal, the journal of the National Academic Advising Association, seeks to enrich the knowledge, skills, and professional development of people concerned with academic advising and student success in higher education. Through its journal and other activities, NACADA is dedicated to the enhancement of student development by supporting the professional growth of academic advisors and the advising profession.
Publications in peer-reviewed journals (preference for NACADA Journal publication)
Experience in academic advising
Interest and strength in quantitative and/or qualitative research methodology
Active membership in NACADA
Terminal degree (preferred)
Review manuscripts for significance, appropriateness, research design, analysis, and quality of writing within 30 days of receipt.
Provide constructive feedback to authors in order to improve manuscripts.
Ability to use Microsoft Word and email with attachments.
Annual attendance at the Editorial Board meeting held during the NACADA National Conference each year. (Preferred)
Editorial Board members serve three-year terms that begin and end at the national conference. An Editorial Board member may serve non-consecutive terms.
Applicants familiar with the field of academic advising who are interested in seeking membership on the Editorial Board should submit an email message to Journals@ksu.edu stating interest in and rationale for serving as a member of the NACADA Journal Editorial Board. A professional resume prepared in Microsoft Word '.doc' format should be attached.
Applications will be considered on a rolling basis until all positions are filled.
Newly Appointed NACADA Leaders
NACADA President Ruth Darling has appointed the following members to leadership positions beginning in October 2003. Congratulations to you all and a BIG thank you for agreeing to serve your organization!
Administrative Division Representative: John Mortensen
Regional Division Representative: Brian Glankler
Journal Co-Editors: Terry Kuhn and Gary Padak
Administrators' Institute Advisory Board:
Susan Campbell (Chair), Alice Reinarz, Rich Robbins, Gene Calderon, Lynn Freeman, Vicki McGillin, Carolyn Collins, Tom Grites, Albert Matheny
Summer Institute Advisory Board:
Wes Habley (Chair), Nancy King, Susan Campbell, John Burton, Tom Kerr, Dorothy Turk, Wanda Martin, Peggy King, Casey Self, Betsy McCalla-Wriggins
In addition, Barbara Bucey is Program Chair for the 2004 National Conference.
CAS Standards Revision Announced
The Council for the Advancement of Standards in Higher Education (CAS) announces the release of a major revision of its landmark publication, the Book of Professional Standards for Higher Education, and an updated version of the CAS Self-Assessment Guides (SAGs). Available August 29, the new 2003 book of standards and guidelines incorporates significant updates, including new general standards with greater detail about desired outcomes in student learning and development. The standards for Academic Advising are presently under going a major revision as well. Newly revised self-assessment guides (SAGs) feature an effective means for measuring how these standards are being met in all 30 functional areas. SAGs are available in both print and interactive CD-Rom formats (both PC and Mac). The CD-Rom also includes a new PowerPoint presentation and an E-learning course to assist institutional staff and faculty members in completing the SAGs.
Further information and online orders are available through the CAS web site ( http://www.cas.edu ). The book, which replaces the 2001 edition, is available separately or in a special package with the interactive CD-Rom including the full set of SAGs. The SAGs are also available for individual purchase and immediate download. The book and SAGs can be ordered from CAS, One Dupont Circle, NW, Suite 200, Washington, D.C. 20036-1188. Telephone: (202) 862-1400, Fax: (202) 296-3286.
The National Academic Advising Association is a member of CAS and strongly endorses these standards to its members. Founded in 1979, CAS is a consortium of educational associations that promotes quality educational practices through the promulgation of standards and guidelines for 30 programs and services in higher education. Individuals and institutions from the 32 CAS member organizations comprise a professional constituency of well over 100,000 professionals. Excellence in educational practice is a central goal of CAS that is achieved through the implementation of standards in all areas of practice in higher education. This vision for excellence is consistent with contemporary goals for accountability and bring an effective approach-the CAS approach-to program assessment. The CAS approach is based on concepts of self-regulation and self-assessment, and all CAS materials are geared to this approach to quality assurance in higher education.
CAS serves higher education programs and services by providing:
- Descriptions of state-of-the-art programs and services
- Designs for programs and service development and assessment
- Criteria for institutional self-studies and preparation for accreditation
- Opportunities for staff development
- Outcomes for student learning and development
- Frameworks for accountability
Visit http://www.cas.edu for orders for all CAS materials, a full account of the work of CAS, links to each member association Internet sites and to the leadership of CAS, and a brief PowerPoint presentation providing an overview of the CAS approach.
The Justification for Case Studies in Advisor Training and Development
Heidi Koring, Chair, Advisor Training and Development Commission
Training and development of
advisors becomes ever more central to the effectiveness of the advising
process with the increasing diversity and complexity of our students'
environments. While there is no 'one-size-fits-all' method for advisor
training and development, case studies are among the most useful items
in the trainer's tool box.
Case studies are an effective part of the training process whether
advisor training takes place as a single workshop, or as series of
continuing in-service meetings, or in formal presentations or informal
discussions. The use of case studies was pioneered by the Harvard School
of Business faculty in the late 1960's. Currently used to enhance
skills development of a variety of populations, case studies add
richness and complexity to advisor training, reflecting the complex
environment of contemporary college students. Case studies not only help
advisors come to grips with the ambiguities and complexities of student
development, but aid them in improving human relations and problem
solving skills. Case studies can be used as exemplars of carefully
defined problems, providing advisor with opportunities to practice
analysis of an advising situation. Presenting a platform for addressing
differences in advising styles, case studies stimulate personal and
professional growth and reflection. Cases can be used with advisors at
all levels of experience, engaging them in discussion and simulating
problem solving in real life situations.
Good cases are realistic and personalized to the advisors' milieu. They
are dramatic enough to engage the participants and ambiguous enough to
allow for multiple interpretations. To prepare effective cases, collect
anecdotes from advisors throughout the academic year. Asking advisors to
reflect in writing on difficult situations can yield rich material for
case studies. When turning the raw material of experience into cases for
training, it is wise to assemble a team of stakeholders to read the
anecdotes and discuss the issues addressed in each. Simultaneously,
develop a list of resources that could help advisors address each case.
Work from the issues creating composite cases that address one main
question and at least one subordinate issue. If you are working from
real experiences, make sure that details are changed so the persons
involved in the original anecdote are not recognizable. Divide cases
into categories by issue for use when planning training events.
Since the traditional case study approach uses small or large group
discussion, begin with advisors enumerating the issues presented in the
case. Discourage any tendency to find easy closure by encouraging
participants to consider the case from different characters' points of
view. Ask 'what if' questions. Consider locus of control and
responsibility issues. What aspects of the case are within the advisor's
locus of control? What aspects are not? Ask probing questions about
each character's motivation. Look for hidden agendas. Use a team
approach to problem solving by encouraging the exploration of resource
and referral possibilities. Discuss how college confidentiality policies
would affect each case. If appropriate, ask if the gender or ethnicity
of characters affect the outcome of the case. Explore several related
cases to develop the best practices or procedures for dealing with a
particular advising challenge, at your institution.
Less traditional delivery methods can be used when
approaching cases. Enlist the cooperation of theater or broadcast majors
to make videos acting out specific cases. At some colleges and
universities, faculty and student organizations are eager to produce
case study vignettes as projects. The NACADA faculty advisor training
video contains eight brief vignettes, six of which show a developing
relationship between a first year student and a new faculty advisor and
two scenarios exploring the needs of adult students. If the training
event includes trainees who are comfortable with each other, have
participants role play cases. Begin an advisor training electronic list
which features one case a month for discussion.
To stimulate your use of case studies with advisors, here's a
scenario that can be adapted to your institution for advisor training:
Lisa is a first year student from a neighboring state.
She attended a competitive 'magnet' school with a 90.3% average and 1200
SAT's (27 Composite ACT). During orientation she tells you she's
considering a pre-veterinary science track because she loves animals.
Her midterm grades are B's and C's in calculus and biology. When she
meets with you at midterm, she slumps in her chair and doesn't make eye
contact. She's lost a lot of weight. Through her hesitant replies, you
learn that math and science are tougher than she expected. She says
she's dumber than she thought. She has a lot of headaches and sleeps a
lot. She's missing classes because she says it doesn't matter if she
goes. She tells you she's thought about going home, but is sure her
family would just say she's a failure. Besides, her parents are getting a
divorce and she's not sure where she would live. She says she knows you
can't help, so maybe she'll just 'give up.'
- What specific problems or issues are raised by this scenario?
- What additional information might you need to handle the scenario? How would you get it?
- What problems or issues would you refer? How?
- What problems of issues could you address yourself? How?
Want to know how others are using case studies? The new monograph, Advisor Training: Exemplary Practices in the Development of Advisor Skills, features several Exemplary Practices utilizing case studies.
Academic Probation, Dismissal and Reinstatement Issues: A Research Challenge
Johanna Pionke, Probation, Dismissal & Reinstatement Issues Interest Group Chair
Why do some students fail to
succeed in college? What interventions are most successful with these
students? There is great demand for research revolving around these
questions. As chair of the Probation, Dismissal & Reinstatement
(PDR) Issues Interest Group, I challenge you to approach your PDR
students from a research perspective.
Students typically do not come to college expecting to
fail. Instead, most enter college with the expectation that they will
have the opportunity gain knowledge that can help them earn a better
living for themselves and their families. Research verifies that
students often believe that there are few reasons why they will not
succeed. They view academic probation or dismissal as something that
will not happen to them.
Bartlett (2002) cites the 2001 Cooperative Institutional Research
Program (CIRP) Annual Survey of Freshman Students showing that 44.1% of
freshmen reported earning 'A' averages in high school. 57.5% of freshmen
estimated their chances of making at least a B average in college as
being very good. 76.5% of students expected to earn a bachelor's degree
while 20.8% thought they had a very good chance of graduating from
college with honors. Only 0.9% felt there was a good chance they might
drop out of college temporarily, while 0.7% felt chances were very good
that they would drop out of college permanently. Among this same cohort
of students, 67.9% rated themselves above average in academic ability,
though only 45% ranked themselves above average in writing ability and
44.2% ranked themselves above average in mathematical ability.
Despite students' positive attitude regarding their
academic abilities, many end up on the academic probation, suspension or
dismissal rolls. At the conclusion of each academic term, advisors,
faculty and administrators review these students' academic progress and
wonder why they were not successful. We look for ways to identify, or
predict, those who are at greatest academic risk so that we may prevent
their downward spiral. We question whether we should intervene with
students who are struggling academically, or if it is better to invest
resources on more successful students.
In journals, books and other publications, we search research for
information relating to academic recovery issues, yet find little
available. At conferences, it's often standing room only' in sessions
discussing intervention programs at other institutions. Yet different
student and program variables affect an institution's intervention
program. Programs vary widely in terms of their requirements, structure,
and level of intrusiveness. Some require a weekly class while others
rely upon regular contact with advisors or mentors. Some intervention
programs utilize group activities and tutorial support services, while
still others require counseling services. Some programs are organized at
the departmental level; others are college wide.
With so many different variables, it is difficult to
attribute student academic success or failure directly to participation
in an intervention program. What roles do student characteristics play
in a student's ability to succeed? Do students' academic preparation,
job and family responsibilities, study skills, or locus of control
affect success? Is there a way to account for these variables?
At the conclusion of each academic term, I challenge you to look for
research questions within the components of your institution's
probation, dismissal and reinstatement procedures. Turn these into
research projects and share your results with the Probation, Dismissal
& Reinstatement (PDR) Issues Interest Group.
Kent State University
Thomas. (2002, February 1). Evaluating Student Attitudes is More
Difficulty This Year. The Chronicle of Higher Education, Volume 29,
Issue 21, p. A35, 4p.
At the NACADA Clearinghouse -
Higgins, Beth (2004, December 1) 'Advising Students on Probation'
Jennifer L. Bloom, Member Career Services Committee Chair
Dear Career Corner: I
just found out that I have been invited to participate in a video
conference interview-do you have any advice on how to approach this
interview?-Signed, Video Neophyte
Dear Video Neophyte:Congratulations
on making it to this preliminary interview stage-it means that the
written materials you submitted caught the attention of the selection
committee. The key now is to prepare like you would for any interview-do
your homework on the institution, the position, and your potential new
boss and colleagues. Please request the full job description from the
search chair as well as other materials that will prepare you for the
interview-strategic plans for the unit and/or the institution, written
materials that the institution distributes to prospective students,
organizational charts, mission statements, etc. You will also want to
contact people in your network of colleagues that are or have been
affiliated with this institution. Find out as much information about the
position, the person who held this position previously and why they
left, and the culture of the unit.
Here are some specific tips concerning the video conference itself.
- Do not use your institution's video facilities for a
position at another university without the full consent of your boss.
Keep in mind that most Kinko's stores have videoconferencing facilities.
- Practice the connection and do a mock interview before the actually interview.
- Establish which party will be responsible for the
reconnection if the connection fails. Always have a telephone number
that you can call on the other end in case of a problem. Do not let a
failed connection faze you.
- Make sure you arrive at the facility early and eliminate all potential distractions.
- Try not to talk over other people-wait for them to complete their sentences or questions before responding appropriately.
- Dress like you would for an in-person interview.
- Have fun-smile, look happy, and share your enthusiasm for this position.
This new NACADA Newsletter feature will be a regular column. Submit questions on-line at http://www.nacada.ksu.edu/AdministrativeDivision/career.htm. Questions will be answered anonymously.
Want to read more about it?
Kennedy, J. L. (1996). Job interviews for dummies. Foster City: IDG Books Worldwide Inc.
Krannich, C. R.,& Krannich, R. L. (1999). 101 dynamite answers to interview questions. Mannassas Park: Impact Publications.
Martin, C. (2001). Interview fitness training: A workout
with Carole Martin the interview coach. San Ramon, California: Interview
Martin, N. A. & Bloom, J. L. (2003). Career
Aspirations & Expeditions: Advancing Your Career in Higher Education
Administration. Champaign, IL: Stipes Publishing.