Assessment FAQs A through Z for Campus
(work in progress; please send feedback! To
kate.alley@sdsmt.edu )

Nature / purpose of academic assessment

A. What is assessment, exactly?
See http://www.hpcnet.org/assessmentbasics on this site

B. What is the purpose of academic assessment for Tech?
The primary reason we assess student achievement and academic programs must be to improve student learning. That said, we also “do assessment” in order to fulfill the requirements of our various accrediting agencies and to meet the expectations of state- and national-level constituencies.

C. What or who gets assessed?
Every academic and academic support unit gets assessed. This means that all academic programs as well as the divisions within Student Affairs and Academic Affairs needs to be assessed for effectiveness to satisfy the requirements for our comprehensive institutional accreditation by the Higher Learning Commission (formerly the NCA).

D. What are some of the most common tools or techniques of assessment?
Here is a listing of commonly used tools:
1. Exit exams
2. Professional certification tests or standardized field tests
3. Exit or other surveys
4. Focus groups or questionnaires
5. Standardized examinations
6. Locally developed exams
7. Oral exams or interviews
8. Behavioral observations
9. Reviews or evaluations by an external examiner
10. Portfolios
11. Simulations / computer modeling
12. Performance appraisals
13. Rubric or primary trait analysis of work

E. Why can’t we just use grades as course or program assessments?
Here are two answers:
1. You can use grades, but only if you have some mechanism for verifying that the grades mean the same thing among all those who assign them for a given assignment or course. This process is often called “norming” and it usually involves multiple instructors working with “primary trait matrices” or some other tool for comparing how a student work artifact is evaluated among instructors. There is an entire book about the use of grades as assessment measures: Walvoord, Barbara, & Anderson, Virginia Johnson. (1998). Effective grading: A tool for learning and assessment. San Francisco: Jossey-Bass.
2. You can’t use grades to verify that the objectives for a given assignment or course have been met because the person assigning the grades is the same person who is offering the grades as verification. So, the bottom line is that, while no one wants to say that instructors can’t be trusted or that grades don’t “mean anything,” grades are not regarded as a convincing assessment measure in and of themselves. (Unless, of course, you verify what the grades mean as described above.) Grades are often used as an assessment measure in conjunction with other measures, but using them is frankly a “red flag” to accrediting agencies.

Big Picture Stuff

F. How many different accrediting agencies do we need to satisfy when it comes to assessment?
The main agencies we deal with at SDSM&T are the Higher Learning Commission or HLC (formerly the NCA) and the Accreditation Board for Engineering and Technology (ABET). Chemistry is accredited by the American Chemical Society. Physics does not have an accrediting agency. Computer Science is accredited by the Computer Science Accreditation Board (CSAB); however, CSAB is moving to become part of ABET. For more on accreditation within specific programs see the
http://www.hpcnet.org/individualprograms section of this site.

G. What is the relation between ABET and the HLC in regards to assessment?
As you may guess, the answer to this question is pretty involved. However, the basic answer is that ABET is more focused on assessment at the program level. Their literature stresses that they evaluate the quality and effectiveness of whole programs. ABET also stresses the necessity of building external constituency input into the assessment cycle (See the ABET “2-loop process” at http://www.abet.org/twoloops.html  

Of course this doesn’t mean that ABET-accredited programs do not look at the learning gains of individual students or that the HLC is mostly focused on assessing and improving the learning of individual students. These are generalizations. You will find, however, in assessment plans for general education or programs in the humanities and social sciences more course-level assessments; analyses of the learning gains of individual, specific students (through the use of reflective surveys or portfolios, for instance), and a stronger reliance on qualitative assessments.

H. Why doesn’t the ACT/CAAP test satisfy our obligation to assess learning in the Gen Ed program?
Well, the results of the CAAP and the ACT/CAAP gains analyses do help us considerably in documenting student learning in some specific areas; however, there are weaknesses in using standardized tests, and foremost among them is the fact that some GEP outcomes are not assessed by the tests. The bottom line is that we need to use more than this one measure to assess how effectively we are achieving our 7 GEP goals (These 7 goals are listed in the attachment below). For the sake of clarity, here is an enumeration of some of the issues that arise with standardized tests, such as the ACT or CAAP:
1. The test may not measure exactly what we teach. For instance, the math component of the CAAP measures college-level algebra but not calculus.
2. The test does not measure certain GEP goals at all, such as effective oral communication.
3. The test measures certain GEP goals only by approximation. For instance the Social Sciences subscore is derived from how the student does in responding to questions after reading passages with social science content.
4. The scores for the CAAP are not “criterion referenced” such that a description (in terms of skills or performance indicators) is given for a given score range. This makes it difficult for some to understand what a score means.
5. The meaning of a score relative to comparison groups or “norms” is dependent upon the makeup of the group from which the norm scores are derived. For instance, some institutions can make their scores look great by comparing their students exclusively to students from open-admission community colleges.
6. It is sometimes difficult to relate scores or even subscores to specific courses or even to specific programs (consider the “critical thinking” subscore, for instance). And it is most often that refinements are made (i.e., the “loop is closed”) at the course and program level.

I. Are faculty members who don’t even teach Gen Ed courses responsible for Gen Ed assessment?
a. “Yes” is the answer, and the reason is that we all form an institution and this institution is committed to instilling the skills, knowledge and attitudes described in the 7 GEP goals in all of its students as a result of their educational experience here. Even though general education courses are taken early in the student’s 4-5 years at SDSM&T, the learning we have identified as “core” is spread across all curricula in the majors.

This necessary connection between learning in the General Education program and learning in the major is built right in to the HLC’s requirements for institutional accreditation. We must show that our “core” skills are fostered across a student’s entire educational experience here.

It is often remarked, however, that a call for “all GEP faculty to walk out on the campus green” would not scare up much of a crowd since faculty often identify primarily with their disciplines.

There is another side to this question, namely, “Do non-engineering faculty share responsibility for satisfying ABET requirements?” The answer here is also “yes,” and for the following reasons:
b. As an institution that is “committed to preparing tomorrow’s engineers and scientists” (as stated in the public radio ad), our institutional curriculum is dominated by our 10 engineering programs. The success of the engineering programs is synonymous with institutional success to a very large degree.
c. ABET has very stringent accreditation criteria in terms of coursework required outside the discipline. ABET requires programs to work with / consult external interests and programs because doing so is key to achieving the benefits of the “two-loop” assessment and continuous improvement process. In short, the engineering programs need the goodwill and help of the non-engineering programs.
d. ABET outcomes a - k stress engineering students’ communication skills and their ability to utilize math and science.

We know our engineering students often have problems with seemingly simple mathematics exercises in many engineering courses. (This is supported by FE exam results.) This finding could result from weak curricular links between math and engineering, or a problem with the exam, or a matter of student pre-college preparation, or something we haven’t yet identified. The point is that the engineering faculty need the help and support of the math faculty in figuring out what might be done to improve student learning.

Similarly, our engineering faculty members generally observe weaknesses in students’ written communication skills. And, similarly, the engineering faculty need the help of the faculty who teach writing in sorting through and addressing the many possible causes of these weaknesses.
e. Lastly, criteria a - k address other skills, such as life-long learning, ethics, and an understanding of engineering in a global context. English, math, humanities and social science faculty have much that they can offer engineering students. And engineering faculty can help students in English, math, humanities and social science courses understand the social context for the skills and concepts they gain in the general education program.

J. What do we do about the learning goals that can't really be measured?
When you find out, write a book and make a million bucks. The answer, of course, is that we cannot prove the acquisition of certain outcomes in students. Assessment in these instances becomes the art of triangulation and multiple measures. Accrediting agencies are well aware of the difficulty of showing, for instance, that a student graduated with the skill of lifelong learning. For years, campuses and programs have adopted laudable (if somewhat abstract) learning outcome goals; now they are doing a great deal of creative thinking about what to observe, measure, or inquire about in their students to see if those goals are being met.

Bottom line is “Don’t jettison the lofty outcome goals; rather, develop several means of detecting the presence of a skill or attitude and establish a pattern of evidence from which you can reasonably infer how well you are meeting the goal.

K. What is the real likelihood that we will fail to be reaccredited just because of assessment?
The answer to this is a “moving target” since enforcement of assessment requirements is being heightened with each passing year. Assessment programs that were regarded as adequate a few years ago are now evoking follow-up visits and focused visits from the HLC. And in 2000 - 2001, ABET evaluation teams assigned the actions “interim review” and “ interim visit” to 27.6% of the 301 programs evaluated that year.

But the question is whether or not assessment deficiencies alone will cause an institution to suddenly lose accreditation, and the answer is generally “no.” Follow-up visits and interim reports, however, are costly and, in the end, far more time consuming and anxiety producing than having effective and efficient processes in place to begin with.

L. The NCA visit is in 2005 - 6 so why the fire drill now?
The current perceived “fire drill” is a rational response to the amount of work we need to do and the amount of time it takes for many assessment processes to unfold. As of September 2002, we have about 24 months before the ABET evaluation team arrives on campus to evaluate 10 SDSM&T programs. And the self-study process for our comprehensive reaccredidation visit by the HLC is something we need to start 1.5 to 2.0 years in advance (i.e., in fall 2003).

Also, it always takes longer than you would imagine to go from achieving consensus about what you want to assess, to actually designing and implementing an assessment, to obtaining and analyzing results, to acting on the analysis, to being able to detect differences as a result of the changes! Even one full “loop” in an assessment process can take many years to achieve from beginning to end. This fall we will begin to start with some assessment processes for our general education program. If we work hard and argue little, we will be able to achieve a full loop process by 2005 - 6.

M. Since when did it become my job to add assessment to my already heavy faculty workload?
“Since the 1980s” is the short answer to that question. And since then, the role educators must also play in investigating and improving student learning has done nothing but grow. Just as few of us were ever taught how to teach before suddenly becoming teachers, most faculty members learn about assessment on the fly and on the job. It’s been part of the changing professional landscape for educators, just like learning about technology. Every year for the last 5 or 6 years, the keynoters at the annual AAHE assessment conferences never failed to make quips about faculty waiting for assessment to “go away.” Now those jokes and quips are seldom heard. The accountability culture is here to stay in K - 12 education, the corporate environment and in higher ed.

Requirements

N. What did the HLC evaluators say about SDSM&T’s assessment plan and processes in their report on their March 11-13, 1996 comprehensive evaluation visit?
Well, the following is the entire section from the HLC evaluation team report (pages 40-42) from the report on assessment:
Assessment
The assessment plan developed by school personnel reflects the integration of technology and available information sources for the support of the improvement of teaching and learning activities. While departments, colleges, and other SDSM&T units have been involved in the planning phases, plan implementation is fragmented and lacks activity coordination, although many faculty members have been carrying out assessment activities in their classrooms. Other faculty members do not have a clear understanding of assessment and view it as an aspect of continuous quality improvement activities. As such, this initial step has not provided the basis for collective implementation at the department and college level, nor does it facilitate interdisciplinary activities within and between colleges. College and institutional (general education) assessment activities are not underway. And it appears that faculty have not taken “ownership” for assessment implementation activities.

The Individualized Educational Program (IEP), a valuable advising tool, incorporates student information generated from standardized tests (ACT, SAT), course work completed, and the student advisor’s comments. This is a main component of the assessment database. Other information sources come from previous surveys of alumni and employers. However, while valuable, these assessment data do not provide relevant information often needed for improvements in teaching and learning found in the classroom, interdisciplinary courses or general education curriculum. Qualitative approaches, such as the use of portfolios and focus groups, which provide information allowing discovery of student skills and insights into the instructional process, are being used by various faculty. The faculty development program, which involves these areas of the teaching and learning process, needs to involve the entire faculty. A broad constellation of assessment information enables faculty to improve student learning. Information should include student life on campus and learning experiences arising from non-classroom activities. With the heavy emphasis on technology at SDSM&T, the expansion of the employment of non-quantitative approaches need the commitment from faculty assessment committees and administration.

The recent requirement by the Board of Regents to administer a “rising Junior” standardized exam appears to be focusing assessment activities away from the faculty, creating the image of a one-time final test, and accentuating accountability as the primary purpose. SDSM&T should have other assessment activities that provide sources allowing for an enhanced/fuller interpretation of the student scores. The emerging focus on a final test, the fragmented implementation of the program, lack of faculty ownership in assessment, and the need to develop a constellation of data sources require the attention of faculty leadership and key staff. If improvements in teaching and learning are to occur, it is necessary to address these issues immediately.

So, what’s the bottom line here? Well, if we isolate the individual points made in the above, we have the following:
· Many faculty members have been carrying out assessment activities in their classrooms
· Student data collected under the Individualized Educational Program (IEP) could be a valuable assessment data.
· Our assessment plan implementation is “fragmented and lacks activity coordination.”
· Some faculty members do not clearly understand assessment.
· Collective implementation of assessment at the department and college level and the use of assessment to facilitate interdisciplinary activities within and between colleges are lacking.
· College and institutional (general education) assessment activities are not underway.
· The faculty have not taken “ownership” for assessment implementation activities.
· The Individualized Educational Program (IEP) is a valuable advising tool, and there is much information in the student database. However, “these assessment data do not provide relevant information often needed for improvements in teaching and learning found in the classroom, interdisciplinary courses or general education curriculum.”
· Some faculty are using qualitative assessment approaches, such as the use of portfolios and focus groups; the use of non-qualitative approaches needs the commitment of the campus assessment committee and the administration in order to be expanded.
· We need to add information about student life on campus and learning experiences arising from non-classroom activities to the assessment data distributed.
· The implementation by the BOR of the CAAP exam may shift responsibility for assessment away from faculty, and may suggest that assessment is all taken care of by the one test.
· We must supplement the CAAP with other assessments that enable “an enhanced/fuller interpretation of the student scores.”
· Lastly, the accrediting team says “If improvements in teaching and learning are to occur, it is necessary to address these issues immediately.”

O. What will the HLC evaluators “look for” when they come in 2006?
It is certain they will look to see how we fared on addressing the needs they pointed out in 1996. Notably, they will want to see a meaningful and fully implemented General Education assessment process. They will want to find faculty educated about and responsible for assessment and the improvement of student learning. And they will want to see much greater collective and cooperative use of both qualitative and non-qualitative assessment methods. For a full picture of the criteria under which we will have our comprehensive evaluation, see http://www.hpcnet.org/accreditationoverview on this site or go to the HLC website at http://www.ncahigherlearningcommission.org/

P. What does ABET require for assessment?
The in-depth answer to this is pretty in-depth. For a thorough answer, go to http://www.hpcnet.org/assessmentengineering&abet and http://www.hpcnet.org/assessmentengineeringoverview on this site or visit the ABET site at <http://www.abet.org/>
The short answer is that under ABET Criterion 3 there are 11 mandatory program outcomes that need to be achieved (and assessed) in each program. In addition, each ABET-accredited program must develop its own set of program objectives (i.e., big-picture goals) and specific program outcomes (i.e., what students know or can do by graduation). ABET also requires that external constituent input be integrated into the entire assessment process for the program.

Q. When will the ABET visit occur and what programs are affected?
The evaluation team will arrive on campus during fall semester 2004, and the following 9 programs will be evaluated:
·Chemical Engineering
·Civil Engineering
·Computer Engineering
·Electrical Engineering
·Environmental Engineering
·Industrial Engineering
·Geological Engineering
·Mechanical Engineering
·Metallurgical Engineering

R. What must individual programs do for assessment?
Again, --a potentially complicated answer. However, the essentials are pretty simple. A program must view itself as an integrated, coherent educational experience designed to produce specific learning outcomes in the students who experience it. The faculty must agree upon and define these outcomes and agree upon and define the performance indicators that will enable them to know if the outcomes have been achieved by the students in the program.

Then, once a functional consensus is reached by faculty about what they want students to learn and how they will recognize success in students, they must design and implement an assessment process for the program.

This process entails devising the means of measuring or detecting the performance indicators identified as evidence of learning. (These “means” are often called assessment instruments or measures.) Then the program faculty members conduct the assessment measures or instruments and analyze or otherwise consider the results of the assessments.

Lastly, the faculty members employ the results of the assessments to drive reflection upon and, perhaps, refinement of the design of the original educational experience. (This is the “closing the loop” part.) Other components of this process include assessing the effectiveness of the program itself and integrating into the assessment mix the input of key external constituencies. These “external constituencies,” by the way, can be both industry councils to service areas within the institution itself (e.g., English, math, chemistry, etc.)

S. What do the academic support units do for assessment?
As mentioned above, academic support units, such as student services, must also have assessment plans. They way they go about assessing effectiveness is slightly different than the way academic units do it. Support units also articulate outcomes, but not all are of the “skills, knowledge, attitude” nature seen in academic setting. For instance, support units might also set outcomes goals for number of students served, the speed with which certain services are rendered, or the percentage of freshman who participated in mentoring or advising initiatives.

T. What is required of me as a faculty member?
Learning about assessment, participating in the development and implementation of your program’s assessment plan and participating in general education assessment, and helping keep everyone’s mind focused on the point of assessment, which is the improvement of student learning. . . All these are part of a faculty member’s professional responsibilities.

U. Whom do I turn to with questions about what is required of my program or me?
Speak with your colleagues on the assessment committees or contact the Director of Academic Initiatives, Kate Alley. This web site also contains much information, as do the many WWW sites devoted to assessment concerns.

Campus assessment structures

V. What are the assessment committees on campus? The Campus-Wide Assessment Committee, the GEP (General Education Program) Assessment Committee, and the Engineering Assessment Committee.

W. Why do we have 3 committees, and what do they do?
We have 3 committees, because all the work that needs to be done cannot be accomplished by one committee. Also, separate groups who can focus on a given challenge can best do the work that needs to be done.

 For instance, we need to develop and implement a GEP assessment program, hence the GEP committee. The 10 ABET-accredited engineering programs have engineering-specific assessment demands placed on them by ABET, including mandated program outcomes common to all. They can make most progress if they collaborate amongst themselves in a focused manner, hence the Engineering Assessment Committee. And then there is all the big-picture administrative, management, planning, and policy tasks that require a campus-wide perspective. The Campus-Wide Committee does that. For more information on each committee and the initiatives they have underway, see the individual committee areas on this site under http://www.hpcnet.org/assessmentcommittees

X. May I attend the meetings?, see the minutes?, or provide input without being an actual member?
Yes. And you are welcomed to do so.

Y. What authority do the committees have?
When it comes to assessment “you” are “they” in terms of wielding authority. Assessment is a collective responsibility. If an institution has a top-down mandated system with authority figures policing the process, it doesn’t have an assessment process; it has a numbers game. So the answer is probably that the committees have as much authority as they have credibility and good will invested in their efforts by all faculty members.

Z. What roles & responsibilities do the Director of Enrollment Services, Chuck Colombe, and the Director of Academic Initiatives, Kate Alley, have?
Well the roles are changing and will continue to change as our assessment processes develop. In general and right now, Chuck is responsible for all things relating to the administration of the CAAP exam, the Freshman Survey, and the Student Satisfaction Inventory. Barb Dolan is also playing an increasingly important role in tracking student success through her work with mentoring and retention under Title III. Kate directs and coordinates campus assessment activities as half of her duties as Director of Academic Initiatives. She works with programs and individual faculty members and manages the work of the assessment committees.