NWCCU MSU Billings Self-Study Report
October 2008

Standard 2.B - Educational Program Planning and Assessment

Helping to strengthen the community, state and region by strengthening higher education are critical to the mission of the University. The perspectives and experiences of business and civic leaders in the community are vital to improving the academic experience for our students and facilitating their successful entry into the workforce.

MSU Billings is fortunate to have strong college and program Advisory Boards composed of dedicated leaders in the community whose expertise is critical to the success of the programs. For example, at the West Campus, to achieve the University’s vision of responsiveness, each program has an advisory board. These boards are made up of managers, business owners, technicians, supervisors of technicians, technical trainers, equipment vendors, and others concerned with the success of the respective programs they are advising. These Boards help the college to respond to the changing needs of the workforce, maintain industry standards, and provide students with opportunities for internships in business and industry. They help to ensure that the curriculum is meeting industry standards. In addition, a 24-member National Program Advisory Board has been integral to the long-range development of the COT. The College of Business Advisory Board is designed to foster a working relationship between the College and the community. The needs of business professionals in the community are a primary focus for the College of Business. Since the individuals operating in local, national, and global markets will know best what business needs the Business Advisory Board is a key asset in helping students succeed in business. Likewise, the Community Advisory Boards in other colleges connect programs with the community, provide feedback on the educational programs, help identify new and unexplored opportunities, and support programs through human and financial resources.

In response to the 1998 NWCCU accreditation review, MSU Billings began the process of reviewing and revising University assessment processes. The first piece tackled was General Education. The NWCCU Accreditation Report indicated that General Education was not being assessed as a program to determine student achievement of desired performance outcomes. In response, faculty began an arduous review of General Education to determine program integrity before focusing on assessment of student outcomes.

This 10-year review process has resulted in the Academic Foundations Program, containing five conceptual areas, designated student outcomes for each and means to assess those outcomes. Academic Foundations is discussed in more detail in Standard 2.C Undergraduate Program.

Using the in-depth review of General Education and its evolution to Academic Foundations as a starting point, the focused existing assessment activities in the Colleges of Allied Health Professions, Arts and Sciences, Business, Education, and Technology, the University expanded assessment throughout the institution. Although MSU Billings has continually operated from a Strategic Plan, the University implemented Continuous Quality Improvement across all divisions — Academic, Administrative and Student Affairs — fall 2005.

Before the implementation of CQI, there were five-year strategic plans with strategic initiatives all based on the institutional mission, vision and core values. University divisions and subdivisions aligned their strategic initiatives to those of the University and reviewed progress on a two- to five-year basis.

With the implementation of CQI, the University reviewed its Core Purpose, Mission, Vision, Core Values and Strategic Initiatives. Each unit in the University aligned its Unit Master Plan with that of the University as a whole. Academic Year 2005-2006 culminated with unit submission of Annual Reports. In Academic Affairs, the unit reports contain three major sections — (1) department alignment with the University Strategic Initiatives and progress on departmental goals; (2) program, faculty and student data; and (3) program alignment with state/national specialty area standards as applicable.

In addition, the University expanded previous institutional efforts to assess constituent satisfaction and published a regularly scheduled rotation of surveys. This effort included the following:

• Student Satisfaction Surveys (Noel-Levitz)
• Alumni Survey (PEG)
• Student and Faculty Engagement Surveys (NSSE, FSSE, CCSSE)
• Employee Morale Surveys (adapted with permission from the University of Wisconsin-Stout) and (disseminated through Survey Monkey)
• Career Services Graduate Survey (longstanding annual MSU Billings survey)

These surveys are referenced throughout the Institutional Report.

The Academic Affairs Annual Reports focus on educational program planning, assessment of program outcomes through student performance and program efficiency/effectiveness. Analyses in the Annual Reports provide direction to faculty and administration for necessary program changes, personnel decisions and student services.

Evidence:

• 1.1 Mission Document
• 1.2 Annual Reports
• 1.3 through 1.7 Surveys and Assessments

Standard 2.B.2 - Educational Outcomes Objectives

Each program publishes objectives/student learning outcomes in the General Bulletin, the COT Catalog or the Graduate Catalog, as appropriate. Program outcomes are reviewed through Annual Reports. Changes in programs result from review of objectives as they relate to student achievement, program/faculty/student data and state/national specialty area standards — the three major sections of the Academic Annual Reports. Necessary programmatic changes go through the University shared governance process — department, college, Undergraduate Curriculum Committee or Graduate Committee, Academic Senate, Provost, Chancellor, Board of Regents. The General Bulletin and Graduate Catalog are in effect for two years with revisions occurring during the current two-year cycle to be published in the next. The COT Catalog is published annually, making program revision more immediately available.

Evidence:

• 2.11 General Bulletin/COT Catalog
• 2.5 Graduate Catalog
• 1.2 Annual Reports

Standard 2.B.3 - Educational Outcomes Planning

The University’s Academic Master Plan (AMP) is the overall guide for educational program outcomes and assessment. In alignment with the AMP, College departments have departmental plans. Taken collectively, the departmental plans become the College plan. The General Bulletin/COT Catalog and Graduate Catalog list objectives for each program with suggested plans-of-study for degree completion. The Office of Information Technology (IT) provides program and faculty data. Program data include numbers of majors, headcounts and student FTE.

Faculty data include faculty loads by semester and student credit hours generated. The University participates in the Delaware study, a national study providing comparative data on faculty efficiency. Programs with state/national standards are reviewed for relevancy in synchronization with standards revisions.

The current University Initiatives have evolved from previous documents. The most recent were developed by a wide representation of University constituencies in summer 2004. In fall 2005, as part of the CQI initiative, the University began the review of its Core Purpose, Mission, Vision, Core Values and Strategic Initiatives. Over the course of two years, the University had multiple venues for participating in the revisions — examples include but are not limited to:

• E-mail
• Back-to-School Conferences
• Summer pizza lunches
• All-University/all-faculty meetings
• Committee meetings

The Academic Master Plan and individual academic departmental plans align with the revised Mission Document. Annual Reports for each program assess program alignment with the Academic Master Plan, program efficiency through faculty data, effectiveness through student data, currency through standards alignment and continued viability through BOR 7-year cyclic reviews. Student outcomes are being assessed through the eCollege assessment system or an alternative approach specific to a discipline. The CQI process is an ongoing evaluation of academic programs in the context of the University.

Evidence:

• 1.2 Annual Reports;
• 2.11 General Bulletin;
• COT Catalog;
• 2.5 Graduate Catalog

College of Allied Health Professions

Mission

The College of Allied Health Professions is dedicated to:

• Preparing competent, caring allied health professionals for Montana’s health care industry;
• Conducting socially significant research to improve the health and well being of people;
• Providing community services aimed at improving the health of Montanans.

The college is five years old and it inherited programs from two other colleges when it was conceived. The College of Education and the College of Professional Studies and Lifelong Learning transferred programs to the college. This, of course, presented its own challenges for creating a common mission and perspective for continued growth. Assessment in light of existing programs is somewhat idiosyncratic to the nature of the programs, but much progress has been made for addressing common approaches for data collection and analysis. As the college enlarges and adds other programs, the existing programs can be used to set a clear model for doing assessment and doing continuous quality review of students, service to the community and scholarship productivity.

Outcome Performance Expectations for Undergraduate Candidates:

The college has three departments, each encompassing different professional program areas. The Health Administration Department offers the bachelor’s degree in health administration. The Rehabilitation and Human Services Department offers B.S. degrees in human services, rehabilitation and psychiatric rehabilitation services and an associate of arts degree in rehabilitation and related services. The third department, the Health and Human Performance Department, offers bachelor’s degrees in Health Promotion, Human Performance, Outdoor Adventure Leadership, along with teaching options in Health and Human Performance. All programs now have competency definition to their curriculum and use these definitions in measuring learning outcomes. For instance, the Outdoor Adventure Leadership Degree, while new and only now admitting students, uses outcome-based assessments for this program that have been aligned with the highly recognized organizations of Project Adventure (PA) and the Association of Experiential Education (AEE).

The degree requirements in the Health Administration Department have, within the last year, undergone curriculum revision. Competency outcomes have been newly defined for the program. These competencies are to be used to measure program outcomes of students.

Determination of Undergraduate Candidate Achievement

Each undergraduate program has different measures, but generally all include:

• GPA in Core Content and Core Professional Courses.
• GPA in Required Major Courses.
• Performance in field projects or internship rotations.
• Preceptor evaluations.

Evidence for all these program outcomes is available in the annual reports for the college.

Outcome Performance Expectations for Graduate Candidates

The college offers a master’s degree in health administration, a master’s degree in rehabilitation and mental health counseling, a master’s degree in athletic training, and a master’s degree in sport, recreation and fitness management.

Programs such as the Master’s programs in Athletic Training and Rehabilitation and Mental Health Counseling meet their specific program accreditation standards and must meet outcomes as specified in their review guidelines. Evidence of success is provided in job placement results and licensure success rates and can be observed in each of their program accreditation self study documents.

The degree requirements in the Health Administration Department have, within the last two years, undergone curriculum revision. Competency outcomes have been defined for the revised MHA curriculum. These new competencies will be used to measure program outcomes of students. Data are still not available to measure the successes and challenges in this program, since it is generally a two year program. In addition, there will be an alumni survey in the fall of 2008 to obtain opinion of the Masters Degree curriculum.

In the Rehabilitation and Mental Health Program, the next step is to connect each course with the specific competencies identified in program accreditation standards for each content area, and add this information to each syllabus. This task is in on our plan of work for 2008-2009. The Program is currently working to adopt the CRC licensing exam as the comprehensive exam that all Masters Degree students in Rehabilitation and Mental Health would take at the end of their course work. The exam is an excellent outcome measure in that the test content was developed by leaders in the field (external validity), has been subjected to tests of internal validity, and reliability, has been normed on performance of all students nationally, scores have been broken down into specific competencies that are linked to course work, and data is provided both annually and summarized over five years.

Determination of Graduate Candidate Achievement

Each graduate program has different measures, but generally all include some form of:

• Competency achievement measures, such as performance exams, course GPA and written online threaded discussions evaluations by instructors.
• Preceptor, student, faculty performance evaluations.
• End project or capstone project.
• In the last year, the inaugural year for graduates in Athletic Training, all AT graduates have passed the national certification exam.
• Master’s degree recipients in the Rehabilitation and Mental Health Counseling program have achieved an 88% success rate in the national licensing exam in 2004, the most recent year of reported data.

Evidence for the achievement can be found in the annual reports and specific program accreditation self study documents.

College of Allied Health Professions Response to Data Analyses

As part of the Athletic Training and the Rehabilitation and Mental Health Counseling accreditation review, changes have been made to reflect reviewer comments and areas of weakness in the curriculum for those not passing the national exam. Satisfactory provisions have been made for both these programs relative to outcomes assessment. Since the Athletic Training Program is a new program and has just undergone rigorous review for accreditation, the curriculum has been reviewed with assessment criteria for three successive years while in preparation for its self study report. This, program in particular, has seen a number of improvements in the offerings that are clearly a result of an outcomes assessment process.

In the Sports management program, changes are being made to offer more courses online since many students are working adults. Also, a consultant in Sport Management Education will be brought in to help assess ways to improve the program and make it better in terms of assessment. In the case of Health Administration, focus group meetings with alumni of the former curriculum and other field faculty of the MHA program were interviewed, and the standards of the Commission on the Accreditation of Health Management Education were reviewed, to help in redesign of the curriculum. The College of Allied Health Professions is a new college and proposals are continuing to be advanced as new programs that will be offered to meet the demands for workforce professionals in the region.

Evidence:

• 1.2 Annual Reports,
• 2.11 General Bulletin,
• 2.5 Graduate Catalog 2.2,
• 2.10 Program Accreditation Reports

College of Arts and Sciences

The CAS provides Academic Foundations for all students through the Academic Foundations Program and content majors for liberal arts students, students entering professional fields and students preparing for graduate school. Also offered are three graduate programs. Although different methods of outcomes assessment are used within programs, all assessment methods can be linked to the electronic Assessment System used for Academic Foundations in spring 2008. Academic Foundations assessment data were entered for fall 2007 and spring 2008 semesters. As of summer 2008, other departments, are engaged with IT to create similar databases for capturing programmatic assessment information. Programs maintain the option of an alternative assessment system if the system is reliable and valid; for example, the Math Department has a UNIX formatted system that tracks student progress.

Assessment is an ongoing attempt toward Continuous Quality Improvement in the College. CQI has allowed for informed academic decision-making, program modifications and improved student learning. Each department in the CAS is involved in outcomes assessment activities using a diversity of discipline-specific assessment tools and the following baseline questions that serve as starting points:

• What is the mission of the department/program?
• What should the students who complete the major/program know and be able to do?
• How does the department determine whether students have achieved the specified learning goals?
• What changes to the major or department have been made, or are being made, as a result of the findings?

Outcomes are measured qualitatively, through student surveys and focus groups, and quantitatively, through scores on pre- and post-tests for students entering specific disciplines. In many cases, pre-testing occurs in the Advising Center or under the aegis of Student Affairs. Often those tests are institutionally modified to yield demographic and student support needs data.

In some cases (including, for example, foreign languages and chemistry) nationally-normed pre- and post-tests are used. In others, performance measures such as final performances, capstone courses, theses, or portfolios are employed.

In addition to the array of discipline/program outcomes measured quantitatively through scores, CAS has assessed the General Education program extensively. Following the last accreditation visit by NWCCU, a General Education Task Force was established composed of faculty representation from each of the five Colleges and led by the CAS Dean. The deliberations of the General Education Assessment Committee resulted in the change of the General Education Program to the Academic Foundations Program currently in use. A summary of activities related to General Education are shown in the chart on the previous page.

Chart 2.3 - Summary of Assessment Activities that Led to Current Academic Foundations Program

In spring and fall 2007, MSU Billings piloted the use of the eCollege course management system in gathering assessment data for Academic Foundations courses. However, as it became apparent that this system would not be capable of generating aggregate reports necessary for guiding decision-making and other technical problems, this project was replaced with an in-house system that is currently in use. The aggregate report on the data collected and analyzed is contained in the exhibit 2.37 and described later in this section.

A faculty member was appointed as the Director of General Education Assessment. The director completed a qualitative analysis of student perceptions of the General Education core. This assessment provided the institution with both a deeper evaluation of the overall academic program and with a more solid basis from which to conduct program modification and integration of general education and discipline-specific instruction. The Assessment Day, held on April 6, 2000, was a marked success.

Subsequent student opinion surveys and input from open meetings in 2000-2001 and follow-up surveys suggested that the purpose of general education was not well understood and the objectives were not readily measured.

As part of the Montana University System, MSU Billings abides by the transfer policies mandated by the Board of Regents. This includes transfer of courses in the general education program taken by students at other campuses of the Montana University System (MUS). A defined general education core with a minimum of 30 credits is mandated. This mandate, along with the recent changes in the transfer policy, impacts the assessment of the value-added component of the MSU Billings general education program.

The General Education Assessment Committee renamed as the Academic Foundations Committee is now a standing committee of the Academic Senate. The Committee is charged with identifying specific problems associated with the quality, delivery and assessment of Academic Foundations.

In collaboration with faculty and administration, the Academic Foundations Committee defined a conceptual framework of measurable learning outcomes, evaluated each course before its inclusion in the program and implemented an assessment model. As a result, a revised General Education program entitled Academic Foundations Program emerged in spring 2007. The Academic Foundations program consists of competencies in three major areas (Skills Development and Application, Cultural Development, and Intellectual Growth and Development), each with categories and subcategories and an embedded assessment model.

The Academic Foundations (AF) program facilitates systemic change addressing concerns of NWCCU (regarding assessment), Montana University System mandates (compliance of a mandated 30- credit transferable general education core), student perceptions (General Education courses academically unjustifiable space fillers), faculty efforts (to make the program rigorous) and institutional commitment to Continuous Quality Improvement. AF assessment allows a consistent examination and measurement of expected student learning outcomes within courses and across the program. It offers a conceptual framework of measurable learning outcomes and competencies in Skills Development and Application, Cultural Development, and Intellectual Growth and Development, each with categories and subcategories.

The Committee worked with departments to ensure proposed courses met expected outcomes of Academic Foundations and selected participatory courses. All courses in the existing General Education Program were excised and the number of course categories was reduced from nine to five. Departments developed courses (new, integrated or revised existing courses) that underwent a rigorous review and approval process by the Committee. The Committee’s work culminated in a matrix of assessable student learning goals centered in three broad areas. All Academic Foundations courses offered in spring 2007 were assessed using eCollege assessment tools. With state mandated contractual limits, the University electronic course management and assessment systems were up for vendor bids. New systems will be implemented spring 2009. Although the vehicle for assessment data warehousing is in transition, the processes across University remain.

The previous General Education program was assessed each year through surveys, and the results were used in the improvement of teaching and learning in the revised program. The results of those surveys are discussed in 2C.

The Academic Foundations program is published in the 2007-2009 General Bulletin. Discipline-specific assessments continue to be diverse in nature. Fundamental student outcomes, including retention and graduation rates and employment-placement upon graduation, continue to be monitored by Academic Affairs in collaboration with Student Affairs, whose staff collect and report persistence and employment data. Changes in the curriculum have frequently arisen from more qualitative measures, however. An example is in the biological and physical sciences, where review of peer science programs and trends led to curricular modifications including the addition of a biology capstone course and increased emphasis on undergraduate research. During the last five years, more than a hundred students participated in research projects in disciplines ranging from history to sciences and presented/published the research results at professional discipline-specific conferences. The research experience has been a gateway to graduate school admission for many of these students. For example, in 2007, eight science students (biology and chemistry majors) were accepted in dental, graduate, Physician Assistant and Veterinary Programs at various prestigious universities (University of Wisconsin, Minnesota, Cleveland, Oregon and Colorado), five students co-authored articles published in refereed journals; seven graduates in the history program were accepted into graduate school; an English major received a full graduate scholarship from the Jack Kent Cook Foundation to pursue a MA program at University of Aberdeen, Scotland; and nine psychology graduates were accepted by graduate schools. MSU Billings has a track record of students successfully completing post-baccalaureate programs. In 2007, two students completed the MD degree, two students graduated with a master’s degree in history and one former history graduate finished law school, one former art graduate completed a PhD program , two graduated with MFA degrees and two former psychology graduates completed PhD programs. As in the previous years, the trend continues in 2008, as attested by the acceptance of MSU Billings students by professional and graduate programs. The departments report the data annually and the college compiles the information for dissemination at the Emeritus Faculty Luncheon. Other examples of assessment are outlined below:

Art

The Department of Art is accredited by NASAD (the last visit in spring 2002 was successful and the next is expected in spring 2012). The Art Program enjoys strong community support as evidenced by participation in exhibitions, scholarship support and other activities sponsored by the department. Competencies are primarily measured through the student’s senior show, portfolio, and verbal defense. Faculty regularly review and act on student evaluations of the program provided to the Department Chair. Changes from this type of feedback include adding a required computer graphics class for degrees, revamping the freshman and sophomore core, addition of an Internet art history class, studying the relation of art history classes to studio classes, examining the art course offerings in the General Education categories, and initiating changes and converting a University Lecturer line in Ceramics to a tenure-track line (2007). The mission of the Department of Art at MSU Billings is to educate students in the understanding, production, and analysis of visual art and culture. This educational process includes development of technical, conceptual and art historical awareness; development or furtherance of visual and tactile insight; development of appropriate verbal and written skills and fostering professional attitudes and goals. The curriculum is designed to prepare students to face the challenges and diverse career opportunities that exist within the discipline and to give students the knowledge, practical skills, and maturity of critical thinking necessary for further study.

To assess program effectiveness, the department uses required senior reviews and senior exhibitions in the Northcutt Steele Gallery, the Yellowstone Art Museum and other museums and galleries in the area. In addition, course work is required in various studio media, art history, theory, and criticism to assure learning outcomes. Advising is mandatory. Other activities include participation in the Art Students’ League and Potters’ Guild, the visiting artist programs and career advisement. Assessment of facilities, equipment and personnel is an ongoing process and has led to considerable improvements in art classrooms when the Liberal Arts Building was renovated in 2002. The remodel greatly improved exhibition spaces, ventilation and lighting in addition to compliance with ADA codes. In the Art Annex, which houses the ceramics and sculpture studios, new kilns have been purchased/built/installed and measures taken to create a safer learning environment for the students.

Biological and Physical Sciences

The department regularly collaborates with the local hospitals, analytical labs and other agencies to deliver the programs. Students in the Hematology, Immunology and Instrumentation labs are exposed to the state-of-the art equipment available at the local hospitals and the analytical labs. Internships provide additional opportunities. The department offers three majors (each with a teaching certification option) and five minors (each with a teaching certification option) in the sciences. The majors are in biology, chemistry and general science. Minors are in biology, chemistry, earth sciences, geography and physics. The majors are comprehensive. For example, the biology major includes a concentration in biology coupled with a balanced study in chemistry and supporting coursework in physics and mathematics. The chemistry major includes a concentration in chemistry coupled with a balanced study of physics and supporting coursework in biology and mathematics. All majors adhere to a broad scope of the discipline offered through a core of courses taken sequentially. Each previous course in the core serves as a prerequisite for the next. Each lecture course in the core has a co-requisite laboratory component. Therefore, basic philosophy, theory, and concepts are re-enforced with the skills in the discipline. In addition to the above, undergraduate research, internship and capstone seminars for the completion of each major assure both comprehensive content and student competence.

Undergraduate Research

Students in the sciences are required to undertake an individual or group research project during their third or fourth year at MSU Billings to provide research experience and training as part of the structured curriculum. In addition to the development of critical observation and thinking skills, research is an important vehicle to develop scientific literacy and communication skills in the sciences. To achieve these goals, students are encouraged to share the results of their research efforts with the scientific community through publication of original work in journals or presentations at regional or national meetings. This experience is invaluable to any student applying to graduate school, seeking employment, or wishing to teach.

The department uses several methods to determine the competency of its majors:

• Students obtaining a major or minor in science are required to obtain a C or better grade in every science course in the program.
• National examinations are given in courses such as organic chemistry to assess competency. The results are used for revisions in the curriculum and its delivery to improve student learning. MSU Billings students’ class average has been at the national average for the last several years with the highest scores ranging between 97-88%. (see exhibit 2.37)
• A one-credit Capstone Seminar in biology (Biol 498) is a required exit course for the graduating seniors. The intent of this course is to assess the integration and synthesis of knowledge and experiences developed through the various courses in the biology program. It also provides students a forum to present results of independent research projects and scientific topics. In addition, opportunities are provided to learn about current research in various scientific fields by attendance at and participation in seminars presented by science faculty and other guest speakers.
• A one-credit internship is required in both teaching majors. Competency of content knowledge is continually assessed when students are working with a faculty supervisor as laboratory teaching assistants in biology, general chemistry, organic chemistry, genetics, earth science, physics, integrated sciences and physical science courses.
• Throughout the curriculum, formative and summative assessment measures are employed. These vary from course to course but encompass a wide gamut of strategies including informal assessment during class, oral examination and written examination using many varied approaches. This is coupled with extensive feedback.

The science faculty regularly review other science programs and national trends (NCLB legislation), and utilize their findings to revise the curriculum requirements. Lack of equipment and available research space were addressed through INBRE and Department of Energy grants and developing partnerships with the local hospitals and analytical labs. Departmental assessment was used to strengthen unit cohesiveness and to build interdisciplinary programs. Assessment findings strengthened the connections between the traditional disciplines of chemistry and biology. Assessment outcomes provided a common assessment across two majors and five minors. Resulting program changes are reflected in the new program tracks (Organism Biology and Ecology and Molecular Life Sciences), new options (Environmental Sciences, Medical Lab Science and Teaching Certification) and an interdisciplinary major in science with an option in teaching.

Music

The Department of Music is accredited by NASM (the last visit in fall 1999 was successful and the next is expected in fall 2009) and offers a Bachelor of Arts degree in music with a teaching option. The program has strong campus and community support as evidenced by the generous donations to the program and attendance at events sponsored by the department. MSU Billings faculty and people in the community give annually for scholarships and maintenance of equipment. The program has several endowed scholarship funds, and grants from the Bair Foundation that have helped purchase a recording set-up for the recital hall. Most recently (2007-2008), the department received donation of $200,000 worth of pianos and $150,000 worth of CD’s and music.

The goal of this program is to offer:

• Comprehensive training in music at the undergraduate level including performance, theory, history and pedagogy
• The opportunity to acquire the knowledge and training necessary to successfully pursue careers in music as teachers and performers
• A liberal arts background in preparation for graduate studies in music
• A broad general education to develop well-rounded individuals with aesthetic values.

Students graduating from the program earn licensure to teach vocal and instrumental music kindergarten through 12th grade.

The visiting NASM team acknowledged the following strengths: stable, competent, amiable, dedicated and collaborative faculty; industrious and capable students; attractive and well maintained facilities; close professional faculty/student relations–mutually respectful; supportive upper administration; a recognition that music is a core discipline within the life of the institution; excellent retention and graduation rates and excellent administration and management at the department level. The visiting team also identified three areas of improvement that were immediately addressed.

A tenure-track faculty member with joint appointment in the College of Education was hired to address and strengthen the teacher education component as identified by the visiting team. This faculty member teaches music education courses, advise music education majors and co-ordinates curriculum with the College of Education. The department has maintained/enhanced these strengths during the last ten years as evidenced by the constant recognition of MSU Billings music students in national and international arenas and high demand across the state for music education majors. Other activities include the addition of a performance option following student and alumni surveys, adoption of a more defined set of procedures and assessment activities outlined below:

End–of–Semester Performance Juries. Students are required to present a juried solo performance for the music faculty at the end of each semester of study for which each faculty member submits a written critique. These critiques are subsequently reviewed by the students and their applied teacher.

• Pre-Recital Jury. One month prior to each student’s required junior and/ or senior recital, a 20-minute performance is presented to the music faculty. Based on this performance, the faculty is charged with the decision to either accept or reject the student’s application to perform the recital.
• Upper-Divisional Recital. Students must present a 20-minute performance for the music faculty to qualify for upper-division music courses. Students failing this jury may reapply in the next semester.
• Qualifying-Performance Option Jury. Students enrolled in the Bachelor of Arts in Music-Performance Option must present a 30-minute recital for the music faculty. This performance will determine their admission to the program.
• Junior Recital. A full recital (50-60 minutes) is required of students seeking the Performance Option. This performance is adjudicated by the music faculty.
• Senior Recital. A full recital (50-60 minutes) is required of all music majors prior to graduation. This performance is adjudicated by the music faculty.

English and Philosophy

The preparation program in English ensures content area competency through three specific courses: English 390-Peer Tutoring, English 394-Internship and English 425-Senior Portfolio.

In English 390-Peer Tutoring, students demonstrate their competency through performance under the direct supervision of English faculty by working as tutors in the Writing Lab, housed in the Academic Support Center. They work in individual and small-group sessions to help students with their writing. In English 394-Internship, students work with a full-time faculty member in a section of English 150-College Composition, to design, deliver, and assess instruction. They have a significant responsibility to the students in the class. These two experiential courses ensure that English majors have mastery of the essential content and have the skills to impart that knowledge successfully to students.

English 425-Senior Portfolio brings together all of the content of the major and provides students a venue to demonstrate that they can write and speak effectively and authoritatively about their profession. Students develop a portfolio of writings that reflects their mastery of the content of the program, and they make a public oral presentation of their best work. The outcomes of this department demonstrate the University commitment to competence in written and oral communication.

Environmental Studies

Banking on the expertise of a Community Advisory Board and an internal Faculty Advisory Board, the Environmental Studies Program at MSU Billings is dedicated to providing well-rounded understandings of our natural systems and the role of humans in those systems. This interdisciplinary major is among the most innovative programs at MSU Billings. It requires students to take courses in a range of subjects, including ecology, environmental ethics, environmental history, geography, and other environmentally related fields. The program is dedicated to community collaboration and experiential learning opportunities for our students. Students learn to investigate questions concerning environmental issues from three approaches:

• Scientific perspectives, as understood through the biological and physical sciences
• Human perspectives, as understood through social sciences and humanities studies
• Policy perspectives, as understood through courses designed around topics such as “Environmental Impact Analyses,” “Living with Predators,” and “Decision and Policy Analysis in Natural Resources.”

As a required part of their curriculum, students must complete a hands-on internship in an area that relates to their field of interest. During their internships, students may work in local, state, or federal agency intern programs or in private firms in the field or in environmental labs. Graduates are prepared to enter the work force in the growing field of environmental occupations in the government and private sectors, or to continue their studies at the graduate level. Entry and Exit assessment is through student performance in courses, internship and the capstone course.

Native American Studies, Sociology & Political Sciences

The disciplines of Sociology, Political Science, Criminal Justice and Native American Studies are housed in one department in addition to the Masters program in Public Administration that is jointly offered with MSU Bozeman. Majors are offered in three of the four disciplines (no major in Native American Studies). Both minor and teaching options are offered in all four; the minor allowed students to specialize in the area of pre-law and public administration, while the teaching option led to a Montana license endorsement. Political science is an integral part of the Broadfie1d Social Science major. The departmental faculty have addressed federal requirements under No Child Left Behind (NCLB) legislation to assure that students pursuing a teaching option are Highly Qualified for their future classrooms. In fall 2007, a teaching certification option was created in the sociology program, a major in political science and a teaching certification option within that major was approved in spring 2008. To meet the needs of the community, the department now offers a new major–criminal justice. This major has a community advisory board that includes a broad representation of law enforcement personnel from the city and state that work collaboratively with faculty and administration to provide guidance to the program.

A program review in Sociology conducted by an outside reviewer led to complete revision of the curriculum. As a result of this assessment, requirements for individual courses were changed and a capstone course added. The discipline of Sociology plays a key role in serving the Academic Foundations requirements of the University and matriculating majors and minors with both Bachelor of Arts and Bachelor of Science degrees. Academic Foundations students learn basic material about how individuals in society interact. Majors and minors build upon this foundation by taking specialized courses in the curriculum. It is expected that the major and minor students will acquire this specialized knowledge while sharpening their abstract thinking, reading, and writing competencies.

The course objectives are clearly spelled out in the syllabi distributed for each course. All upper-division Sociology courses now contain writing components in the form of essay examinations and term paper requirements. Within the next year the course objective and methods for addressing learning objectives and outcome based assessment will be more precisely addressed in redone syllabi and catalogue course descriptions.

Outcome assessment occurs through exams, feedback from student assessments and, very importantly, through the capstone seminar required of all Sociology majors. This course exposes students to professional practices from the major areas of the discipline they have studied and requires them to prepare and submit a written portfolio of the papers they have written for Sociology courses for review.

Unlike many other disciplines, students who pursue a Sociology major have the opportunity to directly enter the work force and employ their knowledge. A large number of the discipline’s graduates take initial jobs working for social service, non-profit agencies, and data analysis. Students have the opportunity to receive an educational background directly linked to a specific career. Sociology majors are provided access to two publications that discuss the multitude of specific career opportunities available to Sociology majors. All Sociology majors who purse majors are directly exposed to workplace settings appropriate to their degree through fulfilling their workplace internship requirement. And the capstone seminar, also required of majors, exposes Sociology students to specific workplace opportunities relevant to their degree. Sociology majors are also encouraged to consider pursuing graduate school to further their development in the discipline.

Based on the feed back from student performance, employers and community organizations, faculty have collectively redesigned degree requirements. These changes include an increase in the credit hours from 34 to 37, the addition of a second required theory course, the elevation of the required methods class from lower division (SOCL 205) to upper division (SOCL 305), an increase in required credit hours (from 14 to 19) and a reduction in elective credits (from 20 to 18). Other changes include increasing the capstone course from 1 credit to 3 credits, and revising the restricted elective categories. The current degree plan contains four categories containing an array of courses having uncertain commonality and large number of non-sociology courses. The new degree plan has a “core” category of lower division course that will provide a base-line of sociological knowledge. Five new categories have been developed containing thematically related courses and from which most non-sociology courses have been eliminated. Sociology faculty feel strongly that these changes both improve the quality of the program and make the degree requirements clearer for students. These changes have been submitted for approval and are intended to be accepted in 2008-2009.

Masters in Public Administration

Competencies are determined through a required Comprehensiv examination and an Applied Research Project.

Comprehensive Examination: In the final semester, MPA students are required to take a written comprehensive examination covering the common course material in their respective programs. The Political Science Faculty read the examinations and determines if the student has mastered the concepts of the curriculum. If the faculty agrees that the student has mastered the material, a letter is sent to the student informing them they have passed their comprehensive examination. If a student should fail the exam, they are permitted to retake the exam the following semester. There are no restrictions on the number of times a student may retake the< examination. The completed exam, with comments from the faculty members, is maintained in the student file as an integral part of the student’s portfolio.

Applied Research Project: As a final step in the process of qualifying for an MPA degree in the joint MSU-MSU Billings MPA program, students are expected to complete an Applied Research Project. This project substitutes for the traditional Mater’s Thesis in MA/MS programs. The requirement examines the student’s ability to apply the material learned in the academic classes to problems facing contemporary governments. The project parallels the responsibility of senior government employees to identify problem areas in their departments or agencies, to define the parameters of the problem, to collect data pertinent to the problem, to analyze that data using standardized methodologies, to develop alternative solutions to problems, and to recommend appropriate courses of action for problem solution. Through this process, students are also assessed on their ability to professionally communicate in writing. Assessment activities have resulted in program enhancements to include an oral presentation of the research project where students will be assessed in their ability to conduct a professional briefing. This program enhancement will be introduced during the 2008-2009 school year.

Students are introduced to the Applied Research Project early in their second semester of the MPA program. During this semester the student is expected to enroll in POLS 551 Research Methods. In this class students are instructed in the methodologies of political research and are required to develop a research proposal for their Applied Research Project as a condition of passing the class. These proposals are evaluated by the course instructor and returned to the student for further evaluation and revisions.

During the final semester of the program, students are expected to enroll in POLS 574 Applied Research Project, where they will begin to finalize their research proposal and conduct the research and data collection. During this process, students are expected to turn in various portions of their individual projects to their faculty project supervisor for review and revision. The faculty supervisor makes comments and suggestions, and returns the draft to the student for revision. Copies of the faculty supervisor’s comments are retained by the faculty supervisor, thus creating a portfolio of student’s work on tie individual project. Beginning in the 2008-2009 school year, copies of the student’s initial proposal in the POLS 551 class will also be maintained as a part of this portfolio.

Once the student and faculty project supervisor agree that the Applied Research Project is completed, copies are forwarded to the faculty in the MPA Program for review and for recommended revisions. Once revisions are finished, the MPA Program faculty and the Department Chair formally evaluate the student’s Applied Research Project. If the faculty accept the student’s work as a professional project paper, the faculty will sign the Applied Research Project. Beginning in the 2008-2009 school year, candidates for the MPA degree will have to conduct a public briefing of their Applied Research Project where faculty and administrators will evaluate the student’s ability in presenting research. Successful completion of this briefing will be required before the faculty will sign the Applied Research Project approval page.

Since the programs inception, five MPA students have completed the Applied Research Project. Copies of these completed projects are maintained in the MPA Coordinator’s office and copies are also shelved in the MSUB Library. Current students are encouraged to read these Project reports when planning their own work. Currently, five MPA candidates are working on their Applied Research Project.

Communication and Theatre

In addition to the Masters program in Public Relations, the department offers undergraduate programs in Communication Arts and Public Relations. Communication courses are also required in Academic Foundations Program. The department participates in and uses National Public Relations Association recommended standards for program delivery and in 1996 was awarded the Program of Excellence Award at the NCA Convention.

Assessment of Undergraduate Programs: Students in B.A (Mass Communication Option) and B.S. Public Relations must complete COMT 470: Capstone Project. These students are required to complete a 120 hour internship or project. At the end of the project students submit a Portfolio as well as a paper. The paper requires the student to evaluate their preparation for the internship/project as well as to comment on the coursework they have taken prior to this course. Similar capstone courses are required in other options of the program.

Employer feedback on recommended student competencies resulted in digital editing, digital camera work and editing being added to program curricula. Student and alumni feedback from questionnaires and focus groups were considered in curriculum revision. Community input was sought during development of the Public Relations program. The department maintains high standards of advising and has created major and program-specific advising worksheets to monitor student progress.

Competencies for students in the Master of Science in Public Relations program are determined through required course work and one of the three required options. All students complete 24 hours of coursework, including 15 hours of core classes. Students are given three options for completing the program.

Thesis: This option is advised for students who are planning to continue their graduate studies. Students who choose the thesis option work with a thesis advisor as they prepare their original research contribution. These students must submit a written thesis to their committee. That committee then hears an oral presentation of the student’s research, asks questions of the student and acts as the final authority over whether the student has successfully met the outcomes of the project. Thesis completion generally takes at least six months, during which time the student is submitting drafts and re-writing given the feedback of the thesis advisor.

Project: This option is advised for students who are intending to work in public relations. Students who choose the project option work with a project advisor to develop and prepare an original professional contribution. These students submit a project to the advisor. The project often goes through several revisions before it is accepted.

Coursework: This option is advised for students who are already working in the public relations field or those students who are interested in continuing with coursework. Students in this category are assessed based on the outcomes in the individual courses.

Modern Languages

The department offers a major and a teaching certification option, a minor and a teaching minor in Spanish. Other languages such as Arabic, Chinese, French, German, Japanese and Latin are offered on a regular basis. In-class proficiency exams and informal feedback from students and alumni have led to the following types of changes:

• Offering graduate courses to service the recertification needs of public school teachers
• Using information to facilitate the hiring of part-time instructors.
• Adding a Spanish accelerated review course for students with two or more years of high school Spanish. Review of these offerings has led to their being dropped.

The major in Spanish requires that all graduating seniors in the traditional Bachelor of Arts program write and defend orally a research paper. Traditionally, members of the community conversant with the material discussed in the senior paper are encouraged to participate in discussion during the oral defense. Seniors must register for SPAN 499, Senior Paper, to receive credit for the paper. A portfolio of writing samples is also kept on file for both traditional and certificate BA’s. Currently, Spanish majors with a certificate use their successful completion of student teaching in Spanish to substantiate their command of material.

All Spanish majors seeking certification are also required to complete an internship in Spanish or to travel abroad in a Spanish speaking country in order to prove their ability in the language. For credit for these classes, students must register for SPAN 490 Internship or LANG 495 Overseas Experience. Traditional BA’s usually complete one or both of these courses, but neither is specifically required.

History

The Department of History offers a major, a teaching certification option, a minor and a teaching minor. History is a required category in the Academic Foundations Program. Assessment of the History Programs has been a constant effort on the part of the departmental faculty and intensified fall 2006. Following a survey, the program was revised to focus more clearly on achievable goals. Perhaps the clearest manifestation of this effort was the revision of history survey courses that were formerly known as World Civilizations to/since 1500.

After several years of trial, the history faculty came to the conclusion that teaching World Civilizations has not been successful because of the breadth of the subject matter. A hybrid course titled “The West and the World to/since 1648” was therefore created. History teaching majors need a background in Chinese and Indian civilizations that they had not been getting in the previous survey courses. The curriculum was revised to require upper-division coursework from a list of world civilization courses. These changes became effective in fall 2007.

The root-level of outcomes assessment for the Department of History has been an effort by each faculty member to identify the particular learning outcomes for each individual course. The goal of this effort is to specifically identify and articulate the enumerated learning outcomes for each course. The department insures that each syllabus includes a list of learning outcomes for the course.

Outcomes are assessed by periodic written examination and by historical research and critical writing on historical projects that become increasingly sophisticated and advanced as students progress through the program.

The Department of History capstone project is a product of the Historical Methodology course. All students are required to research and write a major research paper of 30 pages, based on primary (original) sources and refined to readiness for publication in an historical journal. Since all majors and minors have to take the course, the final product is the culmination of student work in the department.

The Department of History has also been diligently working on an assessment plan for the Academic Foundations courses. The Department’s “Academic Foundations Courses Assessment Plan” includes the primary assessment tools that each survey course will employ to guide the faculty to comply with the Academic Foundations outcomes.

Mathematics and Computer Science

The program in mathematics follows in philosophy and function the recommendations of the Committee on the Undergraduate Program (CUP) of the Mathematical Association of America (MAA). The program in mathematics, teacher certification option, meets all the standards of NCATE and OPI. The program in computer science follows the model curriculum laid out by the Association for Computing Machinery (ACM). Statistical Methods I and I1 follow the recommendations and meet the guidelines set down by the American Statistical Association (ASA) and MAA joint committee on teaching introductory statistics courses.

Utilizing Mathematics Association of America standards, the curriculum was recently revised. The Mathematics Department uses continuous assessment to determine the content knowledge of its graduates. The forms of assessment are traditional: graded homework, time-constrained examinations and classroom presentations.

The mathematics program emphasizes applications of mathematics and computational mathematics, in accordance with that College of Arts and Sciences strategic plan. The department’s application-oriented curriculum emphasizes computational mathematics. The unique aspect of the program is faculty intent to incorporate computational mathematics, usually involving Mathematica, throughout the curriculum. As a result, most lower-division courses (e.g., Math 112 and Math 113) use Mathematica homework problems as assessment tools, while upper-division courses often require Mathematica or other computer-generated projects. The use of computational facilities allows richer, more complex questions and problems than would otherwise be the case. The result is that assessments based on computer problems and projects provide a truer gauge of students’ depth of understanding of the mathematical content of their courses.

State of the art computer software is employed in virtually all courses. Computer software and hardware are maintained by the IT staff. The use of software in mathematics, statistics, and computer science courses is monitored continuously and reviewed annually by the Department of Mathematics. Students majoring in mathematics have the opportunity to work as student tutors in the Academic Support Center (ASC). Math 390, Peer Tutoring, is a course that is available to students who wish to tutor and receive credit for it. This course is recommended for prospective teachers. The Math Department has a UNIX formatted system that tracks student progress.

An emerging method of assessment involves student participation in a senior seminar. Over the past several years, the department has offered students the opportunity to study mathematical topics outside the usual curriculum in a seminar setting. The purpose of the seminars has been to prepare students for graduate school, but both teaching and non-teaching option students have participated. As part of their participation in the seminars, students have demonstrated the breadth and depth of their mathematical knowledge and their understanding of the unity of the subject.

For this reason, the department decided to formalize the seminars as a required capstone course.

A change to the teaching option program is the return of Math 421-Abstract Algebra, replacing Math 371-Numerical Computation. This change was made to ensure that the program complies with revised mathematics standards. The revised program, including both the senior seminar and Abstract Algebra, became effective fall 2003.

During the 2007-2008 academic year, the Department of Mathematics continued to make progress in implementing continuous assessment of beginning, core and lower level mathematics classes. Contract faculty incorporate continuous assessment using Math XL, a computer assessment system. The same assessment technology is also incorporated in the online courses to assure uniform outcomes whatever the format of the courses. Learning objectives for all mathematics majors and minors were established.

Learning objectives for statistics and computer science minors have been revised. The standard format for Mathematics Department syllabi currently includes course (learning) goals and objectives. Faculty are reviewing all courses to ensure that goals and objectives are consistent with program learning outcomes.

Program goals are assessed on a course- wise basis by continuous assessment (using Math XL) and by comprehensive final examinations. In addition, a senior seminar, together with other upper-division courses, is used to assess students’ ability to communicate mathematically and to master new mathematics on their own. Student performance on examinations and in the senior seminar is used to assess the program and provide a basis for modification and improvement of the program. A recent example of program modification is the change in prerequisites for Discrete Mathematics. Discrete Mathematics now has Computer Science 101 as a prerequisite in order to ensure students are able to complete computer assignments relevant to the discrete mathematics curriculum.

Psychology

A graduate program and an undergraduate program are offered.

The Department of Psychology is committed to frequent and informative student evaluation to determine if required coursework is educationally sound. Some courses (such as 206-Research Design) require project development, analysis and public presentation. Such activity allows faculty to evaluate the educational outcomes of courses.

In addition, job tracking after graduation helps to determine if the combination of coursework and community experience required are furthering student careers. Some faculty do additional teaching evaluations beyond the required University evaluations to assess the effectiveness of their courses. Online courses are evaluated both by the students and faculty peers. Feedback from these evaluations is used by faculty in changing course content and presentation.

Assessment activities that have resulted in positive changes include:

• In 2006, the Psychology department began to formalize the feedback from the capstone course (Psych 499) to the lower division courses 101, 205 and 206. This feedback prompted several curricular revisions in laboratory courses. These changes are designed by the faculty to improve performance in writing and research skills. The labs now include more quantitative analysis, research design (including more practical experience) and statistics (including more methods and analyses).
• The department continues to give a sample of seniors a portion of the GRE and is discussing how to use this information. Results indicate that MSU billings students performed above the national average in the area of clinical psychology.
• A new formal training program for graduate students who will be teaching in the department has been implemented.
• Students working on research projects are more involved with the IRB as a result of assessment
• Internships at the graduate and undergraduate level have been substantially reorganized.
• The graduate internships have been completely routed through Dr. Woolston (a faculty member and a practitioner in the community) for quality control and learning outcomes assessment. Guidelines have been agreed upon recently.
• Undergraduate interns are assigned to many of the clinics and organizations in town that provide excellent learning opportunities in the community. Many of the graduate students have jobs with local clinics and non-profit organizations.
• Two new assessment courses in the graduate program have been introduced and the undergraduate testing course reorganized in response to employment opportunities in the community involving assessment.
• More conversations with graduate schools to better understand how undergraduates and graduates can better apply to doctoral programs.
• A former graduate student is employed as a counselor in almost every clinic in town that has helped the department to support undergraduate students involvement in the community.

Assessment of the Graduate Program in Psychology: Academic competency is determined from performance in a required core of four courses. Students have an option to demonstrate clinical competency by choosing either an internship or a thesis. Progress and performance is monitored and feedback provided.

Over 90% of students completing the graduate program are accepted in to doctoral programs.

Evidence:

• 1.2 Annual Reports;
• 2.11 General Bulletin; COT Catalog;
• 2.5 Graduate Catalog;
• 2.38 Capstone Projects in Environmental Studies

College of Business

As an integral part of its work towards AACSB accreditation, the College of Business developed an assessment plan to demonstrate assurance of student learning. The COB developed a list of learning goals and learning objectives that derive from the COB mission statement. The assessment plan includes alignment of learning objectives with the common curriculum for all COB students. The plan identifies points in the curriculum at which assessment takes place, the tasks to be assessed and the measurement instruments to be used. The assessment process includes courseembedded measurement and national examinations.

Development of the student’s knowledge of the traditional and functional areas of business, the behavioral and social sciences, the management sciences, and the systems approach to problem solving is of key importance to business programs.

Chart 2.4 - College of Business AACSB Learning Goals and Objectives

Integrated throughout is an emphasis on developing the ability to make effective operational and administrative decisions.

Additionally, intensive study in a chosen option provides the student with indepth knowledge necessary for an understanding of the global economic and social systems as well as their relationships to the individual and the organization.

Each program is designed to provide the undergraduate student with the background to effectively participate in business, industry, government, education, and other organizations and institutions that require management and administrative competence. The programs also provide the basic undergraduate education required for admission to graduate study in professional fields such as management, law, urban planning, and health administration.

Following is an up-to-date report (6/20/08) on the College of Business Assurance of Learning Procedures and Current Status.

This report outlines the procedures and current status of the assurance of learning process towards AACSB accreditation for the MSUB College of Business.

1. Developing Learning Goals and Learning Objectives.

At the AACSB Continuous Improvement Conference (attended September, 2006) and the AACSB Assessment Seminar (attended November, 2006) it was learned that AACSB has a format for the wording of the Learning Goals and Learning Objectives. During Fall semester, 2006, the assessment teams in the COB re-evaluated and re-worded the Learning Goals and Learning Objectives.

The faculty resolved to accept the Learning Goals and Learning Objectives on February 21, 2007. With the exception of a few word changes, these are the Learning Goals and Learning Objectives that were sent to the Provost on February 15, 2006.

Status: Done

2. Developing Alignment Matrix

According to the guidance from consultants and the presenters at AACSB Applied Assessment Seminar (attended January, 2006); an alignment matrix should be a component of our report to AACSB.

The alignment matrix identifies where each of the learning objectives is taught, reinforced, and/or applied in our business core courses. According to AACSB, each course must teach, reinforce, and/or apply at least one of the objectives.

Work on the alignment matrix began in the Fall semester, 2006, with input from faculty and information from an early report to AACSB.

The COB faculty added content and student assignments on ethics and international topics to business core classes during spring semester, 2008.

Status: Done

3. Developing the Assurance of Learning Plan

The following matrix outlines the tasks involved in developing an assurance of learning plan and presents a proposed plan for the COB.

Developing the Assessment Instrument/Metric involves designing a rubric (instrument) with appropriate traits to be assessed or a metric (such as a passing grade on all or part of an exam or quiz).

Identifying an Assessment Point involves selecting a course in which assessment takes place. Conducting assessment in a course is a preferred approach because the intent is to use a course-embedded assignment as the assessment task. Not all assessment may be done this way. If standardized tests are used, it is still recommended to identify a course in which the students are notified about the test. AACSB recommends that the assessment point be the senior capstone course. It may not be feasible to conduct all assessment at that point in the curriculum.

Identifying an Assessment Task involves, first, looking for a course-embedded assignment that students are currently being graded on so that students are already motivated to perform well. If no course-embedded assignment is available, work with teacher of the course to develop an assignment that addresses the learning objectives. If that fails, find another course with a course-embedded assignment that addresses the objective.

The assurance of learning plan is organized in an Assessment Matrix. When the plan is complete for any one objective, assessment can take place. The COB dean and faculty agreed to administer to students in the capstone courses, Mgmt 488 (onsite students) and Bus 485 (online students), the ETS Major Field Test in business to assess business discipline areas.

Status: Done

Chart 2.5 - College of Business Assessment Tasks

4. Conducting the Assessment

This step of the process involves logistics and scheduling. For example, getting the video equipment to record the oral presentations, making arrangements with test vendors and obtaining permission to purchase, getting permission from faculty member to use assignments, getting the assessment team together to assess with rubrics, to name a few.

Status: Done except for the following in process: Assessment of oral presentations for online students (LG1, LO1).

5. Analyzing the Results and Closing the Loop

The data has to be gathered and recorded/deposited electronically so that faculty, administration, and accreditation teams have access to the data.

Status: COB website has to be updated.
ETS test was administered to onsite and online students at the end of spring 2008. Results have not yet been received.

The following improvements are suggested from meetings with faculty:

LG1, LO1 – Oral presentations from the onsite students met expectations in all traits. No improvements are suggested at this time.

LG1, LO2 – Written communication from onsite and online students indicates much improvement is needed. Lorrie Steerey, Don Larson, and Jenny Leonard volunteered to serve on a Writing Task Force to develop guidelines for faculty to reinforce better writing skills to be implemented in 2008-09.

LG2, LO1and LO2 – Critical thinking tests and written assignments of onsite and online students indicate that improvement is needed. A faculty workshop is planned for the fall 2008 retreat to help faculty design assignments that ask students to demonstrate critical thinking skills to be assigned in 2008-09.

LG3, LO1 and LO2 – CompXM test results for onsite and online students indicate that improvement is needed, particularly in the accounting area. Improvements to be implemented in fall 2008 include online homework to provide more practice in Acct 233. In spring 2008 a comprehensive exam in Acct 234 was administered to identify topics to reinforce in Bus 315 during the next academic year.

LG4, LO1 – Written assignments of onsite and online students indicate that some improvement is needed, particularly with regard to analyzing alternatives in an ethical situation because assignments did not ask students to do so. Assignments will be modified in the next academic year to address this objective. In spring 2008 faculty implemented changes to the core business classes to include more coverage and more assignments in ethics across the core curriculum. Improvement is expected in future assessments as students are exposed to ethical decision-making for more than one semester during their college curriculum.

LG5, LO1, trait 1 – Term papers from onsite and online students indicate that the majority of students meet expectations regarding uses of technology to solve problems or make decisions, but not necessarily in a business context. Current assignments will be modified or new assignments will be developed by the instructor and a faculty team in fall 2008.

LG5, LO1, trait 2 – Written case assignments on ethics in technology for online students indicate that some improvement is needed. As ethics in the curriculum is being emphasized more across the core curriculum, improvement is expected in future assessments as students are exposed to ethical decision-making for more than one semester.

LG5, LO2 – A change in the core curriculum was implemented in fall 2006 to ensure that all graduates have the necessary skills. Students must pass a competency test or are required to take MIS 225 as a remedial course. Students must pass the core competency test or the course to graduate.

LG6, LO1 and LO 2 and LO3 – Written case assignments and essay test questions from onsite and online students both indicate that much improvement is needed. In spring 2008 faculty implemented changes to the core business classes to include more coverage and more assignments of international topics across the core curriculum. Improvement is expected in future assessments as students are exposed to international topics for more than one semester during their college curriculum.

In addition, the College of Business (COB) has implemented the use of the Educational Testing Service (ETS) instrument. Furthermore, the university began using the Noel Levitz Student Satisfaction Inventory (SSI) instrument in 2006. The results indicated the COB was below standards in advising.

Since that time we have required mandatory advising, provided increased tutoring within Student Help Services and hired a coordinator of student and faculty services. The result is a proactive student oriented culture in the COB. The 2008 satisfaction level in advising showed a significant improvement. The 2008 SSI results indicated of the seventy-nine (79) satisfaction indicators, the COB was above the national average in seventy-eight (78).

Graduate Program in Business

The Master of Science in Information Processing and Communication (MSIPC) was established by the Board of Regents in November 1993..It never achieved its enrollment or graduation expectations. It reached a maximum of twelve annual graduates in 1997-1998, then the number dropped significantly with six being the highest. The program was deleted by Board of Regent action in November 2002, with a teach-out expectancy of four students graduating in 2003-2004, two in 2004-2005, and none graduating since.

Evidence:

• 1.2 Annual Reports;
• 2.11 General Bulletin;COT Catalog;
• 2.5 Graduate Catalog

College of Education

Mission

The College of Education is dedicated to:

• Preparing competent, caring and committed professionals for Montana’s schools;
• Conducting socially significant research to improve the human condition;
• Providing community services aimed at improving the quality of life experienced by Montanans;
• Providing graduate education designed for continuing development of professionals.

Outcome Performance Expectations for Undergraduate Education
Candidates (Initial Licensure):

Components of the COE Assessment System for Initial Licensure. The COE has replaced its multi-celled Conceptual Frameworks for both undergraduate and graduate students. The Conceptual Framework for students, either undergraduate or graduate, who are seeking initial licensure comprises the ten Interstate New Teacher Assessment and Support Consortium (INTASC) standards with attention to Montana’s Indian Education for All Act.

Standard 1: Content Pedagogy. The teacher understands the central concepts, tools of inquiry, and structures of the discipline he or she teaches as well as the historical-legal-philosophical foundations of education. The teacher creates learning experiences that make these aspects of subject matter meaningful for students.

Standard 2: Student Development. The teacher understands how children learn and develop, and can provide learning opportunities that support a child’s intellectual, social, and personal development. MSU Billings teacher education candidates understand differences among groups of people and individuals. In the context of human similarity, candidates are aware of United States and global diversity, respect and value differences, recognize that students and their families may hold different perspectives and strive to meet individual student needs.

• Standard 3: Diverse Learners. The teacher understands how students differ in their approaches to learning and creates instructional opportunities that are adapted to diverse learners. Montana educators understand and teach with attention to the cultures of Montana Indian nations.
• Standard 4: Multiple Instructional Strategies. The teacher understands and uses a variety of instructional strategies to encourage student development of critical thinking and problem solving.
• Standard 5: Motivation & Management. The teacher uses an understanding of individual and group motivation and behavior to create a learning environment that encourages positive social interaction, active engagement in learning, and self-motivation.
• Standard 6: Communication & Technology. The teacher uses knowledge of effective verbal, nonverbal, and media communication techniques to foster active inquiry, collaboration, and supportive interaction in the classroom.
• Standard 7: Planning. The teacher plans instruction based upon knowledge of subject matter, students, the community, and curriculum goals.
• Standard 8: Assessment. The teacher understands and uses formal and informal assessment strategies to evaluate and ensure the continuous intellectual, social, and physical development of the learner.
• Standard 9: Reflective Practice: Professional Development. The teacher is a reflective practitioner who continually evaluates the effects of his or her choices and actions on others and who actively seeks out opportunities to grow professionally. MSU Billings teacher candidates demonstrate professional dispositions both on and off campus.
• Standard 10: School & Community Involvement. The teacher fosters relationships with school colleagues, parents, and agencies in the larger community to support students’ learning and well-being.

These 10 standards are assessed in classes and during field experiences. They are easily understood by teacher education candidates and observable by faculty, field mentors and field supervisors.

Determination of Undergraduate Candidate Achievement (Initial Licensure)

The undergraduate program has the following four transition points with requirements as listed.

Transition 1 — Application to the Teacher Education Program

• GPA in Core Content and Core Professional Courses
• Writing Sample

Transition 2 — Preparation for Student Teaching

• GPA in Required Major Courses
• Performance in Jr Field
• PRAXIS II Content
• Professional Dispositions

Transition 3 — Preparation for Graduation

• GPA in Required Major Courses
• Performance in Student Teaching
• PRAXIS II Pedagogy
• Professional Dispositions

Transition 4 — Application for Licensure

• GPA in Required Content Courses
• PRAXIS II Content
• Observed Content Delivery in Clinical Practice

Outcome Performance Expectations for Graduate Candidates (Advanced Studies)

Components of the COE Assessment System for Advanced Studies. Graduate students, whether seeking initial licensure or pursuing advanced studies, must demonstrate the following six competencies, which replace a multi-celled Conceptual Framework with easily understood, observed and assessed performance outcomes.

Advanced Conceptual Framework. As part of their professional and pedagogical development, graduate students will:

• Locate, read, and evaluate relevant professional and academic literature (Addresses Framework Area #1: Research and Professional Inquiry from previous COE Advanced Conceptual Framework)
• Demonstrate understanding of similarities and differences in human learning and development (Addresses Framework Area #2: Human Development and Learning from previous COE Advanced Conceptual Framework)
• Demonstrate mastery in subject area content knowledge, understanding, and skill in applying that knowledge and understanding (Addresses Framework Area #3: Professional Knowledge Base from previous COE Advanced Conceptual Framework)
• Demonstrate professionalism that extends beyond technically accurate knowledge and effective skills
• Integrate extant theory and evidence into one’s professional activities
• Critically analyze policy and one’s own practice in light of professional standards and applicable policy (Addresses Framework Area #4: Professionalism from COE Advanced Conceptual Framework)

These outcomes assume competence related to the Interstate New Teacher Assessment and Support Consortium (INTASC) Standards required for undergraduate and graduate initial licensure candidates.

Determination of Graduate Candidate Achievement (Advanced Studies)
Graduate Advanced Studies students face the following two transition points
in their progress toward achieving a graduate degree, with the requirements as indicated:

Transition 1 — Graduate Plan-of-Study

• PRAXIS II Content and Pedagogy or GRE
• Undergraduate GPA
• Professional Dispositions
• Transition 2 — Graduate Program Completion
• Graduate GPA
• Internship Performance
• Action Research Project
• Professional Dispositions
• Capstone or Thesis

College of Education Response to Data Analyses

COE Assessment System Revision
As part of the 2002 NCATE accreditation process, the College of Education revised the Teacher Reflective Practice Model into Undergraduate and Graduate Conceptual Frameworks (Initial Conceptual Framework-ICF, Advanced Conceptual Framework-ACF). Each framework had three levels of performance; the Initial Conceptual Framework for undergraduates has four conceptual areas, while the Advanced Conceptual Framework for graduate students had five conceptual areas. It became apparent a third framework was necessary for outcomes expected of graduate candidates pursuing initial teacher licensure, the Initial Masters Conceptual Framework (IMCF). The COE developed an award- winning assessment framework through a student portfolio review process. Each cell of the appropriate framework was rated on a three-point rubric. The bases of assessment were candidate artifacts and professional reflection on those artifacts.

Implementation of the assessment process occurred with development of an MS ACCESS data warehouse. Analysis of the ACCESS system led to the major overhaul. This is because the process was not providing information valuable in programmatic decision making:

• Candidates were lost in the details. They did not understand how the details fit into an overall framework.
• Faculty were not assigning rubric scores consistently. Some based their scores on the quality of the artifact, others on the appropriateness of the artifact to a cell in the framework, still others on the candidate’s reflection.
• The scores were not discriminating between candidates who were mastering the concepts of the frameworks and those who were not.
• The ACCESS data base, constructed by a student who graduated and left campus, could not be queried to provide meaningful data. The only data available were specific to an individual or grossly aggregated scores. There was no way to query at specified transition points or to assess individual progress over time.

The College of Education revised assessment process has been in transitional implementation since fall 2007. The College of Education is developing and disseminating undergraduate and graduate Guide Sheets for students, Advising Center advisors and faculty advisors so that all stakeholders are fully aware of the COE assessment expectations. In addition to the above general requirements, students meet accreditation standards in specific program areas through the state Professional Education Preparation Program Standards. PEPPS is based on national professional standards for various content areas—arts and sciences areas, reading, special education, early childhood, educational technology and school counseling.

Undergraduate Candidate Achievement (Initial Licensure)

Transition 1—Application to the Teacher Education Program Students who have not completed Academic Foundations or maintained the required GPA have the option of a provisional admission contract. This allows them to enroll in upper division courses while they complete courses or work on raising their GPA. Students may continue in the Teacher Education Program for a maximum of three provisional contracts. They cannot enroll in either the Junior Field Experience or Student Teaching on a provisional contract.

Written products that are not acceptable when reviewed by the faculty advisor, are revised by the student.

The COE does not consolidate, report or review these data as they are not considered pertinent in terms of program revision. Faculty advisors work individually with student to guide their provisional efforts toward program admission.

Transition 2 — Preparation for Student Teaching To be licensed in Montana, Elementary Education majors must earn an overall score of 139 as part of the Montana three-pronged state assessment. Over 90% of MSU Billings candidates pass the test. Those who do not meet the 139 required score work with faculty advisors on a remediation plan. Student feedback indicated that they had little or no knowledge of economics, one of the areas on the test. Social Studies methods instructors added this topic to the course syllabus.

Included in exhibit 2.44 are scores from the few Secondary Education majors who took the state PRAXIS II content pilot test. AY 2008-2009, PRAXIS II content tests will be required of all Secondary Education Majors. CAS has been working with COE in faculty test review to determine which tests are most applicable to MSU Billings programs. Montana, as a state, is moving toward a three-pronged assessment for licensure for Secondary Education majors, similar to the requirement for Elementary Education majors.

Evidence: Exhibit 2.44 Praxis II Summary Data

Transition 3 — Preparation for Graduation
During student teaching, all education candidates will be required to take PRAXIS II PLT tests of pedagogy, beginning AY 2008-2009. Results of performance will provide data informing efficacy of methods courses and field experiences.

Observations of student teaching by site mentors and university supervisors guide decisions regarding performance in clinical practice. An unsatisfactory observation evaluation leads to a growth plan for the student teacher that is collaboratively developed by the Director of Licensure Standards and Clinical Practice, the site mentor, the university supervisor, the faculty advisor and the candidate.

Summary data on student performance is in the appendix. These data are being shared with the Clinical Practice Committee for analysis and program recommendations to the COE faculty. Evidences of Professional Growth — student teaching work samples — have been reviewed by either the LSCP Director or the Dean and results shared with faculty.

This process is not effective because candidates are not receiving feedback on their work. In AY 2008-2009, the reviews will be completed throughout the student teaching semester by faculty advisors so that guidance can be provided to candidates. Overall scores from the required 4 to 6 work samples in three areas will be provided to the Clinical Practice Committee for review and program recommendations to faculty.

Transition 4 — Application for Licensure
Once students successfully pass the PRAXIS II content exams and student teaching, 100% of them are eligible for licensure.

Graduate Candidate Achievement (Advanced Studies)

Transition 1 — Graduate Plan-of-Study Students who are provisionally admitted to a graduate program can maintain their candidacy for a maximum of one semester. If they are not successful in maintaining a 3.25 GPA, they can no longer enroll in graduate courses or complete their degree program.

Transition 2— Graduate Program Completion The COE initiated a COE Graduate Committee AY 2007-2008 with the intent of focus on graduate program integrity and quality. Currently graduate programs in Early Childhood Education, Reading Education, Special Education and School Counseling are aligned with national content standards. School Counseling conducts beginning, mid and ending program benchmarks but the checks do not currently have rubrics and no data have been collected related to the number of candidates who are counseled into an alternative career based on benchmark determinations. Only School Counseling has an internship form aligned with standards and judged with a rubric for performance.

The COE is in the process of strengthening candidate performance review and electronically data basing results for COE Graduate Committee review, analysis and recommendations to the faculty.

COE graduate capstone courses align with the Advanced Studies Conceptual Framework. Candidates have to meet the capstone performance outcomes assuring ACF outcomes as part of their plans of study in order to complete their graduate program.

Other Data Sources

Site Mentor/University Supervisor Feedback
Site mentors and university supervisors indicated that MSU Billings candidates did not know how to differentiate instruction for individual students with out-of the- ordinary educational needs. Based on this feedback, the COE teamed teachers in several iterations of an on-line differentiated instruction course and hosted faculty presentations in COE faculty meetings. A special education faculty person teams with regular education faculty to teach the elementary education junior field experience seminar. Field feedback now indicates that MSU Billings pre service teachers do a good job in meeting the needs of individual learners in elementary and secondary classrooms.

Employer/Recruiter Surveys
The COE has summarized data from Career Services surveys of school district recruiters who employ MSU Billings education graduates. These data are ready for review by the Clinical Practice and Knowledge/HQT committees.

Past reviews have indicated that candidates required more knowledge in assessment and in reading programs. Methods faculty strengthened curriculum based assessment in their courses and student teaching Evidences of Professional Growth demonstrate that students understand and can do this type of assessment. Standardized assessment interpretation remains a weak area. ETP faculty are adding this component to educational foundations courses.

The disconnect between reading theory and reading curricula currently being used in elementary classrooms were identified as an issue and have been addressed by adding the practical, in-use-curricula to the reading program. While students are not necessarily ready to implement a specific reading program without hiring district in-service, they are able to intelligently discuss the different research based models.

Faculty Research

Several faculty, alone and in collaboration, have conducted survey research with graduates. A consistent area of concern with first-year teachers is classroom management and discipline. A stand-alone course cannot provide context for this topic but on-the-job training during student teaching results in unnecessary anxiety. The COE Dean has contacted a consultant with expertise in building the internal locus of control in learners. He has presented to the College of Education Council and received positive response. He will participate in the COE August Retreat to introduce his conceptual basis to faculty. The hope is that this introduction will grow into an ongoing relationship providing professional development for faculty and valuable models for classroom use.

Evidence:

• 1.2 Annual Reports;
• 211 General Bulletin; COT Catalog;
• 2.5 Graduate Catalog

College of Professional Studies and Lifelong Learning

The CPSLL located in downtown Billings has served as the outreach arm of the university and the “incubator” for new program development. Formed in 1996, the college evolved from the former Center for Continuing Education, Summer Session and Community Service. Specifically, the college was formed to house new ventures, develop innovative and entrepreneurial programs, promote lifelong learning opportunities for its constituencies and provide a number of Universitywide functions.

While the college is not divided into formal departments there are several areas of concentration. Professional Studies includes the core academic programs delivered by the College in partnership with the other MSUB collegiate units including credit program development and delivery. Lifelong Learning includes processes for extension credit and non-credit programming. Workforce Development offers customized training for organizations and delivers programs on a contract basis. Finally, the College maintains a conferencing unit which offers an array of services from simple room rental to a turnkey conference operation. It should be noted that for certain initiatives all of these units may work together for successful program delivery. Within the conference center are two interactive television network classrooms which can link the learner with sites all over the world.

The college has administered both the Health Administration programs prior to its development into the College of Allied Professions and the MSU Billings online program course development and delivery. Health Administration has moved to the College of Allied Health Professions and online delivery to the eLearning Operations (online Hub). Both have developed data gathering procedures. For the past ten years, MSU Billings has used eCollege for online program/course delivery and the Learning Management System (LMS) and will be using Desire2Learn beginning spring 2009. Like the eCollege, this system contains measures for evaluation of the technical aspects of the course, teacher/pupil interaction, time online for students, and time online for instructors.

The CPSLL has a system in place for continuing education and outreach workshop evaluation. Whether offered for continuing education or University credit, CPSLL courses undergo review by academic departments to assure quality and integrity of the offering. Those offered for undergraduate or graduate credit are graded courses with expectations of similar rigor to courses offered through the other colleges. Faculty review, advisement, delivery, and/or involvement in the courses assure that they are compatible with regular program course offerings.

Evidence:

• 1.2 Annual Reports;
• 2.8 CPSLL Operations Guide;
• Standard 2.G.1

College of Technology

Mission

The mission of the College of Technology is to be the College of first choice, dedicated to the development of workforce capacity by providing top quality learning opportunities and services to meet a variety of career choices and customer needs by being responsive, flexible and market-driven.

Outcome Performance Expectations for Applied Certificate and Associate Degree Candidates
The COT offers 35 academic programs including 18 Associate of Applied Science (AAS) degrees, 10 Applied Certificates, 4 Associate of Science (AS) transfer degrees, and one Associate of Nursing (ASN) degree. The academic programs are organized by disciplines into four center divisions, including general education and learner support; industry and computer technology; transportation and business; and nursing, health and safety. Sixteen academic programs are either certified or accredited by national and regional organizations, including: Montana State Board of Nursing, National Council for State Boards of Nursing-Practical Nurse, National League for Nursing Accrediting Commission (NLNAC), National Council for State Boards of Nursing-Registered Nurse, American Association of Medical Assistants (AAMA), American Academy of Professional Coders (AAPC), American Registry for Radiologic Technicians (AART), National Registry EMTParamedic (NREMT-Paramedic), American Health Information Management Association (AHIMA), Commission on Accreditation of Allied Health Education Programs (CAHEP), Automotive Service Excellence (ASE), National Automotive Technical Education Foundation (NATEF), Center for Applied Process Technology< (CAPT), A+ Programming Language Certification (CompTIA A+), Network+ Certification (CompTIA Network+), Network Server + (CompTIA Server+), Microsoft Certified Professional (MCP), Microsoft Certified Desktop Professional (MCDST), Cisco Certified Network Associate (CCNA), Cisco Certified Network Professional (CCNP), Microsoft Certified Systems Engineer (MCSE), National Competency Testing Institute (NOCTI), and National Center for Construction Education Research (NCCER).

Since 2007, student learning objectives have been developed at the programmatic level and course level for all programs offered at the West Campus. In addition, academic programs regularly undergo review and redefinition in response to changing needs, changing resources, and continuous quality improvements. Fourteen industry program advisory councils (PACs) meet either annually or biannually (depending on the program) to review curricula, employment data, resource issues, and program goals. The advisory councils provide recommendations to program faculty and College administration.

During AY2008, 100 percent of all PACs met. Some committees operate at a much higher level than others. The College administration has established a goal to elevate all PACs to the same level of functionality. A PAC manual was developed and distributed to PAC members. In addition, the College administration has identified the need for all PACs to be chaired by a community member. A strong National Advisory Board comprising of 24 senior level business and community leaders and employers provides strong support and consistent feedback to the college and its programs.

Beginning in 2007, each program initiated an annual program report coordinated through the University CQI initiative, and incorporating suggestions and modifications recommended by PACs, as well as those internally identified. The annual reports are used to summarize the work and efforts conducted within the program, identify space and equipment priorities, and list goals for the upcoming year.

The past decade has brought many changes in the West Campus’s curricular offerings; these changes parallel local and national shifts in the economy. Some programs have been discontinued in the last decade because career opportunities are not as available or entry level wages are not high enough to warrant investment in training; for example, AS Word Processing; AAS Culinary and Hospitality Management; and the AAS in Medical Assistant is currently be taught out. New programs preparing students for service, information, and the energy economy including: AS Fire Science, ASN Registered Nurse, AAS Radiologic Technology, AS Human Resources, AS Applied Supervision, AAS Construction Technology, AAS< Process Plant Technology, AAS Power Plant Technology, AAS Desktop Network Support, and AAS Computer Programming and Application Development.

Program specific outcome performances/assessment benchmarks differ. In nursing, as is noted below, there are lists of specific student outcomes required by the State Board of Nursing for accreditation, which range from student demonstration of commitment to professional development through student demonstration of ability to insert an IV or give an injection. In the construction program, specific skills outlined in the curriculum are tested through construction of a house which is then inspected and marketed to the public. Meeting the inspection provides each student cohort with a validation of the skills learned, and individual competencies in multiple areas are demonstrated one-by-one to the instructor and scored as the project proceeds. Listings of other individual program outcomes may be found in the COT Catalog (Exhibit 2.11) as required by Standard 2.B.2, and can also be provided to the accreditation team upon request. Outcomes of the general education courses required in any given program (not all programs – certificate programs, for example – require completion of the general education core) are the same as those required in the four-year East Campus Academic Foundations courses.

Assessment Methodologies for Published Program Outcomes
All COT programs are evaluated for effectiveness on a regular basis. The rigor and internal engagement of faculty with these evaluative efforts differ from program to program, however, and ensuring consistency and the engagement of all faculty remains a goal. We are discovering that it is much easier to implement meaningful assessment strategies as new programs are developed, as opposed to working out assessment strategies for programs that have been in place for some time. Partly, this is due to the difficulty of overcoming established expectations and teaching/ evaluation patterns.

The new ASRN program, which was subjected to a very rigorous review by the State Nursing Board, is a good example of a program which includes strong assessment, review, and curricular modification triggers. As the program was developed, an assessment matrix was built which includes a timeline for review and program modification recommendations (exhibit 2.10).

In addition, examinations and checklists targeting specific student outcomes and skills – for example, demonstrated ability to insert an IV, and demonstrated commitment to professional development – are built into the program. Industry feedback via the required preceptorships and clinicals help us assess students’ progress with respect to skills required locally. Finally, performance on a nationally-normed examination, the NCLEX, serves as a “reality check” for the knowledge and outcomes being delivered both by local healthcare experiences and our curricula. This level of assessment: regularly scheduled internal program review; built-in benchmarking against local industry practice and expectations; and nationally-normed examination – yields a strong level of confidence that we will know when we meet, exceed, or fall short of the specified program outcomes.

Not all programs can offer that level of confidence in their outcomes. Several programs (including some newer additions, such as Radiologic Technology) have strong assessment systems in place. Examples include:

LPN Program: The LPN program began using competitive enrollment criteria in spring 2006. As a result, program retention has increased from a low of 67% in the fall of 2004 to 96% in spring 2007. Prior to the approval of the new ASRN Nursing Pathways Model and implementation of the program in January 2008, the nursing program did not administer either employer surveys or student graduate surveys. Beginning with spring 2008, the program implemented a pilot survey distribution for employer and student employment surveys. The data will be collected to measure the attainment of the learning outcomes for the program graduates.

While the new RN program does not yet have cohort graduates whose outcomes can be assessed, a remarkably rigorous evaluative structure is in place, mandated by the Montana State Board of Nursing. The assessment rubric for the program as a whole is attached as Exhibit 2.10. The graduate surveys and employer surveys for the ASN program are matched to the nursing program evaluation model for future National League for Nursing Accrediting Commission (NLNAC).

Chart 2.6 - Retention of COT LPN Students
(Cohort Defined by Initial Entrance into NURS 206: Nursing Fundamentals)

While the new RN program does not yet have cohort graduates whose outcomes can be assessed, a remarkably rigorous evaluative structure is in place, mandated by the Montana State Board of Nursing. The assessment rubric for the program as a whole is attached as Exhibit 2.10. The graduate surveys and employer surveys for the ASN program are matched to the nursing program evaluation model for future National League for Nursing Accrediting Commission (NLNAC).

Chart 2.7 - AAS Practical Nursing Program Input/Output Data Students NCLEX-PN pass rates and Student Graduate Numbers

Radiologic Technology: The first class of students entered the Radiologic Technology Program in January, 2005. Since that time, three cohorts have graduated from the program and the fourth cohort is currently in attendance. The graduate placement and general success of the program has been excellent. Yearly surveys are taken of all program graduates, as well as their employers, approximately six to eight months after graduation. Employer satisfaction results from the surveys have been mostly very good and excellent. Results of the surveys include:

• 100% of the surveyed graduates obtained employment as radiologic technologists
• 100% of the surveyed graduates passed the national ARRT certification examination
• Program retention to date is 95%

The Radiologic Technology Program goal is to strive for constant improvement through feedback received from graduate surveys, employer surveys, student course evaluations and the Radiologic Technology Advisory Committee. The Program Advisory Committee consists of the directors of radiology from both hospitals, both clinical instructors, two students from each of the current cohorts and supervisors or representatives from the major employers of radiologic technologists in the community. The following list includes some major changes which have resulted from the program assessment process:

• Clinical experience in the program has increased in two steps from 1135 to 1620 hours of clinical. The program received strong support for this change from all involved, including radiology students, clinical instructors, program director, and unanimous support from the entire Advisory Committee.
• Three major curriculum revisions have been completed since the start of the program. These revisions changed the total clinical hours and also removed and added courses to strengthen the program. All curriculum revisions were discussed and approved by the Radiology Advisory Committee with the program director and clinical instructors were the main drivers of this change.
• Employer surveys resulted in program changes including increasing the length of radiology modality rotations, increased operating room student rotations
• Program reduction in cohort size from 16 per year to 14. Discussions at the Radiology Advisory Committee meetings prompted pursuing the reduction in the cohort size to keep the clinical instructor student ratio at an effective level, and also to address the concern that the region is becoming saturated with technologists, seriously affecting students’ ability to find positions after graduation.
• Student evaluations prompted the change of text for a radiology course.
• Clinical Instructor Meetings are held with the program director two to three times per months and frequently include the radiology student representatives. There is frequent, usually daily communication with the program director and between the clinical instructors.

Another result of the assessment process has been the development of a Bachelors of Applied Science with Emphasis in Radiology Management through MSU Billings College of Allied Health‘s 2 + 2 program. This degree provides both local and rural radiologic technologists that have an associate degree the opportunity to complete sixty specified credits and complete the bachelor’s degree with courses which are all available online.

Automotive Collision and Repair, Technology, and Diesel Programs:
Transportation programs (Automotive, Collision Repair and Diesel Technology) are all NATEF accredited, which signifies that the curriculum covers all content recommended by the national accrediting body. The programs are rigorously evaluated by external reviews on a minimum of a five year review cycle. Internally, programs track student progress using written and laboratory exams designed by the national organization. The Collision repair program has implemented an endof- program exam developed, delivered and compiled by the national entity. These results are then reported to the faculty by the accrediting group. Every successful graduate of the technology programs is prepared to attempt the national certification exams. Additional student success is compiled through student self reporting of certification exam pass rates.

NATEF does not report the test results to the post-secondary institutions. All data concerning test results is only available from student self-report. The program department chair has tracked NATEF test results for the past two years and has determined the West COT Campus students are averaging near 70 percent pass rates. As a result, the Transportation faculty are developing a one credit certification preparatory course specifically designed to provide preparation for the NATEF examinations. A requirement of this new course will include student self-report of their scores.

Although other programs of study (Paramedic, Medical Coding and Insurance Billing, Radiologic Technology, Automotive Technology and Autobody Collision Repair) prepare students for national certification or licensure, our Practical Nursing Program is currently the one with exam pass rates readily available through state databases.

The following academic programs reported the collection of data on student learning outcomes in their 2008 annual reports, and noted that they are using the data collected to guide program improvement: AAS Accounting Technology, AS Applied Supervision, AAS Automotive Technology, AAS Automobile Collision and Repair, AAS Computer Desktop Networking Support, AAS Computer Programming and Application Development, AAS Computer Systems Technology, AAS Construction Technology, AAS Diesel Technology, AAS Drafting and Design, AS Fire Science, AS Human Resources, AS Medical Coding and Insurance Billing, AAS Practical Nurse, AAS EMT-Paramedic, AAS Process Plant Technology, AAS Radiologic Technology, and Cert. Welding.

The following academic programs reported in their 2008 annual reports that they are not collecting data on learning outcomes to guide program improvement:

• AAS Administrative Assistant.
• AAS HVACR.
• Medical Administrative Assistant.

These areas are, therefore, our most immediate priority as we continue to work toward a culture of quality improvement and effective assessment. Each of these areas will need to make progress toward development of outcomes assessment in order to demonstrate program strength and viability. In West Campus Business programs, which as a long-established program has struggled with maintaining perceived program strengths while modifying practice to include more rigorous evaluation, the Deans of the COB and COT have opened discussions and exploration of a formal 2+2 pathway, which would require some rethinking and restructuring of curricula. This provides an excellent opportunity to incorporate assessment and outcomes measurement; such metrics will be a necessary part of demonstrating student ability to transfer to the upper-division courses within the COB. Transfer and articulation of programs into four-year degree paths are one tool the COT can use to build stronger assessments for all programs as we move forward.

Achieving comprehensive excellence in assessment remains our objective. The college has come a long way in the last several years in changing its culture and strengthening its ability to prove the excellence of its work on many levels. On-going work between the Union-Management Team and the University administration should yield a mutually acceptable system of faculty peer-review and evaluation, which will be an additional and critical component of our assessment efforts.

The depth and breadth of faculty engagement in the assessment process at the West Campus has varied. During AY 2006-2007, much greater definition of student learning outcomes and student learning outcomes assessment occurred as part of the evolution of the West Campus from a two-year vocational-technical college to a community college. The new position of Associate Dean for Student Learning was filled in July 2006 with a major goal for the Associate Dean to begin working with program faculty to identify overall program and individual course student learning outcomes. As a result of this effort, each academic program at the COT established both program and course outcomes. As part of the validation process, these newly identified outcomes were presented to Program Advisory Committees (PAC) and have now been included in the College of Technology 2007-2008 General Catalog. Assessment of how well successful students meet the program learning outcomes is being built incrementally for every program in the College.

Systematic assessment at the program level is routine in the nursing, health, and safety programs and the industry programs where the accrediting organizations requires such a level of assessment. However, many of the non-licensure/accredited disciplines are either just starting to make progress toward the goal of systematic assessment or are in the very early stages of establishing these goals.

College faculty and administration also began looking carefully at program advisory committee participation and working to increase the value of input received from it. As the annual reports for several programs note, advisory committee engagement is excellent in some areas and much less strong in others. Advisory committees are intended to have a significant impact on curricular change (the primary “process outcome” of assessment) by offering a reality check from the businesses and industries in which the skills we teach will be applied. Thus a strong advisory committee has a direct impact on student learning (the primary goal or ultimate outcome of assessment) by ensuring that students learn the employment skills that most of them are seeking.

Additional measures of external program assessment include Developing a Curriculum (DACUM) studies and periodic program accreditation and certification. DACUM studies were conducted for two existing programs, Drafting and Design Technology and Welding/Metal Fabrication. The intent of engaging in this process was to determine the current efficacy of these two programs. The third DACUM study conducted in this time period assisted in the creation of curriculum for a new program, Power Plant Technology. The studies proved their worth and have been shared with internal and external constituencies to demonstrate the College’s commitment to maintaining current and relevant programs of study.

Our ultimate objectives for internal assessment measures in each of these divisions include student ratings of instruction and peer (faculty) review of programs. Several program faculty use student portfolio development to assess student learning outcomes in courses and programs. These include Drafting and Design Technology, Applied Supervision and Human Resources.

Areas that use final projects or major assignments to assess overall grasp and integration of student knowledge include Computer Programming and Applications Development, Construction Technology, Drafting and Design Technology, Fire Science, and Mathematics.

A list of programs which use external evaluation or professional examination to complement internal evaluative strategies has been presented in the Exhibit 2.10.

Improvements in the advisement area, intended to impact student outcomes and improve student experience, have included new student orientation programs, mandatory student advising, college success classes, faculty-led early intervention and placement testing.

Indicators of the success of program curricula impacting student learning include:

• External nationally/regionally-normed examinations
• Student placement and employer satisfaction
• Outcomes of student portfolio development
• Outcomes of final/major student projects
• Retention and graduation rates

Indicators of the success of the larger COT environment in impacting student
experience include:

• Retention and graduation rates
• Program growth and enrollment increases
• Student survey data [see results from University-wide Student Satisfaction Inventory and the Community College Survey of Student Engagement (CCSSE)

Employment placement statistics are another important part of program assessment. Career Services data reveal that the COT’s placement rate is 96%.

Chart 2.8 - College of Technology Graduate Data

This provides us with a sense of business and industry faith in our graduates and programs, and of the economic value to our students of their degrees.

Evidence:

• 1.2 Annual Reports;
• 2.11 General Bulletin/ COT Catalog;
• 2.5 Graduate Catalog

Back to Top

Standard 2

Standard 2 Breakdown

2.A.1 Faculty Expertise

2.A.2 Institutional Instructional Niche

2.A.3 Instructional Publications

2.A.4 Degree Designators

2.A.5 Educational Program Schedules

2.A.6 Educational Program Formats

2.A.7 Curriculum Design

2.A.8 Library Resources

2.A.9 Student Performance Measures

2.A.10 Experiential Learning Credit

2.A.11 Program Modification Policies & Procedures

2.A.12 Program Elimination Policies & Procedures

2.B Educational Program Planning and Assessment

2.B.2 Educational Outcomes Objectives

2.B.3 Educational Outcomes Planning

- College of Allied Health Professions

- College of Arts & Sciences

- College of Business

- College of Education

- College of Professional Studies & Lifelong Learning

- College of Technology

2.C Undergraduate Program

2.C.2 - 2.C.3 General Education Requirements

2.C.4 Credit Transfer Policies

2.C.5 Academic Advising

2.C.6 Developmental Education

2.C.7 - Developmental Education

2.C.8 State Licensure

2.D Graduate Programs

2.D.1 Graduate Program and University Mission

2.D.2 - Graduate Program Assessment

2.E.1 Graduate Faculty and Resources

2.E.2 - Graduate Program Growth

2.E.3 - 2.E.4 Graduate Faculty Credentials

2.E.5 Graduate Program Delivery

2.F.1 Graduate Records and Academic Credit

2.F.2 Graduate Program Admission Criteria

2.F.3 Graduate Program Governance

2.F.4 Graduation Requirements

2.F.5 Transfer Credit Policy

2.F.6 Graduate Program Credit for Experience

2.G.1 Continuing Education and Special

2.G.2 Continuing Education Course Offerings

2.G.3 Continuing Education Extension Programs

2.G.4 Continuing Education Organization

2.G.5 Accessibility of Programs

2.G.6 Continuing Education Fee Structure

2.G.7 Continuing Education Credit Policies

2.G.8 Continuing Education Review Procedures

2.G.9 Prior Learning Assessment

2.G.10 Transfer and Articulation Agreements

2.G.11 Student Learning Assessment

2.G.12 Travel/Course Study Policies

2.H.1 Non-Credit Programs and Courses

2.H.2 Non-Credit Program Policies and Procedures

2.H.3 Continuing Education Credit Policies