All of Oregon Tech's programs conduct annual assessment of their program student learning outcomes (PSLOs) -- the key knowledge and skills that programs are designed to impart and which set students up for professional success in their discipline. In program assessment, faculty examine student work with respect to these PSLOs and ask whether the program as a whole is adequately preparing students. In areas where improvement may be called for, faculty then develop and implement plans to modify courses, curricula, or student support and eventually re-assess to confirm that changes achieved the desired results.
Program assessment is a way of making sure we honor our commitment to our students and continuously improve in the instruction we deliver in our programs. In each academic program, a program assessment coordinator leads these efforts with the collaborative involvement of all program faculty.
Changes to Assessment Processes for 2018-2019
Program Reports -- No More LiveText! A New Report Guide!
After a review of program assessment reports which saw a decline in quality of assessment work, numerous concerns about poor usability and rigidity of the LiveText system, and LiveText's new status within a corporate merger all led Oregon Tech to terminate its contract with LiveText. All data and student work has been extracted, if programs need access to any of it.
Because program assessment reports will no longer be entered into LiveText, programs will return to preparing their own report documents (in Word, LaTeX, or their preferred editor), as programs did in 2016 and earlier.
To support programs in structuring their reports, we have developed a new Program Assessment Report Guide. which provides directions and examples for programs in constructing their annual program assessment reports. This guide is not a straight-jacket that must be rigidly followed -- programs should adapt the format to serve their purposes -- but outlines the key areas that should be included.
Embedded within the Program Assessment Report Guide is Oregon Tech's program assessment report rubric. Each winter term, the Assessment Commission will evaluate program reports using this rubric and provide feedback to each program on areas to improve in the next year's assessment work.
This guide is a living document -- we welcome feedback, suggestions, and critiques as you use it.
New assessment software -- Portfolium -- is available for use!
ESLO Assessment: Upload work only, score at assessment days!
In past years, we have asked the faculty who teach courses to score student work using the ESLO rubric. Unfortunately, this means that different people use the same rubric in different ways, compromising the validity of the data.
We will run several Assessment Days during the year for collaborative scoring of work (dates TBA), and ask that each program contribute 2-3 hours of faculty time to these days during the year. We also hope that the discussions that occur in these sessions lead to a greater sharing of ideas between faculty.
Program Assessment Report Guide
Key Deadlines for Program Assessment Coordinators
At Convocation and early in fall term, program assessment coordinators should:
- Program Assessment:
- wrap up the previous year's assessment activity by reviewing results from the previous year and planning for improvements to be implemented in response to assessment data.
- plan for the current year's assessment activity by identifying courses or experiences where student work should be collected during the coming academic year, and setting up plans for faculty to collect and evaluate that work.
- ESLO Assessment: Work with program faculty to identify course(s) to be used for ESLO assessment.
By October 31, 2018
- Student Exit Survey: Work with program faculty to review and, if necessary, update your program's questions in the student exit survey.
, submit to Barb Meng (firstname.lastname@example.org):
- Program Assessment: Your Program Assessment Report (see Program Assessment, below)
- ESLO Assessment: Which course(s) from your program will be used for ESLO assessment (see ESLO assessment, below).
- Student Exit Survey: Any changes to your Student Exit Survey content. (see Student Exit Survey, below). If we don't hear from you, the previous year's content will be used.
Responsibilities of Program Assessment Coordinators
Program assessment coordinators coordinate program assessment activity, including:
Plan: Each year, update your program's annual PSLO assessment cycle to identify courses and assignments/activities used for program assessment and faculty responsibility for scoring and analyzing data (or for collecting data to "close the loop" and evaluate the effectiveness of improvements).
Collect: Ensure that program faculty score and analyze student work as assigned (preferably collaboratively), and that student work and scores are collected (in Portfolium or via other departmental methods).
Analyze: Ensure that data from program assessment is shared with and discussed by all program faculty. Useful questions to consider as you reflect on the results together:
Reflection on Results
- For which assessments did students meet our department’s target? For which did they not?
- What surprises us about this data?
- What does this data reveal about our students’ strengths? What does it reveal about our students’ weaknesses?
Reflection on Curriculum:
- How can we strengthen the development of our students in these outcome areas, from their first year to graduation?
- What specific changes to courses or curricula would be needed to support these outcomes better within the curriculum? (If so, who would implement them and when?)
Reflection on Assessment Process:
- Should changes be considered to our outcomes and/or how we assess them?
- Do our outcomes and the way we assess them (courses, assignment, rubrics) capture evidence of our students becoming the sort of graduates we hope to produce – both with the knowledge skills to succeed immediately post-graduation and over the course of their career?
Act: Ensure that your program/department identifies a plan of action (to improve courses, assessment processes, etc.) in response to your findings and discussion of assessment data.
Report: Ensure that the above work is documented in your annual program assessment report, due by October 31 of each year. See the Program Assessment Report Guide (above) for more details.
Coordinate ESLO assessment activity within your program, including:
- Plan: At the start of the academic year, work with program faculty to identify a course in your program to collect work for ESLO assessment. Inform the Office of Academic Excellence what course has been chosen by October 31
- Collect: As that course is taught, help make sure program faculty upload ESLO work to Portfolium. Ensure that work is uploaded during the term the course is taught
- Analyze: For 2018-2019, we are asking each program to contribute 2-3 hours of faculty time to assessment days for scoring and discussion of ESLO work during the academic year
For 2018-2019, the ESLO being assessed is Ethical Reasoning:
Ethical reasoning is the process of recognizing which decisions require ethical judgments, determining potential reasonable courses of action, finding support for potential courses of action, and then selecting the course of action best supported.
- Theory: Demonstrate knowledge of different ethical theories and codes.
- Recognition: Recognize decisions requiring ethical judgments.
- Logic: Demonstrate knowledge of the logic of ethical reasoning.
- Judgment: Make and support plausible ethical decisions.
Please see the Ethical Reasoning ESLO webpage for more information.
Ethical Reasoning Rubric
At your convocation departmental meeting, discuss with your program faculty:
- Confirm the program-integrated practice course from which you will provide data for institutional assessment of Ethical Reasoning. Identify course number(s), instructor(s), and term(s).
- If a program is offered at multiple sites or modes, comparable assessment should be done at each site or mode.
- You should ideally select a course taken by students in which student work in the context of the discipline can be evaluated on as many of the criteria as possible. This course may be either at the introductory (first-year) level, or at the upper-division. Do not feel as though you should redesign an assignment because it's not a perfect fit to the rubric.
Student Exit Survey
Program assessment coordinators are the primary point of contact for coordination of the programmatic portions of the Student Exit Survey.
Each year, coordinators are asked to confirm or update your program's student exit survey questions and student solicitation message. (If you don’t update this, the same questions and solicitation message will be re-used from the previous year.) Communicate any updates to the Office of Academic Excellence by October 31
The Student Exit Survey is distributed around week six of each term based on a FAST report that pulls information from the Registrar's Office on the submission of Petition to Graduate. Summary results for each program are delivered during the summer to each program assessment coordinator and department chair.
Professional Development Opportunities
Although not required for assessment coordinators, participating in the following opportunities will help give program assessment coordinators and other interested faculty a deeper view into the assessment process, and are likely to provide ideas to use in program assessment and course improvement: (voluntary but highly recommended)
* Program Assessment Report Review: Each year, the Assessment Commission invites all coordinators to participate in our systematic review of program assessment reports. Invitations to this will be sent out as review sessions are scheduled during winter term.
* ESLO Assessment and Engagement: ESLO committees, in coordination with Assessment Commission and the Commission on College Teaching, hold occasional workshops to support faculty in understanding ESLOs and supporting students in ESLOs. Look for these opportunities as they arise throughout the academic year.
* Assessment Days: Several periods during the academic year will be set aside for collaborative scoring and discussion of student work sampled for ESLO assessment. We ask each program to contribute several hours of faculty time to this activity over the course of the academic year, and to invite other faculty to participate as well.