22

Outcome Assessment Tool

Paul Stamm

June 28, 2012 15:08

None

Follow

I realize what I am suggesting would be a significant addition to your programming/data storage/manipulation – but to my limited understanding this seems like a logical extension of your already wonderful platform.  Furthermore given the huge costs associated with the existing marketers of “outcome assessment services” this type of addition would be a God-send.

The accreditation services are pushing for this type of data driven outcomes assessment (Higher Learning Commission, North Central Association and the National Council for the Accreditation for Teacher Education).  I have outlined below what I think a Populi version of assessment might look like – but I would also recommend looking at Mayville State university’s document on assessment. http://www.mayvillestate.edu/about-msu/Pages/AccreditationAssessment.aspx This is a pretty good overview of how they are approaching the demands on their school

How I would envision it in Populi . . .

When creating a new “Lesson” the faculty member is given the option of a check-box for “Approved Outcomes Document.”   Selecting that item offers a list of “Institutional Goals” and “Desired Outcomes.”  These would have been entered by the school’s “Curriculum Coordinator.”  They might look something like this . . .

                I.  Statement of Goal

Paragraph of explanation of goal.  Paragraph of explanation of goal.  Paragraph of explanation of goal.  Paragraph of explanation of goal.  Paragraph of explanation of goal.  Paragraph of explanation of goal.  Paragraph of explanation of goal.  Paragraph of explanation of goal. Paragraph of explanation of goal.  Paragraph of explanation of goal.  Paragraph of explanation of goal. 

IA. Individual Outcome

IB. Individual Outcome

IC. Individual Outcome

 

The faculty member would have between a six and two dozen such choices.  They would mark all of those goals that apply to this specific lesson.  At the end of the year the Curriculum Director would be able to have a report showing how many times each of the individual outcomes was addressed within the selected lessons. 

A similar connection would be made to self-selected assignments and tests.  When preparing a test or assignment the faculty member would have and additional check-box called “Approved Outcomes Assessment.”   Selecting that item offers the same list of “Institutional Goals” and “Desired Outcomes.”  The faculty member would again mark all of those goals/outcomes that were evaluated by the assignment/test.  At the end of the year the Curriculum Director would be able to have a report showing how many times each of the individual outcomes was addressed within the selected evaluations (tests/assignments).  It would be best to have some correlation to the grades on these assignments.  So the results might show something like,

IC Standard – Of the 234 lessons sampled, IC was a component in 23 lessons (10%).  The standard was evaluated in 2468 student assessments with an average student score of 87.567%

Obviously additional ways to slice the data would be helpful (comparing freshmen to seniors, by departments, by declared degrees or majors, by course levels).  There would also be some benefit to being able to draw a digital or printed report and have the underlying documentation also available.  So for Standard IC – here are all of the lessons as presented with the Populi or the actual test.  In a absolute best case, although probably not critical, here are the written assignments generated by the assessment and electronically submitted to the faculty member.

• Facebook
• Twitter
• LinkedIn

31 comments

• 1
  
  Patrick Friesen July 20, 2012 17:08

  I could see this being employed by many schools not just as a way to comply with accreditation requirements but also better the institution. A strong "Me too"

• 1
  
  Terry Hiebert August 30, 2012 22:41

  Anything to save time with Learning Outcomes Assessment would be appreciated.

• 0
  
  Joel Badal September 06, 2012 17:42

  Yes we need it.

• 2
  
  Michael DeLashmutt May 21, 2014 15:45

  I totally agree.  As a dean at a former institution, I nearly purchased a tool called eLumen (http://elumen.info) to do more or less this kind of work.  It'd be much cleaner if integrated into the LMS/SIS.  

  As an aside, I think most approaches to outcomes assessment are just too complicated. If you guys wanted to design a tool which works well with what Barbara Walvoord describes in Assessment Clear and Simple, you'd be on your way to a goldmine!

  We need to be able to easily identify institutional, program, degree, and degree component learning outcomes (major, gen ed, etc. - or in Populi language, course groups) which are met through evidence gathered at the course level. Assessment data in support of learning outcomes can be direct (student work) or indirect (student perceptions) and either qualitative (review of an essay) or quantitative (compiled through gpa analysis).  Some shops like to do outcomes assessment for all courses all the time, others prefer to do sampling (a couple of courses used as indicators for a couple of learning outcomes each year) while other schools prefer to use a big formative assessment exercise like a portfolio to test program outcomes over time.  What matters, more than anything, is that you have outcomes, that you have a way to assess that these outcomes are being met, and that you have a mechanism for reflecting on outcomes attainment as an institution (closing the feedback loop, as they say!).  

  At the course level, you could declare learning outcomes as a property of the course (similar to a course description) and optionally align some of your assignments with the course learning outcomes (essay A tests the student's achievement of outcome C).  I think that course level outcomes assessment should be  rolled up into major or program outcomes (ENGL 101, assignment C is an indicator of program-level outcome A).  

  As an institution, we need to know at least three things about outcomes: 1)  which program outcomes are satisfied by which courses 2) which assessment satisfy the course level outcomes; 3) how well are students meeting these outcomes (sliced by program, major, student demography, level, and over time).  

  Given that most of Populi's clients are part of the ATS family of schools, I would think that there's some wisdom to be found in working with them on this project, too.
• 2
  
  James May 22, 2014 22:24

  Just a heads-up: I ordered the book @michael recommended and when it arrives we'll be doing some research and try to come up with a game plan for working Outcomes into Populi - thanks all for the comments/recommendations.

• 1
  
  Randy Siebold November 20, 2014 00:27

  This is HUGE! Because of our move towards accreditation, we are wanting to be more clear about our desired ends—what we are trying to achieve (Goals, Objectives, Student Leaning Outcomes, etc.)—as well as how we are achieving it (classes, mission trips, etc.) and to what level our students are achieving our desired ends. Populi is already collecting much of our "how" and "to what level." The added components would be to add a place to include our desired ends (for institutional, program and activity levels) as well as some sorting mechanism for collecting the relevant data together (most of which is already in Populi now).

  Currently, we are looking for a secondary application to move data from Populi and organize it for us. More money, bigger pain. We would love to see Populi put the same simple-solution philosophy to this as well!

  Either we work with 

• 0
  
  Susan Carlino December 16, 2014 18:15

  I wondered if you, James, have looked into working outcomes into Populi, and are any closer to coming up with a game plan.  Would you mind updating us?

• 4
  
  James January 09, 2015 20:02

  Hi Susan,

  Sorry for the delay - the holidays are a crazy time around here! The short answer to your question is that yes, I carefully read Assessment Clear and Simple and we are closer to a game plan, but unfortunately I can't give a timeline on when we'll be able to actually start work on this.

  Here's the big picture so far:

  1) Add the ability to define "goals" or "outcomes" at the institutional, department, course group, and specialization levels (e.g. "Students will be able to think critically"). Each level of goals should be able to use goals from a higher level and add sub-goals.

  2) Allow courses/course instances/lessons/assignments/discussions to be linked to the goals they support.

  3) Add grading rubrics to Populi for use in essays/papers.

  4) For assignments, be able to link a) a grading rubric criteria (e.g. "Grammar", scale of 1-5 on a paper), or b) just a particular assignment with no rubric (an online test, for example) to one or more goals. Each linked rubric/assignment score will be aggregated as a "measurement" underneath the relevant goals.

  6) Allow files to be uploaded for each group of goals (e.g. institutional, Engineering Degree, etc) for each academic year to keep track of yearly summary reports, assessment meeting minutes, etc.  Accreditors would have access to these along with all the other data for each goal.

  7) Someday maybe build surveys that can be sent to non-Populi users and linked as measurements to one or more goals (not a first-version feature).

  8) Someday maybe allow an "assessment rubric" to be attached to any assignment (in addition to the "grading rubric" certain assignment types might already have). Then allow multiple faculty members to evaluate all students' work (or a sample) for that assignment using the assessment rubric, and report the result as another measurement under one or more goals.  This would allow for "student portfolios", but is also probably not a first-version feature.

  All this is very exciting, and I look forward to when we have time to start on it!  The development team here always has many, many, irons in the fire and so I can't comment on when that will be, though.  We'll mark this feature "Planned" when work actually commences.

  Thanks!
  James

• 2
  
  James January 09, 2015 20:15

  One more thing... specific questions on course evaluations should be linked and aggregated under goals, as well.

• 1
  
  Mike Krauss March 30, 2017 04:13

  Wow, we WHOLEHEARTEDLY AGREE! Assessment from course-level section-by-section up to department goals to program to institution is HUGE for us. It's crucial right now in our pursuit of accreditation! We've spent considerable money on a system that was disappointing and lackluster. It would have been so nice to have this integrated into Populi. As described/suggested above, the process would have been considerably streamlined. I can't believe this doesn't have more votes!!

• 0
  
  Daniel Moore September 12, 2017 21:33

  We would have keen interest in this feature. If the needle has moved forward and you'd like a pilot test parter, let me know.
  ~Dan
  Aquinas Institute of Theology, Director of IT

• 1
  
  Paul Stamm May 10, 2018 20:28

  Has there been any advancement in the linking of Learning Outcomes to Populi's other functions? 

  The material James speaks off, sounds wonderful, but rather more complete and involved than I would have originally intended.  I think it should have been relatively simple to have SLOs by Department linked to individual courses, and hence to individual units and measuring techniques within the courses.  If I could pull a report that this or that SLO was actually taught in a specific course it would be a HUGE blessing.  Then if I could also show that the SLO was somehow tested (quiz, exam, paper, speech) within courses that alone would save us hundreds of hours and thousands of HLC questions!  Those statistics alone would make a world of difference.  Of course the ideal would be tying it all back to individual students - but even without that it would be a great blessing.

• 0
  
  Adam Sentz May 10, 2018 20:41

  @Paul - We've got a key component pretty far along in development. Once that is completed it will be possible for us to build out outcomes.

• 0
  
  Mike Krauss July 19, 2018 21:08

  @Adam - How close is this to beta-testing?  As is everyone here, we're very interested in this and would be happy to help beta-test!

• 0
  
  Adam Sentz July 20, 2018 16:35

  @Mike - Thanks for your interest. We don't always do public betas on new features, but if we decide to here we'll keep you in mind.

• 0
  
  Daniel Moore July 20, 2018 16:37

  Greetings, Adam, Mike (and Paul). Adam, we're certainly interested in beta-testing as well, should you go that route.

• 0
  
  Adam Sentz July 20, 2018 16:41

  @Daniel - Thanks! In this particular instance we aren't planning to, but we appreciate you interest as well. We'll post an update here once it's available so you can be among the first to try it out and let us know what you think. 

• 0
  
  Daniel Moore July 20, 2018 16:43

  Many thanks in advance, Adam.

• 0
  
  Adam Sentz August 06, 2018 16:49

  The key component I mentioned is now available.

  EDIT: Fixed the link.

  Edited by Adam Sentz August 06, 2018 17:47
• 0
  
  Joel Badal August 06, 2018 16:53

  Key Component link is not working.

• 0
  
  Daniel Moore August 06, 2018 16:55

  Thanks guys. I think this is the proper link.
  
  https://populi.co/blog/2018/08/rubrics/

• 2
  
  Jonathon Beeke February 27, 2019 15:59

  Would really like to see program objectives linked to rubrics, assignments, etc. This would really help in the whole assessment piece!

• 0
  
  Jonathon Beeke February 27, 2019 17:36

  Here is long video (about 30 minutes), but apparently Canvas is able to do this.https://community.canvaslms.com/videos/3890-using-canvas-outcomes-for-programmatic-accreditation

  This is GOLD for accreditation visits!!

• 1
  
  Daniel Moore May 03, 2019 22:32

  Greetings, checking in. Any progress towards goals, objectives and outcomes, now that rubrics have been in place for a couple of terms? Many thanks in advance.
  Gratefully,
  ~Dan
  Aquinas Institute of Theology, Director of IT

• 1
  
  Adam Sentz May 03, 2019 22:38

  @Daniel - We don't have anything to announce right now, but when we do we'll announce it here.

• 1
  
  Shawn Ragan August 14, 2020 21:21

  Just checking in on this. We are currently trying to determine how we will assess learning outcomes, and having some system in Populi would be fantastic (and save us from having to implement yet another solution for course and program assessment).  

• 1
  
  Dr. Kathryn Helleman (Faculty) January 20, 2021 14:49

  Adding my voice to this thread in support of Populi support from learning outcomes integration. Currently, we're collecting data externally to Populi. Integration would create a significant savings in time and the need for faculty to access and external process. The availability of live rubrics suggests to me that Populi is moving in this direction. Any updates? 

• 0
  
  Larry Udry November 18, 2021 18:18

  It sounds like this is a pretty common problem. I too add my voice to the chorus. FYI We have massive numbers of outcomes that we have to track for each ESL student each 8-week term for accreditation purposes. Basically a checklist of outcomes for each level in our ESL program. Could the mapping function of Populi be used or adjusted ( or upgraded ?) to facilitate mapping outcomes for a system of courses?

• 4
  
  Bruce Coats April 11, 2022 19:40

  Ten years after the initial request. Is there movement on the matter? It is also nearly that long since Moodle had a rudimentary outcome assessment tool. Surely, there is a way to do something, isn't there?

• 0
  
  Bill Thies December 04, 2023 20:15

  This would help provide some ability to manage a competency-based education program in the system.