Programmatic Partial-Credit Put-In-Order Grading

Back in Autumn, 2009 we introduced online tests with several question types. Certain types—multiple choice for example—are simple for a computer program to grade. You just tell it what the correct answer is, and if the student marks that answer, they get the credit. Other question types, however, aren’t so simple: put-in-order, for example. Different instructors have different methods and rationales for how to assign credit for a partially correct put-in-order question, so replicating how a human would grade one is no mean feat. Or so we’ve learned over the last five years.

Our first method was simple: we'd evaluate each item in turn, and if it was before or after the item that it was supposed to be before or after then it would be counted as correct.

This did a pretty good job approximating how an instructor might give partial credit by focusing on the order of items as opposed to their placement. This method stayed in place until we discovered its principal flaw. Though highly unlikely, it was possible for an incorrect response to receive 100% credit provided each item was next to at least one correct neighbor.

When an instructor brought this to our attention last Winter (over four years since we added online tests!) we quickly revised our methodology to focus on placement. This seemed like a simple, reasonable method: imagine a teaching assistant lining up an answer key next to the student’s response and marking incorrect any item that didn’t match the key.

In reality, though, this sometimes proved much harsher than an actual professor would be, especially if the question had a larger number of options.

In fact, not long after the update an instructor showed us a rather harshly-graded 25-item put-in-order question; Populi counted 13 options as correct when the instructor would have counted 24. In light of this, we sought out another approach. The best programmatic method for giving partial credit on put-in-order questions would need to take into account more than simple placement in order to better replicate how a human teacher would grade and avoid being overly harsh or generous. After testing every method we could think of, here’s what we came up with.

The new method aims to give as much credit as is reasonable (as most instructors would) by focusing on what we’re calling chains—that is, two or more correctly-ordered items in a row. First, we locate the longest chain. Then, we use it to figure out whether or not other chains before it or after it are in order. Anything not in a chain is incorrect.


This method worked well overall, but there were a couple wrinkles to iron out. One was that the first and last items in the list are at a disadvantage when it comes to chaining: each has only one neighbor to chain with, and so are less likely to be counted as correct. This was solved by treating the top and bottom boundaries of the list as non-credit positions. In other words, if the first or last item is in the correct position it always counts as being in a chain, and receives credit.

The other wrinkle: a response could have more than one longest chain. Depending on which chain you started with it was possible to come up with a different number of points. Here, starting with the first chain leads to a lower score:


We solved this by grading the question multiple times, as it were. We look at each chain, and then look at the position of each chain in the answer. We then see which chain to use as a starting point to grant the most credit.

  • Above, starting with eight-nine-ten  would cause Populi to mark the other two chains wrong (because one-two-three and four-five-six don't come after eight-nine-ten). This would result in a score of 30% of the possible points.
  • Below, starting with one-two-three lets us then say that four-five-six is correct (because it comes after one-two-three). This lets us mark two chains correct, giving the student more credit for the question (60%)—and thus, is preferable to the other option. Starting with four-five-six results in this same score.


There's also a very, very remote possibility that the most credit would be awarded by starting with the second-longest chain. So we also try grading using every longest chain and every chain with one fewer item than the longest chain, just in case.

We’re happy to announce this as the new (and hopefully final) method for assigning partial credit to put-in-order questions.

Now, partial credit is, after all, just an option on a feature. But the time we spent working it out and building it is worth it. Professors rely on Populi to save them time with the mundane things (like test-grading), but some of the mundane things are hard to nail. It's actually quite a challenge to replicate a teacher's intuition with rigid, literal software code.

As a bonus, we now show which items were marked incorrect in the test history view so students and teachers can see how the partial-credit grade was derived.

Editor's Note: As you suspected, Yes—the title of this article can be sung to the tune of Supercalifragilisticexpialidocious from Mary Poppins.

Way more free file storage for everyone

June 18, 2013.

April 7, 2014.


Doing what we do best, we're pleased to announce that we've yet again given everyone more free file storage. Have a gander at our Pricing page if you don't believe us.

  • The Small plan just got a little less small with 1,000—count 'em: one thousand—free gigabytes of file storage.
  • The Medium plan is even medium-er with 2,000 free gigs of file storage.
  • And the Large plan? Yup, it's now larger: 3,000 gee-bees of file storage, included free.
  • If you manage to go over your limit, additional storage costs just 10 cents per gigabyte per month.

If you used to pay for storage above your plan's limit, you're probably paying nothing for those files now. And if you've been holding back from uploading files for fear of passing your limit, well, fear no longer—each plan has a lot of free storage.

In case you didn't know, you can do a lot of stuff with files. You can embed audio and video files in your courses so your students can stream them anywhere on almost any device. Faculty and students can upload and exchange assignment files—and instructors can even annotate documents with feedback. Plus there's all the humdrum stuff—files in the Activity Feed, the Files app, profile pictures, application files... and so on.

Since files are such an important part of how our customers use Populi, we're happy to give everyone more opportunity to take full advantage of the functionality—without having to pay extra.

Improved tuition schedules, refund policies, and IPEDS reporting

Early in the morning of Friday, August 1, we released a number of upgrades to student billing together with a smattering of  improvements to other areas of Populi. Here's the low-down:

Tuition schedules

We gave tuition schedules some new features and changed how they relate to courses and programs.

  • Tuition brackets now let you combine flat rate and per-unit charges. Say you want to charge full-time students $5000 for 12-18 credits and then additional $295 for each credit over 18—you can now do that with one tuition bracket.
  • Schedules are no longer linked to academic programs, but rather to individual courses. This lets you get real specific with how you charge your students for particular courses.
  • Programs can now have multiple default tuition schedules. So if you have one schedule that charges $500/credit for science courses and another that charges $400/credit for humanities, you can attach both to, say, your Undergraduate Program. Students in the Undergrad program would then get charged according to how many science or humanities courses they enrolled in.

Refunds are altogether better and more usable than they used to be:

  • Refund policies have been separated from tuition schedules; you now add them directly to student accounts, and they directly govern any automatic refunds the student is to receive.
  • Refunds now issue as a simple credit—previously, there was a two-step process involving a full credit and then a new charge for the non-refunded amount.
  • You can now limit how far into the term a student is eligible for a refund.
Other assorted student billing items

New fee rules for course delivery method and term name (e.g., the fee only triggers in the Fall term) give you new options for automating fees.

You can now edit the posted date of a transaction at the time that you add it. So, when you invoice charges, you can change the posted date; or when you run a financial aid batch disbursement, you can change the posted date for those transactions. And so on...


The preset IPEDS reports in Academics > Reporting got a couple of nice upgrades.

  • We added the Financial Aid survey to the IPEDS Winter Collection.
  • We now lets you export IPEDS reports in XML format. This means that you can upload your Populi-generated data directly to NCES—no more copy-pasting of hundreds of rows of numbers from Excel!


Read about all of the week's updates (including bugfixes) in the Release Notes.

Electronic checks

You can now use Populi to accept electronic checks for online payments!

Screen Shot 2014-05-05 at 5.50.51 PM

Electronic checks let you accept online payments straight from the payer's checking account. To pay, just enter a routing number, an amount, and some other basic info—like what you'd find on a paper check—and click Pay Now. It's pretty simple for the payer. On your end, the fees are lower than credit/debit card transactions, and the turnaround time is a fraction of that of paper checks.

To get set up with electronic checks, you'll need to sign up for, an add-on product for payment gateway accounts. If you're already using Populi with for credit card processing, you're most of the way there. Just visit's Merchant Services page and start your application. Once approved, get in touch with Populi Support and let us know you'd like to enable electronic checks.

Read more about setting up Online Bill Pay in the Populi Knowledge Base.

Coming soon: better tuition schedules, refund policies, and more

The Populi development team is hard at work on improvements to student billing. Here's a quick look at what's coming...

Tuition Schedules

We've re-written tuition schedules from the ground up.

  • Tuition brackets will let you combine per-unit and flat-rate charges. For example, you'll be able to set up a bracket that charges a flat amount from 12-16 credits and an additional per-credit charge for everything above 16 credits. This isn't possible with our current tuition schedules, which rigidly enforce a distinction between flat-rate and per-credit brackets.
  • Tuition schedules will no longer be tied directly to academic programs. Rather, they'll be connected to individual catalog courses. When selecting courses, you'll have options to match the schedule to specific courses, a particular department, or a particular program.

7-25-14 New TS

Refund policies

We're separating refund policies from tuition schedules. That will let you:

  • Attach them directly to student accounts, or as a default to individual tuition schedules.
  • Set up one, single policy for your school and be done with it—unlike the current system, which requires you to duplicate the refund policy with every tuition schedule you set up.
  • Better understand how the refund policy affects non-tuition charges like fees.

Most important, refunds will make a lot more sense.

  • Currently, if a student gets an 80% refund on a $1000 tuition bill, Populi 1) credits the student $1000 and 2) then adds a new $200 tuition charge.
  • After the update, it will simply credit the student $800.

7-25-14 Refund policy

A few other things

We're adding new fee rule types for clinical hours and term name (which lets you charge a particular fee every Fall term, for instance).

The new recalculate function will let you instantly update a student's pending charges.

7-25-14 recalc

Getting ready for the release

We're planning to get this out there to our customers by August 1st. If you have any questions about it, please contact Populi Support.

New course and gradebook features

The Populi development team released a heap of highly-requested assignment and gradebook features last night, including:

  • Extra credit assignments
  • Attendance assignments
  • Drop lowest grade options
  • Student feedback on all assignment types
  • Document viewer with annotations on file assignments
  • Improved gradebook navigation
  • New global minimum attendance settings in Academics