Discussion: View Thread

  • 1.  Online exams are a hard problem - here is one solution

    Posted 07-03-2020 04:37:00 PM
    Edited by Munir Mandviwalla 07-13-2020 10:14:20 AM

    Online exams are a hard problem. Commercial solutions ignore a lesson of cyber-security research:  Perfect security is impossible; managing risk and costs is more important. Of the two online proctoring market leaders: Choice 1 is one-on-one creepy stalking and choice 2 is an invasive surveillance system. To me, the costs are too high.

    Online exams should follow these principles: 1. Minimize cheating risk, 2. Respect students dignity and privacy, 3. Maintain rigor, and 4. Do not increase grading time.

    Following the above principles, I designed an alternate approach, which will work for about 40 students:

    1. Exam has a time limit and is only available on a specific date/time for the set time, and displays a countdown counter. This is possible in most LMS.
    2. Exam has two components:
      • Standard multiple-choice questions. Randomize the question order, as well as the answer choices. Most LMS offer this capability as an option.
      • Provide a deck template in which the student has to move elements around to create something new. For my UX design class, I provided pre-built interface objects to build a new UX, which the student could drag and drop, and then on the next slide, I provided space for a narrative. A similar approach is possible for other types of content (e.g., data elements, code snippets, conceptual objects). Most LMS also offer file upload.
      • At the top of exam, add the following sentence: I agree to follow the XXX University Student Code of Conduct, I understand that the exam will be recorded, and I meet all the requirements (see below). I am ready to start this exam.
    3. Set up a zoom session with recording enabled which the students are required to attend during the exam date/time. List the following requirements:
      • Well lighted table or desk with no other materials such as books, notes, phones, or people in the vicinity. 
      • The exam is closed book, closed notes. You are not allowed to use email, Internet, smartphone or any other device or person to gain help during the exam. 
      • Disable any virtual backgrounds.
      • A computer with webcam turned on.
      • Up to date web browser.
    4. During the exam, watch the students. This has the added advantage in that the instructor is available to answer any questions, and all hear the Q&A.


    The time limit, randomized multiple choice, zoom session, design question submitted on a deck, and requirement to (re)accept code of conduct minimizes but does not eliminate cheating risk, while respecting students, preserving rigor, and minimizing grading time. (bonus: grading is more efficient!).

    One of the students wrote the following in my teaching evaluations:

    "the online exam was the best I had taken all semester. no AI proctoring either which helped with the entire class stress levels."

    The above is not rocket science, I am sure others did something similar or other approaches that are more creative? Can this be adapted for larger classes?

    #Learning Action

    ------------------------------
    Munir Mandviwalla
    Temple University
    ------------------------------​​


  • 2.  RE: Online exams are a hard problem - here is one solution

    Posted 07-03-2020 05:02:00 PM
    A useful protocol.

    I am thinking of having a test bank of questions and each student gets a random subset of these in random order. This makes it harder to cheat, but is more work for the instructor.

    ------------------------------
    Richard Watson
    University of Georgia
    ------------------------------



  • 3.  RE: Online exams are a hard problem - here is one solution

    Posted 07-08-2020 12:40:00 PM
    Alternatively, think carefully about the intended learning outcomes and the best means of assessing them (in many cases I suspect that an unseen examination with no access to notes may only be focusing on the student's ability to memorise rather than necessarily demonstrate their competence in the subject matter).

    Some resources - https://info.lse.ac.uk/staff/divisions/Eden-Centre/Assessment-Toolkit/Selecting-assessment-methods-based-on-skills and https://info.lse.ac.uk/staff/divisions/Eden-Centre/Assessment-Toolkit/Assessment-conditions/Exams

    ------------------------------
    Edgar Whitley
    Dr
    London School of Economics
    London
    (44)2079557410
    ------------------------------



  • 4.  RE: Online exams are a hard problem - here is one solution

    Posted 07-11-2020 01:31:00 PM
    I long ago adopted a policy of open book exams and skills testing, but I am struggling to find a way to get students to watch videos.

    In my regular classes, I get student to present on relevant topics and then we have a Q&A. 

    When we went to an online class last semester, I asked the students to prepare a video of their presentation. I quickly learned few in the class watch these. As a result, a learning opportunity was lost. Giving a quiz on the videos is the only solution I can identify to resolve the problem. 

    Any ideas?


    ------------------------------
    Richard Watson
    University of Georgia
    ------------------------------



  • 5.  RE: Online exams are a hard problem - here is one solution

    Posted 07-13-2020 09:47:00 AM
    Two, indirect, thoughts, which ultimately comes back to what learning outcomes you want in relation to the videos produced by other students.  We have students produce videos (and share them publicly if they want).  This doesn't necessarily mean that they will watch them.  We do, however, have them present a "live" (formative) version where they get feedback from their fellow students in the seminar group (~15 students in total) so some students will have seen the (early) version.  More generally, you might want to explore peer review - either as part of the formative work or even as part of the final assessment.

    (My reflections on a course (MG4G7) that does a bit of both can be found here https://info.lse.ac.uk/staff/education/Sharing-examples-of-LSE-online-teaching)

    ------------------------------
    Edgar Whitley
    Dr
    London School of Economics
    London
    (44)2079557410
    ------------------------------



  • 6.  RE: Online exams are a hard problem - here is one solution

    Posted 07-13-2020 10:13:00 AM
    Edited by Munir Mandviwalla 07-13-2020 10:13:55 AM
    Peer review is a good tool which I have used in online classes, and it affords an important side-benefit, it also builds engagement. However, I question the inherent value of videos in the first place. Why are they needed? One way transmission only works for certain types of content, e.g., learning how to operate machinery. Talking heads in HD video by themselves has never really worked for the majority. However, using them as a means to build engagement affords more promise.

    ------------------------------
    Munir Mandviwalla
    Temple University
    ------------------------------



  • 7.  RE: Online exams are a hard problem - here is one solution

    Posted 07-13-2020 04:48:00 PM
    Thanks to Edgar and Munir for their advice.

    I will use a peer review model.

    ------------------------------
    Richard Watson
    University of Georgia
    ------------------------------



  • 8.  RE: Online exams are a hard problem - here is one solution

    Posted 07-14-2020 05:04:00 AM
    Learning Action

    I find videos to be useful if students are creating it with a particular user or group of users in mind.  This makes the video more targetted to a purpose.  It is not always one way transmission as it also tests students' understanding of the software, the business problem and the users.  Students who watch also learn what makes a good business case and demonstration.  I think peer formative assessment is much more useful than summative.  My observation is that peer summative assessment tends to value more (give higher score) to the popular student, the funny, the extrovert and markdown the least popular, introvert, shy and hesitant English speakers who tend to be sided by their peers.  Personally I prefer, peer formative assessment for this reason and discuss the scores in class so students know that they are responsible for their voting.

    ------------------------------
    Amany Elbanna
    Dr.
    Royal Holloway, University of London
    Egham
    ------------------------------



  • 9.  RE: Online exams are a hard problem - here is one solution

    Posted 07-19-2020 06:53:00 PM
    Edited by Harminder Singh 07-19-2020 06:58:31 PM
    Thanks for the suggestions, Munir. One issue is that not all students may be able to meet this condition for the Zoom exam: "Well lighted table or desk with no other materials such as books, notes, phones, or people in the vicinity." Some students work on their studies on the shared dining table/kitchen counter, because they do not have their own room or their own desk. They may also live with younger siblings or older family members, who may interrupt their studying/exam time. I think it is important to cater for conditions which do not have the potential to exclude anyone, which is why I am not keen on having closed book online exams.

    Harminder Singh