Giving teachers more time for teaching and students more time for learning
Life just improved for our tutors and our first year students. Sussex University has introduced the policy and technology for online submission of essays and return of feedback, removing administrative tasks and making time for value-added activities such as teaching and learning. This is the start of a phased implementation of a three-year initiative to have online submissions for all undergraduate text-based assignments.
Ever since we have had a Learning Management System (LMS), Moodle, it has been possible for tutors to set up a “dropbox” (also known as the “assignment”) feature which allows students to upload essays online. Many of our tutors used the dropbox for formative exercises.
However, up until this year the University did not have a policy that allowed the dropbox to be used for assessments that would contribute to the students’ grades (aka summative assessments). The students had to print their essays and submit paper documents to their department office. This required them to spend time and money printing and coming on to campus to just submit their essays. Sometimes this was during assessment periods when no actual teaching was going on and they had no other reason to come on to the University. Tutors had to pick up their paper-based essays, mark them up with feedback and manually match candidate numbers to input grades on to the student record system.
The University has now rewritten its policy so that all suitable text-based assessments of first year students are submitted online. The policy specified that:
- Our LMS would be used by students as the interface to upload essays
Student view of the Assignment block
- Turnitin software would be used for plagiarism detection
- Grademark would be used for tutors to give a raw mark and leave feedback for students
- The grades and the institutional rules for grading would remain in our student records system and be passed to the systems that need it
We have been engaging in the technical development of this project since February. The University made funds available for a project manager, Catherine Jones, and a technical consultant with an infrastructure background, Stuart Nixon, to help us with the job. It went live at the start of the 14/15 academic year and we are now moving on to phase two of the project.
Integrating Turnitin
Turnitin is online software that compares submitted texts with texts that already exist on their databases. The primary purpose of the software is to find similarities between student texts and text that already exist in order to find cases of plagiarism.
At Sussex we already had a neat integration with Turnitin. We had a site in our LMS that students could use formatively. The purpose of which was to allow students to become familiar with the expectations within Higher Education with regards to referencing other people’s work, while avoiding plagiarising.
The Essay Checker site, a site available to all students for learning about academic standards and referencing
However, we did not have a policy for tutors to use Turnitin to check for the originality of students’ work. This project introduced that policy.
Students upload files to the Moodle assignment tool. On refreshing the Moodle page the student gets access to the report which shows which parts of the text Turnitin has detected matches. Students then get the opportunity to delete and reupload the file up until the due date for the assignment.
Student view having uploaded their file
When the due date is passed the tutor sees the uploaded file in Turnitin with the originality report. This report can help them find where and how students have used texts that were not their own.
Integrating Grademark
Grademark is software developed by Turnitin for online marking and giving students feedback. In some respects it is very powerful. It allows tutors to leave a grade for their students, leave general text or audio feedback, comment directly (in-line) on the student’s submission, and set up marking rubrics.
At Sussex we haven’t used it in earnest before, but the policy created in this project expects all first year text-based submissions in all departments and schools be marked using this system. Student submissions become available to tutors after the assignment due date through Moodle submission list, which links to Grademark where the tutor can mark them and leave feedback. The marks are returned to Moodle when the tutor selects to update marks.
Tutors’ list of submissions marked in Grademark (anonymised by candidate number)
Integrating our student records system
Much of the logic around summative assessments exists in the student record system (SRS), including who has permissions to mark, conflation of grades, which students have extensions and which have special circumstances. Trying to recreate this logic in Moodle seemed pointless, so with our colleagues, Martin Scolding and Karen Tiernan, we developed web services to reference the rules that were needed within the Assignment interface.
Submission dates and times were passed from the Moodle Assignment tool to the SRS. Grades too were pushed from Grademark, through Moodle into the SRS. The view of conflated grades were already available through Moodle. This integration had already been carried out in an earlier project.
Ensuring robustness, scalability and good performance
To ensure the smooth running of the service there are a number of technical details that are worth highlighting.
During assessment periods we are catering for times when there may be thousands of assignment submissions. This will require our server infrastructure to be robust at times of high load. The standard Moodle uploader uses the same Apache web server for serving web pages as it does to handle assignment uploads. This means the two compete for resources. We therefore decided to use Nginx to process submission uploads leaving the Web server free to process web pages.
The Moodle integration that we had with Turnitin used their old API. In order to ensure the long term future of our system we updated our integration. We have two separate automated cron jobs running in the background to retrieve the similarity report, one which sends the file to Turnitin and returns an identifier so it can be linked to, and another that returns a percentage similarity score.
We use web services (Glassfish) to retrieve data from our SRS rather than direct queries. This allows us to be confident that if the SRS changes its database structure it is the responsibility of the providers to ensure that the webservice still returns the same data. It also means that other services can use the service and they will get the same data. The disadvantage is that a web service has a slight hit on performance.
In order to reduce the amount of calls we made to the web service we use memcache technology to store the data. Depending on the substance of the data the time we store it varies from a minute to fifteen minutes.
Further improvements
In general we have a drive towards asynchronicity. This is the processing of multiple jobs at the same time. Stuart Nixon, our consultant, has put together this neat presentation (scroll down on the second slide). Although we haven’t implemented all the ideas in the presentation that is the direction we are going.
One idea we are in the process of implementing is the queuing of background jobs to send files to Turnitin and receive similarity scores. Submitting files to Turnitin scores can take up to 20 seconds. This means that on a day when we were expecting 4000 submissions it could take upto twenty four hours for them to be submitted to Turnitin and even longer for them to get a similarity report. We are therefore looking into replacing our automated crons with a queuing system that will be able to pick off jobs and run them in parallel.
Monitoring and error handling
Because of the criticality to students and institutions of assessments it was important that we had a good audit trail of what was happening. We therefore added more advanced logging than is possible in native Moodle. We stored all logs together including Apache logs, upload results and performance, Web service responses and Turnitin and Grademark processing. In order to query these logs we used an advanced logging interface tool called Kibana which allows us to query the logs quickly and easily.
Image showing Kibana
Lessons learnt
A lessons learnt blog post is definitely warranted for this project and we will keep you updated on how it is going. Maintaining a highly critical system has certainly had its challenges.
As always comments welcome.
One Comment