BLUE DOG EXPOSES POOR ONLINE ASSESSMENT METHODOLOGIES
Tuesday, 14 April 2009
Blue Dog Training is fast becoming recognised as a leader in online learning and assessment because of its phenomenal growth and high quality outcomes. At the recent Partners in Smart Training & Assessment Conference on the Gold Coast one of Blue Dog's Directors Tim Harris was asked to present a session on the varying quality of the technologies and methodologies used in online training.
Tim accepted the challenge by demonstrating and exposing 3 commonly used online assessment methodologies that are neither fair, valid nor reliable. He then showed 2 of the assessment technologies that Blue Dog uses that do satisfy the AQTF and common sense assessment requirements. He had ten minutes to do this.
Tim showed the audience that a traditional VET practitioner's assessment toolkit needs to be much broader and deeper to deal with the requirements of electronic and online assessment. Assessment needs to be more than just well written questions addressing knowledge and skill. It needs to cater to the available technologies and how they can be used not only by the assessor but also by the technologically savvy students of today. Online assessors now need to have an understanding of the technologies and use them to support valid assessment rather than as tools which invalidate the whole process. Without this understanding the technology can impact on the validity and fairness of the assessment which has been designed with the best of intentions.
Tim chose to demonstrate the following examples of online assessment where technology has made the process invalid and unfair.
'Certificate of Clicking'. This is an assessment tool that can have good and valid questions and learning content but the whole exercise is made totally invalid as all the student has to do is continue to click the "Next" button until he/she ends up on the Certificate page. To see an example of a "Certificate of Clicking" type assessment click here.
'Diploma of Persistence' is a slightly more advanced assessment tool than the Certificate of Clicking. This is a tool where good questions and activities are presented and when the user selects an incorrect response they are redirected to the question so that they can select/guess another response. This carries on until the correct response is selected. Tim showed that by about question 3 the students often do not even read the question, they just keep guessing and clicking until they select the correct response and move onto the next question. To see an example of a "Diploma of Persistence" type assessment click here.
'Degree in Luck' is another widely used online and intranet assessment tool. This is where a number of questions are selected from a data bank of questions; say 10 questions from a databank of 50 questions. Tim showed how this assessment methodology is flawed as some students by sheer luck may get all the easy questions or all the questions on one element or bad luck all the difficult questions. This assessment tool lacks in reliability and validity and therefore shouldn't be used as a valid summative assessment. To see an example of a "Degree in Luck" type assessment click here.
Tim did not want to just expose poor assessment examples without offering some examples of good assessment so he then demonstrated two of the assessment tools from the suite of assessment technologies Blue Dog has developed. These included technology to prevent just guessing or clicking by the student to gain success without the knowledge required to progress.
These tools are more complicated to create but the effort is worthwhile knowing that the tools are valid and are designed to assess for the student as opposed to other methodologies that are designed as assessment of the students.
There are many more assessment tools that are used online that can easily be manipulated to give invalid outcomes. Tim said that he believed that online assessment was here to stay but with the plethora of poor and invalid assessment technologies being currently used, he felt that VET practitioners need to be given the technological understanding to be able to identify them.