Maria João Loureiro

IODL & ICEM 2010, Anadolu University – Loureiro, Pombo and Moreira

The quality of peer assessment in a wiki based online context

CIDTFF, DE, Universidade de Aveiro

The quality of peer assessment in a wiki based online context:

a qualitative study in a post-graduation module

Maria João Loureiro




topics

context




context of the study - Portugal

AveiroOporto

Coimbra




context (U. Aveiro)• created in 1973• 10 to 11.000 students• courses (grad and pos-grad)

teachers education sciences, math, …engineering health…

• member of the European Consortium of Innovative Universities




contextPhD program in Multimedia in Education module

Distance Education (DE)

blearning regime face-to-face and at a distance group work

module aim - develop research competences related with literature reviewing

search, select, systematize, synthesize and literature related with DE;communication, collaborative work and assessment competences (self- and peer- assessment)




Context (literature)focus on learner self-directed and collaborative learning -> increase attention to alternative/authentic/integrated assessment methods HE assessment methods are quite narrow – assessment students’ outcomes test based – students are not implicatedpeer assessment (PA) can empower students (critical thinking, reflection)reliability and validity of PA – no consensusliterature on the quality of online (web 2.0) peer assessment scarce

Boud and Falchikov (2007), Joordens et al. (2009), Li et al. (2009, 2010), Peng (2010), Rourke et al. (2008), Sluijsmans et al. (2004), Topping (2000, 2008, 2010),

Zundert et al. (2010)




Context (literature)PA benefits:

promote higher order thinking, developing students’ critical thinking, communication, lifelong learning, problem solving… increase the amount of feedback, from teachers and from peersenhance students’ sense of ownership, responsibility and students’ motivation, since they find it useful, attractive and enjoyablesupport active and autonomous learning can avoid the involvement of free-riders in group work, i.e, facilitate the identification of individual contributions




Context (literature)PA pitfalls:

students’ attitudes toward PA may hinder the above mentioned benefits –students’ unconfident towards PA - judging friends – increase students’ anxiety and resistant toward PAPA is time-consuming – training, preparation, and monitoring are neededPA can be subjective – students might be too hard, critical or boasting, or not used to assess their peers

PA advantages increase – formative feedback




course organisation and development 1st week online- Diagnosis of students’ perceptions about how to do literature

review and about “Distance Education” (individual).- Familiarisation with the module guide and discussion of the

proposed activities, including assessment (individual).- Individual readings of recommended literature and or other

materials related with theoretical and methodological issues to be taken into account to do a literature review.

- Use of the synthesis sheet – instrument developed to help students to analyse and document their readings (individual).




course organisation and development 1st face-to-face session - Teacher, tutor and students’ presentation and definition of work

groups (classroom). - Negotiation of the module activities and schedule and of the

assessment framework (classroom).- Session about online literature search tools (made by a

library technician).- Presentation of tips on how to do a literature review (teach).- Discussion and syntheses of the students’ readings about how to

do a literature review (groups).- Information search (groups).




course organisation and development 2nd weekonline- 1st ind.reflection about the ongoing activities and students’

learning- Definition of the literature review themes related with DE and of

the work plan (groups)- Formative assessment of the work plan (teachers)3rd and 4th weeksonline- 2nd ind.reflection about the activities and the developed

competences- Development of the 1st version of the literature review (gr.)- Formative assessment of the ongoing work (teachers and PA) - Revision of the paper and preparation for its presentation

(groups).




course organisation and development 2nd face-to-face session- Presentation, discussion and assessment (teachers

and peer summative assessment) of the groups work.- Self and peer assessment of the developed

collaborative competences (intra group) and final reflection concerning students own learning during the module (individual)

- Module assessment (individual).




MethodologyQualitative approach - study recent phenomena (like the quality of online PA) in their own contexts Data gathering and analysis techniques:

observation mediated by the wiki (PA was delivered by the students using the wiki) – content analysis (2 researchers, reliability 100%, external validation) inquiry (a questionnaire was applied to evaluate the module in the final face to face session) – descriptive statistics




Criteria Indicators

Use of the negotiated criteria

Students used the negotiated criteria or words with the same significance explicitly.

Adequacy of the chosen vocabulary

Concepts are used with rigour and differentiated.

Provision of constructive feedback

criticismsPA text includes criticism (both positive and negative), pointing out aspects that could be improved, without making suggestions.

questionsPA text presents questions for reflection concerning, for example, the theme or the development of the literature review.

suggestions for improvement

PA text includes suggestions that can lead to the improvement of the literature review, like, new readings, proposing ways to systematize the corpus of the review....

Fairness of the score the PA includes a score which should be in accordance with the given feedback

Adequacy of the final considerations

A synthesis of the main positive and negative aspects of the review is presented.




Examplesnegotiated criteria

“(…) the topics and sub-topics that underlie the work are well framed [i.e. the structure is adequate]. Its coherence and rigour are remarkable and the group demonstrates synthesis competences”.

adequacy of the chosen vocabulary “the paper does not present significant gaps in what concerns cohesion, textual coherence or even orthographic or syntactic mistakes.”

constructive feedback “the work does not include the research question that guides the development of the work”




%0 20 40 60 80 100

Use of the negotiated criteria

Adequacy of the chosen vocabulary

Provision of constructive feedback(criticisms)

Provision of constructive feedback(questions)

Provision of constructive feedback(suggestions for improvement)

Fairness of the score

Adequacy of the final considerations




0 2 4 6 8 10 12 14 16

Col. work should be assessed as a whole with the samemark for all elements

Col. work should be assessed with several components

Col. work should be assessed individually, by teachers andby colleagues

Col. work should be assessed only by teachers

The personal reflections should be published in themodule's site

Self and peer assessment of the work groups should bepublished in the module's site

Self and peer assessment of develloped competences(intra-groups) should be published in the module's site

Number of students

completely disagree disagree agree completely agree don't know




Final thoughtsPA could have more quality (suggestions, questions, …)Results

In line with the literature – lack of quality can be explained by some of the pitfalls of PA (students’ attitudes towards PA, uncomfortable, lack of confidence, unreliability + fear of being exposed in the social web – in this case while agreement about publishing PA)

PA needs training




Contributions framework to assess the quality of PA in web 2.0 online contexts, for some of the criteria, quality PA can be achieved reports students’ perceptions and attitudes towards the assessment of collaborative work, including PA




Thank you for your attention

questions???




coursesorganization

Ramos (2004)

Documents

Maria João Loureiro