Tools and Evaluation Techniques to Support Social Awareness in CSCeL: The AVATAR - Lambropoulos Culwin

Editor's Notes

1. Tricot: external representation of learning
2. The first HCI-Ed stage is the understanding of the context, key concepts and tasks prior to their translation into design. The understanding of the context is attained by identifying issues within it and the key factors that are at play with the associated learning values.  
3. e.g. feeling of community the learner experiences (Tu & McIsaac, 2002), the degree to which a person is perceived as a ‘real person’ in a mediated communication (Gunawardena, 1995), Social presence theory is defined as the ‘degree of salience of the other person in the interaction and the consequent salience of the interpersonal relationships…’ (Short et al., 1976:65), Rafaeli (1990) argues that social presence is a subjective measure of the presence of others, while ‘interactivity’ is the actual quality of a communication sequence or context. salience-=projection, salient=outstanding (important, striking)
4. The second HCI-Ed stage is iterative design based on reflections upon processes and outputs from the previous stage. The design can be defined as an individual and collective activity, finalized by a project to develop a physical and symbolic artefact. The problem is rarely well defined.
5. In descriptive statistics , a quartile is any of the three values which divide the sorted data set into four equal parts, so that each part represents one fourth of the sampled population.
6. The fourth HCI-Ed stage involves building prototypes in order to acquire feedback from design. Exploratory prototypes are useful to allow different groups of users/learners to comment upon design. Such prototypes might be thrown away or might evolve towards a final version. For example, an initial version is useful for an early evaluation; and early access can be provided to different groups of users/learners as well as developers and e-tutors for community previews. Pre-release prototypes shown at community reviews will facilitate the final tweaking of the system.
7. The fifth HCI-Ed stage is the evaluation of system’s quality by measuring the degree to which the system meets its purpose as defined in the first two stages. Live evaluation of the system in use is of extreme importance. The avatars are divided into three regions: head, arms and legs, with a colour scheme indicating the levels of participation this was in an attempt to allow other users to see very quickly what kind of forum member they were communicating with. The baseline study was conducted on a Moodle server at the Greek School Network and this was used to determine the level of participation. There were 8 topics, 28 new discussions and 77 messages sent by 45 participants, 32 of whom were passive and 13 active, making an average of about 6 messages per poster. In the case study the prototypes of the various tools were tested. There were 56 participants and 5 e-tutors including the first author (Niki Lambropoulos). The majority (43) were Greek teachers and the rest (13) were from different parts of the world. In the tools evaluation discussion, there were 33 sent messages and 10 active users with an average of about 3 messages per poster. The comments were analysed and on the basis of these discussions, the tools were redesigned and tested in a focus group of 3 male Greek teachers who were also Moodle developers. None of them had participated in any of the previous studies. The case study and focus group active participants completed an online questionnaire to evaluate the tools. It used a 1-5 Likert scale and focused on efficiency, effectiveness and satisfaction (ISO DIS 9241-11, 1994), as well as enjoyability, learnability and imagination (Zaharias, 2004; Silius et al., 2003). Overall, the tools scored 4.2 out of 5 indicating that the participants were satisfied with the tools. The most important changes that came out of this fieldwork are described in the next section. Six participants were passive in the first study, 1 was a null passive participant, 4 were low passive participants and 1 was medium. Of these 6, 2 became active participants in the second study. However, 5 who were active became passive. There were 9 passive participants in the second study: 5 low, 2 medium and 2 high. Figure 7 shows an overview of the participation levels that appeared in the second study. The calculation for the second case study was based on P50, who posted 12 messages. There are 1 low, 5 medium and 4 high participation learners. The active participation data in the first case study were obtained from the system logs. The calculation for the first case study was based on the most active e-learner, P37, who posted 36 messages. Overall, there were 29 active participants in the first case study: 22 low active participants, 5 medium and 2 high. There were 26 active participants in the second case study: 16 low, 5 medium and 5 high. Figure 8 illustrates these results.
8. The final HCI-Ed stage is the study and research of all phenomena surrounding human-computer interactions. These can be supported by observing the context from different perspectives. Such multi-disciplinary approach is essential to effective HCI-Ed. Interface and contextual information are needed to facilitate the acquisition of structured and unstructured data in numeric, textual and visual formats to reveal converged patterns. Statistical analysis can work on numeric data from questionnaires and logs as well as quantified qualitative analysis. Qualitative analysis can work on textual data, open questions, focus groups and interviews; visual data can provide pictorial perspective of the e-learning phenomena.
9. New PU Q was developed.
10. Tricot: external representation of learning
11. The right methodology gives us the tools to investigate the research context and in fact, to suggest the criteria and metrics for internal and external evaluation and feedback. Ethnotechnology is a perspective rather than a methodology. This means that all approaches are legitimate to acquire relevant information from the situated context, in this case the Greek teachers. No presence indicated stillness and thus, change was needed. The techniques were associated with the aims, for example, an e-learning environment can provide data for human-human and human-computer interactions by quantitative analysis and grounded theory can be used for message analysis. The collaborative e-learning analytical framework has been already suggested for evaluation.
12. http://pagesperso-orange.fr/andre.tricot/Earli_Tricot_Slides.pdf Tricot, A., (2007). Utility, usability and acceptability: an ergonomic approach to the evaluation of external representations for learning. EARLI Symposium « Understanding the role of external representations in supporting learning » , Budapest, August 28 – September 1.