New Opportunities in Assessment: A P2PU-Hosted Workshop
Please Note!
We are moving this page to the P2PU wiki.
Please make any future edits at http://wiki.p2pu.org/assessment-workshop
September 20-21 (Mon-Tue), 2010
Carnegie Foundation for the Advancement of Teaching
Palo Alto, CA.
Peer 2 Peer University, in collaboration with Aspiration, the Mozilla Foundation, the Carnegie Foundation, and the Shuttleworth Foundation, is organizing a workshop about Assessment on the Open Web. Specifically, we are focusing on new opportunities and models for authentic assessment in P2PU and other virtual learning communities.
"Authentic assessment" can be understood as observation and representation of an individual's performance of real-world tasks that demonstrate meaningful application of essential knowledge and skills.
The growth of the Internet and new models for peer learning offer the opportunity to construct and evaluate new systems of authentic assessment. Indeed, there are already novel and interesting systems of peer-review, reputation, accreditation, and evaluation that arose because of the new capacities for social interaction on the open web (the so-called web 2.0 ecosystem). For example, there are the rating schemes on Amazon, the reputation systems on Stack Overflow, or the aggregated recommendations that people collect on LinkedIn. People converse, collaborate, create, and share at a scale and with an ease that was previously unimaginable. However, many of these tools are not overtly intended for education.
Thus the question now is: what affordances do these new capacities provide for learning assessment?
Overarching goals of the workshop:
We would like to develop two or three fleshed-out plans for action which: a) assess specific skills of interest, b) leverage existing Internet-based systems for reputation-building and assessment, and c) show promise for implementation in the peer-oriented P2PU environment as well as other virtual learning communities.
Possible workshop activities and outcomes:
- Understanding the environment: Annotated review of different Internet-based systems for reputation-building and assessment, with special consideration of their potential for use in P2PU and other virtual community learning environments.
- Identifying relevant skills: Detailed examination of mechanisms for identification of desirable skills, as determined for example by online and direct surveys in communities of interest, or textual analysis of job adverts and recommendations.
- Assessing relevant skills: Identification of possible mechanisms for evaluating the skills in a manner which capitalizes on existing tools and the peer-oriented P2PU environment as well as other virtual learning communities.
- Web 2.0 research meets assessment: Pathways to recruitment of collaborators from within and beyond the educational research field who are interested in further research, development, and evaluation of these systems.
Additional possible activities:
- Building a prototype: Specific recommendations for analyzing the efficacy of the existing assessments of key skills which are embedded in the Fall suite of P2PU School of Webcraft courses.
- We could also look ahead to future semesters and develop templates for identifying relevant skills, implementing key assessments, and evaluating their utility and impact.
- These efforts are precursors to conversations about accreditation, including badge systems, personal portfolios, and so on. We may have time to consider the connections between assessment and accreditation for online learning environments.
- Though our focus here is on online learning, we believe that the issues at hand transcend learning contexts, and we might be able to consider ways of building formal/informal or physical/virtual collaborations among people and institutions.
Tentative Agenda
Day 1:
- Introductions. Background on P2PU and virtual environments for learning generally. Background on web-based systems for assessment, reputation-building, and accreditation, including pilot efforts within P2PU.
- Breakouts focused on framing authentic assessment and open accreditation in the context of virtual learning environments.
- Evaluation of some subset of desirable skills which may be amenable to different forms of assessment. One option is to apply methods of backwards course design within the current structure of P2PU to determine promising assessment options which provide visible evidence of competency gains.
- Repeat this process in breakout groups for other domains of interest.
Day 2:
- Revisit determinations made the day before (organized overnight by convening staff). Focus on the challenges of implementation, with special attention to the P2PU system.
- Identify key principles for implementation of suggested authentic assessments in virtual learning environments. What additional tools or resources might be needed? Can we assemble a robust collection of web-based resources which allow students to operate effective personal learning environments, or do additional features need to be added to P2PU (or other existing platforms)?
- Mock up promising assessment systems as they might be manifest in P2PU. Consider inputs and outputs, and map on the research designs (to be able to evaluate the effectiveness of the assessment models). Identify questions and methods of interest and personnel who want to assume leadership roles.
- Final thoughts, reflections, and basic plans for follow-up conversations and workshops.
Help us build the agenda
Our ask for potential participants / Send us some information:
- What are some of the new tools and mechanisms that are emerging on the open web, which are not currently conceptualized in the context of assessing learning?
- What are things that we weren't able to assess in the past (or unwilling) - the so called soft skills - that we might be able to assess now with these new techniques?
Organization
- Overall facilitation, organization and logistics - Aspiration, Allen Gunn
- Content and substance curation - Ahrash Bissell, Philipp Schmidt
Readings / Research
Tools
More background
Peer to Peer University (P2PU) is a system that builds on the new interactive capacities of the Internet by explicitly fostering a social environment for learning. It stands to reason that P2PU should be able to capitalize on the reputation and assessment systems that have already been created and tested in other domains of the Internet. However, there is much we do not know, and there remains a significant need to research and compile options for alternative assessment and accreditation. In considering these options, we need to be mindful of both the risks and benefits of utilizing different approaches for learning, the specific competencies and metrics of interest, plans of action for adoption and endorsement, and suitable early-adopter disciplines and communities. This work is necessary to contextualize the choices made for P2PU going forward, and we hope the outcomes will be useful to virtual learning environments generally.
This workshop represents one component of a burgeoning research effort focused on learning opportunities and outcomes in virtual environments. For example, a major benefit of open accreditation is the potential to capture data from the (formative) evaluation process which can inform us as to the validity and reliability of different types of assessments for different needs. In this manner, authentic assessments can be developed and implemented in appropriate courses such that competency-based open accreditations fall out naturally from the course structures. It seems likely that some subset of the suite of evaluations oriented towards open accreditation will be superior to existing accreditation processes, especially for "soft" skills. We imagine that future work will include reports on how alternative credentials via an open accreditation system can extend beyond an immediate community of learners to signal competence, preparedness, or literacy within new contexts of engagement. In addition, this workshop and work to follow is intended to set the stage for both focused discussions on further developing assessment models in P2PU, as well as broader discussions around open accreditation and authentic assessment generally.
Email invitation text
Dear _________,
Peer 2 Peer University, in collaboration with the Mozilla, Carnegie, and Shuttleworth Foundations, is organizing a workshop about Assessment on the Open Web. I am writing to find out if you might be interested in participating.
Your work on ________ seems highly relevant to what we plan to discuss. By way of background, here is some text copied from our planning doc:
"The growth of the Internet and new models for peer learning offer the opportunity to construct and evaluate new systems of authentic assessment. Indeed, there are already novel and interesting systems of peer-review, reputation, accreditation, and evaluation that arose because of the new capacities for social interaction on the open web (the so-called web 2.0 ecosystem). For example, there are the rating schemes on Amazon, the reputation systems on Stack Overflow, or the aggregated recommendations that people collect on LinkedIn. People converse, collaborate, create, and share at a scale and with an ease that was previously unimaginable. However, many of these tools are not overtly intended for education.
Thus the question now is: what affordances do these new interactive capacities provide for learning assessment and alternative accreditation?"
The meeting is slated for Sep 20-21 (Mon-Tue) at the Carnegie Foundation for the Advancement of Teaching in California. We would pay for travel, lodging, food, but do not have any money for stipends.
We are most interested in adapting existing systems for online interaction in the service of online assessment and learning. While we believe these things can be further adapted for offline use, that is not the primary focus at this time. However, we are expecting much of this work to ultimately get parlayed into formal classroom settings, and we are interested in working with anyone doing so. Also, while there are good online assessment tools available, these are mostly modeled on traditional assessment methods, and we are explicitly focusing on non-traditional models for this workshop. And the focus will definitely be on Peer-to-Peer University (P2PU), rather than on online instruction writ large. Ultimately the goal is for P2PU to be somewhat representative of virtual learning communities generally, but to ground this two-day meeting we thought we should keep it focused. Finally, we are hoping that participants will take an interest in further research and testing of the ideas that emerge from the workshop over the longer term, helping to both build P2PU as a working platform but also utilizing P2PU itself as a research environment.
Any comments or questions welcome! Feel free to reach me at _________ with questions. Because the timeframe for meeting planning is short, it would be fantastic if you could send word yeah or nay as soon as possible.
Best,
___________