Take a look at the limitations section and say a little more there to clarify the idea
finish by 4PM CDT
A CASE STUDY IN PARAGOGICAL EVALUATION
The paragogy principles provide guidelines on best practices for building suc-
cessful peer learning experiences. In this section we will apply these principles to
evaluate the lessons learned from our work at P2PU as facilitators in 2010 & 2011.
For each of the principles we run through an After Action Review to look at
how well the principle was implemented.
CHANGING CONTEXT AS A DECENTERED CENTER
:: Mapping system dynamics and semantics
1. Review what was supposed to happen.
We both organized multiple courses: Collaborative Lesson Planning Fall 2010 and
Winter 2011 (co-organized with Dr. Majorie King); DIY Math; Math for Game Designers; Open Governance and Learning (co-organized with Marisa Ponti); and Shaping P2PU. Participants were supposed to interact and learn about the subject matter (footnote to one of our syllabi?).
2. Establish what happened.
Due to critically low participation, the mathematics courses did not run to
completion. Participation in the other courses was minimal, but suficient
for them to run the entire 6 week session. In an effort to understand these
events, the theory of paragogy was born.
3. Determine what was right or wrong with what happened.
In the more active courses, there were nice examples of learning by course
participants.11 This was common across P2PU exemplified in Dan Diebolt's graphical analysis of coure participation, which showed that participation was generally uneven and falling.12
4. Determine how the task should be done differently the next time.
Our best course experiences happened when we were planning to work
through the material ourselves no matter what, which was not the case in the more experimental courses. A blend between organizer commitment to the material, and some gentle prompting to follow through on commitments can go a long way.The P2PU ecology contains an implicit rubric for learning; it would be good to make this explicit: from member signs-up for a course to its completion, peers go through a cycle. This should be possible to evaluate for quality. Footnote: (i.e. http://open-governance-and-learning.posterous.com/implementing-paragogy-further-reflections)
https://wiki.mozilla.org/Drumbeat/SoW-engagement-ladder
https://wiki.mozilla.org/index.php?title=Drumbeat/SoW-engagement-ladder&oldid=305791
P2PU could then help by implementing more check points throughout the cycle.
METALEARNING IS A FONT OF KNOWLEDGE
:: Transparency, accountability, and tone
1. Review what was supposed to happen.
Support for community members was offered as a P2PU course, in mailing
lists, via weekly phone calls, in a Q&A issue tracker, and via a few other
channels. Participants in courses were hoped to learn how to contribute in a
useful fashion if they did not know already.
2. Establish what happened.
Core members do hold themselves accountable, but this behavior is not nec-
essarily transferred or communicated to new members, for whom account-
ability is low.
3. Determine what was right or wrong with what happened.
People "at the top" are doing a lot of work, keeping the project moving
forward. To date, however, community members have no "formal" account-
ability to one another.
4. Determine how the task should be done differently the next time.
It is typical for online communities to have strictly enforced community
norms. A nice goal for P2PU would be to create and distribute some well-
defined OER that discusses these, along with other "best practices" informa-
tion for organizers and participants. The current Course Design Handbook is one starting point, but it is not yet a sufficiently clear guide.13 Learning how to learn (and how to support others in their learning) is no simple task. It would be good if there were better ways to share information about what to expect.
peers provide feedback that wouldn't be there otherwise. (diff from heutagogy, change globally)
:: Dealing with problems in a respectful way
1. Review what was supposed to happen.
Discussions about P2PU happen in the community mailing list and other
places mentioned above; and bug reports should of been sent to the Lighthouse tracker.
2. Establish what happened.
Discussions happend in many places including and beyond the mailing list (e.g. in courses) to the point it was difficult to keep track of. There was some talk about using the Lighthouse tracker for organizational matters, but that hasn't taken off. Earlier experiments with using a different tracker or a spreadsheet to keep track of organization-level tasks were undersubscribed.
3. Determine what was right or wrong with what happened.
Apart from development work, it is hard to tell what's happening around
P2PU. Presumably participants who have identified critical and unsolvable
problems simply leave.
4. Determine how the task should be done differently the next time.
In a traditional university, there are typically a lot of ways to discuss and resolve prob-
lems without dropping out. P2PU is working on a peer support model. Footnote: new helpdesk.
LEARNING IS DISTRIBUTED AND NONLINEAR
:: Design considerations
1. Review what was supposed to happen.
People are supposed to choose and assemble suitable learning resources
(blogs, OER, etc.) for their courses, in which everyone is supposed to learn
something.
2. Establish what happened.
This is essentially what happened, but it is hard to measure when and
whether knowledge was gained.
3. Determine what was right or wrong with what happened.
The organization is striving to handle the complexity of life online. The sys-
tem is explicitly in an experimental "beta" stage, and people who participate
in betas are guinea pigs (and should know this). Quality control has a some-
what precarious meaning in a beta or "eternal beta", but this makes life
interesting.
13 http://wiki.p2pu.org/w/page/27905271/Course-Design-Handbook
14 http://p2pu.lighthouseapp.com/dashboard
4. Determine how the task should be done differently the next time.
In terms of measuring learning, P2PU would have to work hard to use any-
thing but "participation" as a proxy value. In terms of broader issues of
quality control, one thought is for P2PU core members to "eat their own
dogfood" by using the platform to organize their activities. Indeed,
everyone involved with the project could use the platform to measure
their "stetch/churn". Footnote: http://calnewport.com/blog/2010/03/15/how-to-become-a-star-grad-student-james-mclurkin-and-the-power-of-stretch-churn/
REALIZE THE DREAM IF YOU CAN, THEN WAKE UP
:: High level roadmap
1. Review what was supposed to happen.
At one time, the high-level vision was arguably a Declaration of Indepen-
dence from Formal Education.15 But perhaps each participant has their own
vision.
2. Establish what happened.
At the time of this writing, the P2PU organization is just convening its first
board meeting: perhaps that is where the high-level vision and roadmap
will be set. For the moment, it's not clear whether the vision has shifted,
digressed, or stayed right on target.
3. Determine what was right or wrong with what happened.
There hasn't been clear communication about progress.
4. Determine how the task should be done differently the next time.
P2PU should build an explicit public roadmap that leads from now up to
the point where the vision is achieved. They should do regular or ongoing
open/public assessments of quality, and refine or adjust the vision as needed.
Don't grow longer than 150.
> Would be interested to see your paper expand more on the underlying factors
> of "critically low participation" (p. 6) in courses.
Just below that, the paper says:
"Our best course experiences happened when we were planning to work
through the material ourselves no matter what. In the future, maybe we
need a system where someone who doesn't show up gets a personal phone
call asking them what happened."
Both "Collaborative Lesson Planning" and "Open Governance and
Learning" had high commitment from the organizers to the subject
matter. Participation wasn't enormous, but there was enough to keep
things rolling. And in Collaborative Lesson Planning, although
Charlie didn't call my phone, he did send me personal emails when I
was late with assignments.
By contrast, the mathematics courses I mainly put together in the
spirit of experimentation. For example, as fun as "Math for Game
Designers" might have been *if* others were engaged, I wasn't
committed to go through the course material on my own ("no matter
what").
A blend between organizer commitment to the material, and some gentle
peer guidance go a long way. I would say these points bear a
similarity to our Principle 5 and Principle 3, by the way.
P2PU could help e.g. by implementing more paragogical check points
along the way through course development, and by making it easier for
everyone to state publicly what their commitments are and what will
help keep them accountable.