P2PU Community Call
7 February 2013
Seminar Discussion
Topic: What does success look like at P2PU?
Agenda
**We are hanging out here: https://plus.google.com/hangouts/_/412966ce2a10d71698bf7bc91c025854dc33477e?authuser=0&eid=110535027738331018997&hl=en-GB - join us!**
Attendees
- Vanessa (vanessa@p2pu.org)
- Jane (janepark@p2pu.org)
- About to share it, Jane, don't panic! - see link above
- Philipp (philipp@p2pu.org)
- Kevin (@ubuntourist)
- Bekka (bekka@p2pu.org)
- Dirk (@riskycud)
- Piet (@bagabot)
- Chris - New Color!
- Leah (@leahmacvie)
- Billy (@billymeinke)
Report Back from Staff (written notes, with time for questions/brief discussion on call)
- Progress
- Badges UX
- Personal Learning Plan--woot
- Assessment Manifesto
- Personal learning plan blog post
- Metrics for end of jan newsletter were higher than average (for us and for industry) YAY--can we see these?
- Media Lab Course
- 16,000 users signed-up (~600 problems)
- 37,000 visits yesterday (most ever) - Mainly from Lifehacker post
- Shuttleworth Gathering will be at Media Lab (hosted by yours truly, but "hosted" doesn't mean much)
- Release!!
- Server upgrade
- New course UX
- CSS framework
- SOO will launch during Open Ed Week! Are you guys ready??? - updated list of courses/challenges @ http://schoolofopen.org
- Jane: can you please be in contact with Susan Topol at Open.Michigan about communication/promotion stuff around School of Open? (stopol@umich.edu) - for sure, Piet?
- Priorities
- School of Badge* program
- School of Data regroup
- Notifications for badges
- Iridescent Courses Review
- Media Lab Course - make 100% certain our "experiment" is robust enough for 16,000(VMG has questions)
- 35 thousand people looked at the site on the 6th of feb.
- VMG: What are the goals / metrics for traffic? How would you conceive of this in terms of Media Lab metrics.
- PS: I think this is good.
- Sysadmin: Setting up backups for p2pu.org
- Putting out fires on p2pu.org after the release
- SOO launch in March
- Process
- VMG is doing her performance review
- Problems
- Ideas/Other
- Some media - Lifehacker, Make blog
- Philipp's keynote in Germany (for those of us learning German) - http://t.co/KyOBgBPJ
Seminar Discussion
"What Does Success Look Like at P2PU?"
In the non-profit world, the measures of success are different to those in the for-profit world.
But non-profits can't fix the entire world. So how do we decide what our measures of success are?
We aim for: (What are the right high-level metrics/markers of success/ (no more than 3) for P2PU as an idea?)
- Impact - We reach people with peer learning
- I think impact should have a
long taill--longitudinal--not just hits or signups but completion, satisfaction, efficacy of projects, etc - With impact, I would more want to know what happened after a course/experiment finished. Signups feels more like it's measuring hype.
- We learn something new (that helps the field) - New idea, Uniqueness, Learning
- Is this one right for us? What's the right title?
- Magic - We created epiphanies and enabled serendipity+1
these sound like organizational goals... at P2PU we: 1) facilitate peer learning all over the world (impact); 2) create unique learning experiences from which we build and share new knowledge (learning/uniqueness/punk); 3) enable learners to create epiphanies and experience serendipity (magic)
- from these goals we can derive specific objectives the organization has within each area. for example, try answering the questions: how does P2PU facilitate peer learning all over the world? what does P2PU *do* to facilitate peer learning? this is also a good point to bring in the values of the organization. the values are less measureable as "things" but they can be used to guide the language of the objectives and the following metrics (next step)
- once you've got an idea of the What and the How within each goal, move on to the "prove it" stage. these are where you talk about metrics and measuring things. for example, if one of the ways P2PU facilitates peer learning all over the world is through development of its Lernanta platform, what sorts of things can we measure about Lernanta? (# of code releases, # of usability tests, # of contributors to the code base, etc.) all of these measures will be proxies for measuring the immeasurable: facilitating peer learning. they can be refined over time.
- remember that when you set out to actually measure things, start with the data you have and see how that can be used to help tell your story. figure out what data you have, then figure out what extra data you'll need; from there it's an iterative process to get better data that's more closely aligned with the goals and objectives of the organization
- other things that can be done: create high-level "indicators" which are essentially aggregates of measurements that give you a quick indication of whether or not the organization is on target with its goals; map all projects and initiatives to the objectives -- remember that most projects don't fit nicely into a single goal or objective, but it's helpful to know how each project is contributing to the overall goals of the org
Questions:
- Why are we measuring? Who are we doing this for?
- Ourselves: It is guiding principals
- it helps us know when to drop something or say "no" to a project or task
- We need to know how we are doing
- We need to know what we're doing too.
- Because at the moment, when we do stuff that does succeed, it feels like a surprise
- This is to some degree the nature of innovation though, you want to experiment a lot without always knowing which idea will succeed
- OER research hub @Open University - so OER community, they want to measure impact of OER - so we're going to have to start measuring SOMETHING
- For our funders - current and future+1
- for prospective participants/learners? (Can't this be used as advertising?)
- Doesn't it just matter that the people "are having a good time and learning stuff"?
- PS: That still implies some form of "measuring" what that means / it just doesn't spell it out
- Find it difficult to think about this stuff, because I don't understand the perspective from which things get measured
- how do we tie these to the mission and values of P2PU? https://p2pu.org/en/pages/about/
- one idea is to use the three high-level "metrics" above and within each "metric" we measure things related to the values of P2PU
- Mission: The Peer 2 Peer University is a grassroots open education project that organizes learning outside of institutional walls and gives learners recognition for their achievements. P2PU creates a model for lifelong learning alongside traditional formal higher education. Leveraging the internet and educational materials openly available online, P2PU enables high-quality low-cost education opportunities. P2PU - learning for everyone, by everyone about almost anything.
- Values: Open, Peer Learning, Community
- Open (how do we define open in our context?)
- everyone can participate
- content is "open"
- model and tech are "open"
- processes are "open"
- possible metrics (hard to measure): reuse of "open" materials? growth of community contributions? references to our materials? republication/replication (falls under reuse), projects that spin off outside platform (eg. MIT MediaLab) (sort of also measures innovation resulting?)
- Community
- governance involves everyone
- volunteer participation all over the place
- cordial (civil, tolerant, respectful, etc.) group of individuals
- strive for quality as a group, helping each other along the way
- possible metrics: list activity/participation, participation in weekly calls, # of courses being run, # of active platform users, # of newsletter click-throughs
- Peer Learning
- peers teach and learn
- everyone contributes something, everyone walks away with something
- individuals take ownership of their learning and support each other
- possible metrics: # of participants and retention, # of courses created, # of courses being actively run
Funder/Project specific metrics
Examples - Let's apply them to past projects (groupwork):
- Projects (let's not do all):
- Hewlett assessment work
- Unique - Very unique, when we started, nobody was talking about badges yet, and while people understood that evaluating deeper learning (and deeper learning type skills) was important, nobody was doing it. And an assessment framework still doesn't exist.
- Impact -
- How well/ how much is our framework translated into actual assessment/feedback? - Brainspace
- How many learners have we reached with it? - Direct
- How many badges? What type of badges?
- -->sure! shouldn't we also define the quality of the badges (ie number of badges that are user-generated vs. P2PU certified / awarded to others- how would you describe "quality"? Trying to do so now--I would say that the P2PU badge certification process would be one metric, another would be how many feedback "volleys" we have been experts and learners, or attempts at the badge
- Magic
- This feels harder
- "When assessment is feedback, there might be a higher chance for magic?"
- Someone getting a job with a badge would qualify as magic (also impact, but pretty cool impact)
- Python MOOC
- Uniqueness--very unique(ha! what does "very" mean though? "how" is it unique?) it was a critique/response to the large-scale MOOCs out there
- Super unique: only used OER, leveraged existing resources and services, cost almost nothing to build ... runs perpetually
- Magic--not a ton of magic here--we didn't facilitate a lot of peer interactions
- I think there was a bit of magic in the openstudy community
- Impact - More than 5000 signed up for the first round
- Not sure where this fits - Magic maybe? - Part of of our goal with the groups is to maximize "engagement" or how frequenty they discuss
- Where does the "long tail" of the project fit? We don't really have a sense of success stories / summative evaluation (I think long tail is slightly misleading - to me it means that within the same category of things, some attract a lot of attention and other similar things attract less attention - and that there are more of those things that attract less attention)
- Long tail might be the flu talking--but I think more longitudinal results is more the texture of what I mean (as long as it starts with long?) ;-) Longitudinal makes more sense to me.Youre hysterical
- School of Webcraft
- Impact:
- lots of people completed to some level
- Mozilla still refer people to the challenges
- Uniqueness:
- Not totally unique - other places offered similar things
- BUT the peer-learning aspect was unique, and the freeness too, and the fun/quirkiness (at the time)
- Magic
- Mozilla certifed badges for this? That's definitely a win. Yes, big appeal for ppl, also part of uniqueness.
- Weren't around to experience the real magic - but I sense the type of magic you would feel in an abandoned haunted house now :-)
- Totally. Although I think it was different at the time. At the time, the magic was very cool. You might even say it was magical...
- School of Webcraft badges
- Lernanta (course platform)
- Impact
- More than 2000 daily visits
- 500000+ unique visitors since launch
- Uniqeness
- Magic
- Don't know - would really like to hear thoughts from other people?
- for lernanta we shun magic - magic is that stuff that happens in the Django ORM - we don't like too much of that :-)
- Current projects
- School of Open
- Unique - not many projects attempting to encompass how open applies across the spectrum.. more niche projects focused on a domain
- Impact - dunno yet in progress
- Magic - everything is aimed to run openly at the same time... still in progress
- possible measures
- sign-ups vs
- active participants
- vs participants retained (# of participants remaining at end of courses)
- # of courses actively run vs courses actively participated in vs challenges existing
- kinds of "open" topics
- # of participants at each week who complete the assignments, join calls, etc.
- # of badges earned (and according to badge) once badges are installed (this won't happen for first iteration)
- qualitative reviews and interviews with course participants, where they reflect on the impact they perceive it has had, initiatives they then influence, etc. - eg. power of open book on CC
- # of volunteers (individuals and organizations)
- other impact measures: press mentions, countries participants are in, social media mentions
- community list activity, community list sign-ups
- long-term benchmarks
- incorporation of courses or "open" education into institutions and schools
- outcomes of specific courses, eg. open policy course resulting in open policies implemented at X number of orgs/inst's/gov'ts
- more people involved in "open" generally? way too vague..
- External projects as guides
Questions to ask:
- Do projects that "feel" like success, check out as success according to the metrics?
- Do interesting stories arise from the things we do?
Map out metrics for existing/ future projects
CHAT TRANSCRIPT FROM GOOGLE HANGOUT
You invited people to the hangout.
Kevin Cole joined group chat.
me
15:57
Hi kevin - welcome
We'll get going in just a few minutes
Kevin Cole
15:57
I need two monitors. Can't watch the video and the etherpad at the same time.
me
15:58
I know - it's a lot of hopping around!
Just a matter of protocol - could you mute your mic when not speaking ? We get a lot of background noise once everyone has joined!
Vanessa Gennarelli joined group chat.
Dirk Uys joined group chat.
Vanessa Gennarelli
15:58
bonjour!
me
15:58
VMG!
Vanessa Gennarelli
15:58
C'est moi!
Kevin Cole
15:58
Semi-solution: If someone "participates" from two computers and uses one to screenshare the etherpad window.
Pieter Kleymeer joined group chat.
Pieter Kleymeer
15:59
I see EVERYBODY.
Vanessa Gennarelli left group chat.
Vanessa Gennarelli joined group chat.
Vanessa Gennarelli left group chat.
Vanessa Gennarelli joined group chat.
Vanessa Gennarelli left group chat.
Jane Park joined group chat.
Vanessa Gennarelli joined group chat.
Jane Park left group chat.
Chris Ewald joined group chat.
Jane Park joined group chat.
Kevin Cole
16:01
(I believe Hangouts is supposed to auto-mute with multiple speakers.)
Leah MacVie joined group chat.
Philipp Schmidt joined group chat.
Philipp Schmidt
16:03
9 people - cool!
Chris Ewald
16:03
almost the max
Vanessa Gennarelli
16:04
Living in mute
Leah MacVie
16:04
Me too!!
Philipp Schmidt
16:04
We should cats on youtube
be cats
silent
Vanessa Gennarelli
16:04
word
Chris Ewald
16:04
let's hold up pictures
Vanessa Gennarelli
16:05
han
hand
Leah MacVie
16:05
Allucidate...?haha
Spell that!
Vanessa Gennarelli
16:05
elucidate
Leah MacVie
16:06
I'm using it in my next paper. I'll look so smart...like Vanessa!
YES LOVE LOVE LOVE the newsletter
Vanessa Gennarelli left group chat.
Vanessa Gennarelli joined group chat.
Vanessa Gennarelli
16:07
googlehangout hates me
Philipp Schmidt
16:07
hand
Vanessa Gennarelli
16:09
BOOYAH
is it?
hand
Dirk Uys
16:09
we need a P2PU map
Vanessa Gennarelli
16:09
OLIVER
thats superlative
Pieter Kleymeer
16:10
I'm scared the course will take over my life.
pro tip: don't use Google Docs for group organizing...
Kevin Cole
16:11
I was one of those who got no automatic confirmation w/ invite code. Then I got TWO invitation codes.
Philipp Schmidt
16:11
Hey Kevin - If you got two, it means you can participate twice.
Kevin Cole
16:11
Vanessa Gennarelli
16:12
lol
Jane Park
16:12
@piet - me too. i have yet to click on the join small groiup link
me
16:12
Wow. Dazzling.
Jane Park
16:12
is @p2pu bekka?
me
16:12
YUP!
Kevin Cole
16:13
Some of the response may be due simply to name recognition. "Creativity" and "inventor of Scratch" go together nicely.
me
16:13
Well, more than me. I just try not to fall over most days
Leah MacVie
16:15
haha- wake up P! Take notes!
Jane Park
16:16
i cant hear antyhing now
Kevin Cole
16:16
Alumni stories of success? "What interesting have you done as a result of course X?" "Have you passed the knowledge onto others?"
me
16:17
@Jane - can you hear anything now?
Billy Meinke joined group chat.
Dirk Uys
16:21
hey billy
Billy Meinke
16:21
hey guys, sry I'm late (traffic)
Philipp Schmidt
16:21
conversation?
are those three "right" for us?
Vanessa Gennarelli
16:22
disgree
Vanessa Gennarelli left group chat.
Vanessa Gennarelli joined group chat.
Leah MacVie
16:24
OK I'm back- I was gone, but I don't think anyone noticed
Hi Billy!
Vanessa Gennarelli left group chat.
Jane Park
16:24
+1 vanessa
Leah MacVie
16:24
Poor Vanessa!
Jane Park
16:24
have some benchmarks to aim for
Billy Meinke
16:24
hey Leah
Philipp Schmidt
16:24
We should be aiming to have reliable Internet at home?
Jane Park
16:24
even if they're the wrong ones
Vanessa Gennarelli joined group chat.
Jane Park
16:25
cuz then we can reevaluate
and change them as we go along
Leah MacVie
16:25
Hi VMG!
Jane Park
16:25
success!
Billy Meinke
16:25
+1 on benchmarks
Leah MacVie
16:25
Planning=good
Chris Ewald
16:25
for me: these metrics are guiding principals that we will design for
*end up designing for
Jane Park
16:26
participants that dont fall into pit of attrition
Vanessa Gennarelli
16:26
exactly
id like to be able to say that peer learning can beat attrition
Philipp Schmidt
16:27
hand
Kevin Cole
16:28
I don't recall if I've asked, but do you partner with much smaller groups like: http://knowledgecommonsdc.org/
Jane Park
16:28
measures of success?
Chris Ewald
16:29
metrics feels qunatitative
Philipp Schmidt
16:29
jane had one already, i copied it into the etherpad
Jane Park
16:29
lets just use whatever the research community uses..
Billy Meinke
16:29
quantitative metrics can still show qualitative patterns
Jane Park
16:29
+1 billy
Leah MacVie
16:30
Can this be at the course/challenge level: Can it be as simple have having the 'course designers' 'transparently' state their goals for the course?
Chris Ewald
16:30
qualities?
Philipp Schmidt
16:30
+1 there is a sometimes arbitrary rift between quant and qualitative work
Leah MacVie
16:30
as* having- my bad
Jane Park
16:31
oh i like indicators
Vanessa Gennarelli
16:31
me too
+1
Chris Ewald
16:32
I think so - and proxy measures is interesting too
Billy Meinke
16:32
and if goals/benchmarks are not met or are met in a different way, that'll guide the next set or group of things that are measured
Jane Park
16:32
proxy anything
Philipp Schmidt
16:33
we can definitely "describe" them
Chris Ewald
16:33
so what woud an indicator and what would a proxy be?
Leah MacVie
16:35
As a tangent: I think there are also certain 'traits' P2PU wants in each course. I developed this 'online course review toolkit' for my instructors: http://www.olc.canisiuscampus.net/workshops/teaching-online/online-course-review-tool/ Something like this might help P2PU 'course designers' develop quality courses that meet certain course goals...must must be flexible enough for individuals to set their own. (Course goals, course objectives vs program goals/program objectives)
Chris Ewald
16:35
it does
Philipp Schmidt
16:35
hand
Chris Ewald
16:37
ok
Pieter Kleymeer
16:38
an example from the project I worked on: http://gorgevitalsigns.org/Reports/VSI_SOG_Scenic2009.pdf
Billy Meinke
16:38
ah, thx for the link Piet, was going to ask for that
Pieter Kleymeer
16:39
all part of this: http://gorgevitalsigns.org/
thanks - it's been a while so I'll let the links/docs speak for themselves unless folks have more questions.
Kevin Cole
16:40
BTW, I think the chat may "go away" when we close. (IIRC, Hangout chats are "off-record".)
Given the number of links flying by, as well as other comments, we should capture the contents BEFORE we sign off.
Billy Meinke
16:40
+1 Kevin
me
16:40
@KEvin - absolutely. I'll make sure it all gets copied and added to the notes.
Jane Park
16:41
i added a couple links under examples in the pad
Vanessa Gennarelli
16:43
LOL
Billy Meinke
16:43
doesn't sound simple
Jane Park
16:43
also it is disingenuous at times
like you have to write something for your funder
Philipp Schmidt
16:43
yes!
Leah MacVie
16:43
Oooh- yeah, Jane**
Philipp Schmidt
16:44
with good funders there is usually some level of adjustability
if they understand why you're making changes
but that also is work
Vanessa Gennarelli
16:44
maybe its this flu
but im a little fuzzy as to how
Vanessa Gennarelli left group chat.
Vanessa Gennarelli joined group chat.
Vanessa Gennarelli left group chat.
Billy Meinke
16:45
this is sad
Jane Park
16:45
vanessa
i think its your connection
Vanessa Gennarelli joined group chat.
Leah MacVie
16:45
Can we all pitch together $5 to buy the poor girl a reliable Internet connection!
Jane Park
16:46
you had the same issues with collaborate!
uhuh
its always the tool
Leah MacVie
16:46
No! No! It is awful!!
Billy Meinke
16:46
you're being singled out by the Google
Kevin Cole
16:47
+Vanessa mayhaps browser or OS issues? (I'm on Linux and Google Chrome, FWIW.)
Vanessa Gennarelli
16:47
it works until i stop talking!
must be the magic
got it
OK thats clear philipp
Billy Meinke
16:47
being "open" is core
+1 P
me
16:53
Kevin: how awesome is your profile pic right now... this will broadcast beautifully!
Pieter Kleymeer left group chat.
Kevin Cole
16:53
I raise again the idea of accessibility / diversity. Not to beat the drum too hard, but to keep it on the radar. How do you reach "marginalized" learners?
Vanessa Gennarelli
16:54
word
i think translation and captions s/b part of that conversation
Billy Meinke
16:55
Gotta run to a CC meeting, will keep the pad open. Talk soon, thx guys.
me
16:55
That's an excellent question. I guess it depends on what kind of things are marginalising them. Are they tech issues (low bandwidth in the developing world for example) or are they usability issues, like having alternative access points like captions
Kevin Cole
16:55
Consider also "low bandwidth" users.
Billy Meinke left group chat.
Kevin Cole
16:56
+P2PU two minds w/ but a single thought.
(I was considering both, but not with any fantastic solutions.)
Erika Pogorelc joined group chat.
me
16:57
Hi Erika!
Erika Pogorelc
16:57
Hi
Dirk Uys
16:58
Hi Erika!
Erika Pogorelc
16:58
Sorry I am really late
Pieter Kleymeer joined group chat.
Erika Pogorelc
16:58
been at work
Jane Park
16:58
i have to run to a meeting as well
bye!
Dirk Uys
16:58
Bye Jane
Vanessa Gennarelli
16:58
Vanessa dances
Chris Ewald
16:58
welcome Erika!
Vanessa Gennarelli
16:59
But I cant talk because googlehangout will kick me out!
Jane Park
16:59
welcome erika!
me
16:59
we can hear you!
Jane Park
16:59
i look forward to work with you, esp around School of Open badges/development..
Chris Ewald
16:59
we did nothing in the past years...it's fine
me
16:59
Yes, you can watch the video on you YouTube soon!
Philipp Schmidt
17:00
Audio is bad for me ... (can't really hear Erika)
Jane Park
17:00
ok i really gotta run... bye!
Jane Park left group chat.
Philipp Schmidt
17:00
bye Jane!
Kevin Cole
17:01
+Philipp, me too. Erika's audio present but poor quality -- very echoic.
Chris Ewald
17:01
sounds good
Leah MacVie
17:02
woohoo!
Vanessa Gennarelli
17:02
yaytime!
Chris Ewald
17:02
great
Leah MacVie
17:02
Thank you! Great hour!
Vanessa Gennarelli
17:02
WELCOME ERIKA
Pieter Kleymeer
17:02
thanks kids
g'bye
Leah MacVie
17:02
Good day! bye!
Pieter Kleymeer left group chat.
Leah MacVie left group chat.
Chris Ewald
17:02
thanks bye all!