WebMaking 101 Badges -- Previous content about badge integration 1.0 is at the bottom and at  http://pad.p2pu.org/ep/pad/view/badge-integration/rev.17725

This document:

• General overview of badges / assessments
• Specific implementation for Webcraft Pilot is now here -> http://pad.p2pu.org/webcraft-challenges-badges

Next steps

• Jessica write up some implementation plan for features (that are fairly general)
  • (JL: I thought Erin/Jessica first draft. If so: http://pad.p2pu.org/webcraft-challenges-badges and then from there I can send out to this group [+ p2pu-dev?])
  • Started: http://pad.p2pu.org/badge-implementation
• Erin to review draft challenges, and decide how to integrate badges
  • Content
  • Mechanics

Call Agenda

• Build challenges UX for release 12 Sep
• Build badges / assessments for next release (late Sep)
  • Webcraft:
    • 12 Sep: 5 challenges up, people can sign-up and start working through them
      • Content-wise: First 5 should be complete day after tomorrow(?)
    • Building on existing functionality as much as possible
    • Each challenge = 1 group [of tasks]
      • So it's a new 'group' type - yes
      • signup? same participants? organizers?
        • logic behind this decision? -> why not create 1 challenge as 1 group instead?
          • people need to be able to sign-up to an individual challenge
          • challenges have sub-tasks
          • dev/connected to each other
      • number of challenges: 15
      • planned to build more going forward (and have community contribute more challenges)
    • Each challenge has a number of steps/tasks
      • Those that don't have tasks right now will have tasks in the future 
      • Example: http://webmaking101.p2pu.org/challenges/1
      • Sub-task --> http://webmaking101.p2pu.org/challenges/1/2/
    • Create a pathway that leads from challenge to challenge
      • Example: http://www.codecademy.com/programming-intro
      • Another example are the missions in openhatch.org but that looks for specific test to assess being done with that task
      • Maybe last task is "next challenge/task" lead-in? 
      • Or maybe last task is "I'm done / submit for assessment / what's next"?
        • "I'm finished with this challenge, look"
        • Maybe they don't even ask for review (auto ask or...)
    • Users can come in at the beginning, or pick any of the challenges
  • BADGES -- What badges do we want to issue?
    • [Erin] Look at the first five challenges that are developed in some detail and  recommendation on which badges to integrate and how
    • [Erin] Same for community badges
    • How to award
      • per task
      • per group/challenge
      • per webmaking 101 (all challenges done)
      • how do skills badges fit into this? (JS might work in many challenges)
        • we can either do levels where you build on levels by going through the challenges (but not sure the challenges align with this) or could just have the JS badge available through any one of multiple badges
        • could we tie a "challenge" badge to various skills badges? 
          • demonstrating that you completed a challenge requires demonstration of the following abilities related to CSS?
          • kind of like the tags in the current draft challenges? (jessica's point)
          • or it could say - you can earn three badges (JS, CSS, and HTML) through this challenge. at the end someone might be awarded one or two, not necessary to get all badges? or necessary?
            • And  maybe can get multiple occurrences of same badge like peer learner. Ad maybe not sending all of them upstream
  • DISPLAY / UX -- How do we display badges?
    • Design/UX
  • ASSESSMENT -- How do we handle assessment (awarding badges on what basis)?
    • What features need to be build to support?
    • Self-assessment
      • Tasks can be marked completed (on a per user basis)?
    • Peer-assessment
      • Challenge for a badge as final task in a challenge
      • (Keep in mind -> Incentives are not trivial to figure out)
    • Guru-assessment (could tie in with "adopt a challenge" feature)
    • Stealth-assessment / automated-assessment
• 15 Sep - Badges launch
• Coordinating between different pieces (Erin, Jessica, John, Jamie, Chloe)
• What are we building, when?
• redirects for the mentioned url?
• Peers will create the work they need to submit inside a group and then submit a link(s) to the badge challenge.

• Badge challenge as a new lerning project kind (together with study group, course).

Todo list:

• Send email to Erin, John, Jessica, Philipp, Arlton, Chloe so they can give feedback ASAP (what I forgot? what has to change? what is not clear? timeline and priority between features? and please answer inline questions if your name is mentioned.)

Vocabulary:

• Badge: what is awarded as recognision for learning or activity on p2pu.org.
• Badge Challenges: what users have to accomplish or complete to earn a badge.
• Assesment Logic: different customizable logics for how badge challenges can be completed and the associated badges awarded.
• Tasks: current task functionality available of p2pu study groups, courses.

Objective (version 0.1):
Build basic assessment functionality into Lernanta for the School of Webcraft's WebMaking 101 course (http://pad.p2pu.org/webcraft-plan). This includes:

• support creation of badge challenges
• promote/display badges, badge challenges, and awarded badges within the P2PU platform.
• allow users to send the badges they earn to the Open Badge Infrastructure (OBI) -- done for osqa pilot badges, needs to be extended for sending lernanta badges
• track and report metrics

Core Features/Functionality:

ROLES

(1) Badge Admins (new role on p2pu)

• To allow schools and p2pu to endorse awarded badges some or all the  initial  badge challenges will be created by people appointed by the  schools.
• We will allow school organizers to select badge challenge creators for the schools.
• These people will need the necessary UI to create and manage challenges and the badges associated with them.
• Creating a badge challenge includes selection and customization of the assesment logic and the badge that will be awarded.
• Badge challenges created by  badge admins of a school will be associated to the school as well as the  corresponding badges and the badge challenge will display information  about which school endorse them).

(2) Study Groups/Course Organizers (existing role)

• Depending on the assesment logic behing them, some badge challenges or all of  them will require the interaction between users (e.g. user submits some  piece of work for review as proof that (s)he deserves the badge, and   someone has to review it).
  • One of the options to allow users to interact with eachother is to allow this interaction to occur inside a study group or course.
    • In order to bring badge challenges into the courses, organizers (maybe   participant too if it  is not a course but a study group) will select  badge challenges  endorsed by their schoolto be displayed as tasks inside their  study groups or courses.
    • The developer(s) implementing  the functionality for integrating badges into  lernanta will make the desicion of how much of the necessary logic  will  be implemented for  all tasks and which part only for the badge challenges.
      • As a first step implement what we find easier given how tasks are implemented at the moment and the refactoring plans to provide pages for schools.
  • John: Will this interaction also occur in the UI for mentoring? Will this interaction occur at the challenge page (i.e. no restrictions in who intereact with who)?
• It is possible that the Web  Making 101 course will  require a few badges not endorsed by the school, to give  a sense of  completion to other   tasks of their course, or to the actual  course.
  • If this happens then  there will be badge challenges and badges not created by badge admins  and associated to specific groups or courses.
  • John: Will this be neccesary or not?
• After asking John we will get a better idea, but from what we know at the moment the first priority here is to allow selecting endorsed badges to be part of the task list of a group or study group.

(3) Study Group/Course Participants (or users that want to get a badge)

• Some badges will be awarded to them automatically (based on metrics tracked in p2pu).
• Others will require users to submit pieces of work to a badge challenge, and review eachother work. Kind of what happens currently with comments in  a  task but now with the clear objective of awarding a badge.

ASSESSMENTS LOGIC

• Stealth assessment:
  • On the first badge pilot we did  not add badges that were awarded based on stealth assessment but the  software that we customized for the pilot  had  badges like these to  encourage certain behaivours among their  users (e.g. http://badges.p2pu.org/badges/23/pundit).
  • This kind of assessment logic will allow us to automatically award badges based on metrics collected inside p2pu
  • We have to see how much customization we can put into them (i.e. if we   can/want rely to a certain degree on superusers, badge admins and group organizers for their customization and/creation) but as a first step  we  can define them at the source code level and let those creating the  badge challenge to choose from the list of stealth assesments we  provide.
  • The initial stealth assesments available will be based on the metrics we are  collecting already collecting and we will have to start collecting data specifically releated to the user behaviour around badge challenges.
  • examples of things we could give recognition for with stealth assesments are:   visited course pages X numbers of days in a week, posted X numbers of   comments, reviewed X numbers of submitions.
  • John: Which has more priority for WebMaking 101: stealth assesments or peer assessment?
• Peer assessment:
  • This kind of assessment logic will allow users to get recognision from their peers
  • The   first badge pilot showed  that up/down votes were not appropiate  because  down votes are  unlikelly to occur (thus the idea of rating the  submited  work) and  that we need to make it simple for peers to evaluate  each  other work  (thus the idea of providing rubrics with the badge   challenges).
  • In   some cases it will allow users to submit their work for review (e.g.  to  receive recognition when they learned something) and in others  users  will nominate their peers directly (e.g. to give recognition to  their  peers for qualities like good team player)
    • John: Do we need the second case for WebMaking 101?
    • Both cases have the roles of "candidate for earning a badge", and "peer evaluating if the candidate should earn the badge"
    • The logic behind them will vary (i.e. be customizable by badge admins) depending on:
      • the  scale defined for votes and and threadholds of votes and number of   peers evaluations to issue the badge (e.g. we can have peers evaluate   from a scale from 1 to five and require a 3 or higher rating from 3   assessors to issue the badge)
      • if a rubric consisting of yes or no questions (checkboxs) is provided to help/automate a bit how rating is given.
      • if a submition is necessary from the candidate that will earn the badge
      • what will be evidence that will be associated to the awarded badge once the challenge is completed by a user.
      • if it has to check that the candidate already have some prerequisit badges.
      • John: which of this if not all are necessary for WebMaking 101?
  • John: please priorize the different configuration options for this kind of assessment.
• Guru assessment:
  • Some badges will require us to be more restrictive with repect to who can evaluate a canditate (i.e. trigger a badge award).
    • John: is that the case for WebMaking 101 badges?
  • This restriction will take place either by requiring that people evaluating   to already have those badges or that they have some role (course   organizers, badge admins, maybe mentors).
  • In other matters this kind of assesment will work like the peer assessment.
  • in  the case that we require users to have the badge in order to rate others the  initial seeds will be provided through honorary assessment (see below).
• Honorary assessment:
  • For   seeding purposes (so guru assessment can start), we will need a way to issue a badge to someone   directly even if they have not completed a challenge (see guru   assessment for the reason we need this)
  • Badge   admins will be able to do this and the evidense will include who   awarded them the badge and why (text filled out by the badge admin).

BADGES

• For badges we need the information that will go into the OBI (name, short   description, image) and a little more (long description) to display in   the badge page which url will go to the OBI when a badge is awarded.

BADGE CHALLENGES

• Creating a badge challenge will include:
  • Filling out a title and description.
  • Selecting and configuring the assessment logic.
  • Chossing the badge that will be awarded to those completing the challenge.

WORK SUBMISSIONS

• Some badges will require candidates to submit a piece of work for review.
• Submissions will use rich text (meaning users can embed things like github gists, ... as part of their submition and upload files).
• we  need versioning (like we have on tasks) for submissions because users   can go back to improve their work (don't think we need to reset the   votes after the user posts a new version if it is clear on the evidence page which votes correspond to each version).
  • if a peer votes again in a different version the old vote will be override in the count but still appear on the evidence
• users should be able to "delete" a work submition (and restore it). data  will  remain on the database but it will not be displayed. -- the # of deleted submissions and versions can be included as a metric too.
  • A message saying this submission was deleted will appear on the evidence page in case the user already sent the badge to the OBI
  • If the user has not send the badge to the OBI the badges earned for this submition will not be displayed or sent to the OBI
  • If the user have not earned the badge the submission will not be visible for review and thus the badge will not be awarded.

AWARDED BADGES

• It's was goes into the OBI.
  • data from the badge (name, short description)
  • the badge challenge (url that points to the page with information about the criteria for awarding the badge) -- at the moment called 'template' on django-obi
  • evidence   (url to a page that has the evidence of how the badge was awarded.   depending on the badge challenge and the assessment logic this will  vary  but as and example it could have all the submissions from the user  and  the ratings + feedback (s)he got)

WALL ACTIVITIES

• Activity   around badges will be on the existing 'Default', 'Learning', and 'All'   filters of the walls (groups/courses, dashboard, profiles)
• A new filter will be included just for activity related to Badges (named 'Badges').
• How the deletion of walls mentioned on http://pad.p2pu.org/webcraft-ux affects this

BADGE CHALLENGE ENDORSMENT

• As   we said before challenges will have a default  endorsment depending on   who created them (someone appointed by a school,  p2pu or inside a   course).
• If   we implement support for badge challenges that created inside study   group courses, as next step we should provide to badge admins:
  • a way to endorse those challenges so they are marked as endorsed by the school)
  • a way to group/course badge challenges to the list of school challenges.
• John: is this needed for WebMaking 101?

METRICS

• We will have to report metrics to evaluate the use of badges in p2pu (to superusers and school organizers)
• It can include:
  • # badges issued.
  • for   those badge challenges that require candidates to submit a piece of   work, # of submissions and number of people that submitted work
  •  for those badge challenges that requiere people to rate submissions, # evaluations and average rating

------ Old text ----

BADGE INTEGRATION IN LERNANTA
0.1 Integration Features
 
Description:
Building  basic assessment and badge functionality into Lernanta for the School  of Webcraft 0.1 project. These assessments and badges will be focused  within the Webmaking 101 course and aligned with tasks. Below represents  the core features required to run a pilot within the Webmaking 101  course.
 
Goals:

• support badge challenges and learner submissions
• support peer assessment
• support guru assessment
• support  stealth assessment (if any, will have minimal stealth assessment in  this round, tied to metrics like site visits, number of assessments,  etc.
• push badges into the Open Badge Infrastructure
• promote/display badges within the P2PU platform
• track and report metrics

 
Core Features/Functionality:
 
ADMINISTRATION
(1) Badge Admin role

• New role in the system that gives people permissions to create and administer badges and a backend interface to do so

 
(2) Create Master Tasks

• Create tasks that include assessments and badges that are then available to pull into a course.
• This interface to create Master Tasks is only available to Badge Admins
• Includes creation of assessment and option of creating the badge or linking to existing badge
• Assessment creation form would include:
  • Assessment Title
  • Assessment Type (peer, guru or stealth)
  • Rubric assessment? yes/no
  • Assessment Instructions/Description
  • Additional Details
  • Badge  Alignment (pick from a list) - pick from the list of available system  badges OR Create and link a badge (linking to the form outlined below)

 
(3) Create badges

• Create custom badges that will be available to choose from when creating a task
• This interface to create badges outside of a course is only available to Badge Admins
• Badge creation form should be similar to the creation form on OSQA, including:
  • Badge title
  • Short Description
  • Long Description
  • Prerequisite badges (choose from existing badges)
  • Is this a peer-to-peer badge?
    • If No:
      • Badge Logic:
        • Has-the-badge  Required? - is the voting restricted to only those who have earned the  badge (plus anyone we designate as an owner below)
        • Votes: Number Badge positive ratings required 
          • NOTE:  Since we are moving to ranges, we need to think about how this  translates into the master settings. Could still be number of badge  positive votes required, but what defines the badge positive vote is the  thing to figure out (i.e. a 3 or higher rating from 3 assessors...)
        • OR  Other? For stealth assessment, we could identify several metrics that  we track and award badges for like, checked into site 5 times a week,  participated in course discussion >x times, etc. I can think through a  few of these for the initial pass, just might take me a little more  time)
  • Type of badge (skill, value, peer, participation, misc.)
  • Owner  of the badge (optional) - this allows us to give only certain people  the ability to vote on  submissions for specific badges. This should not  be a full list of all users in OSQA, but instead a list of roles:  (check all that apply) Course Organizer (of the course the badge is used  in), Course Admins, Badge Admins
  • Expiration date? (not required for pilot) 
    • If Yes: calendar to pick date
  • Badge image file

 
(4) Issue a badge to a user from administration

• For seeding purposes, we will need a way to issue a badge to someone directly without the formal assessment channels.
• From  Badge Admin interface, could have a button/link to issue the badge  which brings up a set of fields to enter the username and a  justification of why they deserve the badge
• The badge should then link back to that justification/endorsement

 
(5) Restrict voting on certain badges to specific roles or those who have the badge

• See  above. Work/submissions for expert level badges and some miscellaneous  badges can only be assessed by certain experts or people who have  already earned the badge.
• This is specified in the badge creation form

 
(6) View metrics
    Who can see these metrics? the admins

• Can come up with the list but # submissions, # assessments, average rating on assessments, # badges issued, etc.

 
TASK CREATION FROM COURSE
(1) Pull a Master Task into the course

• Choose from a list of the available Master Tasks to pull into a course
  • What are some examples of "Master Task"?  these would be the P2PU-developed, vetted assessments built as master  tasks that could then be pulled into the course. Basically, instead of  relying on the course designer (content person) to build in the  assessments that we have designed, we can just build them and let  him/her pull them directly in. Also, when we move the assessments/badges  beyond the web-making 101 course, this would be a way for us to make  designed, endorsed assessments that course organizers can pull into  courses.

 
(2) Add an assessment to a task (optional)

• See above, same flow as task creation in the admin interface
  • What's the difference between an assessment and a badge? Get a badge by doing the assessment on a task?  The badge is the end thing that you get if you are successful at  completing the assessment. An assessment might be a series of questions  or exercises that you have to submit work for. If reviewers/assessors  rate the work high enough, you then earn the badge. So the badge is a  symbol of mastery of a particular topic, with the evidence being the  successfully completed assessment. There may be multiple assessments  that lead to one badge - essentially different pathways to that badge. I  can see us doing this early on as we are trying to research/vet our  assumptions/approaches.

 
(3) Link an existing badge to a task

• Choose from existing badges and add to a task

 
(4) Create a badge from task building

• Option to create a new badge from within the task creation process
  • Course  organizers can create any badge? So we could have a badge per task or a  badge for a course (I'm guessing there's a task that's 'finish course'  and awards the badge?)  The model we want to explore in this round of the pilot is the ability  to have a badge per task. There won't necessarily be 1:1 in the  webmaking 101 course but we will have more granular badges, each tied  directly to a task. 
• Links to the badge creation form (see above)

 
WORK SUBMISSION
(1) Submit work to a task for assessment

• Users should be able to submit responses or answers to the assessment tasks
  • So  a task is flagged as an assessment (gives a badge) vs just a regular  "task" page? What about discussion between class mates? Or is that not  allowed/encouraged on assessment tasks?  Yes I guess. Some tasks will be assessments + badges but there will be  other ones which are just tasks like go read this or debate this  question that are not assessments with badges tied to them.
• We could allow them to post files or require them to just post elsewhere and link to it in the submission
  • I believe right now we don't have file storage so it would need to be elsewhere. Yeah, so just have them post elsewhere and link back to it. We did this for the OSQA pilot and it seemed to work fine.
• This submission + assessment feedback should be its own unique URL that the badge will later link to.

 
(2) Format submitted work (links, code, etc)

• Typical WYSIWYG stuff, but also allow for special code formatting since many will post code
• Embedding? That would grow with Lernanta's embedd-ablility too. That would be cool.
  • Agreed  on the coolness of this. Perhaps when we allow for file uploads that  this would grow too. Plus the WYSIWYG functionalty right now is not the  best for code.

 
(3) Edit own submission to an assessment/badge task

• Allow learners to go back and edit their own submissions
• Note: If there is any assessment on the work, they should not be able to delete it. 
  • If  people have already voted, shouldn't it clear out the votes? Or maybe  the person resubmit for votes as a new submission to preserve history?
  • If someone can go back later and update, then they could delete in a way.
  • Can assessers update their votes then?
  • Good  questions, I think we should try to keep it as simple as possible. What  we want is to be able to keep all the data together so you could see  that someone submitted it, got a low rating, and then updated it or  submitted again and got a new rating. We don't want to override or lose  old data on the ratings/feedback. But my fear about making them resubmit  is that it gets lost within the chain of other submissions, instead of  keeping everything together.

 
(4) View recent activity in the course wall

• Show assessment and badge activity on the course wall including if someone earned a badge, posted new work, etc.
  • Can we 'follow' a badge like on OSQA so that we see that someone has submitted for review?  I had this as one of the features below and Philipp suggested pushing  to later releases since we are only rolling this out in one course where  everyone will be focused on the same task at the same time. When we  have more badges in more courses, we definitely want people to be able  to subscribe or follow.
• Course-specific, so would show all the assessment/badge activity on the course wall

 
ASSESSMENT
(1) Check off elements of the rubric

• As an assessor, use check boxes aligned with each element of the rubric to indicate which elements were covered
• Rubrics  will change in length so this should just provide the rubric with check  boxs aligned with each element when someone is assessing the work.  NOTE: might require us to have a specific action like a button that says  "assess this work" which then gives them the rubric with checkboxes  allows them to review and check boxes
• This information should be captured and provided with the assessment

 
(2) Rate the submission

• Rate the work submitted (after reviewing/checking the rubric) on a scale of 1-4
• 1:  Does not meet any of the criteria; 2: Does not meet most of the  criteria; 3: Meets Most of the critera; 4: Meets all of the criteria  (note, scale is subject to change)
• The rating will be what determines if this is a vote for the badge or not. We need to come up with the cut off point for what==badge and what doesn't
  • Is this up to the badge creator?  yes, as it stands now. We could dictate what this should be for  different types of badges, but for now, we are the only one creating  badges anyway. 
• Comment/justification required with the rating
• NOTE: all of the asessment information, including the rating and the rubric checks/unchecks should all be saved with the badge

 
(3) Comment on a submission (without voting)

• Optional - but some people may want to give feedback without voting

 
(4) Rate value of the assessment feedback

• Allow the community to rate the rater and vote on the feedback
• This also would be a range and required comment: 1=Not helpful at all, 2=Not very helpful, 3=Helpful, 4=Very Helpful (or something like that) 

 
(5) Give a peer a badge (with required justification/comment)

• Allow learners to give a badge directly to a peer
• Must enter the peer username and a comment of justification for the badge
• Peer badges accumulate so that you might have the Collaborator badge x5
  • But  we could have 5 different badges named "Javascript" because we got them  in different courses and their requirements may be different?  Yes, skill badges don't accumulate. We are definitely trying to avoid  multiple Javascript badges - and maybe we don't want to let course  organizers create badges when they already exist - something to think  about for future iterations when this is open to more courses than  webmaking 101. One thing we should do is require a unique name for each  badge so we can't have 2 with the same name. And maybe some way to flag  as P2PU/Mozilla endorsed.

 
USER PROFILES
(1) View each user's badge collection

• On user profile, view their badge collection
• Should  the user be able to choose which badges to show and which order to show  them in? [JL] I don't think this is necessary for the pilot, but  eventually having management features in the profile would make sense  b/c P2PU wants this to be a destination community as well

 
(2) Click through each badge to view the user's work/evidence behind the badge

• Each badge should lead to the evidence/work submitted/endorsements/review comments
• So  if Philipp and I have the same badge, on his profile, his badge would  lead to his work and all associated evidence and comments, and from my  profile, my badge would link to mine (including if someone just typed it in via the admin interface? -- probably would look different though.)  Not sure if I am understanding this correctly but if we give a badge to  someone directly, to seed the community or whatever, through the admin  interface, we should have to provide a justification/comment of why that  person deserves this badge and the badge should link to that  justification.

 
SYSTEM
(1) Track votes and issue badges

• Track votes on all submissions and issue badges when necessary thresholds have been reached

 
(2) Track other logic data and issue badges

• For the stealth assessments (to be defined), track necessary data and issue badge when threshold is crossed

 
(3+) Open badge infrastructure issuer requirements

• Storing all the necessary information for the badge metadata: http://openbadges.lighthouseapp.com/projects/80353/badge-manifest
• Maintaining communication channels to validate issued badges
• Utilize the issuer Brian built to push badges into the OBI

 
 
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Old Doc below right? yep - all the old stuff below. only working with stuff above this line.

 
Description:
To  build basic assessment and badge functionality in to Lernanta. The  features identified should be the minimal to suffice for a second phase  of the pilot, but should also be features that will carry through to  later iterations and build outs. 
 
Feedback from Philipp:

• Looks great!
• Only  have one bigger suggestion: roll assessments into tasks for now (and  tasks with assessments issue badges) rather than splitting them and  handling separately (more flexible, but also a lot more complicated). we  have an opportunity to curate this much more, as we will be developing  the content / tasks for next round of SoW ourselves. we can design the  tasks to have assessment features - so they lead to badges. 
  • there  was talk about a 1-to-1 mapping -- basically, a task is a badge. Not  sure if that's in stone though. But then we'd have "got a badge for  reading the syllabus" since "tasks" are so diverse right now. [JL]
• (Still not all the way through ... but coming back)

 
Key Goals:

• support badge challenges and learner submissions
• support peer assessment
• support guru assessment
• support stealth assessment
• show badges in P2PU profile (and throughout the site)
• manage (P2PU, or maybe others as well?) badges through P2PU profile dashboard
• support metrics?

 
Definitions:

• assessments / badge challenges - tasks that are tied to a badge and require some type of review or assessment of submitted work
• badge - image with associated metadata, awarded after successful completion of an assessment or by peers
• system-level badges - badges that are defined by P2PU admins (and school organizers) and are the top-level, endorsed badges (Mozilla and/or P2PU branded)
• system-level tasks/assessments - tasks that are defined by P2PU admins and are accessible to all course organizers for pulling into their course
• peer assessment  - review and voting of submitted work by anyone in SoW community; this  is typically the type of assessment for basic level badges
• guru assessment  - 'has-the-badge' assessment - review and voting by someone who has  already earned the badge; this is typically the type of assessment for  expert level badges. We will seed the community with gurus until there  are enough users that have earned the badge.
• stealth assessment - system tracking and issue of badges based on predefined logic threshold

 
Core Features/Functionality:
 
ADMINISTRATION
(1) Create system-level badges
Philipp  -> I think of these as normal P2PU "Tasks", but with assessment /  voting / etc. built into the Task page. Almost like you can turn badges  on or off for a Task. Easier to build that way so we can roll out much  more broadly to more (all?) tasks lead to some form of badge. But I can  see how that is difficult for some badges - for example, if you design  an interesting "challenge" that is not entirely focused on JS, how do  you know it could lead to a JS badge? Is that something we can solve  through hand-holding - and reviewing tasks carefully as they are  developed. We'll have more control as we will be more actively  developing the tasks/content ourselves. 
Right,  and there will be badges that I think we would want to encourage or try  out different assessments for, no? Regardless, I don't think this is a  pilot phase thing since we will be determining the badges/assessments  for the Web 101 course. It will be fine to just build them into each  task.

• We  could do something like tagging/algorithmic suggestions that won't be  perfect or we could do hand coded "This is a javascript task. Here are  javascript badges." Kind of like how you get suggested other posts in a  blog based on keywords. [JL]
• Create custom badges that will be available for all course organizers to use
• Each  badge could be aligned with several different system-level assessments,  although for the pilot, there will probably only be one assessment per  badge
• Badge creation form should be similar to the creation form on OSQA, including:
• Badge title
• Short Description
• Long Description
• Prerequisite badges (choose from existing system-level badges)
• Is this a peer-to-peer badge?
  • Is the peer-to-peer badge a badge one can give to another for doing X?
• If No:
• Badge Logic:
  • Has-the-badge  Required? - is the voting restricted to only those who have earned the  badge (plus anyone we designate as an owner below)
  • Votes: Number Votes required 
    • Since  we are moving to ranges or subvotes, we need to think about how this  translates into the master settings. Could still be number of badge  positive votes required, but what defines the badge positive vote is the  thing to figure out (x of y right on the rubric or a 2 or higher  rating...)
  • OR  Other? For stealth assessment, we could identify several metrics that  we track and award badges for like, checked into site 5 times a week,  participated in course discussion >x times, etc. I can think through a  few of these for the initial pass, just might take me a little more  time)
• Owner  of the badge (optional) - this allows us to give only certain people  the ability to vote on  submissions for specific badges. There may be  other ways to do this with roles or other permissioning, but need some  way to restrict assessment (voting) to certain people for expert level  badges and other misc badges.
• Type of badge (skill, value, peer, participation, misc.)
• Expiration date? (not required for pilot) 
  • If Yes: calendar to pick date
  • PB: I think that pilot badges are more vulnerable and less well defined so perhaps we need to set an expiration date?
  • What  about adding a issue-date rather than specifying an expiry-date. That  way the market can decide when a JS badge is stale, rather than us.  Also, expiry introduces a lot of complexity into all dimensions (on the  display side, etc.)
  • Issue  date must be captured and stored with the badge, yes. This is a  required piece of the badge metadata for the Open Badge Infrastructure
    • +1 to issue date [JL]
• Badge image file

 
(2) Create system-level assessments (P2PU-created assessments) as core tasks

• Create custom tasks that are available system-wide for integration into courses and are aligned with a particular badge
  • Assessment creation form would include:
  • Assessment Title
  • Assessment Type (peer, guru or stealth)
  • Rubric assessment? yes/no
  • Assessment Instructions/Description
  • Additional Details
  • Badge Alignment (pick from a list) - pick from the list of available system badges
    • What's badge alignment mean? Link to the task that is in a group/course itself?

Note:  this should mirror the task creation form as much as possible. There  may be customizations for these system-level tasks like badge alignment,  but may make sense to build that into the task model anyway
PB:

• I  think that assessments would benefit from a short description as a  title may not be long enough to explain to users what they need to do in  the task
  • This task description (what needs to be done) is different from the assessment criteria (what needs to be measured by assessors)
  • both task description and assessment criteria should be visible to the user.
• do we want assessment criteria to have a marking rubric table? (for when we have reviewers to make sure we keep quality?)
  • How detailed should this be and do we need to  support that detail within the technology? eg.
    • Page validates (yes / no)
    • Appropriate use of semantic tags (good, ok, bad)

 
 
(3) Assign/link system-level badges to system-level assessments/tasks

• See  above. Each assessment only has one badge, but each badge could have  multiple assessments. For the pilot, only need a 1:1 alignment.
• Would prefer to just build as one (task=assessment=>badge)  for now. Splitting them introduces a lot of complexity. Rather, add a  switch to the Task that makes it a badge issuing task (that includes  some form of verification/assessment).

 
(4) Restrict voting on certain badges to specfic people or roles

• See  above. Work/submissions for expert level badges and some miscellaneous  badges can only be assessed by certain experts or people who have  already earned the badge.
• These  will all be "has-the-badge" permissions but we may need to manually  assign these permissions to seed the community before people have  started to earn the badge (+1 for seeding)

 
COURSE/STUDY GROUP CREATION/ADMINISTRATION
(1) Pull in system level assessments/badges as a task

• Course organizers should be able to choose from existing system-level tasks to add into their course
  • How  do they find them? What do we need on the assessments/badges to be  easily findable? Should anything be unique? For example, two assessments  that are titled "Install Git" should not be allowed?
  • List or by tag?

 
(2) Create a badge

• Potentially post-pilot
• Allow course organizers to define badges themselves.
  • If  they create a badge, is it tied to the course or a set of tasks within  the course or similar to peer-to-peer? Or maybe we allow them to create  and then a badge admin awards to a list of people (for now)?
  • It  would be tied to a course and not available for other people to use in  other courses unless we promote it. This is the distinguishing factor  between these course-created/specific badges and the system-level  badges. But we won't need this for the pilot since we are creating all  the content for the course.
• Badge  creation form would be very similar to the admin badge creation form  (just will need to make sure it is super clear and explained well)

 
(3) Assign a badge to a task

• Potentially post-pilot
• Course organizers could create their own task (assessment) and align with one of their created badges
• Note:  for the pilot, and maybe for beyond, I don't think we want to allow  course organizers to create their own assessments for the system-level  badges. It could get messy in understanding which are  P2PU/Mozilla-endorsed badges/assessments and which are  community-created. So in this case, they would only be able to choose  from the badges that they (or other course organizers?) have created)
  • +1 on system-level vs community-created

 
COURSE/STUDY GROUP USER EXPERIENCE
(1) Submit work to an assessment/badge task

• Users should be able to submit responses or answers to the assessment tasks
• We could allow them to post files or require them to just post elsewhere and link to it in the submission
• This submission + assessment feedback should be its own unique URL that the badge will later link to.
  • Pippa:  isn't one of the strengths of the badge the connection to the original  work / portfolio - possibly a q. out of scope for this pilot phase, but  should there be an archive of the work associated with the badge? If  linked files are no longer there surely the badge is less valuable? Is  this our responsibility or the badge earner's?
  • Philipp: good question - i don't think we've solved that issue so far. +1
  • Erin:  yes we have in OSQA, badges link back to the learners submission and  the votes/comments. We need to do that here unless we don't want to have  personal information behind the badge. Some issuers are just putting up  a generic page with information on how the badge was earned (not  specific to the particular learner). This is an issuer decision on how  to handle this. I think linking to the actual work is much more valuable  and powerful but it comes back to if there are any privacy concerns on  the P2PU side.

 
(2) Format submitted work (links, code, etc)

• Typical WYSIWYG stuff, but also allow for special code formatting since many will post code
  • Embedding? That would grow with Lernanta's embedd-ablility too. That would be cool.

 
(3) Edit own submission to an assessment/badge task

• Allow learners to go back and edit their own submissions
  • What if it's to delete their work for which they got a badge. Is that ok? Should we only allow within a timeframe?
  • That's  a good question. If there is any assessment on the work, they should  not be able to delete it. Maybe not even edit it? Need to think about  this.

 
(4) Vote up or down on work submitted by others (with required comment)

• Some ability to rate the work submitted
• Could replicate the OSQA functionality which allows either a vote up or down 
• Require a comment as justification/endorsement/explanation on any vote

 
(4) For rubric aligned assessments: Vote on each element of the rubric

• Instead  of just a summative yes/no vote on the total work, the design group  suggested makign the votes more granular and aligned with the rubric so  that someone votes yes/no on each element of the rubric (with a comment  for each) and then there is some predefined threshold of x out of y  right (or z% yes)  in the rubric = a total + vote
• Require a comment as justification/endorsement on each rubric element vote
• For the pilot, we would just set that the person has to get more than ~70% of the rubric correct
• This will force us to be more rigorous about our rubrics but I think that's a good thing

 
(4a) For non-rubric assessments or for summative assessments : Rate the submission 1-3

• 1-3 is predefined as Does not meet most requirements, Meets most requirements, Exceed requirements
• Design  group suggested a range instead of yes/no vote so that there isn't that  social barrier of saying no. This also can clearly outline how to  improve.
• Eventually  we might want this to be rate each element of a rubric too and we can  test one versus the other. For the pilot, I think having two options of  bigger assessments with rubrics and general rating of the overall work  would be good. 
• We would need to preset the badge to say that some average of 1-3 is required for a badge positive vote.
• Require a comment as justification/endorsement with the rating

 
(5) Comment on a submission (without voting)

• Optional - but some people may want to give feedback without voting

 
(6) Vote up or down on comments/feedback

• Allow the community to rate the rater and vote on the feedback
  • Optional just comment like "Thank you"? Or always require a voting when commenting on the feedback?
  • This  is a good question - I would say only allow voting on the feedback. We  probably want to do the range on this instead of binary voting. 

 
Philipp: I LOVE this, but maybe post pilot?
I think the peer badges are an important piece of the new model.
(7) Give a peer a badge (with required justification/comment)

• Allow learners to give a badge directly to a peer
• Must enter the peer username and a comment of justification for the badge
• Peer badges accumulate so that you might have the Collaborator badge x5
  • Is this the only badge type that does accumulate?
  • Yes, I think so at this point

 
Philipp: Suggest to move to post pilot - not sure we know enough about how this should work
(8) Subscribe to/follow certain assessments (tasks)

• Sign up to receive email notifications if there is activity (new submissions or new votes) on a particular task/assessment

 
(9) View recent activity in the course wall

• Show assessment and badge activity on the course wall including if someone earned a badge, posted new work, etc.
  • Badge that is tied to the course? Or badges of participants?
  • Yes, badge this is tied to a course

 
SYSTEM-WIDE
(1) View all badges with links to active assessments

• There  should be a view into all of the active badges in the system (meaning  those badges that have at least one assessment aligned) with a listing  of the assessments (as links out to those assessments)
• Should be organized by type of badge
  • peer-to-peer, etc or 'tag' style (javascript, programming, etc)?
  • skill, value, peer, participation
• Should indicate how many people have each badge
  • So 'most popular' badges? 
  • well  yeah, popularity is one way to view it, but also rarity is important  too, so they harder (and possibly more valuable) badges to earn would  have less. This is how Khan academy does it: http://www.khanacademy.org/badges/view

 
USER PROFILES
(1) View each users badge collection

• On user profile, view their badge collection
  • Should the user be able to choose which badges to show and which order to show them in? [JL]  I don't think this is necessary for the pilot, but eventually having  management features in the profile would make sense b/c P2PU wants this  to be a destination community as well

 
(2) Click through each badge to view the user's work/evidence behind the badge

• Each badge should lead to the evidence/work submitted/endorsements/review comments
• So  if Philipp and I have the same badge, on his profile, his badge would  lead to his work and all associated evidence and comments, and from my  profile, my badge would link to mine

 
SYSTEM
(1) Track votes and issue badges

• Track votes on all submissions and issue badges when necessary thresholds have been reached

 
(2) Track other logic data and issue badges

• For the stealth assessments (to be defined), track necessary data and issue badge when threshold is crossed

 
(3+) Open badge infrastructure issuer requirements*

• *Need to talk to Brian to specifically get all of these details
• Includes: passing correct metadata for each badge

 
Questions:

• What's the promotion process from course organizer created badge to system-level badge?
  • we need to get (actual) experts to review before they are promoted 
    • SoW content advisory board / ambassadors?
• Should  we/how can we incorporate stealth assessment in early stages? These can  just be system-level assessments and badges that we create, but would  require different logic than just votes. Would be data tracking within  the course or learning platform...

 
Additional Features for Later Iterations:

• Points  versus yes/no vote - allow more precise assessments by scoring against a  rubric and sum either leading to Yes/No or badges tied to # points.  Would probably want to have both options, straight up Yes/No and points.
• Subtasks within assessment task - ability to rate/vote on more incremental parts of the assessment and then sum up for a total 

 
(Ideas that could help simplify the design and examples of use -- Zuzel):

• The  non-stealth assessment badges are created by group organizers by adding  the necessary meta-data to a task (image, booleans, ...).
• The collaborative feature is used to restrict edition if desired.
• First level comments can be marked as submission by the authors.
• Replies to submissions have an optional vote (following the peer-assessed, guru-assessed, .. logic).
• Any task including the ones that are badges can be imported to any group, but clones do not correspond to the same badge.
• Groups with non-moderated signup need to be supported.
• School organizers can choose which badges are endorsed by them from groups with non-moderated signup.

Examples of use:

• Someone  from the school of webcraft staff creates a Web Development Skills  group with non-moderated signup and marks it as featured for the school.  They add some gurus as organizers to help them create non-stealth  badges which can not be edited by participants (non-collaborative tasks  with the necessary meta data). The school add these badges to their list  of endorsed badges which is displayed on the school home page. The  school set of badges can be enriched by the organizers of the Web  Development Skills group since they can clone succesful badges developed  from other groups of the school.
• A  smaller group uses the same features to create an assessment not  related to a school but that allows to recognize what they learned when  they participated in the group, and thus make a difference between  people who just joined the group and people who demonstrated that they  got knowledge from participating in the group.

 
Open Badge Infrastructure Alpha Plans:
 
What's Done:

• All of the communication pathways between the issuer and hub done
• Core hub piece done
• Backpack stuff (frontend) not done at this point
• No display site integration at this point (badges will be displayed on the P2PU profile)

 
Issuer:

• Brian building a test issuer in django - still needs work to be production-ready (doing back-channel communication)
• Probably need to start over to build the Lernanta production-ready issuer technology
• Brian can build - will help to build with the real use case, can expose issues, etc.
• Need to get Brian access to the data - alpha.p2pu.org (anonymized version, with 'user1', 'user2')
• Brian will fork Lernanta and let Zuzel know when ready to integrate
• General timing: End of next week or before the next Lernanta release
• Badges being displayed from OSQA now

 

• Where hosting infrastructure?
  • django, mongo
  • Brian will test on Ubuntu but dev on Mac for now

 

• Show only certain badges?
  • Will be part of backpack
  • But issuer maybe could have some of this as well to help with lookup efficiency
  •