Copied from http://etherpad.mozilla.com:9000/webcraftassessment

SCHOOL OF WEBCRAFT :: JANUARY ASSESSMENTS AND BADGES

:::::SUMMARY:::::::::::::::::::::::::::::::::::::::::::::::::::::::::
We  are conducting a pilot project in the January session of the School of  Webcraft to incorporate some assessments and associated badges.  We hope  to learn more about the viability of badges, reactions to various  assessments/badges, which skills are important to the community and  if/how to scale the badge system.  The badges and assessments targeted  for the pilot are outlined below and cover the following:

Badge types:

• SoW defined skill badges
• Course defined skill badges
• SoW defined community badges
• OSQA community behavior badges (tentative)
• Other badges

Assessment types:

• Peer assessment
• Guru/badge owner assessment
• Course organizer assessment
• System assessment (tentative)

Badge Issuers:

• Peers
• Course Organizers
• Gurus
• System
• Us ("Other" badges)

:::::BADGES/ASSESSMENTS::::::::::::::::::::::::::::::::::::::::

SOW-DEFINED SKILL BADGES
Based  on work with and suggestions from the Webcraft community, we have  selected 3 skills to build assessment and badge options for in the  pilot. These three reflect important skills and include a basic level  badge, and two expert badges to cover a range of skill level. Also, the  JS badges are hard skills, whereas the Accessibility badge is considered  a "softer" skill.  

1)  Javascript Expert badge

Description:
The Javascript Expert badge recognizes high-quality, advanced Javascript knowledge and skills.

Justification: 
Javascript  is an essential language for web developers, with applicability across  many courses. We also want to encourage people to learn advanced skills  and aim for the expert level.

An Expert Javascript programmer is someone who:

• can intelligently decide between JavaScript frameworks and justify their decision.
• consistently  develops JavaScript to Douglas Crockford's recommended style guide OR  an alternative one that provides solid justification for its principles.
• regularly reviews their code quality using JSLint, FireBug or similar tool.
• implements JavaScript in conjunction with other technologies.
• recognises the benefits and limitations of JavaScript and can clearly explain this to clients and peers.
• understands how JavaScript is interpreted by the browser.
• is aware of vulnerabilities within JavaScript and develops secure code.
• can clearly describe the DOM event flow
• intelligently optimises code for efficiency and speed.
• implements graceful degradation as a default and understands the range of browsers and devices that will implement their code.
• is aware of the historical and technology context.
• can  read and interpret relevant technology standards and are aware of how  they can contribute and provide feedback to these standards.
• shares  their knowledge with other developers and release their code for the  scrutiny of others (contributes to the js community).
• practices regular code reviews with colleagues and peers.
• clearly documents how their technical solution will respond to client requirements.
• recognises the range of users accessing sites with their code and understands their varying needs.

Assessment Details:

• Learners will seek out the badge and submit work for assessment.
• The  artifact for assessment is a functioning website or js code for some  purpose.  We expect that many will have existing work to submit, but we  may need to offer a couple challenge ideas for people to use to build  something new.
• With  the submitted work/artifact, the learner must submit a paragraph+  describing their work and why they should get the JS Expert badge
• The  badge will be aligned with several courses (TBD in early Jan) and  courses members can submit to the course community for guru assessment.   For those seeking the badge outside of the specific courses, we are  considering asking several webcraft community gurus to do the  assessments. 

Assessment Challenges/Submission options:

• Challenge:  Submit a link to original code (authored and submitted by you) that  demonstrates expert Javascript, as well as a write-up explaining the  code and why you consider yourself a JS Expert. 
• Challenge:  Create a wrapper that does not exist/someone has not produced before and explain your steps and final outcome.

Assessment Type: GURU (aka, HAS THE BADGE)

Assessment Rubric:
Submitted code...

• is at expected expert level, exhibiting advanced js techniques and considerations
• is bug free (runs in the browser without errors)
• fits  within Douglas Crockford's recommended style guide OR an alternative  one that provides solid justification for its principles
• is secure, avoids vulnerabilities
• exhibits graceful degradation
• is optimized for efficiency and speed
• is clearly documented, instructive of approach and choices

Assessment Process

• Learner submits work and paragraph(s) through the dedicated OSQA environment
• Peers who have already earned the badge (aka 'gurus') will assess the submission, vote Yes or No and leave comments for feedback
• If Yes, the badge will be issued. 
• If No, the submitter will receive the feedback and can revise work and resubmit
• Once badge is issued, it is linked back to the work/endorsements

Notes:  

• For  those JS Experts not in a related course, they can submit through the  Badge page and we will have some JS community gurus who will review and  assess.  In this case, one Yes is sufficient to award the badge. 

2) Javascript Basic badge
Description:
The Javascript Basic badge recognizes mastery of initial Javascript skills.  Aligned with Intro to Javascript course(s).

Justification:
Basic  skills are a critical foundation for Javascript mastery and competency.  This badge is an entry level badge that can be a stepping stone to the  JS Expert badge, or could help intermediates or experts revisit and  refresh their core skills**.

Assessment Details:

• Learners will seek out the badge and submit work for assessment.
• The artifact for assessment is the code and documentation for the following (or similar) challenges: http://en.wikiversity.org/wiki/Web_Design/JavaScript_Challenges

Assessment Type: PEER

Assessment Rubric:
Submitted code...

• meets the requirements of the challenges
• provides the expected behaviors or output
• is bug free (runs in the browser without errors)
• is clearly documented

Assessment Process:

• Learner submits code for the challenges to the course peer group
• Peers review the submission against the rubric
• Peers vote Yes or No and provide an endorsement/feedback (140 characters?)
• Once gets 3 Yes votes, badge is issued, along with the feedback to the learner*
• If 3 Yes votes are not achieved, learner can use the feedback to revise work and resubmit for peer group to review
• Once badge is issued, it is linked back to the work/endorsements

Requirements:

• Mechanism for submitting code for assessment (wiki page?)
• Mechanism for peer review (recognition that there is something to review, voting capability, commenting/feedback)
• Mechanism for learner to receive the peer review feedback (private? public?)
• Mechanism for issuing badge
• Badge infrastructure, where badge can be stored and linked back to the work

Notes:

3) Accessibility Expert badge
Description:
The  Accessibility Expert badge recognizes attitudes, perspectives, values  and practice oriented around the importance of accessibility and  universal design.

Justification:
Accessibility  is an important set of considerations for any developer or designer and  being aware of users' needs is an important part of creating useful and  valuable products.  There is no exact equation for accessibility,  instead it requires awareness and attitudes around acceptance and  inclusion. 

An Accessibility Expert is someone who:

• is aware of the physical, technology and social challenges which can affect a website's accessibility.
• demonstrates awareness of the design and development responses and approaches which make web content more accessible.
• develops accessible websites as a default.
• can communicate the legal, financial and humanitarian benefits  and responsibilities of accessible web sites to clients.
• develops meaningful accessibility tests for their designs.
• cooperates with end-users of different backgrounds to get feedback on their sites.
• can  read and understand W3C and relevant local standards regarding web site  accessibility and are aware of how they can contribute and provide  feedback to these standards.
• works with design personas that reflect a range of ability and access situations.
• continuously updates their skills and knowledge. 
• contributes to the development community and seeks feedback.
• can evaluate websites and content for accessibility, recognize and propose solutions for issues.
• develops  an understanding of alternate user perspectives by experiencing their  own web sites on mobile devices, via screen-readers, with slow download  speeds.

Assessment Details:
Learners will seek out the assessment and point people to blog or artifact demonstrating their attitudes and experiences.
While this badge will be aligned with/promoted within a specific course, anyone can submit.

Assessment Challenge/Submission:
Write a reflective essay in a blog post or some other shareable form on the Web about the following:

• Why accessibility is important
• What types of people should we be designing for
• Experiences with challenged users and/or accessibility technologies
• How you build accessibility into the work you do (with examples)

Assessment Type: GURU
Accessibility  evangelists in the community will be issued the badge to seed the  community, then they will review submissions and determine who has  earned the badge. Once someone earns the Accessibility Expert badge,  they then become one of the assessors, creating an expanding set of  evangelists for accessibility.

Assessment Rubric:
Submission...

• demonstrates  comprehensive awareness of the physical, technology and social  challenges which can affect a website's accessibility.
• demonstrates awareness of the design and development responses and approaches which make web content more accessible.
• communicates the legal, financial and humanitarian benefits and responsibilities of accessibility
• demonstrates awareness of multiple perspectives
• exhibits accessibility-oriented attitudes
• provides examples of work than demonstrates accessibility orientation

Assessment Process:

• Learner submits post / reflection
• Guru reviews it against the rubric
• Guru decides if badge should be issued:
  • If Yes, Guru provides endorsement and badge is issued
  • If No, Guru provides feedback to learner
• Once badge is issued, it is linked back to the reflection/endorsements
• Once badge is issued, the learner can now assess/issue this badge

(4) Accessibility Foundations badge
Description:
The Accessibility Foundations badge recognizes general awareness and understanding of accessibility and universal design.

Justification:
Accessibility  is an important set of considerations for any developer or designer and  being aware of users' needs is an important part of creating useful and  valuable products.  There is no exact equation for accessibility,  instead it requires awareness and attitudes around acceptance and  inclusion. 

An Accessibility Foundations is someone who:

• is aware of the basic physical, technology and social challenges which can affect a website's accessibility.
• communicates the importance of accessibility and universal design
• understands  a number of strategies that they can use in their own code and work to  improve the accessibility of websites and products

Assessment Details:
Learners  will seek out the assessment and answer questions in a narrative either  on their own blogs or within the assessment/badge environment
This badge will be aligned with/promoted within specific courses

Assessment Challenge/Submission:
Option 1:  Write a reflective essay in a blog post or some other shareable form on the Web to answer the following:

• In your own words, define the following: universal design, usability, accessibility?
• What are 3 strategies you can take to improve the accessibility of a website?
• Write  a paragraph to a potential client about why he should budget for  accessibility and universal design aspects of the project.

Option  2: Pick a website (that no one else before you has chosen) and assess  it for accessibility. Post the link to the site, and a narrative  explaining:

• The  general level or compliance with accessibility and universal design  principles. Please explain the principles and use examples from the  site/code.
• Recommendations you have to improve the accessibility.
• A  paragraph that you could send to the website owner explaining your  findings and convincing them of the importance of your recommendations.

Assessment Type: PEER

Assessment Rubric:
Submission...

• demonstrates general awareness of the physical, technology and social challenges which can affect a website's accessibility.
• demonstrates general awareness of the design and development responses and approaches which make web content more accessible.
• communicates the overall value and importance of accessibility considerations

Assessment Process:

• Learner submits post / reflection
• Peers review it against the rubric and vote Yes/No and leave justification/feedback comment with vote
• At 3 Yes votes, the badge is issued. 
• If 3 Yes votes are not achieved, learner can see the feedback and revise/resubmit if desired

COURSE ALIGNED SKILL BADGES
We  are planning on creating some badges/assessments aligned with January  sessions.  We will ask the course organizers as part of the orientation  process and/or initial weeks to help us identify additional badges and  assessments that fit within the course topics. We are hoping to get at  least 3-4 more skill oriented badges/assessments.  

Suggestions so far:
PHP Basic (modeled after the Javascript Basic)
PHP Expert (modeled after the Javascript Expert)
CSS Basic (modeled after the Javascript Basic)
More suggestions?  Please send them our way: https://spreadsheets.google.com/viewform?hl=en&formkey=dHZPOXVNLVV1azN3OFk5MW83UEktdEE6MQ#gid=0

Update:  based on the interest through the mailing list, looks like HTML5 and  CSS3 are hot skills that we might consider working with relevant course  organizers to add into the pilot.

SOW-DEFINED COMMUNITY BADGES
These  badges are meant to help build community and recognize 'softer'  skills.  We have defined 5 initial badges based on work with LinkedIn  recommendations, as well as discussions with the Webcraft community.  These badges will be available for peers to issue to other peers, along  with an endorsement.

For all of the following badges:

Assessment Details:
Badges are awarded by peers and not sought out explicitly by learners.

Assessment Type: PEER

Assessment Process:
After experience with the learner, peers can decide to award the badge based on the rubric.
Peer(s) issues the badge with an endorsement justifying the badge

Requirements:

• Mechanism for issuing badge and providing endorsement (accessible to all SoW participants)
• Badge infrstructure

Notes:
If  multiple give me the same badge, should it accumulate (so my Team  Player badge may have a score or indication that I received it 3x).

Badge Ideas (note: these are not necessarily the badge names; also open to other suggestions or ideas):

1) Team Player / Good Teammate

Rubric:

• Collaborates well
• Helps push work forward within the team, willing to do work for the shared goal
• Exhibits sense of team spirit and shared purpose
• Motivates other team members

2) Communicator

Rubric:

• Writes clearly and concisely
• Accurately communicates concepts
• Can explain concepts to various audiences, including nontechnical people
• Maintains tone of helpfulness and respect

3) Problem Solver / Innovator

Rubric:

• Thinks critically, weighs options
• Solution-oriented
• Has solved problem or challenge to push work forward

4) Community Builder

Rubric:

• Actively participates in the (course) community
• Participates through social tools
• Provides helpful and interested comments to others posts

5) Peer Mentor

Rubric:

• Helps explain concepts or approaches
• Volunteers extra time to help others understand topics
• Answers questions and provides feedback on others work

OSQA BADGES
These  badges would be awarded automatically based on behavior within the SoW  OSQA forums, capitalizing on stealth assessment, as well as promoting  behaviors that are beneficial for the community.

Badge ideas (still in progress):

• Commentator: Left 10 comments
• Enthusiast: Visited the site each day for 30 consecutive days
• Community Guru:  Answered x questions with x votes up

Notes:

• Erin going through the existing StackOverflow badges to see what types of behaviors we might want to start with
• This is tentative based on if we can get the technology up and running in time to build in some badges
• Need to see how to make these badges work with the others (is OSQA another issuer into the Hub/Backpack?)

 
OTHER BADGES
These are some other ideas of badges that have come up in various conversations or calls.

1) Course Organizer Badge
This  badge would be awarded to anyone who organizes a course (and has  organized a course in the past, if still involved).   It could be a way  to encourage more people to organize courses. We would issue this badge  to current course organizers, and previous ones can submit to get the  badge and we can verify and issue.

2) P2PU Veteran
For  those people who have taken P2PU courses before, and are back for more,  they could get the P2PU Veteran badge (involved in 2+ courses). Someone  could submit for the badge and list the courses they have taken and we  could issue the badge accordingly.

3) Peer-to-Organizer Badge
For  someone who has taken a course and then organized / facilitated a  course.  We would adverstise this badge this session and hopefully  encourage people to think about organizing  a course that they are  currently taking.  

Benefits:

• Facilitating the course is an excellent form of assessment - put your knowledge to test
• Could encourage lots of people to organize a course, have multiple 'sections' for popular topics
• Could  encourage higher quality courses as people reflect on what worked or  didn't work in their experience taking the course, while they are  planning for their own facilitation

Limitations:

• Could have overwhelming course creations (although this would be a good problem to have)
• Could  actually encourage lower quality courses if people really aren't ready  to facilitate (more pressure on us to have a stellar orientation)
• May push out core facilitators

Note:
Philipp  had an idea of potentially having a 201 course in April or later  session taught by a community guru, where the only way to get into the  course would be to teach a 101 course on the same topic in the previous  section(s).  Could do something like this to determine the group to help  us build asynchronous courses that can be repurposed. 


:::::EVALUATION:::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
A  critical piece of the pilot is the evaluation of the badges and  assessments.  We will define a more in-depth plan for the evaluation but  some key metrics will be:

• How many times the assessment was sought out (if applicable)
  • Within a course
  • Outside of a course
• How many times the badge was issued
• What is the reaction from the community - are they satisfied with the badge/assessment?
• Was the badge/assessment gamed in any way.  If so, how?
• How valuable is the work 'behind' the badge (end of session evaluation by Mozilla?)

:::::TECHNOLOGY:::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
Many  of the ideas and decisions around which badges/assessments to include  may be driven by the technology solution that we have in place for the  January sessions.  

Currently, we are considering two parallel processes to manage the assessments and issue badges

NOTE: we plan to have more finalization around these over the next couple weeks

1) OSQA ENVIRONMENT
We are utilizing the OSQA environment to manage the badges and assessments.  
for a more detailed description of the OSQA requirements and open questions, see: 
http://etherpad.mozilla.org:9000/osqabadges

Details:

• Badges/challenges for badges will be questions within the course (tag) section of OSQA or in the wider community
• Users will submit their work/artifacts/reflections as a reply to the question
• Peers or gurus will assess and vote up or down replies (essentially saying yes this meets the rubric, or no it doesn't)
• At some threshold, the badge would be awarded
• Badges will be ported from the OSQA environment to the P2PU profile at the end of the session

2) P2PU Environment (Manual)
The  community oriented badges may be more manual within the P2PU  environment.  Still under development - trying to see if we can use OSQA  for this...

Details

• Peer has a positive experience with another learner
• Peer posts recommendation within the P2PU profile
• We manually issue badges at the end of the session based on the recommendations

:::::OUTREACH:::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::

• Identify which courses that we know want to include our predefined badges in (after Jan 12)
• Reach out to the course organizers that we want to use our badges within
• Dedicated orientation session for course organizers
• Reach out to all course organizers to see if they have ideas for badges (across orientation and first two weeks of courses)

:::::FEEDBACK:::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
Please feel free to add your feedback inline, or use the space below.  Thanks!!