Learning Challenges Metrics
- These metrics can be tracked by challenge set or individual challenges
- Some of the metrics require advance setup/ addition of google analytics keywords
Metrics for Hewlett interim report -> http://pad.p2pu.org/metrics-hewlett
Conversion rate
- Users who hit the site
- Google Analytics
- M1a Logged In Users
- M1b Anonymous Users
- (From 15 Jan 2012 -> can run separately for not-logged in and logged-in users)
- For older data -> N = all visitors (data from google analytics) accessing:
- School of Webcraft home page
- Challenge set home pages
- Challenge home pages
- Challenge Task pages
- M2 Lernanta logs -> We also have total number of "only anonymous" visits
- IP address, URL, date/time
- M3 Share of users who are logged in/ registered but never start a challenge
- Logged in users only who visit a challenge related page at some point but never start a challenge
- File 1 - All visits to relevant pages with userid
- File 2 - Userid, time challenge is started
- Result: Take all userid from file 2, and remove them from file 1 (and the opposite is people who started a challenge)
- M4 Share of users who start a challenge
- M5 Share of users who complete a challenge
- Only users who started a challenge at least 3 days prior to reporting
- [Future] Share who abandon a challenge
- Users who are inactive for 10 days
- Users who manually "leave" a challenge
- M6 Share of users who complete more than one challenge (2, 3, 4, ...)
- M7 Share of users who complete a set of challenges
- [Only for challenge sets] Avg number of Challenges completed per learner
- N = Users who completed at least one challenge
== Example Report -> Webmaking 101: 1 Nov - 1 Feb 2012
M1a
Logged in users who visited challenge related pages (nov 1 - feb 1): 705
IP addresses of logged in users: 952
M1b
Anonymous 1 (Number of distinct IP addresses accessing the site) who visited challenge related pages Nov - 1 Feb 2012: 2847
M3
Users (unique users, not IP) who visited a page related to challenges (nov 1 - feb 1) but did not started a challenge: 280
M4
[i assume this is "all times" - correct?]
Users who started at least one challenge (all times) 548
Users who started at least one challenge between nov 1 and feb 1: 428
Number of users who started at least one challenge from challenges 3 to 7 (nov 1 - feb 1): 162 (this is to compute: what percentage of people who took one or more of the challenges #3 to #7 were th superblogger badge is available actually applied for it)
15 out of 81 who offered help reviewed a submission (this to compute %)
M5, M6, M7
Number of Users Who Completed the Challenges (all times):
'webmaking-101-introduce-yourself': 209,
'webmaking-101-writing-html-by-hand': 156,
{'webmaking-101-choose-your-weapon': 144,
'webmaking-101-html-hunting-in-the-world-around-you': 71,
'webmaking-101-your-domain': 40,
'webmaking-101-your-hosting': 26,
'webmaking-101-youre-live': 22}
Number of Users who completed at least one challenge (all times): 273
Number of Users who completed at least 2 challenge (all times): 142
Number of Users who completed at least 3 challenges (all times): 113
Number of Users who completed at least 4 challenges (all times): 67
Number of Users who completed at least 5 challenges (all times): 37
Number of Users who completed at least 6 challenges (all times): 22
Number of Users who completed the 7 challenges (all times): 14
Avg number of completed challenges among the people who completed at least one challenges: 2.446886446886447
Avg number of completed challenges among the people who started challenges: 1.2167577413479054
Number of adopters: 81
From the adopters how many assessed at least one submission: 15
Number of badge awards, Number of Users who received badges:
skill badge html-basic 9 9
community badge wave 27 24
stealth badge beta-hero 280 280
stealth badge html-is-all-around-you 72 72
submission-enabled community badge super-blogger 21 18
skill badge publishing 2 2
stealth badge your-webspace 27 27
skill badge hosting 2 2
stealth badge you-re-live 23 23
submission-enabled community badge creative-presentation 9 8
stealth badge introduce-yourself 213 213
stealth badge domain-name 41 41
stealth badge text-editor 146 146
community badge helpful-feedback 4 4
skill badge dns-basic 4 4
stealth badge writing-html-by-hand 161 161
stealth badge webmaking101-basic 15 15
Comments:
Total of comments to challenges: 1011
Total comments (nov1-feb1): 760
Total number of comments(nov1-feb1) per challenge:
{'webmaking-101-choose-your-weapon': 137,
'webmaking-101-html-hunting-in-the-world-around-you': 50,
'webmaking-101-introduce-yourself': 346,
'webmaking-101-writing-html-by-hand': 139,
'webmaking-101-your-domain': 53,
'webmaking-101-your-hosting': 23,
'webmaking-101-youre-live': 12}
Preparing script for generating csv files:
Checking for duplicate rows:
badge_awards_raw_data.csv 2943 1056 ---------- skill badge,publishing,user43004,2012-02-11 18:55:42
challenge_completion_users.csv 668 668 ---------- user40560,webmaking-101-choose-your-weapon
challenges_click_start_leave_button_and_adopter_creator_learner_status_raw_data.csv 1147 1147 ---------- webmaking-101-writing-html-by-hand,user43652,2012-02-13 12:43:06,,learner
challenges_click_start_leave_button_raw_data.csv 1146 1146 ---------- webmaking-101-introduce-yourself,user43652,2012-02-13 07:13:33,
challenge_tasks.csv 7 7 ---------- webmaking-101-youre-live,4
comments_raw_data.csv 1011 1009 ---------- webmaking-101-introduce-yourself,,user43409,2012-02-08 14:08:21
registered_users_visits_raw_data2.csv 14544 14480 ---------- /es/schools/school-of-webcraft/sets/webmaking-101/,2012-02-09 01:44:07,user43450,189.180.4.169
registered_users_visits_raw_data.csv 14266 14202 ---------- /es/schools/school-of-webcraft/sets/webmaking-101/,2012-02-09 01:44:07,user43450
task_completion_raw_data.csv 2726 2726 ---------- webmaking-101-writing-html-by-hand,hello-world,user43652,2012-02-13 12:48:02
users_hit_site_raw_data.csv 15076 14942 ---------- /es/schools/school-of-webcraft/sets/webmaking-101/,2012-02-12 22:45:59,189.157.73.204
users_ip_address.csv 8987 8987 ---------- user42095,168.77.194.213
Preparing script for generating metrics given csv files:
Badges
- Number of Badges issued over time (organized by badge type)
- Number of badges issued per learner (badge categories: making, understanding)
- Number of blog posts that were generated & comments //
- Number of adopters
Possible future metrics:
- Users who register after visiting a relevant page
- Users are anonymous (we log their IP)
- They visit a relevant page (tracking by IP address)
- They register (tracking by IP again)
Older Notes:
- Starting with the ones in bold
Background on other people's work on metrics:
---------- Dashboard ---------------
1) account creation
2) comments
3) joined a group
4) created a group
Question (Philipp):
- Do we need true realtime? Why don't we just pull this from the DB once a day (or once per hour)? I can already look at all these metrics on the Django admin.
Mockup: https://gomockingbird.com/mockingbird/#6d1kbro
- big number on the mockup is "today" how many new users, comments, users joining groups, groups
- actual is total for this month
- pace is extrapolated based on data so far this month
- guess based on a linear trend total so far in the month / days passed * number days
- last is actual at the end of the previous month
- the last part is the color
- should be red if we are more than 5% behind the previous month
- white if it's ?5% to +5%
- and green if more than +5%
Comments (Philipp)
- I think first priority is tracking the right metrics in any (even if ugly dashboard) way.
- Would like to see monthly, weekly trends as well, rather than snapshots
Metrics/SoW Metrics
- Next steps
- Create list of key metrics
- Share with community
- Turn into features (where needed) - start reporting
- What are metrics useful for?
- Gauge overall health
- Drive our decisions - platform develompent, support structures, etc.
- give boards / funders something easy to view: < Mozilla Foundation board are interested :-)
- Metrics we are interested in:
- Users
- total registered users
- new sign-ups over time (per day)
- users on site / logged-in (today)
- % people "following" other people
- avg number of people they follow
- % of completed profiles
- Participants
- total number of participants
- total number of participants per school
- % [and absolute numbers] of user base that participates in at least 1 course (for all of p2pu)
- distribution of number of courses per user
- Followers
- total
- avg/per study group
- total per school
- Study Groups
- courses_total
- list w state, days in that state etc.-- ability to identify stale courses
- courses_per_school
- number of participants per course
- number of followers per course
- study group creation over time
- frequency of participation (per person, and overall)
- courses in each "state" (draft/sandbox/open/closed etc.)
- how long does a course stay in draft
- what percent of courses convert from sandbox/development to running
- Activity
- how often are people commenting/executing "actions" on the site
- activity per course
- activity per person
- average delta response time between comments per course
- Text Analysis
- most frequent words uses
- per person
- per course
- over time :D
- User Location
- Google Analytics:
- Unique visitors
- IP (location)
- Search terms that lead people to the site
- Visitor locations
- Bounce rate
- Average time on the site
- Conversion rate from visitor to registered user
- Can we tell if someone is a "new" visitor"? (google tracks new visits, if we trust this)
- Conversion rate from sign-up to action:
- Parked for now:
- Badges
- Tasks created / completed
- Revisit once task completion features exists
- over time will groups identify core tasks that when completed recognise "group completion"?
- This mean we should show that we completed now or is this for after we get this in the platform? And optional on a task figures in how? Bonus? :)
- How frequently do we record this information?
- real time, rolling day/week/month
- yay! Yeah I can see this as a report page for the admins (I want it to be public) <3 this ok, report page for all ;) [Needs discussion how/what part of this we make public.]
participation - want a good way to normalize that - eg. total participants, total existing courses, and time is an important metric.
Original Metrics Pad
Course enrolment:
* At start of course
* Per course
* Each week throughout the course
Number of applications vs. accepted participants
* Please add to this spreadsheet
* https://spreadsheets0.google.com/ccc?hl=en_GB&key=tmtS-PcwNqHsFmEUJTlJm1Q&authkey=CMqhwuMO&hl=en_GB#gid=4
Drop outs over time
* Number of registered participants who logged into p2pu.org each week
* # of participants who posted into the course
* etc.
Applicants from X different countries
Users from X different countries
Course participants from X different countries
Web traffic from X different countries
Web traffic per week
* Visits
* Pages / visit
* Time on site
Number of emails sent through p2pu.org / week
Number of unsubscriptions per week for all courses
Number of unsubscriptions per course/ratio participants