Rethinking Assessment

To many in higher education they see problems as nails and testing as the only hammer in our kit. Yet we stand at a time where we can re-mediate assessment using new technologies and old definitions of what it means to learn.

I serve on the Tech Fluency (TF) affinity group at Southern Connecticut State College. TF is a tier one competency in our liberal education program (our general ed program but with more hoops and loftier goals). The goals is to ensure all students have the minimum tech skills they will require after college. In our current (permanent) budget crisis we have been asked to review the effectiveness of LEP.

We developed a series of rubrics instructors could use in their classroom. The newly appointed LEP assessment committee decide our approach was “too subjective.” They suggested a common task, filling out a spreadsheet was their recommendation, that students could complete in a controlled and supervised  environment.

This set me off. I responded (probably with not enough to delay and too much acerbic snark) with some of the following comments.

Objective Assessments are a Hoax

Subjective flickr photo by EVRT Studio shared under a Creative Commons (BY-NC-ND) license \
I am not anti-testing. Most of my research revolves around item design and testing. Yet I think what set me off was the belief that some assessments are objective.  Those who rely on standard measures ignore the bias inherent in statistical models and deciding as “what counts as learning.” They look at rubric scored items as being “too subjective” yet ignore the error variance, the noise in their models. I say bring the noise. It is in outliers where we see interesting methods and learning.

Technology Assessment Lack Ecological Validity

We were asked to create a shared assessment. It’s just the task sounds like 1996 wants their Computer Applications textbook back. We have to move beyond, “These kids don’t know spreadsheets” as the only critique in our self-assessment. There is so much more in the competencies beyond the basics of Excel.
The idea that you do anything in tech under supervision and sitting alone in a crowded room is the wrong approach to assessment. What we are calling cheating will be required collaboration for anyone doing any thing with tech in any field.
This is why I think a digital credentialing platform is the correct path forward. If you  begin by mapping pathways and  rubrics similar to ours or better yet even more fine grained criteria we could develop a system where faculty still had the freedom to design (hopefully co-design) a pathway for students.

Students Should Drive Assessment

 I think we   should  involve the students as stakeholders to a much higher degree in any assessment.We do this by helping stduents tell their story. This also  shifts responsibility onto them to build the data trails we need.

 Purpose of Assessment

I also took issue with the the purpose of our assessment. If the goal is to evaluate the effectiveness of our LEP program then why is our gut reaction to assess each student individually.

Its not that the approach of of a learning artifact (the spreadsheet assessment we were asked to develop) is the wrong path in terms of overall  measurement.
Let us as faculty assess the individual and let machines surface the patterns at class, school, and system level.

Technological Solutions

I think the Academy long term should push off most system wide assessment onto machines. Its way more effective and correlates so highly with well trained human raters.
If we scored could score a batch with high inter-rater reliability once, and laser honed the criteria, much of this could be machine scored and credentialed with minimum faculty involvement. Faculty could build whatever assessments they wanted to on top of the task. It really wouldn’t matter to LEP assessment.
This isn’t fantasy. It’s usually $7-10 a user (for the scoring).

Moving Forward

All measurement and all grades are subjective. Yet I think we have a chance to rethink the academy by empowering learners through assessment. Its time to kill the Carnegie Credit hour.
In fact across the state of Connecticut we have been discussing how to seamlessly transfer students between seventeen community colleges and four universities. Plus students would like to receive credit for work that would demonstrate competencies in our Tier One classes. If we really wanted to think about Transfer Articulation we would forget about tracking credit hours and think of each student as an API. If we had the matching criteria, or even a crosswalk of offerings, it would be a matter of plug and playing the assertions built into our credentialing platform.
This would also allow students to apply previous work they completed in high school or outside of school and get credit for meeting the technology fluency competencies. We can use the new endorsement feature in the Badge 2.0 specification so local schools, computer clubs, or even boot camps could vouch for the independence of student work. The learning analytics can help us with our programmatic review and tracking student knowledge growth.

Some Examples

As some example I threw together a quick prototype that could be using technology fluency and one that could be used in our writing intensive classes (I see very little light between writing and technology)
Want to start a fight in the open badges community? Bring up rigor. Challenge the point of participation badges or say something like, “I got the I blinked while breathing badge.”

I admit I am guilty of questioning badges for low hanging fruit. During the Connected Educator Month last October you could get a badge for attending almost any event. I wondered, often out loud on Twitter, if this approach made sense.

Christina Cantrill of the National Writing Project often pushed backed. Christina made the argument that the evidence for the participatory bags is just that, evidence. She explained to me that different badges have different value and you could even have leveling up badges. As Christina explained it, rigor means nothing, it is relevance of the badge to the community that matters most.

As I have become more and more swayed towards Open Badges (due to the evangelizing efforts of Ian O’Byrne and Doug Belshaw) I keep returning back to this question: What do we get when badging for participation versus badging for competencies?

VIF International

I found my answer yesterday during the The Badge Alliance Teacher Badges open call. Mark Otter and Julie Keane shared their learning platform that they use as part of their efforts in teacher preparation and global education.


The call was great. Seeing badges in the wild, specifically connected to teacher preparation and professional development, really helped to formulate my thinking. To date VIF International has awarded 592 badges to teachers in their community.

Behaviors versurs Competencies

I am sure this debate has played out int eh badging research community for quite some time. Like I said, I am a recent convert so I am stumbling into  many of the lessons others have already  learned.

Mark and Julie make a distinction between competencies and the behaviors that they would like to see in their community.Teachers can earn badges when they share evidence that demonstrates competencies.  This often includes  lesson plans.

To recognize behaviors that help support online communities and professional development Mark and Julie created a point system that translates into stars. So you can earn points for things such as logging in, posting to a forum, and commenting on another post. These are not behaviors  that provide evidence of growth. Yet these behaviors are crucial to building critical mass in online learning spaces.

I think as I start to develop badges, specifically for summer Gear Up Programs I help run, I will try to bake in a similar approach. Lets keep badges for competencies and use other metrics to reinforce behavior.


Update: I submitted my badge application and recieved feedback from Doug.

This “just for fun” badge actually documents the successful work flow of a badge. Doug created the badge, explained the competencies being addressed, described the evidence needed. I then submitted my material. Doug reviewed it and left me this feedback:



I then went back and revised my original post. If you want to track my edits look at my source code I left the old version in html comment form.

Doug Belshaw issued a challenge. As part of the Mozilla Web Literacy Map roll out he encouraged folks to submit artifacts that would demonstrate competencies on three areas of the map,

Screen Shot 2014-04-25 at 13.57.10


I am submitting my badge under the working title of Wisdom, Virtue, Sincerity, Valor, and Austerity in Online Spaces.



  • Accessing the web using the common features of web browsers– I try to teach students how to to explore the web by creating and remixing videos and think alouds of online data: I created videos of students reading online that I allow others to use
  • Using hyperlinks to access a range of resources on the web- I try to find and share links to open education resources such as this one. I was looking for OER sources on the web, and put out a call on Twitter. This link came back to me.
  • Reading, evaluating, and manipulating URLs I wrote a dissertation on differences in searching and evaluating online sources.
  • Recognizing the visual cues in everyday web services- I make online minilessons to teach students how to search the internet and research ways to teach credibility.


  • Using keywords, search operators, and keyboard shortcuts to make web searches more efficient –My  dissertation research focused on improving search results.
  • Finding real-time or time-sensitive information using a range of search techniques- I write about the need to teach and read socially complex texts.


  • Researching authorship and ownership of websites and their content- I create online materials to teach students to focus on authorship.


  • I use my brain as my virus and phishing detector.


Composing for the web

  • Inserting hyperlinks into a web page-  This post
  • Embedding multimedia content into a web page-I can embed multimedia into posts.
  • Creating web resources in ways appropriate to the medium/genre- I write in a variety of places using the norms of thos sites such as Medium.
  • Identifying and using HTML tags- I used the comment tags so people could track the revision history on this post.


  • Identifying and using openly-licensed work- The image I remixed for the header on this post used two openly licensed images.
  • Combining multimedia resources and Creating something new on the web using existing resources- I make remixes using popcorn.

Design and Accessibility

  • Iterating on a design after feedback from a target audience– I got feedback from Doug and then revised this post. This website is the rough draft of my life
  • Improving the accessibility of a web page by modifying its color scheme and markup- I try hard to choose color schemes that allow those with red/green color blindness to differentiate.
  • Demonstrating the difference between inline, embedded and external CSS- I spent this semester trying to use Thimble to teach myself CSS. I have created my first page that no longer uses HTML tables but relies on CSS containers. The actual page isn’t live yet (Department website but containers work

Coding and Scripting

  • Composing working loops and arrays and Using a script framework- I dabbled in javascript when creating a simulated web environment. I do not know java but if I can stare at code long enough I see patterns, kind of like poetry, and can then edit the code. I made changes to the timing and feedback loops.



  • Tracking changes made to co-created web resources– This is the first collaborative story I wrote in Gdocs with my 6th graders.
  • Co-creating web resources- Ian and I edit the digital texts and tools page. Please join us and add your stuff.
  • Configuring notifications to keep up to date with community spaces and interactions– Much to my wife’s chagrin as things chirp and beep all day long.
  • Using synchronous and asynchronous tools to communicate with web communities, networks and groups– I use asychnronous and synchronous chat in my teaching.


  • Encouraging participation in web communities– I encourage folks to be digital residences.

Community Participation

  • Using constructive criticism in a group or community setting -I use online communities on Google+ for Feedback.
  • Defining different terminology used within online communities- I use the discourse of specific affinity spaces and use these spaces for learning.


  • Identifying rights retained and removed through user agreements– I added the Creative Commons plug in to this site.


  • Distinguishing between open and closed licensing- I use only open lecensed images on this site.

As we live online we navigate a sea of myriad rivers merging. Those who use the web literacy map can guide multiple streams of information.

As educators we need to draw a map (of the territory such as the Web Literacy Map) using creativity and all means available to you. To [further] illustrate this point, when even the roads are unknown, enter the online spaces, and familiarize yourself with the languages and practices. Determine which areas have steep learning curves, which areas are wide open, and measure the width of roads to understanding.


Last week faculty asked me to present some ideas I had on using portfolios in our Liberal Education Program (commonly referred to as general education classes at other institutions). I went into the meeting and discussed that badges and not portfolios made more sense.  I believe when we want a competency based program, such as ours,  we need to think more in terms of online presence rather than online portfolios.

So instead I discussed why Open Badges makes more sense for our University rather than a proprietary portfolio system.

Our Liberal Education Program

Calling our LEP classes, general ed, does a disservice to our students and those committed faculty who designed the program. Yes they are classes every student must take but the LEP builds on a foundational belief in the importance of liberal studies. The program contains 24 separate goals over three tiers of classes (click here for more information).  These goals are broken into competencies, knowledge and values.

Screenshot 2014-04-28 at 7.55.06 PM

As a rough summary tier one classes include competencies, tier two include focus on knowledge and values,  and tier three integrates all three elements into a final outcome.

Our tier one competencies briefly introduce the Areas of Knowledge and  Experience and Discussions of Values while maintaining a focus on the development of intellectual skills, or competencies.

I volunteer on the technology fluency committee which is a core competency. In this tier students have to:

1. Common Tasks – Solving problems, accessing information, and  communicating information and ideas using appropriate technologies

A. Students will be able to engage in electronic collaboration.

B. Students will be able to use and create structured electronic documents.

C. Students will be able to do technology-enhanced presentations.

D. Students will be able to use databases to manage information.

E. Students will be able to use spreadsheets to manage information.

F. Students will be able to use graphical and multimedia technologies. 

2. Focus – Using emergent or recently developed technologies (hardware orsoftware) to address specialized tasks

A. Students will have the ability to perform basic operations in at least on ecurrent technology platform, or

B. Students will acquire advanced level skills in three out of six of the Common Tasks listed in (1).

3. Future Technological Change – Navigating and adapting to futuretechnological developments

A. Students will be able to use electronic tools to navigate, to compare orcontrast, to research and to know enough to evaluate the technology as a tool.

4. Broader Implications – Being cognizant of ethical and social implications of revolutionary technologies, including but not limited to their impact on security, privacy, censorship, intellectual property, and the reliability of information

A. Students will be familiar with major legal, ethical, privacy and security issues in information technology.

The purpose of tier one classes is to introduce and teach these competencies. They are then reinforced in tier two and tier three classes. This spiral of complexity in the curriculum design makes badging a better assessment system than portfolios.

Why Badges

The tech fluency and the embedded competency of information fluency represent a broad goal. To establish and display competency students need more than a GPA. Having a B in a class tells me nothing. Yet if I can look at a meta-level badge and know student earned badges in the three areas of tech fluency I can build off of that learning in my tier two classes.

So the badging system would involve different levels of badges. Students would have to earn the common task badge (I know need a better name) would have to point to urls that document each of the common tasks.

They could then submit requisite materials for the other three badges. Once those are all submitted the students could apply for the meta level tech fluency badge.

I believe the badges will then support the tier two classes. Teachers trying to put a larger focus on general knowledge, can look back at the badges students collected in tier one and cater projects to individual students.

Why Open Badges over Portfolios

Portfolios do not seem to work at our University. We make students buy into a program called TK20, but this is not in the support of learning. Its just a way to generate pretty tables for NCATE. At the end of the semester students upload the prerequisite documents, educators grade them, and there they sit collecting digital dust until we need the artifacts for NCATE.

Portfolios are often proprietary. We could use the system embedded into our overpriced LMS but why not ensure assessments  are owned by students and our 100% portable.

Presence matters more than portfolios. Learning is a social event. I do not want to silo off engagement into a system. I would rather students be able to document their education where ever they learn.

Open badges would allow students to document learning in lieu of class. We have many students, especially in tech fluency, that believe they meet the requirements all ready. If we had an open badge system students could then submit documents, without being in a proprietary system, that demonstrate their competencies. If they could earn the meta level technology fluency badge they would not have to complete the course.

Next Steps

First I have to convince a slew of committees open badges makes sense and explain how it could be implemented into our liberal education program.

Then we would have create the road map of competencies for each badge.

We would hten design the layout of the badges.

Finally for this to work open badges require the help from our great IT team on building the back end support so we can use the open API.


The competency based liberal education program and badges make sense. We want to document students abilities for life long learning. Why not create a system that reflects these values in both practice and theory?