In the years leading up to his death, my mentor Clay Christensen battled many ailments. He wondered often about when things were all said and done, as a deeply religious, man, how would he be viewed when he had his final interview with God?
He reached a conclusion that, regardless of one’s religious leanings, has significant implications for how we shape outcomes-based data in higher education—and a need to figure out how to capture individual learners’ desires for progress in those metrics.
Before sharing that conclusion, it helps to have some background on Clay’s broader views about data.
Clay was always suspect of data. First, he was fond of pointing out that data are created by humans. Data aren’t some objective things that magically exist in the world. Humans must create categories, observe phenomena, and make decisions about what the “data” are. There’s subjectivity in data, no matter how we might wish otherwise, as data are based on constructs that humans have created at one point or another.
Second, data, as he was fond of saying, are, by definition, backward-looking. They are unable, by definition, to tell you anything about the future in and of themselves. And by the time data become available, the phenomenon has already passed. Even for those that talk about “real-time” data, they are, by definition, somewhat of an oxymoron. What’s more, data are only convincingly and clearly available about the distant past after people have had time to sort through them and scrub them.
What did this have to do with Clay’s final interview with God?
As Clay considered all these observations about data, he decided that God probably didn’t need data like humans did. We need data to simplify and make sense of what is a complicated and messy world, to organize things in different ways, and be able to understand a big picture that has lots of moving parts underneath it.
But God, he concluded, had an infinite mind, as opposed to us humans with our finite and limited minds. And God was able to see beneath the data and look at the individual stories in the world and understand each story and instance. N of 1 didn’t bother God, in other words.
From this, Clay concluded that when he was before God, he wouldn’t be judged based on the raw number of people that he had influenced or touched or the scale of his reach. Instead, God would judge Clay by each individual interaction that he had had with humans in the world. And God would be able to make sense of all those individual interactions and how Clay had acted. Clay, of course, would have passed with flying colors far more than most of us.
But that, of course, isn’t the point of my story. Nor is my story about whether God exists and has an infinite or finite mind.
The point is that we often mistake data for the phenomena or the outcomes that we’re trying to create rather than seeing data for what they are: a representation and abstraction of something that humans have created. Data are not objective reality.
And yet, we also know that data have an incredible power to motivate how organizations optimize themselves to make the data “look good.” Organizations will go to great lengths to produce outcomes around data—financial, environmental, student, and so forth.
In thinking about this, there may be a fundamental problem with this desire to optimize the data in postsecondary education and training. That’s because the metrics and data on which an institution might optimize are fundamentally supply-side in nature—that is, they view the world from the perspective of those institutions that deliver education.
The problem is that using data, in that case, might not match the demand-side of the equation. What is success for an individual learner in their specific circumstances? What happens when what success looks like for a particular individual doesn’t match up with what is right for the institution?
How can we cease judging institutions on supply-side metrics that capture the average and instead evaluate them based on how they facilitate the individual progress each learner desires in their lives? How do we capture learner voice in outcomes-based metrics?
It’s easy to think that this mismatch must not happen all that often, or that my question isn’t all that pressing, because the majority of metrics in accredited higher education are still input-based. Indeed, nine out of the 10 categories accreditation tracks are inputs because of regulation.
Although I agree much of our metrics are too input-based—that’s been a major thrust of my writing for over a decade now—I suspect the mismatch between individual learner and institution may occur more than people think.
A few stories help illustrate.
Guild Education, where I’m an advisor, works with working adult employees and helps them gain further education and upskilling opportunities through a range of academic partners. It has a unique data set because its concern isn’t just how the working adults it serves progress through their education program, but also—and more importantly—what promotions and pathways open for those individuals at the employers with whom Guild works.
One trend is that a working adult will enroll in a particular college program to acquire more skills and earn a promotion. The program is multiple years, but after one year, the individual has learned many new skills and then earns a promotion. That promotion can mean an increase in tens of thousands of dollars of salary, which is very meaningful for front-line employees and their families.
At this point, the employee presses pause on their enrollment in the learning program. According to traditional metrics, this would be a dropout or a stopout, which would be considered a bad outcome. But is it? The employee just received a meaningful promotion after all.
Guild is certainly not the first to recognize this phenomenon. Community college presidents have long complained about this as well. Those complaints helped inform our creation of the standards behind the nonprofit Education Quality Outcomes Standards Board—to try to allow institutions not to duck accountability, but to accurately claim the value they create for students.
But let me go a step further.
In our book Choosing College, we identified five different “Jobs to Be Done” for why students “hire” a postsecondary program.
One of those Jobs we identified was “Help me get into my best school.” Success for the learner here is all about getting into school, not what happens next.
Then there was “Help me step it up,” which is all about helping someone step into the next thing in their life when they felt like time was running out, they had to act now or never, and people were depending on them. This is like the Guild working adult trying to earn a promotion. The fulfillment of this Job was not about the credential per se. Education was a means, not the end, to something better.
A third Job is “Help me extend myself,” which is all about the challenge and love of learning and a personal pursuit that may or may not have anything to do with a promotion or completion or anything else. Think MOOCs or skimming a book.
A fourth Job is what we call “Help me get away.” A story helps illustrate the complexity of this circumstance.
A student named Juan, about whose story the design firm IDEO told us, didn’t know why he was in college. He enrolled because the No Excuses charter school that he had gone to said he had to attend college and had helped him every step of the way with his application and enrollment. This is what we call another Job— “Help me do what’s expected of me”—and one for which I’m not convinced colleges should be enrolling students. If they do enroll students, then helping them discover purpose and passion should be the success metric.
With that said, given Juan’s uncertainty about why he was enrolled and lack of purpose, when the experience created a significant financial toll on his family, Juan decided to take action and drop out to move to a new stage of his life. He, in essence, moved to what we call the “Help me get away” Job, in which students are looking to get away from a current job; break a current habit; or leave home and their family, town, or a particular relationship—to escape college. Although to many this might sound irresponsible because dropping out of college is seen as a “bad” outcome, it was a responsible decision given that Juan was moving through the experience in an aimless fashion, racking up debt.
So, what was success here? Enrollment of a low-income student by the school was probably held up as success from the supply side. But was it for the student? Dropping out was viewed as a failure from the supply side. Was it for the student?
There is tension between the supply-side metrics of what a good school is and what demand-side metrics—the outcomes desired by each individual learner—look like. Solving that puzzle is important.
One possibility is for schools to be far more precise about the Job to Be Done they serve. They should not strive to be all things to all people.
In our research, we’ve seen that when organizations optimize around one Job—the demand side, not the supply side—which features and functions are actually relevant become obvious. Organizations don’t keep up with their “competitors” in their particular product category just because they added a new feature. They add only what’s relevant for the Job they are set up to serve—and can strip out the rest. An organization’s structure will even change over time to mirror helping people accomplish the Job.
In the instances where we match the suppliers of education with the specific progress a learner is trying to make in a struggling circumstance—along with some other greater specifics around the learner and their desires—perhaps we can more accurately measure progress through the learner’s lens of what they are trying to accomplish and better optimize accordingly.
But I’m not sure.
It’s a puzzle worth tackling, though, because if institutions can match their supply-side metrics to what learners are actually trying to accomplish, then they will optimize their operations and organizational structure around the learner’s progress, rather than what the government or a news magazine or our traditional supply-side structure says is important.
