Post Category → data issues

data

Question The Numbers

The PISA results for 2015 were released this week. US students were ranked below the average of other countries. And their scores dropped from 2012, the last time the test was administered.

Cue the panicky headlines!

The stories, of course, are largely written by news staff who barely know what PISA stands for (Programme for International Student Assessment) and who have probably only read the executive summary. The stories I’ve read barely bothered to report precisely what the test was assessing (other than math, reading, and science) or none questioned whether it was a valid assessment.

On the other hand, Yong Zhao is someone who understands PISA and and other standardized testing systems, and he asks three questions that should be essential for anyone trying to put these numbers (or those of any standardized test) in context.

Does it accurately measure what matters?

How does it assess skills needed for an uncertain future with such certainty?

Does its results in reading, math, and science accurately capture the domains of expertise each individual needs for successful participation in the future society anywhere in the world?

Although Zhao’s reaction to the PISA results for the Post’s Answer Sheet blog is a little convoluted in trying to tie these test results to the failure of polling to predict the recent elections in the US and UK, he does finally arrive at the cautionary conclusion all those headline writers should have reviewed before pushing publish on their stories.

PISA’s confidence in the predictive power of its assessment also comes from the past. The subjects it chose to assess — science, reading, and math — have long been believed as important for success in life all over the world. They have been the core subject matter schools teach worldwide with the belief that they are essential for living in the modern age. But will these subjects turn out to help today’s 15-year-olds some 10, 20 or 30 years later? Are they the right candidates for all people in the future, or might different individuals need different sets of skills and knowledge?

PISA not only tries to use its test results in these subjects to predict what skills and knowledge 15-year-olds will need to succeed in the future, it also disseminates, based on these scores, education policies and practices it believes will equip children with these skills and knowledge. It has the potential to affect the livelihood of hundreds of millions of children, hence the entire world.

The consequences are serious. The stakes are so high. Therefore, we must question the quality of the PISA results before eagerly jumping to conclusions. Don’t read too much into it.

Whenever someone collects a large pile of data, especially about kids, question the numbers.

Personalized Learning by Facebook

If there’s one thing Facebook is great at it’s collecting and using data on “members”. Those skills are why the company is attractive to advertisers. So why not have the same programmers who built that attention-grabbing system create “a powerful tool that could reshape how students learn”?

That tool is called Basecamp, a joint project with the Summit charter school network, and is described in this Post article as a program that “tailors lessons to individual students using software that tracks their progress”. More personalized learning.

And personalized learning systems requires lots and lots of data to do the job.

But it also captures a stream of data, and Bilicki had to sign a consent form for her children to participate, allowing their personal data to be shared with companies such as Facebook and Google. That data, the form said, could include names, email addresses, schoolwork, grades and Internet activity. Summit Basecamp promised to limit its use of the information — barring it from being used, for example, to deliver targeted ads — but Bilicki agonized over whether to sign the form.

Question: if they promise not to use the data to deliver targeted ads, why is it being shared with Facebook and Google? Two of the largest distributors of targeted advertising?

Anyway, currently about 20,000 students in 100 charter and public schools are providing that data as the company is racing to have their product ready by the beginning of the school year next fall. A product that will compete with similar personalized learning systems from dozens of edtech startups.

Although the reporter tries to put a positive face on this story – starting with a headline claiming the software “shows promise” – there are so many things wrong with this project beyond the involvement of Facebook. Like this:

“There’s a lot of hype,” said Joel Reidenberg, a Fordham University law professor who researches student privacy. “In effect, they are experimenting on children.”

Then there’s the fact that the developers have very little evidence of the effectiveness of personalized learning systems.

“We really don’t know that much about personalized learning,” said Monica Bulger, senior researcher at the Data and Society Research Institute in New York.

Which applies to all the other companies on the hunt for venture capital to develop their version of personalize learning.

Not addressed in this story, of course, is whether the curriculum being “personalized”, how it is presented, and the pacing is appropriate for every child. Or if their learning from these systems will be meaningful enough to persist past the spring exams.

But I suppose none of these concerns are important as long as schools can boost test scores, administrators can keep their jobs, and investors are paid their profit.

At least in this case, a billionaire (Zuckerberg) is paying the bills. And Summit is the one organization in the world immune to potential data loss.

“We’re offering this for free to people,” she [Diane Tavenner, chief executive of Summit] said. “If we don’t protect the organization, anyone could sue us for anything — which seems crazy to me.”

I’m convinced.

Understanding Data

We’ve been told that all students should learn to code, in part because business will have a lot of coding jobs to fill. (“1.4 million openings by 2020] but only 400,000 computer science graduates with the necessary skills to fill the positions”

Now, according to one report, we also need to have kids learn data analytics because… jobs.

By 2018, the United States alone could face a shortage of 140,000 to 190,000 people with deep analytical skills as well as 1.5 million managers and analysts with the know-how to use the analysis of big data to make effective decisions.

Here’s a better idea.

Rather than turning K12 schools into training academies for whatever industry is feeling slighted this week, let’s help students graduate with a good understanding of how the real world works. How code makes their magic devices possible. How data impacts their lives. How they can have more control over all of it.

In this case, that means a solid awareness of how all those little bits of information are collected and used, too often misused, by corporations, organizations and governments. Especially the personal data they themselves generate, knowingly and not, in and out of school.

Then we can worry about the staffing problems of the Oceans of Data Institute.