Skip to content

This Method Acting, Well, I Call That Teaching

Deborah Netolicky (@debsnet) has written a response to the idea that I shared shared on Twitter that understanding a teacher’s development can be understood through the lens of art. In her post, Deborah reflected on this idea of teaching as art. This post is a response to her post.

olsen-this

Deborah begins her post by asking, “How can we appreciate an artist’s work or know an artist’s worth?”

I’ve been fascinated by this idea of understanding development through the lens of art for a while. At the Perezhivanie Symposium at the beginning of 2015, Michael Michell presented detailed Lev Vygosky’s love of the theatre. Vygotsky’s first PhD was on art (apparently Vygotsky questioned the quality of this research later), and he was also a prolific theatre critic, writing for the local newspaper. What is particularly interesting is that Russian theatre thanks to the ideas of Konstantin Stanislavski, was experiencing a huge shift at the time, due to the development of the Stanislavski Method, or what is now known as Method Acting.

Method acting (or the Stanislavski method) is famed for enabling actors to deliver powerful and compelling performances, as it doesn’t just focus on the technique of acting, but also the emotion of the role. The actor seeks to understand the motivations of the character they are playing, their motivations, their beliefs, and the essence of who they are. Actors who use the method acting technique, famously, might try not to break character between scenes or performances, as they seek to become the actor for the duration of the production or screening.

What is particularly interesting about method acting, is that when the actor takes on a role, they are not just interpreting the role, they are also interpreting the world through the lens of the character. For an actor, say playing the role of a violent drunken abusive character, the actor doesn’t just consider how this person feels and responds, they also consider how the world responds to them, in the present and also in the past. How did they get to where they are? What are the scars? What might have been the pivotal moments in their life? Where did they hope their life might have been? How has the world conspired to get them to where they are? An obviously, the period in which the play or movie is set is also a consideration. What was important then? Why was it important? What does this mean for what we value currently? What have we lost, and what have we gained?

Further, the way this actor understands the world through the lens of this character, gives us insight into how the actor views the real world. While the actor seeks to emotionally immerse themselves in role, experiencing the world through the character’s eyes and anticipating the character’s emotional responses, the actor exposes their understanding of the world. As such this fictitious world, shines the spotlight on the real world.

This lens on development doesn’t just illuminate a method actors development as an actor, but also development of the actor as a person. Something of course, the method actor knows cannot be separated. When we step back and look at a method actor’s career, we might also consider the types of roles they accept. What might the roles they take on tell us about how they understand the world, and how they view their own careers? Looking over a method actors career and the roles they take on and how they interpret those roles, might we be able to recognise change in their understanding, and as educators might this be a window into their development?

Of course, it isn’t just actors whose art and work provides us with a vantage point for understanding of their work. Picasso’s paintings allow us to understand his view of the world, of love and loss, the Spanish Civil War, fascism, Catholicism, and other world events and world views of the time. In music, Bob Dylan’s development as an artist obvious. From the folk singer singing covers, the celebrated pilgrimage to Woodie Guthie, the folk protest singer, the electric sellout, the born again Christian, and lately the celebrator of classic Americana music. Note, like Vygotsky I do not promote the view that development occurs in stages. Though, myself, I am comfortable understand a person’s development through periods that are unique to their development, such as the well-document periods of development of the life of Bob Dylan.

 

If we sought to understand a teacher’s practice as their art, how might we interpret it? Immediately, we begin to understand, through their practice, what the teacher believes about teacher identity and role. We’d consider what the teacher believes about students, and their capacity. We’d begin to understand what the teacher believes about the role of school and the wider community. Just as for the method actor, their teaching practice, their art, illuminates their beliefs, their world view, and their understanding.

What about the wider environment that influences teachers? Increasingly, we’re seeing a culture of performativity, what does a teacher’s art tell us about this? To other measures of teacher quality? To presence of computers and other technologies? Maybe, rather than seeking to reduce the understanding of who teachers are, and their development, we might seek to understand how they see the world through the lens of their practice. Further, as we get a picture of the period of development (again not maturational stages, but periods unique to their development), we gain a clear understanding for future developmental possibilities.

As such, the teaching environment and how the teacher responds to it, speaks more clearly of their development, than any skills or competencies that could ever be observed in a lesson, or deduced from a test. Particularly, the lens of critical current issues, but not through their response in totally.

For it is in disruption, crisis and the unexpected that defines these periods. Dylan’s visit to Woodie Guthrie. The war around Picasso. The performativity, globalisation, and new technologies around the teacher.

 

9/11 changed America. How Americans viewed themselves, and how they viewed the rest of the world. Shortly after, in 2002, Conor Oberst through his band Bright Eyes released the song Method Acting. Conor signs about watching this “shocking bit of footage” and “trying to make out the meaning.”  But, Conor isn’t trying to make out the meaning for America, he is trying to make out the meaning for himself, as an artist concerned with justice and humanity. Ultimately, Conor doesn’t make sense of the attack, which he doesn’t even explicitly mentioned in the lyrics. Because that isn’t the point, the point rather is how does make sense of himself. He stays determined to “keep the tapes rolling,” and to “keep strumming those guitars” for what is important is to keep a “record of our failures, we must document our love.”  For to Conor his life is “not a movie, no private screening. This method acting, well, I call that living.” Ultimately, his response as an artist to this terrible period for America is to understand it, the way he always has, by living it. In his words, it’s not method acting, it’s living.

 

I’m not that interested in trying to understand a teacher’s worth. Rather, I’m interested in who they are, what they are currently developmentally capable of, and what their future development possibilities might be. Similarly, to Conor, this method acting, well, I call that teaching living.

 

Poor research and ideology: Common attempts used to denigrate inquiry

Time to read: 5 minutes

A few months ago a prominent Melbourne University academic tweeted “Pure discovery widens achievement gaps” citing the paper “The influence of IQ on pure discovery and guided discovery learning of a complex real-world task” I was immediately dubious of this research, as research that is commonly quoted showing that the inquiry learning doesn’t work, is usually fundamentally flawed. I’m not a proponent of “pure discovery learning” per se, but I feel this type of research, and the reporting on this type of research is designed to denigrate all inquiry learning. In an attempt to leave only teacher instructional approaches standing – why these researchers don’t instead prove a theoretical basis for instruction is beyond me.

So I took a look at the research to see if educators should have any confidence in its reported findings.

TLDR: No, we shouldn’t haven’t any confidence in this research, and it does not show that pure discovery or inquiry approaches widen achievement gaps.

 

Not surprisingly, this research fails the good educational research test as it doesn’t use a learning theory. That is, the research does not attempt to justify a theoretical basis for its findings. The researcher does use two other (non-learning) theories though, to defend the research, notably game theory, and control value theory. In essence the author uses these theories to defend the research design, yet for some reason he does not believe a learning theory is also required? I find this baffling.

Why doesn’t the author believe that a learning theory is required to define the scope of the research, given that the research is about learning? Why does the author believe that theory is required to explain games, and emotional attainment?

Anyway, let’s look at the research, as I’m always interested in how this type of research is used to investigate inquiry learning, or in this case pure discovery learning.

The author defines pure discovery learning as learning occurring “with little or no guidance. Essentially, knowledge is obtained by practice or observation.” The author spends considerable time explaining how pure discovery occurs in so much of our lives, with ATMs and iPhones requiring people to use them correctly without instructions. He continues, in explaining how Texas Hold’em poker requires people to “use multiple skills to reason, plan, solve problems, think abstractly, and comprehend complex ideas” which are similar to real life pure discovery learning situations.

The author explains: “The poker application used for this study was Turbo Texas Hold’em for Windows, version four copyright 1997–2000 Wilson Software. This is a computerized simulation of a 10-player limit hold’em poker game.”

Wait!!! What????

A computer simulation of a game you play with real people is a suitable method for exploring pure discovery?

Texas Hold'em Archive.orgInterestingly enough, you can play Texas Hold’em, the software used in this research in your browser thanks to archive.org. (Note: If you’re using a mac use Function + right arrow when it asks you to press End to play.) In playing Texas Hold’em you will discover just what an poor attempt of simulating the playing of poker against nine other simulated people this really is.  It appears that the study data used in this paper is actually from a previous study by the same author, “Poker is a skill” dated 2008. The 2008 date still doesn’t explain why such old DOS software was used! In this paper the author explains that 720 hands of Texas Hold’em over six hours is equivalent to thirty hours of casino play, with real people as opponents. That is 6 hours playing against a computer is supposedly the same as playing 30 hours against real people!

If you play the simulation at archive.org it is easy to see how 30 hours of real play can be achieved in 6 hours using this simulator. Turns made by your computer opponents fly past with short text messages popping up briefly on the screen.  Two groups of students used this old software. The researchers provided the instruction group with instructions of a specific poker strategy, the pure discovery group were, for some reason, given documents detailing the history of poker! The success of players was determined by the money that they had won (or lost), though it should be noted that the participants were not playing for real money. Instead the highest ranking players were promised to be playing to a chance to be part of a raffle for an iPod. This was intended to place meaning, to the otherwise valueless money, each player was playing for.

So this study designed is to simulate a real life complex problem, yet it doesn’t even simulate a real life game of poker!  The participants were not playing against real people. The participants were not playing for real money (though their success was measured as if they were.) And… the participants were playing five times faster than real poker is played.

All of this should make anyone question how the author could possibly argue that this research design can possibly be described as “pure discovery,” as commonly used in real life situations. Interestingly, though the author identifies differences between the instruction and control groups. Neither group learned to play poker to the point where they didn’t lose money. Further both groups played many more hands, more than twice as many hands as poker experts are reported to recommend that “good” poker players play. That is, neither group exhibited the one of the main traits of good poker players, folding around 85% of the time, and only playing 15% of the time.

 

How might research of pure discovery be better designed?

Duh.

 

Playing with real people would allow a learner using pure discovery to observe, and seek to understand the decision making of other poker players. Depending on the relationship with the players the learner might ask questions of their opponents, seeking to clarify rules and strategies. Other players might also intervene in the play, offering advice and pointing out pivotal moments in the hand, and pivotal decisions being made by other players. If the participants played against real people, surely they would’ve noticed that they were playing many more hands (much more than twice as many) than their more skilled opponents? Though the computer might be able to simulate the logic of poker, it cannot and does not simulate the interactions between the players, a critical feature of playing any game, and especially poker.

Given that the instruction group did not learn to play Texas Hold’em poker to a satisfactory level, it is obvious that instructional strategies used did not work. Of course, it must be noted neither did the control group, who were left to battle the computer opponents on their own, armed only with a document on poker history. To suggest players playing against real opponents, using pure discovery or other inquiry approaches would also fail to learn to play poker satisfactorily, is obviously outside of the scope of the research, as the researcher did not explore this.

Of course, the lack of a learning theory is what has also led the researcher to his narrow definition of successful learning. Did the author ever consider why people play poker? Is money the only indicator of successful play? Or do people also play games for fun? Are the social aspects of playing with friends an important part of being a poker player?

A more complete understanding of what makes a poker player, a poker player, would consider other indicators, traits, characteristics and motivations.  Did the study participants continue playing poker after the study had finished? Did they enjoy playing poker?  Do they intend to teach friends? Do they feel playing poker with their friends strengthens their friendships? (Not that they were given this opportunity.) Have they developed their own theories and strategies they intend to try out in the future? What do they know about poker?

I believe a better understanding of poker players and the reason people play poker, would greatly improve this poor study. It would also provide further evidence for the worth of learning to play poker by playing poker with friends.  Not that this is an earth-shattering conclusion! After all isn’t that how we all learn to play any new game? Or maybe you’re the one out on the oval by yourself with a ball, a sheet of paper documenting the history of football!

 

Driven By Ideology?
To suggest that individuals playing Texas Hold’em against a computer mirrors inquiry that happens in our schools in complete nonsense.

To suggest that this research proves pure discovery “widens the achievement gap” is complete nonsense.

To suggest that learning poker by yourself on a computer playing against a simulation has anything at all do with student learning and real inquiry is nonsense.

Do academics who favour high levels of teacher instruction really expect us to believe that inquiry classrooms operate the same way that people learn to play poker individually on their computer?

Do academics who favour high levels of teacher instruction really believe that playing poker on your own against a computer tells us anything about how teacher professional development or teacher pre-service training should be designed?

Do academics who favour instruction really believe that a piece of paper with strategies on them is really the best way to learn anything?

Do academics who favour instruction really believe learning is just about knowing, and not about experiencing with others?

Do academics who favour instruction really believe we’re that gullible?

Is there evidence that Positive Education improves academic performance? No

Screen Shot 2016-07-20 at 8.23.50 AM

Time to read: 5 minutes

Lately there has been quite a bit of talk in education circles about social aspects of learning, particularly well-being, grit, growth and other mindsets, positive psychology and other social and emotional programs.

My personal opinion is that these are a tacit recognition by proponents of direct instruction, that their belief that learning and development is a linear cognitive approach of memorising skills is insufficient. Maybe they are starting to understand that development is highly individual in nature, it is not linear or maturational, and that it is a complex transition to qualitatively new understanding of concepts, new motivations, new relationships with others and the world, new directions, and new results?

Unfortunately, rather than reexamining the more appropriate learning theories of Vygotsky, Piaget and other dialectical approaches to development, these instructionists blindly continue down their misguided path co-opting bits and pieces into their flawed framework. Rather than design learning and teaching so that it IS social, they attempt to teach social as if it was a seperate discrete unit to other learning.

One such model is the Visible Wellbeing Instructional Model. Rather than admitting direct instruction (Visible Learning) and performativity (Visible Thinking) don’t work. They’ve now misunderstood the fundamental aspect of the idea that all learning is social from Vygotsky and Piaget, and instead tried to stuff it into their broken Visible Learning and Visible Thinking model in the hope that it will fix it.

How do they justify this?  Well according to them, Positive Education has been shown to increase student academic results by 11 percent.

Unfortunately for the Visible Wellbeing Instructional Model, this is simply untrue.

 

In 2011, Durlak, Weissberg, Dymnicki, Taylor, and Schellinger released their meta-analysis of social and emotional interventions. Notice that their paper is concerned with school based interventions, not a study of social and emotional practices that are embedded in standard learning and teaching practice. Their finding that is widely reported as evidence that these programs improve academic results is found in the abstract where they write:

“Compared to controls,  SEL (Social Emotional Learning) participants demonstrated significantly improved social and emotional skills, attitudes, behavior, and academic performance that reflected an 11-percentile-point gain in achievement.”

Seems clear cut right? Wrong!

If you, like me, and seemingly subsequent researchers who quote this research took “compared to controls” means compared to those who didn’t participate in these programs you’d be wrong, because that’s not at all what they are saying… Let’s read the paper further.

In Table 5, they specify the results of their meta analysis:
Skills 0.57
Attitudes 0.23
Positive Social Behaviours 0.24
Conduct 0.22
Emotional Distress 0.24
Academic Performance 0.24

Though I’m not a fan of effect sizes, as I believe they are completely flawed, consider what John Hattie in the book Visible Learning says about effect sizes:

“Ninety percent of all effect sizes in education are positive (d > .0) and this means that almost everything works. The effect size of d=0.4 looks at the effects of innovations in achievement in such a way where we can notice real-world and more powerful differences. It is not a magical number but a guideline to begin discussion about what we can aim for if we want to see student change.”
(Hattie, p15-17 quoted by http://visiblelearningplus.com/content/faq)

You might notice that all except one of Durlak et al effect sizes fall below Visible Learning’s guideline for beginning discussion about them. The only one is Skills (0.57) so according to their figures only worth of Social and Emotional Interventions are to develop social and emotional skills. Everything else atttitudes (0.23), positive social behaviours (0.24), conduct (0.22), emotional distress (0.24), and academic performance (0.24) fall a fair way below the Visible Learning cut off.

You’re probably wondering, where the 11% gain in academic improvement comes from, in light of its small effect size. To solve this one, we need to keep reading the paper.

“Aside from SEL skills (mean ES = 0.57), the other mean ESs in Table 2 might seem ‘‘small.’’ However, methodologists now stress that instead of reflexively applying Cohen’s (1988) conventions concerning the magnitude of obtained effects, findings should be interpreted in the context of prior research and in terms of their practical value (Durlak, 2009; Hill, Bloom, Black, & Lipsey, 2007).”
Durlak, Joseph A., et al. “The impact of enhancing students’ social and emotional learning: A meta‐analysis of school‐based universal interventions.”Child development 82.1 (2011): 416.

The mean effect sizes in Table 2 (Table 2 contains the same figures as above and broken down into further groups, such as class by teacher, class by non-school) do seem “small,” because they are small! Very small, so small Hattie would no doubt suggest you should ignore social and emotional programs, unless you’re teaching social and emotional “skills” (0.57).

But what do the author’s mean when they say “instead of reflexively applying Cohen’s (1988) conventions”?   I looked up the definition of reflexively… the Webster-Merriam dictionary gives the following meaning:

“showing that the action in a sentence or clause happens to the person or thing that does the action, or happening or done without thinking as a reaction to something”

Now I’m not a methodologist, like Durlak whose other paper is provided as a reference about why the effect size of the social and emotional intervention shouldn’t be understood by the effect size happens because of the intervention. Yet, it does seem a bit of a stretch (to a non-methodologist), to find what the methodologist is an appropriate method of determining its practical value.

Table 5

What the authors did, as far as I can tell as a non-methodologist, in order to “interpret the practical value of social and emotional interventions” is compare the results to other social and emotional interventions.

I’ll say that again, the 11% improvement in academic results is not compared to control groups who did not have interventions at all, they are 11% gains over students in other social and emotional type programs, and all students experience less improvement than those who did not participate in social and emotional programs.

We can see clearly from the last line of the table that the figure 11% was produced by comparing the effect size of 0.27 to four other studies with effect sizes of 0.29, 0.11, 0.30 and 0.24.

I’ve taken a quick look at these studies. They describe: 1) Changing Self Esteem in Children, 2) Effectiveness of mentoring programs for youth, 3) Primary prevention mental health programs for children and adolescents, and 4) Empirical benchmarks for interpreting effect sizes in research.

I must admit (as a non-methodologist) that I don’t understand why or how the fourth study “Empirical benchmarks for interpreting effect sizes in research” fits the criteria of “prior research” given that, from as far as I can tell it has nothing to do with social and  emotional programs. But what that particular research does describe is that typical effect sizes for elementary school are 0.24 and middle school 0.27.  On that research alone the effect sizes are either level or slightly above expected, hardly a ringing endorsement, nor a source of much faith in the 11 percentage points of academic improvement.

A rudimentary understanding of mathematics also suggests the extremely low effect size (0.11) of study into “Effectiveness of mentoring programs for youth” greatly increased the difference between the study in question and the “prior research.” I’d suggest if that study was deemed not fit the “practical value” of the study then the 11 percentage points figure would’ve been much lower.

So, it seems clear to me the 11 percentage points of academic improvement is determined by comparing it to previous similar studies which didn’t work as well. Any other measure would not have produced the same results.

 

Of course, to Vygotsky or Piaget these results would not be surprising. For they know you can’t reduce learning and development to individual traits instead we can only understand it as a complex system.  Maybe, the Visible Wellbeing Model is trying to move towards Vygotsky and Piaget? If so, they’re doing it wrong. By attempting to identify and promote three traits of teacher effectiveness, teacher practice, and wellbeing, they’re not seeing them as a system but rather three individual traits together. Yet, at the same time they’re only measuring one trait… test scores. And when you only measure one trait, guess what, the only traits that matter are that trait!

For Positive Education and wellbeing to ever produce an effect size that is substantial, what is measured would need to change, just as they did to produce the contrived 11% figure. But can what Visible Learning effect sizes deem important change? Could they decide what matters while still believing in “evidence”?

Such is the conundrum that the Visible Wellbeing Model finds itself in? Theoretical baseless, considering test scores only worthwhile, what it finds are worthwhile aren’t what they know are worthwhile… No wonder most of us still listen to Vygotsky and Piaget!

 

Personally, I believe the learning and development is social, so this post is not to belittle the wellbeing movement but rather to suggest reducing social and emotional to skills to be learned though programs and interventions is, in my opinion, a missed opportunity. Further, to think we can bolt on wellbeing in order to improve test scores, is to misunderstand how our students actually learn and develop.

 

Incidentally, Inquiry-based learning in the incredibly flawed Visible Learning meta analysis comes in at 0.35, maybe it is time it replace Positive Education, with an effect size of 0.27 as one of the three components of their model?

On Evidence, Research and the Need for Theory

Time to read: 4 minutes

Who is the best swimmer? Swimmer A or Swimmer B?

Swimmer A is part of a regular swimming squad, they can swim 500 metres in a swimming pool using all of the major strokes. Swimmer B can’t swim 500 metres but regularly swims in the ocean, mostly just old school freestyle with their head out of the water, but they can body surf and dive under a wave. They understand about tides and rips, and makes great decisions about when and where to swim. When Swimmer A visits the beach their parents keep a careful watch over them. Swimmer B’s parents are happy for them to go to the beach with their friends unsupervised, as they know their child is a safe, strong and experienced swimmer.

Quantitative approaches might identify Swimmer A as the best swimmer. They might hypothesize that the distance a swimmer is able to swim is the crucial measure of swimming ability. They might not even consider that understanding the relative safety of surf conditions is important, with all their field research of swimming occurring in suburban swimming pools. Qualitative approaches might observe both swimmers at the beach, and come to vastly different conclusions, finding distance as an unreliable measure instead understanding swimming as a far more complex set of skills and characteristics.

Of course, believers in quantitative evidence (test scores) might try to point out that their research (tests, surveys, experiments) are much better designed than my simple example above. They might point to a curriculum as being able to define what needs to be tested, suggesting that well designed quantitative evidence (test scores) are better than qualitative data. But how can we know this is true? How can we know that the evidence that is captured is able to measure what it claims to be able to measure?

In short we can’t.

This is where learning theory steps in.

A learning theory determines how learning and teaching designed, and how development of learners can be understood. The learning theory defines the rationale and process.  We cannot have evidence without it. The theory defines what the evidence is proving or disproving. It explains why a measure of swimming distance is or isn’t a suitable measure. Also, theory defines what it isn’t measuring.

So what makes a good learning theory?

Believable educational research and evidence must pass ALL five of the following tests.

 

1. Is there a learning theory?
If there is not an acknowledged learning theory which the researcher is using appropriately, then do not believe the evidence at all. A learning theory has a theoretical basis, we are not talking about the methodology or approach. The curriculum is the curriculum it is not a learning theory.

Research and evidence that use meta-analysis approaches, research that simply uses test scores or pre and post tests usually fail this test. Research from fields other than education always fail this test. Sadly, research that fails this test is the worst type of educational research, and probably makes up more than 90% of all published research on learning and teaching. It should all be thrown out.

 

2. Does the learning theory explains how learning/development occurs?
At the heart of learning/development is a change in the learner. How does the learning theory define this change? Does the theory assert that learning/development is an individual trait, such as distance swum, or is it a combination of traits such as decision making, experiences, and other characteristics. If the research is making claims about things that the learning theory does not view as  being important indicators then the evidence cannot be believed. Trusted evidence reports only on areas the learning theory views as being important.

Research that uses frameworks such as TPACK and SAMR fail this test, they actually fail the first test as well but people don’t see this. Research that suggests student progress can be reported according to months and years behind usually fail this test. Research that uses purely quantitative assessments usually fail this test. Research claiming to disprove other pedagogical approaches often also fail this test as they attempt to use the measures of one theory and apply them to another. Research proving or disproving the usefulness of specific technologies often fail this test.

 

 

3. Does the learning theory explain the relationship between teaching and student learning/development?
At the heart of learning and teaching is a belief that well designed pedagogy produces a desired change in the learner. In assessing learning to gain evidence, we are ultimately assessing the effectiveness of all the important facets of the learning and teaching process. Does the theory make a clear rationale for why the factors highlighted in the evidence are actually important facets of the learning and teaching process? If the research and evidence seeks to make claims about learning and teaching design that the theory does not make clear as being crucial then the evidence should be dismissed. Trusted evidence reports only on areas that the learning theory asserts are important.

Research and evidence on engagement, grit, flow, and other motivations or character traits often fail this test. Research on teaching approaches usually fail this test if there are reporting on facets of teaching outside the scope the learning theory.

 

4. Does the learning theory align with the study?
Of course, the learning theory doesn’t need to justify that studying a trait, or a combination or unit of traits/characteristics/functions is theoretically accurate.  The research also needs to demonstrate why the studied trait(s)/characteristics/functions appropriate for the specific learning/development. As such the appropriate use of the learning theory is crucial to understanding whether evidence is trustworthy.  If the learning and teaching process or design is being studied, the evidence of the research are limited to that specific process or design and cannot be extended to other learning and teaching processes and designs.

Research into the suitability of teaching strategies, media, and specific technologies, by assessing student comprehension often fails this test.  Research around knowing “what universally works” in all situations always fails this test.

 

5. Does the learning theory in its entirety accurately explain learning/development?
Of course, the biggest test is, if there is actually a learning theory being used to support the research and evidence, does the learning theory make pedagogical sense!  Learning theory must align with how we know learning actually happens. Learning theory shouldn’t be separated from reality, it should fit completely with our everyday experiences of how people learn. This is not to say that learning theory shouldn’t be complex! Complex ideas need complex theories, but complex does not mean we need to make a leap of faith in order to believe or understand them.

Research that uses cognitive load theory and other “how the brain works” rationales usually fail this test.

 

Footnote: I know it is tempting to take notice of bad research and evidence which we agree with, but please don’t do it. We need to throw away all bad research even when we agree with the “evidence.” 

Why coding is the vanguard for modern learning

Time to read: 20 minutes

Note: Sorry this post is so long, I do have to start writing shorter posts… also I’m publishing this as is and plan to edit and improve the post over time, largely to clarify ideas that may be difficult for non-coder to understand. Please excuse the typos and other errors.

 

Screen Shot 2015-12-31 at 8.23.02 AM

Coding is coming to all of our schools with a lot of hype, inevitably there is push back from many different quarters for a variety of reasons… In my view these push backs are uninformed.  The code I know is the vanguard for modern learning, and for this reason, in my view, anyone interested in learning should take notice, and seek to understand, what is happening.

 

The Importance of the Learning Environment

We get so attune to curriculum and assessment as it is done in schools, we can begin to think that all learning, no matter how it occurs, produces the same results and the same outcomes. Nothing could be further from the truth.

It is fairly obvious that you can’t learn to play golf out in the ocean, and that you can’t learn to surf on the golf driving range. It is less obvious however that the learning environment in which, say math is learned, has the same effect on the developmental possibilities, not just the pedagogical effectiveness. Too often when we think about pedagogy, we think about it as if the outcomes of learning are fixed regardless of the pedagogical approach that is used. Our curriculum is so entrenched in school that we take it on face value that its progression points are natural and irrefutable. We cannot fathom, for example, that memorising time tables might not be required for every approach to learning maths, that there might be learning approaches and learning environments where students can develop deep understanding and the ability to use complex mathematical concepts and yet not be able to recall their times table facts in a fraction of a second.

It would be remiss of me to write about the learning environment without beginning with Vygotsky, who writing in The Problem of Development says that “the environment is the source of development.” The environment doesn’t just dictate the quality of development it dictates the actual development. In looking at the development of children, Vygotsky uses the development of speech to explain how the environment is crucial, and the source of development.

Seymour Papert’s vision for LOGO was as an immersive learning environment to experience and learn mathematics, and similarly to Vygotsky, Papert also spoke of learning [mathematics] in terms of learning a language. Papert worked towards creating a learning environment where students could learn not only as naturally as children learn french in France but also as completely as they learn french.

Papert argued that “school french” was very different, not only in form but also in content, from french spoken in France. In a french class in an english speaking country students are exposed to a small number of french speakers, and the differences between these speakers are minimal, in France learners are exposed to many different french speakers. In french class the vocabulary is static, in France not so. In France slang, idioms and vagaries of language (dependent on the speakers age and geography) are frequently used, in french class not so. Learning french in France isn’t just better because it is immersive, it is better because it is the real thing, french in french class is not the same learning environment and has vastly different, and poorer, qualitative learning outcomes.

LOGO wasn’t designed to teach students the school maths curriculum, LOGO was designed to produce vastly different and qualitatively better mathematical learning outcomes. LOGO enables students to pose mathematical based hypotheses, school maths doesn’t. School maths requires students to memorise time tables and algorithms to be successful, LOGO doesn’t. LOGO enables students to test their hypotheses themselves, school maths has the answers in the back of the textbook. LOGO isn’t a better way to learn school maths, it is better way to learn real maths. The result of learning maths in the LOGO environment to learning maths from a maths textbook produces qualitative different results, they aren’t two roads to the same end, they are two roads to different ends.

LOGO isn’t the only mathematical learning environment that enables qualitatively different results to the usual school math. Many schools, for example, use play based learning, such as playing shop, to enable children to explore mathematical concepts through buying, selling and the exchange of goods. Yet it would be to misunderstand the play-based learning environment to view it only through the curriculum progression points of patterns, sequences and the usual associated strategies. Instead you’ll most likely find, that these children are understanding and using traditionally more complex mathematical concepts such as supply and demand in order for them to play the game of shop. The mathematical play based environment enables completely different learning outcomes for these children that are not possible when children learn mathematics in traditional maths classrooms. Just because children can learn these different, historical more complex mathematical concepts, in LOGO and play-based learning environments does not mean it is possible that it is possible for all environments to produce these outcomes, just like the golf course and the ocean, different environments produce completely different learning outcomes.

Papert though it must be noted didn’t see the computer as a device for learning math, he saw the computer as an instrument whose music is ideas!

Of course, our understanding (or rather our lack of understanding) of mathematics makes it difficult to fully understand the magnitude of the what is possible in play-based, or LOGO based mathematical learning environments as opposed to traditional mathematics classroom approaches. When we reduce maths to the content outlined in the curriculum, and not the development of students who can expertly use maths to solve problems and make good decisions, we no longer need to care about how maths is best learned. Even worse, as we’re seeing with the current debate about coding, when we focus on the content and its merits, we engage in silly debates about which content has more value than other content, as opposed to what are the characteristics and motivations of the students we’re graduating and what problems will they want to, and what will they be able to solve.

Consider how fairy tales are used in classrooms to enable children to experience heightened emotional experiences, such as fear, sadness, elation, empathy, and so on, in a contrived, completely artificial situation. Without the contrived environment that fairy tales make possible, children wouldn’t have opportunities to learn to understand and express how they feel, and instead parents and teachers would need to wait until heightened emotions are present due to real life concerns. As such fairy tales offer developmental possibilities that are unique, and not possible in other learning environments. Not to mention much more appropriate and effective, and not to mention prepare children to emotionally cope with real life crises.

 

Given that when we change the learning environment, we change the possible learning outcomes, it would be reasonable to suggest that searching for new learning environments that offer qualitatively improved developmental possibilities would be a major focus for educators. Rather than trying to improve the effectiveness of existing learning environments, such as our traditional maths approaches, we should rather be taking a leaf from Papert’s book and pursuing new, qualitatively better, learning environments.

The web, by an order of magnitude, is a qualitatively better learning environment that what most learners currently have. Sure we can try to understand it in terms of traditional learning environments, but to do so is to completely underestimate the developmental possibilities. Possibilities that were previously unimaginable and impossible.

Coding, programming, software development (whatever we are calling it today) have always driven this new learning environment, as they sought to create the web we now have. Driven by their own learning and work needs these modern learners really are the vanguard for modern learning, and to learn how they learn is, in my opinion, currently the best way to understand the modern learning environment that is the web. I’m calling this learning, modern learning (borrowed from my friend Bruce Dixon), and not life long learning, or even just learning. I’m doing this to make it perfectly clear, that the learning outcomes from learning how to learn in the coding environment and with the current software development tools is qualitatively different to learning how to learn without the environment of coding. Modern learning is not possible without modern tools and modern learning environments.

 

Not that all coding is a magic bullet, and educators (usually with good intentions) can design learning environments that include coding but omit what actually makes it powerful learning. Just as educators might play maths games to help students memorise number facts, or use LOGO to teach recursion, or use edu-games for drill and practice.

 

The Importance of Tools in Development
If the learning environment is crucial, the tools we use are just as crucial. Tell a blind person that their walking cane is just a tool, a deaf person that their hearing aide is just a tool, or person who wears a prosthetic limb that it is just a tool. Tools enables us achieve what would otherwise be developmentally impossible. Tools enable the blind to see, the deaf to hear, and the incapacitated to walk.

Of course, it isn’t just using a tool that is developmentally important, it is using the tool to achieve success in activities that were previously impossible that makes tools developmentally important. Communication and sharing via the Internet has remarkably changed coding, but equally apparent are the tools that have developed and are now what makes software development as we now know it possible.

From open source to design patterns to devops to test driven development to MWWV frameworks to continuous integration to whatever is next on the horizon, new tools are produced that in turn produce new ways of working which in turn produce new beliefs and practices about software development. How coders now work and code is far removed from how they worked only a few years ago, and what is certain is that this trend will continue.

 

In an attempt to make sense of how developers are learning to develop, I am proposing loose learning environment types, as a way of understanding how learning to code isn’t acquiring skills but rather how coding enables development that is not possible without coding. As I do, I highlight the important developmental role tools play in each environment. Hopefully, this process will enable other educators to understand why many of us are excited about coding.

Note: I’m not going to switch from referring to code and coders to (software) development and (software) developers, which is a more accurate description of what coders do. However this does also pose a problem as I’m more interested in exploring the development of the developer (as a learner), as opposed to the development (of products) by the developer!

 

1. Feedback-Rich Learning
At its most basic, learning to code is learning to write scripts to solve computational problems.  To understand what a variable is, to learn to use if else statements and create logic loops, understand what functions are and what functions are commonly available, and how to write our own functions. This might be a simple line of code that splits first names and surnames in an excel column into two seperate columns, it might allow us to make a game in Scratch, build a redstone powered machine in Minecraft, or invent and create Facebook.

Code enables us to experiment. To imagine solutions or creations, and then see if we can in fact make them. Code enables us to solve problems that would otherwise be beyond our capacity because they would take far too long, or be difficult not to introduce mistakes. The computer language provides precision and consistency to a level beyond what we can usually do by hand.

The feedback that a coder receives as they code is immediate. Almost from the first single line of code, a developer can run their code and see the results. They receive feedback on what is working as expected and what isn’t. Papert’s LOGO and then later the very similar MicroWorlds, and now with Scratch, began with turtle drawing, drawings that we created by following the path of the path of the LOGO turtle. Scratch has since replaced the turtle with a much more Internet acceptable cat!

Screen Shot 2015-12-23 at 10.19.09 AM

The screenshot above shows the Scratch editor and interface. Coder select commands from the middle pane, drag them into the right pane to create the program, and then observe the results of the program in the left pane. Modify the code blocks on the right, click the green flag at the top of the left pane and immediately see the results of the code.

In this way coding bring mathematics to life. No longer is the truth sealed in the pages at the back of the text book, now the truth can be exposed click of the run or refresh button. For educators who understand the importance of feedback, the way students can test their mathematical ideas and immediately get feedback on these ideas should be compelling. Programming languages designed for kids like Scratch (shown above), Kodu Game Lab, and even Minecraft redstone don’t require typed code and therefore don’t need to provide additional methods beyond visual and audible checks.

Browsers, for example, via their developer tools enable coders to check whether their code is running as expected using a number of different tools. Looking at a browser’s developer tools is like lifting the bonnet on the car to see how the engine running. All modern browsers have developer modes which let you inspect the DOM by right clicking on the area of the screen that you’re interested in, shown below in the left pane. This shows the structure of the underlying html, and also allows designers to see the CSS markup that dictates the design elements, shown below in the smaller right pane. Some browsers also allow you to manually change the html and css markup in order to view the changes immediately without needing to refresh the browser window.

Screen Shot 2015-12-26 at 8.26.54 AM

Browsers also show the files that they use to build the page, this can be useful on a simple level to discover if all of the expected files are being successfully retrieved, because the filename or location isn’t correct, or some other server error.

Screen Shot 2015-12-26 at 8.35.20 AM

 

If schools wish to develop learners who can use new ways of experimentation and feedback they need to encourage learning environments in which students can use code to solve real problems. Of course, feedback on code also comes in more traditional avenues. The cost of obtaining feedback, in both time and money, is now extremely low and many organisations now use experimentation and feedback in place of detailed pre-planning. Eric Ries a leading thinker in entrepreneurship says learning is the only useful unit of progress for startups. Harold Jarche a leading thinker in personal knowledge management explains that “learning is the work, and the work is learning.” It must be noted that teaching is also headed down the feedback path with many education experts suggesting teachers can improve their practice by obtaining feedback on their effectiveness from their students.

Of course there are many other learning environments where feedback is also rich. Blogging is celebrated for the way it facilitates the exchange of ideas and provides feedback. YouTube, Twitter and other social media allows offers rich forms of feedback. Wearable technologies being used are being used in some universities to provide a rich source of feedback. Yet, none of these environments even come close to the feedback-rich environment that is coding

 

Unfortunately, when you start to understand how feedback-rich the environment is in which coders code, you begin to understand why educators don’t understand its power. As educators we’re obsessed with students mastering the single best strategies (assuming we actually believe there is more than one appropriate strategy.) Mottos such as “fail = first attempt in learning” sound as if teachers understand modern learning, yet failing isn’t necessarily feedback, you can fail and not learn anything. It also suggests feedback only happens at the conclusion of the learning activity, as opposed to throughout it.

Similarly, feedback from teachers, particularly when given as explicit instruction isn’t informative it is authoritative. Feedback-rich learning informs why something works or doesn’t work, or at least allows the learner to discover more on why an activity didn’t work as expected, as opposed to textbook learning which is only concerned with right and wrong.

 

2. Reuse-Rich Learning
Free and open source software and its derivatives (such as Creative Commons) have made the web that we now have possible. Without open source we wouldn’t have the web that we have today. Software development, as it is commonly now done, particularly creating apps and developing for the web, builds on the work of others. Unless the work is simple and trivial, a developers unique code is often only a small percentage of the code base. Open source licensing and its culture of reuse, makes this possible.

A typical website today will most likely build upon numerous open source code. A typical website might run on, or on variants of, the LAMP stack, and use backend and front-end frameworks, javascript libraries, css frameworks, icon libraries, as well as other code sourced from blogs, github and question and answer sites. All these pieces used by the developer to make a well functioning website, that would not be possible to create, due to complexity and time, without them.

WordPress, one of the greatest examples of building a business based on open source, is a wonderful model of how software architecture can promote reuse. Like 25% of the Internet, this site is a WordPress site, I haven’t written any of it’s code but if I wanted to change the functionality of the site it is easy to do. Which is why WordPress is so popular, developers use it to create all sorts of websites that were not part of the original WordPress blogging vision.

How does WordPress do this?

WordPress provides two main ways to modify it’s functionality, themes and plugins. It is crucial that WordPress provides these methods otherwise changes to the existing WordPress code would be lost every time an upgrade for WordPress is released. WordPress separates the presentation code, as themes, from the core code that provides the base functionality. This allows theme designers to easily customise the look on feel of the WordPress website by only replacing certain functionality.  If you’re happy with how are part of WordPress looks or works then leave it, if you’re not happy with how it looks or works write code to change it. And as a result tens of thousands of plugins and themes have been created and shared for other WordPress users to use.

The major way plugins and themes build on or change WordPress’ functionality is through its system of hooks and filters. WordPress theme files uses standard WordPress template tags to fetch and display the appropriate content to the browser. To display the title of the blog post the theme developer uses <?php the_title() ?>, to display the blog post’s content (text and pictures) the developers uses <?php the_content() ?>.

Say for example, I wanted to put a message at the bottom of each and every blog post on this blog.  I could edit every blog post or I could write a simple plugin that changes the post’s text before it is displayed. The code might look something like this:

add_filter( ‘the_content’, ‘thanks_for_reading’ );

function thanks_for_reading( $content ) {
$content = $content.”<p>Thanks for reading! If you liked the post, I’d love to hear from you by way of a comment.</p>”;
return $content;
}

I’m not going to step the reader through the code, for this is not the point of the article. The point is to demonstrate how WordPress is designed to be easily modified. The WordPress developers expect other developers will want to and need to modify the functionality of WordPress, and they have designed WordPress to make this really easy to do. WordPress has thousands of filters and hooks through which its functionality can be modified in order to meet the needs of the developer. It also has a replaceable theme and modifiable short code system that developers can also modify, but that is beyond the scope of this post…

Of course it should also noted that WordPress, like all web code, uses a lot of code created by other developers. They don’t just expect others to use their code, they also use the code of others.

jQuery is a javascript library that provides developers with a range of powerful functions for manipulating (showing, hiding, animating, searching) webpages and responding to user actions, such as clicking and dragging. It is used by a large proportion of all websites. If you visit BootStrap or Font Awesome you probably recognise what you see there. Interface components (buttons, menus, alerts, drop downs) and icons to create beautiful and functional websites, that work on both computers and mobile devices.

It would be wrong to assume that the code that reused by programmers is limited to large projects packaged for others and produced by organised teams. GitHub contains a staggering 2 million active repositories, which coders have used to share their code. GitHub makes it easy for anyone to grab a copy of someone else’s code (this is known as a fork) and make changes to it, enabling them to modify the code to suit their specific needs.

The reuse that happens in software development isn’t just limited to code. In 1994, the Gang of Four famously release the influential book Design Patterns: Elements of Reusable Object-Oriented Software, which presented 23 design patterns. A design pattern is a tried and tested reusable solution to a common software development problem. Typically a design pattern would also describe the rationale including pros and cons of the solution as well as the solution itself. As such, design patterns enable developers to focus on design decisions unique to their needs as opposed to designing solutions for common problems, such as connecting to a database.

The usefulness of design patterns outside of programming have been explored by others in a variety of other domains: a wiki of game design patternsbusiness model patterns, and even pedagogical patterns, tough note I’m not vouching for the quality of the pedagogical patterns! GitHub shows even more promise, examples such as GitHub for government, for writing a novel, and creating documentation that describes the standards that run the web.  Open source software licensing has inspired Creative Commons licensing where over 1 billion licensed works such as photos, writing, and other digital products have been licensed by their creators that allow others to remix and reuse, just as open source code is remixed and reused.

 

It would be easy to view reuse-rich learning, as simply saving time and effort, this is to completely misunderstand what is happening. Lawrence Lessig describes a remix and reuse culture as an ecology of creativity. For developers remixable and reusable code and design patterns offer opportunities to solve problems and create solutions that are outside their current developmental scope. When coders use jQuery they can create sites beyond their level of skill, when developers use BootStrap’s components they can create functional and visually beautiful sites that they wouldn’t be able to otherwise create, and when developers draw on solutions of known design patterns they can have confidence in a design solution without needing to exhaustively test its suitability.

Reuse-rich learning environments therefore allow learners to engage in successful activity beyond their current developmental possibilities. Which might not seem to be a big deal, except for the fact that is how our students need to learn outside of school. If we want to develop our students as learners who will be able to successfully operate in fields where as Howard Jarche says learning is THE work, we need to prepare them.

 

Screen Shot 2016-01-25 at 10.30.40 PM

The image above shows a Scratch sprite that can be saved (and then shared and imported) individually.

Of the programming languages designed for kids and education, Scratch is the only one I know of that is designed to encourage reuse and remixing. Most educationally designed programming languages let you share the whole file, however Scratch allows you to share and reuse the individual sprites containing images and code. Strangely, and disappointingly, though Scratch doesn’t allow you to share the stages in the same way. There are a lot of YouTube videos showing redstone programming creations, which is a poor form of reuse, hopefully down the track MineCraft will find ways to encourage reuse and remixing. You can see how I used reusable when I was a teacher Scratch sprites here, you can see how I used games design patterns to structure a course on designing games with Kodu Game Lab here.

Screen Shot 2016-01-25 at 10.36.17 PM

The image above shows a description of a Big Boss game design pattern found in Kodu Game Lab course.

Screen Shot 2016-01-25 at 10.37.29 PM

The image above shows a screenshot of the website the students used to share and remix Scratch sprites as they created maze games.

When you start to understand how reuse-rich the environment is that developers are developing software in, it is easy to understand why it educators might not want to understand them. Schools and educators are obsessed with individual knowledge and individual learning. Sure we might use collaborative activities but only so individual can learning how to collaborate…. individually.

 

3. Opinionated Learning
In the same way that code is shared for the reuse and betterment of all, so are tools and processes. Most tools and processes that are highly opinionated, that require (or even force) the developer to work in a certain way. Where workflows once dictated how developers choose tools to increase their productivity, increasingly developers are choosing tools based on ideological beliefs about the best way to solve complex problems.

Tools are still evaluated on how well they work, but now they are also evaluated on how well they fit and promote a developers philosophy of development. This has been most apparent in the uptake of GItHub and the move towards agile and continuous delivery as opposed to a linear or waterfall approach. Similarly, web-development in the last few years has seen a huge ideological shift in the rise and rise of single-page frameworks and the node.js server.

AngularJS (created at Google), BackBone and Ember appeared on the web development scene at roughly the same time, and all positioned themselves in contrast to jQuery. These frameworks appeared as a response to the need to create websites that acted more like apps. Rather than clicking links and fetching a whole new webpage from the server, these frameworks enabled the creation of single page webapps. They were all preceded by jQuery mobile, yet jQuery mobile, over time was largely scorned, because it doesn’t enforce a architectural structure. Angularjs et al do force a structure, specifically the Model-View-Controller structure. The Model View Controller is an architectural pattern (similar to the design patterns mentioned earlier) that segments the code base into three distinct areas, the model (the underlying data), the controller (the logic), and the view (the display.)

Screen Shot 2016-01-25 at 9.52.47 PM

The image above shows the MVC architectural pattern (taken from wikipedia)

Proponents of MVC frameworks, eschewed jQuery and its lack of clear design architecture, believing their code was not only quicker to create but also less likely to have bugs, easier to understand and maintain due to its superior underlying architecture. The supremacy of these MVC frameworks didn’t last that long, largely due to the concern over the speed of MVC frameworks particularly when displaying large amounts of content on mobile devices. Facebook famously stated that web apps would never be able to compete with native (downloaded from the app store) apps, that was until they created their own framework, which they called React.

Interestingly, though not surprisingly, the creators of React didn’t just create a framework they also created and released a new architectural pattern, which they named Flux. The team behind React knew they couldn’t just expect developers to use React because it is faster (and it is apparently a lot faster), they also knew they needed to communicate why they believe it is architecturally and philosophically superior.

Screen Shot 2016-01-25 at 9.53.14 PM

The image above shows Flux’s architectural pattern (taken from the Flux documentation.)

Over the last five years in web programming we’ve witnessed a shift in the web development community from jQuery to AngularJS (and its variants) to React. This has been a shift in the use of tools, but more importantly a philosophical shift. Not that everyone agrees with these shifts and philosophies, but I think it is fair to say that most do. Architecturally opinionated frameworks are the only opinionated tools that coders use, increasingly developers are using a range of opinionated tools to test, manage external libraries, and even tools to manage these tools.

 

Obviously, all tools are opinionated, for example design thinking is philosophically based on user’s point of view, and lean startup entrepreneurship is based on the philosophy of reducing waste while searching for a viable business model. Yet, what is unique about coding is the speed in which current philosophies are being questioned and new philosophies are emerging.

bmlloop

The image above shows Lean Startup’s Build Measure Lean loop.

 

Unfortunately, when you start to understand how opinionated the learning environment of developers is, you begin to understand why educators don’t get it. In the classrooms of our schools, the teacher is the point of truth, the expert. Students aren’t able to use the tools and architectures they use, in fact schools usually don’t want to let them know there are philosophical and design choices they can make. They are taught that there is a single right way to write a narrative or a persuasive text, or to solve a mathematical problem.

 

4. Continuously Evolving Learning
There are two reasons why the philosophies of web development are evolving. Firstly of course, is the continual advancements in the underlying technologies, but equally important is that developers are making their own tools to solve philosophical problems. It should be no surprise to learn that AngularJS was initially created a single developer to meet his own need, or that React was created by Facebook because web apps couldn’t compete with native apps. React needed AngularJS, in same way that AngularJS needed jQuery, neither of them were created from a theoretical or philosophical vacuum, new philosophies are created in the response of the shortcomings of the previous philosophies.

Don’t be fooled though into thinking though it is only the big companies and rock star programmers creating these tools, this is certainly not true.

It is because programmers can create the tools to challenge old philosophies and develop new philosophies. If you can believe that software is eating the world, and a quick look in any of the app stores or conference exhibition halls would suggest this is true, then it stands to reason that many domains will continue to be redefined (and philosophies shifted) by software-based tools.

From consumers to creators to (opinionated) tool makers.

In my view, increasingly we will witness a differentiation from those who can create new tools and those who can’t. Tools enable learners to succeed where they cannot succeed without the tools. Tool use also enables development (Vygotsky proved this a long time before the Internet), the development that results from the use of tools, in turn enables the possibility of future higher forms of development.

Coding therefore not only provides our students with opportunities to use and develop opinionated tools within the activity of coding, and to develop as such as modern learners. It also provides them with skills and experience in creating opinionated, philosophical based tools that can be used to transform other areas of learning. What sort tools? I can’t wait to see!

 

Unfortunately, when you start to understand how continuously evolving the learning environment of developers is, you begin to understand why educators don’t get it. Continuous evolution doesn’t fit, nor is it even slightly desirable in our curriculum model. What’s on the test is what matters, and what’s on the test doesn’t need to change.

The sad fact is that most teachers aren’t pedagogues, they’re soothsayers. They’re able to predict the future, not because they can, but because they don’t really believe what really matters in learning changes. They don’t really believe that our students might need new tools, or that creating new tools is even possible.

If teachers saw themselves as pedagogues rather than soothsayers, they’d stop making predictions of the future, such as “everyone doesn’t need to code,” but instead start to understand how technologies create new pedagogies and change exisiting ones. They’d read Papert’s Mindstorms, and seek to understand constructionism, the learning theory he developed and promoted. They’d also investigate Siemens’ Connectivism, a second learning theory that is inspired by technology… they might even start creating their own (opinionated) tools!

Evaluating expert advice on schools and learning

Time to read: 3 minutes

Note: I’ve already blogged about the criteria that I use to assess the quality of formal education research, this post builds on those ideas to present an ethical method for evaluating education ideas that come from other sources.

There has been a bit of discussion lately about whose research to believe. Some have suggested that Kevin Donnelly and Wiltshire shouldn’t be considered experts. Others have suggested that teachers shouldn’t be considered researchers. I don’t like these suggestions. I don’t like them at all. Firstly, I don’t think whether Donnelly and Wiltshire are experts or not, or whether teachers are researchers, changes the validity of their arguments whatsoever. Experts and researchers, are labels and have nothing to do with the arguments the person is making.  Secondly, evaluating arguments and findings based on the person making the arguments, as opposed to the arguments themselves, opens us to the whims of the experts and researchers we put our faith in. It also prohibits us from learning from others we disagree with.

Also, a few weeks ago during a some what robust discussion on Twitter, someone quizzed me on my experience as a teacher, their argument was that if I hadn’t taught my opinions on schools shouldn’t be listened to. When I refused to answer their questions about my experience, they took it as me admitting that I’d never taught… which was untrue, I refused to respond, not because I hadn’t taught (I have), but because I don’t think anyone’s arguments should be evaluated by anything except the strengths of their arguments. (Note: If you’re still confused about my teaching experience and are desperate to know, you’ll find me on linkedin, but please don’t try to use my experience to win an argument against me! If you want to argue with me, argue against my beliefs not against who I am or what I have or haven’t done.)

My point is. If we need to resort to trying to discredit someone just because we disagree with them, then the ethical thing to do is to construct better arguments. While it might be beneficial (for the majority in schools) in the short term to have Donnelly and Wiltshire labelled as non-experts and therefore have their ideas dismissed out of hand, the better response to understand where Donnelly and Wiltshire are coming from and what are the fundamental differences that we have with them. Not only does this approach enable us to articulate a counter narrative, it also enables us to have a clearer understanding of what we believe about learning and teaching. Of course, the third, and often better, option is just to ignore!

Of course constructing a counter narrative can be hard, particularly when many commentators, such as Donnelly and Wiltshire hide their true beliefs behind outrageous language and exaggerated examples.

Knud Illeries provides a useful model for understanding and comparing the different approaches to learning with his Three Dimensions of Learning. There are a few variations of Illeries’ model but basically he suggests how we believe people learn can be mapped against two axis, content to incentive, and individual to social. You simply, pick a point within the triangle that best represents what you believe about learning.

illeris-1

Illeries model is useful in that it enables us to identify some of core differences between what different people believe about learning (and teaching.) While, I wouldn’t recommend using this model to adequately explain the differences between the various educational theories and theorists as some try to do, it does help us to start identifying where we agree and disagree with others, which in turn enables us to construct and articulate a well formed argument.

So where do Donnelly and Wiltshire sit on Illeries Three Dimensions of Learning?

Again, pulling some quotes from the article that has incited so much angst

“kumbaya” childish and emotive but suggests they don’t view learning as being social, rather solely individual.

“progressive new age fads” just ignore this, it just emotive and meaningless.

“blames on the fact students have been handed autonomy” again suggests an individual view of learning but also suggests they see content being far more important than incentives and emotions.

“It’s a good idea to have self-discovery but kids need to have knowledge” more evidence that content is far more important than emotions/incentives, but maybe the authors aren’t on the absolute extreme?

illeris-3

Reading that newspaper article I think it is safe to place Donnelly and Wiltshire as viewing learning as an individual pursuit that focusses entirely on content. As such rather than attempting to argue with them about students are “rolling around on the floor” and whether schools are too “kumbaya” we would be better suited arguing about why we believe emotions and motivations are crucial to learning, or why we believe learning requires social interactions, or both. After all, that’s what really matters, and what Donnelly and Wiltshire fail to see when they see kids rolling on the floor and teachers at the side rather than at the front….

Further, while Donnelly and Wiltshire are quick to critique other approaches they are not as open to revealing their own view of how schools should operate. Sure they mention the teacher at the front (which fits with where I’ve placed them on Illeries model) but what else do they want to our schools to be? If they don’t want kids rolling on the floor, what do they want? Kids in rows? In silence? Reading solely from text books and/or doing dictation? Placing them on a model like this helps us to start anticipating what they might believe and asking those questions of them?

For me, these questions are much more important then whether they are experts or not!

The purpose of this article isn’t to unpack and disprove Donnelly and Withshire’s ideas, so I’ll leave that to others. I hope that I have presented a framework that you may find useful. This approach might be less useful however when discussing learning and teaching with those whose beliefs are similar to our own. In that case other models might be more useful, for instance, Tom has some really nice work unpacking different beliefs about inquiry learning, I’ll try to post about it soon.

As for subtle differences in highly instructional models, sorry, you’re on your own, maybe ask Donnelly and Wiltshire!

 

 

Oh, one last thing, if you can’t work out where an expert fits in Illeries’ model my best is that they’re being deliberately ambiguous and probably can be placed up there near Donnelly and Wiltshire…

The toxic myth of good and bad teachers

Time to read: 13 minutes

There are a number of claims made by various people about the effect on a student having a good teacher versus having a bad teacher. Most of these claims are nonsensical, and rather than increasing the likelihood of improvement in schools, they do a great deal of damage to teachers, students and schools, and make school improvement much less likely.

Why?

Because there aren’t many good teachers and there aren’t many bad teachers, most teachers are just average. We know this, because we know that teacher quality, measured across all teachers, results in a normal distribution, a bell curve. Sure we’d find that there are a few high performing teachers at the top end and a few low performing teachers at the bottom, but the far majority would be in the middle with not that much separating them. If you’re a teacher who is reading this post, I’ve got bad news for you, you’re almost certainly an average teacher. Just as I am almost certainly an average teacher. While we’d like to think that we’re high performing, compared to our colleagues, the actual evidence points to the contrary.

It would be same if we measured the quality of carpenters, golfers, doctors, lawyers, public servants, scientists, whoever… but for some reason we don’t complain about the quality of other professions. Teacher bashing has become a convenient excuse for far too many critics.

A few week or so ago, The Age newspaper identified some of the A+ teachers helping students to 40+ in VCE here in Victoria, Australia. In this article five teachers, whose VCE results stand out clearly above other teachers, are identified as great teachers. One teacher had ten out of the top 14 students in VCE Sociology, another taught 17 of the top 33 students for his Business, and another taught 2 of the top 8 students in their Australian History. The results these teachers have achieved are exceptional, and clearly it is impossible to believe that every teacher could produce these kind of results, but it is also clearly wrong to suggest that just because they aren’t producing these kinds of results that they are bad teachers.

Obviously, every parent whose child is undertaking VCE would love to have teachers who produced these results. Yet to believe that it is possible for the majority of teachers to produce results similar to these four teachers in this article is plainly wrong. It is impossible for most teachers to have a number of students in the top bracket of students. Of course it shouldn’t be surprising that there are teachers who produce results like these, rather it is to be totally expected, as obviously when we consider the distribution of teacher quality, its distribution is shaped like a bell curve.

Empirical_Rule

This is why statements from people like John Hattie are so misleading. According to Hattie “teachers account for a variance of 30% in student achievement.” I’m not convinced this is true but even if it is, is Hattie describing the maximum variance within to the two limits of the bell curve? If he is then the 30% variance only applies to a tiny fraction of exceptional good teachers compared with the tiny number of exceptionally poor teachers. For the far majority of teachers, whose quality lies in the middle of curve, the variance will be non-existent. Sure there may be a theoretical maximum 30% difference within the absolute best and the absolute worse, but for the far majority the variance will be close to zero.

hattie-30

Looking at the graph above then Hattie’s 30% maximum variance within good and bad teachers, even if it is true, is vastly overstated. Three standard deviations from the mean of 15% falls at 6% and 24%, which cover the middle 99.7% of teachers. As such there is only 18% difference within the middle 99.7% of teachers. Looking at two standard deviations which account for 95% of all teachers, slims the variance to 12%! While the middle 68% of teachers (1 standard deviation) only shows a 6% variance in student achievement, a far cry from the stated 30%!

Of course Hattie might not believe that teacher quality fits a normal distribution, but if so he needs to justify why he believes this and how many good and bad teachers there is in our schools. He may also suggest that the absolute maximum difference is more than 30% and the 30% figure correlates to the third or even the second standard deviation from the mean, but if that was the case, then surely that would be the figure promoted or he would explain how many teachers are subject to this variance. But he doesn’t so I believe it is safe and proper to fit a normal distribution to his variance claims.

Dylan Wiliam tries to similarly tries to promote the myth of good and bad teachers when speaking at the ALT-C conference in 2007 he said, “If you get one of the best teachers, you will learn in six months what an average teacher will take a year to teach you. If you get one of the worst teachers, that same learning will take you two years.” Again, I’m not agreeing with these figures, in fact I highly doubt them unless of course Wiliam is speaking about pure memorisation and direct instruction, which he well may be.

Wiliam gives us a little more information than Hattie though. He suggests that his data doesn’t fit a normal distribution, where the average would be in the middle of the lower and upper bounds. If Wiliam’s data fitted a normal distribution then the average would be 15 months instead of 12 months, as 15 months is 9 months more than the lower bound of 6 months, and 9 months less than the upper bound of 24 months.  As such, Wiliam’s assertion fits what is called a positively skewed distribution, as shown below.

positive-skewed-data

By graphically representing Wiliam’s figures, it is obvious that his claims are overblown. It is clear that the far majority of teachers produce about the same results, and the teachers at the lower and upper ends of the impact are in the tiny minority. Also according to Wiliam, most teachers, that is more than half, are not producing a year’s worth of learning a year! While the mean is 12 months, in a positively skewed distribution the mode will be less than the mean. In Wiliam’s world, more than 50% don’t produce a year’s worth of learning in one year… and somehow it is their fault??

 

Even if the maximum difference within the best and the average teaching is 18 months, then the actual variance of what is an average teacher, and the variance within the far majority (1, 2 or 3 standard deviations from the mean) would be much, much smaller, and again just like Hattie’s 30% variance extremely overstated for the majority of teachers and students. Even if Hattie’s and William’s figures are correct, the point to their message must be that this difference does not occur regularly in our classrooms but rather it is an extremely rare exception.

 

The cumulative effect of Hattie, Wiliam and others suggesting that these rare and extreme examples of teacher quality variance are in fact common occurrences, results in teacher quality being viewed as a much bigger problem than it is. Yes, if their figures are accurate, it should be a big concern that 0.015% of teachers impact student learning outcomes much less than others (probably only measure solely through test scores) but it is a rare problem rather than systemic problem than many people believe, and it should be seen and treated as such. Also, it should be recognised that the problem of variance in professional quality is not unique to teaching but rather occurs at the same distribution and to the same degree in every profession.

Atul Gawande posed the question “What happens when patients find out how good their doctors really are?” in his 2004 article Under The Bell Curve. Gawande describes the efforts over  117 cystic fibrosis clinics across the US over the last 60 years. We’d like to think that when we go to hospital we would get the same quality of care and would have the same expected outcomes regardless of which hospital we attend and which doctor attends to us. Yet, Gawande tells us that isn’t the case at all, there are good doctors and bad doctors, with the good hospitals in 1997 reporting life spans 16 years above the average for cystic fibrosis patients! Gawande points us to the bell curve and reports that the far majority of doctors and hospitals however are average and their patients have much shorter life expectancies.

Gawande then explores how hospitals reacted to the news that the care they were providing to their patients was average, and the efforts they used to increase the quality of the average majority. While there have been substantial improvements, Gawande insists that the bell curve remains, and will always remain, and the difference in life expectancy for cystic fibrosis (and patients will other life threatening illnesses) will always be dependent on the quality of the care they receive.

Gawande finishes his article examining himself as a surgeon. What if he found out that he was just average, or worse? For Gawande however, the problem of being average isn’t as big as settling for being average, something I assume that Gawande admits that he would rather quit being a surgeon than doing.

So do the doctors and hospitals how provide average quality care for cystic fibrosis patients at their clinics want to improve want to improve? One would hope so, but simply identifying them as average doesn’t mean they are happy being average as there is no evidence to suggest this. The bell curve in itself does and cannot distinguish those who want to improve from those who don’t. In fact, Gawande points to patients who chose to stay with their average doctors because of the care they feel built up over a number of years.

It is distressing for teachers to acknowledge the bell curve. After all we all want to view ourselves as being good teachers as opposed to being average but a realistic understanding of how skills and knowledge are distributed across of a cohort forces us to face this unwelcome truth.

Of course it would be easier if we actually could measure teacher quality which would allow us to measure, identify and quantify good, bad and average teachers. The problems is that we don’t have an universal way of understanding teacher quality, while various groups have tried they haven’t done a good job of this. The previous government here in Victoria, unsuccessfully tried to implement a system where school principals would rate their teachers from 1 to 5 in order to identify 20 to 40% of them as being underperforming and not eligible for pay promotion. Clearly those suggesting this system believed that teacher quality in Victorian schools didn’t fit a normal distribution but rather a negatively skewed distribution. Of course, the only reason they had for this was budgetary.

This is where the toxic nature of talking about good and bad teachers is revealed. After all does it matter more about the actual distribution of teacher quality, or does it matter more about what people believe the distribution looks like? What happens when a myth is propagated that teacher quality doesn’t fit a bell curve but rather fits a negatively skewed distribution?

Furthermore, in the absence of appropriate data we do what most people do, we assume that we are a good teacher and therefore we are the definition of a good teacher. And if we’re not teachers ourselves, when base our view on good teachers on the teachers we had when we were at school. It’s almost as if we say, “I might not know what teacher quality is, but I know a great teacher when I see one.” Which might sound reasonable… but in reality these ideas have an incredibly narrow view of what a teacher is, and quickly descend into discrimination and teacher bashing.

Discrimination and teacher bashing? How?

Well, some people believe that to be a mythical great teacher you need to be a highly passionate caring teacher. In this narrative great teachers are in the mould of Miss Honey from Roald Dahl’s “Matilda” with a rare gift to inspire and connect with their students. These people point to inspirational teachers who taught them when they are in school, or the inspirational teacher they believe themselves to be.

This narrow understanding of teacher quality creates unrealistic expectations, it really is impossible for every teacher, in most schools, to have an amazing rapport with each and every student. As a result quality teaching becomes a teacher who displays their passion for teaching by working long hours and having teaching as their only real priority. Who is always positive and never has a bad day!

While we’d all like every teacher to be passionate about teaching, but discrimination happens when we expect every teacher to be only thinking about teaching and willing to put in every hour they can. Single-parents and others for a range of reasons, who are unable or unwilling to devote every waking hour to teaching are quickly labeled as bad teachers, who should be moved on, overlooked for promotion or discriminated in other ways. People with problems in their personal lives, or suffer from medical conditions might not always project this image of the inspirational teacher, and when we’re on the hunt for bad teachers these people can soon be in our sights…

Others believe that to be a mythical great teacher you need high level knowledge and skills. A good teacher is so much smarter and more knowledgeable than a bad teacher. Pretty soon though we’re lining up those teachers we don’t think are knowledgeable enough and moving them on. Tests have recently be proposed here in the Australia to check that new teachers are literate enough teach, despite them having passed their teaching degree and all their school teaching placements.

Older teachers who are not up with technology might be the first to go. Next might be women who have taken maternity leave and have a big gap in their experience or who are not able to (in our eyes) balance family/work. Next might be those who aren’t on Twitter day and night, attending professional conferences whenever they can in order to keep their skills up to date.

We’re all too quick to blame and label those teachers who aren’t just like us. Rather than celebrating diversity and considering what it might offer our students and our education system, we see diversity and being undesirable. We see diversity as being different from good, and we blame those for not being exactly like our picture of an ideal teacher, and we make erroneous assumptions about them.

Again this something that Dylan Wiliam gets really wrong when he says, “if we create a culture where every teacher believes they need to improve, not because they are not good enough but because they can be even better, there is no limit to what we can achieve.” While this might sound reasonable, sort of, where is Wiliam’s evidence that every teacher doesn’t currently want to improve? My confident guess is that teacher’s desire to improve is also distributed as bell curve, and Wiliam’s assertion that there are many teachers that don’t want to improve is misguided and overstated.

William’s attempt to link a teacher’s desire to improve to the variance of teacher quality is also false. You cannot overcome the bell curve by wishing it away, no more than every golfer can play as well as professional golfers if only they wanted to improve! It is silly. And why does Wiliam’s faith in limitless potential derive from? Surely finding better approaches for learning and teaching is where limitless potential might be found, such as via new pedagogical approaches afforded by modern technologies?

But those who talk about good and bad teachers don’t want to find new pedagogical approaches, they’re happy with the system we’ve got. And shame on anyone who can’t be a good teacher in their system and can’t reap good results using their approaches. According to these experts, it’s not the bell curve that’s the problem, it is the teachers themselves.

Not only does this lead to discrimination, with anyone who doesn’t fit their mould being labelled a bad teacher. It also leads to not focussing on what could actually improve student learning outcomes. While we try to narrow the quality gap, whether it be Hattie’s 30% or Wiliam’s year and a half year, we’re not concerned with why all teachers can’t successfully teach in Hattie and William’s systems. We’re not looking for pedagogical approaches, (constructivism anyone?, inquiry anyone?) that might not be so susceptible to such variances due to teacher quality.

Consider the Measures of Effective teaching project whose goal is to identify effective teaching. I’m still at a loss why you wouldn’t just use test scores as a predictor of future test scores, unless of course you’re trying to pretend that student learning isn’t just about test scores. Of course, if you want to try to pretend that you can measure effective teaching beyond test scores you can then appear to agree that learning isn’t just about test scores, which I guess is why METS suggest approaches that weigh test scores somewhere between 33% and 50%…

In order for every student to achieve success we need learning and teaching approaches that are suitable for average teachers. We need to recognise that education of our students is far more than test scores. That is the first stage and until we’ve done that we need to lay off teacher quality. If Hattie, Wiliam, and others do believe that education is all about test scores then they need to be honest and upfront about that before we start labelling teachers as good and bad.

How many good and bad teachers there actually are matters a lot. Take for example, the report: Great Teaching, Inspired Learning What does the evidence tell us about effective teaching? where the authors say:  “Modelling by the US economist Erik Hanushek estimates that if a student had a good teacher as opposed to an average teacher for five years in a row, the effect would be sufficient to close the average performance gap associated with low-socioeconomic status.”

But how likely is it that a student had a good teacher as opposed to an average teacher five years in a row? If we want the results that Hattie and William suggest the best of the best teachers can achieve, then we’re looking at the teachers above the third standard deviation or 0.015% of teachers. How likely is it that a student would have these teachers for five years in a row? We can working this out by multiplying 0.15 with itself five times

0.015 x 0.015 x 0.015 x 0.015 x 0.015 = 0.00000000007%

This is so unlikely you wonder why Hanushek would even bother suggesting this.

If we believe that teacher quality fits with a normal distribution how many standard deviations are we going to choose to identify good teachers, that is, where do we set the bar? Say we set the middle 68% as the average (one standard deviation) which means the top 16% will be good teachers? How likely is a student to have a good teacher five years in a row? Only 0.0001%! Alternatively, if we believe that only 80% of teachers are good, then only 30% of students will have a good teacher for five years in a row. And where do Hattie, Wiliam and their peers set the bar, for what is tolerance of teacher quality for which their pedagogical approaches work?

We have two choices, first we follow the path of Hattie, Wiliam and their peers who think that our pedagogical approaches are set in stone and appropriate and our teacher variance is the problem, or we can decide that teacher variance shouldn’t impact student learning, rather instead our pedagogical approaches should ensure all students equally experience learning success. Make no mistake, a focus on teacher quality is incompatible with a focus on pedagogical innovation and improvement, and conversely a focus on pedagogical innovation and improvement is incompatible with a focus on pedagogical quality. We need to choose which focus we believe offers the biggest gains for increasing student learning and equity.

I believe that we need to find, and that we can find, learning and teaching approaches that work for almost all (99.85%) teachers. If we can find pedagogical approaches that work for 99.85% of teachers, then 99.25% of students will have access to exemplary learning experiences for five years in a row. This will not only result in better learning outcomes but also a system that is more inclusive, equitable and more diverse.

Improving our pedagogical approaches so that they work effectively for all students and teachers is a complex task, and one that we won’t be able to solve while we continue to apportion the blame on bad teachers.

For me, the choice is clear. We need to stop speaking about good and bad teachers, we need to stop worrying about teacher variance, and instead focus on what might actually make a difference in the lives of our students focussing on developing higher quality learning and teaching approaches that are not limited by the variance of teacher quality.

 

Footnote: By the way, the bell curve as it relates to good and bad school leaders is also true. Sure there may be a tiny great school leaders, and tiny few terrible ones, but most of them are just part of the average majority…. as for doctors, public servants, politicians, car drivers, golfers, singers, ….

 

Update: Feedback from Andrew Worsnop suggests that I’m misusing Hattie’s 30% figure, I’ve expanded the section on Wiliam’s figures as it makes the same point. I haven’t changed my writing on Hattie’s 30% though, as I’m not sure that I agree with Andrew that I am misconstruing what Hattie is saying about the 30% variance in teacher quality/impact.

 

Image credit:  A visual representation of the Empirical (68-95-99.7) Rule based on the normal distribution. http://commons.wikimedia.org/wiki/File:Empirical_Rule.PNG Creative Commons Attribution-Share Alike 4.0 International license.

What can Visible Learning effect sizes tell us about inquiry-based learning? Nothing.

Time to read: 8 minutes

I haven’t read Visible Learning, I’ve only skimmed through it a couple of times, largely because the book isn’t aimed at me. Visible learning and its effect sizes are only useful to inform learning and teaching in highly instructional settings, I’m not interested in highly instructional settings. I think they’re a poor substitute for authentic learning, I think students would be much better served inquiry-based learning and teaching approaches, if Visible Learning was about that I’d read it. It isn’t so I haven’t and I won’t.

Which is why I was very disappointed to read Dan Haesler’s reporting on his interview with John Hattie, here, here, and here, where John reportedly critiques student-centred learning, inquiry, 21st century skills, and constructivism. Except for constructivism where this poor paper is cited as evidence, (I’ll explain why the paper is extremely poor later,) there is no evidence quoted that  suggests to me that the claims John makes doesn’t come from Visible Learning’s meta analysis. These statements, about inquiry-based learning and 21st century skills (though it isn’t my favourite term) have compelled me to write this post in order to challenge their validity.

Statements like this from John, deeply worry me..

“We have a whole rhetoric about discovery learning, constructivism, and learning styles that has got zero evidence for them anywhere.” Note, I’m not defending learning styles!

…and this next statement is worrying as  well…

 “I’m just finishing a synthesis on learning strategies, it’s not as big [as others he’s done] there’s only about 15 – 20 million kids in the sample, and one of the things that I’ve learnt from the learning strategies, and a lot of them include the 21st Century skill strategies is that there’s a dirty secret.” 

If the synthesis that John is speaking about is using the same approach as Visible Learning’s meta analysis, then the dirty little secret is that the research is invalid and can’t be trusted.

There has been a bit of talk about the maths in Visible Learning, to me this is a distraction from the real problem of the book. You only have to get to the second page of the preface of the book to find the first huge problem, as we read…

“This is not a book about qualitative studies. It only includes studies that include basic statistics (means, variances, sample sizes.) Again this is not to suggest that qualitative studies are not important or powerful just that I have had to draw lines around what can be accomplished in a 15 year writing span.” Visible Learning (preface xi)

The next section outlines an even bigger, insurmountable problem with Visible learning’s design and findings…

“It is not a book about criticism of research, I have deliberately not included much about moderators of research findings based on research attributes (quality of study, nature of design) again not because they are unimportant (my expertise is measurement and research design), but because they have been dealt with elsewhere by others.” Visible Learning (preface xi)

If you’re not interested in instructional teaching approaches and you reach a passage similar to either of the two above, my advice would be to put the book down and walk away, and advise others to do so.

These two decisions, to omit qualitative studies and not require the study design to match the object of study renders its findings virtually useless. Firstly, this restricts the definition of impact to numbered results (obviously this almost always means test scores) and secondly it allows these numbers (test scores) to measure the impact of things that may not even be seeking to impact. In short they’re using test scores to measure things that may not be able to be measured meaningfully with test scores. Furthermore, they are disregarding other research that measures these same things against their actual claims. In Visible Learning all that matters is presumably badly designed (more on that later) test scores, in classrooms we know that is simply not true. Visible learning’s design makes it incompatible with a large number of learning theories, approaches and strategies, yet unfortunately it doesn’t admit this, and it still calculates an effect size for them.

In Visible Learning’s world of research,the impact of inquiry-based approaches can be measured by how well a student does on a badly designed and irrelevant test. Does that mean the impact of 21st century skills can be measured by an irrelevant test? Does that mean that the impact of constructivism can be measured against an irrelevant test? It is as if that all that matters is the test.

The Visible Learning study design chooses not to require the object of study (eg inquiry-based learning, 21st century skills ) to be evaluated against it benefits. Instead it allows the researchers to test inquiry-based learning, 21st century skills and whatever else to be tested against what the researcher deems important, say for example the ability to pass a math test. Furthermore, the requirement that the studies produce a number surely results in favouring studies that align non-instructional approaches with instructional outcomes. Visible Learning then omits surely better designed non-instructional approaches but excluding qualitative research. The end result is that for non-instructional approaches Visible Learning has to be omitting well designed research while including poorly designed research.

To see that these questions aren’t just hypotheticals, lets look at an example where these two failings 1) the reliance on numbers and an irrelevant test, and 2) a misalignment between study design and study object cause meaningless results that are then touted as evidence. Why don’t we briefly look at the paper that John purportedly himself suggests is a major investigation into constructivismShould There Be a Three-Strikes Rule Against Pure Discovery Learning? by Richard Mayer.

Let’s specifically look at his third strike supposedly taking down Seymour Papert’s vision of discovery learning, of course we’ll conveniently side step around Mayer’s wrong assertion that Papert promoted constructivism when in fact he promoted constructionism. Does Mayer examine all of Papert and MIT kindergarten’s studies and seek to replicate them? Of course not, instead he makes the same design mistake that Visible Learning makes, trying to apply an instructional theoretical research approach to constructionism/constructivism/discovery learning.  Mayer refers to findings of two similarly deeply flawed studies, studies that seek to test what the researchers think the students should know against what they actually have learned. Actually that’s not true, the two studies do not seek to find out what the students learned at all, that might have would’ve been a better study…

The first Kurland & Pea (1985) study, which provides no rationale that the fundamental programming concepts are indeed fundamental, where is the author’s explanation that the impact of constructivism/constructionism/discovery learning can be accurately measured by testing these fundamental concepts. This study is flawed because it wrongly associates worth of the approach against a test that has nothing to do with the worth of the approach.

Where is it asserted anyway that LOGO is designed to teach a predefined set of fundamental programming concepts? Absolutely nowhere, the authors just made it up. They’ve used a badly designed test student’s knowledge of recursion! Why recursion? Simply because the authors think it is important, not because the purpose of LOGO is to teach recursion (spoiler, it is not) or because the purpose of discovery learning is to learn recursion (again spoiler, it is not.) If using a test wasn’t bad enough to evaluate LOGO and discovery learning, they’ve made it even worse by limiting the test to recursion. Sure if you want to students to learn recursion quickly do some research but don’t try to extrapolate the results or misconstrue the results to make unfounded claims against LOGO and discovery learning.

A proper study would assess impact based on the overt goals of constructivism/constructionism/discovery learning, but this study didn’t, instead it wrongly measured impact against a measure suitable for instructional approaches using a very bad test that produced very bad numbers.

The design of the second study Fay and Mayer (1994) is even worse, and its findings should be believed even less. The researchers taught programming using two approaches but they only tested using one approach (bonus points if you guess which approach was used to measure impact.)

Now there’s a radical idea, someone quick do research into the effectiveness of direct instruction to deliver on the promise of constructivism!

The third study, Lee & Thompson (1997), is a thesis and too long for me to be bothered reading, yet a flip through the first twenty pages doesn’t fill me with optimism that the research is using constructivism/constructionism/discovery learning to understand and assess the impact of constructivism/constructionism/discovery learning. In fact it again seems like a study using guided instruction design to compare guide instruction with constructionism. One could guess that the study might be comparing the approach of LOGO with an approach of LOGO plus a worksheet… by asking the students to complete a worksheet!

 

The Mayer paper that John cites and its three examples show why impact of constructionism and LOGO can only be adequately understood through the theoretical lens of constructionism. Anything else is to do a huge disservice to Seymour Papert, constructionism and educational research. To reduce LOGO to “discovery learning” an believe that its value can be assessed by testing students on a specific externally defined knowledge of recursion is both ridiculous and poor research. I don’t know whether these studies contributed to the Visible Learning effect sizes but I’m sure many studies like these did, studies which supposedly prove LOGO, constructivism, constructionism, discovery learning, and everything else outside of instruction don’t work as well as instruction.

If Visible Learning effect sizes did take study design into account then it would not be open to these errors, and its effect sizes would be more believable. It doesn’t and therefore I cannot see how anyone can have any confidence in the effect sizes.

Visible learning and its effect sizes probably adequately report on the impact of instructional approaches but it cannot possibly adequately or in good faith report on the effect on student learning of using LOGO, constructivism, constructionism, discovery learning, or anything else that is based on a learning theory that is not based on instruction.  If you want to know if research finds evidence into any of these things go and find research that is uses the basis of theory as a lens of understanding, there is lots out there. Unfortunately these well designed and trustworthy research isn’t included in Visible Learning because it is qualitative research (it has to be by the nature of the object of the research) and therefore is excluded from Visible Learning’s meta analysis and therefore does not contribute to Visible Learning’s effect sizes.

The same is true for John’s claim that his forthcoming meta analysis illuminates the “dirty little secret” that 21st century skills don’t work. If in this forthcoming study 21st century skills are not evaluated against the purpose of 21st century skills then the study is flawed and should not be trusted. If the forthcoming study uses the same study design as the Visible Learning then it too will be deeply flawed, and John’s claims about 21st century skills should be ignored.

 

Finally, just in case you’re not convinced by my critique of these research papers and the importance of using the theory of learning to measure the impact of the theory of learning. Lets examine a more everyday critique of inquiry-based learning. I’m not sure if it was the author, the sub-editor or John Fleming the interviewee, that came up with the opening paragraphs of the article Schools cool to direct instruction as teachers ‘choose their own adventure‘ but they are gold….

“WHEN teaching your four-year-old to tie their shoelaces, do you give them four pairs of shoes and tell them to try different techniques until they work it out? Or do you sit down and show them how to do it: make the bunny ears and tie the bow, watch while they try it, lending a helping finger if required, and then let them practise until they can do it on their own?

The two approaches illustrate different teaching styles used in classrooms. The first describes a constructivist method in which a child “constructs” their own understanding through discovery or activities, also referred to as student-centred learning.”

Of course, it is absurd that any rational person, let alone a teacher, wouldn’t use direct instruction to teach a child to tie their shoelaces. Unless of course you’re this guy, and you might point out that most people tie their shoelaces incorrectly.

 

Now if your assessment of a child’s ability to tie their shoelaces is by being able to replicate the adults (wrong) method, say by testing them, then our use of direct instruction is working wonderfully well. If your assessment on the success of direct instruction is based on how many kids are running around the school grounds with their laces undone, or how often they have to stop and retie them, then maybe direct instruction isn’t doing so well.

If direct instruction cannot even teach children to tie their shoelaces correctly, what can it possibly be trusted to get right?  If direct instruction fails to make obvious that the teacher is teaching incorrectly when tying shoelaces, what else are teachers using direct instruction getting wrong? In maths? In english? In everything?

 

Would a better study design show a more accurate effect size for direct instruction? A study design that looked beyond a simple numbered value produced by a test? For us to have any faith in academic research, I’d like to believe yes.

 

What if kids did use an inquiry approach to learn to tie our shoelaces? What if we did as the above article suggests and give kids four pairs of shoes and asked them to work out the best method?

My bet is that we’d see benefits above that of actually having everyone being able to tie their shoelaces correctly. Maybe we’d see students who were less accepting that there is a single right way? Maybe we’d see kids believe less that when something goes wrong its because they hadn’t followed the proper process accurately?  Maybe we’d see kids being more critical consumers of the purportedly correct information they’re presented with?

Maybe we’d see a whole range of things… but while we continue to use instructional measures (predefined narrow tests) to measure impact we’ll never know.

 

Note: I have updated this post for clarity since publishing, and I will probably make further updates over the next couple of weeks, as I receive feedback.

Pre-service Teachers and the Terrible, Horrible, No Good, Very Bad Test

Time to read: 3 minutes

Yesterday, the Teacher Education Ministerial Advisory released a new report on pre-service teacher education titled Action Now: Classroom Ready Teachers, for mine this is a very flawed report mainly because it is based largely on assumptions and biases that are not disclosed, its findings largely benefit the authors of the report, and additionally lays the blame squarely on others. The authors do a very good job trying to pretend that this is a fair and balanced report though, as they continually make the claim that their recommendations are all evidence-based. So much so that the word evidence is used 146 times in the whole document! While the report authors do provide a definition of “evidence-based teaching practice” in the glossary (interestingly this complete term is only used twice, with one being a heading), the authors do not at any stage define what good and bad evidence is, despite their reliance on it.

If only there was way that we could infer what the authors may view as being good (or even acceptable) evidence? I could propose my own definition of what good evidence is and then seek to identify each of their uses meets my criteria but I think that there is possibly a better (and much quicker way.) Take for example Recommendation 13: “Higher education providers use the national literacy and numeracy test to demonstrate that all pre-service teachers are within the top 30 per cent of the population in personal literacy and numeracy.”  A recommendation that according to the explicit goals of the report has “been chosen on the basis that they are practical, based in evidence and calculated to succeed.”

Lets examine Recommendation 13 against the report’s own criteria…

… they are practical…

In the scheme of things making universities administer a test seems reasonably practical, there might even be a couple of research papers in it! Of course there isn’t any evidence given that a single current pre-service teacher, or that future pre-service teachers wouldn’t test in the top 30 per cent, so it could be argued that administering a test that is passed 100% of the time is practical.

So, is Recommendation 13 practical? I’ll give this criteria half a mark.

Taking about evidence leads us to the next criteria.

… based in evidence…

Unfortunately Recommendation 13 doesn’t look as promising when we apply their based in evidence criteria. There is no evidence in the report (or that I could find elsewhere) that supports the notion that teachers need to be in the top 30 per cent in order to be classroom ready. I’m not sure you’d find many people who would argue that teachers don’t need to literate and numerate, but where is the evidence for using the top 30 per cent as a benchmark? Also, there is no evidence that teachers who are currently teaching in our schools and are deemed to be high quality teachers are in the top 30 per cent. Surely this would be the first piece of evidence on which this recommendation would be based, that there is a strong correlation between teacher quality and scores in the proposed literacy and numeracy test.

So, is Recommendation 13 based on evidence? Sadly I’d have to say no, zero marks.

… calculated to succeed…

Given the lack of evidence for the test in the first place obviously predicting its success is difficult. I assume that success wouldn’t be measured by Higher Education’s ability to administer a test, in that case it would be a no brainer to agree that the recommendation would succeed. Yet, I don’t think that is how success would or should be measured! How else could success be measured?  Success could be better measured by pre-service teachers in the top 30 per cent at the time of taking the test remaining in the top 30 per cent over time. Again I can’t see how such success could possibly be guaranteed that a one off test could possibly predict that a teacher will remain in top 30 per cent of the population in literacy and numeracy over their entire career, especially given the short term knowledge that these types of tests test.

Sorry, for Recommendation 13 based on the likely to succeed criteria, again I can’t give any marks.

My total score for Recommendation 13: “Higher education providers use the national literacy and numeracy test to demonstrate that all pre-service teachers are within the top 30 per cent of the population in personal literacy and numeracy.” based on their own criteria of “is it practical, is it based on evidence, and is it likely to succeed” I can only give it half a mark out of three. It is fair to say that the author’s understanding of evidence is a far short of my understanding of good evidence.

 

Paradoxically, of course, there is also no evidence that pre-service teachers who might not test as being in the top 30 per cent at the time of graduation would not later test as being in the top 30 per cent. Given the focus and time spent on literacy and numeracy based work, I assume that all teachers would perform well on this sort of test. This highlights the most terrible, horrible, no good, very bad part of this recommendation, beyond failing its own criteria, it displays a fundamental lack of understanding of how people learn. Teachers will increase their proficiency in literacy and numeracy as they teach because they engage in wider range of authentic activities that rely heavily on the fundamentals of literacy and numeracy. A single test can only test the immediate past and cannot possibly give any insight into the future, understand a person’s development in terms of a ranking against peers in even worse.

 

However you look at Recommendation 13, using the report’s own criteria, or what we all know about learning and teaching, this is a terrible, horrible, no good, very bad test.

Teacher Quality and the Purpose of School

Time to read: 1 minute

Disclaimer: This post is not intended to be a deconstruction of the purpose of school and the three roles of schooling, I’ll save that post for another time!

To round out this series of posts on teacher quality I thought I should address the final elephant in the room that those who speak of good and bad teachers need to acknowledge… the purpose of school. If you haven’t read the other posts you may wish to read Who are the Best Teachers?, Assessing Teacher Development (or anything really) and A Better Understanding of Teacher Quality.

Those that teacher bash talk about teacher quality like to pretend that schools have a singular universal purpose, that a school’s sole purpose is to prepare students for their professional lives by teaching them what they need know, and how to think.

Yet when we reflect on the purpose of school without thinking about teacher quality we realise that the purpose of school isn’t so narrow. Sure preparing students for the workforce is a major role but so is developing the social aspects of our students, developing in them a morale compass, an appreciation of culture and history, and how to develop constructive and meaningful relationships. Students engage in leadership programs, work and play in teams, and participate in many other activities and programs for the sole purpose of social and emotional development. There is also a third aspect to the purpose of schools, that is, to develop students as unique individuals. Yes, we believe we need to teach specific forms of knowledge and ways of thinking, yes we believe that we need to develop the student socially, but we also believe that students are individuals with unique talents, and aspirations. As such as teachers we provide opportunities for student choice. We highlight that there are multiple paths rather than a single preferred path to tackle a problem. We encourage students to reflect and assess their own learning. We have student-led conferences rather than teacher-directed reporting.

Yet, when experts, school leaders, and teachers talk about teacher quality they almost always ignore the three facets of the purpose of schooling and focus solely on teaching of content, skills, and ways of thinking. This is because assessment focusses solely on this singular aspect of schooling, but in reality the other two roles of schooling might be held in a high regard by a teacher, the school, and the wider school community. Making pronouncements about the quality of a teacher without understanding the how the teacher and their school views the purpose of their school, is completely unfair bordering on immoral. When talking about teacher quality in this way, we devalue the other two roles of school as well as devaluing teachers and teaching.

When we talk about teacher quality we can only do so ethically in light of what they and their school community believe about the purpose of school, and the weighting of the three roles of school. If we don’t properly understand the purpose of a specific school we cannot possibly determine whether an individual teacher is “good or bad.”