## Understanding the limitations of NAPLAN

Disclaimer:
I am not a statistician. Though I studied statistics at university, it was a very long time ago. Further, I know very little, beyond what is mentioned in this post, about standardised testing. I’m probably not using the correct terms when I speak about test errors.  Read everything below with a critical eye, and take my claims with a grain of salt… and sorry for the typos, at the end of writing I just wanted to push “publish.”

Part of my job at school this year has been looking at NAPLAN data. This isn’t something that anyone really wants to do. This is post is about what I’ve discovered. I’m using dummy data for privacy reasons, yet all findings mentioned in this post is in line with what I found using data from my school.

Introduction
People more qualified than me have written about the limitations of NAPLAN, in particular Margaret Wu, who makes it clear:

Still, it is hard for many school leaders to understand what this means for their yearly NAPLAN data, and even harder to communicate to other stakeholders. As such, I created a tool that allows school leaders and data nuts to visualise how NAPLAN errors our data.

Major Types of Errors
There are two types of major errors which hang ever present over NAPLAN data 1) student error, and 2) test extrapolation error.

Student Error
Student error acknowledges the fact that if a student gets 18 out of 25 in a spelling test this week, that doesn’t mean that they’ll get 18 out of 25 in a spelling test next, or the week after that, and so on. Of course, we wouldn’t expect someone who gets 18/25 to get 2/25 the next week either. So we know it is a predictor. Wu suggests +/- 12.5% is a good guide. Which means that our student who this week spelt 18/25 words correctly will probably get 15 to 21 words correct next week. Student error acknowledges that students sometimes make silly mistakes, and also, sometimes they get lucky and answer a question correctly that they usually wouldn’t.

Test Extrapolation Error
The second error accounts for the ability of the test to accurately represent the development that it is seeking to assess. Take for example, NAPLAN’s 2017 Year 3 spelling test.

In this test there were 26 questions. If you answer zero out of 26 correctly you are in band 1. If you answered any question correctly, you are in band 2 or higher. In fact, 3 questions cover the entire band 2, 6 cover band 3, 5 cover band 4, 5 cover band 5, and 8 further questions cover bands 6 and above. Given that each band is supposed to represent a year’s worth of learning, how well can 3, 4 or 5 questions (in the case of bands 2, 3, 4, and 5) represent all the words that students in these bands should know? How well does each of these words, represent 20% of the total words of bands 4 or 5? Can we really believe the test can determine that a student who answers two of the five band 4 words correctly knows about 40% of the band 4 words?

Of course, this is assuming NAPLAN’s tests have increase in linear difficultly where once students encounter a word they cannot spell they cannot correctly answer any of the harder words in the test. In this year’s spelling test this could not be further from the truth.

As part of our toolkit for analysing NAPLAN data, we were provided a spreadsheet with the test questions ranked in order of difficulty. We were also provided with the state percentage correct for each test question. For the spelling tests in years 3 and 5 (I didn’t check years 7 and 9), what NAPLAN believed was the order of difficultly did not correlate with the perceived linear difficultly of the words. For example, when tested while 91% of the state correctly answered the first and third easier questions, only 56% of state correctly answered what was considered to be the second easiest question!

In fact, according to Victorian state averages, band two contained the first, third and seventh hardest words. Band 3 which is supposed to have the 4th, 5th, 6th and 7th hardest words, in fact contained the 4th, 10th, 14th and 16th hardest words. And, rather than the 8th, 9th, 10th, 11th, 12th and 13th hardest words, band 4 contained the 2nd, 8th, 15th, 20th and 21st hardest words. Therefore, a student who is identified in band 2, having only been able to answer the three easiest question, may in all likelihood have answered what was perceived to have been the hardest question in band 3!

It is true that actual difficultly and perceived difficultly in numeracy, reading and grammar line up much more accurately than spelling in the NAPLAN. Yet, none of these anywhere close to being accurate to the point where a question can pinpoint a specific point in a student’s development.

The test extrapolation error therefore describes the difference between the test and reality. How well can the test, and in NAPLAN’s case maybe only three or four questions represent a student’s actual development?

When accounting for the these two errors it is clear that an individual student’s NAPLAN score is an approximation of the student’s development. How accurate an approximation is open to debate.

NAPLAN’s Scaled Scores

NAPLAN uses scaled scores to convert, in the case of the 2017 year 3 spelling test, into NAPLAN bands. A scaled score of between 0 and 270 places a student in Band 1, a scaled score of between 271 and 322 places the student in Band 2, and so on with each subsequent band, up to Band 9, comprising 52 points. Twelve correct answers in a test doesn’t place a student in the same band. Each and every test is scaled differently with the raw score in a test is uniquely converted to the scaled score.

ACARA publish equivalence tables each year, apparently the 2o17 tables are due out in December, which show how the raw score (number of correct answers) is translated into a NAPLAN band for each test. As you can see the 2016 year 3 spelling test was scored a bit differently to the 2017 test, with a student needing three correct answers to achieve band 2.

What is not apparent, that while the scaled scores are precise to five decimal places, students cannot get any score, instead they get the score that corresponds with their raw score. In the case of the above table, two students with raw scores of 3 and 4, are given scaled scores of 296.25472 and 316.08247 respectively. There is no way to achieve a score between these two points. Considering that this equates to 20 points and a year’s worth of learning is 52 points, NAPLAN cannot, and is not, precise. For those who believe NAPLAN can show learning to a year or months, this gap of 20 points is a gap of 4.6 months! Hardly precise.

It is true, that in the middle the equivalences tables, each question usually has a scaled score gap of 9 or 10 points. Though this still equates to two to three month increments, even if we believe their aren’t any student errors or extrapolation errors.

Let’s look further at the table.

I’m going to leave most of data in this table to the real statisticians, but I will focus on Scale SE column. NAPLAN defines a scaled standard error for each scaled score. In the table above, you can see a raw score of zero, translates to a scaled score of 182.95034 with a scale standard error of 61.60665. The scale standard errors, quickly reduce from this high of 61 to 36, then 30, and for many of the tests become as low as 20 or 21.

ACARA does not, anywhere that I can find, communicate what the scale standard error is compensating for, though, I would assume that it is an attempt to allow for the test extrapolation error. They do, though make it clear, in some documents, that the scale standard error correlates to a single standard deviation. Which means that there is a 68% likelihood that a student’s development falls within 1 standard deviations of the scaled score, a 90% likelihood within 1.64 standard deviations, 95% likelihood within 1.96 standard deviations.

If we assume that this error is normally distributed, then a student with raw score of 4 in the 2016 year 3 spelling test, would be 68% likely to have a NAPLAN score between 291 and 251. They would be 90% likely to have a NAPLAN score between 275 and 257, which is just under two NAPLAN years difference. And, 95% to have a NAPLAN score between 267 and 365, a difference of 98 NAPLAN points or nearly two NAPLAN years!

Visualising NAPLAN Errors
Take, for example, a fictitious student who had a raw score of 14 on the 2015 year 3 reading test. I’m using 2015 as a example, because later I’ll look at relative growth, between a student who sat the year 3 test in 2015, and the year 5 test in 2017.

This score of 14 rounds to a scaled score of 310 (band 2), with a scale standard error of 26.42. Remember band 2 begins at 270 and ends at 322. Using NAPLAN’s scaled score and scale standard error, I randomly generated 10,000 scores within the normal distribution, in order to see the range of possibilities and their likelihood. This is the result, the last found columns show bands 1 to 4, with the green highlight indicating the band that NAPLAN places the student in. Note: The code also highlights in orange when probabilities are greater than 20%, an arbitrary decision.

With an initial raw score of 310, it is unsurprisingly, most likely (61%) that the student is in band 2. However, as the student sits close to the top of band 3 (310 as opposed to 322) there is also a 32% chance they might be actually be in band 3. There is also a small chance, that they are also in bands 1 (7%) or 4 (1%)!

What about a student that sits just in band 3? In the same test if a student has a raw score of 15 they will be allocated a scaled score of 323.23804, 13 more NAPLAN points that a student (a quarter of a NAPLAN year) than the student with a raw score of 14. Random sampling 10,000 times suggests a 50% likelihood that a student just in the band, by 1.23804 points, is in actually in that band.

There is around a 46% probability that the student is in the band (band 2) below, and again a small chance that the student is actually in band 1 (2%) or band 4 (2%).

The probabilities of a student who sits in the middle of band 2 (297 points), looks like this:

Much more likely, 67% to be in the band than the students on the edge of the band, but also quite likely 15% and 17% that they are in the bands below or above. As such, it is relatively easy to imagine how likely an individual student’s results are accurate. NAPLAN’s rule of thumb, as we’re using their errors, might be, if a student sits in the middle of the band then there is about a two in three chance that they’ve been place in the correct band, if they are on the edge it is about 1 in 2.

And, in a cohort of 20, 30, 50, or more students, that is a lot of students who are probably placed in the wrong band.

ACARA would probably argue that in writing and spelling unlike reading and numeracy, the scale standard errors, in the middle of the table, are around 19 not 25. Yet, even then this only appears to increase the accuracy by about 4%.

What if NAPLAN underestimates the test error?

I wanted to explore who bigger errors effect NAPLAN’s results, as personally I’m sceptical that 25 scale standard error points are enough to adequately account for all of the errors in this type of test. As I discussed in the first section of this post.

The three fictitious students we are following are shown below with their relative probabilities based on the official NAPLAN errors.

If the standard error is increased by 10 (see table below), the confidence that the bands are accurate, drops considerably. For the student in the middle of band, from 66% to 52%, for the student just in the band from 49% to 44%, and for the student near the top of the band, from 49% to 40%.

When the test is increased by a further 10 points, to 20 points the results for these three students are:

And, a further 10 points, taking the errors to a bit more than a full NAPLAN year.

If NAPLAN’s scale standard errors are accurate, then the results at the single student level are really only a guide, with a best being a two in three chance of placing the student in their actual band. Yet, if these standard errors, which are usually around the number of points that equates to two questions, are underestimated then the accuracy of NAPLAN is even worse.

When considering 1) the simple errors (or correct guesses) that a student might make in the test, and 2) the ability for 3, 4 or 5 questions to test and assess a whole band’s learning, are we satisfied that a scaled error of 19 to 26 accounts for these? If NAPLAN published their research maybe we would be able to more accurately assess these questions, but until then I, for one, am highly sceptical.

Understanding Relative Growth
One of the measures that many Victorian schools are being encouraged to include in their four year strategic plans are NAPLAN relative growth measures. But can they be trusted?

NAPLAN describes relative growth as such:

The above table seems to suggest that relative growth is understood as being relative to a student’s prior performance NAPLAN. Yet, in reality relative growth is concerned with a student’s performance relative to other students. The points range is dependent on the actual scores of the students and differs from test to test, and question to question.

NAPLAN relative growth compares a student’s improvement in NAPLAN over a two year period.  To do so, a student is compared with all other students who obtained the same raw scores two years prior. For example, a student with a raw score of 14 in the year three reading test is compared with all of the other students who also scored 14 in the same test. The scores from this group in the test two years later, are analysed to determine the raw scores that correlated with the bottom 25% and the top 25%. The students who score below the 25% mark are determined to have produced low growth, the students above the 75% mark are determined to have high growth, and the students between the marks are determined to have produced average growth.

NAPLAN does not release the 25% and 75% for each question, however they can be somewhat reverse engineered based on the reports they data they do release to schools. For reading year 3 2015 to year 5 2017, the relative growth looks something like this, based on the data I obtained from my school, using data from 59 students, and no doubt is not 100% accurate. Note: the students placed below are dummy data.

In the above table, low growth is shown in red, average growth in yellow, and high growth in green. The y-axis show the raw scores and bands for the year 3 test, and the x-axis shows the year 5 test raw scores and bands.

As with the indicative NAPLAN table, it is clear that the range of scores for average growth narrows as the raw scores increase. A student with a raw score of 14 in year 3 needs a raw score of between 12 and 20 to have average growth, and a raw score of 21 or more to produce high growth. As such, for students with lower scores in the year 3 test, the range of subsequent scores for the students with average growth is about nine questions.

Another student who has a raw score of 35 in year 3, needs a raw score between 32 and 36 to produce average growth, and 37 or higher to produce high growth. That is, 50% of all students who scored in 35 in the year 3 test, score in the range of 32 to 36, a five question spread, in the year five test.  This average range which can be as narrow as four or five questions for higher performing students, make relative growth a worrisome measure for school effectiveness, especially when we factor in the errors previously discussed.

Let’s now look at how relative growth applies to our three example students. In 2015, they sat the year 3 reading test and their results and probabilities are shown below.

In 2017, they sat the year 5 reading test and their results and probabilities are shown below.

In the table below, our three fictitious, example students were:
1. Jane Smith, who had a raw score of 13 in year 3, and 10 in year 5, resulting in her being categorised as having low growth.
2. Kristi Lidija, who had a raw score of 14 in year 3, and 14 in year 5, resulting in her being categorised as having average growth.
3. John James, who had a raw score of 15 in year 3, and 21 in year 5, resulting in him being categorised as having average growth.

Note: As ACARA doesn’t publish these figures, I can only guess where the 25% and 75% marks sit. For example, I reasonably confident that the 25% mark for year 3 raw score of 14 is that a year 5 raw score of 11 is low growth, while a year 5 raw score of 12 is average growth. However, I’ve had to guess the scale scores (based on previous year’s data) and I estimate they are about 12 NAPLAN points apart. Is the 25% mark half way between these two scaled scores? Who knows? All these educated guesses, make the following probabilities only estimates.

We can see in the above table that all three students sit close to edge of the ranges, in my school’s real data this was very much the case. This is also more likely for students in the higher bands where the average range can narrow to only four or five question, as described at the start of this section.

In the case of the student with low growth, we can see below, that the probability that they are correctly identified as low growth is approximately 62%. If this student had answered one less question correctly in the year 3 test, or one more question correct in the year 5 test, then they would have be categorised as having average growth. As per the previous examples, these probabilities have been calculated by random sampling.

Similarly to students who scaled scores are next to the band limits. Students whose relative growth places them next to limit, results in probabilities around 50%.

In practice, NAPLAN relative growth is a so unreliable that I cannot believe that it is a suitable measure and I would personally discourage anyone from using it. The narrow range of questions that define average growth, compounded by the error inherent to NAPLAN’s testing method make it an extremely unreliable measure.

Note: The code I use only has relative growth limits for reading 2015 year 3 to 2017 year 5, as it is time consuming to reverse engineer. I also use another method, but this method is much less reliable. Hence the different results when you use the two relative growth reports in my code.

Five-Year Trend Analysis
I actually started my NAPLAN journey looking at the five year trend analysis. The code needs to run on a server and is much less user friendly, so I haven’t released it as yet. My findings suggest that five year trends are more accurate than other NAPLAN data, but at the most they in the ball park of around 80% accurate in accurately predicting the trajectory of the graph from year to year. Over the summer, I seek to include this into the code base and update this post.

A demo of the code used in this project can be found at:
https://www.richardolsen.me/naplan/getting-started

https://github.com/richardolsen/naplan-vis

Any comments, corrections and suggestions are appreciated.

## No, you don’t understand what it’s like to be a man. The case for inquiry in our schools…

In one way it is great to see the focus on post-truth at the moment, but laying the blame at facebook and journalism is wrong, it is not their fault, it is the fault of instructionist teaching approaches. Instructionist teaching approaches are the ones that produce narrow views based solely on the entrenched, usually white, middle-class, and male, views. They teach that knowledge is unchanging and unchallengeable. They teach that perspective doesn’t matter.

Inquiry approaches, however, teach the opposite.

In Situating Constructionism, Seymour Papert makes two arguments for constructionism, his flavour of student directed inquiry learning:

1) “The weak claim is that it suits some people better than other modes of learning currently being used.”

2) “The strong claim is that it is better for everyone.”

Starting with Papert’s asserted stronger claim, why does he believe that inquiry approaches are better for everyone? Papert, correctly understands, that different learning approaches view learning through from different perspectives, and that these vastly different perspectives result in vastly different outcomes, for the student as an individual, and as society as a whole. To reinforce, why this stronger claim was indeed so strong, Papert referenced the hope of feminism and Africanism. In his talk, “Perestroika and Epistemological Politics” [this is a must read] which was presented in Sydney in 1990, Papert explains this further when he claims that instruction cannot ever combat racism, discrimination, misogyny, and other ills of society. Rather, instructionist approaches reinforce what we have good or bad.

As such, Parpert argues that constuctionism is better for everyone because it is likely to bring about justice and equity. While instruction reinforces inequality, inquiry challenges it. In our technology amplified post-truth world, our prejudices and beliefs are never challenged, our erroneous beliefs are constantly reinforced from friends and others who hold similar beliefs and perspectives. Papert argued that this was also true in 1990, before the Internet, Facebook and Twitter.  Papert though, did not believe that truth (as opposed to post-truth) could change society, why, because it never had. Instead, Papert believed that the only way real change could happen is by using new ways of thinking. New ways of thinking based on inquiry approaches to learning, knowledge, and understanding.  For instance, Papert believed feminist pedagogy and feminist ways of thinking were the only ways to challenge and overcome sexism and misogyny. Similarly, Papert believed the only way to overcome racism and apartheid (remember Papert was speaking as a South African in 1990) was to adopt Africanist ways of thinking.

Turning to Papert’s weak argument, who are the people that constructionism suits better? Of course, that’s clear from the stronger argument, constructionism suits the disadvantaged and the discriminated. The inference of the weaker argument is that instruction does suit some people well. Some people benefit from the sexist elements of our society, such as being more likely to be paid more, and promoted more often. Some people benefit from racism. Some people benefit from alternative facts. Some people benefit from denying climate change…

And, some people benefit from instruction.  Yet, Papert would argue that all of the people who benefit from instruction, would benefit from inquiry approaches to learning even more. Yes, even those who now benefit from direct instruction, would derive greater benefits from inquiry approaches.

Whether we are concerned with the world’s move towards the right, or any other of the ills of society, I’m siding with Papert, rather than trying to rewind the post-truth world, we need to embrace new (inquiry-based) ways of thinking. All other solutions have never, ever worked.

As for the title of this post, yes I’m using it ironically, just as it’s used in this wonderful pop song…

## How is learning to speak different from learning to read and write?

It appears de rigueur at the moment to make bold proclamations, usually based on flimsy evidence, about what students need to know, in order to learn this or that. For example, people pushing phonics make claims about the essential knowledge readers need to have about reading. Proponents of direct instruction make claims about the incompatibility of using play or inquiry to learn specific scientific concepts of mathematics, while others outline the non-negotiable knowledge that student writers apparently need to know in order to write.

Recently, I’ve encountered people trying to justify their beliefs in specific essential learning by citing “biological primary and secondary knowledge,” ignoring that fact that Vygotsky, Piaget, and others have differentiated spontaneous and non-spontaneous concepts for eighty years! Though it is disappointing that these academics don’t refer to those who came before them, I feel that this is great opportunity to have discussions about what, when, and how people form scientific concepts. This is the subject of this post.

Vygotsky outlines the differences between spontaneously developed and non-spontaneously developed concepts in Thinking and Speech Chapter 6 The Development of Scientific Concepts in Childhood. He also describes Piaget’s position, and the differences between the two. It is a fairly heavy read, but worthwhile, and I will use examples from this chapter in this post. [Note: The Russian word obuchenie has been poorly translated as “instruction” in English translations of Vygotsky’s writings, instead according to Moll 1992 substitute “teaching/learning” or Wertsch 1988 “teaching-learning processes” whenever “instruction” is used.]

There is general agreement that there are two types of concepts, spontaneous and non-spontaneous, which are often more commonly referred to as everyday and scientific concepts. In this post, I will attempt to outline my understanding of Vygotsky’s position using his examples, from the position of cultural-historical theory.

Even the most devout direct instructionists admit that everyone learns to talk spontaneously. That is, young children form concepts of oral language spontaneously through their everyday interactions with their parents and others, all the way to becoming fluent speakers. Yes, there are many scientific concepts that form our understanding of language. For example, through the study of language we form scientific concepts around the construction of words certain prefixes and suffixes, which provides us the ability to understand unfamiliar words. However, no one that I’ve encountered has yet to claim that specific scientific concepts need to be explicitly taught, and known by children, before they can learn to talk.

Vygotsky uses the contrasting examples of brothers and Archimedes principle, to illustrate the differences between everyday and scientific concepts. We form the concept of a brother through many and varied experiences of brothers through interactions with them. Compare our understanding of a brother with the scientific definition of brother, obtained from the google, a brother is “a man or boy in relation to other sons and daughters of his parents.”  Further, compare this with Archimedes principle from wikipedia “Archimedes’ principle states that the upward buoyant force that is exerted on a body immersed in a fluid, whether fully or partially submerged, is equal to the weight of the fluid that the body displaces and acts in the upward direction at the center of mass of the displaced fluid.”

We can see clearly, and Vygotsky explains as such, that everyday concepts and scientific concepts have opposite strengths. A child’s understanding of what a brother is rich in real-life experiences with brothers, yet they most likely have difficulties defining what a brother is in precise terms, something  Wikipedia also has difficulty with! However, a student’s understanding of Archimedes principle is rich in the abstract. They can learn such scientific definitions by heart and apply them to problems and relate them to other scientific concepts. Yet the weakness of scientific understanding is the ability to associate it with real objects. By simply knowing the principle can they use it to explain what happens to a ship being unloaded at the dock, a helium filled balloon, or a person floating?  Initially, probably not.

Scientific concepts and everyday concepts coexist, they don’t replace each other. Studying the literature featuring sibling rivalry doesn’t replace our everyday concepts of a brother, rather the everyday and scientific concepts connect to enable, as Vygotsky says, the “mastery of the higher characteristics of the everyday concept.”

Another example. I’m currently teaching my oldest child to drive. Cornering speed, understanding the speed in which the car can navigate a corner safely, is extremely complex and difficult to master. Different corner shapes, different road and weather conditions, the width of the road, and the speed zone, all play a part, and largely can only be learned through experience of different corners in different conditions. Some corners taken to cautiously, some taken to fast, and some taken just right. For a beginning driver, actually for any driver, feeling safe is a good indication of whether the speed is suitable, but I’ve suggested to my daughter that if she needs to break in the second half of the corner then she’s travelling too fast. By using this clearly defined scientific concept, she can assess whether she chose the right speed for a corner, and modify her driving in the corner.

Of course, this scientific concept of not needing to break in the second half of the corner, coexists with the everyday concepts developed spontaneously during the thousands of corners she’ll navigate during her 120 hours of learner practice, as required here in Victoria. There is, however, a limit to the scientific concept of slow in, fast out cornering. It would be ludicrous and negligent by me, as her driving instructor, to simply teach her the scientific concept, without the everyday experiences of cornering, as both a passenger and a driver. Without the opportunity to form concepts about driving spontaneously, there isn’t the necessary developmental foundation for the scientific concept to form, and my rule of thumb would be just meaningless words. Again, as Vygotsky says, the scientific concept allows the higher aspects of everyday concepts to emerge.

Some will argue that some things can only be learned scientifically first due to their nature.

Vygotsky points out that the way we learn to speak a second language does not occur the same spontaneous way. When we learn a second language we learn it scientifically first, and everyday second. I’ve already discussed that everyone accepts we learn to speak spontaneously, this is not true of how we learn a second language. In fact, Vygotsky shows, we learn it completely opposite to our first language. In our first language, we can communicate with others right from the beginning, yet we don’t understand grammar and the like. In second languages, we learn the rules of grammar first, and can only communicate with others, and develop spontaneous concepts through oral interactions, is a long way down the learning path.

This is because we learn the scientific concepts of a second language in the context of the everyday concepts of our first language. Scientific concepts of a second language still come after everyday concepts, but they are the everyday concepts of our first language.

Naturally some will argue that reading, writing, mathematics, music, and everything else is closer to learning to speak a second language, than learning to speak a first language. They’ll argue that their subject can’t be learned spontaneously, it needs to be learned scientifically. This is not true. Papert for example, showed how students learn mathematics spontaneously.

Sadly, we’re beginning to see this approach in the teaching reading and writing becoming more widespread. The inability of students to use a phonetical approach to read their own writing demonstrates the flaw in thinking. I’ve seen first hand how the scientific concepts students used to write words on paper, doesn’t enable them to read their own words back. They struggle to use the scientific concepts to write, and then they find it impossible to use the scientific concepts to read their own writing. This is in contrast to much younger students who scrawl a mix of letters, lines and symbols on paper, and read them back clearly and confidently.

Without the opportunity to spontaneously form concepts of writing, through their own form of writing to communicate their ideas, they don’t have the basis to form the scientific concepts. Like a student driver who has never been in a car before, and yet who finds themselves heading towards a corner they don’t have the slightest clue whether to hit the accelerator or the brake! It is not the lack of the scientific concept that is the issue, it is the lack of everyday concepts to give the scientific concept meaning.

The ever bigger danger of the scientific concept first approach, is that the scientific concepts we’re teaching our students might not actually be scientific concepts. Without spontaneously developed everyday concepts it is impossible to tell whether promoted scientific concepts are in fact pseudo concepts. Whether or not the slow in, fast out concept is actually a scientific concept can be ascertained through the everyday concepts already formed, but without those everyday concepts we’re lost.

In this way, for scientific concepts not only need a broad range of everyday concepts to be formed but they also need a broad range of everyday concepts so that they can be adequately understood. As such, teaching the scientific concept of phonetic awareness through non-words appears a particularly futile approach, as it severs the relationship between the scientific concept (phonetic awareness) and the everyday concepts (known words). Rather, strengthening a student’s understanding of scientific concepts of reading and writing, first requires a strengthening and expanding of their experience of everyday concepts.

Footnote.
It is often asked whether teachers need to know learning theory, or whether they just need to know good practice and what works. It is a legitimate question. I do believe an understanding of everyday and scientific concepts show that they two, everyday practice and theory, are not only related but they are dependent upon each other. A strong understanding of pedagogy reveals the “higher characteristics” of teaching, while a strong understanding of pedagogy is dependent on a broad range of spontaneously formed concepts during actual learning and teaching.

## Confessions of a failed edtech entrepreneur

A while ago I reflected on being a stay at home dad. Now that I’ve been back teaching for six months, I think the time is right to reflect on another interpretation of my last four years, that of a failed edtech entrepreneur.

The closing of ideasLAB due to the restructure of the Victorian Education Department was extremely disappointing. Sadly, it is hard to imagine that working for Bruce Dixon for four years, will be topped in my professional life. ideasLAB was formed in 2009, when the ideas of the lean startup were just starting to gain traction. We tried to use some of these ideas at the lab, in order to be more likely to make decisions that offered real value for schools. My voluntary departure payout, which included an undertaking not to work in a government school for twelve months, gave me an opportunity to pursue the entrepreneurial dream. A dream, I ended up pursuing for four years.

Unlike most educators and academics I don’t have scorn for edutech companies. Though, their frequent media releases announcing their plans to “save education” do get old.  Rather, I believe that (unfortunately) education entrepreneurs are our best hope for improving pedagogy. In our current climate of data and performativity, school leaders seem unwilling or unable to try alternative learning and teaching approaches. This is despite the clear failings of current mainstream teaching and learning approaches. External companies, and consultants, appear to offer alternative paths, where school leaders can implement new approaches in their schools without putting their own neck on the line.

To understand the current entrepreneur, and their education startups, you need to understand the lean startup methodology. Lean startup, which has its roots in lean production and agile software development, was conceived and popularised by Eric Ries in 2008. Essentially, lean startup believes that startups are businesses looking for a viable business model. It assumes that an entrepreneur’s assumptions about their product and customers are wrong, and that they will most likely need to be radically changed before a viable business model is discovered. These beliefs, along with their tools and processes, are well established in software development, making a receptive environment for lean startup beliefs, tools and processes. Over the last ten years, lean startup has become the dominate entrepreneurship approach from startups to academia to multinationals.

What makes lean startup a fantastic approach, is that it assumes that the chosen approach is most likely flawed. The lean startup assumes that the entrepreneurs understanding is incomplete, that their assumptions are wrong, and they technologies change the business environment. It allows entrepreneurs to make good decisions in environments of extreme uncertainty and constant change. A lack of time and money is the enemy of the startup and lean startup provides a proven methodology to counter these.

Unfortunately, lean startup has problems. Major problems. When success is defined solely in dollars and cents, problems occur. Uber and AirBnB, and the rise of the sharing economy, are classic cases of this. Despite the rhetoric, the only metrics edtech entrepreneurs have or care about is sales. Not that sales are necessarily a bad thing but when they become the sole focus educational worth becomes lip service. Education startups, using the lean startup approach, won’t build their product until someone will buy it. Education startups cold call schools to sell a product that doesn’t exist. While this is great for product-market fit it doesn’t have anything relevance for understanding whether their product will actually improve student learning or work as claimed.

Despite slogans about improving education and learning, education startups don’t have metrics for this. The only measure of success they have is whether someone is willing to buy their product, and how far their product is towards hockey stick growth. Edtech startups, using lean startup, have resulted in a race to the bottom. They prey on the fears of school leaders. They make outlandish claims about improving learning when in reality they usually push outdated behaviourism, and over collect student data.

For lean startup to have a place in education, it needs to tackle the questions 1) what’s worth doing, and 2) what works. This should be easy as lean startup adherents know that it is most likely that what they believe is wrong. Yet, currently, edutech companies don’t believe, or want to believe, that what they are offering might not be good for students.

Until that day, most if not all edutech startups are just wasting everyone’s time.

And, my confessions:

I never found product-market fit.

I built a solution that no one wanted.

I continue to believe, that my idea still might work.

Even though, deep down, I know I’m wrong.

## A Stay At Home Dad, No More

Sorry that this post is a bit self-indulgent.

The youngest of my three children finished primary school today. I went and picked him up for the last time. Next year he’ll be catching the bus with his sisters to secondary school, and I’ll no longer be needed.

For the last four years, I’ve been a stay home at home dad. It has been an absolute privilege to be able to spend so much time with the kids. I’ve played countless hours of games with them, both before and after school. I’ve had time to coach Ned’s soccer team. I’m also a much better cook. I play guitar every day, and recently I play the piano as well.

Occasionally, people ask me what I do. I usually give a bit of a shrug and mumble something about my PhD, or the business idea I’m trying to get off the ground. Yet, these are hollow excuses. I wash the bed sheets. I clean the shower. I cook dinner and empty the dishwasher. I ferry kids to sport and music practice. And, somehow, these activities fill my day.

There hasn’t been a single moment that I’ve regretted staying home these last four years. Yes, we’re already discussing the house renovations that we hope to undertake with two wages next year… and I’m sure I’ll love working again… but I’m going to miss just hanging out with the kids.

## Ain’t nothing new about the post-truth world

In one way it is great to see the focus on post-truth at the moment, but laying the blame at facebook and journalism is wrong, it is not their fault.

It is the fault of instructionist teaching approaches.

In Situating Constructionism, Seymour Papert makes two arguments for constructionism, his flavour of student directed inquiry learning, as opposed to instructionist learning:

1) “The weak claim is that it suits some people better than other modes of learning currently being used.”

2) “The strong claim is that it is better for everyone.”

Starting with Papert’s asserted stronger claim, why does he believe that inquiry approaches are better for everyone? Papert, correctly understands, that different learning approaches view learning through from different perspectives, and that these vastly different perspectives result in vastly different outcomes, for the student as an individual, and as society as a whole. To reinforce, why this stronger claim was indeed so strong, Papert referenced the hope of feminism and Africanism. In his talk, “Perestroika and Epistemological Politics” which was presented in Sydney in 1990, Papert explains this further when he claims that instruction cannot ever combat racism, discrimination, misogyny, and other ills of society. Rather, instructionist approaches reinforce what we have good or bad.

As such, Parpert argues that constuctionism is better for everyone because it is likely to bring about justice and equity. While instruction reinforces inequality, inquiry challenges it. In our technology amplified post-truth world, our prejudices and beliefs are never challenged, our erroneous beliefs are constantly reinforced from friends and others who hold similar beliefs and perspectives. Papert argued that this was also true in 1990, before the Internet, Facebook and Twitter.  Papert though, did not belief that truth (as opposed to post-truth) could change society, why, because it never had. Instead, Papert believed that the only way real change could happen is by using new ways of thinking.  For example, Papert believed feminist pedagogy and feminist ways of thinking were the only ways to challenge and overcome sexism and misogyny. For example, Papert believed the only way to overcome racism and apartheid (remember Papert was speaking as a South African in 1990) was to adopt Africanist ways of thinking.

Turning to Papert’s weak argument, who are the people that constructionism suits better? Of course, that’s clear from the stronger argument, constructionism suits the disadvantaged and the discriminated. The inference of the weaker argument is that instruction does suit some people well. Some people benefit from the sexist elements of our society, such as being more likely to be paid more, and promoted more often. Some people benefit from racism. Some people benefit from instruction as well.  Yet, Papert would argue that these people who benefit from instruction, would also benefit from inquiry approaches.

Whether we are concerned with the world’s move towards the right, or any other of the ills of society, I’m siding with Papert, rather than trying to rewind the post-truth world, we need to embrace new (inquiry-based) ways of thinking.

All other solutions have never, ever worked.

## This Method Acting, Well, I Call That Teaching

Deborah Netolicky (@debsnet) has written a response to the idea that I shared shared on Twitter that understanding a teacher’s development can be understood through the lens of art. In her post, Deborah reflected on this idea of teaching as art. This post is a response to her post.

Deborah begins her post by asking, “How can we appreciate an artist’s work or know an artist’s worth?”

I’ve been fascinated by this idea of understanding development through the lens of art for a while. At the Perezhivanie Symposium at the beginning of 2015, Michael Michell presented detailed Lev Vygosky’s love of the theatre. Vygotsky’s first PhD was on art (apparently Vygotsky questioned the quality of this research later), and he was also a prolific theatre critic, writing for the local newspaper. What is particularly interesting is that Russian theatre thanks to the ideas of Konstantin Stanislavski, was experiencing a huge shift at the time, due to the development of the Stanislavski Method, or what is now known as Method Acting.

Method acting (or the Stanislavski method) is famed for enabling actors to deliver powerful and compelling performances, as it doesn’t just focus on the technique of acting, but also the emotion of the role. The actor seeks to understand the motivations of the character they are playing, their motivations, their beliefs, and the essence of who they are. Actors who use the method acting technique, famously, might try not to break character between scenes or performances, as they seek to become the actor for the duration of the production or screening.

What is particularly interesting about method acting, is that when the actor takes on a role, they are not just interpreting the role, they are also interpreting the world through the lens of the character. For an actor, say playing the role of a violent drunken abusive character, the actor doesn’t just consider how this person feels and responds, they also consider how the world responds to them, in the present and also in the past. How did they get to where they are? What are the scars? What might have been the pivotal moments in their life? Where did they hope their life might have been? How has the world conspired to get them to where they are? An obviously, the period in which the play or movie is set is also a consideration. What was important then? Why was it important? What does this mean for what we value currently? What have we lost, and what have we gained?

Further, the way this actor understands the world through the lens of this character, gives us insight into how the actor views the real world. While the actor seeks to emotionally immerse themselves in role, experiencing the world through the character’s eyes and anticipating the character’s emotional responses, the actor exposes their understanding of the world. As such this fictitious world, shines the spotlight on the real world.

This lens on development doesn’t just illuminate a method actors development as an actor, but also development of the actor as a person. Something of course, the method actor knows cannot be separated. When we step back and look at a method actor’s career, we might also consider the types of roles they accept. What might the roles they take on tell us about how they understand the world, and how they view their own careers? Looking over a method actors career and the roles they take on and how they interpret those roles, might we be able to recognise change in their understanding, and as educators might this be a window into their development?

Of course, it isn’t just actors whose art and work provides us with a vantage point for understanding of their work. Picasso’s paintings allow us to understand his view of the world, of love and loss, the Spanish Civil War, fascism, Catholicism, and other world events and world views of the time. In music, Bob Dylan’s development as an artist obvious. From the folk singer singing covers, the celebrated pilgrimage to Woodie Guthie, the folk protest singer, the electric sellout, the born again Christian, and lately the celebrator of classic Americana music. Note, like Vygotsky I do not promote the view that development occurs in stages. Though, myself, I am comfortable understand a person’s development through periods that are unique to their development, such as the well-document periods of development of the life of Bob Dylan.

If we sought to understand a teacher’s practice as their art, how might we interpret it? Immediately, we begin to understand, through their practice, what the teacher believes about teacher identity and role. We’d consider what the teacher believes about students, and their capacity. We’d begin to understand what the teacher believes about the role of school and the wider community. Just as for the method actor, their teaching practice, their art, illuminates their beliefs, their world view, and their understanding.

What about the wider environment that influences teachers? Increasingly, we’re seeing a culture of performativity, what does a teacher’s art tell us about this? To other measures of teacher quality? To presence of computers and other technologies? Maybe, rather than seeking to reduce the understanding of who teachers are, and their development, we might seek to understand how they see the world through the lens of their practice. Further, as we get a picture of the period of development (again not maturational stages, but periods unique to their development), we gain a clear understanding for future developmental possibilities.

As such, the teaching environment and how the teacher responds to it, speaks more clearly of their development, than any skills or competencies that could ever be observed in a lesson, or deduced from a test. Particularly, the lens of critical current issues, but not through their response in totally.

For it is in disruption, crisis and the unexpected that defines these periods. Dylan’s visit to Woodie Guthrie. The war around Picasso. The performativity, globalisation, and new technologies around the teacher.

9/11 changed America. How Americans viewed themselves, and how they viewed the rest of the world. Shortly after, in 2002, Conor Oberst through his band Bright Eyes released the song Method Acting. Conor signs about watching this “shocking bit of footage” and “trying to make out the meaning.”  But, Conor isn’t trying to make out the meaning for America, he is trying to make out the meaning for himself, as an artist concerned with justice and humanity. Ultimately, Conor doesn’t make sense of the attack, which he doesn’t even explicitly mentioned in the lyrics. Because that isn’t the point, the point rather is how does make sense of himself. He stays determined to “keep the tapes rolling,” and to “keep strumming those guitars” for what is important is to keep a “record of our failures, we must document our love.”  For to Conor his life is “not a movie, no private screening. This method acting, well, I call that living.” Ultimately, his response as an artist to this terrible period for America is to understand it, the way he always has, by living it. In his words, it’s not method acting, it’s living.

I’m not that interested in trying to understand a teacher’s worth. Rather, I’m interested in who they are, what they are currently developmentally capable of, and what their future development possibilities might be. Similarly, to Conor, this method acting, well, I call that teaching living.

## Poor research and ideology: Common attempts used to denigrate inquiry

A few months ago a prominent Melbourne University academic tweeted “Pure discovery widens achievement gaps” citing the paper “The influence of IQ on pure discovery and guided discovery learning of a complex real-world task” I was immediately dubious of this research, as research that is commonly quoted showing that the inquiry learning doesn’t work, is usually fundamentally flawed. I’m not a proponent of “pure discovery learning” per se, but I feel this type of research, and the reporting on this type of research is designed to denigrate all inquiry learning. In an attempt to leave only teacher instructional approaches standing – why these researchers don’t instead prove a theoretical basis for instruction is beyond me.

So I took a look at the research to see if educators should have any confidence in its reported findings.

TLDR: No, we shouldn’t haven’t any confidence in this research, and it does not show that pure discovery or inquiry approaches widen achievement gaps.

Not surprisingly, this research fails the good educational research test as it doesn’t use a learning theory. That is, the research does not attempt to justify a theoretical basis for its findings. The researcher does use two other (non-learning) theories though, to defend the research, notably game theory, and control value theory. In essence the author uses these theories to defend the research design, yet for some reason he does not believe a learning theory is also required? I find this baffling.

Why doesn’t the author believe that a learning theory is required to define the scope of the research, given that the research is about learning? Why does the author believe that theory is required to explain games, and emotional attainment?

Anyway, let’s look at the research, as I’m always interested in how this type of research is used to investigate inquiry learning, or in this case pure discovery learning.

The author defines pure discovery learning as learning occurring “with little or no guidance. Essentially, knowledge is obtained by practice or observation.” The author spends considerable time explaining how pure discovery occurs in so much of our lives, with ATMs and iPhones requiring people to use them correctly without instructions. He continues, in explaining how Texas Hold’em poker requires people to “use multiple skills to reason, plan, solve problems, think abstractly, and comprehend complex ideas” which are similar to real life pure discovery learning situations.

The author explains: “The poker application used for this study was Turbo Texas Hold’em for Windows, version four copyright 1997–2000 Wilson Software. This is a computerized simulation of a 10-player limit hold’em poker game.”

Wait!!! What????

A computer simulation of a game you play with real people is a suitable method for exploring pure discovery?

Interestingly enough, you can play Texas Hold’em, the software used in this research in your browser thanks to archive.org. (Note: If you’re using a mac use Function + right arrow when it asks you to press End to play.) In playing Texas Hold’em you will discover just what an poor attempt of simulating the playing of poker against nine other simulated people this really is.  It appears that the study data used in this paper is actually from a previous study by the same author, “Poker is a skill” dated 2008. The 2008 date still doesn’t explain why such old DOS software was used! In this paper the author explains that 720 hands of Texas Hold’em over six hours is equivalent to thirty hours of casino play, with real people as opponents. That is 6 hours playing against a computer is supposedly the same as playing 30 hours against real people!

If you play the simulation at archive.org it is easy to see how 30 hours of real play can be achieved in 6 hours using this simulator. Turns made by your computer opponents fly past with short text messages popping up briefly on the screen.  Two groups of students used this old software. The researchers provided the instruction group with instructions of a specific poker strategy, the pure discovery group were, for some reason, given documents detailing the history of poker! The success of players was determined by the money that they had won (or lost), though it should be noted that the participants were not playing for real money. Instead the highest ranking players were promised to be playing to a chance to be part of a raffle for an iPod. This was intended to place meaning, to the otherwise valueless money, each player was playing for.

So this study designed is to simulate a real life complex problem, yet it doesn’t even simulate a real life game of poker!  The participants were not playing against real people. The participants were not playing for real money (though their success was measured as if they were.) And… the participants were playing five times faster than real poker is played.

All of this should make anyone question how the author could possibly argue that this research design can possibly be described as “pure discovery,” as commonly used in real life situations. Interestingly, though the author identifies differences between the instruction and control groups. Neither group learned to play poker to the point where they didn’t lose money. Further both groups played many more hands, more than twice as many hands as poker experts are reported to recommend that “good” poker players play. That is, neither group exhibited the one of the main traits of good poker players, folding around 85% of the time, and only playing 15% of the time.

How might research of pure discovery be better designed?

Duh.

Playing with real people would allow a learner using pure discovery to observe, and seek to understand the decision making of other poker players. Depending on the relationship with the players the learner might ask questions of their opponents, seeking to clarify rules and strategies. Other players might also intervene in the play, offering advice and pointing out pivotal moments in the hand, and pivotal decisions being made by other players. If the participants played against real people, surely they would’ve noticed that they were playing many more hands (much more than twice as many) than their more skilled opponents? Though the computer might be able to simulate the logic of poker, it cannot and does not simulate the interactions between the players, a critical feature of playing any game, and especially poker.

Given that the instruction group did not learn to play Texas Hold’em poker to a satisfactory level, it is obvious that instructional strategies used did not work. Of course, it must be noted neither did the control group, who were left to battle the computer opponents on their own, armed only with a document on poker history. To suggest players playing against real opponents, using pure discovery or other inquiry approaches would also fail to learn to play poker satisfactorily, is obviously outside of the scope of the research, as the researcher did not explore this.

Of course, the lack of a learning theory is what has also led the researcher to his narrow definition of successful learning. Did the author ever consider why people play poker? Is money the only indicator of successful play? Or do people also play games for fun? Are the social aspects of playing with friends an important part of being a poker player?

A more complete understanding of what makes a poker player, a poker player, would consider other indicators, traits, characteristics and motivations.  Did the study participants continue playing poker after the study had finished? Did they enjoy playing poker?  Do they intend to teach friends? Do they feel playing poker with their friends strengthens their friendships? (Not that they were given this opportunity.) Have they developed their own theories and strategies they intend to try out in the future? What do they know about poker?

I believe a better understanding of poker players and the reason people play poker, would greatly improve this poor study. It would also provide further evidence for the worth of learning to play poker by playing poker with friends.  Not that this is an earth-shattering conclusion! After all isn’t that how we all learn to play any new game? Or maybe you’re the one out on the oval by yourself with a ball, a sheet of paper documenting the history of football!

Driven By Ideology?
To suggest that individuals playing Texas Hold’em against a computer mirrors inquiry that happens in our schools in complete nonsense.

To suggest that this research proves pure discovery “widens the achievement gap” is complete nonsense.

To suggest that learning poker by yourself on a computer playing against a simulation has anything at all do with student learning and real inquiry is nonsense.

Do academics who favour high levels of teacher instruction really expect us to believe that inquiry classrooms operate the same way that people learn to play poker individually on their computer?

Do academics who favour high levels of teacher instruction really believe that playing poker on your own against a computer tells us anything about how teacher professional development or teacher pre-service training should be designed?

Do academics who favour instruction really believe that a piece of paper with strategies on them is really the best way to learn anything?

Do academics who favour instruction really believe learning is just about knowing, and not about experiencing with others?

Do academics who favour instruction really believe we’re that gullible?

## Is there evidence that Positive Education improves academic performance? No

Lately there has been quite a bit of talk in education circles about social aspects of learning, particularly well-being, grit, growth and other mindsets, positive psychology and other social and emotional programs.

My personal opinion is that these are a tacit recognition by proponents of direct instruction, that their belief that learning and development is a linear cognitive approach of memorising skills is insufficient. Maybe they are starting to understand that development is highly individual in nature, it is not linear or maturational, and that it is a complex transition to qualitatively new understanding of concepts, new motivations, new relationships with others and the world, new directions, and new results?

Unfortunately, rather than reexamining the more appropriate learning theories of Vygotsky, Piaget and other dialectical approaches to development, these instructionists blindly continue down their misguided path co-opting bits and pieces into their flawed framework. Rather than design learning and teaching so that it IS social, they attempt to teach social as if it was a seperate discrete unit to other learning.

One such model is the Visible Wellbeing Instructional Model. Rather than admitting direct instruction (Visible Learning) and performativity (Visible Thinking) don’t work. They’ve now misunderstood the fundamental aspect of the idea that all learning is social from Vygotsky and Piaget, and instead tried to stuff it into their broken Visible Learning and Visible Thinking model in the hope that it will fix it.

How do they justify this?  Well according to them, Positive Education has been shown to increase student academic results by 11 percent.

Unfortunately for the Visible Wellbeing Instructional Model, this is simply untrue.

In 2011, Durlak, Weissberg, Dymnicki, Taylor, and Schellinger released their meta-analysis of social and emotional interventions. Notice that their paper is concerned with school based interventions, not a study of social and emotional practices that are embedded in standard learning and teaching practice. Their finding that is widely reported as evidence that these programs improve academic results is found in the abstract where they write:

“Compared to controls,  SEL (Social Emotional Learning) participants demonstrated significantly improved social and emotional skills, attitudes, behavior, and academic performance that reflected an 11-percentile-point gain in achievement.”

Seems clear cut right? Wrong!

If you, like me, and seemingly subsequent researchers who quote this research took “compared to controls” means compared to those who didn’t participate in these programs you’d be wrong, because that’s not at all what they are saying… Let’s read the paper further.

In Table 5, they specify the results of their meta analysis:
Skills 0.57
Attitudes 0.23
Positive Social Behaviours 0.24
Conduct 0.22
Emotional Distress 0.24

Though I’m not a fan of effect sizes, as I believe they are completely flawed, consider what John Hattie in the book Visible Learning says about effect sizes:

“Ninety percent of all effect sizes in education are positive (d > .0) and this means that almost everything works. The effect size of d=0.4 looks at the effects of innovations in achievement in such a way where we can notice real-world and more powerful differences. It is not a magical number but a guideline to begin discussion about what we can aim for if we want to see student change.”
(Hattie, p15-17 quoted by http://visiblelearningplus.com/content/faq)

You might notice that all except one of Durlak et al effect sizes fall below Visible Learning’s guideline for beginning discussion about them. The only one is Skills (0.57) so according to their figures only worth of Social and Emotional Interventions are to develop social and emotional skills. Everything else atttitudes (0.23), positive social behaviours (0.24), conduct (0.22), emotional distress (0.24), and academic performance (0.24) fall a fair way below the Visible Learning cut off.

You’re probably wondering, where the 11% gain in academic improvement comes from, in light of its small effect size. To solve this one, we need to keep reading the paper.

“Aside from SEL skills (mean ES = 0.57), the other mean ESs in Table 2 might seem ‘‘small.’’ However, methodologists now stress that instead of reflexively applying Cohen’s (1988) conventions concerning the magnitude of obtained effects, findings should be interpreted in the context of prior research and in terms of their practical value (Durlak, 2009; Hill, Bloom, Black, & Lipsey, 2007).”
Durlak, Joseph A., et al. “The impact of enhancing students’ social and emotional learning: A meta‐analysis of school‐based universal interventions.”Child development 82.1 (2011): 416.

The mean effect sizes in Table 2 (Table 2 contains the same figures as above and broken down into further groups, such as class by teacher, class by non-school) do seem “small,” because they are small! Very small, so small Hattie would no doubt suggest you should ignore social and emotional programs, unless you’re teaching social and emotional “skills” (0.57).

But what do the author’s mean when they say “instead of reflexively applying Cohen’s (1988) conventions”?   I looked up the definition of reflexively… the Webster-Merriam dictionary gives the following meaning:

“showing that the action in a sentence or clause happens to the person or thing that does the action, or happening or done without thinking as a reaction to something”

Now I’m not a methodologist, like Durlak whose other paper is provided as a reference about why the effect size of the social and emotional intervention shouldn’t be understood by the effect size happens because of the intervention. Yet, it does seem a bit of a stretch (to a non-methodologist), to find what the methodologist is an appropriate method of determining its practical value.

What the authors did, as far as I can tell as a non-methodologist, in order to “interpret the practical value of social and emotional interventions” is compare the results to other social and emotional interventions.

I’ll say that again, the 11% improvement in academic results is not compared to control groups who did not have interventions at all, they are 11% gains over students in other social and emotional type programs, and all students experience less improvement than those who did not participate in social and emotional programs.

We can see clearly from the last line of the table that the figure 11% was produced by comparing the effect size of 0.27 to four other studies with effect sizes of 0.29, 0.11, 0.30 and 0.24.

I’ve taken a quick look at these studies. They describe: 1) Changing Self Esteem in Children, 2) Effectiveness of mentoring programs for youth, 3) Primary prevention mental health programs for children and adolescents, and 4) Empirical benchmarks for interpreting effect sizes in research.

I must admit (as a non-methodologist) that I don’t understand why or how the fourth study “Empirical benchmarks for interpreting effect sizes in research” fits the criteria of “prior research” given that, from as far as I can tell it has nothing to do with social and  emotional programs. But what that particular research does describe is that typical effect sizes for elementary school are 0.24 and middle school 0.27.  On that research alone the effect sizes are either level or slightly above expected, hardly a ringing endorsement, nor a source of much faith in the 11 percentage points of academic improvement.

A rudimentary understanding of mathematics also suggests the extremely low effect size (0.11) of study into “Effectiveness of mentoring programs for youth” greatly increased the difference between the study in question and the “prior research.” I’d suggest if that study was deemed not fit the “practical value” of the study then the 11 percentage points figure would’ve been much lower.

So, it seems clear to me the 11 percentage points of academic improvement is determined by comparing it to previous similar studies which didn’t work as well. Any other measure would not have produced the same results.

Of course, to Vygotsky or Piaget these results would not be surprising. For they know you can’t reduce learning and development to individual traits instead we can only understand it as a complex system.  Maybe, the Visible Wellbeing Model is trying to move towards Vygotsky and Piaget? If so, they’re doing it wrong. By attempting to identify and promote three traits of teacher effectiveness, teacher practice, and wellbeing, they’re not seeing them as a system but rather three individual traits together. Yet, at the same time they’re only measuring one trait… test scores. And when you only measure one trait, guess what, the only traits that matter are that trait!

For Positive Education and wellbeing to ever produce an effect size that is substantial, what is measured would need to change, just as they did to produce the contrived 11% figure. But can what Visible Learning effect sizes deem important change? Could they decide what matters while still believing in “evidence”?

Such is the conundrum that the Visible Wellbeing Model finds itself in? Theoretical baseless, considering test scores only worthwhile, what it finds are worthwhile aren’t what they know are worthwhile… No wonder most of us still listen to Vygotsky and Piaget!

Personally, I believe the learning and development is social, so this post is not to belittle the wellbeing movement but rather to suggest reducing social and emotional to skills to be learned though programs and interventions is, in my opinion, a missed opportunity. Further, to think we can bolt on wellbeing in order to improve test scores, is to misunderstand how our students actually learn and develop.

Incidentally, Inquiry-based learning in the incredibly flawed Visible Learning meta analysis comes in at 0.35, maybe it is time it replace Positive Education, with an effect size of 0.27 as one of the three components of their model?

## On Evidence, Research and the Need for Theory

Who is the best swimmer? Swimmer A or Swimmer B?

Swimmer A is part of a regular swimming squad, they can swim 500 metres in a swimming pool using all of the major strokes. Swimmer B can’t swim 500 metres but regularly swims in the ocean, mostly just old school freestyle with their head out of the water, but they can body surf and dive under a wave. They understand about tides and rips, and makes great decisions about when and where to swim. When Swimmer A visits the beach their parents keep a careful watch over them. Swimmer B’s parents are happy for them to go to the beach with their friends unsupervised, as they know their child is a safe, strong and experienced swimmer.

Quantitative approaches might identify Swimmer A as the best swimmer. They might hypothesize that the distance a swimmer is able to swim is the crucial measure of swimming ability. They might not even consider that understanding the relative safety of surf conditions is important, with all their field research of swimming occurring in suburban swimming pools. Qualitative approaches might observe both swimmers at the beach, and come to vastly different conclusions, finding distance as an unreliable measure instead understanding swimming as a far more complex set of skills and characteristics.

Of course, believers in quantitative evidence (test scores) might try to point out that their research (tests, surveys, experiments) are much better designed than my simple example above. They might point to a curriculum as being able to define what needs to be tested, suggesting that well designed quantitative evidence (test scores) are better than qualitative data. But how can we know this is true? How can we know that the evidence that is captured is able to measure what it claims to be able to measure?

In short we can’t.

This is where learning theory steps in.

A learning theory determines how learning and teaching designed, and how development of learners can be understood. The learning theory defines the rationale and process.  We cannot have evidence without it. The theory defines what the evidence is proving or disproving. It explains why a measure of swimming distance is or isn’t a suitable measure. Also, theory defines what it isn’t measuring.

So what makes a good learning theory?

Believable educational research and evidence must pass ALL five of the following tests.

1. Is there a learning theory?
If there is not an acknowledged learning theory which the researcher is using appropriately, then do not believe the evidence at all. A learning theory has a theoretical basis, we are not talking about the methodology or approach. The curriculum is the curriculum it is not a learning theory.

Research and evidence that use meta-analysis approaches, research that simply uses test scores or pre and post tests usually fail this test. Research from fields other than education always fail this test. Sadly, research that fails this test is the worst type of educational research, and probably makes up more than 90% of all published research on learning and teaching. It should all be thrown out.

2. Does the learning theory explains how learning/development occurs?
At the heart of learning/development is a change in the learner. How does the learning theory define this change? Does the theory assert that learning/development is an individual trait, such as distance swum, or is it a combination of traits such as decision making, experiences, and other characteristics. If the research is making claims about things that the learning theory does not view as  being important indicators then the evidence cannot be believed. Trusted evidence reports only on areas the learning theory views as being important.

Research that uses frameworks such as TPACK and SAMR fail this test, they actually fail the first test as well but people don’t see this. Research that suggests student progress can be reported according to months and years behind usually fail this test. Research that uses purely quantitative assessments usually fail this test. Research claiming to disprove other pedagogical approaches often also fail this test as they attempt to use the measures of one theory and apply them to another. Research proving or disproving the usefulness of specific technologies often fail this test.

3. Does the learning theory explain the relationship between teaching and student learning/development?
At the heart of learning and teaching is a belief that well designed pedagogy produces a desired change in the learner. In assessing learning to gain evidence, we are ultimately assessing the effectiveness of all the important facets of the learning and teaching process. Does the theory make a clear rationale for why the factors highlighted in the evidence are actually important facets of the learning and teaching process? If the research and evidence seeks to make claims about learning and teaching design that the theory does not make clear as being crucial then the evidence should be dismissed. Trusted evidence reports only on areas that the learning theory asserts are important.

Research and evidence on engagement, grit, flow, and other motivations or character traits often fail this test. Research on teaching approaches usually fail this test if there are reporting on facets of teaching outside the scope the learning theory.

4. Does the learning theory align with the study?
Of course, the learning theory doesn’t need to justify that studying a trait, or a combination or unit of traits/characteristics/functions is theoretically accurate.  The research also needs to demonstrate why the studied trait(s)/characteristics/functions appropriate for the specific learning/development. As such the appropriate use of the learning theory is crucial to understanding whether evidence is trustworthy.  If the learning and teaching process or design is being studied, the evidence of the research are limited to that specific process or design and cannot be extended to other learning and teaching processes and designs.

Research into the suitability of teaching strategies, media, and specific technologies, by assessing student comprehension often fails this test.  Research around knowing “what universally works” in all situations always fails this test.

5. Does the learning theory in its entirety accurately explain learning/development?
Of course, the biggest test is, if there is actually a learning theory being used to support the research and evidence, does the learning theory make pedagogical sense!  Learning theory must align with how we know learning actually happens. Learning theory shouldn’t be separated from reality, it should fit completely with our everyday experiences of how people learn. This is not to say that learning theory shouldn’t be complex! Complex ideas need complex theories, but complex does not mean we need to make a leap of faith in order to believe or understand them.

Research that uses cognitive load theory and other “how the brain works” rationales usually fail this test.

Footnote: I know it is tempting to take notice of bad research and evidence which we agree with, but please don’t do it. We need to throw away all bad research even when we agree with the “evidence.”