On Evidence, Research and the Need for Theory

Time to read: 4 minutes

Who is the best swimmer? Swimmer A or Swimmer B?

Swimmer A is part of a regular swimming squad, they can swim 500 metres in a swimming pool using all of the major strokes. Swimmer B can’t swim 500 metres but regularly swims in the ocean, mostly just old school freestyle with their head out of the water, but they can body surf and dive under a wave. They understand about tides and rips, and makes great decisions about when and where to swim. When Swimmer A visits the beach their parents keep a careful watch over them. Swimmer B’s parents are happy for them to go to the beach with their friends unsupervised, as they know their child is a safe, strong and experienced swimmer.

Quantitative approaches might identify Swimmer A as the best swimmer. They might hypothesize that the distance a swimmer is able to swim is the crucial measure of swimming ability. They might not even consider that understanding the relative safety of surf conditions is important, with all their field research of swimming occurring in suburban swimming pools. Qualitative approaches might observe both swimmers at the beach, and come to vastly different conclusions, finding distance as an unreliable measure instead understanding swimming as a far more complex set of skills and characteristics.

Of course, believers in quantitative evidence (test scores) might try to point out that their research (tests, surveys, experiments) are much better designed than my simple example above. They might point to a curriculum as being able to define what needs to be tested, suggesting that well designed quantitative evidence (test scores) are better than qualitative data. But how can we know this is true? How can we know that the evidence that is captured is able to measure what it claims to be able to measure?

In short we can’t.

This is where learning theory steps in.

A learning theory determines how learning and teaching designed, and how development of learners can be understood. The learning theory defines the rationale and process.  We cannot have evidence without it. The theory defines what the evidence is proving or disproving. It explains why a measure of swimming distance is or isn’t a suitable measure. Also, theory defines what it isn’t measuring.

So what makes a good learning theory?

Believable educational research and evidence must pass ALL five of the following tests.


1. Is there a learning theory?
If there is not an acknowledged learning theory which the researcher is using appropriately, then do not believe the evidence at all. A learning theory has a theoretical basis, we are not talking about the methodology or approach. The curriculum is the curriculum it is not a learning theory.

Research and evidence that use meta-analysis approaches, research that simply uses test scores or pre and post tests usually fail this test. Research from fields other than education always fail this test. Sadly, research that fails this test is the worst type of educational research, and probably makes up more than 90% of all published research on learning and teaching. It should all be thrown out.


2. Does the learning theory explains how learning/development occurs?
At the heart of learning/development is a change in the learner. How does the learning theory define this change? Does the theory assert that learning/development is an individual trait, such as distance swum, or is it a combination of traits such as decision making, experiences, and other characteristics. If the research is making claims about things that the learning theory does not view as  being important indicators then the evidence cannot be believed. Trusted evidence reports only on areas the learning theory views as being important.

Research that uses frameworks such as TPACK and SAMR fail this test, they actually fail the first test as well but people don’t see this. Research that suggests student progress can be reported according to months and years behind usually fail this test. Research that uses purely quantitative assessments usually fail this test. Research claiming to disprove other pedagogical approaches often also fail this test as they attempt to use the measures of one theory and apply them to another. Research proving or disproving the usefulness of specific technologies often fail this test.



3. Does the learning theory explain the relationship between teaching and student learning/development?
At the heart of learning and teaching is a belief that well designed pedagogy produces a desired change in the learner. In assessing learning to gain evidence, we are ultimately assessing the effectiveness of all the important facets of the learning and teaching process. Does the theory make a clear rationale for why the factors highlighted in the evidence are actually important facets of the learning and teaching process? If the research and evidence seeks to make claims about learning and teaching design that the theory does not make clear as being crucial then the evidence should be dismissed. Trusted evidence reports only on areas that the learning theory asserts are important.

Research and evidence on engagement, grit, flow, and other motivations or character traits often fail this test. Research on teaching approaches usually fail this test if there are reporting on facets of teaching outside the scope the learning theory.


4. Does the learning theory align with the study?
Of course, the learning theory doesn’t need to justify that studying a trait, or a combination or unit of traits/characteristics/functions is theoretically accurate.  The research also needs to demonstrate why the studied trait(s)/characteristics/functions appropriate for the specific learning/development. As such the appropriate use of the learning theory is crucial to understanding whether evidence is trustworthy.  If the learning and teaching process or design is being studied, the evidence of the research are limited to that specific process or design and cannot be extended to other learning and teaching processes and designs.

Research into the suitability of teaching strategies, media, and specific technologies, by assessing student comprehension often fails this test.  Research around knowing “what universally works” in all situations always fails this test.


5. Does the learning theory in its entirety accurately explain learning/development?
Of course, the biggest test is, if there is actually a learning theory being used to support the research and evidence, does the learning theory make pedagogical sense!  Learning theory must align with how we know learning actually happens. Learning theory shouldn’t be separated from reality, it should fit completely with our everyday experiences of how people learn. This is not to say that learning theory shouldn’t be complex! Complex ideas need complex theories, but complex does not mean we need to make a leap of faith in order to believe or understand them.

Research that uses cognitive load theory and other “how the brain works” rationales usually fail this test.


Footnote: I know it is tempting to take notice of bad research and evidence which we agree with, but please don’t do it. We need to throw away all bad research even when we agree with the “evidence.”