Thursday 18 December 2014

Sorting the wheat from the (research) chaff. A rough guide.

A few teachers have commented to me this year that they would have no idea how to determine whether a new piece of research is “any good” or represents a change of practice that they should adopt. While it’s probable that many health science practitioners would also say they struggle with the task of critically appraising new research, I think this is a particular challenge for teachers, who historically, have not been taught about research methods and data analysis in their pre-service training. This leaves teachers vulnerable to “the next new thing” that policy-makers decide to introduce, and makes it hard for them to argue their corner with any confidence.

No blog post can adequately stand in for two, three, four or more years of research methods training, but I thought it might be helpful here to sign-post a few key points to de-mystify some of the research landscape for teachers.

Here’s my Top-10 questions to keep in mind when reading about new research:

1.      Where is the study published?
a.     The optimum answer to this  question is “In a peer reviewed journal”. By “peer review” we mean that the researchers sent their manuscript to an academic journal editor, the editor considered its general suitability for the journal, and then nominated a couple of academic “peers” to conduct a detailed review. This process is often conducted on a double-blind basis – i.e. the reviewer does not know the author’s identity and vice versa. However some journals use an open-review process. The distinction is not important for our purposes here. What matters is the level of scrutiny the paper receives, in terms of the theoretical logic behind its rationale, its method, data collection, analysis and interpretation.

 As any academic will attest, this can be a bruising process and we often need to don a metaphorical rhino hide before opening the email with a subject line “MS 2014XYZ Decision” or similar. Reviewers rarely spend a lot of time on the study’s strengths, highlighting instead its flaws and limitations. This is not a game for the faint-hearted.

The upside though is that most published papers have undergone considerable revision by the time they go to print, and the researchers may have had to patiently and painstakingly address a myriad of queries and challenges to their argument.

b.     But not all journals are created equal. Academics are in the know about esteem hierarchies and metrics such as impact factors. Universities are in the know about these as well, and bring considerable pressure to bear on academics to publish in high-impact journals. One problem with this is that such journals may only be read by other academics and never by practitioners “on the ground” – so while it’s gratifying to have your research cited by other researchers, it may not be translating into meaningful, real-life change.

c.       Media reports, blogs and websites often provide accessible, easy to read summaries of research, but should not be the primary source of research. If they are the primary source, you should remember that the rigorous peer-review process outlined above has almost certainly not taken place.

2.      Who are the authors?
a.     Is there a well-qualified academic on the research team? Particular knowledge of research methodology, data collection, analysis and interpretation is needed in order to conduct rigorous research. Look for evidence that this exists in the research team (e.g. a university-affiliated team leader).

b.     Are there potential or actual conflicts of interest for any of the team? Examples of this might be someone employed by a particular publishing house being part of a team that is evaluating an intervention in which that publishing house has a commercial stake. You should also look at the funding source(s) and ask yourself whether there might be vested interests in the data telling a particular story.

c.      What else has this team published? What do we know about their ideological stance / bias? (everyone has one!)

3.      What was the context of the study?
a.    What country was the study conducted in? You might read a fabulous report of a rigorous piece of research that was conducted in Uzbekistan, and be quite confident that it is tight and well-controlled.  But if there are significant differences between the Uzbeki educational context and your own, you might want to think carefully before adopting any recommended changes.

b.    What is the policy framework in which the study was conducted? Are there particular teaching approaches that are explicitly or implicitly associated with this setting?

c.     What are the demographic  characteristics of the sample? Here we need to think about socio-economic status (SES) factors, ethnicity, culture, religious influences, age and gender characteristics and any other wider influences on the context that might be relevant. The authors might tell you that the study was conducted in “10 schools with similar socio-economic characteristics”, but this doesn’t help you very much if you don’t know what those characteristics were – i.e. were the schools in a disadvantaged area, or were they middle or high-SES? This has important implications for the extent to which findings can be generalised beyond the study  – no matter how rigorous the study itself may have been.

4.      How clearly was the research question stated?
a.    Some studies are highly specific with respect to their purpose and this can be easy to see even from the title. Unfortunately, though, some research studies are a bit like fishing expeditions – the researchers pack their gear and head out into the wild to see what they can find. While it is absolutely appropriate for qualitative studies to take a broader sweep around “exploring and understanding” a phenomenon, you should always have a clear sense of what the researchers are examining and why.

5.      How adequate was the sample and the description of the intervention?
a.     Here we’re interested in issues like sample size (e.g. the number of teachers, students, schools etc) included. However there is no simple absolute answer to the question “How many is enough?” Sample size should, however be based on some kind of “power analysis” – a statistical consideration of the nature of the questions asked and the number of participants needed to test an hypothesis. If for example, you wanted to know about differences in vocabulary size between four year olds and eight year olds, we would expect that age would have a “big effect” and we would need a relatively smaller sample than if we were studying the differences in vocabulary between four year olds and four-and-a-half year olds – here there will be more developmental blurring between the two groups, and so to find an age effect (assuming one actually exists), a larger sample would be needed.

b.    Is there any potential bias/distortion due to sampling processes? An obvious issue in schools-based research is the (usual) requirement for parent/guardian consent. However it may be that parents from non-English speaking backgrounds cannot adequately understand the Information Sheet and Consent Form, and so decide (quite reasonably!) to not complete them and return them to the school. This will then introduce a systematic bias into the sample, and means findings can really only be generalised to other groups of similar composition.

c.     In studies that involve any kind of pre-post comparison (e.g. collection of baseline data at Time 1, an intervention phase, and collection of follow-up data at Time 2), it’s important to think about retention of participants over time, and most importantly to look at the characteristics of participants who were lost to follow-up. Often, these are from minority groups or have some other defining characteristic (e.g., frequent suspensions due to behaviour problems) that might in itself influence the Time 2 scores.

d.    If it is an intervention study, how were participants allocated to study arms (research Vs control?) Ideally this should occur via a process of randomisation, so that potentially confounding variables (e.g. ethnicity, IQ) are equally distributed across study arms and so are “cancelled out” in the analysis. In some medical research, it is possible to conduct “double blind” trials, in which neither the participants nor the researcher interacting with them is aware of who is in which group. This is harder to do in schools, for obvious reasons, but in general, you should look for evidence that the researchers did not influence the allocation of individuals or schools to one study arm or the other. 


e.   Also with intervention studies, ask yourself about the basis of the intervention. Does it have a theoretical rationale that draws on previous research, or is it just someone's idea about what might work? There's unfortunately been way too much of the latter in education. You should also ask whether the intervention was delivered as intended (so-called fidelity) and whether anything else might have happened during the intervention that could independently account for an apparent improvement in student performance.

6.      How suitable are the measures for the questions asked?
a.     If I told you that I was going to measure children’s IQs, and then proceeded to take out a tape measure and record their head circumferences, I think you would rightly howl me down for using an inappropriate measure of IQ. Fortunately, extremely poor choices such as this are not common, though it is common for researchers to select assessment tools that others consider to lack validity (accuracy) or reliability (consistency and trustworthiness). We also need to consider how current the measures are and whether they are widely known and well-regarded.

b.    Who conducted the assessments / measurements?  Just as we don’t want doctors interviewing their own patients about the acceptability of a new treatment, we don’t want teachers assessing their own students. Humans are prone to all sorts of conscious and unconscious bias, whether as the observer (see Rosenthal Effect) or as the observed (see Hawthorne Effect).

7.      How clearly are the results presented?
a.     Are all of the results presented, or just some of them?

b.    Often it’s necessary to have a good grasp of statistics to wade through this section of a paper, so don’t be put off if you don’t feel you bring the necessary background knowledge to the table. If necessary, consult with someone who is more confident with this territory, but persevere with other sections of the paper. Many academics would probably privately admit that they don't give this part of the paper the focus they should – which is a shame as it’s often the most difficult to write!

8.      When results are discussed, are a range of possibilities canvassed to account for the findings, or do the authors just stick with their original hypothesis?

a.     Unfortunately there is a well-known bias in what gets published, and findings that don’t sit well with researcher bias and/or the prevailing zeitgeist often just don’t see the light of day. Happily though, that is beginning to change, and academics and journal editors alike are a little more open to publishing findings that might be unexpected. Here we want to see a range of possibilities being canvassed, and the importance of future replication studies being noted. The language used should be appropriately cautious and circumspect, e.g., "These findings suggest....", or "Our results are consistent with the notion that .....".

9.      Are limitations acknowledged and addressed?
a.    All research has limitations, and most researchers are acutely aware of this when they submit a paper to a journal (if they weren't beforehand, the review process normally fixes that!). So you should expect that some limitations and their potential importance are considered (e.g.  small or biased sample, limited follow-up time).

10. Are implications for theory, practice, policy and/or further research stated?
a.      The purpose of research is to effect change  - in at least one of theory, practice, and policy. So the authors should present some ideas about the implications of their work (without over-reaching of course) and should make constructive suggestions as to how other researchers can advance the field even further.


This is by no means an exhaustive guide to critical appraisal and nor is it intended to be such. It is however, intended to guide the novice and instil some confidence that even without detailed statistical knowledge, you can still be an astute consumer of new research.

Remember too, that we rarely change tack on the basis of one study. Instead we rely on consistent trends in well-conducted research, to guide policy and practice – so look out for systematic reviews or meta-analyses, both of which pool findings on a particular question and synthesise the current state of the evidence.

I would recommend that all teachers bookmark the Macquarie University MUSEC Briefings page, as this open-access site provides reliable, independent assessments of a range of approaches that may or may not be well-supported by research evidence.


As in all things though, it is wise to remember that when something seems too good to be true ..... it probably is. 

"Trust me, I'm a researcher" is never enough.



(c) Pamela Snow 2014

Thursday 11 December 2014

Dear Santa

It's been another busy year! Have I been "good"? Well, that's always open to interpretation, as this T-shirt reminds us -




However I can tell you that I've enjoyed being more active on Twitter this year than I was in the previous three years after I created a Twitter handle and had absolutely no idea why, or how I was meant to use it. Somehow I think I just assumed that creating a social media presence would mean that I was...... well..... "present", somewhere....... socially. However it wasn't like that Santa. In the early days, I didn't Tweet, no-one followed me and I made very limited inroads into working out who / which organisations I should be following.

Oh sure, Twitter was very helpful, by sending me intermittent emails, telling me that so-and-so "had tweets for me", and suggesting Twitter handles that I should follow, using clever algorithms derived from my ghost-like activity on the platform.

But you see, Santa, I just. didn't. get it. Twitter that is - why would I tweet? (Who was listening?). What would I tweet about? (I am really only interested in the inner-most thoughts and daily routines of my nearest and dearest and had seen some truly mind-boggling over-sharing via Facebook). And who would want to interact with me via this medium? (I know who I know, but I don't know who I don't know). Total confusion and bewilderment.

It turns out Santa, that what I needed was some Direct Instruction. I know, I know, that's not popular anymore, and you of all people know how fashions and fads come and go. However I was fortunate to find Professor Dorothy Bishop's helpful blogpost A gentle introduction to Twitter for the apprehensive academic which I sat down and digested in full, and then I understood enough of the why, what and how of Twitter to really get into it. 

A year on, Santa, I find myself feeling much more connected with an international community that shares my interests - though not always of course my perspectives, which is just fine, and is often quite informative in its own way. 


I don't mind people disagreeing with me, but Santa I do mind when people reject good science in favour of propping up an ideological position that entrenches disadvantage for children who are not even at the developmental starting line when they commence school. 


So Santa - I'm wondering if you could pop a copy of the latest edition of the Australian Journal of Learning Disabilities into the sacks of academics all around the world (but especially in Australia) who teach our next generation of primary school teachers? The paper entitled 

What teachers don't know and why they aren't learning it: addressing the need for content and pedagogy in teacher education by Louisa Moats is this year's must-have for all teacher educators.

Santa I think this could be a wonderful gift to the children of the world, and who knows, your job might be made easier too, because you'll receive such well-composed letters in future years! Though some of them are already pretty cute, as you'd have to agree.


Tuesday 9 December 2014

The Speech Language Pathologist-Teacher Tango


Image source: http://www.freedigitalphotos.net/ 

Because of my interest in the language-literacy nexus and school success, I was very interested to read the following in a recent post on the Speech Language Literacy Lab blog: 

Service delivery models have changed substantially over the last few years. While small pull-out groups are appropriate in many situations, keeping students in their classrooms is more of a priority than it has been in the past. As SLPs, we are often told by administrators that we should be "pushing-in" to general education, and providing services in the classroom. 

This post called to mind some thought-provoking conversations I had with various UK colleagues during my recent visit there. Professor Courtenay Norbury (Royal Holloway, London) was lamenting the fact that Speech Language Therapists in the UK are increasingly providing a secondary consultation service to teachers and classrooms, spending very little time working 1:1 with those children whose poor language skills pose serious and imminent threats to school success, both academically and socially. Courtenay and I discussed the fact that classrooms are educational, not therapeutic environments, and teachers (or indeed teaching assistants) cannot be expected to provide the kinds of specialised, individually-tailored intervention that children with significant language difficulties need, in order for them to engage with the curriculum and form social connections with peers. If SLTs (SLPs in Australia and the USA) are operating more at Tier 1 in a Response to Intervention (RTI) framework, with occasional Tier 2 work and virtually no Tier 3 interventions, this has serious implications for (a) the extent to which such children have their educational trajectories altered, and (b) the development and maintenance of professional SLP skills in ameliorating complex expressive and receptive language disorders in young children. Courtenay observed that a consultation-only model is a sure-fire route to redundancy of the specialist knowledge and skills that sit within the SLP profession.

Some commentators have referred to SLPs providing their 1:1 services to children in "homeopathic doses" -  a charge that most clinicians would recoil from, yet it is hard to argue the other corner if a sufficient intervention frequency and intensity cannot be demonstrated. It will also be difficult to establish the efficacy of SLP interventions if they are conducted in ways that promote classic Type II errors, if not statistically, then certainly in the minds of administrators and policy makers. Remember the dinosaurs?

The other conversation that stands out for me in relation to SLPs and teachers working together on the issue of early literacy instruction is the one I had with Professor Bill Wells at the University of Sheffield. When I commented during a presentation that teachers have, in recent years, received uneven pre-service preparation on the linguistic basis of the transition to literacy, Bill rightly asked me whether I thought SLPs learn enough about how reading should be taught during their pre-service education. While I can't call on any empirical data to answer this question, my guess is that an audit of SLP curricula would reveal similar gaps and unevenness to that which has been reported in teaching curricula concerning linguistic precursors to reading.

So - if we are wanting SLPs and teachers to meet in the middle, then it really will take two to tango.

Dancing Tango

Faculties of Education need to ramp up their curricula with respect to linguistic precursors to literacy (vocab., phonemic awareness, narratives, syntactic complexity and so on) and Speech Language Pathology curricula are going to need to cover historical, epistemological, and pedagogical approaches to reading instruction. If this isn't done, too much time will be lost in trying to deal with turf issues and find a common language between professions. If it is done, however, the children who really need them might have better chances of receiving those Tier 2 and 3 services that genuinely impact on their language, academic, and social struggles.

I am absolutely all for SLPs and teachers working collaboratively at Tier 1, sharing knowledge, both of theory and of individual children. At Tiers 2 and 3, however, SLPs need to be able to offer targetted services to children whose language needs will never be met in the context of the mainstream classroom. Advocacy for this needs to come both from the education sector and from SLP/SLT peak bodies. Failure to do so will "dumb down" the skill-base of the SLP profession and will entrench developmental disadvantage for those children whose language skills are not adequate to meet the rapidly changing academic and social demands of the classroom.


Image source: http://www.freedigitalphotos.net/ 


(C) Pamela Snow 2014

Wednesday 12 November 2014

Re-visiting the Language House

In a blog-post back in December last year I shared an image of a "Language House"  that I often use in professional development sessions with teachers.

As a result of a recent very enjoyable after-dinner discussion with Sally, a Speech-Language Pathologist  in Madison, Wisconsin, I've slightly modified this, to take in the importance of such "houses" being built on solid ground - in this case, the social contexts and purposes for language use in the real world.

I have pasted the updated image below - I think it will make our house both more solid and better able to resist a metaphorical force-10 gale, whether academically or socially.




(C) Pamela Snow 2014

Thursday 6 November 2014

Language skills and the social gradient



I have recently returned from a very busy 5-week overseas trip as part of my sabbatical leave this semester. This trip enabled me to visit a small handful of researchers and academic departments whose interests intersect with mine with respect to language development/disorders, early literacy instruction, and psychosocial aspects of being an adolescent with a language-learning disability. I spent most of my time in the UK, but also had some time in the USA, the latter predominantly dealing with issues pertaining to language competence of young people in the youth justice system.

My trip provided some wonderful opportunities to reflect on contemporary issues and challenges pertaining to the language-learning impairment field, and to consider ways in which current research agendas in various centres are tackling different aspects of some of the “big questions” of our time.

For me, one of the most compelling issues for researchers, policy makers, and practitioners is the social gradient that exists with respect to early language exposure, and to language skills and academic success during the school years (and no doubt beyond, but data is lacking on this).


Many readers of this blog would be familiar with the work of Hart and Risley in the USA in the mid 1990s. In a nutshell, these workers recruited 42 families who were stratified across three layers of socio-economic status (SES) spectrum – parents from professional backgrounds, those from working class backgrounds, and those who were recipients of state welfare. At the outset of the study, these families had an infant who was 9 months old, and over a period of 36 months, Hart and Risley tape-recorded an hour per month of child-directed parental language. Their quantitative findings are summarised in the figure below, however it is must be noted that there were important qualitative differences across the SES spectrum too. Most notably, children of professional parents tended to experience parental language that was more encouraging, and they also received more expansions and elaborations on their utterances.



Although Hart and Risley’s work in this area is probably the most well-known, it’s important to note that similar findings have been reported by other workers as well, e.g. Hoff (2003), Locke et al. (2002), Spencer et al. (2012), and Weisleder and Fernald (2013). Also, it must be remembered that of course some families who are struggling economically never-the-less have the human and social capital to provide linguistically enriching environments for their children. Conversely, some high-income families struggle in this regard. So SES correlates with, rather than determines the nature of the verbal input that children receive. Studies such as these are not about "parent-blame", irrespective of the fact that the tabloid media will mis-use them in that way from time-to-time. Rather, this line of inquiry is simply about being data-driven and then considering applications of the data - as should be the case in all social sciences research 

In our work on young offenders, we have reported high rates of language impairment on formal testing, and of course young offenders as a group overwhelmingly come from low SES backgrounds. However we have also examined the language skills of non-offending controls from similar SES backgrounds and have found that in spite of being an average of two years younger, the controls out-performed their offending peers across the board on language measures.

So – low SES alone does not account for poor language skills in vulnerable young people; nor, incidentally, are such differences accounted for on the basis of nonverbal IQ. Other factors, such as chaotic family structure, disrupted education, and developmental comorbidities such as behavioural difficulties all need to be considered.

Perhaps language is not quite as robust to the effects of environmental disruption / deprivation as we might previously have believed. 

What is inescapable, however, is the fact that there is a social gradient with respect to early language exposure and at least mid-range language outcomes such as academic achievement. Schools, teachers, clinicians and policy makers all know that some schools cater to communities that are economically and socially disadvantaged and this disadvantage manifests as lower achievement overall on tasks that require strong verbal skills. And it’s hard to think of many aspects of an academic curriculum that don’t require strong verbal skills.   

However inescapable such a conclusion may be though, it raises many more questions than it answers, for example:

        What is the basis of this gradient? Exactly what is it about being poor, and/or less well educated, that reduces the quantity and quality of language experiences made available to young children? Is this purely a social-environmental phenomenon that can be “rectified” by changing the behaviour of key adults in a child’s immediate networks?

     Is it possible that what looks like a social phenomenon actually reflects an underlying, genetically transmitted disadvantage with respect to oral language skills? I am no expert on genetics (far from it!), but of course we must recognise that many human characteristics are genetically influenced, if not determined – e.g. height, body morphology, eye colour, skin tones, and so on. There is an established link between genetic factors and  language skills.  Is this link enough, however to account for a stratification of society with respect to capacities for acquiring and using language?

        If the social gradient is biologically determined, is there any point trying to intervene to improve the life-chances of children with “deficient” language skills? A nihilistic view of the world would suggest that such interventions are unlikely to be beneficial and/or economically justified. The evidence on interventions such as Head Start (the USA) and Sure Start (the UK) is not exactly bursting with reasons for excitement over the benefits of targeting “at-risk” communities and providing a verbal “top-up” prior to school entry.

        If a child comes from an economically disadvantaged community/family/school and has oral language skills that place him/her in a clinical range on standardised language measures, do we simply attribute the language profile to social circumstances? Can such a child also have what would, under other circumstances be referred to as a “specific language impairment” (SLI)? If the answer to the last question is “yes”, how do we determine which poor children have “language impairments”, Vs having language skills that are commensurate with their position on the social gradient, albeit woefully inadequate when it comes to academic and social success at school?

These are all big questions facing child language researchers, policy makers, and practitioners in speech-language pathology as well as education. These questions are also central to the current terminology debate concerning children with language difficulties.

My view is that while it is important for researchers to vigorously investigate and classify the various aetiological pathways that lead to compromised language skills, we cannot forsake the academic and social outcomes of large numbers of children simply because we know they are “poor”.  In fact, evidence would suggest that such children may derive particular benefits from rigourous application of evidence in the early years’ classroom. For example, a pilot investigation we recently conducted in Victoria suggests that targeting teacher knowledge and practice with respect to early oral language and literacy development has a direct and beneficial effect on the performance of children in disadvantaged schools. A larger scale investigation is now attempting to replicate this finding, and I hope I can report positive outcomes via this blog in due course.

References / Further reading


Hart, B. & Risley, T. (1995). Meaningful Differences in Everyday Parenting and Intellectual Development in Young American Children. Baltimore: Brookes.
Hoff, E. (2003). The specificity of environmental influence: Socioeconomic status affects early vocabulary development via maternal speech. Child Development (74), 1368-1378. 
Jordan, J. A., & Coulter, L. (2016). Individual Differences in Speech and Language Ability Profiles in Areas of High Deprivation. Child Care in Practice, 1-14.
Locke, A., Ginsborg, J. & Peers, I. (2002). Development and disadvantage: Implications for the early years and beyond. International Journal of Language and Communication Disorders, 37, 3-15.
Roy, P. & Chiat, S. (2013). Teasing apart disadvantage from disorder. The case of poor language. In C.R.Marshall (ed.) Current Issues in Developmental Disorders (pp. 125–150)London: Psychology Press.
Spencer, S., Clegg, J., & Stackhouse, J. (2012). Language and social disadvantage: A comparison of the language abilities of adolescents from two different socio-economic areas. International Journal of Language & Communication Disorders, 47, 3, 274–284.
Weisleder, A. & Fernald, A. (2013). Talking to children matters: Early language experience strengthens processing and builds vocabulary. Psychological Science, 24(11), 2143-2152.




(c) Pamela Snow 2014