Priscila Corção B. Moraes1,2¶, Bruno Figueiredo Damásio2¶, Gabriel Cardoso Medeiros de Lima1,3¶, Felipe Kenji Sudo1&, Paulo Eduardo Luiz de Mattos1,4&
1 D’Or Institute of Research and Education, Rio de Janeiro, RJ, Brazil.
2 Institute of Psychology (IP), Federal University of Rio de Janeiro (UFRJ), Rio de Janeiro, RJ, Brazil.
3 Veiga de Almeida University, Rio de Janeiro, RJ, Brazil.
4 Institute of Psychiatry (IPUB), Federal University of Rio de Janeiro (UFRJ), Rio de Janeiro, RJ, Brazil.
¶ These authors contributed equally for this work: Conceptualization, Data curation, Formal analysis, Investigation, Methodology, Resources, Writing – Original Draft Preparation.
& These authors also contributed equally for this work (Senior Authors): Conceptualization, Methodology, Project Administration, Resources, Supervision, Writing – Review & Editing.
Received: 03/14/2019 – Accepted: 11/16/2019
Address for correspondence: Priscila Corcao B. Moraes. Instituto D’Or de Pesquisa e Ensino, Centro de Neuropsicologia Aplicada (CNA). Rua Diniz Cordeiro, 30, 2º andar, Botafogo – 22281-100 – Rio de Janeiro, RJ, Brazil. Telephone: +55 () 2295-3796/+55 (21) 2295-0806. E-mail: [email protected]
Background: Parents and teachers can be valuable sources of information for characterizing children’s ADHD-related impairments in different environments. However, evidence indicated that those categories of informants often provide conflicting responses in formal assessment scales, which may challenge diagnostic decisions regarding the condition. Objective: We aimed to investigate reliability rates between parents and teachers of children and adolescents with and without ADHD using SNAP IV. Methods: 199 children and adolescents aged 6 to 17 years were evaluated for ADHD symptoms using parent-rated and teacher-rated SNAP IV scales. Intraclass correlation coefficients were analyzed for ADHD domains (inattention and hyperactivity/impulsivity), as well as for defiant-oppositional behavior. Results: Reports from parents and teachers showed low reliability for all ADHD domains. Parents’ scores on the SNAP IV were higher than those of teachers. Parents and teachers provided highly discrepant responses concerning to the presence and severity of ADHD in children and adolescents, which might result from intrinsic aspects related to their daily functioning in different settings. Discussion: Clinicians should consider those trends in parental and teachers’ responses when interpreting results from informant-based instruments for detecting ADHD.
Moraes PCB et al. / Arch Clin Psychiatry. 2020;47(2):35-9
Keywords: Reproducibility of results, attention-deficit disorder with hyperactivity, Psychiatric Status Rating Scales, attention-deficit and disruptive behavior disorders.
Attention-deficit/hyperactivity disorder (ADHD) is a life-long and potentially disabling neurodevelopmental disorder characterized by inattention and/or hyperactivity-impulsivity1. During the last decades, along with its establishment as a clinically relevant condition in the scientific and medical community, massive public interest for the disorder propelled large-scale pursuit for screening and parental guidance by millions of concerned individuals and their families2. Overall prevalence rates for ADHD have been estimated to be as high as 7% of children and adolescents and at least half of those individuals may present persistent symptoms into adulthood3,4. However, as awareness of ADHD increased, emerging allegations of overdiagnosis and overmedication, notably relative to school children, have been shifting public opinion towards skepticism for the importance of this clinical entity5. Controversies centered around ADHD assessment are abundant, especially regarding the validity of the diagnostic approaches6,7.
Recommended practices for detecting ADHD, according to panel of experts, include a systematic investigation of the subjects’ history of symptoms, including age of onset, psychiatric comorbidities and evidence of impairments in at least two areas (school, work, home and interpersonal contacts)8. Gathering information from multiple respondents, such as family members, teachers or coworkers has been recommended to allow identification of functional impairments in different environments1,9,10. In special, considering that self-reports of children about their own behavior can be inaccurate and that adolescents tend to underestimate their symptoms10,11, collecting collateral impressions of the presence of ADHD clinical features in a variety of scenarios may be crucial for the assessment of this population. Among validated informant-based instruments for this purpose, the Swanson, Nolan and Pelham rating scale – 4th version (SNAP IV) is possibly the most widely used one in research settings1,12,13. It comprises twenty-six questions, corresponding to the list of ADHD symptoms, as depicted in the DSM-51. Those include nine items aiming at detecting inattention, six assessing hyperactivity, and three measuring impulsive behavior.
Nonetheless, gaps in the knowledge regarding the best methods to assess ADHD are far from being filled. Several studies have suggested that reports from parents and teachers are often conflicting or weakly correlated, mainly regarding the presence and magnitude of clinical features. This shortcoming might challenge the interpretation of the results and raise serious doubts about the instruments’ clinical utility, including for distinguishing the ADHD presentations (predominately inattentive type, predominately hyperactivity/impulsive presentation, and combined type)14,15. Detailed analyses of the responses suggested that teachers tended to be more benevolent in scoring symptom severity and that they might consistently report lower frequency of hyperactivity than parents14,15. Possible sources of divergences across informants may result from different cultural and environmental aspects, such as discrepancies between less structured home settings compared to typical schools and to varying amount of time spent with the children15. Moreover, it is known that raising a child with ADHD might demand more time and energy on caring that with non-ADHD children, which can lead to parental burnout16.
Several strategies have been proposed in the literature to overcome those inconsistencies and to clarify whether informant-based tools are valid approaches to assess the disorder. Among them, mathematically-derived indices obtained from comparing responses from different sources for the same instrument have been largely used17. Inter-rater agreement, for example, measures the matching rates of the absolute scores across evaluators for one specific test17. However, as previously mentioned, results from parents and teachers in standardized scales are generally highly discordant, which corresponds to low inter-rater agreement. Alternatively, inter-rater reliability could be employed to investigate consistency of classification patterns across different raters for one particular instrument, regardless of the absolute scores17. Noteworthy, inter-rater agreement and reliability are often uncorrelated, which means that high scores for one measure do not necessarily predict changes in the other index17.
In the present study, we aimed to investigate the reliability rate of responses of parents and teachers of children and adolescents aged 6 to 17 years in the SNAP IV. We hypothesized that despite low agreement, reliability might allow further information on the validity of the instrument as an assessment tool.
Participants were recruited at the Centro de Neuropsicologia Aplicada, D’Or Institute of Research and Education (IDOR) in Rio de Janeiro, Brazil. Most subjects were volunteers referred to the service by health professionals (physicians, psychologists and speech therapists), whereas others sought our institute due to recommendation from school personnel (9.5%) or upon spontaneous demand (20.3%).
Participants were included if the following criteria were fulfilled: (i) age between 6 to 18 years, corresponding to school-aged children and adolescents; (ii) regularly attending school, as informed by parents and teachers; (iii) living with one parent or both and (iv) all procedures were adequately completed. Exclusionary criteria were: (i) epilepsy, (ii) any psychotic disorder, (iii) current illicit drug use as reported by parents or self reported.
Patients and parents underwent an initial medical interview, followed by neuropsychological tests including WISC-IV18 and all subtests (intelligence, attention, working memory, vocabulary, reasoning, abstraction, visual perception, constructional abilities, dexterity) and CPT – Continuous Performance Test (attention). The Swanson, Nolan, and Pelham, version IV (SNAP-IV) scale was completed by one of the parents and one school teacher12. This instrument contains 26 items which evaluate ADHD domains and comorbid oppositional-defiant behavior (ODD), distributed as follows: Inattention (9 items), Hyperactivity/impulsivity (9 items) and Opposition (8 items). Existence of more than one respondent for the scale (for example, when the two parents or more than one teacher completed it separately) was resolved by randomly withdrawing one of the repeated versions. All individuals from the sample had collateral reports from both parents and teachers.
Diagnoses of ADHD and other comorbid or concurrent mental disorders were based on results from the whole assessment protocol and were made according to the DSM-5 criteria by trained psychiatrists1. Upon this gold standard classification, sample was divided into groups with and without ADHD.
All parents provided a written informed consent prior to enrollment in this study. The research has been approved by the Ethics Committee of Instituto D’Or de Pesquisa e Ensino (IDOR), Rio de Janeiro, RJ, Brazil.
For the present study, reliability was defined as a measure of consistency among evaluators in the ordering or relative position of performance evaluations irrespective of the absolute value of each evaluator’s rating19. In other words, it reflects the relative pattern similarity between two or more sets of ratings19. To evaluate the reliability between parents’ and teachers’ reports on the symptoms of inattention, hyperactivity/impulsivity and challenging/oppositional behavior, intraclass correlation coefficients (ICC) were calculated through one-way randomized analysis and selected measures mean of intraclass correlation. Values lower than 0.5, between 0.5 and 0.75, between 0.75 and 0.9 and greater than 0.90 were considered as representing poor, moderate, good and excellent reliabilities, respectively20. The Cronbach’s α was used to investigate the internal consistency of the ADHD domains in the scale across responders21. To examine which of the respondents observed higher and lower levels of symptoms, mean scores given by categories of observers were compared, using t-test analyzes of repeated samples. Bootstrapping (1,000 re-sampling, with 99% confidence interval) was implemented, aiming at correcting for possible deviations from the normal distribution. Statistical analysis was performed using SPSS for Windows version 19.
Classification and sociodemographic characteristics of the sample
From an initial sample of 225 children and adolescents, 199 attended the eligibility criteria and were included in the study. Age ranged from 6 to 17 years (M = 10.60; SD = 3.23) and schooling varied between 0 years and 11 years (M = 4.17; SD = 3.12). Diagnoses of the sample were as follows: ADHD (n = 77); Autism Spectrum Disorders (n = 15); Communication Disorders (n = 10); Learning Disorders (n = 39); Conduct Disorder and Oppositional Defiant Disorder – ODD (n = 3); Intellectual disability (n = 20) and other DSM-5 diagnoses (n = 9). 15 individuals presented no psychiatric disorder and 11 showed subthreshold attentional deficits, which did not meet the DSM-5 diagnostic criteria for ADHD or for any other conditions. Responses for ODD items were available for 189 participants, 73 of which had ADHD. Of note, individuals could score points in the ODD subscale without reaching diagnostic threshold. Subsequently, the sample was divided into two groups: (i) ADHD (n = 77) with ages ranging from 6 to 16 years (M = 10.06; SD = 3.03) and schooling between 0 and 10 years (M = 3.73; SD = 2.93) and (ii) non-ADHD (n = 122) with ages ranging from 6 to 17 years (M = 10.91; SD = 3.31) and schooling between 0 and 11 years (M = 4.42; SD = 3.21). Sociodemographic variables were not significantly different between the two groups.
Differences between parents and teachers’ responses
Parents’ reports indicated higher mean rates for symptoms of inattention and hyperactivity than teachers in both ADHD (p < 0.001) and non-ADHD samples (p < 0.001). On the other hand, no group difference for the severity of ODD symptoms was detected across informants. Table 1 depicts those results.
Assessment of internal consistency and inter-rater reliability
For most SNAP IV items, a good internal consistency was observed for both ADHD and non-ADHD groups. On the other hand, a Cronbach’s α of 0.71 for parents’ reports of Inattention in ADHD group indicated a lower, but acceptable, internal consistency. Moreover, for all SNAP IV subscales, inter-rater reliability was fair, as indicated by low ICC values. Those results are summarized on Table 2.
In line with previous reports22,23, the current study showed that parent-rated and teacher-rated SNAP IV adequately measured ADHD domains, as well as ODD symptoms, as indicated by a satisfactory internal consistency for all the subscales. In contrast, inter-rater reliability was low for all the instrument’s subscales, which demonstrated that perception of symptoms was largely divergent across parents and teachers. Precisely, parents rated symptoms of inattention and hyperactivity as significantly more frequent and severe compared to teachers in the instrument, whereas such discrepancy was not verified for the ODD items.
Evidence of lower perception of children’s cognitive, emotional and behavioral problems among teachers in comparison to parents had been previously described in the literature24,25. Those heterogeneities could be associated with intrinsic differences in their roles with respect to the children. It would be plausible to admit that parents and teachers participate in the children’s lives and routines in a complementary, but far from coincident way. Students spend predetermined limited time at schools and teachers must follow structured educational programs, which contrasts with a less formally regulated home environment15. Hence, it could be predicted that divergent expectations regarding the children’s behavior and particular strategies to deal with potential problems in each setting would be identified26. Convergingly, studies have suggested that rating agreement was greater when informants had equivalent relationships with the child. For instance, scores were more correlated among parents than across parents and teachers, suggesting that the nature of the liaison and the setting may influence the awareness of the symptoms27.
Other factors that might contribute for the disparity between responses ought to be discussed. Pervasive child difficulties and behavioral problems related to ADHD may result in considerable parental distress. Reports of feelings of irritation, isolation and exhaustion are common among those individuals, as well as experiencing marital conflicts, career setbacks and neglecting other children due to recurrent demands from the affected child28. In contrast, chronic occupational stress caused by students’ misbehavior or low motivation tend to elicit a cynical attitude towards particular students or students in general on teachers29. Those results imply that dealing with ADHD children might evoke paradoxical effects on parents and teachers: whereas the former may respond with frustration and a trend for excessive complaining, the other group may react with less distress. Finally, parents often acknowledge that children are more attentive to activities related to leisure than to school tasks, which usually endorses their impressions of students’ irresponsible attitude towards school and raises doubts about the ADHD diagnosis. Diversely, teachers’ observations are restricted to a more structured classroom situation, and responses to ADHD scales were probably made simply through comparisons across peers of the same age and gender30.
The present study has its strengths, as it analyzed the reliability of SNAP IV across parents and teachers of ADHD subjects, for which the literature only provided agreement studies. However, some limitations should also be highlighted. Firstly, parental information may not be homogeneous between mothers and fathers. Mothers consistently report their children as having more problems of inattention and hyperactivity than fathers, therefore, the choice of the parental informant may impact considerably on the estimates of ADHD symptoms31. In addition, our sample has been drawn from one clinic, so data should be interpreted carefully when extrapolating to other situations. Finally, the presence of mood and stress-related disorders among collateral responders were not analyzed. Considering that informant-based scales could be highly influenced by the participant’s mental health, validity of our findings could be undercut by this limitation.
Diagnostic guidelines for ADHD in children, such as the DSM-5, strongly recommended that mental health practitioners collected reports from different informants, for the investigation of potential functional impairments at home and at school. For this purpose, formal validated instruments have been widely employed in both clinical and research practices. Our findings indicated that, for the SNAP IV, not only agreement, as previously demonstrated in the literature, but also reliability may be low regarding parents and teachers’ impressions about the presence and severity of ADHD among children. A trend for overly perceiving the children’s symptoms among parents and a much less emphatic report from teachers might be expected. With those results, we suggest that evaluators consider those traits of parental and teachers’ responses when interpreting results from informant-based instruments for detecting ADHD.
Our findings should be interpreted in light of some limitations. Our sample was collected from a private outpatient clinic where clientele belongs to middle, upper-middle and upper classes. For this reason, findings cannot be generalized to other settings. On the other side, our sample mirrors what is seen in most private settings in the country. In addition, referred clinical samples may present different clinical profiles than non-clinical epidemiological ones.
The authors declare that there is no conflict of interests regarding the publication of this article.
1. American Psychiatric Association. Diagnostic and Statistical Manual of Mental Disorders [Internet]. American Psychiatric Association; 2013. Available from: https://psychiatryonline.org/doi/book/10.1176/appi.books.9780890425596
2. Davidovitch M, Koren G, Fund N, Shrem M, Porath A. Challenges in defining the rates of ADHD diagnosis and treatment: trends over the last decade. BMC Pediatr. 2017;17(1):218.
3. Thomas R, Sanders S, Doust J, Beller E, Glasziou P. Prevalence of attention-deficit/hyperactivity disorder: a systematic review and meta-analysis. Pediatrics. 2015;135(4):e994-1001.
4. Biederman J, Petty CR, Evans M, Small J, Faraone SV. How persistent is ADHD? A controlled 10-year follow-up study of boys with ADHD. Psychiatry Res. 2010;177(3):299-304.
5. Gualtieri CT, Johnson LG. ADHD: Is Objective Diagnosis Possible? Psychiatry (Edgmont). 2005;2(11):44-53.
6. Coelho C, Bastos CL. Avaliando o TDAH. Psychiatry On Line Brazil. 2011;16(12).
7. Matte B, Rohde LA, Turner JB, Fisher PW, Shen S, Bau C, et al. Reliability and Validity of Proposed DSM-5 ADHD Symptoms in a Clinical Sample of Adults. J Neuropsychiatry Clin Neurosci. 2015;27(3):228-36.
8. Kooij JJS, Bijlenga D, Salerno L, Jaeschke R, Bitter I, Balázs J, et al. Updated European Consensus Statement on diagnosis and treatment of adult ADHD. Eur Psychiatry. 2019;56:14-34.
9. NICE guideline. Attention deficit hyperactivity disorder: diagnosis and management [Internet]. 2018.
10. Dvorsky MR, Langberg JM, Molitor SJ, Bourchtein E. Clinical Utility and Predictive Validity of Parent and College Student Symptom Ratings in Predicting an ADHD Diagnosis. J Clin Psychol. 2016;72(4):401-18.
11. Coutinho G, Mattos P, Schmitz M, Fortes D, Borges M. Concordância entre relato de pais e professores para sintomas de TDAH: resultados de uma amostra clínica brasileira. Arch Clin Psychiatry (São Paulo). 2009;36(3):97-100.
12. Mattos P, Serra-Pinheiro MA, Rohde LA, Pinto D. Apresentação de uma versão em português para uso no Brasil do instrumento MTA-SNAP-IV de avaliação de sintomas de transtorno do déficit de atenção/hiperatividade e sintomas de transtorno desafiador e de oposição. Rev Psiquiatr Rio Gd Sul. 2006;28(3):290-7.
13. Wagner DJ, McLennan JD. An Alternative Approach to Scoring the MTA-SNAP-IV to Guide Attention-Deficit/Hyperactivity Disorder Medication Treatment Titration towards Symptom Remission: A Preliminary Consideration. J Child Adolesc Psychopharmacol. 2015;25(10):749-53.
14. Martoni AT, Trevisan BT, Dias NM, Seabra AG. Funções executivas: relação entre relatos de pais, de professores e desempenho de crianças. Temas Psicol. 2016;24(1):173-88.
15. Narad ME, Garner AA, Peugh JL, Tamm L, Antonini TN, Kingery KM, et al. Parent-teacher agreement on ADHD symptoms across development. Psychol Assess. 2015;27(1):239-48.
16. dosReis S, Myers MA. Parental attitudes and involvement in psychopharmacological treatment for ADHD: a conceptual model. Int Rev Psychiatry. 2008;20(2):135-41.
17. Shweta, Bajpai RC, Chaturvedi HK. Evaluation of inter-rater agreement and inter-rater reliability for observational data: An overview of concepts and methods. J Indian Acad Appl Psychol. 2015;41(Special Issue 3):20-7.
18. Wechsler D. Escala Wechsler de Inteligência para Crianças (WISCIV): manual de instruções para aplicação e avaliação. São Paulo: Casa do Psicólogo; 2013.
19. Matos DAS. Confiabilidade e concordância entre juízes: aplicações na área educacional. Estud Aval Educ. 2014;25(59):298.
20. Koo TK, Li MY. A Guideline of Selecting and Reporting Intraclass Correlation Coefficients for Reliability Research. J Chiropr Med. 2016;15(2):155-63.
21. Tavakol M, Dennick R. Making sense of Cronbach’s alpha. Int J Med Educ. 2011;2:53-5.
22. Stevens J, Quittner AL. Factors influencing elementary school teachers’ ratings of ADHD and ODD behaviors. J Clin Child Psychol. 1998;27(4):406-14.
23. Correia Filho AG, Bodanese R, Silva TL, Alvares JP, Aman M, Rohde LA. Comparison of Risperidone and Methylphenidate for Reducing ADHD Symptoms in Children and Adolescents With Moderate Mental Retardation. J Am Acad Child Adolesc Psychiatry. 2005;44(8):748-55.
24. Sellers R, Maughan B, Pickles A, Thapar A, Collishaw S. Trends in parent- and teacher-rated emotional, conduct and ADHD problems and their impact in prepubertal children in Great Britain: 1999-2008. J Child Psychol Psychiatry. 2015;56(1):49-57.
25. Segal-Triwitz Y, Kirchen LM, Shani Sherman T, Levav M, Schonherz-Pine Y, Kushnir J, et al. Parents’ and Teachers’ Perceptions of Abnormal Attention Span of Elementary School-Age Children. Isr J Psychiatry Relat Sci. 2016;53(2):33-8.
26. Andrade AC, Flores-Mendoza C. Transtorno do Déficit de Atenção/Hiperatividade: o que nos informa a investigação dimensional? Estud Psicol. 2010;15(1):17-24.
27. Efstratopoulou M, Simons J, Janssen R. Concordance Among Physical Educators’, Teachers’, and Parents’ Perceptions of Attention Problems in Children. J Atten Disord. 2013;17(5):437-43.
28. Corcoran J, Schildt B, Hochbrueckner R, Abell J. Parents of Children with Attention Deficit/Hyperactivity Disorder: A Meta-Synthesis, Part I. Child Adolesc Soc Work J. 2017;34(4):281-335.
29. Skaalvik EM, Skaalvik S. Dimensions of teacher burnout: relations with potential stressors at school. Soc Psychol Educ. 2017;20(4):775-90.
30. Johnston C, Weiss MD, Murray C, Miller NV. The Effects of Instructions on Mothers’ Ratings of Attention-Deficit/Hyperactivity Disorder Symptoms in Referred Children. J Abnorm Child Psychol. 2014;42(3):479-88.
31. Caye A, Machado JD, Rohde LA. Evaluating Parental Disagreement in ADHD Diagnosis: Can We Rely on a Single Report From Home? J Atten Disord. 2017;21(7):561-6.