Chapter 3 Test Fairness

3.1 Introduction

The Smarter Balanced Assessment Consortium (Smarter Balanced) has designed the assessment system to provide all eligible students with a fair test and equitable opportunity to participate in the assessment. Ensuring test fairness is a fundamental part of validity, starting with test design. It is an important feature built into each step of the test development process, such as item writing, test administration, and scoring. The 2014 Standards for Educational and Psychological Testing (AERA, APA, & NCME, 2014, p. 49) state, “The term fairness has no single technical meaning, and is used in many ways in public discourse.” It also suggests that fairness to all individuals in the intended population is an overriding and fundamental validity concern. As indicated in the Standards (p. 63), “The central idea of fairness in testing is to identify and remove construct-irrelevant barriers to maximal performance for any examinee.”

The Smarter Balanced system is designed to provide a valid, reliable, and fair measure of student achievement based on the Common Core State Standards1 (CCSS). The validity and fairness of student achievement measures are influenced by a multitude of factors; central among them are:

  • a clear definition of the construct—the knowledge, skills, and abilities—intended to be measured;
  • the development of items and tasks that are explicitly designed to assess the construct that is the target of measurement;
  • delivery of items and tasks that enable students to demonstrate their achievement on the construct; and
  • the capturing and scoring of responses to those items and tasks.

Smarter Balanced uses several processes to address reliability, validity, and fairness. The fairness construct is defined in the CCSS. The CCSS are a set of high-quality academic standards in mathematics and English language arts/literacy (ELA/literacy) that outline what a student should know and be able to do at the end of each grade. The standards were created to ensure that all students graduate from high school with the skills and knowledge necessary for post-secondary success. The CCSS were developed during a state-led effort launched in 2009 by state leaders. These leaders included governors and state commissioners of education from 48 states, two territories, and the District of Columbia, through their membership in the National Governors Association Center for Best Practices (NGA Center) and the Council of Chief State School Officers (CCSSO).

The CCSS have been adopted by all members of the Smarter Balanced Consortium. The Smarter Balanced content specifications (Smarter Balanced, 2017a,b) define the knowledge, skills, and abilities to be assessed and their relationship to the CCSS. In doing so, these documents describe the major constructs—identified as “claims”—within ELA/literacy and mathematics for which evidence of student achievement is gathered and that form the basis for reporting student performance.

Each claim in the Smarter Balanced content specifications is accompanied by a set of assessment targets that provide more detail about the range of content and depth of knowledge levels. The targets serve as the building blocks of test blueprints. Much of the evidence presented in this chapter pertains to fairness to students during the testing process and to design elements and procedures that serve to minimize measurement bias (i.e., Differential Item Functioning, or DIF). Fairness in item and test design processes and the design of accessibility supports (i.e., universal tools, designated supports, and accommodations) in content development are also addressed.

3.2 Definitions for Validity, Bias, Sensitivity, and Fairness

Some key concepts for the ensuing discussion concern validity, bias, sensitivity, and fairness and are described as follows.

3.2.1 Validity

Validity is the extent to which the inferences and actions based on test scores are appropriate and backed by evidence (Messick, 1989). It constitutes the central notion underlying the development, administration, and scoring of a test, as well as the uses and interpretations of test scores. Validation is the process of accumulating evidence to support each proposed score interpretation or use. Evidence in support of validity is extensively discussed in Chapter 1.

3.2.2 Bias

According to the Standards for Educational and Psychological Testing (AERA, APA, & NCME, 2014), bias is “construct underrepresentation or construct-irrelevant components of tests scores that differentially affect the performance of different groups of test takers and consequently affect the reliability/precision and validity of interpretations and uses of test scores” (p. 216).

3.2.3 Sensitivity

“Sensitivity” refers to an awareness of the need to avoid explicit bias in assessment. In common usage, reviews of tests for bias and sensitivity help ensure that test items and stimuli are fair for various groups of test takers (AERA, APA, & NCME, 2014, p. 64).

3.2.4 Fairness

The goal of fairness in assessment is to assure that test materials are as free as possible from unnecessary barriers to the success of diverse groups of students. Smarter Balanced developed the Bias and Sensitivity Guidelines (Smarter Balanced, 2012c) to help ensure that the assessments are fair for all groups of test takers, despite differences in characteristics that include, but are not limited to, disability status, ethnic group, gender, regional background, native language, race, religion, sexual orientation, and socioeconomic status. Unnecessary barriers can be reduced by:

  • measuring only knowledge or skills that are relevant to the intended construct;
  • not angering, offending, upsetting, or otherwise distracting test takers; and
  • treating all groups of people with appropriate respect in test materials.

These rules help ensure that the test content is fair for test takers and acceptable to the many stakeholders and constituent groups within Smarter Balanced member organizations. The more typical view is that bias and sensitivity guidelines apply primarily to the review of test items. However, fairness must be considered in all phases of test development and use.

3.3 Bias and Sensitivity Guidelines

Smarter Balanced strongly relied on the Bias and Sensitivity Guidelines in the development of the Smarter Balanced assessments, particularly in item writing and review. Items must comply with these guidelines in order to be included in the Smarter Balanced assessments. Use of the guidelines will help the Smarter Balanced assessments comply with Chapter 3, Standard 3.2 of the Standards for Educational and Psychological Testing. Standard 3.2 states that “test developers are responsible for developing tests that measure the intended construct and for minimizing the potential for tests being affected by construct-irrelevant characteristics such as linguistic, communicative, cognitive, cultural, physical or other characteristics” (AERA, APA, & NCME, 2014, p. 64).

Smarter Balanced assessments were developed using the principles of evidence-centered design (ECD). ECD requires a chain of evidence-based reasoning that links test performance to the claims made about test takers. Fair assessments are essential to the implementation of ECD. If test items are not fair, then the evidence they provide means different things for different groups of students. Under those circumstances, the claims cannot be equally supported for all test takers, which is a threat to validity. As part of the validation process, all items are reviewed for bias and sensitivity using the Bias and Sensitivity Guidelines prior to being presented to students. This helps ensure that item responses reflect only knowledge of the intended content domain, are free of offensive or distracting material, and portray all groups in a respectful manner. When the guidelines are followed, item responses provide evidence that supports assessment claims.

3.3.1 Item Development

Smarter Balanced has established item development practices that maximize access for all students, including English Learners (ELs), students with disabilities, and ELs with disabilities, but not limited to those groups. Three resources—the Smarter Balanced Item and Task Specifications (2015c), the General Accessibility Guidelines (Smarter Balanced, 2012a), and the Bias and Sensitivity Guidelines (Smarter Balanced, 2012c)—are used to guide the development of items and tasks to ensure that they accurately measure the targeted constructs. Recognizing the diverse characteristics and needs of students who participate in the Smarter Balanced assessments, the states worked together through the Smarter Balanced Test Administration and Student Access Work Group to incorporate research and practical lessons learned through universal design, accessibility tools, and accommodations (Thompson, Johnstone, & Thurlow, 2002).

A fundamental goal is to design an assessment that is accessible for all students, regardless of English language proficiency, disability, or other individual circumstances. The intent is to ensure that the following steps were achieved for Smarter Balanced.

  • Design and develop items and tasks to ensure that all students have access to the items and tasks. In addition, deliver items, tasks, and the collection of student responses in a way that maximizes validity for each student.
  • Adopt the conceptual model embodied in the Accessibility and Accommodations Framework (Smarter Balanced, 2016c) that describes accessibility resources of digitally delivered items/tasks and acknowledges the need for some adult-monitored accommodations. The model also characterizes accessibility resources as a continuum ranging from those available to all students, those that are implemented under adult supervision only, and those for students with a documented need.
  • Implement the use of an individualized and systematic needs profile for students, or Individual Student Assessment Accessibility Profile (ISAAP), that promotes the provision of appropriate access and tools for each student. Smarter Balanced created an ISAAP process that helps education teams systematically select the most appropriate accessibility resources for each student and the ISAAP tool, which helps teams note the accessibility resources chosen.

Prior to any item development and item review, Smarter Balanced staff trains item writers and reviewers on the General Accessibility Guidelines (Smarter Balanced, 2012a) and Bias and Sensitivity Guidelines (Smarter Balanced, 2012c). As part of item review, individuals with expertise in accessibility, bias, and sensitivity review each item and compare it against a checklist for accessibility and bias and sensitivity. Items must pass each criterion on both checklists to be eligible for field testing. By relying on universal design to develop the items and requiring that individuals with expertise in bias, sensitivity, and accessibility review the items throughout the iterative process of development, Smarter Balanced ensures that the items are appropriate for a wide range of students.

3.3.2 Guidelines for General Accessibility

In addition to implementing the principles of universal design during item development, Smarter Balanced meets the needs of English Learners (ELs) by addressing language aspects during development, as described in the Guidelines for Accessibility for English Language Learners (Smarter Balanced Assessment Consortium, 2012b). ELs have not yet acquired proficiency in English. The use of language that is not fully accessible can be regarded as a source of invalidity that affects the resulting test score interpretations by introducing construct-irrelevant variance. Although there are many validity issues related to the assessment of ELs, the main threat to validity when assessing content knowledge stems from language factors that are not relevant to the construct of interest. The goal of these EL guidelines was to minimize factors that are thought to contribute to such construct-irrelevant variance. Adherence to these guidelines helped ensure that, to the greatest extent possible, the Smarter Balanced assessments administered to ELs measure the intended targets. The EL guidelines were intended primarily to inform Smarter Balanced assessment developers or other educational practitioners, including content specialists and testing coordinators.

In educational assessments, there is an important distinction between content-related language that is the target of instruction versus language that is not content related. For example, the use of words with specific technical meaning, such as “slope” when used in algebra or “population” when used in biology, should be used to assess content knowledge for all students. In contrast, greater caution should be exercised when including words that are not directly related to the domain. ELs may have had cultural and social experiences that differ from those of other students. Caution should be exercised in assuming that ELs have the same degree of familiarity with concepts or objects occurring in situational contexts. The recommendation was to use contexts or objects based on classroom or school experiences, rather than ones that are based outside of school. For example, in constructing mathematics items, it is preferable to use common school objects, such as books and pencils, rather than objects in the home, such as kitchen appliances, to reduce the potential for construct-irrelevant variance associated with a test item. When the construct of interest includes a language component, the decisions regarding the proper use of language becomes more nuanced. If the construct assessed is the ability to explain a mathematical concept, then the decisions depend on how the construct is defined. If the construct includes the use of specific language skills, such as the ability to explain a concept in an innovative context, then it is appropriate to assess these skills. In ELA/literacy, there is greater uncertainty as to item development approaches that faithfully reflect the construct while avoiding language inaccessible for ELs.

The decisions of what best constitutes an item can rely on the content standards, the definition of the construct, and the interpretation of the claims and assessment targets. For example, if the skill to be assessed involves interpreting meanings in a literary text, then the use of original source materials is acceptable. However, the test item itself—as distinct from the passage or stimulus—should be written so that the task presented to a student is clearly defined using accessible language. Since ELs taking Smarter Balanced content assessments likely have a range of English proficiency skills, it is also important to consider the accessibility needs across the entire spectrum of proficiency. Since ELs, by definition, have not attained complete proficiency in English, the major consideration in developing items is ensuring that the language used is as accessible as possible. The use of accessible language does not guarantee that construct-irrelevant variance will be eliminated, but it is the best strategy for helping ensure valid scores for ELs and for other students as well.

Using clear and accessible language is a key strategy that minimizes construct-irrelevant variance in items. Language that is part of the construct being measured should not be simplified. For non-content-specific text, the language of presentation should be as clear and simple as possible. The following guidelines for the use of accessible language were proposed as guidance in the development of test items. This guidance intended to work in concert with other principles of good item construction. From the EL Guidelines (Smarter Balanced Assessment Consortium, 2012b), some general principles for the use of accessible language were proposed as follows.

  • Design test directions to maximize clarity and minimize the potential for confusion.

  • Use vocabulary widely accessible to all students, and avoid unfamiliar vocabulary not directly related to the construct (August, Carlo, Dressler, & Snow, 2005; Bailey, Huang, Shin, Farnsworth, & Butler, 2007).

  • Avoid the use of syntax or vocabulary that is above the test’s target grade level (Borgioli, 2008). The test item should be written at a vocabulary level no higher than the target grade level, and preferably at a slightly lower grade level, to ensure that all students understand the task presented (Young, 2008).

  • Keep sentence structures as simple as possible while expressing the intended meaning. In general, ELs find a series of simpler, shorter sentences to be more accessible than longer, more complex sentences (Pitoniak, Young, Martiniello, King, Buteux, & Ginsburgh, 2009).

  • Consider the impact of cognates (words with a common etymological origin) and false cognates (word pairs or phrases that appear to have the same meaning in two or more languages, but do not) when developing items. Spanish and English share many cognates, and because the large majority of ELs speak Spanish as their first language (nationally, more than 75%), the presence of cognates can inadvertently confuse students and alter the skills being assessed by an item. Examples of false cognates include: billion (the correct Spanish word is millones; not billón, which means trillion); deception (engaño; not decepción, which means disappointment); large (grande; not largo, which means long); library (biblioteca; not librería, which means bookstore).

  • Do not use cultural references or idiomatic expressions (such as “being on the ball”) that are not equally familiar to all students (Bernhardt, 2005).

  • Avoid sentence structures that may be confusing or difficult to follow, such as the use of passive voice or sentences with multiple clauses (Abedi & Lord, 2001; Forster & Olbrei, 1973; Schachter, 1983).

  • Do not use syntax that may be confusing or ambiguous, such as using negation or double negatives in constructing test items (Abedi, 2006; Cummins, Kintsch, Reusser, & Weimer, 1988).

  • Minimize the use of low-frequency, long, or morphologically complex words and long sentences (Abedi, 2006; Abedi, Lord, & Plummer, 1995).

  • Teachers can use multiple semiotic representations to convey meaning to students in their classrooms. Assessment developers should also consider ways to create questions using multi-semiotic methods so that students can better understand what is being asked (Kopriva, 2010). This might include greater use of graphical, schematic, or other visual representations to supplement information provided in written form.

3.4 Test Delivery

In addition to focusing on accessibility, bias, and sensitivity during item development, Smarter Balanced also maximizes accessibility through test delivery. Smarter Balanced works with members to maintain the original conceptual framework (Smarter Balanced Assessment Consortium, 2016c) that continues to serve as the basis underlying the usability, accessibility, and accommodations (Figure 3.1). This figure portrays several aspects of the Smarter Balanced assessment resources—universal tools (available for all students), designated supports (available when indicated by an adult or team), and accommodations (as documented in an Individualized Education Program (IEP) or 504 plan). It also displays the additive and sequentially inclusive nature of these three aspects.

  • Universal tools are available to all students, including those receiving designated supports and those receiving accommodations.

  • Designated supports are available only to students who have been identified as needing these resources (as well as those students for whom the need is documented as described in the following bullet).

  • Accommodations are available only to those students with documentation of the need through a formal plan (e.g., IEP, 504). Those students also may access designated supports and universal tools.

A universal tool or a designated support may also be an accommodation, depending on the content or grade. This approach is consistent with the emphasis that Smarter Balanced has placed on the validity of assessment results coupled with access. Universal tools, designated supports, and accommodations are all intended to yield valid scores. Use of universal tools, designated supports, and accommodations result in scores that count toward participation in statewide assessments. Also shown in Figure 3.1 are the universal tools, designated supports, and accommodations for each category of accessibility resources. There are both embedded and non-embedded versions of the universal tools, designated supports, or accommodations, depending on whether they are provided as digitally delivered components of the test administration or provided locally separate from the test delivery system.

Conceptual Model Underlying the Smarter Balanced Usability, Accessibility, and Accommodations Guidelines

Figure 3.1: Conceptual Model Underlying the Smarter Balanced Usability, Accessibility, and Accommodations Guidelines

3.5 Meeting the Needs of Traditionally Underrepresented Populations

Members decided to make accessibility resources available to all students based on need rather than eligibility status or other designation. This reflects a belief among Consortium states that unnecessarily restricting access to accessibility resources threatens the validity of the assessment results and places students under undue stress and frustration. Additionally, accommodations are available for students who qualify for them. The Consortium utilizes a needs-based approach to providing accessibility resources. A description as to how this benefits ELs, students with disabilities, and ELs with disabilities is presented here.

3.5.1 Students Who Are ELs

Students who are ELs have needs that are unique from students with disabilities, including language-related disabilities. The needs of ELs are not the result of a language-related disability, but instead are specific to the student’s current level of English language proficiency. The needs of students who are ELs are diverse and are influenced by the interaction of several factors, including their current level of English language proficiency, their prior exposure to academic content and language in their native language, the languages to which they are exposed outside of school, the length of time they have participated in the U.S. education system, and the language(s) in which academic content is presented in the classroom. Given the unique background and needs of each student, the conceptual framework is designed to focus on students as individuals and to provide several accessibility resources that can be combined in a variety of ways. Some of these digital tools, such as using a highlighter to highlight key information, are available to all students, including ELs. Other tools, such as the audio presentation of items or glossary definitions in English, may also be assigned to any student, including ELs. Still, other tools, such as embedded glossaries that present translations of construct-irrelevant terms, are intended for those students whose prior language experiences would allow them to benefit from translations into another spoken language. Collectively, the conceptual framework for usability, accessibility, and accommodations embraces a variety of accessibility resources that have been designed to meet the needs of students at various stages in their English language development.

3.5.2 Students with Disabilities

Federal law requires that students with disabilities who have a documented need receive accommodations that address those needs and that they participate in assessments. The intent of the law is to ensure that all students have appropriate access to instructional materials and are held to the same high standards. When students are assessed, the law ensures that students receive appropriate accommodations during testing so they can demonstrate what they know and can do, so that their achievement is measured accurately.

The Accessibility and Accommodations Framework (Smarter Balanced, 2016c) addresses the needs of students with disabilities in three ways. First, it provides for the use of digital test items that are purposefully designed to contain multiple forms of the item, each developed to address a specific access need. By allowing the delivery of a given access form of an item to be tailored based on each student’s access need, the Framework fulfills the intent of federal accommodation legislation. Embedding universal accessibility digital tools, however, addresses only a portion of the access needs required by many students with disabilities. Second, by embedding accessibility resources in the digital test delivery system, additional access needs are met. This approach fulfills the intent of the law for many, but not all, students with disabilities by allowing the accessibility resources to be activated for students based on their needs. Third, by allowing for a wide variety of digital and locally provided accommodations (including physical arrangements), the Framework addresses a spectrum of accessibility resources appropriate for math and ELA/literacy assessment. Collectively, the Framework adheres to federal regulations by allowing a combination of universal design principles, universal tools, designated supports, and accommodations to be embedded in a digital delivery system and through local administration assigned and provided based on individual student needs. Therefore, a student who is both an EL and a student with a disability benefits from the system, because they may have access to resources from any of the three categories as necessary to create an assessment tailored to their individual need.

3.6 The Individual Student Assessment Accessibility Profile (ISAAP)

Typical practice frequently required schools and educators to document, a priori, the need for specific student accommodations and document the use of those accommodations after the assessment. For example, most programs require schools to document a student’s need for a large-print version of a test for delivery to the school. Following the test administration, the school documented (often by bubbling in information on an answer sheet) which of the accommodations, if any, a given student received; whether the student actually used the large-print form; and whether any other accommodations, such as extended time, were provided. Traditionally, many programs have focused only on students who have received accommodations and thus may consider an accommodation report as documenting accessibility needs. The documentation of need and use establishes a student’s accessibility needs for assessment.

For most students, universal digital tools will be available by default in the Smarter Balanced test delivery system and need not be documented. These tools can be deactivated if they create an unnecessary distraction for the student. Other embedded accessibility resources that are available for any student needing them must be documented prior to assessment. The Smarter Balanced assessment system has established an Individual Student Assessment Accessibility Profile (ISAAP) to capture specific student accessibility needs. The ISAAP tool is designed to facilitate selection of the universal tools, designated supports, and accommodations that match student access needs for the Smarter Balanced assessments, as supported by the Usability, Accessibility, and Accommodations Guidelines (Smarter Balanced Assessment Consortium, 2017c). The ISAAP tool2 should be used in conjunction with the Usability, Accessibility, and Accommodations Guidelines and state regulations and policies related to assessment accessibility as a part of the ISAAP process. For students requiring one or more accessibility resources, schools will be able to document this need prior to test administration. Furthermore, the ISAAP can include information about universal tools that may need to be eliminated for a given student. By documenting need prior to test administration, a digital delivery system will be able to activate the specified options when the student logs in to an assessment. In this way, the profile permits school-level personnel to focus on each individual student, documenting the accessibility resources required for valid assessment of that student in a way that is efficient to manage.

The conceptual framework shown in Figure 3.1 provides a structure that assists in identifying which accessibility resources should be made available for each student. In addition, the conceptual framework is designed to differentiate between universal tools available to all students and accessibility resources that must be assigned before the administration of the assessment. Consistent with recommendations from Shafer and Rivera (2011); Thurlow, Quenemoen, and Lazarus (2011); Fedorchak (2012); and Russell (2011), Smarter Balanced is encouraging school-level personnel to use a team approach to make decisions concerning each student’s ISAAP. Gaining input from individuals with multiple perspectives, including the student, will likely result in appropriate decisions about the assignment of accessibility resources. Consistent with these recommendations, one should avoid selecting too many accessibility resources for a student. The use of too many unneeded accessibility resources can decrease student performance.

The team approach encouraged by Smarter Balanced does not require the formation of a new decision-making team. The structure of teams can vary widely depending on the background and needs of a student. A locally convened student support team can potentially create the ISAAP. For most students who do not require accessibility tools or accommodations, a teacher’s initial decision may be confirmed by a second person (potentially the student). In contrast, for a student who is an English language learner and has been identified with one or more disabilities, the IEP team should include the English language development specialist who works with the student, along with other required IEP team members and the student, as appropriate. The composition of teams is not being defined by Smarter Balanced; it is under the control of each school and is subject to state and federal requirements.

3.7 Usability, Accessibility, and Accommodations Guidelines

Smarter Balanced developed the Usability, Accessibility, and Accommodations Guidelines (UAAG) for its members to guide the selection and administration of universal tools, designated supports, and accommodations. All ICAs (Interim Comprehensive Assessments) and IABs (interim assessment blocks) are fully accessible and offer all accessibility resources as appropriate by grade and content area, including ASL, braille, and Spanish. It is intended for school-level personnel and decision-making teams, particularly Individualized Education Program (IEP) teams, as they prepare for and implement the Smarter Balanced summative and interim assessments. The UAAG provides information for classroom teachers, English development educators, special education teachers, and related services personnel in selecting and administering universal tools, designated supports, and accommodations for those students who need them. The UAAG is also intended for assessment staff and administrators who oversee the decisions that are made in instruction and assessment. It emphasizes an individualized approach to the implementation of assessment practices for those students who have diverse needs and participate in large-scale assessments. This document focuses on universal tools, designated supports, and accommodations for the Smarter Balanced summative and interim assessments in ELA/literacy and mathematics. At the same time, it supports important instructional decisions about accessibility for students. It recognizes the critical connection between accessibility in instruction and accessibility during assessment. The UAAG is also incorporated into the Smarter Balanced Test Administration Manual (Smarter Balanced, 2017i).

According to the UAAG (Smarter Balanced, 2014, p. 2), all eligible students (including students with disabilities, ELs, and ELs with disabilities) should participate in the assessments. In addition, the performance of all students who take the assessment is measured with the same criteria. Specifically, all students enrolled in grades 3 to 8 and high school are required to participate in the Smarter Balanced mathematics assessment, except students with the most significant cognitive disabilities who meet the criteria for the mathematics alternate assessment based on alternate achievement standards (approximately 1% or less of the student population).

All students enrolled in grades 3 to 8 and high school are required to participate in the Smarter Balanced English language/literacy assessment except:

  • students with the most significant cognitive disabilities who meet the criteria for the English language/literacy alternate assessment based on alternate achievement standards (approximately 1% or fewer of the student population), and
  • ELs who are enrolled for the first year in a U.S. school. These students will participate in their state’s English language proficiency assessment.

Federal laws governing student participation in statewide assessments include the Elementary and Secondary Education Act - ESEA (reauthorized as the No Child Left Behind Act - NCLB of 2001), the Individuals with Disabilities Education Improvement Act of 2004 - IDEA, and Section 504 of the Rehabilitation Act of 1973 (reauthorized in 2008).

Since the Smarter Balanced assessment is based on the CCSS, universal tools, designated supports, and accommodations, the Smarter Balanced assessment may be different from those that state programs utilized previously. For the summative assessments, state participants can only make available to students the universal tools, designated supports, and accommodations consistent with the Smarter Balanced UAAG. According to the UAAG (Smarter Balanced, 2014, p. 1), when the implementation or use of the universal tool, designated support, or accommodation is in conflict with a member state’s law, regulation, or policy, a state may elect not to make it available to students.

The Smarter Balanced universal tools, designated supports, and accommodations currently available for the Smarter Balanced assessments have been prescribed. The specific universal tools, designated supports, and accommodations approved by Smarter Balanced may undergo change if additional tools, supports, or accommodations are identified for the assessment based on state experience or research findings. The Consortium has established a standing committee, including members from the Consortium and staff, that reviews suggested additional universal tools, designated supports, and accommodations to determine if changes are warranted. Proposed changes to the list of universal tools, designated supports, and accommodations are brought to Consortium members for review, input, and vote for approval. Furthermore, states may issue temporary approvals (i.e., one summative assessment administration) for individual, unique student accommodations. It is expected that states will evaluate formal requests for unique accommodations and determine whether the request poses a threat to the measurement of the construct. Upon issuing temporary approval, the petitioning state can send documentation of the approval to the Consortium. The Consortium will consider all state-approved temporary accommodations as part of the annual Consortium accommodations review process. The Consortium will provide to member states a list of the temporary accommodations issued by states that are not Consortium-approved accommodations.

3.8 Provision of Specialized Tests or Pools

Smarter Balanced provides a full item pool and a series of specialized items pools that allow students who are eligible to access the tests with a minimum of barriers. These accessibility resources are considered embedded accommodations or embedded designated supports. The specialized pools that were available in 2018-19 are shown in Table 3.1.

Subject Test Instrument
ELA/literacy ASL adaptive online (Listening only)
Closed captioning adaptive online (Listening only)
Braille adaptive online
Braille paper pencil
Math Translated glossaries adaptive online
Stacked Spanish adaptive online
ASL adaptive online
Braille adaptive online
Braille hybrid adaptive test (HAT)
Spanish paper pencil
Braille paper pencil
Translated glossaries paper pencil

Table 3.2 and Table 3.3 show, for each subject, the number of online items in the general and accommodated pools by test segment (CAT and PT) and grade. Items in fixed forms, both online and paper and pencil, are not included in the counts shown in these tables.

Grade Online CAT General Online CAT ASL Online CAT Braille Online PT General Online PT ASL Online PT Braille
3 867 58 260 38 NA 10
4 823 61 241 44 NA 10
5 787 47 257 50 NA 12
6 811 56 251 38 NA 10
7 735 51 209 48 NA 16
8 815 54 259 50 NA 16
11 2,612 121 466 56 NA 16
Grade Online CAT General Online CAT ASL Online CAT Braille Online CAT Glossaries Online CAT Spanish Online PT General Online PT ASL Online PT Braille Online PT Glossaries Online PT Spanish
3 1,234 333 370 229 388 95 35 35 30 50
4 1,325 336 317 226 384 116 41 32 29 43
5 1,268 367 332 232 389 105 41 41 28 51
6 1,147 343 347 246 378 92 23 23 28 39
7 1,047 317 323 233 335 92 20 20 17 25
8 915 298 257 208 301 79 25 19 39 34
11 2,610 660 492 370 672 70 17 17 16 28

Table 3.4 and Table 3.5 show the total score reliability and standard error of measurement (SEM) of the tests taken by students requiring the accommodated pools of items. The statistics in these tables were derived as described in Chapter 2 for students in the general population.

The measurement precision of accommodated tests is in line with that of the general population, taking into consideration the overall performance of students taking the accommodated tests and the relationship between overall performance level and measurement error. Measurement error tends to be greater at higher and lower deciles of performance, compared to deciles near the median (Table 2.29). To the extent that the average overall scale scores of students taking accommodated tests, shown in Table 3.4 and Table 3.5, fall into higher or lower deciles of performance (see tables in Section 5.4.3), one can expect the corresponding average SEMs in Table 3.4 and Table 3.5 to be larger than those for the general population (mean SEM in Table 2.29). To the extent that average SEM associated with the accommodated tests tend to be larger for this reason, one can also expect reliability coefficients in Table 3.4 and Table 3.5 to be smaller than those for the general population (see total score reliabilities in Table 2.4 and Table 2.5). Any differences in reliability coefficients between general and accommodated populations must also take into account differences in variability of test scores. Even if the groups had the same average scale score and measurement error, the group having a lower standard deviation of scale scores would have a lower reliability coefficient. In some cases, such as grade 3 ASL, the standard deviation of scale scores, 77.1, is much lower than that of the general population grade 3 ELA/literacy test-taking population (91.4 in Table 5.7).

Statistics concerned with test bias are reported for braille and Spanish tests in Chapter 2 and are based on simulation. Reliability and measurement error information for accommodated fixed forms, both paper and pencil and online, are the same for regular fixed forms, as reported in Table 2.10 and Table 2.11, since the accommodated and regular fixed forms use the same items.

Type Grade Count Mean SD Rho Avg.SEM SEM.Q1 SEM.Q2 SEM.Q3 SEM.Q4
Braille 3 15 2430 66.1 0.875 23.3 24.8 22.0 22.7 23.5
4 18 2482 111.9 0.940 27.1 31.0 25.2 24.5 26.8
5 27 2494 92.3 0.921 25.8 28.1 24.3 24.0 26.6
6 23 2452 79.5 0.885 26.8 31.8 25.8 24.6 24.5
7 21 2518 100.9 0.916 28.4 34.5 26.6 25.0 26.4
8 19 2528 80.1 0.886 26.9 29.4 27.0 25.8 25.4
HS 17 2556 122.5 0.918 33.8 43.0 30.0 29.8 30.2
ASL 3 194 2309 77.1 0.838 30.2 39.9 30.7 26.7 23.4
4 215 2321 86.9 0.845 32.9 43.2 31.8 29.4 26.4
5 210 2370 89.1 0.876 30.4 38.8 30.5 26.6 25.1
6 217 2392 93.4 0.880 31.5 39.4 32.7 28.3 25.3
7 265 2405 83.5 0.844 32.5 39.7 33.5 30.3 26.4
8 257 2423 89.3 0.855 33.2 42.5 33.3 30.0 26.8
HS 246 2453 100.9 0.866 36.4 45.0 37.4 33.2 29.8

Type Grade Count Mean SD Rho Avg.SEM SEM.Q1 SEM.Q2 SEM.Q3 SEM.Q4
Braille 3 12 2458 69.7 0.934 17.9 19.3 17.3 17.0 18.0
4 18 2438 122.0 0.940 25.3 41.6 20.8 18.3 17.6
5 25 2473 92.5 0.931 23.6 31.0 24.2 19.0 19.2
6 20 2407 117.5 0.883 36.2 60.6 35.4 28.0 21.0
7 20 2469 108.6 0.909 31.2 45.0 32.0 25.8 22.0
8 16 2431 92.1 0.820 37.9 50.5 40.8 33.2 27.2
HS 9 2461 118.0 0.825 45.4 67.0 38.5 36.0 29.5
ASL 3 188 2330 83.8 0.909 24.2 34.9 24.4 20.0 17.4
4 212 2362 89.3 0.888 27.8 42.3 28.3 21.8 18.4
5 209 2393 82.0 0.839 31.2 44.6 31.7 27.0 21.3
6 210 2383 94.0 0.817 37.5 56.1 39.0 31.5 23.3
7 254 2385 96.4 0.765 43.3 67.4 44.9 34.7 25.9
8 254 2410 98.2 0.805 41.5 56.7 43.4 37.3 28.2
HS 239 2450 94.8 0.733 46.3 67.4 48.5 39.1 29.9
Spanish 3 8,038 2390 83.8 0.939 20.1 26.2 19.5 17.5 17.0
4 7,087 2422 86.2 0.929 22.0 30.4 21.1 18.7 17.9
5 5,956 2441 91.4 0.908 26.4 36.6 27.4 22.8 18.7
6 4,402 2426 98.1 0.878 32.0 47.9 32.5 26.0 21.4
7 4,008 2420 95.0 0.822 37.4 56.1 37.4 31.0 24.6
8 3,650 2428 88.9 0.801 38.2 51.5 39.4 34.6 27.3
HS 3,262 2447 86.6 0.686 45.9 66.0 47.0 39.6 31.0
TransGloss 3 10,918 2400 80.4 0.937 19.7 24.9 19.1 17.5 17.1
4 11,310 2432 81.8 0.929 21.1 27.5 20.2 18.5 18.0
5 10,037 2449 85.0 0.903 25.4 34.1 26.4 22.4 18.7
6 8,861 2449 100.3 0.904 29.2 42.7 29.2 23.9 20.8
7 7,598 2452 106.9 0.883 34.1 50.5 34.2 28.6 22.9
8 7,372 2466 115.3 0.897 35.5 49.1 37.2 31.0 24.5
HS 10,777 2502 122.3 0.879 39.5 59.4 41.2 32.8 24.6

The braille statistics in Table 3.5 include both the fully adaptive online braille test and the Braille hybrid adaptive test (Braille HAT). More specific information about the Braille HAT is available online (Smarter Balanced, 2017h). Documentation concerned with blueprint fulfillment, item exposure, measurement precision, and bias of the Braille HAT is currently under review for public release.

3.9 Differential Item Functioning (DIF)

DIF analyses are used to identify items for which groups of students that are matched on overall achievement, but differ demographically (e.g., males, females), have different probabilities of success on a test item. Information about DIF and the procedures for reviewing items flagged for DIF is a component of validity evidence associated with the internal properties of the test.

3.9.1 Method of Assessing DIF

DIF analyses are performed on items using data gathered in the field test stage. In a DIF analysis, the performance on an item by two groups that are similar in achievement, but differ demographically, are compared. In general, the two groups are called the focal and reference groups. The focal group is usually a minority group (e.g., Hispanics), while the reference group is usually a contrasting majority group (e.g., Caucasian) or all students that are not part of the focal group demographic. The focal and reference groups in Smarter Balanced DIF analyses are identified in Table 3.6.

Group Type Focal Group Reference Group
Gender Female Male
Ethnicity African American White
Asian/Pacific Islander
Native American/Alaska Native
Special Populations Limited English Proficient (LEP) English Proficient
Individualized Education Program (IEP) No IEP
Title 1 (Economically disadvantaged) Not Title 1

A DIF analysis asks, “Do focal group students have the same probability of success on each test item as reference group students of the same overall ability (as indicated by their performance on the full test)?” If the answer is “no,” according to the criteria described below, the item is said to exhibit DIF.

Different DIF analysis procedures and flagging criteria are used depending on the number of points the item is worth. For one-point items (also called dichotomously scored items and scored 0/1), the Mantel-Haenszel statistic (Mantel & Haenszel, 1959) is used. For items worth more than one point (also called partial-credit or polytomously scored items), the Mantel chi-square statistic (Mantel, 1963) and the standardized mean difference (SMD) procedure (Dorans & Kulick, 1983, 1986) are used.

The Mantel-Haenszel statistic is computed as described by Holland and Thayer (1988). The common odds ratio is computed first:

\[\begin{equation} \alpha_{MH}=\frac{(\sum_m\frac{R_rW_f}{N_m})}{(\sum_m\frac{R_fW_r}{N_m})}, \tag{3.1} \end{equation}\]

\(R_r\) = number in reference group at ability level m answering the item right;
\(W_f\) = number in focal group at ability level m answering the item wrong;
\(R_f\) = number in focal group at ability level m answering the item right;
\(W_r\) = number in reference group at ability level m answering the item wrong; and
\(N_m\) = total group at ability level m.

This value is then used to compute MH D-DIF, which is a normalized transformation of item difficulty (p-value) with a mean of 13 and a standard deviation of 4:

\[\begin{equation} -2.35ln[\alpha_{MH}]. \tag{3.2} \end{equation}\]

The standard error used to test MH D-DIF for significance is equal to

\[\begin{equation} 2.35\sqrt{(var[ln(\hat{\theta}_{MH})])}, \tag{3.3} \end{equation}\]

where the variance of the MH common odds ratio, \(var[ln\hat{\theta}_{MH})],\) is given for example in Appendix 1 of Michaelides (2008). For significance testing, the ratio of MH D-DIF and this SE is tested as a deviate of the normal distribution (\(\alpha = .05\)).

The statistical significance of MH D-DIF could alternatively be obtained by computing the chi square statistic:

\[\begin{equation} X^2_{MH} = \frac{(|\sum_m R_r-\sum_m E(R_r)|-\frac{1}{2})^2}{\sum_m Var(R_r)}, \tag{3.4} \end{equation}\]


\[\begin{equation} E(R_r) = \frac{N_r R_N}{N_m,Var(R_r)} = \frac{N_r N_f R_N W_N}{N_m^2 (N_m-1)}, \tag{3.5} \end{equation}\]

\(N_r\) and \(N_f\) are the numbers of examinees in the reference and focal groups at ability level m, respectively, and \(R_N\) and \(W_N\) are the number of examinees who answered the item correctly and incorrectly at ability level m, respectively. Smarter Balanced uses the standard error to test the significance of MH D-DIF.

The Mantel chi-square (Mantel, 1963) is an extension of the MH statistic, which, when applied to the DIF context, presumes the item response categories are ordered and compares means between reference and focal groups. The Mantel statistic is given by:

\[\begin{equation} X^2_{Mantel} = \frac{(\sum_{m=0}^T F_m - \sum_{m=0}^T E(F_m))^2}{\sum_{m=0}^T \sigma_{F_m}^2} \tag{3.6} \end{equation}\]

where \(F_m\) is the sum of item scores for the focal group at the \(m^{th}\) ability level, E() is the expected value (mean), and \(\sigma^2\) is the variance. Michaelides (2008, p. 5) provides additional detail pertaining to the computation of the mean and variance.

The standardized mean difference used for partial-credit items is defined as:

\[\begin{equation} SMD = \sum p_{Fk} m_{Fk} - \sum p_{Fk} m_{Rk}, \tag{3.7} \end{equation}\]

where \(p_{Fk}\) is the proportion of the focal group members who are at the \(k^{th}\) level of the matching variable,\(m_{Fk}\) is the mean item score for the focal group at the \(k^{th}\) level, and \(m_{Rk}\) is the mean item score for the reference group at the \(k^{th}\) level. A negative value of the standardized mean difference shows that the item is more difficult for the focal group, whereas a positive value indicates that it is more difficult for the reference group.

To get the effect size, the SMD is divided by the total item group (reference and focal groups pooled) standard deviation:

\[\begin{equation} SD = \sqrt{\frac{(n_F-1)\sigma_{y_F}^2+(n_R-1)\sigma_{y_R}^2}{n_F + n_R - 2}}, \tag{3.8} \end{equation}\]

where \(n_F\) and \(n_R\) are the counts of focal and reference group members who answered the item, and \(\sigma_{y_F}^2\) and \(\sigma_{y_R}^2\) are the variances of the item responses for the focal and reference groups, respectively.

Items are classified into three categories of DIF: “A,” “B,” or “C” according to the criteria shown in Table 3.7 (for dichotomously scored items) and Table 3.8 (for partial-credit items). Category A items contain negligible DIF. In subsequent tables, category A levels of DIF are not flagged as they are too small to have perceptible interpretation. Category B items exhibit moderate DIF, and category C items have significant values of DIF. Positive values favor the focal group, and negative values favor the reference group. Positive and negative values are reported for B and C levels of DIF. Negative and positive DIF at the B level are denoted, respectively, B- and B+. Likewise for C-level DIF.

DIF Category Definition
A (negligible) |MH D-DIF| < 1
B (slight to moderate) |MH D-DIF| \(\ge\) 1 and |MH D-DIF| < 1.5
Positive values are classified as “B+” and negative values as “B-”
C (moderate to large) |MH D-DIF| \(\ge\) 1.5
Positive values are classified as “C+” and negative values as “C-“

DIF Category Definition
A (negligible) Mantel chi-square p-value > 0.05 or |SMD/SD| \(\le\) 0.17
B (slight to moderate) Mantel chi-square p-value < 0.05 and 0.17 < |SMD/SD| \(\le\) 0.25
C (moderate to large) Mantel chi-square p-value < 0.05 and |SMD/SD| > 0.25

Items flagged for C-level DIF are subsequently reviewed by content experts and bias/sensitivity committees to determine the source and meaning of performance differences. An item flagged for C-level DIF may be measuring something different from the intended construct. However, it is important to recognize that DIF-flagged items might be related to actual differences in relevant knowledge and skills or may have been flagged due to chance variation in the DIF statistic (known as statistical type I error). Final decisions about the resolution of item DIF are made by the multi-disciplinary panel of content experts.

3.9.2 Item DIF in the 2018-19 Summative Assessment Pool

Table 3.9 and Table 3.10 show DIF analysis results for items in the 2018-19 ELA/literacy and mathematics summative item pools. The numbers of items with moderate or significant levels of DIF (B or C DIF) in the summative pools were relatively small. Items classified as N/A (not assessed) were items for which sample size requirements were not met. Most of these cases occurred for the Native American/Alaskan Native focal group. These students comprised only about 1% of the total test-taking population.

Grade DIF Category Female Male Asian White Black White Hiapanic White Native American White IEP NonIEP LEP NonLEP Title1 NonTitle1
3 N/A 0 167 91 2 637 77 29 0
3 A 912 693 781 888 271 815 859 913
3 B- 8 30 38 24 12 23 29 10
3 B+ 4 29 12 7 4 9 5 0
3 C- 0 2 2 3 0 0 2 1
3 C+ 0 3 0 0 0 0 0 0
4 N/A 0 131 91 4 578 67 18 0
4 A 868 696 752 845 295 776 821 876
4 B- 12 19 36 35 11 36 41 11
4 B+ 8 35 8 3 5 10 6 1
4 C- 1 4 1 2 0 0 3 1
4 C+ 0 4 1 0 0 0 0 0
5 N/A 0 126 83 1 605 57 34 0
5 A 821 651 733 808 242 763 757 843
5 B- 22 43 28 40 12 32 55 17
5 B+ 14 35 14 9 3 5 8 2
5 C- 4 2 1 4 0 5 6 0
5 C+ 1 5 3 0 0 0 2 0
6 N/A 0 145 69 0 567 46 59 0
6 A 817 656 752 827 287 781 738 854
6 B- 22 32 39 29 7 30 62 11
6 B+ 18 23 7 7 6 3 4 3
6 C- 3 5 0 5 1 8 5 0
6 C+ 8 7 1 0 0 0 0 0
7 N/A 0 133 90 2 464 57 111 0
7 A 726 626 678 769 328 714 649 792
7 B- 30 21 28 27 8 30 40 14
7 B+ 35 19 9 4 7 6 5 1
7 C- 6 1 1 5 0 0 2 0
7 C+ 10 7 1 0 0 0 0 0
8 N/A 0 136 122 0 621 93 138 0
8 A 791 668 722 842 263 747 671 873
8 B- 35 42 32 29 4 43 63 16
8 B+ 40 31 11 11 2 4 11 0
8 C- 7 3 2 5 0 3 7 1
8 C+ 17 10 1 3 0 0 0 0
11 N/A 99 882 1,113 104 2,570 1,350 1,650 99
11 A 2,395 1,649 1,522 2,405 119 1,297 974 2,504
11 B- 117 86 34 147 1 28 51 81
11 B+ 49 66 23 28 5 18 18 9
11 C- 15 3 2 11 1 1 2 3
11 C+ 21 10 2 1 0 2 1 0

Grade DIF Category Female Male Asian White Black White Hiapanic White Native American White IEP NonIEP LEP NonLEP Title1 NonTitle1
3 N/A 0 132 40 0 1,258 2 0 0
3 A 1,292 1,058 1,175 1,231 69 1,270 1,267 1,306
3 B- 16 34 48 50 0 37 29 20
3 B+ 16 81 58 42 1 16 27 3
3 C- 4 4 2 3 0 2 2 0
3 C+ 1 20 6 3 1 2 4 0
4 N/A 0 137 129 0 1,258 17 1 0
4 A 1,386 1,150 1,226 1,371 173 1,354 1,358 1,414
4 B- 29 32 41 39 2 63 45 23
4 B+ 24 99 37 26 7 5 29 4
4 C- 1 5 3 2 0 1 6 0
4 C+ 1 18 5 3 1 1 2 0
5 N/A 0 133 125 0 1,123 1 18 0
5 A 1,322 1,071 1,174 1,325 235 1,275 1,302 1,360
5 B- 29 21 28 31 4 63 29 11
5 B+ 19 122 45 15 7 23 20 2
5 C- 3 5 1 2 1 10 4 0
5 C+ 0 21 0 0 3 1 0 0
6 N/A 0 125 267 0 1,172 111 71 0
6 A 1,179 985 927 1,187 67 1,069 1,118 1,209
6 B- 28 22 17 31 0 41 25 21
6 B+ 29 67 21 15 0 13 21 7
6 C- 2 5 3 2 0 5 2 2
6 C+ 1 35 4 4 0 0 2 0
7 N/A 0 191 325 1 1,063 174 160 0
7 A 1,081 825 786 1,064 75 919 936 1,091
7 B- 35 8 5 47 0 28 18 43
7 B+ 22 82 17 21 1 16 18 4
7 C- 1 3 2 6 0 1 1 1
7 C+ 0 30 4 0 0 1 6 0
8 N/A 0 240 210 0 955 120 251 0
8 A 968 678 743 948 37 824 708 971
8 B- 17 19 22 28 1 30 24 20
8 B+ 8 42 17 14 1 12 8 3
8 C- 1 4 1 4 0 3 1 0
8 C+ 0 11 1 0 0 5 2 0
11 N/A 500 1,714 1,222 501 2,666 1,946 2,184 500
11 A 2,053 833 1,400 2,070 13 690 463 2,115
11 B- 51 15 16 60 1 20 11 40
11 B+ 56 79 36 42 0 22 18 20
11 C- 12 1 3 2 0 0 1 2
11 C+ 8 38 3 5 0 2 3 3

3.10 Test Fairness and Implications for Ongoing Research

The evidence presented in this chapter underscores the Smarter Balanced Consortium’s commitment to fair and equitable assessment for all students, regardless of their gender, cultural heritage, disability status, native language, and other characteristics. In addition to these proactive development activities designed to promote equitable assessments, other forms of evidence for test fairness are identified in the Standards (2014). They are described and referenced in the validity framework of Chapter 1.