Introduction and Overview

Technical Report Approach

This report is intended to provide evidence in support of the validity and reliability of the Smarter Balanced interim assessments for the 2022-23 school year. Three types of interim assessments are described in this report: interim comprehensive assessments (ICAs), interim assessment blocks (IABs), and focused interim assessment blocks (Focused IABs). Information about the overall system, which includes the summative assessment and Tools for Teachers, is provided for context. The interim assessments are routinely updated by replacing items and adding more tests, but such updates subsequent to the 2022-23 school year are not represented in this report.

Demonstration of validity and reliability is an ongoing process. Validity and reliability evidence provided here include evidence from the initial pilot and the field test phases as well as evidence from more recent operational assessments. Members generally do not provide interim assessment response data or scores to the Consortium for analysis. Consequently, much of the evidence in this report focuses on the development of test items and characteristics of test forms.

The interim assessments are designed to support teaching and learning throughout the year. They may be administered in a standard manner or teachers may use interim items or tasks as a basis for class discussion or individual feedback. In addition, the assessments may be administered to the same students several times during the year and teachers may administer individual assessments at any grade level (e.g., a teacher may administer a grade 4 interim assessment to students in grade 3). Reliability and validity evidence provided in this technical report holds only for tests administered the first time using standardized administration, assuming that students have not been exposed to the items and each student does his or her own work. When teachers use interim items or tasks as a basis for class discussion or individual feedback prior to administration, student scores may not have the same properties described in this report.

Smarter Balanced provides a customizable Online Summative Assessment Test Administration Manual (TAM) for summative assessments that may be used for standardized interim assessment administrations. Some states have developed interim assessment test administration manuals of their own. Members can usually find the appropriate customized version on their state’s assessment portal. Beginning with the 2019–20 administration, Smarter Balanced provided a customizable Interim Assessment Guide that members may use to create or update their state-specific interim assessment TAMs.

The Standards for Educational and Psychological Testing (American Educational Research Association, American Psychological Association, & National Council on Measurement in Education, 2014), hereafter referred to as the Standards, was used as the foundation for developing the necessary validity evidence. With respect to Smarter Balanced, this information is necessary for understanding the degree to which the Consortium is meeting its goals, and in some cases, what further tasks remain to improve the system as it evolves operationally.

Overview of the Smarter Balanced Assessment System

The Smarter Balanced Assessment System includes a set of balanced components designed to meet diverse student needs for all Consortium members. It consists of three components: interim assessments designed to support teaching and learning throughout the year, a suite of tools and resources in Tools for Teachers that support classroom-based formative assessment practices, and end-of-year summative assessments designed for accountability purposes. This system provides valid, reliable, and fair assessments of the deep disciplinary understanding and higher-order thinking skills increasingly demanded by a knowledge-based global economy. The system is based on the belief that assessment must support ongoing improvements in instruction and learning experiences for students that lead to outcomes valued by all stakeholders. Smarter Balanced supports the goals of its members who seek to ensure that all students leave high school prepared for post-secondary success in college or a career through a planned sequence of educational experiences and opportunities. The system was grounded in the strong foundational assessments, policies, and procedures of its members, including supports and resources from institutions of higher education and workplace representatives. The Consortium expanded on these proven successes to create a high-quality, balanced assessment system based on the state standards in English language arts/literacy (ELA/literacy) and mathematics.

Smarter Balanced staff provide expert guidance and facilitate member-driven decisions regarding the maintenance and enhancement of the system as required to fulfill its mission to improve teaching and learning. Smarter Balanced members retain flexibility in how to customize the system so that it may best be used as part of their approach to improving their local educational systems. The Smarter Balanced assessment system strategically uses a variety of item types, including performance tasks, to sample the full range of the 3-HS grade level ELA/literacy and mathematics standards. The Consortium also deploys essential accessibility resources that are embedded in the test to ensure fair and accurate assessment of all students, including students with disabilities, English language learners, and low- and high-performing students. The Smarter Balanced system includes the following components:

  • Summative assessments that determine students’ progress toward college and career readiness in ELA/literacy and mathematics. The summative assessments are given at the end of the school year and consist of two parts: a computer adaptive test and a performance task. These secure summative assessments incorporate a variety of item types, including technology-enhanced items, items that require constructed response, and performance tasks. Items are deliberately designed to measure specific content. The assessments include writing at every grade and ask students to solve multi-step, real-world problems in mathematics.
  • Interim assessments that allow teachers to check student progress throughout the year, and provide them with information they can use to improve instruction and help students meet the challenges of college and career readiness standards. These tools are used at the discretion of schools and districts and can be administered using standardized and non-standardized administration to support student learning. Teachers can employ them to check students’ progress on specific concepts at strategic points during the school year. There are three types of interim assessments: 1) ICAs test the same content as summative assessments and report scores on the same scale as the summative assessment, 2) IABs assess smaller bundles of related concepts and provide more detailed information for instructional purposes, 3) Focused IABs that assess no more than three assessment targets to provide educators with a finer grained understanding of student learning. The ICAs, IABs, and Focused IABs draw from the same bank of interim items and performance tasks. Test questions are developed using the same rigorous methods as those items found on the summative assessment.
  • Tools for Teachers provides educators with instructional supports to learn about and implement the formative assessment process. This online collection consists of high-quality instructional and professional learning resources and formative and accessibility instructional strategies contributed by educators for educators. These resources are aligned with the intent of the standards and help educators implement the formative assessment process to improve teaching and learning. Educators can use the materials to engage in professional learning communities, differentiate instruction for learners’ needs and preferences, engage students in their own learning, improve assessment literacy, and design professional development opportunities. Tools for Teachers also connects instructional resources to IABs. Educators use the Interim Connections Playlist before, during, and after using an interim to identify learning success criteria and to help move student learning forward.
  • The Content Explorer is a web application that provides a user-friendly interface to explore details about how the content is organized on, and how the test questions are developed for, our summative and interim assessments.
  • The Smarter Annotated Response Tool (SmART) helps educators better understand how student writing is scored on Smarter Balanced assessments and support writing instruction in their school or classroom.

In summary, the Smarter Balanced learning and assessment system is grounded in a sound Theory of Action. This system promotes research-supported classroom practice and incorporates a balanced set of technology-enabled tools, innovative assessments, and classroom support materials intended to work coherently to facilitate teaching and learning.

Overview and Background of the Smarter Balanced Theory of Action

Smarter Balanced supports the development and implementation of learning and assessment systems that reshape education in member states to improve student outcomes. Through expanded use of technology and targeted professional development, the Theory of Action calls for the integration of learning and assessment systems, leading to more informed decision-making and higher-quality instruction, ultimately increasing the number of students who are well-prepared for college and careers.

The ultimate goal of Smarter Balanced is to ensure that all students leave high school prepared for post-secondary success in college or a career as a result of increased student learning and improved teaching. This approach suggests that enhanced learning will result when high-quality assessments support ongoing improvements in instruction and learning. A quality assessment system strategically “balances” summative, interim, and formative components (Darling-Hammond & Pecheone, 2010). An assessment system must provide valid measurement across the full range of performance on common academic content, including assessment of deep disciplinary understanding and higher-order thinking skills increasingly demanded by a knowledge-based economy.

Six Principles Underlying the Smarter Balanced Theory of Action

The Smarter Balanced Assessment System is guided by a set of six principles shared by systems in high-achieving nations and in some high-achieving states in the U.S.

  1. Assessments are grounded in a thoughtful standards-based curriculum and are managed as part of an integrated system of standards, curriculum, assessment, instruction, and teacher development. Curriculum and assessments are organized around a well-defined set of learning progressions along multiple dimensions within subject areas. Formative assessment processes and tools and interim assessments are conceptualized in tandem with summative assessments, all of which are linked to the state standards and supported by a unified technology platform.

  2. Assessments produce evidence of student performance on challenging tasks that represent the state standards. Instruction and assessments seek to teach and evaluate knowledge and skills that generalize and can transfer to higher education and multiple work domains. These assessments emphasize deep knowledge of core concepts and ideas within and across the disciplines — along with analysis, synthesis, problem-solving, communication, and critical thinking — thereby requiring a focus on complex performances and specific concepts, facts, and skills.

  3. Teachers are integrally involved in the development and scoring of assessments. While many assessment components are efficiently scored with computer assistance, teachers must also be involved in the formative and interim assessment systems. This is in order to understand and teach in a manner that is consistent with the full intent of the standards while becoming more skilled in their own classroom assessment practices.

  4. The development and implementation of the assessment system is a state-led effort with a transparent and inclusive governance structure. Assessments are structured to improve teaching and learning. Assessments are designed to develop an understanding of learning standards, what constitutes high-quality work, to what degree students are approaching college and career readiness, and what is needed for further student learning.

  5. Assessment and reporting provide useful information on multiple measures for all stakeholders. Reporting of assessment results is timely and meaningful in order to guide instructional and professional development decisions. Interim results can offer specific information about areas of performance so that teachers can follow up with targeted instruction, students can better target their own efforts, and administrators and policymakers can fully understand what students know and can do.

  6. Design and implementation strategies adhere to established professional standards. The development of an integrated, balanced assessment system is an enormous undertaking, requiring commitment to established quality standards in order for the system to be credible, fair, and technically sound. Smarter Balanced continues to be committed to developing an assessment system that meets critical elements required by U.S. DOE Peer Review Guidance, relying heavily on the Standards as its core resource for quality design.

The primary rationale of the Smarter Balanced assessments is that these six principles can interact to improve the intended student outcomes (i.e., college and career readiness).

Purpose of the Smarter Balanced Assessment System

The Smarter Balanced purpose statements are organized into three categories: (a) summative assessments, (b) interim assessments, and (c) Tools for Teachers for formative assessment instructional and professional learning supports. This report provides information about the interim assessments. The purposes of the summative and Tools for Teachers resources are also stated in this section to provide context for interim assessments as a component of a balanced assessment system.

Summative Assessments

The purposes of the Smarter Balanced summative assessments are to provide valid, reliable, and fair information about:

  • students’ ELA/literacy and mathematics achievement with respect to the state standards in grades 3 to 8 and high school;

  • whether students prior to grade 11 have demonstrated sufficient academic proficiency in ELA/literacy and mathematics to be on track for achieving college readiness;

  • whether grade 11 students have sufficient academic proficiency in ELA/literacy and mathematics to be ready to take credit-bearing, transferable college courses after completing their high school coursework;

  • students’ annual progress toward college and career readiness in ELA/literacy and mathematics;

  • how instruction can be improved at the classroom, school, district, and state levels;

  • students’ ELA/literacy and mathematics proficiencies for federal accountability purposes and potentially for state and local accountability systems; and

  • student achievement in ELA/literacy and mathematics that is equitable for all students and targeted student groups.

Interim Assessments

The purposes of the Smarter Balanced interim assessments are to provide valid, reliable, and fair information about:

  • student progress toward the mastery of the skills in ELA/literacy and mathematics measured by the summative assessment;

  • student performance at the claim or cluster of assessment targets so teachers and administrators can better measure students’ performance against end-of-year expectations and adjust instruction accordingly;

  • individual and group (e.g., school, district) performance at the claim level in ELA/literacy and mathematics to determine whether teaching and learning are on target; and

  • teacher-moderated scoring of student responses to constructed-response items as a professional development vehicle to enhance teacher capacity to evaluate and interpret student work aligned to the standards.

Formative Assessments and Tools for Teachers Resources

The purposes of the Smarter Balanced instructional formative assessment process resources are to provide tools and resources to:

  • improve teaching and learning by incorporating explicit learning goals and success criteria;

  • help teachers monitor their students’ progress in real time throughout the school year;

  • illustrate how teachers and other educators can use formative assessment data to engage students in monitoring their own learning and building student agency;

  • help teachers and other educators align instruction, curricula, and assessments to the learning standards and grade-level expectations;

  • assist teachers and other educators in using the interim assessments to improve instruction at the individual student, classroom, school, and district levels;

  • offer professional development and strategy resources for how to use the formative assessment process in daily routines to improve instruction and decision-making in the classroom;

  • provide accessibility resources and strategies to support student access to content during instruction; and

  • directly connect to interim assessment reports and individual interim assessment items to facilitate data-informed instructional decision making.

Overview of Report Chapters

A list of the chapters included in this report is given below, followed by descriptions of the purpose of evidence in each chapter to provide context for teachers, parents, and other stakeholders.

Chapter Number Chapter Title
1 Validity
2 Reliability, Precision, and Errors of Measurement
3 Test Fairness
4 Test Design
5 Test Administration
6 Reporting and Interpretation

Chapter 1: Validity

Although all of the information in this report provides validity evidence, Chapter 1 provides information about test purposes and the overall approach to showing how scores provide validity evidence. The information in this chapter answers questions such as:

  • For what purpose was the interim assessment designed to be used?
  • What evidence shows that test scores are appropriate for these uses?
  • What are the intended test score interpretations for specific uses?

Evidence bearing on these questions does not change with each administration or testing cycle.

Chapter 2: Reliability, Precision, and Errors of Measurement

The degree of accuracy and precision of scores contributes to evidence about appropriate interpretations and uses of test scores. Decisions must be made with full understanding of measurement error and reliability. Chapter 2 presents information about how the test performs in terms of measurement precision, reliability, classification consistency, and other technical criteria. The information is based on facts about the interim fixed forms, and on simulated student response data. Information presented in this chapter can answer questions such as:

  • How accurate and reliable are Smarter Balanced interim test scores?
  • Are Smarter Balanced test scores equally accurate and reliable for all students?

Chapter 3: Test Fairness

Test fairness concerns whether test scores can be interpreted in the same way for all students regardless of race, gender, special needs, and other characteristics. Evidence for test fairness includes documentation of industry-standard procedures for item development and review, appropriate use of professional judgment (e.g., bias review of items), and statistical procedures for detecting potential bias in test items. Information presented in Chapter 3 can answer questions such as:

  • How were test questions and tasks developed to ensure fairness to all students?
  • How is the test administered so that each student can demonstrate their skills?
  • How does one know that the test is fair to all students?

Chapter 4: Test Design

Test design is predominantly focused on the content validity of the test. Tasks and items must represent the domain of knowledge and skill as intended. For Smarter Balanced assessments, test design includes the relationship of claims and targets to the underlying state standards, item development, test blueprints, and scoring procedures. Information in Chapter 4 can answer questions such as:

  • What’s on the test? Is it consistent with stated test purposes?
  • Does each student get a set of questions that fully represents the content domain?

Chapter 5: Test Administration

Part of test validity rests on the assumption that assessments are administered in a standard manner. Unlike the summative assessment, which is administered in only standardized fashion, interim assessments can be administered in non-standard as well as standardized fashion. For standardized administration, the Consortium provides a common administration template that members may customize up to a point for specific use. Chapter 5 describes the customizable Smarter Balanced Online Test Administration Manual. The information in this chapter can answer questions such as:

  • What are the conditions for test administration to assure that every student was afforded the same chance for success?
  • How was the test administered to allow for accessibility for all students?

Chapter 6: Reporting and Interpretation

Reports based on test scores are among the most public-facing features of an assessment program. They must be useful and accurate, supporting the decisions and purposes for which the assessment was designed while discouraging inappropriate conclusions and comparisons. Chapter 6 provides examples of the Smarter Balanced suite of reports and interpretive information and discusses intended uses of report information. Information in this chapter can answer questions such as:

  • What information do Smarter Balanced reports of the interim assessments contain?
  • What do scores mean?
  • How can the reports best be used by teachers and parents?

Acknowledgments

The Consortium acknowledges individuals who contributed time and expertise to our mission, including members of the:

  • Technical Advisory Committee (TAC), who provide guidance on technical assessment matters pertaining to validity and reliability, accuracy, and fairness,

  • Students with Disabilities Advisory Committee, who provide feedback to Smarter Balanced staff, workgroups, and contractors to ensure that the assessments provide valid, reliable, and fair measures of achievement and growth for students with disabilities, and

  • Performance and Practice Committee, who assess the efficiency of Smarter Balanced assessments to meet their designed purpose and to deepen overall stakeholder investment.

Current members of these committees are listed on our website.

References

American Educational Research Association, American Psychological Association, & National Council on Measurement in Education. (2014). Standards for educational and psychological testing. American Educational Research Association.
Darling-Hammond, L., & Pecheone, R. (2010). Developing an internationally comparable balanced assessment system that supports high-quality learning. ETS Center for K-12 Assessment; Performance Management.