#QuestionTheWeb Research Plan

2.0 Research Plan

This developmental goal research grant will occur over four years. The first two years will be formative design studies. In Year Three, we conduct a switching replication design pilot study. Year Four involves data analysis and a cost/benefit study.

2.1 Research Goals

This research proposal responds to the IES call by addressing the national crisis in writing scores. We will develop a technology-based intervention employing a theoretical framework for reading and writing the web, moving teachers and students toward an understanding that meaning making is socially-situated, fluid, and dynamic. Interventions in both argumentative writing and source evaluation are essential to assess how an author’s point of view may influence their textual choices. We believe the best way to understand the role that identity plays is to empower students to take control of shaping their own academic identity, online, and then interrogate how perspectives influence the use and belief of claims and evidence.

Year	Goal	Method	Outcomes
1	Develop Intervention Learning Activities Test and refine student blogging platform	Formative Design utilizing field notes and cognitive labs	Establish baseline data (e.g., students’ initial writing sample, self-efficacy assessment; use of critical evaluation skills) Revise existing phase I learning activities Refine existing social reader. Test student use of multiple features on blogging platform.
2	Iterative design of curriculum materials	Formative Design utilizing field notes, basic learning analytics, and cognitive labs	Establish baseline data and track over time to ensure efficacy of our theory of change Reliability estimates Revise existing Phase II learning activities Record, edit, and publish biased think alouds.
3	Pilot Study	Switching Replication Design	t-test for mean comparison Repeated Measures for effect of time and order
4	Cost and Feasibility Study	Analyzing Year Three data	Research ready for dissemination Findings ready for a Level 5 Measurement grant

2.2 Baseline Assessments

2.2.1 Analytic Writing Continuum.

As a significant component of this intervention study, students will be writing about the effects of social media and publishing their work to their own personal website. The writing itself will vary in terms of topic, length, frequency, purpose, and audience. Because of the vast number of entries that students publishing on the open web could lead to—and because we need a clear, coherent method for evaluating the quality of such writing across time for individual participants and for the group as a whole—it is imperative that we have a proven, flexible tool for writing assessment. Moreover, because we aim to engage educators in the process, we need a tool that will serve multiple purposes.

In an effort to bring an accurate, yet nuanced, approach to writing assessment, the National Writing Project developed its “Analytic Writing Continuum” (AWC) as a tool “appropriate for use across genres of writing and grade levels” (Bang, 2013). Built on the foundation of the 6+1 traits model and “[u]sing both holistic and analytic procedures, the AWC is a robust instrument that measures all the important aspects of writing, consistently across multiple raters and scoring sessions” (Bang, 2013).

DevelopmentThe National Writing Project’s (NWP) Analytic Writing Continuum was initially developed based on the 6+1 Writing Traits (Culham, 2003) to support research on the impact of NWP programs on student writing achievement (Swain & LeMahieu, 2012). After a review of writing rubrics from several English-speaking countries, the 6+1 Traits Writing Model was selected for adaptation because of its focus on attributes of writing valued by NWP, focus on writing quality, potential to address a range of grade levels and writing genres, and a scale sensitive enough to detect differences. NWP revised the rubric to focus on the writing rather than on the reader or the writer, expand the number of scale points, and emphasize what is in the writing rather than what is not there. NWP also conducted a series of studies to test the reliability and validity of the AWC (Bang, 2013).

ReliabilityIn 2013, NWP developed the AWC for Source-Based Argument (AWC-SBA) to focus on specific features of source-based argument writing. In adapting the AWC, NWP reviewed extant argument writing rubrics (e.g., the Smarter Balanced and PARCC rubrics). The AWC-SBA retain the AWC’s basic structure rooted in the “six traits” of writing but each has a particular focus on the attributes related to source-based argument writing. The AWC-SBA measures four attributes: content (e.g., quality of reasoning and strength of evidence); structure (e.g., organization to enhance the argument); stance (e.g., tone, establishment of credibility); and conventions (e.g., control of usage, punctuation, spelling, capitalization, and paragraphing). The AWC-SBA has been used in three large-scale scorings in large-scale scorings (n>5,000) and performed similarly to the original AWC. For example, reliability estimates for the AWC-SBA ranged from 89%–92% on each attribute (Gallagher, Arshan, & Woodworth, 2017.

Scoring Procedure

In order to reduce bias when the writing is scored and to control costs for participating LEAs, the AWC-SBA will not be scored until the end of Year Four. This will include the three assessments delivered in Year One, Year Two, and Year Three. This approach requires only one group of raters to be trained and reduces cost of scoring. While this approach does not allow for the use of instument as formative data we will draw on other writing samples and transcripts of dialogue for data that identifies pedagogical enhacement to our models.

2.2.2 Critical Online Information Literacies

Critical Online Information Literacies Assessment is a previously validated multiple choice assessment of website evaluation. The Critical Online Information Literacies (COIL) instrument is based on measures developed by Kiili, Laurinen & Marttunen, (2008), Brem, Russell, & Weems (2001), and Leu et al. (2010). The final instrument is delivered using an online survey tool. The items measured each of the following constructs: author, bias, publisher, and source.

DevelopmentThe COIL assessments was developed over two iterations (author, 2013; O'Byrne, 2011). There are two types of items. The first consisted of screenshots of websites. These were either single websites or four screenshots. The second type of item required students to open links to four different websites. These websites included secondary links to author and publisher information. All of the websites involved, either static or dynamic, are recreations of actual websites. Each test item udnerwent rigorous content validation procedures. The isntrumnent was refined after each adminsistration

ReliabilityThe COIL contained forced response answers that included entire screenshots of websites. This violates basic principles of comprehension assessment that call for short distractors (Fuchs, Fuchs, & Maxwell, 1988; Keenan & Betjemann, 2008) However, in each administration of the COIL the reliability had a coefficient alpha that exceeded 70% (McVerry, 2013) which will meet the needs of a developmental research grant.

2.2.3 Writing Self-Efficacy.

The Self-Efficacy for Writing Scale (SEWS) (Bruning et al., 2013) will be used to estimate writing efficacy. The measure contains items that assess load on three scales: (1) self-efficacy for writing ideation, (2) writing conventions, and (3) writing self-regulation. More specifically, following Bruning et al.’s design, the instrument will include 16 items that examine the three aforementioned dimensions of self-efficacy for writing: (1) Ideation (5 items); (2) Conventions (5 items); and (3) Self-regulation (6 items). Following recommendations by Bandura (2006) and Pajares and Schunk (2001), the Self-Efficacy for Writing Scale (SEWS; Bruning et al., 2013) includes items answered on a Likert scale from 0 (I'm not sure I could do) to 100 (I'm totally sure I could do).

Survey of Internet Use and Online Reading.

This online survey identifies frequency and type of Internet use inside and outside school, as well as assesses knowledge and skill related to Internet-based reading and writing activities. With funding from a previous IES project, it was developed, field tested, revised, and then administered to approximately 1600 middle-grade students in several urban and rural school districts under the supervision of Dr. James Witte at Clemson University, a nationally recognized expert in online survey development. Samples appear in Appendix (X). This survey has been previously validated and used in numerous studies (cit)

2.3 Formative Design in Year One

We will use a series of design experiments (Brown, 1992; Cobb, Confrey, DiSessa, Lehrer, & Schauble, 2003) to build upon our existing model of teaching argumentative writing with students composing texts and posting them to their own domain/blog. Our design experiments are distinctive in that they engage teachers as “insiders” who collaborate with the research team, using outcome data to revise the curriculum and tools before experimentation.

In Year One, we begin with our pedagogical goal to improve the self efficacy and ability of adolescents to critically evaluate sources and use this evidence in argumentative writing.

2.3.1 Year One Research Question

In a formative experiment, we must identify essential components of the intervention to be studied a priori to implementation and data collection (Reinking & Bradley, 2008), yet iterate on this understanding as you identify variables and strategies that enhance your pedagogical goal. In Year One we will ask, “What are the essential components a hybrid writing environment needed to improve argumentative writing when we give every student their own domain?”

Therefore, the main thrust of Year One will be to identify the variables that support the improvement of argumentative writing, specifically when every student has access to their own website. In Year One, we also seek to iterate on the learning activities planned in the two phase intervention. The instructional design decision made a priori are based on our theoretical alignment and the results of previous research.

2.3.3 Overview of Research Activities. The formative design experiment in Year One of the study will follow a three phase model adapted from Reinking and Bradley (2004, 2008; see also Reinking & Watkins, 1998): (a) gathering baseline data; (b) implementing and developing the intervention based on data collection and analysis; (c) retrospective analysis.

Setting: The formative design study will take place in one school district in Connecticut in two classrooms. We will begin with a one day training for teachers on the technology of blogging and setting up their own personal websites. We will then provide a second day overview on the interventions. Then we will follow an iterative design process.

The participants will also have access to an online coach who will offer pre-recorded tutorials on an as needed basis. Teachers will also be able to attend open office hours that are hosted online. These will be drop in sessions using video chat software.

Materials: The project will begin by creating a set of learning materials designed to scaffold students as they begin to publish from their own domain. A sequence of learning goals will be developed at the first Scientific Advisory Board.

Figure 1. Strategy for Development And Modification Of Each Learning Activity: Our Iterative Process

1. Initial input by the Research team drawing upon our theory of change.
2. Definition of assessment framework and scope of learning sequence in conjunction with the Scientific Advisory Board (SAB).
3. Development of a corpus of approximately twice the number of learning activities needed for 4 weeks of lessons with two one-hour lessons a week.
4. Analysis in the cognitive lab of any tutorial accompanying the lesson.
5. Interview with the teacher before the lesson
6. Observation of Hybrid Writing Space. An examination of participatory learning practices that support learning argumentative writing
7. Analysis of student writing. Looking for increase used of claims, evidence, and source attribution.
8. Correlation between COIL Scores and Writing Analysis. Examining the relationship between the frequency of claims, evidence, and source attribution to scores on an assessment of critical evaluation skills.
9. Analysis by Scientific Advisory Board. 
10. Selection of “best” versions After the learning cycle we will further refine the learning activities.

Phase 1: Gathering Baseline Data. We will begin by first gathering background data on how the school previously taught source evaluation and argumentative writing. Interviews will be conducted with the teachers and classroom observations conducted

We will then administer the baseline assessments of writing efficacy and the COIL assessment of critical evaluation. The Critical Online Information Literacies Inventory will be scored to identify key areas of instruction. The ACW, which is out student outcome measure will be scored after the three year data collection

Phase II: Implementing and Developing the Intervention. Once the learning activities are developed we will focus on the delivery of the learning activities. These activities will center around the role of identity and social media. Students will then practice sourcing skills as they read about “The effect of social media” while making argumentative writing pieces across a variety of modes. They will then create “their most credible self” while building an About Me Page.

The developing of these lessons will occur over an eight-week period. The graduate student and project manager will work with the teacher and act as participant observers by intentionally designing lessons in the hybrid learning spaces. They will specifically identify variables that improve both the learning activities and the literacy spaces created over the eight-week period.

After the first iteration, the PIs and graduate researchers will use the data collected to revise the learning activities. The lessons will then again be delivered to another class. However, these students will have access to their websites since the beginning of the study. This will allow us to understand the impact new technology has on the learning environments and outcomes.

Phase II: Implementing and Developing the Technology. As we iterate on the lesson plans we will also be developing the blogging platform for students to use. As stated we are not creating a “new” platform or learning management system. Instead we are encouraging students to blog from their own web domain using a customized theme we create, including both the blogging tools and the updated RSS reader.

While we are using mature web standards and will be using well-established, open source solutions, we will be doing some custom development in Year One. We will use industry standard methods to evaluate accessibility and use of the tools. In the Year One formative design, the goals will be to test and develop:

• Functioning webmentions (comments) and fragmentions (annotations)
• A social reader with APIs for micropub (writing) and microsub (reading) which is web based, and therefore available on every desktop and mobile device

 These blogging tools use very specific types of metadata directly in the html file of every post, and this metadata is called microformats. This has been a tool used on the web for over thirteen years. Using this metadata, students will be able to send comments back and forth to each other from their own websites. This uses a technology called webmentions. They will be able to highlight and annotate each other’s blogs from their own website using a technology called fragmentions.

The students will also have a social reader that allows them to curate all the posts from their classmates and the teacher. They will be to add other feeds using a syndication technology called RSS. This tool will be catered to the purpose of source evaluation. For example, we will have a tag for evaluating the claims and evidence used by their peers. Students will highlight blog posts of their peers and then judge the credibility of the evidence. These special posts can be pulled directly into a social feed reader.

The teacher will also be given a social reader with additional tools for tracking the progress. This includes counts of publishing and commenting, the number of claims evaluated by students, relational maps that illustrate which students interact together. The teacher will also be able to deliver feedback and comments to their students directly from their social reader.

In the Year One formative design study we will work on refining existing tools through an iterative process that focuses on the environment of the user interaction as well as the user experience. Weekly design sprints will be used based on a roadmap developed a priori to start but that is then revised based on the data collected. The overall software development goal for the Year One formative design is refinement of existing technologies to meet our classroom needs.

Data Collection and Analysis. The graduate assistants will act as participant observers. While the classroom teacher will remain the primary instructor and deliver the lessons included in the intervention, the graduate assistant will provide assistance. A graduate assistant will be in attendance for the twice a week delivery. The principal investigators will provide just in time coaching and feedback.

Qualitative data. Qualitative data will include (a) field notes recorded and analyzed by graduate research assistants who have had explicit training in gathering and analyzing qualitative data; (b) teacher logs with observations and reflections about #QuestionTheWeb, which are particularly useful for our awareness of possible effects and issues that arise during times when a member of the research team is not present; (c) weekly interviews/debriefings (15-30 minutes) with teachers conducted by a member of the research team, which will be recorded and transcribed; and (d) periodic focus-group interviews with students.

Immediately after each #QuestionTheWeb lesson, a research assistant who either observed or who led the lesson will write reflective field notes. Student artifacts such as digital presentations and various assignments will also be included in data collection and analysis. These data will primarily guide development, but will also include theoretical notes aimed at generating broader pedagogical theories and generalizations across contexts related to implementing a digital tools to teach argumentation. As noted, the research team will meet at least every two week to conduct preliminary retrospective analyses aimed at generating broader theoretical understandings and at sharing observations across sites focusing on determining (a) factors that enhance or inhibit #QuestionTheWeb success in achieving its goals, (b) what modifications or adaptations were made in response to that data, and (c) whether the modifications or adaptations produced desirable results.

In addition, with input from classroom teachers, we will identify 2-4 writers in each class as focal students for cognitive labs. These students will offer feedback on the lessons and on the user experience of having a blog on their own domain and a hybrid learning environment that allows students to curate, read, and comment on the blog post of their peers. We will also have students work through tutorials and lesson plans while thinking out loud.

Quantitative data collected during the intervention will include the administration of the COIL Assessment at three different time points. At the start of the intervention, the four-week mark, and the eight-week mark. Students will also complete the AWC at this time. However, these assessments will not be scored until the end of the funding period. The COIL assessments will then be correlated with periodic content analysis of student posts.

The students will also complete self-efficacy measures at three different time points during the intervention. As part of year one we will check our factor loadings to see if the hypothetical model holds. While a survey with a single item construct would be more ideal for analysis we will work with the SAB to understand the model design possibilities and decisions of what scales to use.

The graduate assistants will analyze student writing and code the frequency of claims and evidence. Furthermore, we will have a record of all the fragmentation students used to annotate websites. These frequency counts can be correlated against the COIL scores. In Year Four of the grant we will test a variety of models for best fit in order to explore the relationship between sourcing skills and argumentative writing.

Phase III: Retrospective Analysis. During this phase the research team will collectively conduct a comprehensive and exhaustive retrospective analysis (Cobb, et al., 2003; Gravemiejer & Cobb, 2006) of all available data. The object of this analysis will be to (a) determine the “active ingredients” of our intervention that are likely to be factors related to its success in any context, (b) consider adjustments that may enhance the intervention in the following years (Years 2 and 3), (c) evaluate findings in relation to existing theory and research, and (d) identify context-specific factors. The members of the Scientific Advisory Board will participate in a portion of these discussions via teleconferencing or online webinar applications. The research team will also determine the conference presentations, technical reports, and publications that will be the focus of each summer’s work.

2.3.4 Desired Outcomes We will use intermittent assessments and triangulate the descriptive data with our field observations to ensure we our theory of change is leading to the desired outcomes. We anticipate, using a repeated measures ANOVA, that there could be an effect size of Cohen’s d=0..3 when examining the difference in mean scores of the frequencies of claims and evidence in student writing at both the four-week mark and the eight-week mark of the intervention.

 After the conclusion of the first year, we will have refined the blogging platform. We will have also collected data to check on instrument reliability to ensure we can measure the effectiveness of our theory of change. Every student and the teacher will have an updated feed reader that allows them to comment and annotate on each other’s work while writing their own post.

2.4 Formative Design in Year Two

In Year Two we will use a similar formative design approach. However, we will concentrate on refining the Phase II lesson plans, developing the bias read-aloud videos to assist students and teachers in source evaluation and argumentative writing.

2.4.1 Year Two Research Questions. In Year Two we ask, “Can encountering bias think alouds and receiving additional scaffolds improve student critical evaluation and argumentative writing skills?” We will shift our focus from developing learning activities about exploring how “I shape truth” to refining the learning activities and technologies for students to consider the implications of how “we shape truth” in online spaces.

2.4.2 Rationale. Given our goal of improving argumentative writing among adolescents, we developed the second year formative design experiment around this goal. Consistent with formative experiments, we intend to learn as much from our failures as our successes, including perhaps identifying thresholds of failure for components of argumentative writing in challenging contexts (see Sloane & Gorard, 2003; Wagner, 1993) like hybrid writing spaces. Our goal will be to gain deep understandings of the factors that enhance or inhibit the effectiveness, efficiency, and appeal and to continue to evaluate the nature of “active ingredients” identified in Year One. However, the goal in Year Two will be to focus on the Phase II learning activities, developing interactive think alouds.

We included biased think alouds in this study due to the demonstrated results that checklists approaches fail to improve sourcing skills. Furthermore, we want our young writers to see argumentation not as a single thesis statement, but as an ongoing societal dialogue. A biased think aloud will be a video recorded avatar that is overlaid on top of an annotated website. For instance, a non-human character could be a representative for the local farming bureau worried about carbon taxes and the cost of fertilizer.

We will also test interactive graphic organizers that will scaffold students argumentative writing. We will include two models for testing the first will be based on the Toulmin model (Toulmin, 2003) and an Argumentative Vee Diagram (Nuassbaum, 2008). These techniques have shown to lead to improvement in writing ability.

2.4.3 Overview of Research Activities. Year Two will be an iterative design process that proceeds through the same four phases as Year One. However, in Year Two we will be in two classrooms across each state, CN and MI, during each of the eight-week periods. This means we will conduct our research in a total of eight classrooms in Year Two. The second phase will begin after the same Phase I Baseline data from Year One is collected.

Setting and Participants: In Year Two of the formative design, the lessons will take place in four different classrooms in four different schools, two in each state. The same cohort of Year One teachers will be involved as well as the addition of a new cohort. Each intervention period will last a total of four weeks. However, all students and teachers will be given access to the blogging platform and the social reader at the beginning of the year. This will let us explore the role teacher experience has in shaping hybrid learning environments to support argumentative learning.

The participants will also have access to an online coach who will offer pre-recorded tutorials on an as-needed basis. Teachers will also be able to attend open office hours that are hosted online. These will be drop in sessions using video chat software.

Material: The Phase II lesson plans revolve around students investigating contemporary issues in social sciences and the humanities. The topical choices will be developed by the research team and then refined by the Scientific Advisory Board. The lessons will consist of eight 45-minute to one hour lessons delivered twice a week.

Phase II: Implementing and Developing the Technology. Phase II development will begin with the Principal Investigators and Graduate Assistants drafting the learning activities. They will then present these drafts to the Scientific Advisory Board.

The scripts for the biased read alouds will include seven sources, and will include the use of five websites per social topic. Two sources will be chosen on each side of an issue, with one additional source considered to be the most neutral. Two avatars will be created, one for each of the positions. The avatars will be pop-up windows with an animated mouse that hovers and highlights. Each avatar to complete a think aloud for two sites they disagree with and one site where they share an opinion. Each avatar in a given social set will read the source considered most neutral.

After each script is written the avatar and screen recordings of website mouse clicks will be recorded. The voice of the avatars will be from the PIs and research assistants. These videos will then be layered as proxies on top of the original sources so users can toggle them on and off in their social reader.

The avatars will then undergo a series of cognitive labs before deployment. In conjunction with the classroom teacher, students will be chosen to participate. Students will be asked to identify key decisions they make when summarizing the conclusions of sources through a think-aloud protocol. The participants will also be asked how about the avatars and their perspectives.

The classrooms teachers will then receive a two-day training on supporting student blogging platforms in the classroom. Then in each of the four settings the participants will engage in the second phase of the learning activities included in the lesson. Similar to Year One, the interventions will take place in two 45-60 minute classes over eight weeks. After eight weeks, a second class in each of the four schools will conduct the intervention.

Phase II: Implementing and Developing the Technology. As stated above, the Phase II technology development focuses on interactive graphic organizers. The interactive graphic organizers will rely heavily on javacscript and the micropub and microsub APIs. Basically, students will be able to visit the domain of the graphic organizer and “fill in a box” this will then publish to their own domain. Alternatively, students could publish from their own domain and this entry will pre-populate on their graphic organizers.

Data Collection and Analysis. The graduate assistants will act as participant observers. While the classroom teacher will remain the primary instructor and deliver the lessons included in the intervention the graduate assistant will provide assistance. A graduate assistant will be in attendance for the twice a week delivery. The principal investigators will provide just in time coaching and feedback.

Qualitative data will be collected by research team members, who include Principal and Co-Investigators, as well as graduate research assistants who have had explicit training in gathering and analyzing qualitative data. Data will stem from (a) field notes of _classroom observations by the research team; (b) weekly interviews/debriefings (15-30 minutes) with teachers; (c) periodic focus-group interviews with students; (d) cognitive labs with individual students and (e) teacher self-reports of observations and reflections about #QuestionTheWeb, which are particularly useful for our awareness of possible effects and issues that arise during times when a member of the research team is not present..

We will also examine field notes to continue our efforts to understand how space shaped the quality of hybrid learning environments. We will meet with the scientific advisory board to review results from Phase I. We will then develop a coding system to encourage the use of specific practices noted in Year One with the goal of increasing the frequency of these techniques in Year Two.

Quantitative data collected in Year Two will be very similar to the data collected in Year One. The AWC will again be administered at three different time points. However, these assessments will not be scored until the end of Year Three. The Critical Online Information Literacies (COIL) assessment will also be delivered during the same assessment battery. We will also identify the frequency of claims and evidence noted in student writing. Finally, we will have frequency counts of specific HTML elements and tags used by students during their blogging. Correlations and a variety of model fits will be tested with these variables. The hypotheses for these statistical analyses will not be developed a priori of the learning activities and the platform features. Instead variables that support desired outcomes will be further investigated for inclusion in the pilot study.

Phase III Retrospective Analysis. As we finish Year II we will examine all the data collected in both formative design years to inform final changes before the pilot study. In year II we did not include the Phase I lessons. These lessons provided an exploration of self while also scaffolding the blogging platform. A comparison of Year Two and One field notes will provide insight into the types of supports students and teachers will need. We will also compare demographic data between the two years. Specifically, we will seek to triangulate the claims and evidence students use with the specific artifacts from learning activities, field notes, and platform uses.

The retrospective analysis will first occur in December after the intervention comes to a conclusion in four classrooms who completed the lessons. Focus will be placed on analyzing the results of cognitive labs, revising the avatar scripts, and developing new bot scripts based on new triggers. Then after the conclusion of Year Two the scientific advisory board will gather for one last time to make final revisions based on the second cycle of learning activities delivered in Year Two.

2.4.4 Desired Outcomes. After the conclusion of the Year Two formative assessment we will have greater evidence that our theory of change leads to desired outcome. First, with a sample size between 100-120 in Year One, we hope to derive reliability estimates of our measure before undertaking the pilot study in Year Two. This will help to ensure we can estimate effect sizes that would indicate we are progressing towards our outcomes in the theory of change.

At the conclusion of Year Two, we will have developed and recorded avatar scripts for four different argumentative topics in the field of humanities and social studies. This will be a total of 32 different bias read alouds.

 

2.5 Pilot Study in Year Three

In Year Three of the grant we will perform a pilot study testing out both the learning activities and the blogging platform developed in the first two years of the pilot study. The goals of the study will be to provide evidence on the relationship between source evaluation, argumentative writing and any possible effects our learning interventions may have.

2.5.1 Participants. This will be an approximate sample size of 300 students. According to Tabachnick and Field (2001), a sample of 300 participants exceeds their guidelines for a t-test using a Cohen’s d effect size of 0.3 and also for any possible regression analysis of 100 + m, where m equals the number of predictors. This sample size is also adequate for a regression model with one dependent variable and three independent variables with an =.05, a desired power size of 0.8, and an anticipated effect size of 0.15 (Sloper, 2010). This anticipated Cohen’s f2 is a medium effect. The estimate of power size was chosen to ensure an adequate effect (Sloper, 2010).

Settings The pilot study will take place in four schools across two states, with two schools in each state. We have chosen to focus on middle school students due to federal child data protection laws and alignment with curriculum goals. There will be one classroom in each school assigned to one of three conditions: Condition A, Condition B, and control. Each group will have 300 students, while this sample size is large for a Goal Two: Development grant it will provide balanced design to strengthen our estimations and allow us to estimate the reliability of our measures when estimating effect sizes to track if our theory of change will lead to our hypothetical educational outcomes.

2.5.2 Measures Measures in the pilot study will include the AWC-SBA, the COIL, a survey of self-efficacy of writing, and the internet use survey. All reliability estimates will be conducted after the Year Two Formative Design.

2.5.3 Methods The pilot study will use a switch replication design with two conditions. Both groups will start with the assessment battery and then be given access to the blogging platform. Group A will begin the eight-week intervention while group B completes normally scheduled classroom activities with the option of using the blogging platform. The control group will complete the battery of assessments in the third testing window.

Group	Period One 10 weeks	Period Two 10 weeks
Condition A	X	Students get blogging platform. Complete Phase I and II	X	X
Condition B	X	Students Get blogging platform	X	Complete Phase I and II	X
Control	X

2.5.4 Analysis The data analysis to examine the effects of instruction on source evaluation and scores on the AWC-SBA will be different models. We will first begin by testing for the classical assumptions for a repeated measures design. First, we will test if either condition had learning gains over above a control group which will consist of the same sample size and be given the three assessments at the final time point. Using a t-test we will report our effect sizes

If there is a significant difference between the means with a medium to large effect size, we will have evidence that our theory of change lead to the expected outcomes. If we do not find a statistical difference we will also establish that our intervention did not hinder students’ progress when compared to their peers. If the control group has a significant greater average then we will have practicality evidence that at some point our interventions did not lead to student growth.

To answer research question one, the dependent variable will be the self-efficacy measure given at three timepoints (Time: 1, 2, 3), and the independent variable will be the order factor that represented the sequence of instruction for each student (Order: blogging platform with intervention, intervention after having blogging platform). Covariates will include a proxy of prior knowledge of the internet using a self report measure of internet frequency which has been used in previous studies (Leu et al., 2015).

To answer research question two, a repeated measures design the dependent variable will be the AWC measure given at three timepoints (Time: 1, 2, 3), and the independent variable will be the order factor that represented the sequence of instruction for each student (Order: blogging platform with intervention, intervention after having blogging platform). We will again use the survey of internet use as a covariate.

To answer research question three, the dependent variable will be scores on the COIL given at three timepoints (Time: 1, 2, 3), and the independent variable will be the order factor that represented the sequence of instruction for each student (Order: blogging platform with intervention, intervention after having blogging platform). We will again use the survey of internet use as a covariate.

In conjunction with the scientific advisory board, a variety of model fits will be tested to see what best explains variance in argumentative writing skills. This will allow us to estimate the effect of time. Other mediating variables that may be included in the model include, prior knowledge, as measured using concept maps, and critical evaluation scores on the COIL. These data are intended to complement and extend the qualitative and quantitative data collected during the intervention phase. These statistical comparisons may be made, but these comparisons are not intended to carry the weight of a randomized experimental design or analysis using HLM procedures, which would be more consistent with a Goal Five project. Furthermore, we do not have the sample sizes to do so. However, it does allow us an opportunity to field-test these procedures, especially in Year Three with an eye toward a subsequent Goal Three project. We also expect that these assessments may be useful, not just for data collection and analysis, but also useful tools for teachers who wish to formatively assess students’ progress.

A switch replication design was chosen for the study given our firm belief that all adolescents will benefit from writing on their own domain and receiving additional instruction in digital literacies. While a quasi-experimental study with a control group would allow us to estimate if treatment groups performed better than no instruction at all, the required sample size for such as suited for a Goal Three research project.

Furthermore, a switched replication design lets us better understand any possible mediators between time and order of instruction. Previous research introducing technology found an initial drop in performance on measures of reading comprehension (Leu et al, 2016). This finding is mirrored in qualitative studies of distributed classrooms using blogging technologies similar to our #QuestionTheWeb intervention. Our proposed model will allow us to explore for any mediators of time and order that should be further explored when we seek to scale up the grant as a Goal Three.

2.6 Practicality, Fidelity, and cost study in Year Four

It is essential that any instruction model to be practical. Beyond the data analysis in Year Four we will also focus on practicality. We define this as having reproducible fidelity and a low cost. Time and money are precious commodities during the school year. For this proposal, we have thought carefully about this issue, recognizing that no instructional model will be adopted unless it is ultimately practical to stakeholders. First, the implementation of any model must have fidelity to ensure its success, yet each classroom is a unique context. We try to address this paradigm by creating choice within a constrained system. We will develop a series of learning activities and a similar tech set-up, but students and teachers can adopt the blogging to their local platforms. To measure fidelity in the Year Three pilot study the research team will use a checklist of “Qualities of Hybrid Writing Spaces” developed in conjunction with the scientific advisory board after the formative design experiments in Year Two.

Second, we have already piloted an initial versions of our model (McVerry, 2015) and draw on a long history of distributed courses. Scholars have already looked closely at the initial version of the model and made changes to it, in order to make it more practical for a whole school context.

We will also conduct a cost analysis in Year Four. We have built in an online coach into our model. While some see this as an expense districts could not afford, we believe non-union, non-employed coaches provide a huge cost saving opportunity. The cost of two districts splitting a part time coach is vastly cheaper than each having a full-time employee with the associated fringe.

To complete a cost benefit analysis in Year Four we will first develop criteria for ascertaining what is considered a cost. If, for example, a graduate assistant or PI must teach in Year Three, the cost of one day for a secondary teacher will be added. The technology costs are fixed and easy to project to scale.

We will then look to our benefits in terms of cost. In order to do this, we will examine the student learning outcomes as measured by the AWC. This will then be compared to other writing instruction curriculums that have had similar outcomes.