News of evaluating commercial educational software in the US

Sun Apr 8 05:24:48 CST 2007

Studies of this kind surely do not help, but there are some good  
counter-arguments already laid out in the responses so far.
The most fundamental problem we have, I suggest, is the origin of the  
study. Note that "Congress asked whether technology was  
effective" (Exec Sum, xv). There's our problem, right there. It's  
like asking whether books are effective, or whether paper is  
effective. It depends... it depends... There will always be dumb  
answers if the question is dumb. With that question, the methodology  
is necessarily experimental - are books effective? - let's test this  
by comparing classrooms with and without selected books, where those  
without may use "other technology products that may have been in  
their classrooms" (Exec Sum, xiv). Would that be a good research design?
So I'd like to add to the critiques so far that: random selection of  
teachers means using teachers who have no experience of it; allowing  
control groups to use other educational technologies means we have no  
idea what is being compared; standard tests are designed to test what  
is learned in standard classrooms; and Congress is asking the wrong  
question.
What would have been the right question to find out whether horseless  
carriages were worth investing in? It's a lot more complex than 'are  
they effective?'.
There is to be an extension of this study, so a robust response ought  
to ensure that it's done in way that will not risk jeopardising the  
education of millions through a poor understanding of how to assess  
the value of new technologies.
Diana

Professor Diana Laurillard
London Knowledge Lab
Institute of Education
23-29 Emerald Street
London WC1N 3QS
020 7763 2162
07789111965
www.lkl.ac.uk
www.noe-kaleidoscope.org/pub/

On 7 Apr 2007, at 19:01, Rory McGreal wrote:

> Bob,
> Economists are still arguing about the cost-effectiveness and/or  
> social
> impact of introducing railroads to the US in the 19th Century. Was the
> horseless carriage really better than the horse and carriage? I  
> believe the
> point is that if the whole world is using technology, why would we  
> allow our
> schools to remain as technology-deprived anachronistic havens that our
> children have to escape from in order to experience the modern world.
> All the best.
> Rory
>
> -----Original Message-----
> From: discussion-bounces at g1to1.org [mailto:discussion- 
> bounces at g1to1.org] On
> Behalf Of Robert Kozma
> Sent: Saturday, April 07, 2007 10:02 AM
> To: 'Sherry Hsi'; 'Tak-Wai Chan'; discussion at g1to1.org
> Subject: RE: News of evaluating commercial educational software in  
> the US
>
> Sherry,
>
> A somewhat different tack is that it is not just the introduction of
> technology but the institution of other significant structural  
> changes that
> use technology in order for there to be a significant impact of  
> ICT.  In
> economic analyses, there was a five-year lag between the widespread
> introduction of ICT in the US economy and its impact on productivity.
> Despite the widespread use of ICT in the US in the early 90's, it  
> was not
> until the late 90's when technology was used to restructure  
> businesses, such
> as the retail sector (in what is sometimes unfortunately called "the
> Wal-Mart Effect"), that US productivity dramatically increased.
>
> Clearly the IES study looked at the use of ICT that was merely  
> pasted on
> existing education practices.  It will take significant ICT-based  
> structural
> reform in education (and probably more than the 5-year lag in  
> business)
> before we see significant results.
>
> Bob
>
>
> ____________________
>
> Robert B. Kozma, Ph.D.
> Emeritus Director and Principal Scientist
> Center for Technology in Learning
> SRI International
> 2151 Filbert St.
> San Francisco, CA 94123
> USA
>
> CTL Website: http://ctl.sri.com
> Personal Website: http://robertkozma.com
>
> Phone +1 415 292 2471
> Mobile +1 415 623 4340
> -----Original Message-----
> From: discussion-bounces at g1to1.org [mailto:discussion- 
> bounces at g1to1.org] On
> Behalf Of Sherry Hsi
> Sent: Saturday, April 07, 2007 6:21 AM
> To: Tak-Wai Chan; discussion at g1to1.org
> Subject: Re: News of evaluating commercial educational software in  
> the US
>
> Hi Tak-wai,
>
> Maybe we could write another response letter and post on our website.
> (I am afraid to suggest this because I can't take the lead on this.)
>
> Here are some things to note:
> --the length of the study was 1 year...pretty short if you expect
> teachers to integrate and make into effective practices.
> --the software they selected to evaluate were
> practice-oriented/drill-kill packages with little to no immediate
> student feedback nor student assessment (with exception to the
> Andersonian cognitive tutor-one of the five packages in the study.)
> --The teacher training was given by the developers in how to use the
> software, not how to use the software effectively for pedagogy or
> content learning. (But then again, you can't do much with limited
> software.)
> --The length of the intervention was on average 17 hours in one
> year--not a big allocation of time.
>
> One could use this study as evidence to show that large-scale
> adoption of poorly designed/shallow software with little professional
> development to teachers shows no impact, thus better kinds of
> designed software (e.g., like WISE, Molecular Workbench, GenScope,
> SimCalc, Geometer's Sketchpad,...), more investment into research on
> online assessment and  meaningful feedback for learners, more
> effective teacher professional development, and more allocation of
> time in the curriculum are all needed to make real improvements.
>
> Or, one could also use this study as ammunition to say all computer
> software and education technology in the classroom is evil and school
> districts are better off investing in more textbooks.
>
> Who wants to take the pen?
>
> -Sherry
>
>>  Hi,
>>
>> A colleague sent me this:
>> http://www.washingtonpost.com/wp-dyn/content/article/2007/04/04/ 
>> AR200704040
> 2
>> 715.html
>>
>> In the article, Elliot expressed his worry.
>>
>> Do you think G1:1 community can help something?
>>
>>
>> Regards,
>> Tak-Wai
>>
>>
>> _______________________________________________
>> connect with G1:1 community -> discussion at g1to1.org
>> subscribe/unsubscribe G1:1 discussion mailing list ->
>> http://mail.g1to1.org/mailman/listinfo/discussion
>
> _______________________________________________
> connect with G1:1 community -> discussion at g1to1.org
> subscribe/unsubscribe G1:1 discussion mailing list ->
> http://mail.g1to1.org/mailman/listinfo/discussion
>
> _______________________________________________
> connect with G1:1 community -> discussion at g1to1.org
> subscribe/unsubscribe G1:1 discussion mailing list ->
> http://mail.g1to1.org/mailman/listinfo/discussion
>
>
> __
>     This communication is intended for the use of the recipient to  
> whom it
>     is addressed, and may contain confidential, personal, and or  
> privileged
>     information. Please contact us immediately if you are not the  
> intended
>     recipient of this communication, and do not copy, distribute,  
> or take
>     action relying on it. Any communications received in error, or
>     subsequent reply, should be deleted or destroyed.
> ---
> _______________________________________________
> connect with G1:1 community -> discussion at g1to1.org
> subscribe/unsubscribe G1:1 discussion mailing list -> http:// 
> mail.g1to1.org/mailman/listinfo/discussion

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.g1to1.org/pipermail/discussion/attachments/20070407/6597df45/attachment.html>