News of evaluating commercial educational software in the US
Deborah Tatar
tatar at cs.vt.edu
Sun Apr 8 05:03:40 CST 2007
The form of our response is really quite important here because we
have to be very simple and clear in our message. I see two
elements to this:
(1) I believe that, while the 1:1 community can and should frame a
response, the most important issue here is not 1:1 computing, or
technology in general, or even experimental validity, but narrowly
conceived outcome measures. There was a recent article in
Educational Researcher claiming that learning has only taken place
when there are changes in long term memory, with an unstated and
unbacked assumption that (a) we can in fact detect such changes using
simple tests and (b) when no such changes are detected through these
simple measures nothing of importance has taken place. Claims like
this unravel the cognitive revolution and devalue achievements in the
learning sciences of the last thirty years.
Of course, the averaging of the different programs under evaluation
is disgraceful, but the battle for equity that Elliot is talking
about will not be won through technology alone or primarily, but
through keeping our eyes on the nature of learning. We believe in
1:1 computing, not because of a blind faith in technology nor because
we think that all 1:1 technologies are good, but because it provides
many pragmatic and pedagogically sound avenues for scaffolding
learning in all its rich complexity.
If scientifically-based evaluation is undertaken, it must be subject
to the same validity considerations as any other scientific
enterprise: not one "gold standard" but replication with variation
across settings, persons, treatments, and constructs.
(2) This leads us to the second point: money. What we have in the
United States is an economically created constriction of competition
on scientific studies. If people want scientifically based research,
they need to pay for it. Millions or billions, they need to pay to
do it right, and doing it right means replication, replication,
replication---with variation.
I note that we have recently submitted the results of our first year
Scaling Up SimCalc study to Science. We can't really talk about the
results until they come out, but they are very nice. However, even
these do not stand alone. These results are built on many previous
studies in different teaching venues with different technological
platforms, variations on curriculum and assessment constructs. They
are also built on a highly successful pilot and a conceptualization
of Scaling Up that includes accounting for variation in outcome as
well as overall success.
In any case, we definitely do not want to create a situation in
which we box ourselves in methodologically or otherwise tie our hands
in non-essential ways.
Cheers,
D.
Deborah Tatar, Ph. D.
Associate Professor of Computer Science and, by courtesy, Psychology
2202 Kraft Dr., Room 123
Blacksburg, VA 2401
(540)231-8457 (office)
(540)231-9218 (fax)
tatar at cs.vt.edu
On Apr 7, 2007, at 2:01 PM, Rory McGreal wrote:
> Bob,
> Economists are still arguing about the cost-effectiveness and/or
> social
> impact of introducing railroads to the US in the 19th Century. Was the
> horseless carriage really better than the horse and carriage? I
> believe the
> point is that if the whole world is using technology, why would we
> allow our
> schools to remain as technology-deprived anachronistic havens that our
> children have to escape from in order to experience the modern world.
> All the best.
> Rory
>
> -----Original Message-----
> From: discussion-bounces at g1to1.org [mailto:discussion-
> bounces at g1to1.org] On
> Behalf Of Robert Kozma
> Sent: Saturday, April 07, 2007 10:02 AM
> To: 'Sherry Hsi'; 'Tak-Wai Chan'; discussion at g1to1.org
> Subject: RE: News of evaluating commercial educational software in
> the US
>
> Sherry,
>
> A somewhat different tack is that it is not just the introduction of
> technology but the institution of other significant structural
> changes that
> use technology in order for there to be a significant impact of
> ICT. In
> economic analyses, there was a five-year lag between the widespread
> introduction of ICT in the US economy and its impact on productivity.
> Despite the widespread use of ICT in the US in the early 90's, it
> was not
> until the late 90's when technology was used to restructure
> businesses, such
> as the retail sector (in what is sometimes unfortunately called "the
> Wal-Mart Effect"), that US productivity dramatically increased.
>
> Clearly the IES study looked at the use of ICT that was merely
> pasted on
> existing education practices. It will take significant ICT-based
> structural
> reform in education (and probably more than the 5-year lag in
> business)
> before we see significant results.
>
> Bob
>
>
> ____________________
>
> Robert B. Kozma, Ph.D.
> Emeritus Director and Principal Scientist
> Center for Technology in Learning
> SRI International
> 2151 Filbert St.
> San Francisco, CA 94123
> USA
>
> CTL Website: http://ctl.sri.com
> Personal Website: http://robertkozma.com
>
> Phone +1 415 292 2471
> Mobile +1 415 623 4340
> -----Original Message-----
> From: discussion-bounces at g1to1.org [mailto:discussion-
> bounces at g1to1.org] On
> Behalf Of Sherry Hsi
> Sent: Saturday, April 07, 2007 6:21 AM
> To: Tak-Wai Chan; discussion at g1to1.org
> Subject: Re: News of evaluating commercial educational software in
> the US
>
> Hi Tak-wai,
>
> Maybe we could write another response letter and post on our website.
> (I am afraid to suggest this because I can't take the lead on this.)
>
> Here are some things to note:
> --the length of the study was 1 year...pretty short if you expect
> teachers to integrate and make into effective practices.
> --the software they selected to evaluate were
> practice-oriented/drill-kill packages with little to no immediate
> student feedback nor student assessment (with exception to the
> Andersonian cognitive tutor-one of the five packages in the study.)
> --The teacher training was given by the developers in how to use the
> software, not how to use the software effectively for pedagogy or
> content learning. (But then again, you can't do much with limited
> software.)
> --The length of the intervention was on average 17 hours in one
> year--not a big allocation of time.
>
> One could use this study as evidence to show that large-scale
> adoption of poorly designed/shallow software with little professional
> development to teachers shows no impact, thus better kinds of
> designed software (e.g., like WISE, Molecular Workbench, GenScope,
> SimCalc, Geometer's Sketchpad,...), more investment into research on
> online assessment and meaningful feedback for learners, more
> effective teacher professional development, and more allocation of
> time in the curriculum are all needed to make real improvements.
>
> Or, one could also use this study as ammunition to say all computer
> software and education technology in the classroom is evil and school
> districts are better off investing in more textbooks.
>
> Who wants to take the pen?
>
> -Sherry
>
>> Hi,
>>
>> A colleague sent me this:
>> http://www.washingtonpost.com/wp-dyn/content/article/2007/04/04/
>> AR200704040
> 2
>> 715.html
>>
>> In the article, Elliot expressed his worry.
>>
>> Do you think G1:1 community can help something?
>>
>>
>> Regards,
>> Tak-Wai
>>
>>
>> _______________________________________________
>> connect with G1:1 community -> discussion at g1to1.org
>> subscribe/unsubscribe G1:1 discussion mailing list ->
>> http://mail.g1to1.org/mailman/listinfo/discussion
>
> _______________________________________________
> connect with G1:1 community -> discussion at g1to1.org
> subscribe/unsubscribe G1:1 discussion mailing list ->
> http://mail.g1to1.org/mailman/listinfo/discussion
>
> _______________________________________________
> connect with G1:1 community -> discussion at g1to1.org
> subscribe/unsubscribe G1:1 discussion mailing list ->
> http://mail.g1to1.org/mailman/listinfo/discussion
>
>
> __
> This communication is intended for the use of the recipient to
> whom it
> is addressed, and may contain confidential, personal, and or
> privileged
> information. Please contact us immediately if you are not the
> intended
> recipient of this communication, and do not copy, distribute,
> or take
> action relying on it. Any communications received in error, or
> subsequent reply, should be deleted or destroyed.
> ---
> _______________________________________________
> connect with G1:1 community -> discussion at g1to1.org
> subscribe/unsubscribe G1:1 discussion mailing list -> http://
> mail.g1to1.org/mailman/listinfo/discussion
More information about the discussion
mailing list