Reips - Standards for Internet-Based, Psychologia użytkownika internetu

[ Pobierz całość w formacie PDF ]
REPRINT, cite as Reips, U.-D. (2002). Standards for Internet-based experimenting.
Experimental Psychology, 49 (4)
, 243-256.
Standards for Internet-Based
Experimenting
Ulf-Dietrich Reips
Experimental and Developmental Psychology, University of Zürich, Switzerland
Abstract.
This article summarizes expertise gleaned from the first years of Internet-based experimental research and
presents recommendations on: (1) ideal circumstances for conducting a study on the Internet; (2) what precautions have to
be undertaken in Web experimental design; (3) which techniques have proven useful in Web experimenting; (4) which
frequent errors and misconceptions need to be avoided; and (5) what should be reported. Procedures and solutions for
typical challenges in Web experimenting are discussed. Topics covered include randomization, recruitment of samples,
generalizability, dropout, experimental control, identity checks, multiple submissions, configuration errors, control of moti-
vational confounding, and pre-testing. Several techniques are explained, including “warm-up,” “high hurdle,” password
methods, “multiple site entry,” randomization, and the use of incentives. The article concludes by proposing sixteen stan-
dards for Internet-based experimenting.
Key words:
Internet-based experimenting, Web experiment, standards, experiment method, psychological experiment,
online research, Internet research, Internet science, methodology
Introduction
services other than the Web (such as e-mail, ICQ,
Telnet, Gopher, FTP, etc.) are rarely conducted.
Many of the issues discussed in this article apply to
experiments conducted via these services as well,
and even to nonexperimental Internet-based meth-
ods, such as Web surveying or nonreactive data col-
lection (for examples see Reips & Bosnjak, 2001).
Web experiments may be used to validate results
from field research and from laboratory experiments
(see Krantz & Dalal, 2000; Pohl, Bender, & Lach-
mann, 2002; Reips, 1997; Reips, Morger, & Meier,
2001), or they may be used for new investigations
that could only be feasibly accomplished in this me-
dium. For instance, in 2000, Klauer, Musch, and
Naumer published an article on belief bias in Psy-
chological Review that contains an example of a
study that reasonably could only be conducted as an
Internet-based experiment, because several thousand
participants were needed to obtain accurate estimates
of model parameters. Similarly, Birnbaum (2001) re-
cruited experts in decision making via a decision
making researchers’ e-mail list and sent these experts
to a Web page where many of them contradicted their
own theories in a Web experiment on choices be-
tween gambles. Experiments such as these would
prove impractical and burdensome if delivered in an-
other medium.
We are in the midst of an Internet revolution in ex-
perimental research. Beginning in the mid-nineties
of the last century, using the world-wide network for
experimental research became the method of choice
for a small number of pioneers. Their early work was
conducted soon after the invention of forms on Web
pages established user-server interaction (Musch &
Reips, 2000). This medium holds the promise to
achieve further methodological and procedural ad-
vantages for the experimental method and a pre-
viously unseen ease of data collection for scientists
and students.
Several terms are used synonymously for In-
ternet-based experiments:
Web experiment
,
on(-)line
experiment
,
Web-based experiment
,
Wo r l d Wi d e We b
(WWW) experiment
, and
Internet experiment
. Here
the term
Web experiment
will be used most often
because historically this term was used first and ex-
periments delivered via the Web are clearly the most
accessible and popular, as experiments using Internet
I would like to thank Tom Buchanan, William C.
Schmidt, Jochen Musch, Kevin O’Neil and an anonymous
reviewer for very helpful comments on earlier versions of
this article.
DOI: 10.1027//1618-3169.49.4.243
” 2002 Hogrefe & Huber Publishers
Experimental Psychology
2002; Vol. 49(4): 243Ð256
 244
Ulf-Dietrich Reips
In addition to the benefit of increased access to
participants, Internet-based experimenting always in-
cludes the possibility to use the programmed experi-
mental materials in a traditional laboratory setting
(Reips, 1997). In contrast, a laboratory experiment,
even if built with Internet software technologies, can-
not simply be turned into a Web experiment by con-
necting it to the Internet. Successful and appropriate
use of the Web medium requires careful crafting and
demands methodological, procedural, technical, and
ethical considerations to be taken into account!
While laboratory experiments can be built directly
with Internet software technologies, it seems wise to
conceptualize experiments as Web experiments
whenever possible, given their many advantages.
Sheer numbers, reduced cost, and accessibility of
specific participants are only a few of the Internet-
specific properties in Web experimenting that create
an environment that has been greeted with great en-
thusiasm by experimental psychologists.
low power. A solution for many of these problems
could be the implementation of experiments as Web
experiments. In total, about eighteen advantages
counter seven disadvantages of Web experimenting
(Reips, 2000, see Table 1).
Why Experimenters Relish Internet-Based
Experimenting
Speed, low cost, external validity, experimenting
around the clock, a high degree of automation of the
experiment (low maintenance, limited experimenter
effects), and a wider sample are reasons why the In-
ternet may be the setting of choice for an experiment.
Seventy percent of those who have conducted a Web
experiment intend to
certainly
use this method again
(the other 30%
maybe
). This result from a survey
of many of the early pioneers in Web experimenting
conducted by Musch and Reips (2000) is indirect
evidence that learning and using the methods of In-
ternet-based experimenting is certainly worthwhile.
Surveyed Web experimenters rated “large number of
participants” and “high statistical power” as the two
most important factors why they made the decision
to conduct a Web experiment.
The survey conducted by Musch and Reips is in
itself a good example that Internet-based research
may be the method of choice if a special subpopula-
tion is to be reached. Access to specific groups can
be achieved through Internet newsgroups (Hewson,
Laurent, & Vogel, 1996; Schmidt, 1997), Web site
guestbooks, chat forums, or topic-related mailing
lists. Eichstaedt (2002, Experiment 1) recruited per-
sons using either Macintosh or Windows operating
systems for his Web experiment via newsgroups de-
voted to the discussion of issues related to these op-
erating systems. The participants performed a Java-
based tachistoscopic word recognition task that in-
cluded words typically used in ads for these com-
puter systems. Word recognition was faster for words
pertaining to a participant’s computer system. Some
target groups may be easier to study via Internet,
because persons belonging to this group will only
reveal critical information under the protection of an-
onymity, for example drug dealers (Coomber, 1997),
or Ecstasy users (Rodgers, Buchanan, Scholey, Hef-
fernan, Ling, & Parrott, 2001).
When and When not to Conduct an
Experiment on the Internet?
Before standards for Internet-based experimenting
can be established, a few words should be devoted to
the question of criteria that should be used in decid-
ing the mode an experiment is best conducted in.
Implementing an Experiment: A General
Principle
Because many laboratory experiments are conducted
on computers anyway, nothing is lost when an ex-
periment is designed Web-ready: It can always also
be used in the laboratory. In
distributed Web experi-
menting
, local collaborators recruit and assist partici-
pants who all log onto the same Internet-based ex-
periment (Reips, 1999).
Solving Long-Standing Issues in
Experimental Research
The experimental method has a long and successful
tradition in psychological research. Nevertheless, the
method has been criticized, particularly in the late
1960s and early 1970s (e.g., Chapanis, 1970; Orne,
1962; Rosenthal, 1966; Rosenthal & Fode, 1973; Ro-
senthal & Rosnow, 1969; Smart, 1966). This criti-
cism is aimed in part at the validity of the method
and in part at improper aspects of its realization; for
instance experimenter effects, volunteer bias, and
When Not to Conduct an Experiment on
the Internet
Obviously, Web experiments are not the most suit-
able method for all research projects. For instance
whenever physiological parameters of participants
Experimental Psychology
2002; Vol. 49(4): 243Ð256
” 2002 Hogrefe & Huber Publishers
Standards for Internet-Based Experimenting
245
Tabl e 1 .
Web Experiments: Advantages, Disadvantages and Solutions (Adapted from Reips, 2000)
Advantages of Web Experiments
Disadvantages with Solutions
(1) Ease of access to a large number of demographi- (1) Possible multiple submissions Ð can be avoided
cally and culturally diverse participants (for an exam- or controlled by collecting personal identification
ple of a study conducted in three languages with 440 items, by checking internal consistency as well as
women from more than nine countries see in this vol- date and time consistency of answers (Schmidt,
ume Bohner, Danner, Siebler, & Samson, 2002);
1997), and by using techniques such as
sub-sampling,
participant pools
, or handing out
passwords
(Reips,
(2) . . . as well as to rare and specific participant pop-
1999, 2000, 2000b). There is evidence that multiple
submissions are rare in Web experiments (Reips,
(3) Better generalizability of findings to the general
1997).
population (Horswill & Coster, 2001; Reips, 1995).
(2) Generally, experimental control may be an issue
(4) Generalizability of findings to more settings and
in some experimental designs, but is less of an issue
situations (because of high external validity, e.g.,
when using between-subjects designs with random dis-
Laugwitz, 2001).
tribution of participants to experimental conditions.
(5) Avoidance of time constraints.
(3) Self-selection can be controlled by using the
multiple site entry technique
.
(6) Avoidance of organizational problems, such as
scheduling difficulties, as thousands of participants
(4) Dropout is always an issue in Web experiments.
may participate simultaneously.
However, dropout can be turned into a detection de-
vice for motivational confounding. Also, dropout can
be reduced by implementing a number of measures,
(8) Ease of acquisition of just the optimal number of
such as promising immediate feedback, giving finan-
participants for achieving high statistical power while
cial incentives, and by personalization (Frick,
being able to draw meaningful conclusions from the
Bächtiger, & Reips, 2001).
experiment.
(5) The reduced or absent interaction with partici-
(9) Detectability of motivational confounding.
pants during a Web experiment creates problems, if
instructions are misunderstood. Possible solutions are
(10) Reduction of experimenter effects.
pretests of the materials and providing the partici-
(11) Reduction of demand characteristics.
pants with the opportunity for giving feedback.
(12) Cost savings of laboratory space, personnel
(6) The comparative basis for the Web experiment
hours, equipment, administration.
method is relatively low. This will change.
(13) Greater openness of the research process (in-
(7) External validity of Web experiments may be lim-
creases replicability).
ited by their dependence on computers and networks.
Also, many studies cannot be done on the Web. How-
(14) Access to the number of nonparticipants.
ever, where comparable, results from Web and labora-
(15) Ease of comparing results with results from a lo-
tory studies are often identical (Krantz & Dalal,
cally tested sample.
2000).
(16) Greater external validity through greater techni-
cal variance.
(17) Ease of access for participants (bringing the ex-
periment to the participant instead of the opposite).
(18) Public control of ethical standards.
are to be measured directly, specialized hardware is
required, or when a tightly controlled setting is im-
portant, then laboratory experiment administration is
still required.
A further basic limitation lies in Web experi-
ments’ dependency on computers and networks hav-
ing psychological, technical, and methodological im-
plications. Psychologically, participants at computers
will likely be subject to self-actualization and other
influences in computer-mediated communication
(e.g., Bargh, McKenna, & Fitzsimons, 2002; Joinson,
2001). Technically, more variance is introduced in
the data when collected on the Internet than in the
laboratory, because of varying network connection
” 2002 Hogrefe & Huber Publishers
Experimental Psychology
2002; Vol. 49(4): 243Ð256
ulations (Schmidt, 1997).
(7) Highly voluntary participation.
246
Ulf-Dietrich Reips
speed, varying computer speed, multiple software
running in parallel, etc. (Reips, 1997, 2000).
On first view one may think that the Internet al-
lows for easy setup of intercultural studies, and it is
certainly possible to reach people born into a wide
range of cultures. However, it is a widespread misun-
derstanding that Internet-based cultural research
would somehow render unnecessary the use of many
techniques that have been developed by cross-cul-
tural psychologists (such as translation Ð back
translation, use of the studied cultures’ languages,
and extensive communication and pretests with peo-
ple from the cultures that are examined). Issues of
access, self-selection, and sampling need to be re-
solved. In many cultures, English-speaking computer
users are certainly not representative of the general
population. Nevertheless, these people may be very
useful in bridging between cultures, for instance, in
cooperative studies based on distributed Web experi-
menting.
Finally, for ethical reasons, many experiments
cannot be conducted that require an immediate de-
briefing and adjunctive procedures through direct
contact whenever a participant terminates participa-
tion.
In the following section we will turn to central
issues and resulting proposals for standards that
specifically apply to Internet-based experimenting.
the user’s computer configuration however (Schmidt,
2000). Combinations of
server-side
and
client-side
processing methods are possible; they can be used to
estimate technical error variance by comparison of
measurements.
General experimental techniques apply to Web
experimentation as well. For example, randomized
distribution of participants to experimental condi-
tions is a measure against confounding and helps
avoiding order effects. Consequently, in every Web
experiment at least one randomization technique
should be used. In order of reliability, these are
roughly: (1) CGI or other server-side solutions, (2)
client-side Java, (3) Javascript, and (4) “the birthday
technique” (participants pick their experimental con-
ditions by mouse-clicking on their birthday or birth-
day month; Birnbaum, 2000; Reips, 2002b).
Generating an Experiment
For several years, Web experimenters created their
experimental materials and procedures “by hand.”
With enough knowledge about HTML and the ser-
vices and structures available on the Internet con-
ducting a Web experiment was only moderately com-
plicated. However, many researchers hesitate before
acquiring new technical skills. Fortunately, several
recently developed applications considerably ease the
development of a Web experiment. For within-sub-
jects designs, Birnbaum (2000) developed
FactorWiz,
1
a Web page that creates Web pages with
items combined according to previously defined fac-
torial designs. WEXTOR,
2
by Reips and Neuhaus
(2002), a Web-based tool for generating and visualiz-
ing experimental designs and procedures, guides the
user through a ten-step program of designing an ex-
periment that may include between-subjects, within-
subjects, and quasi-experimental (natural) factors.
WEXTOR automatically creates the experiments in
such a way that certain methodological requirements
of Internet-based experimentation are met (for exam-
ple, nonobvious file naming [Reips, 2002a] is imple-
mented for experimental materials and conditions,
and a session ID is generated that helps identify sub-
missions by the same participant).
Checks and Solutions for
Methodological Challenges in Web
Studies
The following section contains methodological and
technical procedures that will reduce or alleviate is-
sues that are rooted within the very nature of In-
ternet-based studies.
Web Experiment Implementation
Data collection techniques on the Internet can be po-
larized into
server-side
and
client-side
processing.
Server-side methods (a Web server, often in combi-
nation with a database application, serves Web pages
that can be dynamically created depending on a us-
er’s input, Schmidt, 2000) are less prone to platform-
dependent issues, because dynamic procedures are
performed on the server so that they are not subject
to technical variance. Client-side methods use the
processing power of the participants’ computers.
Therefore, time measurements do not contain error
from network traffic and problems with server avail-
ability are less likely. Such measurements do rely on
Recruitment
A Web experiment can be announced as part of the
collection of Web studies by the American Psycho-
1
factorWiz.htm
2
Experimental Psychology
2002; Vol. 49(4): 243Ð256
” 2002 Hogrefe & Huber Publishers
 Standards for Internet-Based Experimenting
247
logical Society.
3
This Web site is maintained by John
Krantz. A virtual laboratory for Web experiments is
the Web Experimental Psychology Lab
4
. This Web
site is visited by about 4500 potential participants
a month (Reips, 2001). Internet-based experiments
should always be linked to the
web experiment list
,
5
a Web site that is intended to serve as an archive of
links and descriptions of as many experiments con-
ducted on the Internet as possible. Other ways of re-
cruiting participants that may be combined with link-
ing to experiment Web sites is the use of online pan-
els, newsgroups, search engines, banners, and e-mail
lists. Also, participants for Web experiments can be
recruited offline (e.g., Bamert, 2002; Eichstaedt,
2002; Pohl et al., 2002; Reips et al., 2001; Rup-
pertsberg, Givaty, Van Veen, & Bülthoff, 2001).
potential through self-selection can be calculated
(Reips, 2000). If self-selection is not found to be a
problem, results from psychological experiments
should be the same no matter where participants are
recruited, and should therefore show high generaliza-
bility.
Dependence on Technology
Limited generalizability of results from Internet-
based research may also arise due to dependence on
computers and networking technology. Web-based
experimenting can be seen as a form of computer-
mediated communication. Hence, differences found
in comparisons between behaviors in computer-me-
diated situations and
face-to-face
situations (see, for
example, Buchanan, 2002; Kiesler & Sproull, 1986;
Postmes, Spears, Sakhel & DeGroot, 2001) need to
be taken into account when results from online re-
search are interpreted.
Generalizability
Self Selection
In many areas of Psychology self-selection is not
considered much of a problem in research because
theory testing is the underlying model of epistemol-
ogy and people are not considered to vary much on
the essential criteria, for example, in research on cog-
nition and perception. However, at least in research
more socially oriented, self-selection may interfere
with the aim of the study at hand and limit its gene-
ralizability. The presence and impact of self-selec-
tion in an Internet-based study can be tested by using
the
multiple site entry technique
(Reips, 2000,
2002b). Via log file analysis it is possible to deter-
mine a Web experiment’s degree of appeal for parti-
cipation for each of the samples associated with re-
ferring Web sites.
The multiple site entry technique can be used in
any Internet-based study (for a recent example of
longitudinal trauma survey research implementing
this technique see Hiskey & Troop, 2002). Several
links to the study are placed on Web sites, in discus-
sion groups, or other Internet forums that are likely
to attract different types of participants. Placing iden-
tifying information in the published URLs and ana-
lyzing different referrer information in the HTTP
protocol can be used to identify these sources (see
Schmidt, 2000). Later the data sets that were col-
lected are compared for differences in relative degree
of appeal (measured via dropout), demographic data,
central results, and data quality, as a function of re-
ferring location. Consequently, an estimate of biasing
Advantages
Apart from the challenges to generalizability men-
tioned above, Internet-based experimenting has three
major advantages in this respect:
(1) Increased generalizability through nonlocal sam-
ples with a wider distribution of demographic
characteristics (for a comparison of data from
several Web experiments see Krantz & Dalal,
2000, and Reips, 2001).
(2) “Ecological” validity: “the experiment comes to
the participant, not vice versa” (Reips, 1995,
1997). Participants in Web experiments often re-
main in familiar settings (e.g., at their computer
at home or at work) while they take part in an
Internet-based experiment. Thus, any effects can-
not be attributed to being in an unfamiliar setting.
(3) The high degree of voluntariness Ð because there
are fewer constraints on the decisions to partici-
pate and to continue participation, the behaviors
observed in Internet-based experiments may be
more authentic and therefore can be generalized
to a larger set of situations (Reips, 1997, 2000).
A high degree of voluntariness is a
trade-off
for po-
tential effects of self-selection. Voluntariness refers
to the voluntary motivational nature of a person’s
participation, during the initial decision to participate
and during the course of the experiment session. It
is influenced by external factors, for example, the
setting, the experimenter, and institutional regula-
tions. If participants in an experiment subjectively
feel that they are participating entirely voluntarily as
3
4
WebExpPsyLab.html
5
” 2002 Hogrefe & Huber Publishers
Experimental Psychology
2002; Vol. 49(4): 243Ð256
  [ Pobierz całość w formacie PDF ]

  • zanotowane.pl
  • doc.pisz.pl
  • pdf.pisz.pl
  • tejsza.htw.pl

    •