Experimental Research Designs

In the previous section, we have discussed at length about the type and nature of experimental studies. This section is entirely devoted to the research designs adopted for carrying out these experimental studies. Experimental research design is an elegant way to find out how well a particular program achieves its goals.

In this section, we discuss a few experimental designs that vary widely to control contamination of the relationship between independent and dependent variables.

Experimental Research Designs

Three broad categories of designs are presented here:

  • Pre-experimental designs
  • True experimental designs, and
  • Quasi-experimental designs.

Pre-experimental Designs

Pre-experimental designs are those designs that do not have any comparison groups. Even if they have so, they fail to meet the requirement of random assignment of these groups.

We will discuss a few pre-experimental designs;

  • Posttest-only design;
  • Pretest-Posttest design; and
  • Static-group comparison.

Posttest-only design

The posttest-only design, also called the one-shot case study design, is the weakest of all designs that fail to control the various threats adequately to internal validity. These designs are most useful for collecting descriptive information or doing small case studies of a particular situation.

It is diagramed as follows:

Experimental Group:XO

The design includes the following steps:

  • Select the subjects
  • Select, the experimental environment
  • Administer the experimental stimulus X
  • Conduct the posttest with measurement O

As we can see, in this design, a program intervention (X) has been introduced, and sometimes after its introduction, a measurement observation (O) is made.

Since there is neither a control group nor a pre­test measurement, there is no possibility of comparing the measurement O with any other measurement.

All that the measurement O can do is that it provides descriptive information.

By environment, we mean whether the experiment is conducted in the field or a laboratory setting or any other environment. The threats to the validity of history, maturation, selection, and experimental mortality cannot be controlled.

The lack of a pretest and a control group makes this design inadequate for establishing causality.

Example

UNICEF, Bangladesh introduced a glucose injection campaign in certain IDD prone areas of Bangladesh.

This is our intervention (X). A year later, a measurement was taken from those who received the injection resulting in the observation ‘O.’

Note that we have no means of controlling the extraneous influences. There should be some measure of what would happen when test units were not exposed to X to compare with the measure when subjects were exposed to X.

Pretest-Posttest Design

This design includes a single experimental group, and it is called a pretest­posttest design with no control group.

Since this design lacks a control group with which to measure extraneous variation, it can be used only when the experimenter can assume that extraneous variation is minimal so that virtually all recorded change in pre and posttest measurements are caused by the intervention (X), the test stimulus.

The design includes the following steps:

  • Select the subjects
  • Select the experimental environment
  • Conduct the pretest with measurement O1
  • Administer the experimental stimulus X
  • Conduct the posttest with measurement O2

Example

In the measurement of glucose level, the blood sample is taken, and the glucose level determined. This is our pretest observation, O1.

Two hours after administering glucose or breakfast, a second measurement was taken. This is our post-test measurement O2. The design is diagrammed as follows:

Experimental Group:O1XO2

Since total variation in pre and posttest scores is being attributed to the causal factor, the formula for this cause is

ΔExpt = O2 – O1

If the experimenter’s assumption is incorrect and the extraneous factors cause a change in the pre and posttest scores, then the experimenter does not know how much of the change in the dependent variable is due to the intervention (X) and how much to uncontrolled factors.

We can address this problem by repeating the experiment and adding a control group with the pre and posttest but without intervention.

This design is subject to several threats to validity-history, testing, maturation, and instrumentation.

Static-Group Comparison

In the static group design, subjects are identified as either an experimental group or a control group.

The experimental group is measured after it has been exposed to the experimental treatment. Look at the following diagram that portrays the design:

static group comparison

Unlike the other two designs, this design adds a control group. The experimental group receives a program intervention (X) followed by a measurement observation (O1). This measurement observation is then compared against a second observation (O2) from a control group that did not receive the program intervention. The results of the static group design are then computed as a difference between the two observations as follows:

ΔExpt = O2 – O1

The broken line ( – – – – – ) is a non-random line, which indicates that no random process was followed to create the two groups.

The addition of a comparison group makes a substantial improvement over the two designs. Its chief weakness is that there is no way to be certain that the two groups are equivalent and that no random process has been followed in creating the two groups.

True Experimental Designs

The major deficiency of the pre-experimental designs is that they fail to provide comparison groups that are truly equivalent. The way to achieve equivalence is through matching and random assignment.

We describe two such designs that fall under this category. These are;

  • Pretest-posttest control group design.
  • Posttest- only control group design.

Pretest-posttest control group design

It is a design in which all subjects are randomly assigned (RA) from a single population to the experimental group and the control group. Both the experimental and the control groups receive an initial measurement observation (O1 and O3 in the accompanying diagram).

The experimental group then receives the program intervention (X), but the control group does not receive this intervention.

Finally, the second set of measurement observations ( O2 and O4 ) are made for both groups.

This design assumes that the effect of all extraneous variables will be the same on both the experimental and the control groups. The design is portrayed as follows:

RAExperimental GroupO1XO2
RAControl GroupO3O4

We enumerate the steps in conducting a pretest-posttest control group design as follows:

Experimental GroupControl Group
1. Select subjects1. Select subject
2. Select an experimental environment2. Select an experimental environment
3. Take the pretest measurement (003. Take the pretest measurement (O3)
4. Administer intervention (X)4. Administer no intervention.
5. Take the posttest measurement (02)5. Take the posttest measurement (04)

To assess the causal effect of the experimental treatment, we proceed as follows:

ΔExpt = (O2 – O1) –  (O4 – O3)

We would expect that, since the experimental group received a special program intervention, O2 would be greater than O4. Also, since both the experimental and control cases were randomly assigned, we would expect that O1 would be equal to O3 on key variables.

Under this assumption, AExpt will be positive, and the amount contributing to this difference will be the true causal effect of the test stimulus. The control difference will equal the experimental difference in the cases in which the causal effect of the test stimulus is zero.

Because of the random assignment of subjects in the groups, this design suffers very little from the problems of validity threats.

Maturation, testing, and regression effects are handled well because one would expect them to be felt equally in experimental and control groups.

Mortality, however, can be a problem if there are different dropout rates in the study groups. The random assignment process well tackles the selection problem.

Posttest- only control group design.

In this design, the pretest measurements of both groups are omitted. Pretests are well established in classical research designs but are not necessary when the randomization process is followed.

The design is diagrammed as follows:

RAExperimental GroupXO1
RAControl GroupO2

The experimental effect is measured by the difference between O1 and O2:

ΔExpt = O1 – O2

The simplicity of this design makes it more attractive than the pretest­posttest control group design.

Internal validity threats from history, maturation, selection, and statistical regression are adequately controlled by the random assignment.

Since the subjects are measured only once, the threats of testing and instrumentations are reduced, but differential mortality rates between experimental and control groups continue to be a potential problem.

Solmon four-group design

The Solmon four-group design is a way of avoiding some of the difficulties encountered in the pretest-posttest design.

This design contains two extra control groups, which serve to reduce the influence of confounding variables and allow the researcher to test whether the pretest itself affects the subjects.

While much more complex to set up and analyze, the design type combats many of the internal validity issues that can plague research. It allows the researcher to exert complete control over the variables and allows the researcher to check that the pretest did not influence the results.

As we will observe from the diagram below, the Solmon four-group test is a standard pretest-posttest design, and the posttest only controls group design.

The various combinations of tested and untested groups with treatment and control groups allow the researcher to ensure that confounding variables and extraneous factors have not influenced the results.

(A)RAExperimental groupO1XO2
(B)RAControl groupO3O4
(C)RAExperimental groupXO5
(D)RAControl groupO6

The first two groups of the Solomon four-group design are designed and interpreted in the same way as in the pretest-post-test design and provide the same checks upon randomization.

The comparison between the posttest results of groups C and D allows the researcher to determine if the actual act of pretesting influenced the results.

If the difference between the posttest results of Groups C and D is different from the Groups A and B difference, then the researcher can assume that the pretest has had some effect on the results.

The comparison between the Group B Pretest and the Group D posttest allows the researcher to establish if any external factors have caused a temporal distortion.

For example, it shows if anything else could have caused the results shown and is a check upon causality.

The Comparison between Group A posttest and the Group C posttest allows the researcher to determine the effect that the pretest has had upon the treatment. If the posttest results for these two groups differ, then the pretest has had some effect on the treatment, and the experiment is flawed.

The comparison between the Group B posttest and the Group D posttest shows whether the pretest itself has affected behavior, independently of the treatment.

If the results are significantly different, then the act of pre­testing has influenced the overall results and needs refinement.

Quasi-Experimental Designs

Quasi-experimental designs are those that do not satisfy the strict requirements of the experiment.

In such designs, subjects to be observed are not randomly assigned to different groups to measure the outcomes, as in a randomized experiment, but grouped according to a characteristic that they already possess.

Some authors distinguish between a natural experiment and a quasi-experiment. The difference is that in a quasi-experiment, the causal factor is manipulated by the researcher, while in a natural experiment, the causal factor varies naturally.

The major disadvantage of quasi-experiments is that they are more open to confounding variables.

We have already pointed out that the best designs are those that control relevant outside effects and lead to valid inferences about the effects of the program.

Unlike an experimental design, which protects against just about all possible threats to internal validity, quasi-experimental designs generally leave one or several of them uncontrolled.

In reality, it is simply impossible to meet the random assignment criteria of true experimental design.

Alongside this, researchers want to avoid the problems of validity threats associated with the pre-experimental designs.

The use of quasi-experimental designs in these circumstances offers a reasonable compromise, which does not have the restriction of random assignment.

It is, in this sense, inferior to a true experimental design but is usually superior to pre-experimental designs. We discuss a few quasi-experimental designs in this section. These are

  • Non-equivalent control group design;
  • Time-series design;
  • Separate sample pretest-posttest design.

Non-equivalent control group design

This is strong and probably the widely used quasi-experimental design. Here there is no random assignment to program and control as there would be in a true experiment.

There are two variants of this design. One is the so-called intact equivalent design, and the other is the self-selected experimental design.

In intact equivalent design, the membership in both groups is naturally assembled.

For example, we may use different classes in a school, different hospital wards, or customers from similar stores. A major issue here is how to make the comparison group as similar to the experimental group as possible.

Matching procedures are sometimes resorted to pairing up members of the experimental and control groups on available measures at the start of the program.

Afterward, when one group has been exposed to the benefits of the program and the other group has not, the difference between them should be attributed to the program intervention.

But matching for obvious reasons is much less satisfactory than randomized assignment on several counts. Not the least that we often cannot define the characteristics on which people should be matched.

That is, we do not know which characteristics will affect whether the person benefits from the program or not.

The second variant, the self-selected experimental design, is a weaker design because it encounters a problem in selecting a comparison group.

People who choose to enter a program are likely to be different from those who do not, and the prior differences (in interest, attitude, desire, norms, values, initiative, etc.) make post-program comparisons between ‘served’ and ‘un-served’ groups risky.

Self-selection problems can sometimes be overcome if the subjects of both experimental and control groups are selected from volunteers. Such selection can be thought of as a true experiment if the volunteers are randomly assigned to either group. The design is diagrammed as follows:

A comparison of pretest results (O1– O3) is one indicator of the degree of equivalence between the experimental and the control groups. If the pretest results are significantly different, there is a real question about the groups’ comparability.

On the other hand, if the pretest observations are similar between groups, there is more reason to believe that the internal validity of the experiment is good.

The non-equivalent control group design is particularly useful in the evaluation of training programs.

The design, however, is threatened by selection effect as well as by the interaction of selection with other factors, and possibly, if groups were selected for extreme scores, the regression effect is an additional problem.

Time-series design

The time-series design is one of the most attractive quasi-experiments. It involves a series of measurements at periodic intervals before the program begins and continuing measurements after the program ends.

It thus becomes possible to determine whether the measures immediately before and after the program are a continuation of earlier patterns or whether they mark a decisive change.

A time-series design is similar to a pre-experimental design except that it has the advantage of repeated measurement observations before and after the program intervention (X).

Examine the following diagram:

Experimental Group:O1O2O3XO4O5O6

Suppose we find that there is no difference between O1, O2, and O3, but then a sudden increase occurs between O3 and O4, which subsequently continues to O5. We can conclude with some degree of confidence that the sudden increase was probably due to the effect of the program intervention (X).

a time series design showing the effect of intervention

Figure: A time-series design showing the effect of the intervention

three-time series displaying the varying significance

Figure: Three-time series displaying the varying significance

The figure is designed to illustrate the impact of intervention more vividly. The three cases shown in these figures will have different significance. Neither the top nor the middle one tells us anything about the effect of the intervention.

The figure in the top displays a monotonous or continuous increase since the beginning of the study. In other words, the intervention does not show any impact on the program.

The same interpretation is true for the figure in the middle. Only the change from B to A in the bottom figure can be attributed to the program effect. Note that this figure is similar.

The time-series design probably protects against almost all threats to validity except history and probably instrumentation threats.

It allows for a more detailed analysis of data and programs impact than the pretest-posttest design because it gives information on trends before and after program intervention. The time-series design helps the researcher avoid making a mistaken conclusion.

A time-series design is particularly an appropriate design to use when a researcher can make multiple measurement observations before and after a program intervention (Fisher et al. 1998: 87).

Once again, we note that the time series design is recommended when the use of a control group is either impossible or not feasible for practical reasons.

Separate sample pretest-posttest design

This design is most applicable when we cannot know when and whom to introduce the treatment, but we can decide when and whom to measure.

Essentially, it involves doing a baseline pretest (O1) with a randomly selected sample from a study population.

Subsequently, a program intervention (X) is introduced, and then a posttest measurement (O2) is made using a second randomly selected sample from the same study population. The design is displayed as follows:

RA: Pretest GroupO1(X)
RA: Posttest GroupXO2

The bracketed treatment (X) is irrelevant to the purpose of the study but is shown to suggest that the experimenter cannot control the treatment.

This is not a strong design because several threats to internal validity stand on its way. The history effect can confound the results by repeating the study at other times in other settings.

In contrast, it is considered superior to true experiments in external validity.

Its strength results from its being a field experiment in which the samples are usually drawn from the population to which we wish to generalize our findings.

Ex-post facto design

Sometimes it becomes difficult to divide the study population into two clear and similar groups.

This may be the case where the entire society consisting of different varieties of people and conditions are involved. It may be necessary to study the entire historical background of a country.

For instance, if a researcher is interested in studying the causes of revolution, which is already in motion, he will not be able to objectively study the exact situation that was prevailing before the revolution started in the country.

He has to depend on the historical background of the country, and this will be studied through what is called the ex-post-facto study design.

In this particular instance, the investigator should select two particular countries-one in which revolution has taken place and the other, where it has not. The countries, otherwise, should broadly be similar.

Then, through a comparative study of the conditions of the two countries, the researcher may be able to find out the causes of the prevailing revolution.

In the ex-post-facto study, the past is studied through the present. But in other types of studies, we try to prognosticate the future from the present.

The most obvious limitation of the ex-post-facto study is the difficulty of finding two similar groups that are comparable. It is also difficult to find an objective criterion of comparison.

Secondly, it is not possible to create artificial conditions or to have controlled conditions for study.

Thirdly, it is not possible to employ a pretest-posttest design in such a study.

Related Posts ⁄