Research Design
Part 1:
Exploring Structure and Standards
by Terri McCartney
View comments on this article
Last month we looked at the steps involved in doing research and
exemplified those steps by exploring two Astrosignatures for psychic
ability set forth by Gaston Mascarenas and Mitchell Gibson. This month
we begin to explore research design. The design is the glue that holds
the research project together. It structures steps three, four and five
of the research process. That is, it structures method, data collection
and analysis. Importantly, the research design addresses the important
issues of validity, transferability, reliability and confirmability
which are the criteria we use to judge the value and usefulness of our
research endeavors.
There are three general types of research designs:
- Experimental
- Quasi-experimental
- Non-experimental
The experimental design is commonly referred to as the “scientific
method” and is characterized by the random assignment of participants
into equivalent test/treatment and control/comparison groups. Random
assignment is a means for removing bias by controlling for factors that
are not examined directly in the study. Quasi-experimental studies
adhere to less random assignment but do use test and control groups.
Non-experimental studies do not use control groups. The experimental and
quasi-experimental designs commonly employ quantitative methods of
analysis while most qualitative methods fall into the category of
non-experimental designs. Instead of offering a statistical analysis,
non-experimental designs offer a detailed descriptive analysis.
Background
Historically, astrological researchers have attempted to adhere to
the experimental design and its Gold Standard, the rigorous scientific
method. Unfortunately, the results have often been a source of despair
for the astrological community. It became evident that a research
approach that assumed a cause-effect relationship—if A then B and if not
A then not B—wasn’t suitable for capturing the multidimensional and
interactive reality astrology describes. The Scientific Experimental
design limits reality to physical objects and linear relationships.
Astrological realities are not limited this way—they encompass abstract
concepts and non-linear, multidimensional relationships. And so it is
felt that astrology can’t be tested quantitatively, because there is no
single cause-effect relationship. And there has long been debate among
quantitative and qualitative researchers—this debate is not unique to
astrology. Yet, what astrological researchers need to be discussing are
research designs that are appropriate to investigating astrology. And
I’m an advocate for designs which integrate both qualitative and
quantitative methods.
We know the basics of doing qualitative research. While astrology may
not have faired well applying the scientific method to its research
endeavors, as a discipline we have a huge body of qualitative data.
What’s qualitative data? All that richly detailed and descriptive
material found in astrological publications that is a natural result of
applying astrology. We’ve been doing our qualitative research all
along—we’ve been out in the field, thick in the trenches of life with
our clients and burning the midnight oil as we perused case studies,
theorizing about meanings and correlations. We just haven’t been
structuring it or calling it research. For example, C.E.O. Carter’s,
An Encyclopedia of Psychological Astrology, provides a compendium of
traits/tendencies and their astrological correspondences born from
Carter’s qualitative investigation: he studied groups of charts sharing
a common trait (such as alcohol abuse) and then proposed astrological
markers or signatures for the group based on his observations. If one
wanted to test the validity of Carter’s observations and determine the
probability the correlations exist more frequently in the charts of
alcohol abusers as compared to the general population or a group of non
abusers, a quantitative approach and a control group would be used.
Quantitative analysis would affirm or disaffirm the qualitative
analysis. You see? Qualitative and quantitative approaches are not
mutually exclusive. In fact, the two approaches compliment each other
and are easily integrated under the structure of one research design.
Essentially, the two are virtually inseparable. Think about it. We use
the quantitative process to assign meaningful numerical values to
qualitative data. And we qualitatively analyze and assign meaning to
quantitative data.
I view the choice of methodology as more a matter of preference than
paradigm. Yet, paradigm influences methodological preferences. The
quantitative versus qualitative division that exists between researchers
is more philosophical than methodological and is rooted more in
differences in world view as well as the human propensity for dualistic,
either/or mind sets. While we’ve been living in a universe collectively
constructed within the framework of the mechanistic scientific paradigm
for two centuries, (where the tangible and intangible are separated),
modern and quantum physics suggest that we exist within a universe that
is holographically constructed (the microcosm contains the whole of the
macrocosm) allowing for the possibility of acausal principles (such as
Jung’s principle of synchronicity)1. These new paradigms grant validity
to our basic, all encompassing astrological tenet: that observable
correlations exist between the cosmos and earthly phenomena. I believe
we all agree that our research designs need to honor our astrological
paradigm and embrace both qualitative and quantitative methods.
In summary, the important issue we need to turn our attention to now
is establishing standards for astrological research designs. Our designs
need to be valuable and useful. The way to judge the value and
usefulness of a design is to establish standards for evaluating the
validity, transferability, dependability and confirmability of its
procedures. These principles guide the development of our design and
help us ensure our research projects adhere to these standards.
The Four Standards for Evaluating the Value and Usefulness of
Research Designs
Validity: How do we estimate validity? We critique the
research process. We review the means used to arrive at conclusions. We
assess the degree to which the evidence supports the conclusions.
Astrological research necessitates multifactor research. That is, it
must strive to account for all the possible interacting variables within
the ultimate understanding of the impossibility of this task. While
thinking in terms of A causes B is likely to render disappointing
results in astrological research, it's meaningful to identify as many of
those multiple factors that influence and interact with B that we're
able to identify.
The validity of a research project increases with random selection of
test groups. When comparison groups are used we must ensure they control
for astronomical anomalies and mirror the test group closely on
hemisphere, birth year, longitude and latitude. The larger the groups,
the better and when possible, it’s best to use data that spans a broad
range of time and location. That is, use data from all over the globe
with broad generational representation. Ensuring data accuracy is also
critical to assuring validity—always use data that has been coded for
its accuracy on the Rodden Scale and cite the source of your data.
Transferability refers to the degree we are able to generalize
or transfer the results of the study to other persons, places,
circumstances or times. Quantitative or statistical analysis uses a test
and comparison group. Transferability is dependent on the validity of
these groups as outlined above. While we don’t have access to all the
people or circumstances that make up our group, it is our job to ensure
we have a representative sample if we intend to generalize our findings.
Quantitative analysis enables researchers to determine the degree of
probability that the research findings can be generalized to other
persons, places, circumstances or times. Qualitative studies, such as
the case study that does not use comparison groups, does not have this
liberty. Qualitative approaches can be generalized to theory only. The
case studies are used to illustrate dimensions of the theory being
advanced but can not be generalized to other persons, times,
circumstances or places.
Dependability is something we estimate. Our judgment of
dependability rests on the consistency of the measures used. We must
ensure that our procedures are measuring the attributes, concepts or
properties we say we are measuring. This is slippery stuff in
astrological research. We have the job of specifying consistent symbolic
meanings for complex, interactive, interdependent factors of one unified
whole. In other words, we have the task of measuring the immeasurable.
We have to find a way to get to those fine distinctions that make each
independent astrological variable uniquely consistent in its
connotations and denotations. We are in desperate need of studies that
assist us in identifying these distinctions at both concrete and
abstract levels of interpretation. It is difficult to define and measure
abstract constructs such as feelings, motivations, attitudes, and
ambitions. I doubt there is a single factor that accounts for such
concepts. If you were to study intelligence, would you limit your
examination to Mercury? Of course not. But we likely don’t all agree on
all the various factors that interact to contribute to intelligence.
Operationalism is the term used to describe this process of representing
constructs and it is the job of the researcher to specify how this is
being done.
The fourth criterion, confirmability, addresses the issue of
objectivity. In recognition that we can’t entirely remove the subjective
influence of the researcher from the research equation, we critique the
research process and findings for subjective bias. For example, we might
ask if the researcher attempted to evaluate charts that disconfirm
his/her expectations. Another way to evaluate for subjective bias is to
consider the degree to which others would corroborate or agree with the
conclusions drawn from the results. It’s healthy to play devils advocate
and consider alternate ways for interpreting the data.
Summary
Qualitative research provides a means for becoming more familiar with
the phenomenon that fascinates us. Qualitative methods allow us a direct
experience with the phenomenon which can then be described in rich
detail. But once we have the direct experience and have formulated our
ideas about associations and correlations, the natural step to take is
to test our theories. That is, we employ both a representative sample of
our phenomenon (the test group) and a control group for comparison and
we do some counting. When we use a comparison group, we have the ability
to test our theories and weed out the least effective among them. We
begin to understand more intimately the dynamic, multi-layered,
multi-faceted relationship than exists among the astrological symbolism.
In this way we improve the practice of astrology. We can’t generalize
our research findings to other persons, places, circumstances or times
unless representative test and control groups are used.
Now that we have examined the standards for critiquing astrological
research designs, I’ll present a multi-method research design next
month--one that integrates qualitative and quantitative methods and
utilizes multi-factor analysis. Using the standards outlined above, we
will evaluate the value and usefulness of the design. Psychic ability
will be explored to exemplify its use. I value having you join me on
this exploration and welcome having your input and feedback.
Notes & References
1 Talbott, Michael, The Holographic Universe. New
York: Harper Collins Publishers, 1991.
Terri McCartney joined AstroDatabank as Research Director in 2004. She was appointed to the NCGR Board as Research Director in May, 2005. Terri is also employed by Astrolabe, Inc. where she provides technical support for their astrology software as well as technical writing. Terri has been a practicing astrologer for 20 years and is passionate about researching astrology to discover the techniques that work best. Terri graduated summa cum laude from Arizona State University where both her undergraduate and graduate studies were in Communication.
Research By Design
Part 2: The AstroSignature Model
|
