Research Design
by Terri McCartney
Exploring Structure and Standards
View comments on this section
Last month we looked at the steps involved in doing research and
exemplified those steps by exploring two Astrosignatures for psychic
ability set forth by Gaston Mascarenas and Mitchell Gibson. This month
we begin to explore research design. The design is the glue that holds
the research project together. It structures steps three, four and five
of the research process. That is, it structures method, data collection
and analysis. Importantly, the research design addresses the important
issues of validity, transferability, reliability and confirmability
which are the criteria we use to judge the value and usefulness of our
research endeavors.
There are three general types of research designs:
- Experimental
- Quasi-experimental
- Non-experimental
The experimental design is commonly referred to as the “scientific
method” and is characterized by the random assignment of participants
into equivalent test/treatment and control/comparison groups. Random
assignment is a means for removing bias by controlling for factors that
are not examined directly in the study. Quasi-experimental studies
adhere to less random assignment but do use test and control groups.
Non-experimental studies do not use control groups. The experimental and
quasi-experimental designs commonly employ quantitative methods of
analysis while most qualitative methods fall into the category of
non-experimental designs. Instead of offering a statistical analysis,
non-experimental designs offer a detailed descriptive analysis.
Background
Historically, astrological researchers have attempted to adhere to
the experimental design and its Gold Standard, the rigorous scientific
method. Unfortunately, the results have often been a source of despair
for the astrological community. It became evident that a research
approach that assumed a cause-effect relationship—if A then B and if not
A then not B—wasn’t suitable for capturing the multidimensional and
interactive reality astrology describes. The Scientific Experimental
design limits reality to physical objects and linear relationships.
Astrological realities are not limited this way—they encompass abstract
concepts and non-linear, multidimensional relationships. And so it is
felt that astrology can’t be tested quantitatively, because there is no
single cause-effect relationship. And there has long been debate among
quantitative and qualitative researchers—this debate is not unique to
astrology. Yet, what astrological researchers need to be discussing are
research designs that are appropriate to investigating astrology. And
I’m an advocate for designs which integrate both qualitative and
quantitative methods.
We know the basics of doing qualitative research. While astrology may
not have faired well applying the scientific method to its research
endeavors, as a discipline we have a huge body of qualitative data.
What’s qualitative data? All that richly detailed and descriptive
material found in astrological publications that is a natural result of
applying astrology. We’ve been doing our qualitative research all
along—we’ve been out in the field, thick in the trenches of life with
our clients and burning the midnight oil as we perused case studies,
theorizing about meanings and correlations. We just haven’t been
structuring it or calling it research. For example, C.E.O. Carter’s,
An Encyclopedia of Psychological Astrology, provides a compendium of
traits/tendencies and their astrological correspondences born from
Carter’s qualitative investigation: he studied groups of charts sharing
a common trait (such as alcohol abuse) and then proposed astrological
markers or signatures for the group based on his observations. If one
wanted to test the validity of Carter’s observations and determine the
probability the correlations exist more frequently in the charts of
alcohol abusers as compared to the general population or a group of non
abusers, a quantitative approach and a control group would be used.
Quantitative analysis would affirm or disaffirm the qualitative
analysis. You see? Qualitative and quantitative approaches are not
mutually exclusive. In fact, the two approaches compliment each other
and are easily integrated under the structure of one research design.
Essentially, the two are virtually inseparable. Think about it. We use
the quantitative process to assign meaningful numerical values to
qualitative data. And we qualitatively analyze and assign meaning to
quantitative data.
I view the choice of methodology as more a matter of preference than
paradigm. Yet, paradigm influences methodological preferences. The
quantitative versus qualitative division that exists between researchers
is more philosophical than methodological and is rooted more in
differences in world view as well as the human propensity for dualistic,
either/or mind sets. While we’ve been living in a universe collectively
constructed within the framework of the mechanistic scientific paradigm
for two centuries, (where the tangible and intangible are separated),
modern and quantum physics suggest that we exist within a universe that
is holographically constructed (the microcosm contains the whole of the
macrocosm) allowing for the possibility of acausal principles (such as
Jung’s principle of synchronicity)1. These new paradigms grant validity
to our basic, all encompassing astrological tenet: that observable
correlations exist between the cosmos and earthly phenomena. I believe
we all agree that our research designs need to honor our astrological
paradigm and embrace both qualitative and quantitative methods.
In summary, the important issue we need to turn our attention to now
is establishing standards for astrological research designs. Our designs
need to be valuable and useful. The way to judge the value and
usefulness of a design is to establish standards for evaluating the
validity, transferability, dependability and confirmability of its
procedures. These principles guide the development of our design and
help us ensure our research projects adhere to these standards.
The Four Standards for Evaluating the Value and Usefulness of
Research Designs
Validity: How do we estimate validity? We critique the
research process. We review the means used to arrive at conclusions. We
assess the degree to which the evidence supports the conclusions.
Astrological research necessitates multifactor research. That is, it
must strive to account for all the possible interacting variables within
the ultimate understanding of the impossibility of this task. While
thinking in terms of A causes B is likely to render disappointing
results in astrological research, it's meaningful to identify as many of
those multiple factors that influence and interact with B that we're
able to identify.
The validity of a research project increases with random selection of
test groups. When comparison groups are used we must ensure they control
for astronomical anomalies and mirror the test group closely on
hemisphere, birth year, longitude and latitude. The larger the groups,
the better and when possible, it’s best to use data that spans a broad
range of time and location. That is, use data from all over the globe
with broad generational representation. Ensuring data accuracy is also
critical to assuring validity—always use data that has been coded for
its accuracy on the Rodden Scale and cite the source of your data.
Transferability refers to the degree we are able to generalize
or transfer the results of the study to other persons, places,
circumstances or times. Quantitative or statistical analysis uses a test
and comparison group. Transferability is dependent on the validity of
these groups as outlined above. While we don’t have access to all the
people or circumstances that make up our group, it is our job to ensure
we have a representative sample if we intend to generalize our findings.
Quantitative analysis enables researchers to determine the degree of
probability that the research findings can be generalized to other
persons, places, circumstances or times. Qualitative studies, such as
the case study that does not use comparison groups, does not have this
liberty. Qualitative approaches can be generalized to theory only. The
case studies are used to illustrate dimensions of the theory being
advanced but can not be generalized to other persons, times,
circumstances or places.
Dependability is something we estimate. Our judgment of
dependability rests on the consistency of the measures used. We must
ensure that our procedures are measuring the attributes, concepts or
properties we say we are measuring. This is slippery stuff in
astrological research. We have the job of specifying consistent symbolic
meanings for complex, interactive, interdependent factors of one unified
whole. In other words, we have the task of measuring the immeasurable.
We have to find a way to get to those fine distinctions that make each
independent astrological variable uniquely consistent in its
connotations and denotations. We are in desperate need of studies that
assist us in identifying these distinctions at both concrete and
abstract levels of interpretation. It is difficult to define and measure
abstract constructs such as feelings, motivations, attitudes, and
ambitions. I doubt there is a single factor that accounts for such
concepts. If you were to study intelligence, would you limit your
examination to Mercury? Of course not. But we likely don’t all agree on
all the various factors that interact to contribute to intelligence.
Operationalism is the term used to describe this process of representing
constructs and it is the job of the researcher to specify how this is
being done.
The fourth criterion, confirmability, addresses the issue of
objectivity. In recognition that we can’t entirely remove the subjective
influence of the researcher from the research equation, we critique the
research process and findings for subjective bias. For example, we might
ask if the researcher attempted to evaluate charts that disconfirm
his/her expectations. Another way to evaluate for subjective bias is to
consider the degree to which others would corroborate or agree with the
conclusions drawn from the results. It’s healthy to play devils advocate
and consider alternate ways for interpreting the data.
Summary
Qualitative research provides a means for becoming more familiar with
the phenomenon that fascinates us. Qualitative methods allow us a direct
experience with the phenomenon which can then be described in rich
detail. But once we have the direct experience and have formulated our
ideas about associations and correlations, the natural step to take is
to test our theories. That is, we employ both a representative sample of
our phenomenon (the test group) and a control group for comparison and
we do some counting. When we use a comparison group, we have the ability
to test our theories and weed out the least effective among them. We
begin to understand more intimately the dynamic, multi-layered,
multi-faceted relationship than exists among the astrological symbolism.
In this way we improve the practice of astrology. We can’t generalize
our research findings to other persons, places, circumstances or times
unless representative test and control groups are used.
Now that we have examined the standards for critiquing astrological
research designs, I’ll present a multi-method research design next
month--one that integrates qualitative and quantitative methods and
utilizes multi-factor analysis. Using the standards outlined above, we
will evaluate the value and usefulness of the design. Psychic ability
will be explored to exemplify its use. I value having you join me on
this exploration and welcome having your input and feedback.
The AstroSignature Model
View comments on this section
I'm energized by our renewed interested in astrology research. If you
subscribe to the ISAR E-newsletter, you are aware of the dialogue that has
been taking place about methodologies for conducting and standards for evaluating
astrology research. It's apparent that as a community, we are united in our
agreement that our research designs and methodologies must honor the integrity
of the astrological paradigm. In this article I will outline the research process
that fuels the AstroSignature research model and critique it by the standards
that ensure quality astrological research.
The Model
The first two steps of the research process involve selecting a topic to
investigate and conducting a literature review. The third step in the research
process, formulating research objectives and methodology, is driven by the research
design. The objective that is inherent to the AstroSignature model is to advance an
astrological signature for the topic being investigated. An AstroSignature sets
forth multiple astrological factors that have been found in the individual horoscopes
of a collective of people or events that share a common experience—be it a vocation,
a disease, trait, preference, event, etc. These multiple factors form the criteria by
which individual horoscopes are analyzed and scored to determine the degree to which they
fit the astrological patterns that are being tested.
In essence, the AstroSignature sets forth a theory about the cosmic factors that are
believed to interact and have influence when a particular phenomenon is present. For example,
if the topic being investigated is psychic ability, the AstroSignature outlines (in the form
of multiple rules) the astrological symbolism believed to be emphasized in the horoscope when
psychic ability is demonstrated. Horoscopes are evaluated by the criteria set forth in the
AstroSignature and assigned a score by how well they fit the AstroSignature for psychic ability.
It is assumed that those people with psychic ability will score well above the average score earned
by those with no demonstrated psychic ability.
AstroSignatures provide us with a means to measure and test our theories about the qualities and
characteristics we attribute to the astrological symbolism. When there are contradictions in opinion
about what astrological factors have the greatest influence in a particular phenomenon, an AstroSignature
can assist us in identifying what is most prevalent.
The challenge is to identify the relevant astrological symbolism that will be used to build the
AstroSignature. This necessitates both a qualitative and quantitative analysis. For example, if our
topic is psychic ability then our analysis involves both case study (a qualitative approach) and
comparing the chart factors of those well-known for their psychic abilities to the horoscopes of a general
population of people with no known psychic ability. It's important to compare to ensure that what we are
finding in the charts of the psychics occurs significantly more frequently than by chance or random error.
Such an evaluation necessitates counting, a quantitative analysis.
In step four, we collect data. We are conscientious about collecting reliable data to
represent both our psychic test group and comparison group because these sample groups and
their size influence the validity of our research endeavor. David Cochrane has published
guidelines for doing astrological research and notes the importance of ensuring that the
experimental group is "highly homogenous."2 That is, narrow down our criteria for selecting
our experimental group. For example, scientist is a broad category and while all scientists
might share similarity in chart dynamics, narrowing the experimental group of scientists
to chemists (a more homogenous group) might render different results than studying
astronomers.
In our example of psychic ability, the Rodden database was used to identify 80 well-known
psychics with reliable birth data (a Rodden rating of B or above). Of these 80 psychics, 64 have
a Rodden rating of A or above. The AstroDatabank software was then used to create a control group
20 times larger than the psychic group (i.e., control group consisting of 1600 records). The
AstroDatabank control group closely mirrors our psychic experimental group in order to control
for uneven distribution of astronomical realities.3 The larger our comparison or control group,
the less likely it is that the things we find to be significant will have occurred by random
error or chance. In the same vein, the greater the homogeny of our test group, the greater the
degree we are able to transfer or generalize our findings to other similar groups.
Once we have our data, our next step is to analyze it both qualitatively and quantitatively.
This fifth step of the research process involves a circular qualitative and quantitative analysis,
one naturally feeding into and supporting the other. A common qualitative technique, the case
study, provides one means for analyzing the data. That is, to study the horoscopes and look for
patterns occurring in the charts of the psychics not often found in the charts of the comparison
group. Another option is to conduct a statistical univariate comparison analysis4 of the
individual astrological factors to identify specific patterns occurring in the charts of the
psychics significantly more often than they are found in the charts of the control group.
You don't have to be a statistical maverick to conduct this sort of analysis, the factor
analysis feature in AstroDatabank 4.0 will do it for you.
Often research is driven by a desire to prove or disprove specific assumptions or
hypothesis. Another approach is to allow the general astrological hypothesis – that there
is a correlation between cosmic realities and human realities – to drive the research
without a specific hypothesis. In essence, it's assumptionless. It is to ask the data to
show us what is significant. That is what a univariate comparison analysis does for us.
It shows us what is occurring significantly more frequently in the charts of the psychics
than it is occurring in our control group. More on this next month when I'll delve more
deeply into the intricacies of data analysis, demonstrate more clearly the circular
relationship that exists between qualitative and quantitative analysis and outline the
process of building an AstroSignature.
Judging the Value and Usefulness of the AstroSignature Model
There are astrologers who have little faith in the quantitative analysis to render any
valuable or useful results. This is understandable in light of our research experience
with quantitative methods in the past. Early attempts involving quantitative methods often
failed to demonstrate significance because only univariate analysis was used. New computer
technology has made it easier to do multivariate analysis that takes into account the
interrelationships that exist among the various parts of the horoscope. The AstroSignature
involves a multi-factor analysis.
Glenn Perry suggests that there are six basic philosophical principles that constitute
general rules of interpretation and "any experimental design that violates even one of
these rules must be considered unsound."5 These six principles are:
- Meaning is a function of context.
- Personality is an emergent property.
- The meaning of chart symbols contains an inescapable ambiguity.
- Astrological phenomena are synchronistic.
- Astrological causation is circular and teleological.
- The horoscope symbolizes an open, evolving, indeterminate system.
In conducting single-factor research, astrologers have violated several of these principles.
The AstroSignature research model involves multi-factor analysis and therefore honors the
complexity of the whole system symbolized by the chart. In essence, we can't isolate factors
in the chart and forget the inherent, multifaceted relationships that exist among the parts
of the whole. Perry's first principle reminds us that the context of the whole is established
by the relationship that exists among its various parts. The second principle follows from the
first to remind us that what we are studying is an emergent property of the whole chart. At the
same time, I believe that our experience also demonstrates that particular parts have greater
input into particular outcomes than do others. Our job is to identify those prevalent multi-factors
and the AstroSignature assists us in doing this.
The third principle cautions us against boxing in our symbols. Astrological symbols
operate on multi-dimensions simultaneously and the result is a wide range of possibilities.
The AstroSignature embraces the multi-dimensional reality symbolized in the chart. It
allows us to score a chart on how strongly it resonates with the multiple factors most
commonly found when the phenomenon or outcome we are studying exists. In simple terms,
while we can't say that one factor such as Mars conjoined the Ascendant is a signature
for boxing prowess, I believe we can set forth a multi-factor signature containing 30-40
astrological factors commonly found in the charts of successful boxers that sets these
boxers apart from the general population. The AstroSignature provides us with deeper insights
into how the diverse parts of the whole symbolized by the chart interact and resonate with the
particular phenomenon or outcomes, such as boxing talent.
Perry's principles four and five emphasize the acausal nature of astrological reality.
The parts of the whole of one chart (the microcosm) simultaneously exist within larger
wholes (macrocosms). There is no identifiable separation between within and without. As
the fundamental astrological tenet states: as above, so below. A state of mutual reciprocity
of influence exists between the inner and the outer. The key point here is that we can't
assume singular or linear lines of cause and effect. With astrology we assume correlations. The
AstroSignature model accounts for the correlations that we are able to identify within the
understanding that it is impossible to account for all possible influences that result in a
particular phenomenon or outcome.
The final principle states that the horoscope symbolizes an indeterminate system. It reinforces
the previous five principles to remind us that neither the microcosm nor the macrocosm is static.
Instead, they are ever evolving in a dynamic relationship. Additionally, there is no way for us to
know just by looking at a chart if it belongs to a person, collective, place, creature, event,
etc. With astrological research we are attempting to define and measure concepts, attributes,
motivations and other intangible, indeterminate variables. And certainly the dependability of the
research findings rest on how well we are measuring what we say we are measuring. This is why it
is imperative to integrate the qualitative analysis with the quantitative to ensure our
AstroSignature is astrologically sensible – that is, it is meaningful within our understanding of
the astrological symbolism.
Summary
In this article we have looked at the AstroSignature research design and demonstrated that it
honors the astrological paradigm and has the ability to provide us with valuable and useful
information. Certainly this has the potential to advance our knowledge of the multidimensional
relationships that exist among the astrological symbolism, enabling us to apply our knowledge more
meaningfully in practice. Next month we will explore in greater detail the integration of
qualitative and quantitative techniques in data analysis and build an AstroSignature for psychic
ability.
Notes & References
1 Talbott, Michael, The Holographic Universe. New
York: Harper Collins Publishers, 1991.
2 Cochrane, D. (2004). Towards a Proof of Astrology:
An AstroSignature for Mathematical Ability. On the Internet at
http://www.astrosoftware.com/Proveast.htm.
3 From the AstroDatabank user manual, following is a
description of how control groups are created: Imagine each of the
four elements of a set of birth data – Month-Day (where the Sun is),
Year, Time, and Place – is a suit in a deck of cards. Imagine that you
deal the cards in rows where it takes four cards to make up a birth data
record. There are columns for Month-Day, Year, Time, and Place. Imagine,
now, that you pick up the cards in the Month-Day column, shuffle them,
and then deal them out again down the column. You will have the same
number of May 3rds and November 10ths in that column as before, but
they will be paired with different years, times and places than they
were before. If you did this for each of the four elements (columns,
in our example) in a birth record, you would have a control group that
has the same distribution of month-days, years, times and places as
the experimental group, but the month-days, years, times and places
will be completely "shuffled".
This shuffling method of building control group is superior to building a
control group based on random birth data. Many people using a control group
built on random birth data elements have ended up with spurious results.
Suppose you built a randomized control group of a thousand records from the
years 1900-2000. We presume that there would be about ten records for each
year. Imagine you had an experimental group of 1000 arthritis sufferers
from the same time span, but 500 were from the 1930's and the rest from
other decades. If you compared their Saturn placements by sign to a
completely randomized control group of a thousand records, you would
see many more Saturn's in Capricorn, Aquarius, Pisces and Aries in the
experimental group than in the control group. One might conclude that
this is significant but it isn't because so much of the data is from
the 1930's when Saturn was in Capricorn, Aquarius, Pisces and Aries.
That is why it is so important to mirror the experimental birth data
in your control group. Any anomalies in the experimental group are
mirrored in the control group so the chances of getting false signals are reduced.
4 A univariate analysis examines astrological factors individually.
5 Perry, G. (1997). Stealing Fire from the Gods, Myth and Method in
Astrological Research. San Rafael, CA: The Academy of AstroPsychology, p.9.
Terri McCartney joined AstroDatabank as Research Director in 2004. She was appointed to
the NCGR Board as Research Director in May, 2005. Terri is also employed by Astrolabe, Inc.
where she provides technical support for their astrology software as well as technical writing.
Terri has been a practicing astrologer for 20 years and is passionate about researching
astrology to discover the techniques that work best. Terri graduated summa cum laude from
Arizona State University where both her undergraduate and graduate studies were in Communication.
|
