Estudios interdisciplinario en Social y Ciencias del Comportamiento.

http://www.huffingtonpost.com/entry/exonerations-2015_us_56ac0374e4b00b033aaf3da9

http://rakesresearch.yolasite.com/resources/Threats_to_Validity.pdf

Degrees::  Psychology, Interdisciplinary Studies in Social and Behavioral Sciences, Interdisciplinary Studies in Arts and Humanities.

Licenciaturas: Psicología, Estudios interdisciplinario en Social y Ciencias del Comportamiento, Estudios Interdisciplinarios en Artes y Humanidades.

Why Do We Need Science?

Humans are poor data gathering machines. We have numerous biases, cognitive flaws, and psychological errors that prevent our unguided minds from grasping reality in any accurate way.

To put it more specifically:

[There are] two countervailing human tendencies of omission and commission: to neglect the logical and statistical strategies of science on the one hand, and to over-utilize intuitive or simplistic strategies on the other

Thus, in order to deal with the deluge of information that our brains take in every second of everyday, we have to structure it in a way that can accurately interpret, explain, and predict reality. Science can do this where other forms of thinking fail. Gut-feelings and common sense are not enough; they may get us somewhere, but not always to the truth.

As we proceed, I will outline how one thinks scientifically (and unscientifically) in order to show you how modern science obtains knowledge about the universe. More importantly, as we continue, we should know why thinking in a scientific way is the best vehicle for obtaining knowledge.

To that end, let us outline what are the unscientific ways of thinking and why we cannot rely on them.

Unscientific Sources of Knowledge

INTUITION

By intuition we mean vague feelings or gut reactions about a question or phenomena. One problem with intuition as a knowledge source is that our intuitions are often wrong. For example, we may have a gut reaction that giving children sugar will make them more hyperactive. However, if we scientifically interpret the data, we find that this is not that case.

A second problem arises because intuitions are feeling-based. Mood, and a host of other psychological and physiological factors influence intuition. A judgment of information may be solely based on how you rolled out of bed this morning, and not what the data says. An “intuitive” judgment about the driving skills of another motorist will differ greatly if that motorist just cut you off.

Finally, ask five different people to make predictions based on intuition and we are likely to receive five different answers. We simply cannot place enough confidence in intuition to accept it as a source of information.

TENACITY

This second source of unscientific information, know also as tradition, includes unquestioned belief in superstitions, truisms, and myths. These forms of knowledge are often passed from generation to generation through cultural mechanisms such as family, media, and religious institutions. As an information source, tradition is used all the time. Like intuition however, many tenaciously held beliefs are inaccurate.

For example, at one time everyone held tenaciously to the beliefs that the Earth was flat; that the Sun revolved around the Earth; that applying leeches to the ill was good medical practice; and that Salem, Massachusetts was plagued with witches.

Scientific breakthroughs in medicine, physics, and genetics, to name a few disciplines, continually expose the erroneous nature of previously held beliefs. Moreover, across and within cultures there is considerable variance in perspectives: What seems obvious to one social group is often rejected as ludicrous by another.

Tenacity, like intuition, is an unacceptable way to answer scientific questions.

COMMON SENSE

Common sense, a third unscientific way of knowing, consists of generating what appear to be obvious answers to scientific questions. Appeals to this sort of knowledge are accompanied by prefacing or supporting remarks such as “it’s obvious that…,” “everybody knows…,” “any halfway intelligent person can see that…,” or “it’s just common sense.”

Common sense, however, is often wrong and people often disagree what even constitutes the common sense thing to do in a particular situation. Furthermore, scientific problems abound for which common sense provides no insight. This is especially true for complex problems.

For example, what is the commonsense way to search for the Higgs Boson? What is the commonsense way to treat pancreatic cancer? There are no existing commonsense answers to such questions.

PERSONAL EXPERIENCE

Personal experience is often used as a knowledge source. We posses a wealth of personal experience and, while experience is an extremely valuable resource, there are three reasons to be cautious about deriving knowledge claims about science based on experiences.

First, personal experience is both subjective and uncontrolled, leaving us susceptible to misperception and misrepresentation of events. We are limited in the amount of information we can process because the quantity of stimuli in any given situation is virtually unlimited. Because of this limitation, we often attend to events and stimuli selectively: We simply do not and can’t pay attention to every sound; we don’t notice everything there is to see; many things go undetected. Rather, we attend to some stimuli and block out others, some of what we do sense, we sense incorrectly, yielding an experience that is necessarily incomplete and inaccurate.

Second, we selectively remember characteristics of experience. Anyone who has every studied for a test realizes that some of the subject matter, although we read it and perhaps even hear it during class, was somehow lost on test day. Thus, our memories of events are usually incomplete and misrepresent events. It is also problematic that our selectivity is driven by strong preconceptions. Meaning we attend, perceive, accept, and recall data that confirm our beliefs and attitudes whereas we tend to ignore, distort, discount, and forget data which disconfirm our beliefs and attitudes. This is the confirmation bias at work.

AUTHORITY

A fifth unscientific source of knowledge, authority, consists of appealing to experts for answers to our questions. We are surrounded by experts and authorities: professors, physicians, attorneys, journalists, economic advisers, stock brokers, mechanics, news anchors, just to name a few. Although experts frequently provide valuable service, there is often disagreement among them, and, of course, they can be wrong.

The more important issue is how the experts gained their knowledge in the first place. If their knowledge was acquired through intuition, tenacity, or experience, it is subject to many of the caveats already mentioned.

Assuming that an expert’s knowledge is the product of scientific inquiry, it is the scientific inquiry, not the expert, that is the source of the knowledge. Well-informed experts can disseminate knowledge but they are not acceptable as oracles of it.

RATIONALISM

A final unscientific of deriving answers is through rationalism, or logic, usually in the form of deduction. Accordingly, knowledge takes the form of conclusions, which are deduced from premises. For example, suppose that (1) watching a scary movie usually makes a person fearful or anxious, and that (2) being fearful or anxious usually causes the person’s heart rate to increase. Applying logic we would conclude, therefore, that watching a scary movie raises a person’s heart rate.

Two major problems are associated with rationalism as a source of knowledge. First, we must consider how the truth of the premises were determined. Logic alone cannot produce premises, and without valid premises, sound conclusions cannot be reached. Second if we apply logic in this form to premises that are not absolutely true, erroneous conclusions will be met even when strictly following the rules of deduction.

Reconsider our example. Suppose that when we said “usually” we meant around 70 percent of the time. Thus, the probability of a scary movie making you fearful is 70 percent and the probability of being fearful increasing your heart rate is 70 percent. What then is the probability that watching a scary movie increases your heart rate? The correct answer is calculated by multiplying the two probabilities together ,[.70x.70], which equals only 49 percent. So, more often than not, watching a scary movie would not increase your heart rate, given the probabilities we assumed.

Logic itself does not provide these crucial probabilities estimates; nor does it produce the premises. As humans who arevery bad at estimating probabilities, relying on pure logic will not get us as far as science can. While logic is an essential tool used by scientists, it alone is insufficient as a knowledge source because its use requires existing knowledge in the form of premises. If the premises are incorrect, so is your logic.

Thinking Scientifically

As we can see from this brief overview of the unscientific sources of knowledge, we need a more precise and objective approach to generating knowledge. This leads us to the scientific perspective of knowledge acquisition.

Regardless of the field of study, those committed to a scientific approach to generating answers to questions, whether theoretical or practical in nature, can almost always be described in terms of a five-step process known as thescientific method. We will quickly review the steps with special focus on how they differ from unscientific ways of gaining knowledge. The steps are:

1. Observe a phenomena that needs to be explained
2. Construct provisional explanations or pose hypotheses
3. Design an adequate test of the hypotheses
4. Execute the test
5. Accept, reject, or modify our hypotheses based on the outcome of our test

CURIOSITY

The first step describes a basic characteristic of scientific thinking,curiosity. We notice that an object let go on Earth always falls straight down. We see that some planets are made almost entirely of gas. We wonder why some animals take care of their young and others do not. When we focus on such observations and feel compelled to explain them, we have engaged in the first step of the scientific method.

CONCEPTUALIZING

It is at this second step that we begin to differentiate scientific from unscientific thinking. Not having found an explanation for a phenomena, a scientist looks for clues in existing research. Under a scientific framework, input from intuition, tradition, experience, common sense, experts, and logic might be incorporated during the construction of a preliminary hypothesis, but we do not at this point accept the validity of those explanations. These unscientific explanations may be correct, but we cannot be sure if we do not test them.

Scientific thinking builds upon the previous thinking of previous scientists, thus ensuring the maintenance of objectivity separate from unscientific forms of knowing.

DESIGNING AND EXECUTING THE TEST

Another feature of the scientific method is the testing of hypotheses that take care to control for all of the possible variables that might confuse the research. Scientists must rule out all competing explanations if they are to show that their explanation is the correct one. Unscientific thinking leaves many possible explanations up for debate, and rarely settles on the truth (or tests for the explanation’s validity).

With all of the possible confusing variables controlled for, the scientist carries out the experiment. If the experiment was designed correctly and carried out in the right way, the scientist should receive objective data about the phenomenon in question.

Just how a scientist carries out a “correct” experiment is a much larger subject, and will not be discussed here. Suffice to say that all scientific research depends on scientists following common experimental rules and procedures to make sure that their data can be replicated by others, is falsifiable, and reflects reality.

RESULTS

The final stage of the scientific process calls for the rejection, acceptance, or modification of the explanation based on an analysis of the data. During this process the scientist takes many things into consideration: statistical significance, experimental error, false positives and negatives, etc.

The power of science resides within this final stage. The ability for science to progressively accumulate knowledge that has been checked, tested, repeated, and verified separates it from all other areas of knowing. Science knows when it is wrong and when it has made a mistake, and the internal mechanisms of peer review etc. then move it forward. This is what separates science from pseudoscience. Pseudoscience, like homeopathy for example, does not incorporate new evidence and indeed proceeds without it. Scientific evidence that clearly rejects the idea that homeopathy could ever work is dismissed and forgotten by the proponents of pseudoscience. This sort of thinking, without the checks and balances of science, then becomes an unscientific form of knowledge, and is not reliable.

Science as a Way of Thinking

Science, as a human enterprise, is the most successful tool ever devised for explaining our universe. It has passed the tests that other forms of thinking do not. This is why science proceeds the way that it does, and why it is so powerful. Adapted from Michael J. Betty’s essay entitled: “Thinking Quantitatively”,published in the book An Integrated Approach to Communication Theory and Research.

1) What is good about this?

2) What is not perfect yet?

3) What I am willing to do in order to get the way I want it?

4) What am I willing to no longer do, in order to have the things the way I want?

5) How can I do (that / what) is Necessary to get this job done and enjoy the process?

START AGAIN ! Even if you feel tired;

if you leave the victory;

although an error hurt you;

even break a business;

but a betrayal strikes you;

but an illusion goes;

although pain burning eyes;

even ignore your efforts;

but ingratitude is the reward;

misunderstanding but cut your laughter;

although everything seems nothing;

START AGAIN!

Threats to Validity
Internal Validity: Reasons why inferences that the relationship between two variables is causal may be incorrect:
1. Ambiguous Temporal Precedence: Lack of clarity about which variable occurred first.
2. Selection: sometimes at the start of an experiment, the average person receiving one experimental condition already differs from the average person receiving another condition. This difference might account for any result
3. History: Events occurring concurrently with treatment could cause the observed effect.
4. Maturation: Naturally occurring changes over time could be confused with a treatment effect.
5. Regression: when units are selected for their extreme scores, they will often have less extreme scores on other variables, which can be confused with a treatment effect.
6. Attrition: Loss of respondents to treatment or to measurement can produce artificial effects if that loss is systematically correlated with conditions.

7. Testing: Exposure to a test can affect scores on subsequent exposures to that test, which can be confused with a treatment effect.
8. Instrumentation: The nature of a measure may change over time or conditions in a way that can be confused with a treatment effect. For example, a spring might become weaker over time and easier to push, artificially increasing reaction times.

9. Additive and Interactive Effects of Threats to Internal Validity: The impact of a threat can be added to that of another threat or may depend on the level of another threat.


Statistical Conclusion Validity
 refers to the appropriate use of statistics to infer whether the presumed independent and dependent variables co-vary.
1. Low Statistical Power: An insufficiently powered experiment may incorrectly conclude that the relationship between treatment and outcome is not significant

2. Violated Assumptions of Statistical Tests: can lead to overestimating or underestimating the size and significance of an effect.
3. Fishing and the Error Rate Problem: Repeated tests for significant relationships, if uncorrected for the number of tests, can artificially inflate statistical significance.
4. Unreliability of Measures: weakens the relationship between two variables and strengthens or weakens the relationships among three or more variables.
5. Restriction of Range: Reduced range on a variable usually weakens the relationship between it and another variable.
6. Unreliability of Treatment Implementation: If a treatment that is intended to be implemented in a standardized manner is implemented only partially for some respondents, effects may be underestimated compared with full implementation.
7. Extraneous Variance in the Experimental Setting: Some features of an experimental setting may inflate error making detection of an effect more difficult.
8. Heterogeneity of Units: Increased variability on the outcome variable within conditions increases error variance, making detection of a relationship more difficult. The more the units in a study are heterogeneous within conditions on an outcome variable, the greater will be the standard deviations on that variable and on any others correlated with it. Possible solution is blocking - measure relevant respondent characteristics and use them for blocking or as covariates.
9. Inaccurate Effect Size Estimation: Some statistics systematically overestimate or underestimate the size of an effect.
Threats to Validity
Construct Validity: The validity of inferences about whether the cause-effect relationship holds over variation in persons, settings, treatment variables, and measurement variables.
1. Inadequate Explication of Constructs
2. Construct Confounding - operations usually involve more than one construct; failure to describe all the constructs may result in incomplete construct inferences.
3. Mono-Operation Bias - using only one operationalization of each construct. Single operations both under-represent constructs and may contain irrelevancies.
4. Mono-Method Bias - when all operationalizations use the same method, that method becomes part of the construct actually studied.
5. Confounding Constructs with levels of constructs - Inferences about the constructs that best represent study operations may fail to describe the limited levels of the construct that were actually studied
6. Treatment sensitive factorial structure - the structure of a measure may change as a result of treatment, change that may be hidden if the same scoring is always used. The structure itself becomes a construct rather than just the treatment, person, etc.
7. Reactive Self-Report Changes - the factorial structure and the level of responses can be affected by whether a person is accepted into the treatment or control group separate from the treatment effect.
8. Reactivity to the Experimental Situation - subjects may try to guess what the experimenter is studying and give results that they think the researcher wants to see. Sometimes referred to as the "placebo effect."
9. Experimenter Expectancies - these expectations become part of the molar treatment package. For example, teacher expectations about student achievement become self fulfilling prophecies.
10. Novelty and Disruption Effects - when an innovation is introduced, it can breed excitement, energy, and enthusiasm that contribute to success, especially if little innovation previously occurred.

11. Compensatory Equalization - when two conditions are being contrasted and the control group is compensated to equal out the inequity.
12. Compensatory Rivalry - public assignment of units to control and experimental conditions can sometimes cause rivalry in which the control group tries to show that it can perform as well as the experimental group.
13. Resentful Demoralization - members of a group receiving a less desirable treatment may be resentful and demoralized, changing their responses to the outcome measures.
14. Treatment Diffusion - the participants in one condition receive some or all of the treatment in the other condition. (Participants try to surreptitiously receive the treatment for its benefit)
External Validity:
Defined as the question, "to what populations, settings, and variables can this effect be generalized?"
1. Interaction of the Causal Relationship with Units: An effect found with certain kinds of units might not hold if other kinds of units had been studied.
2. Interaction of the Causal Relationship over treatment variations: An effect found with one treatment variation might not hold with other variations of that treatment, or when that treatment is combined with other treatments, or when only part of that treatment is used.
3. Interaction of the Causal Relationship with outcomes: An effect found on one kind of outcome observation may not hold if other outcome observations were used.
4. Interaction of the Causal Relationship with Settings: An effect found in one kind of setting may not hold if other kinds of settings were to be used.
5. Context Dependent Mediation: An exp

Statistical Concepts:
  Central Tendency (mean, median, mode)

  Variability : (Variable )Something that the  researcher/experimenter can measure.
 

Statistics (what are they and what are the different types) Statistical Concepts: A measure of central tendency is a single value that attempts to describe a set of data by identifying the central position within that set of data. As such, measures of central tendency are sometimes called measures of central location. They are also classed as summary statistics. The mean (often called the average) is most likely the measure of central tendency that you are most familiar with, but there are others, such as the median and the mode.

Rectangular Distribution : The scores in a rectangular distribution are all about equally frequent or probable. An example of a rectangular distribution is the theoretical distribution representing the six possible scores that can be obtained by rolling a single six-sided die.

Bimodal Distribution: In the case of a bimodal distribution, two distinct ranges of scores are more common than any other. A likely example of a bimodal distribution would be the heights of the athletes attending the annual sports banquet for a very large high school that has only two sports teams: women’s gymnastics and men’s basketball. If this example seems a little contrived, it should. Bimodal distributions are relatively rare, and they usually reflect the fact that a sample is composed of two meaningful subsamples.

Normal Distribution: The nice thing about the normal distribution is that if you know that a set of observations is normally distributed, this further improves your ability to describe the entire set of scores in the sample.

Hypothesis vs. Theory: So what is the Relationship between theory and data you might ask?  Well the first relationship is Deduction
Theory: A set of logically consistent statements about some psychological phenomenon that best summarizes existing empirical knowledge of the phenomenon organizes this knowledge  in the form of precise statements of the relationship among variables  provides a tentative explanation for the phenomenon serves as a basis for making predictions about behavior, from theory to actual research.

Hypothesis: Prediction about specific events that is derived from the theory.

Different types of hypotheses

Probability Theory: Probability Theory: From the classical perspective, the probability of an event is a very simple thing: It is (a) the number of all specific outcomes that qualify as the event in question divided by (b) the total number of all possible outcomes.

Different Types of error (Type I vs. Type II)
Type I Error: incorrectly rejecting the null hypothesis when it is, in fact, correct.

Type II Error: occurs when we fail to reject an incorrect null hypothesis.
Effect Size and Significance Testing

Effect Size: the magnitude of the effect in which they happen to be interested.
Important Statistical Notation

Process of publishing scholarly articles

Distinction between scholarly and unscholarly sources
Content Validity Does the method used actually seem to measure what you intended? For example, does an IQ test actually measure levels of intelligence, or is it measuring ability to solve puzzles?
-     To ensure content validity, a panel of experts (on IQ for example) may be asked to assess the measure for validity.
Concurrent validity How well does the measure agree with existing measures? For example, does our IQ test agree with established tests of IQ?
-     We can ensure concurrent validity by testing participant with both the new test and the established test. If our test has concurrent validity, there should be high agreement between the scores on both measures.
Construct validity Is the method actually measuring all parts of what we are aiming to test? For example, if we use a maths test to test intelligence, we are missing out on other factors involved such as linguistic ability or spatial awareness.
-     To maintain construct validity, we need to define what it is we are aiming to measure, and ensure that all parts of that definition are being measured.
Predictive validity Is our measure associated with future behaviour? For example, if someone scores high on our IQ test, we would expect them to perform well in GCSE exams, or do well in their career. This is similar to concurrent validity.
-     We can investigate predictive validity by following up our participants to see if future performance is similar to performance on our measure.



Reliability (different types) Reliability:  refers to the consistency of a measure. Validity: is the extent to which a test measures what it claims to measure. There is Three main features that need to be included to measure are:


Control: refers to how well the experimenter controlled the experiment. The control is important because without control, researchers cannot establish cause and effect.
Realism: is where psychological research is provided information about how people in the real world behave. If an experiment is too controlled, too artificial or situation, participants can act differently than they would in real life.
Generalizability: is the primary objective of psychological research is producing results that can be generalized beyond the experiment setup.

Validity (different types)

Internal And External Objectivity can be achieved form a thorough review of the literature and the development of a theoretical framework.  The literature review should be presented so that the reader can judge the objectivity of the research questions. Purpose of Research Design Provides the plan or blueprint for testing research questions and hypotheses. Involves structure and strategy to maintain control and intervention fidelity.
 Variable: Something that the researcher/experimenter can measure.

  Independent Variable: The variable the experimenter has control over, can change in some way to see if it has an effect on other variables in the study.

Dependent Variable: The variable that is measured to see if a place: change takes.

Theory: Definition of a theory: A set of logically consistent statements about some psychological phenomenon that best summarizes existing empirical knowledge of the phenomenon organizes this knowledge in the form of precise statements of the relationship among variables provides a tentative explanation for the phenomenon serves as a basis for making predictions about behavior. So we have our method, we have our assumptions, so how do we actually do research? Relationship between theory and data

Hypothesis: Prediction about specific events that is derived from the theory. Induction: Logical process of reasoning from specific events to the theory (either confirming or disproving the theory).
Definition: A way of knowing characterized by the attempt to apply systematic, objective, empirical methods when searching for causes of natural events. Probabilistic Statistical determinism: Based on what we have observed, is the likelihood of two events occurring together (whether causal, predictive, or simple relational) greater than chance? Objectivity: without bias of the experimenter or participants. Data-driven: conclusions are based on the data-- objective information. Data-driven: conclusions are based on the data-- objective information.

Generalisability: The aim of psychological research is to produce results which can then be generalised beyond the setting of the experiment. If an experiment is lacking in realism we will be unable to generalise. However, even if an experiment is high in realism, we still may not be able to generalise.


Control Variable: The variable that is not manipulated that serves as a comparison group from the other variables in the study. This third variable is used to ensure that the independent variable, when manipulated is actually having an effect on the dependent variable. For example, if a similar change occurs in the control variable as the dependent variable, this indicates that the change may not be the result of the independent variable manipulation and may be a natural change in the variable. In a experiment the researcher manipulates the independent variable to see if it has an effect on the dependent variable.

Sample Variance: The sample variance equals the mean squared deviation from  . A small   means that the observed values cluster around the average, while a large variance means that they are more spread out. Thus, the variance is a measure of the “spread” in the sampled values.

Selection Threat -- Differences in groups that exist before treatment or intervention begins. Especially problematic when participants selected for belonging to a specific group. Differences may be due to initial differences in groups – not treatment or intervention.

Method: A technique used to analyze data. Commonly, a method is aligned with a particular strategy for gathering data, as particular methods commonly require particular types of data. “Method” is therefore commonly used to refer to strategies for both analyzing and gathering data.

Methodology: A body of practices, procedures, and rules used by researchers to offer insight into the workings of the world.

Insight: Evidence contributing to an understanding of a case or set of cases. Comparative-historical researchers are generally most concerned with causal insight, or insight into causal processes.

Treatment Diffusion - This occurs when a comparison group learns about the program either directly or indirectly from program group participants. In a school context, children from different groups within the same school might share experiences during lunch hour. Or, comparison group students, seeing what the program group is getting, might set up their own experience to try to imitate that of the program group. In either case, if the diffusion of imitation affects the posttest performance of the comparison group, it can have an jeopardize your ability to assess whether your program is causing the outcome.

Threats to internal validity

Ambiguous Temporal Precedence: Ambiguous Directionality -- When the independent variable is not manipulated, the direction of the influence is not always clear (e.g., impact of a therapist empathy on client outcome – does therapist get warmer b/c client improves or vice versa?)
Ambiguous Directionality: How to fix it?
Unless the temporal order is clear, the directionality is difficult to determine.
Be clear in statements of causality and describe when causality is unclear.


Selection:  -- Differences in groups that exist before treatment or intervention begins. Especially problematic when participants selected for belonging to a specific group. Differences may be due to initial differences in groups – not treatment or intervention.

History: History Threat -- Events outside experimental manipulation influence collective or individual participation “Many change-producing events may have occurred between Observation 1 and Observation 2 “ (Source: Internet Presentation)  “History is more …[likely] the longer the lapse between the pretest and posttest.” (Source: Alliant

Maturation: Maturation Threat - Normal developmental changes in participants between pre and post tests. Gains/Losses over time may be due to normal maturation.


Regression:  Regression Threat -- The tendency to drift towards the mean as one takes a test multiple times. Especially problematic when choosing “extreme” scores which can “regress” toward the mean due to influences other than treatment.   “Occurs when individuals are selected for an intervention or treatment on the basis of extreme scores on a pretest. Extreme scores are more likely to reflect larger (positive or negative) errors in measurement (chance factors). Such extreme measurement errors are NOT likely to occur on a second testing.” (Source: Internet Presentation)

Attrition: Attrition Threat – In a longitudinal study where you need to measure responses from a participant at multiple time points, attrition refers to when participants drop out or leave a study, causes problems as it reduces available data, in particular when it is different across groups.

In a longitudinal study where you need to measure responses from a participant at multiple time points, attrition refers to when participants drop out or leave a study, causes problems as it reduces available data, in particular when it is different across groups.  When participants drop out or leave a study, causes problems as it reduces available data, in particular when it is different across groups.

         
Testing: Testing Threat -- Changes in a test score due to taking it more than once, through familiarization, recall.

Instrumentation: Instrumentation threat: How do we fix it?
Randomly assign participants to two groups, one treatment group & one control group.
Researchers can expect similar effects in treatment group versus control group over time.  Or, use tests rather than observation, ratings, or interviews.

Additive and Interactive Effects: This is when two internal validity threats are present, and at least one of the threats’ effects is dependent on the other threat or is greater because of the other threat.

External Validity:This is when the results found are not generalizable to the population of interest.

Selection (Interaction of Causal Units): This is a threat when the participants used in a study are of a specific type such that the results found with them might not apply (be generalizable) to other participants, but researchers imply this.

Interaction of the causal relationship over treatment variations:  :  This is a problem when the research study has a TREATMENT, but a weird variation of the treatment is used, and results found may not be generalizable to other variations of that treatment.

a.  Example: Researchers test a manualized CBT treatment. This CBT treatment is meant to be 12 weeks, but researchers did a 7 week version of it, so the results may be different. Effects found with this treatment are not necessarily the same at results that would have been found with the treatment they were trying to test (the 12 week treatment). OR Researchers combined CBT with another treatment, which may find different results. OR They only used part of the treatment, such as the Cognitive portion of the treatment.
b.  Remedy: Use the whole treatment as it was meant to be given.
c.  Risk: Low. Researches would likely admit that they administered a treatment differently than it was supposed to be administered.
Look in: Methods, measures, and Limitations.

Interaction of the causal relationship with outcomes: This is similar to the idea of mono operation bias, but just for the DV.  This is a threat when the outcome (usually just one Dependent Variable) was measured a certain way (usually with only one measure). It is a problem when measuring the outcome in a different way could have given different results.


Interaction of causal relationship w/settings: This is a threat when the research was done in a particular setting (environment), and results may not be generalizable to a different setting. By setting we are talking about specific research setting such as laboratory, school, home, internet, ect. NOT geographic location like Texas or New York. Geographic locations are more of a selection threat because we are talking about different types of people with results that may not generalize to other types of people. Here we are talking about a setting where results may not generalize to other settings.


Context-dependent Mediation: This is a threat when a mediating variable may not mediate the same way in a different situation (different setting, or participants, or treatment, or task, ect). This is only when the study has a mediating variable, and this relationship (the variable mediating the relationship between two other variables) may not be the same in a different situation.  Example: Suppose a study used only males to study depression. They found that owning more video games makes them super excited and causes sleeplessness, which later causes depression in these people. So, sleeplessness mediates the relationship between video games and depression. However, this may not be the case if females were used. Owning video games may bore them, which may make them depressed, but it doesn’t cause sleeplessness because they aren’t excited about the games. So the video games cause depression for them, but this isn’t mediated by sleeplessness.  Or if different video games were used, then sleeplessness may not mediate the relationship the same way.
1.Remedy: Try to use generalizable situations.
2.Risk: RARE. The study must have a mediating variable in it.
Look in: Methods, Results, and Conclusions

Interaction of History and Treatment: This is different than the Internal Validity threat History. That is when an event affects participants and causes the results found. THIS one is when results found from a study are not generalizable to other time periods. This usually has to do with whole eras. It mostly applies to old studies, when times were different, and results found may not be the same today.


Construct Validity Threats This is when researchers did not actually measure the construct they intended to measure.

Inadequate explication of constructs: This is when researchers do not get at a construct all the way. This could be when researchers only measure part of a construct. Or when they measure the wrong construct. Or when they use a measure with low reliability or validity. Or when they use a measure that is too broad to get at the measure they want. Or when they only use part of a measure but don’t check the reliability and validity of this portion of the measure.

Construct confounding: There is a construct that the researchers have not accounted for that is responsible for causing or affecting the results and relationship found. There is a construct here that is unaccounted for (missing). The must some evidence for this, good reason to believe this is a problem.


Mono-operation bias: This is a problem when only one instrument (measure) is used to measure ONE construct. For any particular construct in the study, if there was only one thing measuring it, this is a problem. It may occur multiple times in a study, but only talk about one of those times when asked to describe a threat.

Mono-method bias: This is the same thing except replace “instrument” or “measure” with “method”. Researchers could use more than one measure for any ONE construct, but if that ONE construct only has ONE method used to measure it, you may not be getting at the construct as well as you could.


Confounding constructs with levels: Mono-method bias: This is the same thing except replace “instrument” or “measure” with “method”. Researchers could use more than one measure for any ONE construct, but if that ONE construct only has ONE method used to measure it, you may not be getting at the construct as well as you could.
                                                                     

Treatment-sensitive factorial structure: This is when participants who are exposed to treatment see a measure in a different way than those who were not exposed to the treatment. For people who have received the treatment, the answers may be broken down into different “factor structures”. People who did not receive the treatment have answers that result in a single factor. The structure of the measure is different between the people who received treatment and the people who didn’t.


*** The following Construct Validity threats are somewhat specific and Medium to Low risk. Look in Methods to find these***

Reactive Self-report changes: This is when people change the way they are answering in a self-report because they want to be accepted to the study of receive treatment, or because they want to appear socially acceptable.

Reactivity to the experimental situation: This is when something in the experimental situation is affecting the way the participants respond. This could include the Hawthorne effect- a reaction to something in the environment. It could also include hypothesis guessing done by the participants, or the placebo effect.

Experimental Expectancies: This is also called the Rosenthal effect.  It is when the experimenter conveys the message of what is wanted from the participant and that leads PT to act in a certain way. Instead of actually measuring the construct naturally in the participant, you are just picking up on how the participant is trying to meet expectations

Novelty & Disruption Effects: This is when participants may respond better to things that are new. So instead of just measuring the construct as it is meant to be, the fact that the treatment or measure is new to them may skew things. OR they may respond particularly poorly or poorly to something that disrupts their routine.

Compensatory Equalization: This is when treatment provides something very desirable (researchers are testing a Free Ice-Cream treatment), and researchers attempt to compensate by giving things to the non-treatment group (vouchers they can use for ice cream after the study). Doing this must then be included as part of the treatment construct description.

Compensatory Rivalry: This is when (competitive) participants who are not receiving treatment may be motivated to show they can do as well as those receiving treatment. This rivalry motivation could change what you are measuring, and must then be included as part of the treatment construct description.
Resentful Demoralization: This is when there is a very desirable treatment (Free Ice Cream Therapy). Participants not receiving this desirable treatment may be so resentful or demoralized that they may respond more negatively than otherwise, and this resentful demoralization must then be included as part of the treatment construct. The people not getting the free ice cream may be upset and report extra depression because they don’t care anymore. Then the results would indicate that the ice cream cured depression way better than the control group, when really the people not getting ice cream were not depressed, but answered that way because they were cranky from not getting ice cream.
Treatment diffusion: This is when participants may are getting extra services outside of the study. For example, a CBT was tested for use with blind people to help them see.  Meanwhile, some people from the CBT group, and some from the control group were getting vision surgery. And others were getting homeopathic or religious treatments. And others were getting some other vision treatments. Now it is difficult to define what services each group got, because they got extra services outside of the study.
Internal Validity
To summarize the Threats to Internal Validity are:
1. Selection Bias (self- selection, existing groups, random assignment)
2. Selection by maturation interaction (a difference that isn't present or observable of a variable (usually a DV) at one point of testing isn't always the same at all points in time)
3. Regression effects
 4. Mortality (differential attrition)
5. Maturation
6. History
7. Testing (practice effects,
8. Fatigue effects, "catching on" effects)
8. Instrumentation9-threats to internal validity
Confidence in cause and effect
Requirements:
Difference in dependent variable DV
Independent variable (IV) BEFORE DEPENDENT VERIABLES (DV)
Extraneous factors (alternative rival hypotheses)
In theory
Two identical groups
Pretest-post design
Pretest-posttest design
Threat to internal validity
 Selection bias
  Self-selection
 Existing groups
  Random assignment
2)  Selection by maturation interaction
3) Regression effects   More concern for pre-post
4) Mortality
  Differential attrition
5) Maturation
6) History
7) Testing
  Practice effects
 Fatigue Effects
  “Catching on effects”
8) Instrumentation
8 threats not exhaustive
Decisions about formation of sample..

Threats to Construct Validity:
Inadequate Pre-operational Explication of constructions.
We didn’t clearly define things before we started. Mono-operational Bias. We use a single measure of a construct that is not complete. Mono-method Bias. Using only one approach to measuring a construct. We only use surveys to assess employee engagement they are self-report and subject to bias as self-report. Hypothesis Guessing. Participants try to guess what we are looking for an act differently. Participants learn they are in a pilot program aimed at improving success so they work harder or report better results.
Must due: External validity randomly select people to study (not randomly assigned). Replication even on small scale over time over sample of a study. Clear about how you select people, how do we get people to this discreetness time settings.
Internal Validity: Confidence in cause and effect Requirements. Difference in Dependent Variable. Independent variable before Dependent Variable. Extraneous factors (alterative rival hypotheses. In theory: Two identical groups Pretest-posttest design.
Makes sure I use the interactions:  selection, settings and history. Interactions selection and treatment. Selected have a different reaction to our program than other programs. Make sure you select high potential for our program. Interactions setting and treatment. Results in one setting may not work in other settings. Interactions History and treatment the results we see today may not be true for other times.
External Validity
So basically External Validity threats arise from
1. Interaction of Selection and Treatment -Those we select have a different reaction our program than others (if we select high potential people, self-fulfilling prophecy will affect the outcome/DV)

2. Interaction of Setting and Treatment - Results in one setting may not work in another. Recognition programs at hospital for doctors wont work for mcdonalds slave workers
3. Interaction of History and Treatment -results we see may not hold true for other points in time. Stock pickers at tech bubble won't get same result of Apple stock as they would now. Difference of timing aka History and Treatment

What is Random Error?

Random error is caused by any factors that randomly affect measurement of the variable across the sample. For instance, each person's mood can inflate or deflate their performance on any occasion. In a particular testing, some children may be feeling in a good mood and others may be depressed. If mood affects their performance on the measure, it may artificially inflate the observed scores for some children and artificially deflate them for others. The important thing about random error is that it does not have any consistent effects across the entire sample. Instead, it pushes observed scores up or down randomly. This means that if we could see all of the random errors in a distribution they would have to sum to 0 -- there would be as many negative errors as positive ones. The important property of random error is that it adds variability to the data but does not affect average performance for the group. Because of this, random error is sometimes considered noise.


What is Systematic Error?

Systematic error is caused by any factors that systematically affect measurement of the variable across the sample. For instance, if there is loud traffic going by just outside of a classroom where students are taking a test, this noise is liable to affect all of the children's scores -- in this case, systematically lowering them. Unlike random error, systematic errors tend to be consistently either positive or negative -- because of this, systematic error is sometimes considered to be bias in measurement.



Reducing Measurement Error

So, how can we reduce measurement errors, random or systematic? One thing you can do is to pilot test your instruments, getting feedback from your respondents regarding how easy or hard the measure was and information about how the testing environment affected their performance. Second, if you are gathering measures using people to collect the data (as interviewers or observers) you should make sure you train them thoroughly so that they aren't inadvertently introducing error. Third, when you collect the data for your study you should double-check the data thoroughly. All data entry for computer analysis should be "double-punched" and verified. This means that you enter the data twice, the second time having your data entry machine check that you are typing the exact same data you did the first time. Fourth, you can use statistical procedures to adjust for measurement error. These range from rather simple formulas you can apply directly to your data to very complex modeling procedures for modeling the error and its effects. Finally, one of the best things you can do to deal with measurement errors, especially systematic errors, is to use multiple measures of the same construct. Especially if the different measures don't share the same systematic errors, you will be able to triangulate across the multiple measures and get a more accurate sense of what's going on

TYPES OF RELIABILITY AND VALIDITY
Instructions:   Match the definitions/examples with the correct type of reliability or validity. Technically there may be more than one correct answer for each definition/example, but each term is intended to be used only once.
___1. Test-retest reliability        
____7. Error
___2. Reliability                                                              
____8. Face validity
___3. Validity                                                                              
____9. Predictive validity
___4. Construct validity                                                
____10. Inter-rater reliability
___5. Criterion validity                                                  
____11. Discriminant validity
___6. Internal reliability/Inter-item consistency              
____12. Content validity

Social Science Research Methods
Distinction between empirical vs. non-empirical ways of knowing
 (UN) SCIENTIFIC THINKING Because someone told us that something is true. 2. Reasoning = A priori method (proposed by Charles Peirce): a person develops a belief by reasoning, listening to others’ reasoning, and drawing on previous intellectual knowledge – not based on experience or direct observation.  So now that we have a firm grounding in some of the basic theories and theorists within psychology, and a basic understanding of the multiple conceptions of personality and how it develops, the logical next step is to explore how we come to these conclusions with regard to models of personality and other psychological concepts. In other words, how do we scientifically ascertain whether these theories hold water, and how we can most accurately quantify and categorize human behavior, while still attempting to allow for the grey area of individual differences?

Well, Psychological Research is defined as the scientific exploration, designed to describe, predict, and control human behavior, and to understand the hard facts of the human mind across diverse psychological and cross-cultural populations.

So the first logical question before engaging in any kind of research is, how do we know something to be true in day to day life?

Well the first way is Authority, because someone told us that something is true. This could be any authority figure, like a professor, or someone in the media. Because someone told us it’s the truth, we often believe it to be so. Obviously, this has issues from a scientific perspective. When engaging in research we can’t rely on what others tell us alone, we need to have hard, replicable data to support it.

The second method is Reasoning. The main method of reasoning is the A priori method (proposed by Charles Peirce): where a person develops a belief by reasoning, listening to others’ reasoning, and drawing on previous intellectual knowledge – not based on experience or direct observation. While this might work when developing opinions in day-to-day life, we can’t say that this gives us hard facts that are generalizable to any population. Opinions derived from this method, however can be said to create a good starting point for research.
(Somewhat) scientific thinking 3. Experience
Empiricism:

knowing by direct observation or experience.

Subject to errors in thinking!
Threats to Internal Validity. So as we start to move towards scientific thinking we get to our third way of knowing, Experience, or knowing by direct observation. While this is all well and good, it is subject to errors in thinking. Indeed, our perceptions may differ between individuals when encountering the same experience, so we cant say that what we come up with will always be accurate. And, in fact there are several well known types of errors in experience-based conclusions and in psychological research in general.
Experience based errors in thinking The first one is an Illusory Correlation, or thinking that one has observed an association between events that either:
 (a) doesn’t exist,
(b) exists but is not as strong as is believed,
or (c) is in the opposite direction from what is believed

Confirmation Bias –  In psychology and cognitive science, confirmation bias (or confirmatory bias) is a tendency to search for or interpret information in a way that confirms one's preconceptions, leading to statistical errors.

Confirmation Bias – In psychology and cognitive science, confirmation bias (or confirmatory bias) is a tendency to search for or interpret information in a way that confirms one's preconceptions, leading to statistical errors. Finally, the last common error in psychological research is the Availability Heuristic is a mental shortcut that relies on immediate examples that come to mind. When you are trying to make a decision, a number of related events or situations might immediately spring to the forefront of your thoughts. As a result, you might judge that those events are more frequent and possible than others. You give greater credence to this information and tend to overestimate the probability and likelihood of similar things happening in the future.
The term was first coined in 1973 by psychologists Amos Tversky and Daniel Kahneman. They suggested that the availability heuristic occurs unconsciously and operates under the principle that "if you can think of it, it must be important." Things that come to mind more easily are believed to be far more common and more accurate reflections of the real world.

For example, after seeing several news reports about car thefts, you might make a judgment that vehicle theft is much more common than it really is in your area. This type of availability heuristic can be helpful and important in decision-making. When faced with a choice, we often lack the time or resources to investigate in greater depth. Faced with the need to an immediate decision, the availability heuristic allows people to quickly arrive at a conclusion.

While it can be useful at times, the availability heuristic can lead to problems and errors. Reports of child abductions, airplane accidents, and train derailments often lead people to believe that such events are much more typical than they truly are.
The scientific method: 4. Scientific Method:

Definition: A way of knowing characterized by the attempt to apply systematic, objective, empirical methods when searching for causes of natural events.

Probabilistic Statistical determinism: Based on what we have observed, is the likelihood of two events occurring together (whether causal, predictive, or simple relational) greater than chance?

Objectivity: without bias of the experimenter or participants.

Data-driven: conclusions are based on the data-- objective information.
Steps of The Scientific Method:
Ask a Question
Do Background Research
Construct a Hypothesis
Test Your Hypothesis by Doing an Experiment
Analyze Your Data and Draw a Conclusion
Communicate Your Results
Scientific Thinking in Research: So what are the CRITERIA FOR SCIENTIFIC METHOD?
Well the first is that its:
Empirical: All information is based on observation.
Once again, that it is Objective: Observations are verified by others.
For it to be organized it has to be Systematic: Observations are made in a step-by-step fashion.
It has to be Controlled: Potentially confusing factors are eliminated.
And finally, results must be made Public: Built on previous research, open to critique and replication, building towards theories.
ASSUMPTIONS ABOUT BEHAVIORS OR OBSERVATIONS:
Lawful: Every event can be understood as a sequence of natural causes and effects.
Determinism: Events and behaviors have causes.
Discoverability: Through systematic observation, we can discover causes – and work towards more certain and comprehensive explanations through repeated discoveries.
10.       Research Question:  Definition of a theory: A set of logically consistent statements about some psychological phenomenon that
a)    best summarizes existing empirical knowledge of the phenomenon
b)    organizes this knowledge in the form of precise statements of the relationship among variables
c)    provides a tentative explanation for the phenomenon
d)    serves as a basis for making predictions about behavior.
11.       From theory to actual research: Definition of a theory:
A set of logically consistent statements about some psychological phenomenon that
best summarizes existing empirical knowledge of the phenomenon
organizes this knowledge in the form of precise statements of the relationship among variables
provides a tentative explanation for the phenomenon
serves as a basis for making predictions about behavior.
Social Science Research Methods  Method: a technique used to analyze data. commonly, a method is aligned with a particular strategy for gathering data, as particular methods commonly require particular types of data. “method” is therefore commonly used to refer to strategies for both analyzing and gathering data.
Methodology: A body of practices, procedures, and rules used by researchers to offer insight into the workings of the world.
Insight: Evidence contributing to an understanding of a case or set of cases. Comparative-historical researchers are generally most concerned with causal insight, or insight into causal processes.
Comparative-historical analysis: A prominent research tradition in the social sciences, especially in political science and sociology. Works within this research tradition use comparative-historical methods, pursue causal explanations, and analyze units of analysis at the      meso- or macro-level.
Epistemology: A branch of philosophy that considers the possibility of knowledge and understanding. Within the social sciences, epistemological debates commonly focus on the possibility of gaining insight into the causes of social phenomena.
Positivism: An epistemological approach that was popular among most of the founding figures of the social sciences. It claims that the scientific method is the best way to gain insight into our world. Within the social sciences, positivism suggests that scientific methods can be used to analyze social relations in order to gain knowledge..

Ethnographic methods: A type of social scientific method that gains insight into social relations through participant observation, interviews, and the analysis of art, texts, and oral histories. It is commonly used to analyze culture and is the most common method of anthropology.
Case Study (Within-case methods): A category of methods used in the social sciences that offer insight into the determinants of a particular phenomenon for a particular case. For this, they analyze the processes and characteristics of the case.
 Ideographic explanation: Causal explanations that explore the causes of a particular case. Such explanations are not meant to apply to a larger set of cases and commonly focus on the particularities of the case under analysis.

BROAD CATEGORIES OF SOCIAL SCIENCE RESEARCH
Basic versus Applied Research
Goal of describing, predicting, & explaining fundamental principles
 of behavior vs. solving real-life problems

 Laboratory Research versus Field Research
Research in controlled laboratories vs. uncontrolled or real-life contexts

 Quantitative versus Qualitative  Research
Descriptive & inferential statistics vs. narrative analysis (e.g., case studies, observational research, interviews):

Statistical methods: The most common subtype of comparative methods. It operationalizes variables for several cases, compares the cases to explore relationships between the variables, and uses probability theory to estimate causal effects or risks. Within the social sciences, statistics uses natural variation to approximate experimental methods. There are diverse subtypes of statistical methods.
Comparative methods: Diverse methods used in the social sciences that offer insight through cross-case comparison. For this, they com- pare the characteristics of different cases and highlight similarities and differences between them. Comparative methods are usually used to explore causes that are common among a set of cases. They are commonly used in all social scientific disciplines.
Experimental methods: The most powerful method used in the social sciences, albeit the most difficult to use. It manipulates individuals in a particular way (the treatment) and explores the impact of this treat- ment. It offers powerful insight by controlling the environment, thereby      allowing researchers to isolate the impact of the treatment
How we conduct empirical research: Key Terms and Concepts
Variable: Something that the researcher/experimenter can measure.
Independent Variable: The variable the experimenter has control over, can change in some way to see if it has an effect on other variables in the study.
Dependent Variable: The variable that is measured to see if a change takes place:
Control Variable: The variable that is not manipulated that serves as a comparison group from the other variables in the study. This third variable is used to ensure that the independent variable, when manipulated is actually having an effect on the dependent variable. For example, if a similar change occurs in the control variable as the dependent variable, this indicates that the change may not be the result of the independent variable manipulation and may be a natural change in the variable.
In a experiment the researcher manipulates the independent variable to see if it has an effect on the dependent variable.

Field vs. Laboratory Research
Experimental Method: takes a representative sample from the larger population and attempts to generalize results of experiment on sample to the population from which it was derived.
Dependent Variable
Independent Variable
Natural sciences: In natural sciences, most experiments have been undertaken in carefully controlled laboratory conditions Seeking laws of physics, etc. Inanimate objects are unlikely to react differently in controlled and uncontrolled conditions The main goal is to avoid contaminating the experiment via the introduction of third variables. Far more exacting and finer measurement is possible under controlled conditions.
Laboratory experiments in social science: Closely controlled social science experiments do have the advantage of limiting the impact of third variables, but the unnatural situation presents problems for applying the findings outside the laboratory “Artificiality” of laboratory setting
Environmental Impact: Human behavior is sensitive to the environment within which it occurs People act differently in a laboratory than in the natural world Several characteristics of laboratories are thought to influence behavior The very third variables controlled in the lab may be the ones that determine behavior in the real world So, findings from laboratory experiments may only be valid for laboratory environments.
An example: New commercials are tested in controlled conditions  Eye tracking Liking for commercials  Influence on purchase interest May try to provide less artificial conditions for study Simulated living room Commercials that test high in lab experiments often do not work very well when used in real marketing campaigns.
So, experiments move out of the lab: Researchers want to retain some of the advantages of the experiment: Ability to manipulate/introduce the independent variable and to control how much of it is presented Time order—which comes first While sacrificing some of their ability to control third variables The goal is to improve our ability to generalize our findings to the real world.
Field Research: One way to do so is to carry out a ‘field experiment’ The researcher still manipulates the independent variable, but she does so within the natural world
For example, police try randomly stopping motorists and testing for alcohol to see if alcohol-related crashes decline.
What problems do we encounter? Naturalistic Observation: “Naturalistic observation involves methods designed to examine behavior without the experimenter interfering with it in any way.”   Prominent Example: Research of Margaret Mead Participant Observation:
Expense Tradeoff between extensive and intensive study Budget constraints on number of sites, etc. Access/permission Some research may present concerns to authorities, citizens, etc. Gain authorization/support prior to entering the field Maintain good relationships with community leaders, etc. throughout the intervention/research.
Capturing ‘natural experiments’ Sometimes unusual or unique events occur
Service disruptions
Political campaigns
School shooting
Institution of Wi-Fi in an area
“Natural experiments” Because most such events are unplanned, the ability to prepare for them is limited May keep a research group, materials and resources ready for certain types of events Must engage in ‘firehouse research’ gathering as much data as possible in a short time Inefficient, and may miss important data However, real-world events, etc. may provide very valuable data—may have both internal and external validity
So what can we conclude? When understanding the distinction between laboratory and field research and decide which one to choose. To do this we much clarify the tradeoffs:
Reliability -  Reliability is the degree to which the findings from a study produce stable and consistent results. For example, if we looked at the same group of people in the exact same way at a different time, would be draw the same conclusions?
Validity - Validity refers to how well does the type of research we engage in measures what it is purported to measure. (e.g., when we try to understand the behaviors of North American people in general, are college students a good group to look at and generalize?)
Important Types of Validity  Internal Validity - One of the keys to understanding internal validity (IV) is the recognition that when it is associated with experimental research it refers both to how well the study was run (research design, operational definitions used, how variables were measured, what was/wasn't measured, etc.), and how confidently one can conclude that the change in the dependent variable was produced solely by the independent variable and not extraneous ones. In group experimental research, IV answers the question, "Was it really the treatment that caused the difference between the means/variances of the subjects in the control and experimental groups?" Similarly, in single-subject research (e.g., ABAB or multiple baseline), IV attempts to answer the question, "Do I really believe that it was my treatment that caused a change in the subject's behavior, or could it have been a result of some other factor?" In descriptive studies (correlational, etc.) internal validity refers only to the accuracy/quality of the study (e.g., how well the study was run-see beginning of this paragraph).
External Validity - The extent to which a study's results (regardless of whether the study is descriptive or experimental) can be generalized/applied to other people or settings reflects its external validity. Typically, group research employing randomization will initially possess higher external validity than will studies (e.g., case studies and single-subject experimental research) that do not use random selection/assignment.
Reliability and Validity and Field Vs. Laboratory Research:  Field Research: High internal validity, low external validity, Low reliability.  Laboratory Research: High external validity, low internal validity, High reliability.
Comparative- Historical methods: Since the rise of the social sciences, researchers have used comparative- historical methods to expand insight into diverse social phenomena and, in so doing, have made great contributions to our understanding of the social world.
Comparative-Historical Analysis: Mahoney and Rueschemeyer (2003) refer to it as comparative-historical analysis in recognition of the tradition’s growing multidisciplinary character. In addition to sociology, comparative-historical analysis is quite prominent in political science and is present—albeit much more marginally—in history, economics, and anthropology.
4 types of comparative-historical research:
• Historical Events Research –focuses on one short historical period (1 case, 1 time period)
• Historical Process Research –traces a sequence of events over a number of years (1 case, many time periods)
• Cross-sectional Comparative Research comparing data from one time period between two or more nations (many cases, 1 time period)
• Comparative Historical Research – longitudinal comparative research (many cases)
Comparative and Historical Research by number of cases and length of time studied:
How do we understand Comparative Historical Research? As the Venn diagram in this Figure depicts, comparative-historical analysis has four main defining elements. Two are methodological, as works within the research tradition employ both within-case methods and comparative methods. Comparative-historical analysis is also defined by epistemology. Specifically, comparative-historical works pursue social scientific insight and therefore accept the possibility of gaining insight through comparative-historical and other methods. Finally, the unit of analysis is a defining element, with comparative-historical analysis focusing on more aggregate social units.
Historical methods, also known as historiography, are the most common analytic techniques used in the discipline of history. They are generally used to explore either what happened at a particular time and place or what the characteristics of a phenomenon were like at a particular time and place.
Comparative-Historical Methods in Comparative Perspective: Similar to statistical and experimental methods, comparative-historical methods employ comparison as a means of gaining insight into causal determinants. Similar to ethnographic and historical methods, comparative-historical methods explore the characteristics and causes of particular phenomena.   Comparative-historical analysis, however, does not simply combine the methods from other major methodological traditions—none of the major comparative methods is very common in comparative-historical analysis.  As a consequence, comparative-historical researchers commonly avoid statistics and simply focus on causal processes. Additional reasons for the limited use of statistical comparison within the comparative-historical research tradition include the limited availability of historical data needed for appropriate statistical analyses and the small number of cases analyzed by comparative-historical researchers.
So what does this tool-kit look like? •  Historically specific. It is likely to be limited to the specific time(s) and place(s) studied, like traditional historical research.
• Narrative. It researches a story involving specific actors and other events occurring at the same time (Abbott, 1994:102), or one that takes account of the position of actors and events in time and in a unique historical context (Griffin, 1992).
• Inductive. The research develops an explanation for what happened from the details discovered about the past.
• Case-oriented. It focuses on the nation or other unit as a whole.
• Holistic. It is concerned with the context in which events occurred and the interrelations among different events and processes: “how different conditions or parts fit together” (Ragin, 1987:25–26).

• Conjunctural. This is because, it is argued, “no cause ever acts except in complex conjunctions with others”  (Abbot, 1994:101).

• Temporal. It becomes temporal by taking into account the related series of events that unfold over time.


Comparative Historical “toolkit” Besides comparative methods, comparative-historical scholars employ several different types of within-case methods: Ethnography
Historical Methods
Idiographic Methods
Nomothetic Explanations
Historical Events Research & Event-Structure Analysis: It often utilizes a process known as Historical Events Research.
Historical events research is research on past events that does not follow processes for some long period of time—that is basically crosssectional—is historical events research rather than
historical process research.

Event Structure Analysis is a qualitative approach that relies on a systematic coding of key events or national characteristics to identify the underlying structure of action in a chronology
of events.
• Oral History can be useful for understanding historical events that occurred within the lifetimes of living individuals.
Steps of Event-Structure Analysis: An event structure analysis requires several steps:
(Example: an auto manufacturing plant that produces SUVs closes in your home town).
• Classifying historical information into discrete events: plant closes, worker strike, corporate buy-out, gas prices increase.
• Ordering events into a temporal sequence: 1. gas prices increase 2. Corporate buy-out 3. Worker strike 4. Plant closes.
• Identifying prior steps that are prerequisites for subsequent events: Oil embargo, political
More on Oral Histories: Another way to get a very rich understanding of how individuals experienced historical events is through oral history.
This requires intensive interviewing of individuals, which is an obtrusive measure. However, there are many oral histories
archived at university libraries and the Library of Congress. You might get lucky and find out that oral histories of community members were conducted during the time of the plant closing. You could qualitatively analyze the oral histories for information about how people perceived the plant closing and its importance to the community.
Historical Process Research: Most likely to be qualitative and case-oriented (traditional history of country X)
• Case could be anything from
– A small community (village)
- Even looking at event’s across the whole world!
Cross-Sectional Comparative Research: Comparisons between countries during one time period can help social scientists identify the limitations of explanations based on single-nation research. These comparative studies may focus on a period in either the past or the present. Could be quantitative (!!!!) or qualitative.
Example: comparing the perceptions of events taking place in the recent conflicts between Ukraine and Russia from media reports from both sides.
Cross-Sectional vs. Longitudinal: Cross-Sectional Research – Takes place by looking at an event at one time point.
Longitudinal Research – Takes places by looking at several cross-sections to gain an understanding of the evolution of events and how they change over time.
These types of research are often utilized in comparative historical analysis but are also utilized in experimental and quasi-experimental research among most of the types of research covered by this course!
Problems With Historical Research: Reliance on Secondary vs. Primary sources of information (data).
Primary Sources – Collecting data from the individual source such as being present when an event (such as an important election) takes place. This is more verifiable because we are seeing the data for what it is.

Secondary Sources – Collecting data from others who have already collected the data such as news papers, magazines, and interviews.
These sources of data are prone to the bias of the source, therefore the data may be somewhat inaccurate.
Example: Hurst Publications and the Spanish American War
The Spanish-American War is often referred to as the first "media war." During the 1890s, journalism that sensationalized—and sometimes even manufactured—dramatic events was a powerful force that helped propel the United States into war with Spain. Led by newspaper owners William Randolph Hearst and Joseph Pulitzer, journalism of the 1890s used melodrama, romance, and hyperbole to sell millions of newspapers--a style that became known as yellow journalism.

The term yellow journalism came from a popular New York World comic called "Hogan's Alley," which featured a yellow-dressed character named the "the yellow kid." Determined to compete with Pulitzer's World in every way, rival New York Journal owner William Randolph Hearst copied Pulitzer's sensationalist style and even hired "Hogan's Alley" artist R.F. Outcault away from the World. In response, Pulitzer commissioned another cartoonist to create a second yellow kid. Soon, the sensationalist press of the 1890s became a competition between the "yellow kids," and the journalistic style was coined "yellow journalism."

Yellow journals like the New York Journal and the New York World relied on sensationalist headlines to sell newspapers. William Randolph Hearst understood that a war with Cuba would not only sell his papers, but also move him into a position of national prominence. From Cuba, Hearst's star reporters wrote stories designed to tug at the heartstrings of Americans. Horrific tales described the situation in Cuba--female prisoners, executions, valiant rebels fighting, and starving women and children figured in many of the stories that filled the newspapers. But it was the sinking of the battleship Maine in Havana Harbor that gave Hearst his big story--war. After the sinking of the Maine, the Hearst newspapers, with no evidence, unequivocally blamed the Spanish, and soon U.S. public opinion demanded intervention.

Today, historians point to the Spanish-American War as the first press-driven war. Although it may be an exaggeration to claim that Hearst and the other yellow journalists started the war, it is fair to say that the press fueled the public's passion for war. Without sensational headlines and stories about Cuban affairs, the mood for Cuban intervention may have been very different. At the dawn of the twentieth century, the United States emerged as a world power, and the U.S. press proved its influence. Adapted from:  http://www.pbs.org/crucible/frames/_journalism.html

View Documentary on the Spanish American War: https://www.youtube.com/watch?v=8g8NpQsmxj4
Yellow Journalism:  https://www.youtube.com/watch?v=0wFrAny77UY.
• Documents and other evidence may have been lost or damaged.
• Available evidence may represent a sample biased toward more newsworthy figures.
• Written records will be biased toward those who were more prone to writing.
• Feelings of individuals involved in past events may be hard, if not impossible, to reconstruct.
Case Study Method: “An empirical inquiry about a contemporary phenomenon (e.g., a “case”), set within its real-world context—especially when the boundaries between phenomenon and context are not clearly evident (Yin, 2009a, p. 18; SagePub, 2014)’
Assumptions: “Among other features, case study research assumes that examining the context and other complex conditions related to the case(s) being studied are integral to understanding the case(s) (SagePub, 2014).”
When to use the Case Study Method: “First and most important, the choices among different research methods, including the case study method, can be determined by the kind of research question that a study is trying to address (e.g., Shavelson & Towne, 2002, pp. 99–106, SagePub, 2014).”
“Second, by emphasizing the study of a phenomenon within its real-world context, the case study method favors the collection of data in natural settings, compared with relying on “derived” data (Bromley, 1986, p. 23, SagePub, 2014)”
Third, the case study method is now commonly used in conducting evaluations (SagePub, 2014).
Concerns and Problems: Often considered the “method of last resort”
Lack of trust in procedures and processes
Inability to generalize results
When done poorly problems increase.
Three Steps in designing a “Case”:
Define the Case
Select one of 4 types of case study designs
Use theory in design work
Data Collection:
 1. Direct observations (e.g., human actions or a physical environment)
2. Interviews (e.g., open-ended conversations with key participants)
3. Archival records (e.g., student records)
4. Documents (e.g., newspaper articles, letters and e-mails, reports)
5. Participant-observation (e.g., being identified as a researcher but also filling a real-life role in the scene being studied)
6. Physical artifacts (e.g., computer downloads of employees’ work)
Evidence from multiple sources:
Triangulation
Literature Review
Direct Observation
* You are always better off using multiple rather than single sources of evidence.
Case Study Protocol: The typical protocol consists of a set of questions to be addressed while collecting the case study data (whether actually taking place at a field setting or at your own desk when extracting information from an archival source).
Importantly, the questions in the protocol are directed at the researcher, not at any field participant. In this sense, the protocol differs entirely from any instrument used in a conventional interview or survey. The protocol’s questions in effect serve as a mental framework,
Collect Data for Rival Explanations: A final data collection topic stresses the role of seeking data to examine rival explanations. The desired rival thinking should draw from a continual sense of skepticism as a case study proceeds.
During data collection, the skepticism should involve worrying about whether events and actions are as they appear to be and whether participants are giving candid responses.
Having a truly skeptical attitude will result in collecting more data than if rivals were not a concern.
Presenting your Case: You need to present the evidence in your case study with sufficient clarity (e.g., in separate texts, tables, and exhibits) to allow readers to judge independently your later interpretation of the data.
Ideally, such evidence will come from a formal case study database that you compile for your files after completing your data collection.
Case Study Data Analysis: Whether using computer software to help you or not, the researcher will be the one who must define the codes to be used and the procedures for logically piecing together the coded evidence into broader themes—in essence creating your own unique algorithm befitting your particular case study. The strength of the analytic course will depend on a marshaling of claims that use your data in a logical fashion.
Your analysis can begin by systematically organizing your data (narratives and words) into hierarchical relationships, matrices, or other arrays (e.g., Miles & Huberman, 1994).
Techniques:
 Pattern-Matching
Open-Ended Questions
Time-Series-Like Analysis
If selecting your case(s) to be studied is the most critical step in doing case
study research, analyzing your case study data is probably the most troublesome.

Much of the problem relates to false expectations: that the data will somehow
“speak for themselves,” or that some counting or tallying procedure will be sufficient to produce the main findings for a case study. Wrong. Instead, consider
the following alternatives.

You actually made some key assumptions for your analysis when you defined
your research questions and your case. Was your motive in doing the case study
mainly to address your research questions? If so, then the techniques for analyzing
the data might be directed at those questions first. Was your motive to derive
more general lessons for which your case(s) are but examples? If so, your analysis
might be directed at these lessons. Finally, if your case study was driven by a
discovery motive, you might start your analysis with what you think you have
discovered.

Now comes a “reverse” lesson. Realizing that key underlying assumptions for
later analysis may in fact have been implicit at the initial stages of your case
study, you could have anticipated and planned the analytic strategies or implications when conducting those initial stages. Collecting the actual data may lead to changes in this plan, but having an initial plan that needs to be revised (even drastically) may be better than having no plan at all.

For instance, one possibility is to stipulate some pattern of expected findings
at the outset of your case study. A pattern-matching logic would later enable you
to compare your empirically based pattern (based on the data you had collected)
with the predicted one. The prediction in a community study might have stipulated that the patterns of outcomes in many different economic and social sectors (e.g., retail sales, housing sales, unemployment, and population turnover) would be “catastrophically” affected by a key event— the closing of a military base in a small, single-employer town (Bradshaw, 1999).

The analysis would then examine the data in each sector, comparing pre-post
trends with those in other communities and statewide trends. The pattern-matching
results should be accompanied by a detailed explanation of how and why the base closure had (or had not) affected these trends. By also collecting data on and then examining possible rival explanations (e.g., events co-occurring with the key event or other contextual conditions), support for the claimed results would be
strengthened even further.

Second, a case study may not have started with any predicted patterns but in fact
may have started with an open-ended research question that would lead to the use of an explanation-building technique. The purpose of the case study here would be to build an explanation for the issue and then again to deliberately entertain rival explanations.

A third technique mimics the time-series analyses in quantitative research. In
case study research, the simplest time series can consist of assembling key events
into a chronology. The resulting array (e.g., a word table consisting of time and
types of events as the rows and columns) may not only produce an insightful
descriptive pattern but also may hint at possible causal relationships, because any
presumed causal condition must precede any presumed outcome condition. Assuming again the availability of data about rival hypotheses, such information
would be used in examining the chronological pattern. When the rivals do not fit
the pattern, their rejection considerably strengthens the basis for supporting your
original claims.

If the case study included some major intervening event in the midst of the
chronological sequence, the array could serve as a counterpart to an interrupted
time series in experimental research. For instance, imagine a case study in which
a new executive assumed leadership over an organization. The case study might
have tracked the production, sales, and profit trends before and after the executive’s ascendance. If all the trends were in the appropriate upward direction, the case study could begin to build a claim, crediting the new leader with these
accomplishments. Again, attending to rival conditions (such as that earlier policies
might have been put into place by the new executive’s predecessor) and
making them part of the analysis would further strengthen the claim.
What about when quantitative data is available? Correlations between single case studies can be used only if there is sufficient data to run a statistical analysis.
Meta Analysis: A study of multiple Case Studies: The logic for such a cross-case synthesis emulates that used in addressing whether the findings from a set of multiple experiments—too small in number to be made part of any quantitative meta-analysis (a study of the results of other studies)—support any broader pattern of conclusions.
The replication or corroboratory frameworks can vary. In a direct replication,the single cases would be predicted to arrive at similar results.
Can we generalize at all from a Case Study? To the extent that any study concerns itself with generalizing, case studies tend to generalize to other situations (on the basis of analytic claims), whereas surveys and other quantitative methods tend to generalize to populations (on the basis of statistical claims).
In summary, to the extent that any study concerns itself with generalizing, case studies tend to generalize to other situations (on the basis of analytic claims), whereas surveys and other quantitative methods tend to generalize to populations (on the basis of statistical claims).
Quantitative and Qualitative Research Methods:
Quantitative Research: This type of research is charged with quantifying (measuring and counting) and subscribes to to a particular empirical approach to knowledge, believing that by measuring accurately enough we can make claims about the object at study.
Qualitative Research: This type of research is charged with the quality or qualities of an experience or phenomenon. Qualitative research rejects the notion of their being a simple relationship between our perception of the world and the world itself, instead arguing that each individual places different meaning on different events  or experiences and that these are constantly changing. Qualitative research generally gathers text based data through exercises with a small number of participants, usually semi structured or unstructured interviews.
Example of Survey Used in Quantitative Research: So as we can see here, this is an example of the, what are known as likert-type scales which are used to numerically quantify a research participant’s response. Each response is assigned a number which can then be entered into a statistical analysis to utilize mathematical principles to generate understandable results based on a smaller sample of people which is taken from the larger population.
Relationship between the Sample and the Population: So when we think about this sample vs. population issue, what are we talking about, well essentially we are saying that since we cant run an analysis on an entire population, a feat that would be nearly impossible we must collect data from the population that is large enough to use the inferential statistical methods, based on rules of probability to make a generalization to what is occurring in the population. So in the case of heavy metal musical preference and dysphoric mood, we are essentially stating that because we can not effectively sample all heavy metal listeners in the world and give them our survey, instead we want to collect a representative sample from the larger population.
When to use Quantitative Research:
 1) When we are looking for a numerical answer
2) When we want to study numerical change
3) When we want to find out about the state of something or to explain a phenomena
4) When we want to test a hypothesis
Advantages and Disadvantages:
QuantitativeQuantitative Advantages:
ü  Concise
ü  Accurate
ü  Strictly Controlled
ü  Replicable
ü  Can indicate causation
ü  Ideally is objective
ü  Quantitative Disadvantages:
ü  Limited understanding of individuality
ü  Groups people into categories
ü  Can be accused of oversimplifying human nature


1. The first situation where quantitative research will fail is when we want
to explore a problem in depth. Quantitative research is good at providing
information in breadth, from a large number of units, but when we
want to explore a problem or concept in depth, quantitative methods
can be too shallow. To really get under the skin of a phenomenon, we
will need to go for ethnographic methods, interviews, in-depth case
studies and other qualitative techniques.

2. We saw above that quantitative research is well suited for the testing of
theories and hypotheses. What quantitative methods cannot do very
well is develop hypotheses and theories. The hypotheses to be tested may
come from a review of the literature or theory, but can also be developed
by using exploratory qualitative research.

3. If the issues to be studied are particularly complex, an in-depth qualitative
study (a case study, for example) is more likely to pick up on this
than a quantitative study. This is partly because there is a limit to how
many variables can be looked at in any one quantitative study, and partly
because in quantitative research the researcher defines the variables to be
studied herself, while in qualitative research unexpected variables may
emerge.

4. Finally, while quantitative methods are best for looking at cause and
effect (causality, as it is known), qualitative methods are more suited to
looking at the meaning of particular events or circumstances.

Qualitative Research: Focus on “language rather than numbers”
“Embraces “intersubjectivity” or how people may construct meaning…”
Focus on the individual and their real lived experience.
Advantages and Disadvantages: Qualitative:


Qualitative Advantages:
Appreciates research participant’s individuality
Provides insider view of research question
Less structured than quantitative approach
Qualitative Disadvantages:
Not always appropriate to generalize results to larger population
Time consuming
Difficult to test a hypothesis


Qualitative Research in Psychology: “Today, a growing number of psychologists are re-examining and re-exploring qualitative methods for psychological research, challenging the more traditional ‘scientific’ experimental approach”
When to use Qualitative Research
 Analysis, or Thematic Analysis (the terms are frequently used interchangeably and generally mean much the same), is particularly useful for conceptual, or thematic, analysis or relational analysis. It can quantify the occurrences of concepts selected for examination (Wilkinson & Birmingham, 2003).
Grounded Theory - is frequently considered to offer researchers a suitable qualitative method for in-depth exploratory investigations. . It is a rigorous approach which provides the researcher with a set of systematic strategies and assumes that when investigating social processes qualitatively from the bottom up there will be an emergence of a working theory about the population from the qualitative data (Willig, 2008, pp. 44).
Discursive psychology and Discourse Analysis: Discourse Analysis: The discursive approach looks to verbal behavior as a more direct means of uncovering underlying cognitions (Harré,1995) rather than assigning a numerical value ‘score’ or scale to a behavior. This approach takes the view that interpretation and empathy are involved in attempting to understand human behavior. Self-report, from people being studied, can then become a valuable resource in its own right.
What do we do if we want the best of both worlds?
 Mixed-Methods Designs - questionnaire) and qualitative (for example, a number of case studies) methods. Mixed-methods research is a flexible approach, where the research design is determined by what we want to find out rather than by any predetermined epistemological position. In mixed-methods research, qualitative or quantitative components can predominate, or both can have equal status. http://www.sagepub.com/upm-data/3869_muij.pdf
Statistics:  are a set of mathematical procedures for summarizing and interpreting observations. These observations are typically numerical or categorical facts about specific people or things, and they are usually referred to as data.
Central Tendency and DispersionCentral:  Tendency  Dispersion  Variability   Range  Sigma.
Shape of Distribution:
Rectangular Distribution
Bimodal Distribution
Normal Distribution
Inferential Statistics: Decisions about what to conclude from a set of research findings (sample date) need to be made in a logical, unbiased fashion, using mathematical principles to make presumptions about the population.
Null Hypothesis
Alternative Hypothesis
Alpha level
Probability Theory: From the classical perspective, the probability of an event is a very simple thing: It is (a) the number of all specific outcomes that qualify as the event in question divided by (b) the total number of all possible outcomes.
Type I Error: incorrectly rejecting the null hypothesis when it is, in fact, correct.
Type II Error: occurs when we fail to reject an incorrect null hypothesis.
Effect Size and Significance TestingEffect Size: the magnitude of the effect in which they happen to be interested.
Important Statistical Notation
Internal Validity – How valid a study is: the more we avoid confounding variables (variables that could interfere in attaining the true results of our study) the higher the level of internal validity.
Attrition Threat – In a longitudinal study where you need to measure responses from a participant at multiple time points, attrition refers to when participants drop out or leave a study, causes problems as it reduces available data, in particular when it is different across groups.
**Confounding Variables specially whether it avoids confounding (more than one possible independent variable [cause] acting at the same time). The less chance for confounding in a study, the higher its internal validity is.
Therefore, internal validity refers to how well a piece of research allows you to choose among alternate explanations of something. A research study with high internal validity lets you choose one explanation over another with a lot of confidence, because it avoids (many possible) confounds.
Attrition Threat – In a longitudinal study where you need to measure responses from a participant at multiple time points, attrition refers to when participants drop out or leave a study, causes problems as it reduces available data, in particular when it is different across groups. When participants drop out or leave a study, causes problems as it reduces available data, in particular when it is different across groups.

“The differential loss of individuals from treatment and/or comparison groups. This is often a problem when research participants are volunteers. Volunteers may drop out of the study if they find it is consuming too much of their time. Others may drop out if they find the task to be too arduous.” (Source: Internet presentation)
 It also can be thought of as“…refer[ing] to the drop out of subjects in an experiment…if people drop out of an experiment they do so for a reason…[perhaps indicates] a problem with either a noxious or an inadequate independent variable” (Kroth, 1988, p. 60).
Attrition: How do we fix it?
The best way to account for this is to randomly assign participants to two groups, one treatment group & one control group.
Researchers can expect similar effects in treatment group versus control group over time.
Treatment Diffusion - This occurs when a comparison group learns about the program either directly or indirectly from program group participants. In a school context, children from different groups within the same school might share experiences during lunch hour. Or, comparison group students, seeing what the program group is getting, might set up their own experience to try to imitate that of the program group. In either case, if the diffusion of imitation affects the posttest performance of the comparison group, it can have an jeopardize your ability to assess whether your program is causing the outcome. Notice that this threat to validity tend to equalize the outcomes between groups, minimizing the chance of seeing a program effect even if there is one.  References
http://www.socialresearchmethods.net/kb/intsoc.php

Treatment Diffusion: How do we fix it? By isolating one group of participants from the other, this mitigates the likelihood that demand characteristics of the contrast between treatment conditions could interfere with the results and threaten the internal validity of the study in this way.
Testing Threat -- Changes in a test score due to taking it more than once, through familiarization, recall.        Also called “pretest sensitization,” this refers to the effects of taking a test upon performance on a second testing. Merely having been exposed to the pretest may influence performance on a posttest. Testing becomes a more viable threat to internal validity as the time between pretest and posttest is shortened.” (Source: Internet Presentation)

The impact of “taking a test, and whether or not any special circumstances existed which may have influenced sores on a subsequent test…taking a test may cause anxiety, stress or self-consciousness the first time one takes it, but such emotions may be absent at posttest time [causing an improvement that is not simply due to the treatment, or IV]”
(Kroth, 1988, p. 60).
Testing Threat: How do we fix it?

Randomly assign participants to two groups, one treatment group & one control group.

Researchers can expect similar effects in treatment group versus control group over time.
History Threat -- Events outside experimental manipulation influence collective or individual participation
“Many change-producing events may have occurred between Observation 1 and Observation 2 “ (Source: Internet Presentation)
“History is more …[likely] the longer the lapse between the pretest and posttest.” (Source: Alliant International University - Internet Presentation)

This was described best by Kroth, 1988 who stated:

“Things happen between a pretest and posttest, especially when there may be 6 months to a year between the two…These …events, or outside influences [confounding variables] may cause changes we see between pretest time and posttest time may be due to these outside factors, not the independent variable” (Kroth, 1988, p. 60)
History Threat: How do we Fix it?
Consistent methods / measurements in longitudinal studies from one time point to the next.
Instrumentation Threat -- Changes in the measurement or procedure over the course of a study (e.g., with observations, coding systems over time, drift systematically). Changes in the measurement or procedure over the course of a study (e.g., with observations, coding systems over time, drift systematically).

“Changes in the way a test or other measuring instrument is calibrated that could account for results of a research study (different forms of a test can have different levels of difficulty). This threat typically arises from unreliability in the measuring instrument. Can also be present when using observers.” (Source: Internet Presentation)

Another possibility is whether the instrument tends to…[vary] all by itself, registering…a spurious increase or decrease which is unrelated to what the good old independent variable is trying to do” (Kroth, 1988, p. 60).
Instrumentation threat: How do we fix it?
Randomly assign participants to two groups, one treatment group & one control group.
Researchers can expect similar effects in treatment group versus control group over time.
Or, use tests rather than observation, ratings, or interviews.
Selection Threat -- Differences in groups that exist before treatment or intervention begins. Especially problematic when participants selected for belonging to a specific group. Differences may be due to initial differences in groups – not treatment or intervention.
“This can occur when intact groups are compared. The groups may have been different to begin with. If three different classrooms are each exposed to a different intervention, the classroom performances may differ only because the groups were different to begin with.” (Source: Internet Presentation)
“…the biases which result from comparing one group of complex and unique people to another group of complex and unique subjects” (Kroth, 1988, p. 60).
Selection Threat: How do we fix it?
Randomly assign participants to two groups, one treatment group & one control group.
Researchers can expect similar effects in treatment group versus control group over time.
Maturation Threat - Normal developmental changes in participants between pre and post tests. Gains/Losses over time may be due to normal maturation.
“Changes in physical, intellectual, or emotional characteristics, that occur naturally over time, that influence the results of a research study.” (Source: Internet Presentation)
 “Internal changes occurring between the pretest … and posttest… which can cause changes in the dependent variable. These are internal changes, like becoming tired or heavier or weaker, less anxious, or more inattentive with the passage of time.” (Kroth, 1988, p. 60)
Maturation Threat: How do we fix it?
Randomly assign participants to two groups, one treatment group & one control group.
Researchers can expect similar effects in treatment group versus control group over time.
Inequitable Treatments Threat -- When participants in one group outperform or underperform relative to another group as a result of study expectation of a certain performance.
Inequitable Treatments Threat: How do we fix it?
Isolating groups.
Blind studies.
Threats to Internal Validity
 Special Treatment Threat -- When groups are treated differently as a result of being in the study – or one group is treated differently from another group b/c of their treatment status.
Special Treatment Threat: How to fix it?
Training of treatment administrators and personnel associated with the study

Statistical Regression Threat -- The tendency to drift towards the mean as one takes a test multiple times. Especially problematic when choosing “extreme” scores which can “regress” toward the mean due to influences other than treatment.
 “Occurs when individuals are selected for an intervention or treatment on the basis of extreme scores on a pretest. Extreme scores are more likely to reflect larger (positive or negative) errors in measurement (chance factors). Such extreme measurement errors are NOT likely to occur on a second testing.” (Source: Internet Presentation)
 This “refers to the fact that a group at the extreme ends of a normal curve has a greater probability of being “closer” to the mean of that curve the next time the group is measured” (Kroth, 1988, p. 60).
Statistical Regression Threat: How to fix it?
The problem is with single assessments. Two randomly assigned groups should equally regress toward the mean.
Randomly assign participants to two groups, one treatment group & one control group.
Researchers can expect similar effects in treatment group versus control group over time.
Interaction with Selection (with other factors)
Also known as: Additive and interactive effects of threats to internal validity.  --
When certain participants (selection) experience drop out, historical effects, testing issues as a group.

One research scholar, Vanbeek states: “No threat is completely isolated, often one threat can lead to another or enhance another, often in vicious cycles.”
Interaction with selection with other factors threat: How to fix it?
Randomly assign participants to two groups, one treatment group & one control group.

Researchers can expect similar effects in treatment group versus control group over time
Ambiguous Directionality -- When the independent variable is not manipulated, the direction of the influence is not always clear (e.g., impact of a therapist empathy on client outcome – does therapist get warmer b/c client improves or vice versa?)
Ambiguous Directionality: How to fix it?
Unless the temporal order is clear, the directionality is difficult to determine.
Be clear in statements of causality and describe when causality is unclear.

Threats to External Validity
What is a threat to External Validity?
This is when the results found are not generalizable to the population of interest.
You may see an effect, but its not the effect you wanted to measure or it doesn’t generalize outside of the small group you are studying!

Selection (Interaction of Causal Units): This is distinguished from selection threat to internal validity. This is the selection threat to external validity. You can think of this one as the interaction of causal units. This is a threat when the participants used in a study are of a specific type such that the results found with them might not apply (be generalizable) to other participants, but researchers imply this.
Example: An article entitled “Depression among Asian Americans” studies depression, but the participants were only from Japan and China, and were upper-middle class. This is a problem because researchers are trying to find out information about Asian Americans in general, but only used Japanese and Chinese upper SES people. The results are not generalizable to all Asian Americans, and depression may work differently with other types of Asians and Asians of other SES’s.  This WOULD NOT be a problem, however, if the article was called “Depression among wealthy Japanese and Chinese Americans”, since researchers then don’t intend for the results to be generalized to all Asian Americans.

Remedy: Either try to have participants that are very representative (don’t miss out on certain people by only doing college students or white people or high SES etc),  or don’t try to generalize to other people.
Risk:  EXTREMELY HIGH. This happens ALL THE TIME in studies.
Look in:  PARTICIPANTS to see who was used for the study. Then look at the title, abstract, and/or discussion to make sure you know who they are trying to generalize the results to and talk about.
Interaction of the causal relationship over treatment variations:  This is a problem when the research study has a TREATMENT, but a weird variation of the treatment is used, and results found may not be generalizable to other variations of that treatment.

Example: Researchers test a manualized CBT treatment. This CBT treatment is meant to be 12 weeks, but researchers did a 7 week version of it, so the results may be different. Effects found with this treatment are not necessarily the same at results that would have been found with the treatment they were trying to test (the 12 week treatment). OR Researchers combined CBT with another treatment, which may find different results. OR They only used part of the treatment, such as the Cognitive portion of the treatment.
Remedy: Use the whole treatment as it was meant to be given.
Risk: Low. Researches would likely admit that they administered a treatment differently than it was supposed to be administered.
Look in: Methods, measures, and Limitations
Interaction of the causal relationship with outcomes: This is similar to the idea of mono operation bias, but just for the DV.  This is a threat when the outcome (usually just one Dependent Variable) was measured a certain way (usually with only one measure). It is a problem when measuring the outcome in a different way could have given different results.
Example: Unintelligent researchers wanted to measure if taking Prozac effects basketball players’ performance. They give them the drug, then measure the percentage of shots they make during a game, and compare this to their old shot percentage. The effects found may have been different if they had measured a different outcome than shot percentage. If they had included assists, blocks, passes, coaches ratings of them, ect, the different results may have been found about the players performance.
Remedy: Use more than one way to measure the DV.
            Risk: Medium. Look for just one DV with just one measure that could have been measured differently. (If the one outcome was height and they used a ruler, do NOT apply this threat, because there is no other way to measure height and get different results).
            Look in: Methods
Interaction of causal relationship w/settings: This is a threat when the research was done in a particular setting (environment), and results may not be generalizable to a different setting. By setting we are talking about specific research setting such as laboratory, school, home, internet, ect. NOT geographic location like Texas or New York. Geographic locations are more of a selection threat because we are talking about different types of people with results that may not generalize to other types of people. Here we are talking about a setting where results may not generalize to other settings.

Example: Researchers wanted to test children’s reactions to their parents instructions. They tested this in a room at a local elementary school, asking parents to tell their kids to complete certain tasks. The kids were super compliant. HOWEVER, the setting here was a school, where kids are accustomed to being compliant. If the study were done at children’s homes, different results may have been found. You cannot generalize this to how children’s reaction to their parents instructions in general, only how they react to them at a school.
Remedy: Try to do studies in the appropriate types of settings you are trying to generalize to or in more than one setting.
Risk: HIGH. Only 1 setting is used in most studies.
Look in: Methods in Procedure
Context-dependent Mediation: This is a threat when a mediating variable may not mediate the same way in a different situation (different setting, or participants, or treatment, or task, ect). This is only when the study has a mediating variable, and this relationship (the variable mediating the relationship between two other variables) may not be the same in a different situation.

Example: Suppose a study used only males to study depression. They found that owning more video games makes them super excited and causes sleeplessness, which later causes depression in these people. So, sleeplessness mediates the relationship between video games and depression. However, this may not be the case if females were used. Owning video games may bore them, which may make them depressed, but it doesn’t cause sleeplessness because they aren’t excited about the games. So the video games cause depression for them, but this isn’t mediated by sleeplessness.  Or if different video games were used, then sleeplessness may not mediate the relationship the same way.
Remedy: Try to use generalizable situations.
Risk: RARE. The study must have a mediating variable in it.
Look in: Methods, Results, and Conclusions
Interaction of History and Treatment:   This is different than the Internal Validity threat History. That is when an event affects participants and causes the results found. THIS one is when results found from a study are not generalizable to other time periods. This usually has to do with whole eras. It mostly applies to old studies, when times were different, and results found may not be the same today.
Example: Researchers in the 40’s found that Japanese-Americans felt very low allegiance to the USA, and reported that their ethnic identity caused them great distress. These results were gathered during World War II, when Japanese-Americans were hated and oppressed by many people in the US in an extreme way. These results do not generalize to modern day as much. These results do not hold in the same way they once did, and Japanese-Americans may feel very differently about their allegiance to the USA, or about their distress over ethnic identity.


RELIABILITY
Reliability refers to the consistency of a measure. A measure is considered reliable if we get the same result repeatedly. A research method is considered reliable if we can repeat it and get the same results.

Coolican (1994) pointed out
“Any measure we use in life should be reliable, otherwise it’s useless. You wouldn’t want your car speedometer or a thermometer to give you different readings for the same values of different occasions. This applies to psychological measures as much as any other.”

A ruler for example would be reliable, as the results could be replicated time after time and the same results would be gained (consistency). If you measure the length of a book on Monday, and your ruler tells you its 25 cm long, it will still tell you its 25cm long on Friday.

An IQ test however may be unreliable, if a person sits the test on Monday and scores 140, and then sits the same test on Friday and scores 90. Even though it can be replicated, it shows low consistency and therefore is an unreliable test.

Some research methods (such as laboratory studies) have high reliability as they can be replicated and the results checked for consistency. Other research methods however (such as case studies and interviews) have lower reliability as they are difficult or impossible to replicate. As they cannot be replicated, we cannot check how consistent the results are.
We will look in more detail of the specific reliability of various research methods throughout the course.
VALIDITY
A study may be high in reliability, but the results may still be meaningless if we don’t have validity. Validity is the extent to which a test measures what it claims to measure.

There are three main aspects of validity that we investigate in psychological research Control, Realism and Generalisability.(p138)

Control
This refers to how well the experimenter has controlled the experimental situation. Control is important as without it, researchers can not establish cause and effect relationships. In other words, without control, we cannot state that it was the independent variable (IV) which caused the change in the dependant variable (DV). The result could have been caused by another variable, called an extraneous variable (EV). These are variables which have not been controlled by the experimenter, and which may affect the DV (see below).


Realism
The whole point of psychological research is to provide information about how people behave in real life. If an experiment is too controlled, or the situation too artificial, participants may act differently than they would in real life. Therefore, the results may lack validity.

   https://s3.amazonaws.com/engrade-myfiles/4086128317569189/Brainology.pdf
Journal of Nervous & Mental Disease:
December 1986
Original Article: PDF Only
Development, Reliability, and Validity of a Dissociation Scale.
BERNSTEIN, EVE M. Ph.D.; PUTNAM, FRANK W. M.D.
Abstract
http://allpsych.com/researchmethods/researcherror/

Dissociation is a lack of the normal integration of thoughts, feelings, and experiences into the stream of consciousness and memory. Dissociation occurs to some degree in normal individuals and is thought to be more prevalent in persons with major mental illnesses. The Dissociative Experiences Scale (DES) has been developed to offer a means of reliably measuring dissociation in normal and clinical populations. Scale items were developed using clinical data and interviews, scales involving memory loss, and consultations with experts in dissociation. Pilot testing was performed to refine the wording and format of the scale. The scale is a 28-item self-report questionnaire. Subjects were asked to make slashes on 100-mm lines to indicate where they fall on a continuum for each question. In addition, demographic information (age, sex, occupation, and level of education) was collected so that the connection between these variables and scale scores could be examined. The mean of all item scores ranges from 0 to 100 and is called the DES score. The scale was administered to between 10 and 39 subjects in each of the following populations: normal adults, late adolescent college students, and persons suffering from alcoholism, agoraphobia, phobicanxious disorders, posttraumatic stress disorder, schizophrenia, and multiple personality disorder. Reliability testing of the scale showed that the scale had good test-retest and good split-half reliability. Item-scale score correlations were all significant, indicating good internal consistency and construct validity. A Kruskal-Wallis test and post hoc comparisons of the scores of the eight populations provided evidence of the scale's criterion-referenced validity. The scale was able to distinguish between subjects with a dissociative disorder (multiple personality) and all other subjects.
(C) Williams & Wilkins 1986. All Rights Reserved.
http://www.sfexaminer.com/sanfrancisco/sf-public-defender-attacks-role-of-criminal-justice-system-in-school-board-members/Content?oid=2919897

http://www.times-standard.com/general-news/20141226/sunday-memorial-to-honor-life-of-joseph-waters

http://www.duggans-serra.com/mobile/obit.php?id=1463998&name=Joseph-William-Waters&loca=Daly-City-CA

https://www.facebook.com/josephwaterspersonaltraining

http://www.sacbee.com/news/politics-government/capitol-alert/

According to the lectures, and the book. I learned that, in the social sciences there is no best kind of research.  I think researchers probably use several methods in order to conduct research. Empirical, all information is based on observation.  Objectivity, Observations is verified by others.  Systematic, observations are made in a step-by-step fashion.  Controlled, potentially confusing factors are eliminated.  Public, built on previous research, open to critique and replication, building towards theories.

 It dependents what king of research or for what purpose you are researching. For example in social science is the science of people or collections of people, such as groups, firms, societies, or economies, and their individual or collective behaviors. It can be classified into disciplines such s psychology, sociology, and economics.  The society very much is more for the “collective”.  I think, using the scientific method is imperative for any kind of research. .

When we consider the advantages and disadvantages of laboratory vs. field research, are there any others that come to mind that were not outlined in lecture?
A)     Field Research/Ethnography: Participant observation is based on living among the people under study for a period of time, could be months or maybe years, and gathering data through continuous involvement in their lives and activities. The ethnographer begins systematic observation and keeps notes, in which the significant events of each day are recorded along with informants and interpretations. These demands are met through two major research techniques participant observation and key informant interviewing.  An example would be the one on the video that Maria has been spending several months with Steve a drug user, and the ethical problem come now, the participant do not realize that their behavior is being observed. Obviously (there is no consent) cannot give voluntary informed consent to be involved in the study.  Steve confesses that he is HIV positive and his partner does not know, there is a confidentiality issue.

2. Are there some things we can do in the field that we just cannot do in the lab and vise-versa?

A)        I learned that clear advantage of laboratory experiments over field experiments is that it is much easier to obtain large amounts of very detailed information from participants in the laboratory. An important reason why laboratory experiments are more artificial than field experiments is because the participants in laboratory experiments are aware that their behavior. One of the advantages of field experiments over laboratory experiments is that the behavior of the participants is often more of their normal behavior. The greatest advantage of field experiments over laboratory experiments is that they are less artificial

 3.      What are your ideas as researchers-in-training for accounting for the disadvantages of each and what problems might you foresee arising with your idea?

A)     I learned that, the method of investigation used most often by psychologists is the experimental method. Some of the advantages of the experimental method are common to both laboratory and field experiments. I would have to know reliability and validity and field vs. laboratory research. To avoid any confounding variables. These are variables that are manipulated/allowed to vary systematically along with the independent variable. The presence of any confounding variables can destroy the experiment, because it prevents from being able to interpret our findings.Question: “History is created by the victor.” Do we believe that comparative historical research gives us a good grasp of what actually happened in the past?
 I do not think so. Because, according to the lecture, the  comparative-historical analysis has four main defining elements. Comparative-historical analysis is also defined by epistemology. Specifically, comparative-historical works pursue social scientific insight and therefore accept the possibility of gaining insight through comparative-historical plus other  methods. Finally, the unit of analysis is a defining element, with comparative-historical analysis focusing on more aggregate social units.
Four main elements:
 Historical Events Research focuses on one short historical period (1 case, 1 time period)
 Historical Process Research –traces a sequence of events over a number of years (1 case, many time periods)
Cross-sectional Comparative Research -- comparing data from one time period between two or more nations (many cases, 1 time period)
 Comparative Historical Research – longitudinal comparative research (many cases) over a prolonged period of time.  Comparative and Historical Research by number of cases and length of time studied.
What are some of the benefits and negatives of the Case Study method. When compared to other types of research reviewed in the course thus far?  Can you think of some specific examples where the case study method might be preferable?
 The Case Study Method, Case study is a” Strategy for doing research which involves an empirical investigation of a particular contemporary phenomenon within its real life context using multiple sources of evidence” (Robson, 1993, p. 146). Case studies are in-depth investigations of a single person, group, event or community.  Depending on the case study   it says whether the research is field or laboratory research, we always loose valuable information about individual variation when we try to collect the information of the experiences, emotions, and behaviors into common experiences that we can measure numerically and generalize across a population. If I understand correctly the lecture, we cannot generalize from a Case Study, must be used three things: Descriptive, Exploratory, and Explanatory.
Consider generalizing the results of research done on a small sample to the general population.

I think I need to consider the Type of design chosen:  Questions the conditions under which the findings be generalized deals with the ability to generalize the findings outside the study to other populations and environments. Purpose of Research Design: Provides the plan or blueprint for testing research questions and hypotheses. Involves structure and strategy to maintain control and intervention fidelity. Accuracy:  Accomplished through the theoretical framework and literature review. All aspects of the study systematically and logically follow form the research questions. Time: Is there enough time for completion of the study. Control: Achieved with steps taken by the searcher to hold the conditions of the study uniform and avoid or decrease the effect of intervening, extraneous, or mediating variables of the dependent variable or outcome. Ensures that every subject receiving the intervention of treatment receive the identical intervention or treatment.

what are some of the benefits and negatives of qualitative and quantitative research?

Variables that occur during the study that interfere with or influence the relationship between the independent and dependent variables.

Intervening and mediating variables are processes that occur during the study.

Objectivity can be achieved form a thorough review of the literature and the development of a theoretical framework.

Instrumentation: changes in equipment used to make measurements or changes in observational techniques may cause measurements to vary between participants related to  treatment fidelity.

Controlling Extraneous Variables Using a homogeneous sample

Using consistent data-collection procedures- constancy.  A homogeneous sample is one in which the researcher chooses participants who are alike – for example, participants who belong to the same subculture or have similar characteristics.  Homogeneous sampling can be of particular use for conducting focus groups because individuals are generally more comfortable sharing their thoughts and ideas with other individuals who they perceive to be similar to them. Patton, M. (2001). Qualitative Research & Evaluation Methods.

when thinking of this,  Could one said to be superior to the other, or are they context specific?

 Consider generalizing the results, according to the lecture, the independent variable is: the variable that the researcher hypothesizes will have an effect on the dependent variable Usually manipulated (experimental study) The independent variable is: manipulated by means of a program, treatment, or intervention done to only one group in the study (experimental group ) The control group gets the standard treatment or no treatment.

The dependent variable is a factor, trait, or condition that can exist in differing amounts or types. Not manipulated and pressured to vary with changes in the independent variable The variable the researcher is interested in explaining. Randomization Each subject in the study has an equal chance of being assigned to the control group or the experimental group.

Assumes that any important intervening, extraneous, or mediating variable will be equally distributed between the groups, minimizing variance and decreasing selection bias.

Testing:  Taking the same test more than once can influence the participant’s responses the next time test is taken.
I understand that: There are two reasons for error that can result in a sample that is different from the population from which it is derived.
These are: The sampling error - chance, random sample bias mistake - constant error due to inadequate design.
Inferential statistics take into account the sampling error.
These statistics skew the sample is not correct. That's a problem of research design.
Inferential statistics refer only to random error. Value P
The reason for calculating an inferential statistics is to get a value of p (p = probability). The p value is the probability that the samples are from the same population with respect to the dependent variable (outcome).
Typically, we are testing the hypothesis samples (groups) differ in the result.
The value of p is directly related to the null hypothesis. The p value determines if the null hypothesis is rejected. It is used to estimate if we think that the null hypothesis is true. The p value provides an estimate of how often we would get the result by chance, if in fact the null hypothesis were true. If the p value is small, reject the null hypothesis and accept that the samples are really different in the result. If the p value is large, I accept the null hypothesis and conclude that the treatment or prediction variable had no effect on the outcome.

Do we believe the peer reviewed publication process outlined in the posted videos is sufficient for the general public to ascertain and  the results of such research?


Let's see if I get this:
It is very important to follow these process outlined in the videos. However it does not guarantee they will accept your paper (the journal ).  According to the videos, do have peer review before sending the paper to a journal, however is important to match journal and paper, make sure you count the word that is accepted by this publication/journal etc, following all the citations the videos are advising which is making me tired and disappointed just by thinking the process.
If is minor revision you should be positive because that means change all of what is advise and may be published. If is major changes, that makes me think do not even bother because you have to rewrite your hard work or star over your paper. Yes, I believe the peer reviewed publication process outlined in the posted videos helps. I am not sure if it is sufficient.....                                                                                                                                                    

What are some of the reasons why the constructs of reliability and validity are important to measure?
                                               
Reliability:  refers to the consistency of a measure. Validity: is the extent to which a test measures what it claims to measure. There is Three main features that need to be included to measure are:  Control: refers to how well the experimenter controlled the experiment. The control is important because without control, researchers cannot establish cause and effect. Realism: is where psychological research is provided information about how people in the real world behave. If an experiment is too controlled, too artificial or situation, participants can act differently than they would in real life. Generalizability: is the primary objective of psychological research is producing results that can be generalized beyond the experiment setup.

How do we use the words reliability and validity in everyday life? What do these words mean? Is there a difference between them or do they mean the same thing?
Reliability and Validity

How do we use the words reliability and validity in everyday life? What do these words mean? Is there a difference between them or do they mean the same thing?


RELIABILITY
Reliability refers to the consistency of a measure. A measure is considered reliable if we get the same result repeatedly. A research method is considered reliable if we can repeat it and get the same results.

Coolican (1994) pointed out
“Any measure we use in life should be reliable, otherwise it’s useless. You wouldn’t want your car speedometer or a thermometer to give you different readings for the same values of different occasions. This applies to psychological measures as much as any other.”

A ruler for example would be reliable, as the results could be replicated time after time and the same results would be gained (consistency). If you measure the length of a book on Monday, and your ruler tells you its 25 cm long, it will still tell you its 25cm long on Friday.

An IQ test however may be unreliable, if a person sits the test on Monday and scores 140, and then sits the same test on Friday and scores 90. Even though it can be replicated, it shows low consistency and therefore is an unreliable test.

Some research methods (such as laboratory studies) have high reliability as they can be replicated and the results checked for consistency. Other research methods however (such as case studies and interviews) have lower reliability as they are difficult or impossible to replicate. As they cannot be replicated, we cannot check how consistent the results are.


A study may be high in reliability, but the results may still be meaningless if we don’t have validity. Validity is the extent to which a test measures what it claims to measure.

There are three main aspects of validity that we investigate in psychological research Control, Realism and Generalisability.(p138)

Control
This refers to how well the experimenter has controlled the experimental situation. Control is important as without it, researchers can not establish cause and effect relationships. In other words, without control, we cannot state that it was the independent variable (IV) which caused the change in the dependant variable (DV). The result could have been caused by another variable, called an extraneous variable (EV). These are variables which have not been controlled by the experimenter, and which may affect the DV (see below).


Realism
The whole point of psychological research is to provide information about how people behave in real life. If an experiment is too controlled, or the situation too artificial, participants may act differently than they would in real life. Therefore, the results may lack validity.

Generalisability
The aim of psychological research is to produce results which can then be generalised beyond the setting of the experiment. If an experiment is lacking in realism we will be unable to generalise. However, even if an experiment is high in realism, we still may not be able to generalise.

For example, the participants may be all from a small group of similar people, meaning low population validity.  Many experiments use white, middle class American college students as participants. What issues with generalisability can you think of?



TYPES OF VALIDITY
Experimental Validity: is the study really measuring what it intends?

INTERNAL VALIDITY refers to things that happen “inside” the study. Internal validity is concerned with whether we can be certain that it was the IV which caused the change in the DV. If aspects of the experimental situation lack validity, the results of the study are meaningless and we can make no meaningful conclusions from them.
-        Internal validity can be affected by a lack of mundane realism. This could lead the participants to act in a way which is unnatural, thus making the results less valid.
-        Internal validity can also be affected by extraneous variables (see below).
Lecture 1: Distinction between empirical vs. non-empirical ways of knowing
(UN) SCIENTIFIC THINKING
Authority
Because someone told us that something is true.
2. Reasoning
A priori method (proposed by Charles Peirce): a person develops a belief by reasoning, listening to others’ reasoning, and drawing on previous intellectual knowledge – not based on experience or direct observation.

(Somewhat) scientific thinking
3. Experience
Empiricism:

knowing by direct observation or experience.

Subject to errors in thinking!

Experience based errors in thinking
Illusory Correlation
Definition:  thinking that one has observed an association between events that
(a) doesn’t exist,

(b) exists but is not as strong as is believed,

or (c) is in the opposite direction from what is believed.


Experience based errors in thinking
Confirmation Bias –

In psychology and cognitive science, confirmation bias (or confirmatory bias) is a tendency to search for or interpret information in a way that confirms one's preconceptions, leading to statistical errors.
Experience based errors in thinking
Availability Heuristic - An availability heuristic is a mental shortcut that relies on immediate examples that come to mind.
As a result, you might judge that those events are more frequent and possible than others and tend to overestimate the probability and likelihood of similar things happening in the future.
So how do we do it right?
So given all of these errors in thinking and understanding, how could we ever hope to accurately observe?

The scientific method
4. Scientific Method:

Definition: A way of knowing characterized by the attempt to apply systematic, objective, empirical methods when searching for causes of natural events.

Probabilistic Statistical determinism: Based on what we have observed, is the likelihood of two events occurring together (whether causal, predictive, or simple relational) greater than chance?

Objectivity: without bias of the experimenter or participants.

Data-driven: conclusions are based on the data-- objective information.

Steps of The Scientific Method
Ask a Question
Do Background Research
Construct a Hypothesis
Test Your Hypothesis by Doing an Experiment
Analyze Your Data and Draw a Conclusion
Communicate Your Results

Scientific Thinking in Research
CRITERIA FOR SCIENTIFIC METHOD:
Empirical: All information is based on observation.
Objectivity: Observations are verified by others.
Systematic: Observations are made in a step-by-step fashion.
Controlled: Potentially confusing factors are eliminated.
Public: Built on previous research, open to critique and replication, building towards theories

ASSUMPTIONS ABOUT BEHAVIORS OR OBSERVATIONS:
Lawful: Every event can be understood as a sequence of natural causes and effects.

Determinism: Events and behaviors have causes.

Discoverability: Through systematic observation, we can discover causes – and work towards more certain and comprehensive explanations through repeated discoveries.


Research Question
Definition of a theory:

A set of logically consistent statements about some psychological phenomenon that

best summarizes existing empirical knowledge of the phenomenon
organizes this knowledge in the form of precise statements of the relationship among variables
provides a tentative explanation for the phenomenon
serves as a basis for making predictions about behavior.
From theory to actual research
Relationship between theory and data

Deduction: Reasoning from a set of general statements towards the prediction of some specific event.  Based on a theory, one can deduct an event or behavior given particular conditions.

Hypothesis: Prediction about specific events that is derived from the theory.

Methodology
Methodology: A body of practices, procedures, and rules used by researchers to offer insight into the workings of the world.

Insight: Evidence contributing to an understanding of a case or set of cases. Comparative-historical researchers are generally most concerned with causal insight, or insight into causal processes.

Comparative-Historical Analysis
Comparative-historical analysis: A prominent research tradition in the social sciences, especially in political science and sociology. Works within this research tradition use comparative-historical methods, pursue causal explanations, and analyze units of analysis at the meso- or macro-level.

Epistemology
Epistemology: A branch of philosophy that considers the possibility of knowledge and understanding. Within the social sciences, epistemological debates commonly focus on the possibility of gaining insight into the causes of social phenomena.

Positivism: An epistemological approach that was popular among most of the founding figures of the social sciences. It claims that the scientific method is the best way to gain insight into our world. Within the social sciences, positivism suggests that scientific methods can be used to analyze social relations in order to gain knowledge...

Ethnographic Methods
Ethnographic methods: A type of social scientific method that gains insight into social relations through participant observation, interviews, and the analysis of art, texts, and oral histories. It is commonly used to analyze culture and is the most common method of anthropology.

Case Study
Case Study (Within-case methods): A category of methods used in the social sciences that offer insight into the determinants of a particular phenomenon for a particular case. For this, they analyze the processes and characteristics of the case.

Ideographic Explanation
Ideographic explanation: Causal explanations that explore the causes of a particular case. Such explanations are not meant to apply to a larger set of cases and commonly focus on the particularities of the case under analysis.

Comparative Methods
Comparative methods: Diverse methods used in the social sciences that offer insight through cross-case comparison. For this, they com- pare the characteristics of different cases and highlight similarities and differences between them. Comparative methods are usually used to explore causes that are common among a set of cases. They are commonly used in all social scientific disciplines.

Statistical Methods
Statistical methods: The most common subtype of comparative methods. It operationalizes variables for several cases, compares the cases to explore relationships between the variables, and uses probability theory to estimate causal effects or risks. Within the social sciences, statistics uses natural variation to approximate experimental methods. There are diverse subtypes of statistical methods.

Experimental Methods
Experimental methods: The most powerful method used in the social sciences, albeit the most difficult to use. It manipulates individuals in a particular way (the treatment) and explores the impact of this treat- ment. It offers powerful insight by controlling the environment, thereby allowing researchers to isolate the impact of the treatment.


Three Steps in designing a “Case”
Define the Case
Select one of 4 types of case study designs
 Use theory in design work
Data Collection

1. Direct observations (e.g., human actions or a physical environment)
2. Interviews (e.g., open-ended conversations with key participants)
3. Archival records (e.g., student records)
4. Documents (e.g., newspaper articles, letters and e-mails, reports)
5. Participant-observation (e.g., being identified as a researcher but also filling a real-life role in the scene being studied)
6. Physical artifacts (e.g., computer downloads of employees’ work)
Evidence from multiple sources
Triangulation
Literature Review
Direct Observation

* You are always better off using multiple rather than single sources of evidence.
Case Study Protocol
The typical protocol consists of a set of questions to be addressed while collecting the case study data (whether actually taking place at a field setting or at your own desk when extracting information from an archival source).
Importantly, the questions in the protocol are directed at the researcher, not at any field participant. In this sense, the protocol differs entirely from any instrument used in a conventional interview or survey. The protocol’s questions in effect serve as a mental framework,
Collect Data for Rival Explanations
A final data collection topic stresses the role of seeking data to examine rival explanations. The desired rival thinking should draw from a continual sense of skepticism as a case study proceeds.
During data collection, the skepticism should involve worrying about whether events and actions are as they appear to be and whether participants are giving candid responses.
 Having a truly skeptical attitude will result in collecting more data than if rivals were not a concern.
Presenting your Case
You need to present the evidence in your case study with sufficient clarity (e.g., in separate texts, tables, and exhibits) to allow readers to judge independently your later interpretation of the data.
Ideally, such evidence will come from a formal case study database that you compile for your files after completing your data collection.

Case Study Data Analysis
Whether using computer software to help you or not, the researcher will be the one who must define the codes to be used and the procedures for logically piecing together the coded evidence into broader themes—in essence creating your own unique algorithm befitting your particular case study. The strength of the analytic course will depend on a marshaling of claims that use your data in a logical fashion.
Your analysis can begin by systematically organizing your data (narratives and words) into hierarchical relationships, matrices, or other arrays (e.g., Miles & Huberman, 1994).
Techniques
Pattern-Matching
Open-Ended Questions
Time-Series-Like Analysis

What about when quantitative data is available?
Correlations between single case studies can be used only if there is sufficient data to run a statistical analysis.
Meta Analysis: A study of multiple Case Studies
The logic for such a cross-case synthesis emulates that used in addressing whether the findings from a set of multiple experiments—too small in number to be made part of any quantitative meta-analysis (a study of the results of other studies)—support any broader pattern of conclusions.
The replication or corroboratory frameworks can vary. In a direct replication,the single cases would be predicted to arrive at similar results.

Can we generalize at all from a Case Study?
To the extent that any study concerns itself with generalizing, case studies tend to generalize to other situations (on the basis of analytic claims), whereas surveys and other quantitative methods tend to generalize to populations (on the basis of statistical claims).

Social Science Research Methods
Lecture 6: Quantitative and Qualitative Research Methods
What’s the difference?
Quantitative Research: This type of research is charged with quantifying (measuring and counting) and subscribes to to a particular empirical approach to knowledge, believing that by measuring accurately enough we can make claims about the object at study.
Qualitative Research: This type of research is charged with the quality or qualities of an experience or phenomenon. Qualitative research rejects the notion of their being a simple relationship between our perception of the world and the world itself, instead arguing that each individual places different meaning on different events  or experiences and that these are constantly changing. Qualitative research generally gathers text based data through exercises with a small number of participants, usually semi structured or unstructured interviews.


Quantitative Research: An Overview
Mathematically based
Often uses survey-based measures to collect data
Often collects data on what is known as a “Likert-scale” a 4-7 point numerical scale which a participant rates agreement
Uses statistical methodology to analyze numerical data

Example of Survey Used in Quantitative Research
Relationship between the Sample and the Population
When to use Quantitative Research
1) When we are looking for a numerical answer
2) When we want to study numerical change
3) When we want to find out about the state of something or to explain a phenomena
4) When we want to test a hypothesis
Advantages and Disadvantages: Quantitative
Quantitative Advantages:
Concise
Accurate
Strictly Controlled
Replicable
Can indicate causation
Ideally is objective


Quantitative Disadvantages:
Limited understanding of individuality
Groups people into categories
Can be accused of oversimplifying human nature
Qualitative Research
Focus on “language rather than numbers”
“Embraces “intersubjectivity” or how people may construct meaning…”
Focus on the individual and their real lived experience.
Advantages and Disadvantages: Qualitative
Qualitative Advantages:
Appreciates research participant’s individuality
Provides insider view of research question
Less structured than quantitative approach
Qualitative Disadvantages:
Not always appropriate to generalize results to larger population
Time consuming
Difficult to test a hypothesis
Qualitative Research in Psychology
“Today, a growing number of psychologists are re-examining and re-exploring qualitative methods for psychological research, challenging the more traditional ‘scientific’ experimental approach”
When to use Qualitative Research
Content and Thematic Analysis
Grounded Theory (Generating Theory from Data)
Discourse and Narrative Analysis
Individual Case Studies
Content and Thematic Analysis
Content and Thematic Analysis - Content Analysis, or Thematic Analysis (the terms are frequently used interchangeably and generally mean much the same), is particularly useful for conceptual, or thematic, analysis or relational analysis. It can quantify the occurrences of concepts selected for examination (Wilkinson & Birmingham, 2003).
Grounded Theory
Grounded Theory - is frequently considered to offer researchers a suitable qualitative method for in-depth exploratory investigations. . It is a rigorous approach which provides the researcher with a set of systematic strategies and assumes that when investigating social processes qualitatively from the bottom up there will be an emergence of a working theory about the population from the qualitative data (Willig, 2008, pp. 44).

Discursive psychology and Discourse Analysis
Discourse Analysis: The discursive approach looks to verbal behavior as a more direct means of uncovering underlying cognitions (Harré,1995) rather than assigning a numerical value ‘score’ or scale to a behavior. This approach takes the view that interpretation and empathy are involved in attempting to understand human behavior. Self-report, from people being studied, can then become a valuable resource in its own right.

What do we do if we want the best of both worlds?
Mixed-Methods Designs - questionnaire) and qualitative (for example, a number of case studies) methods. Mixed-methods research is a flexible approach, where the research design is determined by what we want to find out rather than by any predetermined epistemological position. In mixed-methods research, qualitative or quantitative components can predominate, or both can have equal status.

Social Science Research Methods
Basic Overview of Statistics
Adapted from Sage Publications: http://www.sagepub.com/upm-data/49259_ch_1.pdf
Working definition
Statistics are a set of mathematical procedures for summarizing and interpreting observations. These observations are typically numerical or categorical facts about specific people or things, and they are usually referred to as data.
Central Tendency and Dispersion
Central Tendency
Dispersion
Of the threats to validity covered thus far in the course, which do you believe is the most harmful or the most likely to occur in psychological and/or sociological research?
External validity
Interactions
Interactions selection and treatment= those we select have a different reaction our program than others. We select high potentials for our pilot; it works better for them than others.

Interactions Setting and treatment= results in one setting may not work in another. Recognition programs that work in a hospital may not work in manufacturing, many not work due settings etc.

Interactions History and treatment= the results we see may not hold true for other points in time. Stock pickers were very successful during the “tech bubble” have not been able to replicate the success since.

Of the threats to validity covered thus far in the course, which do you believe is the most harmful or the most likely to occur in psychological and/or sociological research?

Threats to Construct Validity : Inadequate Pre-operational Explication of constructions. We didn’t clearly define things before we started. Mono-operational Bias. We use a single measure of a construct that is not complete. Mono-method Bias. Using only one approach to measuring a construct. We only use surveys to assess employee engagement they are self-report and subject to bias as self-report. Hypothesis Guessing. Participants try to guess what we are looking for an act differently. Participants learn they are in a pilot program aimed at improving success so they work harder or report better results.

Internal Validity: Confidence in cause and effect Requirements. Difference in Dependent Variable. Independent variable before Dependent Variable. Extraneous factors (alterative rival hypotheses In theory: Two identical groups Pretest-posttest design. makes sure the interactions selection, settings and history. Interactions selection and treatment. Selected have a different reaction to our program than other programs. Make sure you select high potential for our program. Interactions setting and treatment. Results in one setting may not work in other settings. Interactions History and treatment The results we see today may not be true for other times.
Due: External validity; Randomly select people to study (not randomly assigned).  Replication even on small scale over time over sample of a study. Clear about how you select people, how do we get people to this discreetness time settings.
Social Science Research Methods
Threats to Validity, Part 2
Social Science Research Methods
Threats to Validity, Part 2
Professor Gavin Ryan Shafron, M.A.
Threats to Internal Validity
Internal Validity – How valid a study is: the more we avoid confounding variables (variables that could interfere in attaining the true results of our study) the higher the level of internal validity.
Threats to Internal Validity
Attrition Threat – In a longitudinal study where you need to measure responses from a participant at multiple time points, attrition refers to when participants drop out or leave a study, causes problems as it reduces available data, in particular when it is different across groups.
Threats to Internal Validity
Attrition: How do we fix it?
Randomly assign participants to two groups, one treatment group & one control group.
Researchers can expect similar effects in treatment group versus control group over time.
Threats to Internal Validity
Treatment Diffusion - When one group becomes aware of the other’s treatment and the effects of the treatment.
Threats to Internal Validity
Treatment Diffusion: How do we fix it?
Isolation of participants
Threats to Internal Validity
Testing Threat -- Changes in a test score due to taking it more than once, through familiarization, recall.
Threats to Internal Validity
Testing Threat: How do we fix it?
Randomly assign participants to two groups, one treatment group & one control group.
Researchers can expect similar effects in treatment group versus control group over time.
Threats to Internal Validity
History Threat -- Events outside experimental manipulation influence collective or individual participation
“Many change-producing events may have occurred between Observation 1 and Observation 2 “ (Source: Internet Presentation)
“History is more …[likely] the longer the lapse between the pretest and posttest.” (Source: Alliant International University - Internet Presentation)
Threats to Internal Validity
History Threat: How do we Fix it?
Consistent methods / measurements in longitudinal studies from one time point to the next.
Threats to Internal Validity
Instrumentation Threat -- Changes in the measurement or procedure over the course of a study (e.g., with observations, coding systems over time, drift systematically).
Threats to Internal Validity
Instrumentation threat: How do we fix it?
Randomly assign participants to two groups, one treatment group & one control group.
Researchers can expect similar effects in treatment group versus control group over time.
Or, use tests rather than observation, ratings, or interviews.
Threats to Internal Validity
Selection Threat -- Differences in groups that exist before treatment or intervention begins. Especially problematic when participants selected for belonging to a specific group. Differences may be due to initial differences in groups – not treatment or intervention.
Threats to Internal Validity
Selection Threat: How do we fix it?
Randomly assign participants to two groups, one treatment group & one control group.
Researchers can expect similar effects in treatment group versus control group over time.
Threats to Internal Validity
Maturation Threat - Normal developmental changes in participants between pre and post tests. Gains/Losses over time may be due to normal maturation.
Threats to Internal Validity
Maturation Threat: How do we fix it?
Randomly assign participants to two groups, one treatment group & one control group.
Researchers can expect similar effects in treatment group versus control group over time.
Threats to Internal Validity
Inequitable Treatments Threat -- When participants in one group outperform or underperform relative to another group as a result of study expectation of a certain performance.
Threats to Internal Validity
Inequitable Treatments Threat: How do we fix it?
Isolating groups.
Blind studies.
Threats to Internal Validity
Special Treatment Threat -- When groups are treated differently as a result of being in the study – or one group is treated differently from another group b/c of their treatment status.
Threats to Internal Validity
Special Treatment Threat: How to fix it?
Training of treatment administrators and personnel associated with the study.
Threats to Internal Validity
Statistical Regression Threat -- The tendency to drift towards the mean as one takes a test multiple times. Especially problematic when choosing “extreme” scores which can “regress” toward the mean due to influences other than treatment.
Threats to Internal Validity
Statistical Regression Threat: How to fix it?
The problem is with single assessments. Two randomly assigned groups should equally regress toward the mean.
Randomly assign participants to two groups, one treatment group & one control group.
Researchers can expect similar effects in treatment group versus control group over time.
Threats to Internal Validity
Interaction with Selection (with other factors)
Also known as: Additive and interactive effects of threats to internal validity.  --
When certain participants (selection) experience drop out, historical effects, testing issues as a group.
Threats to Internal Validity
Interaction with selection with other factors threat: How to fix it?
Randomly assign participants to two groups, one treatment group & one control group.
Researchers can expect similar effects in treatment group versus control group over time
Threats to Internal Validity
Ambiguous Directionality -- When the independent variable is not manipulated, the direction of the influence is not always clear (e.g., impact of a therapist empathy on client outcome – does therapist get warmer b/c client improves or vice versa?)
Threats to Internal Validity
Ambiguous Directionality: How to fix it?
Unless the temporal order is clear, the directionality is difficult to determine.
Be clear in statements of causality and describe when causality is unclear.

Social Science Research Methods
Threats to Validity, Part 3


Threats to External Validity
What is a threat to External Validity?
This is when the results found are not generalizable to the population of interest.
You may see an effect, but its not the effect you wanted to measure or it doesn’t generalize outside of the small group you are studying!
Threats to External Validity
Selection (Interaction of Causal Units): This is distinguished from selection threat to internal validity. This is the selection threat to external validity. You can think of this one as the interaction of causal units. This is a threat when the participants used in a study are of a specific type such that the results found with them might not apply (be generalizable) to other participants, but researchers imply this.
Example: An article entitled “Depression among Asian Americans” studies depression, but the participants were only from Japan and China, and were upper-middle class. This is a problem because researchers are trying to find out information about Asian Americans in general, but only used Japanese and Chinese upper SES people. The results are not generalizable to all Asian Americans, and depression may work differently with other types of Asians and Asians of other SES’s.  This WOULD NOT be a problem, however, if the article was called “Depression among wealthy Japanese and Chinese Americans”, since researchers then don’t intend for the results to be generalized to all Asian Americans.
Selection (Interaction of Causal Units): This is a threat when the participants used in a study are of a specific type such that the results found with them might not apply (be generalizable) to other participants, but researchers imply this.
Threats to External Validity
Remedy: Either try to have participants that are very representative (don’t miss out on certain people by only doing college students or white people or high SES etc),  or don’t try to generalize to other people.
Risk:  EXTREMELY HIGH. This happens ALL THE TIME in studies.
     Look in:  PARTICIPANTS to see who were used for the study. Then look at the title, abstract, and/or discussion to make sure you know who they are trying to generalize the results to and talk about.
Threats to External Validity
Interaction of the causal relationship over treatment variations:  This is a problem when the research study has a TREATMENT, but a weird variation of the treatment is used, and results found may not be generalizable to other variations of that treatment.
Threats to External Validity
Remedy: Use the whole treatment as it was meant to be given.
Risk: Low. Researches would likely admit that they administered a treatment differently than it was supposed to be administered.
Look in: Methods, measures, and Limitations
Threats to External Validity
Interaction of the causal relationship with outcomes: This is similar to the idea of mono operation bias, but just for the DV.  This is a threat when the outcome (usually just one Dependent Variable) was measured a certain way (usually with only one measure). It is a problem when measuring the outcome in a different way could have given different results.
Threats to External Validity
Remedy: Use more than one way to measure the DV.
     Risk: Medium. Look for just one DV with just one measure that could have been measured differently. (If the one outcome was height and they used a ruler, do NOT apply this threat, because there is no other way to measure height and get different results).
     Look in: Methods
Threats to External Validity
Interaction of causal relationship w/settings: This is a threat when the research was done in a particular setting (environment), and results may not be generalizable to a different setting. By setting we are talking about specific research setting such as laboratory, school, home, internet, ect. NOT geographic location like Texas or New York. Geographic locations are more of a selection threat because we are talking about different types of people with results that may not generalize to other types of people. Here we are talking about a setting where results may not generalize to other settings.
Threats to External Validity
Remedy: Try to do studies in the appropriate types of settings you are trying to generalize to or in more than one setting.
     Risk: HIGH. Only 1 setting is used in most studies.
     Look in: Methods in Procedure
Threats to External Validity
Context-dependent Mediation: This is a threat when a mediating variable may not mediate the same way in a different situation (different setting, or participants, or treatment, or task, ect). This is only when the study has a mediating variable, and this relationship (the variable mediating the relationship between two other variables) may not be the same in a different situation.
Threats to External Validity
Remedy: Try to use generalizable situations.
     Risk: RARE. The study must have a mediating variable in it.
     Look in: Methods, Results, and Conclusions
Threats to External Validity
Interaction of History and Treatment:   This is different than the Internal Validity threat History. That is when an event affects participants and causes the results found. THIS one is when results found from a study are not generalizable to other time periods. This usually has to do with whole eras. It mostly applies to old studies, when times were different, and results found may not be the same today.
Threats to External Validity
Remedy: Replicate studies over time to see if they still apply.
     Risk:  Rare.
     Look in: Date of publication, Methods

Hi class,  For this week we will be continuing our discussion of validity and in particular threats to ensuring validity. As we’ve reviewed, when designing a study of any kind, we want to ensure that as much reliability and validity can be secured to be able to state that what we are observing is in fact more likely than not what happens in the general population, rather than some artifact that is not the result of our treatment or intervention we are studying. So when considering this we have to review some of the basic concepts of what can go wrong when we implement a study.
Now as we might remember, internal validity refers to how well an experiment is done, especially whether it avoids confounding (more than one possible independent variable [cause] acting at the same time). The less chance for confounding in a study, the higher its internal validity is.
Therefore, internal validity refers to how well a piece of research allows you to choose among alternate explanations of something. A research study with high internal validity lets you choose one explanation over another with a lot of confidence, because it avoids (many possible) confounds.
Attrition Threat – In a longitudinal study where you need to measure responses from a participant at multiple time points, attrition refers to when participants drop out or leave a study, causes problems as it reduces available data, in particular when it is different across groups.
When participants drop out or leave a study, causes problems as it reduces available data, in particular when it is different across groups.

“The differential loss of individuals from treatment and/or comparison groups. This is often a problem when research participants are volunteers. Volunteers may drop out of the study if they find it is consuming too much of their time. Others may drop out if they find the task to be too arduous.” (Source: Internet presentation)

It also can be thought of as“…refer[ing] to the drop out of subjects in an experiment…if people drop out of an experiment they do so for a reason…[perhaps indicates] a problem with either a noxious or an inadequate independent variable” (Kroth, 1988, p. 60).
Attrition: How do we fix it?
Randomly assign participants to two groups, one treatment group & one control group.
Researchers can expect similar effects in treatment group versus control group over time.
Attrition: How do we fix it?
The best way to account for this is to randomly assign participants to two groups, one treatment group & one control group.
Researchers can expect similar effects in treatment group versus control group over time.
Treatment Diffusion - This occurs when a comparison group learns about the program either directly or indirectly from program group participants. In a school context, children from different groups within the same school might share experiences during lunch hour. Or, comparison group students, seeing what the program group is getting, might set up their own experience to try to imitate that of the program group. In either case, if the diffusion of imitation affects the posttest performance of the comparison group, it can have an jeopardize your ability to assess whether your program is causing the outcome. Notice that this threat to validity tend to equalize the outcomes between groups, minimizing the chance of seeing a program effect even if there is one.
References
http://www.socialresearchmethods.net/kb/intsoc.php

Treatment Diffusion - When one group becomes aware of the other’s treatment and the effects of the treatment.
Treatment Diffusion: How do we fix it?
Isolation of participants.
Treatment Diffusion: How do we fix it?
By isolating one group of participants from the other, this mitigates the likelihood that demand characteristics of the contrast between treatment conditions could interfere with the results and threaten the internal validity of the study in this way.
Testing Threat -- Changes in a test score due to taking it more than once, through familiarization, recall.  Testing Threat -- Changes in a test score due to taking it more than once, through familiarization, recall.

Also called “pretest sensitization,” this refers to the effects of taking a test upon performance on a second testing. Merely having been exposed to the pretest may influence performance on a posttest. Testing becomes a more viable threat to internal validity as the time between pretest and posttest is shortened.” (Source: Internet Presentation)

The impact of “taking a test, and whether or not any special circumstances existed which may have influenced sores on a subsequent test…taking a test may cause anxiety, stress or self-consciousness the first time one takes it, but such emotions may be absent at posttest time [causing an improvement that is not simply due to the treatment, or IV]”
(Kroth, 1988, p. 60). Testing Threat: How do we fix it?
Randomly assign participants to two groups, one treatment group & one control group.
Researchers can expect similar effects in treatment group versus control group over time.
Testing Threat: How do we fix it?
Randomly assign participants to two groups, one treatment group & one control group.
Researchers can expect similar effects in treatment group versus control group over time.

History Threat -- Events outside experimental manipulation influence collective or individual participation
“Many change-producing events may have occurred between Observation 1 and Observation 2 “ (Source: Internet Presentation)
“History is more …[likely] the longer the lapse between the pretest and posttest.” (Source: Alliant International University - Internet Presentation)
History Threat -- Events outside experimental manipulation influence collective or individual participation
“Many change-producing events may have occurred between Observation 1 and Observation 2 “ (Source: Internet Presentation)
“History is more …[likely] the longer the lapse between the pretest and posttest.” (Source: Alliant International University - Internet Presentation)
This was described best by Kroth, 1988 who stated:
“Things happen between a pretest and posttest, especially when there may be 6 months to a year between the two…These …events, or outside influences [confounding variables] may cause changes we see between pretest time and posttest time may be due to these outside factors, not the independent variable” (Kroth, 1988, p. 60)
History Threat: How do we Fix it?
Consistent methods / measurements in longitudinal studies from one time point to the next.
Instrumentation Threat -- Changes in the measurement or procedure over the course of a study (e.g., with observations, coding systems over time, drift systematically
Changes in the measurement or procedure over the course of a study (e.g., with observations, coding systems over time, drift systematically).

“Changes in the way a test or other measuring instrument is calibrated that could account for results of a research study (different forms of a test can have different levels of difficulty). This threat typically arises from unreliability in the measuring instrument. Can also be present when using observers.”
(Source: Internet Presentation)

Another possibility is whether the instrument tends to…[vary] all by itself, registering…a spurious increase or decrease which is unrelated to what the good old independent variable is trying to do” (Kroth, 1988, p. 60).
Instrumentation threat: How do we fix it?
Randomly assign participants to two groups, one treatment group & one control group.
Researchers can expect similar effects in treatment group versus control group over time.
Or, use tests rather than observation, ratings, or interviews.
Selection Threat -- Differences in groups that exist before treatment or intervention begins. Especially problematic when participants selected for belonging to a specific group. Differences may be due to initial differences in groups – not treatment or intervention. Selection Threat -- Differences in groups that exist before treatment or intervention begins. Especially problematic when participants selected for belonging to a specific group. Differences may be due to initial differences in groups – not treatment or intervention.
“This can occur when intact groups are compared. The groups may have been different to begin with. If three different classrooms are each exposed to a different intervention, the classroom performances may differ only because the groups were different to begin with.” (Source: Internet Presentation)
“…the biases which result from comparing one group of complex and unique people to another group of complex and unique subjects” (Kroth, 1988, p. 60).
Selection Threat: How do we fix it?
Randomly assign participants to two groups, one treatment group & one control group.
Researchers can expect similar effects in treatment group versus control group over time.
Maturation Threat - Normal developmental changes in participants between pre and post tests. Gains/Losses over time may be due to normal maturation. Maturation Threat - Normal developmental changes in participants between pre and post tests. Gains/Losses over time may be due to normal maturation.

“Changes in physical, intellectual, or emotional characteristics, that occur naturally over time, that influence the results of a research study.” (Source: Internet Presentation)

“Internal changes occurring between the pretest … and posttest… which can cause changes in the dependent variable. These are internal changes, like becoming tired or heavier or weaker, less anxious, or more inattentive with the passage of time.” (Kroth, 1988, p. 60)
Maturation Threat: How do we fix it?
Randomly assign participants to two groups, one treatment group & one control group.
Researchers can expect similar effects in treatment group versus control group over time.
Inequitable Treatments Threat -- When participants in one group outperform or underperform relative to another group as a result of study expectation of a certain performance.
Inequitable Treatments Threat: How do we fix it?
Isolating groups.
Blind studies.
Special Treatment Threat -- When groups are treated differently as a result of being in the study – or one group is treated differently from another group b/c of their treatment status.
Special Treatment Threat -- When groups are treated differently as a result of being in the study – or one group is treated differently from another group b/c of their treatment status.
Special Treatment Threat: How to fix it?
Training of treatment administrators and personnel associated with the study.
Special Treatment Threat: How to fix it?
Training of treatment administrators and personnel associated with the study.
Statistical Regression Threat -- The tendency to drift towards the mean as one takes a test multiple times. Especially problematic when choosing “extreme” scores which can “regress” toward the mean due to influences other than treatment.
Statistical Regression Threat -- The tendency to drift towards the mean as one takes a test multiple times. Especially problematic when choosing “extreme” scores which can “regress” toward the mean due to influences other than treatment.

“Occurs when individuals are selected for an intervention or treatment on the basis of extreme scores on a pretest. Extreme scores are more likely to reflect larger (positive or negative) errors in measurement (chance factors). Such extreme measurement errors are NOT likely to occur on a second testing.” (Source: Internet Presentation)

This “refers to the fact that a group at the extreme ends of a normal curve has a greater probability of being “closer” to the mean of that curve the next time the group is measured” (Kroth, 1988, p. 60).
Statistical Regression Threat: How to fix it?
The problem is with single assessments. Two randomly assigned groups should equally regress toward the mean.
Randomly assign participants to two groups, one treatment group & one control group.
Researchers can expect similar effects in treatment group versus control group over time.
Interaction with Selection (with other factors)
Also known as: Additive and interactive effects of threats to internal validity.  --
When certain participants (selection) experience drop out, historical effects, testing issues as a group.
One research scholar, Vanbeek states: “No threat is completely isolated, often one threat can lead to another or enhance another, often in vicious cycles.”
Interaction with selection with other factors threat: How to fix it?
Randomly assign participants to two groups, one treatment group & one control group.
Researchers can expect similar effects in treatment group versus control group over time
Ambiguous Directionality -- When the independent variable is not manipulated, the direction of the influence is not always clear (e.g., impact of a therapist empathy on client outcome – does therapist get warmer b/c client improves or vice versa?)
Ambiguous Directionality: How to fix it?
Unless the temporal order is clear, the directionality is difficult to determine.
Be clear in statements of causality and describe when causality is unclear.
Ambiguous Directionality: How to fix it?
Unless the temporal order is clear, the directionality is difficult to determine.
Be clear in statements of causality and describe when causality is unclear.


Page 5 of 3 treats to extern ….
Interaction of the causal relationship over treatment variations:  This is a problem when the research study has a TREATMENT, but a weird variation of the treatment is used, and results found may not be generalizable to other variations of that treatment.

      Example: Researchers test a manualized CBT treatment. This CBT treatment is meant to be 12 weeks, but researchers did a 7 week version of it, so the results may be different. Effects found with this treatment are not necessarily the same at results that would have been found with the treatment they were trying to test (the 12 week treatment). OR Researchers combined CBT with another treatment, which may find different results. OR They only used part of the treatment, such as the Cognitive portion of the treatment.
Example: Unintelligent researchers wanted to measure if taking Prozac effects basketball players’ performance. They give them the drug, then measure the percentage of shots they make during a game, and compare this to their old shot percentage. The effects found may have been different if they had measured a different outcome than shot percentage. If they had included assists, blocks, passes, coaches ratings of them, ect, the different results may have been found about the players performance.
Example: Researchers wanted to test children’s reactions to their parents instructions. They tested this in a room at a local elementary school, asking parents to tell their kids to complete certain tasks. The kids were super compliant. HOWEVER, the setting here was a school, where kids are accustomed to being compliant. If the study were done at children’s homes, different results may have been found. You cannot generalize this to how children’s reaction to their parents instructions in general, only how they react to them at a school.
Example: Suppose a study used only males to study depression. They found that owning more video games makes them super excited and causes sleeplessness, which later causes depression in these people. So, sleeplessness mediates the relationship between video games and depression. However, this may not be the case if females were used. Owning video games may bore them, which may make them depressed, but it doesn’t cause sleeplessness because they aren’t excited about the games. So the video games cause depression for them, but this isn’t mediated by sleeplessness.  Or if different video games were used, then sleeplessness may not mediate the relationship the same way.
Interaction of History and Treatment:   This is different than the Internal Validity threat History. That is when an event affects participants and causes the results found. THIS one is when results found from a study are not generalizable to other time periods. This usually has to do with whole eras. It mostly applies to old studies, when times were different, and results found may not be the same today.

Example: Researchers in the 40’s found that Japanese-Americans felt very low allegiance to the USA, and reported that their ethnic identity caused them great distress. These results were gathered during World War II, when Japanese-Americans were hated and oppressed by many people in the US in an extreme way. These results do not generalize to modern day as much. These results do not hold in the same way they once did, and Japanese-Americans may feel very differently about their allegiance to the USA, or about their distress over ethnic identity

Threats to Validity

Internal Validity: Reasons why inferences that the relationship between two variables is causal may be incorrect:

1. Ambiguous Temporal Precedence: Lack of clarity about which variable occurred first.

2. Selection: sometimes at the start of an experiment, the average person receiving one experimental condition already differs from the average person receiving another condition. This difference might account for any result

3. History: Events occurring concurrently with treatment could cause the observed effect.

4. Maturation: Naturally occurring changes over time could be confused with a treatment effect.

5. Regression: when units are selected for their extreme scores, they will often have less extreme scores on other variables, which can be confused with a treatment effect.

6. Attrition: Loss of respondents to treatment or to measurement can produce artificial effects if that loss is systematically correlated with conditions.

7. Testing: Exposure to a test can affect scores on subsequent exposures to that test, which can be confused with a treatment effect.

8. Instrumentation: The nature of a measure may change over time or conditions in a way that can be confused with a treatment effect. For example, a spring might become weaker over time and easier to push, artificially increasing reaction times.

9. Additive and Interactive Effects of Threats to Internal Validity: The impact of a threat can be added to that of another threat or may depend on the level of another threat.

Statistical Conclusion Validity: refers to the appropriate use of statistics to infer whether the presumed independent and dependent variables co-vary.

1. Low Statistical Power: An insufficiently powered experiment may incorrectly conclude that the relationship between treatment and outcome is not significant

2. Violated Assumptions of Statistical Tests: can lead to overestimating or underestimating the size and significance of an effect.

3. Fishing and the Error Rate Problem: Repeated tests for significant relationships, if uncorrected for the number of tests, can artificially inflate statistical significance.

4. Unreliability of Measures: weakens the relationship between two variables and strengthens or weakens the relationships among three or more variables.

5. Restriction of Range: Reduced range on a variable usually weakens the relationship between it and another variable.

6. Unreliability of Treatment Implementation: If a treatment that is intended to be implemented in a standardized manner is implemented only partially for some respondents, effects may be underestimated compared with full implementation.

7. Extraneous Variance in the Experimental Setting: Some features of an experimental setting may inflate error making detection of an effect more difficult.

8. Heterogeneity of Units: Increased variability on the outcome variable within conditions increases error variance, making detection of a relationship more difficult. The more the units in a study are heterogeneous within conditions on an outcome variable, the greater will be the standard deviations on that variable and on any others correlated with it. Possible solution is blocking - measure relevant respondent characteristics and use them for blocking or as covariates.

9. Inaccurate Effect Size Estimation: Some statistics systematically overestimate or underestimate the size of an effect.

Threats to Validity

Construct Validity: The validity of inferences about whether the cause-effect relationship holds over variation in persons, settings, treatment variables, and measurement variables.

1. Inadequate Explication of Constructs

2. Construct Confounding - operations usually involve more than one construct; failure to describe all the constructs may result in incomplete construct inferences.

3. Mono-Operation Bias - using only one operationalization of each construct. Single operations both under-represent constructs and may contain irrelevancies.

4. Mono-Method Bias - when all operationalizations use the same method, that method becomes part of the construct actually studied.

5. Confounding Constructs with levels of constructs - Inferences about the constructs that best represent study operations may fail to describe the limited levels of the construct that were actually studied

6. Treatment sensitive factorial structure - the structure of a measure may change as a result of treatment, change that may be hidden if the same scoring is always used. The structure itself becomes a construct rather than just the treatment, person, etc.

7. Reactive Self-Report Changes - the factorial structure and the level of responses can be affected by whether a person is accepted into the treatment or control group separate from the treatment effect.

8. Reactivity to the Experimental Situation - subjects may try to guess what the experimenter is studying and give results that they think the researcher wants to see. Sometimes referred to as the "placebo effect."

9. Experimenter Expectancies - these expectations become part of the molar treatment package. For example, teacher expectations about student achievement become self fulfilling prophecies.

10. Novelty and Disruption Effects - when an innovation is introduced, it can breed excitement, energy, and enthusiasm that contribute to success, especially if little innovation previously occurred.

11. Compensatory Equalization - when two conditions are being contrasted and the control group is compensated to equal out the inequity.

12. Compensatory Rivalry - public assignment of units to control and experimental conditions can sometimes cause rivalry in which the control group tries to show that it can perform as well as the experimental group.

13. Resentful Demoralization - members of a group receiving a less desirable treatment may be resentful and demoralized, changing their responses to the outcome measures.

14. Treatment Diffusion - the participants in one condition receive some or all of the treatment in the other condition. (Participants try to surreptitiously receive the treatment for its benefit)

External Validity: defined as the question, "to what populations, settings, and variables can this effect be generalized?"

1. Interaction of the Causal Relationship with Units: An effect found with certain kinds of units might not hold if other kinds of units had been studied.

2. Interaction of the Causal Relationship over treatment variations: An effect found with one treatment variation might not hold with other variations of that treatment, or when that treatment is combined with other treatments, or when only part of that treatment is used.

3. Interaction of the Causal Relationship with outcomes: An effect found on one kind of outcome observation may not hold if other outcome observations were used.

4. Interaction of the Causal Relationship with Settings: An effect found in one kind of setting may not hold if other kinds of settings were to be used.

5. Context Dependent Mediation: An exp

		https://www.youtube.com/watch?v=N40-_dlXIKI&feature=youtube_gdata_player
		https://www.youtube.com/watch?v=RgLHTKGc0S8&feature=youtube_gdata_player
		https://www.youtube.com/watch?v=2puogI0uSX4&feature=youtube_gdata_player
		https://www.youtube.com/watch?v=9qs76Q8tO5g&feature=youtu.be
		https://www.youtube.com/watch?v=E2Q3-OHyeUA
		https://www.youtube.com/watch?v=NggFVnMCB-M
	Field Research: Margret Mead Documentary	https://www.youtube.com/watch?v=K2FhWyulpb8
	Types of Psychological Research: Discovering Psychology	http://www.learner.org/series/discoveringpsychology/02/e02expand.html
	Problems with Secondary Sources in Historical Research: The Spanish American War & the Media Documentary	https://www.youtube.com/watch?v=8g8NpQsmxj4
	Ongoing issues with journalistic accuracy: The current case of NBC Anchor Brian Williams	http://www.cbsnews.com/videos/nbc-anchor-brian-williams-apology-fails-to-silence-critics/
	What is "Yellow Journalism?" - Overview Video	https://www.youtube.com/watch?v=0wFrAny77UY
		https://www.youtube.com/watch?v=VnggWn8mc1Y&feature=youtu.be
	A Case Study in Psychology: Genie Wiley	https://www.youtube.com/watch?v=KfOlPK2P_G8
	Lecture Series on Case Studies: Graham R Gibbs at the University of Huddersfield pt 1	https://www.youtube.com/watch?v=gQfoq7c4UE4
	Lecture Series on Case Studies by Graham R Gibbs at the University of Huddersfield pt 2	https://www.youtube.com/watch?v=o1JEtXkFAr4
	Lecture Series on Case Studies by Graham R Gibbs at the University of Huddersfield pt. 3	https://www.youtube.com/watch?v=b5CYZRyOlys
		https://www.youtube.com/watch?v=aN_VAs0vu4M&feature=youtu.be
	Video lecture from Dr. Kristine Florczak on Quantitative research	https://www.youtube.com/watch?v=eRDergihcPw
	Qualitative and Mixed Methods Research	https://www.youtube.com/watch?v=l5e7kVzMIfs&index=19&list=PLzNyCQk_pbAnNSLAudbheGsVuh9ZyW_Nk
	Lecture 7a	Social Science Publication Format: APA https://www.youtube.com/watch?v=pdAfIqRt60c
	Lecture 7b	https://www.youtube.com/watch?v=o4L3WaqBttk
	Basic concepts in probability video	https://www.youtube.com/watch?v=uzkc-qNVoOk
	Supplementary lecture on basic statistical concepts	https://www.youtube.com/watch?v=v-vCEae31PU
		http://web.cortland.edu/andersmd/STATS/normcurv.html
		http://www.abs.gov.au/websitedbs/a3121120.nsf/home/statistical+language+-+measures+of+shape
		https://www.youtube.com/watch?v=klRWllrdpcg&list=PLzNyCQk_pbAnNSLAudbheGsVuh9ZyW_Nk&index=21
		https://www.youtube.com/watch?v=1wGIieGeQ9U&list=PLzNyCQk_pbAnNSLAudbheGsVuh9ZyW_Nk&index=24
	Video on social science research publication pt 2	https://www.youtube.com/watch?v=klRWllrdpcg&list=PLzNyCQk_pbAnNSLAudbheGsVuh9ZyW_Nk&index=21
	Distinguishing between scholarly and popular sources	https://www.youtube.com/watch?v=-61AJb2VEMA&list=PLzNyCQk_pbAnNSLAudbheGsVuh9ZyW_Nk&index=22
	Reliability and Validity: Video 1	https://www.youtube.com/watch?v=DS8Hw0Ort4w&list=PLzNyCQk_pbAnNSLAudbheGsVuh9ZyW_Nk&index=29
	Reliability and Validity: Video 2	https://www.youtube.com/watch?v=LolwQXYjuh8&list=PLzNyCQk_pbAnNSLAudbheGsVuh9ZyW_Nk&index=33
	Reliability and Validity: Video 3	https://www.youtube.com/watch?v=CZQlqVswAq8&list=PLzNyCQk_pbAnNSLAudbheGsVuh9ZyW_Nk&index=34
	Threats to External Validity	https://www.youtube.com/watch?v=dfZ_FjLD7JM&list=PLzNyCQk_pbAnNSLAudbheGsVuh9ZyW_Nk&index=37
	Threats to Construct Validity	https://www.youtube.com/watch?v=O4vIsTxpVgc&list=PLzNyCQk_pbAnNSLAudbheGsVuh9ZyW_Nk&index=38
	Threats to Internal Validity	https://www.youtube.com/watch?v=F7kjR30tEAc&list=PLzNyCQk_pbAnNSLAudbheGsVuh9ZyW_Nk&index=39
	Threats to Statistical Conclusion Validity Lecture	https://www.youtube.com/watch?v=b3txRcXZwzY&feature=em-upload_owner#action=share