Testing Natasha

Can a seventeen-year-old girl truly "see" inside a person's body? Ray Hyman and colleagues conducted tests to search for the truth inside The Girl with X-Ray Eyes.

Our assignment might seem straightforward. A seventeen-year-old Russian girl, Natasha Demkina, says she can look at people and "see" the status of their internal organs. The Discovery Channel asked Richard Wiseman, Andrew Skolnick, and me to test her claim for their television program, The Girl with X-ray Eyes. You might think that testing Natasha's claims would be routine. The test of a psychic claim, however, is rarely cut-and-dried. Most such claims do have much in common. Each also offers unique challenges. We had to conduct the test of Natasha's claim to fit the constraints of a television program. We had only a month to devise a protocol that would be acceptable to all parties. After everyone agreed to the procedure, we had less than a week to locate a testing site in New York City and to find seven willing and suitable test subjects. [1]

The Claim and Its Support

Monica Garnsey, director and producer of the program, told us how Natasha operates and what she claimed to do. Many news sources and reports on the Internet described her accomplishments. (This information was consistent with what we observed when Natasha diagnosed volunteers at the Open Center in New York City the day before the test.) Garnsey e-mailed us the following information from Russia, where she was taping material for the television program:

I double-checked a few things with her last night. Since the age of ten, a few days after having a religious dream, and also having had an operation to have her appendix removed that went wrong, swabs were left in her and she had to have another operation, Natasha has claimed to be able to see into people. . . . Natasha can see through clothing, but not see what someone is holding behind their back. She cannot see inside people if she shuts her eyes. Daylight is better. She does not need to talk to them to diagnose. She can also diagnose from a photograph. She usually scans people all over first, by making them stand up fully clothed and looking them up and down; delivers a general diagnosis; and then goes into more detail when the patients have discussed their concerns with her. She says she can certainly see ribs, heart, lungs, initially in general "like in an anatomy book," but can see right down to the cell level if she concentrates. She says that she can examine the whole body, but it can give her a bad headache if she does too much. The idea of restricting the test to the chest area appeals [to her], though her claims extend further than that.

Natasha's story is like thousands of other accounts. Alleged psychics and their supporters make claims that, if true, defy the physical limitations and laws of modern science. The proponents support the reality of these claims with testimonials of outstanding successes. They argue for the reality of the claim passionately and unreservedly. Although some proponents have had scientific training, none of the supporting evidence comes from well-controlled scientific studies.

In the long history of psychical research, not one of these claims has produced convincing scientific evidence for the existence of paranormal ability (see Joe Nickell's column in this issue, p. 18). A few researchers have claimed that they did have scientific proof for a paranormal claim. Scrutiny by other scientists, however, showed that the "scientific proof" had serious flaws. Furthermore, none of these claims could be independently replicated.

Natasha Demkina poses for photographs after being tested by CSICOP and CSMMH for the Discovery Channel program The Girl with X-ray Eyes. Her friend, Svetlana Skarbo, who acted as her translator, holds a cell phone over which they had sent and received text messages from unknown parties during the test (in violation of test protocols.) On the left is Barrie Cassileth, Ph.D., Chief of Integrative Medicine Services at Memorial Sloan-Kettering Cancer Center, who helped to recruit subjects for this preliminary study. Photo credit: Andrew A. Skolnick.

The evidence supporting Natasha's abilities comes from selected anecdotes of reactions to her readings. No matter how subjectively compelling, the context of such readings makes it impossible to separate how much of the apparent success is due to such possibilities as: guessing; external clues from the client's physical appearance and observable behavior; feedback from the client's spoken and bodily reactions; or actual paranormal powers. A meaningful test would allow Natasha to show her powers and, simultaneously, control for guessing and the use of normal sensory clues.

Problems With Testimonial Support of Natasha's Claims

The stories told by Natasha's proponents are consistent with her having X-ray vision. This does not show that she does have X-ray vision because the same stories are consistent with many other alternatives. Two possibilities are the following: 1) her statements have no connection with the client's condition but appear to do so because of luck, selective reporting, and/or other reasons that I will discuss; or 2) her statements accurately reflect the subject's condition, but this information comes through normal means such as the subject's appearance and behavior. Consider, first, the ways that her statements can falsely appear to describe the patient's condition.

For Discovery Channel publicity photos following CSICOP-CSMMH's test, Natasha Demkina examines the seven volunteer test subjects. The subjects are wearing opaque glasses to prevent communication through eye movement. The head test proctor, Ray Hyman, is sitting in the bottom right corner. Photo credit: Andrew A. Skolnick.

She Might Have No Knowledge About the Client's Condition But Get Credit Anyway

Natasha has been giving readings to a steady flow of clients for more than six years. By now the number of such readings is huge. Her supporters naturally emphasize the most striking examples of apparent hits. The number of diseases and internal parts that could be defective is limited. Some conditions, such as cancer and heart problems, are more common than others. We should expect that her supporters will find some examples of "correct" diagnoses. With so many diagnoses, a certain number will match the client's condition just by chance.

To evaluate a diagnostic procedure properly we need to clearly decide what is a "hit" and what is a "miss." Most important, we should set the criteria before we know the outcome. In Natasha's readings, no clear and objective standards were ever established. This allows for her generally vague utterances to be retrofitted to what the client or observer knows to be true. An example of such retrofitting occurred when Natasha was doing a reading in London. Dr. Chris Steele, described by The Daily Mail (January 29, 2004) as one of her champions, was observing. The newspaper quotes him as saying, "Natasha doesn't know any medical terms at the moment. With one person this week she was trying to describe a kidney stone, and her translator came up with the words, 'sand' and 'gravel' before I suggested stones. When kidney stones start off, they do look like sand." Dr. Steele gives her credit for correctly diagnosing kidney stones. Yet we have no idea what Natasha was "seeing" or what she had in mind. Dr. Steele made the medical diagnosis, not Natasha.

Other features of Natasha's readings foster the illusion of accuracy. When she tells clients something that agrees with previous medical diagnoses, they credit her with a hit.

Similarly, when she tells the clients something that disagrees with previous medical diagnoses, they still credit her with a hit; the clients and her supporters argue that she picked up on something that the medical professionals missed. We witnessed some examples of this when we watched her giving readings to volunteers at the Open Center in New York. She told one volunteer that she saw a problem with his right shoulder. After the reading, this volunteer told Monica that he had not previously realized something was wrong with his shoulder. Neither his previous medical examinations nor anything in his experience suggested something was wrong with his shoulder. I thought, as a result, he might be skeptical about Natasha's claim. Instead, he was impressed. He decided she had detected a problem that neither he nor his doctors had noticed.

Natasha Demkina stands between her friend Svetlana Skarbo (who served as her translator instead of the one hired by Discovery Channel) and Richard Wiseman, who helped to design and conduct the CSICOP-CSMMH test. Photo credit: Andrew A. Skolnick.

Possibilities of Natasha Picking Up Clues by Non-paranormal Means

I have described just some ways that testimonials can appear to support Natasha's claim even if she is picking up no information about her clients. Those possibilities would suffice to make such testimonials useless as evidence for her ability. The testimonials become even more suspect when we realize how the circumstances of her readings allow her to pick up information about her client without having X-ray vision. Natasha is looking directly at her client when she does her diagnosis. This means that we cannot rule out the possibility that she is picking up clues from subtle (and not-so-subtle) client reactions. To make matters worse, the clients begin a session by asking Natasha questions about their concerns. This provides obvious clues about their condition. I watched one reading where the client began asking Natasha about her back. This narrows considerably the number of possibilities that Natasha needs to consider. Natasha can also gain considerable information from verbal exchanges with the client.

Another source of clues is how the clients react, both verbally and nonverbally, to her statements. Some of her clients say that they find it unsettling when Natasha is staring at them. This could enhance the tendency for individuals to react to her statements with subtle, unwitting bodily movements, breathing changes, pupil dilations, and other signs of emotional and cognitive states. Although psychological research has documented how humans frequently provide unconscious clues to their current thoughts and emotions, most people seem unaware of this possibility. The research also shows that subtle clues can influence us without our consciously realizing it.

One classic case involved the German horse Clever Hans. [2] In the early twentieth century, Hans became a celebrity in Germany and throughout the world. People could ask him questions about addition, the identity of musical pieces, about foreign words, spelling, and many other topics. Hans would answer by tapping his hoof or by nudging an alphabet board with his nose. He usually was correct. Prominent educators certified that he had the intelligence and competence of a thirteen- or fourteen-year-old German student. Oskar Pfungst, a German psychologist, investigated Hans with exemplary thoroughness. He eventually discovered that Hans was clever only in having "horse sense." Typically, a questioner would focus on the horse's right hoof, which Hans used to tap out the answer. When questioners focused on the hoof, they would almost imperceptibly lean forward and become tense as they watched the horse tap out the answer. This slight leaning and tensing were Hans's cues to begin tapping. When Hans had tapped the appropriate number of times, the questioner would unconsciously relax and move his or her head upwards very slightly. Often this movement was one millimeter or less. This was Hans's clue to stop tapping.

Pfungst then carried out experiments to confirm this finding. He played the role of Hans. He would invite people to stand beside him and think of a number. Pfungst would then begin tapping with his right hand. He would stop when he thought he detected a very slight bodily movement-usually a very slight displacement of the subject's head. These movements were extremely subtle, rarely more than a millimeter in extent. Pfungst amazed his volunteers, stopping his tapping at the number they had in mind.

Pfungst tried this experiment with twenty-five persons ranging in age from five years to adult. He succeeded in picking up cues from all but two of them. They insisted they were unaware of giving him any information. Pfungst used the same method to divine other kinds of thoughts the subjects had in mind. The subjects again denied that they had provided any clues about what they were thinking. Other psychological experiments have confirmed these results. Some skilled performers have made careers out of pretending to read minds when, in fact, they were relying upon subtle and unwitting clues provided by their volunteers.

Some reports supporting Natasha's claim describe outcomes consistent with the possibility that she is picking up such clues. For example, a Russian reporter says that he became a convert to Natasha's cause when she found the exact spot on his arm where he had fractured his wrist many years before. In another case, a reporter from a British tabloid validated Natasha's ability when Natasha succeeded in identifying the location of the fractures she had received in an accident. Both cases seem ideal for picking up the sorts of clues that Pfungst found that most people provide without realizing they are doing so.

What I have just written does not show that Natasha lacks X-ray vision. We do not know from the evidence offered by her proponents whether she does or does not have a paranorm-al capacity to see into people's bodies. What we do know is that the accounts that seem to support Natasha's claim are consistent with both normal and paranormal possibilities. We also know that nonparanormal mechanisms can and do operate in the real world. We do not know that paranormal ability, such as that claimed for Natasha, exists. So far, no one has displayed such ability with scientific credibility. Given these two possible explanations for Natasha's apparent successes, rationality tells us to bet on the nonparanormal one. We should demand convincing evidence that is scientifically acceptable before we give credence to the paranormal claim.

The Test Protocol

With input from Richard and me, Andrew wrote the test protocol, titled "Test Design and Procedures for Preliminary Study of Natasha Demkina." The goal was to make every aspect of the test explicit. The protocol stated how we would conduct the test and how we would interpret the results. We wanted all parties to be clear about what would and would not be considered a "successful" outcome. What makes a scientific experiment or a test meaningful is just such an explicit commitment to the interpretation of the outcome before we observe the data. This is a critical distinction between the post hoc interpretation of testimonial evidence and the prior commitment to specified outcomes of a meaningful test. Natasha's defenders apparently fail to grasp this essential point.

The written protocol protects the interests of all parties. Natasha and her supporters had the opportunity to study the document, to suggest modifications, and finally to agree or disagree with its provisions. The protocol also protects the investigators against a variety of false accusations about how we conducted the test.

We made sure to include in the protocol the statement that the "test is not in any way a definitive test. Deciding the truth of Natasha's claims with comfortable certainty is too simple and brief. It can only help to decide whether further studies of Natasha's claimed abilities are warranted." This statement is worth elaborating. Understanding what the test can and cannot do is essential. Even under ideal circumstances this test could not clearly decide if Natasha does or does not have X-ray vision. Any scientific hypothesis-especially a paranormal one-cannot be confirmed or disconfirmed by one test or one experiment. Scientific investigation requires a series of experiments. Each new experiment builds on the results of previous ones. The more we learn from the early experiments, the better we can understand what we need to control and what we can safely ignore. If the hypothesis is implausible and/or controversial-as Natasha's claim certainly is-then the original investigators must replicate their findings. In addition, independent investigators must also replicate the findings before they gain scientific credibility.

We knew that our test could not distinguish between two possibilities: (1) she can make correct matches using external clues; or (2) she can make correct matches using paranormal X-ray vision. The alternatives we could control or reduce were that she gets correct matches just by luck or that her correct matches are due to those factors that make vague statements seem like hits.

We were also aware that our test could only detect a large effect. Natasha's claim can be considered in several contexts. The testimonials imply that she is highly accurate. This has practical consequences. If clients are depending upon her for medical diagnoses, Natasha's readings should be reliable. Otherwise, she can do much harm. Of course, Natasha could possess paranormal powers, but they could be weak and erratic. Such unreliable and weak ability would be useless for medical diagnosis, but would still be of theoretical interest. We lacked the resources and time to try to detect such a weak effect. We used all our resources to obtain seven subjects. If we had been trying to test for a moderate or weak effect, we would have had to use many more subjects. Given the constraints of our task, this was impossible. Our test, then, was aimed at detecting a large effect. We reasoned that if she possessed the reliability of diagnosis that her proponents claimed, our test would reveal this. Such an effect would encourage us to investigate her abilities in more detail.

The outcome of the test could be from zero to seven correct matches. We set the criterion for success at five correct matches. We clearly stated this criterion in the test protocol and all parties agreed to this in advance. Although Natasha's mother says that her daughter never makes a mistake, we did not want to demand that Natasha perform perfectly. We wanted to give her some margin for error. Keep in mind that if she got five or more correct this would be consistent with her having the X-ray power that she claims. Yet it would also be consistent with the possibility that she was matching the target condition by normal means such as the appearance and behavior of the subjects.

The Test

Richard Wiseman, Andrew Skolnick, and I collaborated in designing the test. We arrived at a mutually satisfactory plan after exchanging several e-mails. The task of finding appropriate subjects, and coordinating the many details was left to Andrew. He had less than one week to accomplish all this. He had to do this from Amherst, more than 350 miles from New York City.

Austin Dacey, executive director of the Center for Inquiry-MetroNY, obtained an excellent set of rooms for the test at the City College of New York and helped recruit several subjects. Dr. Barrie Casselith, Chief of Integrative Medicine Service at Memorial Sloan-Kettering Cancer Center, helped us with the daunting task of assembling seven appropriate and willing subjects. On the morning of the day of the test we learned that two of the subjects had withdrawn. Again, Andrew and Austin saved the day by finding two replacements at the last moment. (Andrew's separate article about certain aspects of the tests follows mine.)

During the test, we seated the seven subjects in a semicircle facing the chair where Natasha sat. Each volunteer had an internal condition that should be easy to detect if Natasha's claim is correct. The target conditions were as follows: One patient had metal surgical staples in his chest from open heart surgery; one had a section of her esophagus surgically removed; one had a large section of one lung removed; one had an artificial hip replacement; one had a missing appendix (we discovered afterwards that another subject also had a missing appendix, which he didn't mention when we recruited him. Natasha chose neither of these two as the one with the missing appendix); one had a large brain tumor removed and now has a large hole in his skull covered by a metal plate; and the final subject had none of these target conditions.

During the test, when Natasha was looking at the subjects, the subjects wore sunglasses whose lenses were covered with opaque tape. This prevented the subjects from knowing when Natasha was looking at them. This also prevented Natasha from picking up clues from their eye movements or pupillary dilations (which are a sign of emotional reaction). Before the test, I instructed and rehearsed the subjects on how to behave. They were to sit as still as possible when Natasha was in the room. If Natasha needed to observe them in a standing position, I would tell Natasha to turn her back while they stood up and when they sat again. We used similar precautions if Natasha needed to look at them in profile. These precautions reduced the possibility of reactions by the subjects from knowing which target condition Natasha was currently studying. We also wanted to reduce external movements (for example, the subject with a hip replacement might give herself away from her efforts to stand or to change the position of her body). [3]

The test room was large and had chairs for our seven subjects, for Natasha and two interpreters. One interpreter was Natasha's friend Sveta Skarbo. We allowed her in the test room to make Natasha feel comfortable. The other interpreter was supplied by the Discovery Channel. Ideally, only I, as the head proctor, Richard Wiseman as my co-investigator, Natasha and the two interpreters, and the seven subjects should have been present during the test. The realities of television production and the requests of Natasha's companions forced us to compromise here, and in some matters of protocol. The test room also included a television crew of three persons from the production company (Shine, Ltd.); Austin Dacey, who was videotaping the proceedings for CSICOP; Joe Nickell as an observer; a still photographer from the Discovery Channel; and Will Stewart, a British journalist living in Russia who was acting as a representative for Natasha. Except for the subjects (and Austin Dacey), everyone in the test room, including myself, was blind to the condition of each subject.

A small room, attached to rear of the test room, was used for briefing Natasha. We could retreat to this room when we wanted to discuss matters out of sight and hearing of the subjects. Because Andrew was in charge of recruiting the subjects and was not completely blind to their conditions, he stayed out of the testing room. He remained in the briefing room during the entire test (which lasted more than four hours). We used this room to brief Natasha before each of the six required matches (once she had made six matches, the seventh was determined by default). Before each trial Andrew gave her a clear description, along with images and diagrams, of the target condition that she was to match to a subject. We also discussed any of Natasha's questions or concerns in this room.

Andrew and I met with Natasha in this room before the test to review the procedure and to remind her about the details of the protocol. She had agreed to this protocol, which Monica had shown her five days previously. We reviewed each condition that we would ask her to detect. She expressed concerns about the removed appendix and the resected esophagus. She was worried that if the appendix had been removed long enough ago it might have grown back. Andrew assured her that appendices do not grow back. Her concern about the resected esophagus was that individuals might normally differ in the length of their esophagus and this could mislead her. Andrew told her that instead of the length she should look for the scar that completely encircled the place where the two ends of the resected esophagus had been surgically joined.

The test consisted of six trials. On each trial Andrew gave Natasha a test card that clearly described, in Russian and English, the condition she was to match to a subject. The card contained an illustration of the target organ or condition. Andrew also showed her relevant illustrations from an anatomy text. When she was satisfied, I accompanied Natasha to the test room, where she sat between the two interpreters and equidistant from each subject. After Natasha had studied the subjects for the given condition, she chose the subject she believed had the specified condition. She would circle the subject's number on the test card and both of us would sign the card. We then returned to the back room to prepare for the next condition and trial.

We wanted to make the test as comfortable and nonstressful for Natasha as possible. I made sure not to rush or pressure her in any way. I gave her all the time she wanted to make each match. She took one hour to make the first match-which was to find the subject who had a large section of the top of her left lung surgically removed. She required more than four hours to complete the matches of conditions to the seven subjects. Throughout this process I repeatedly asked her if she was comfortable and if we could do anything to make the process more agreeable to her. She could ask for a break in the proceedings whenever she wished. Her mother had decided to remain outside both the test and briefing rooms because she wanted to be with Natasha's younger sister. Midway through the proceedings, Natasha told us she would feel better if her mother could be in the briefing room. I immediately agreed to her request. [4]

The Outcome

Natasha succeeded in correctly matching four target conditions out of a possible seven. Our protocol required that Natasha get five or more correct matches to "pass" our test.

Understandably, Natasha's supporters were disappointed. They expressed their misgivings about the test on the television documentary, in media interviews, on Web sites, and through e-mails. They accused the testers of bias and of deliberately manipulating the procedure to prevent Natasha from succeeding. Natasha has complained that if she had gotten five correct she would have been a success. Isn't four close enough?

Our answer is that five was the minimum score that everyone agreed upon. It was also the minimum score that would convince us of a possible ability to diagnose subjects with sufficient reliability to be useful. We designed our test to detect a large effect. We were looking for something that would distinguish Natasha's claims from many similar ones. We wanted a good reason to justify using the additional time and resources to investigate her ability further.

Although Natasha's score did not meet our criterion for "success," it is possible that she can pick up information about the subject's condition. Some of her choices might show some accuracy on her part, although of a low level. If this is true, her correct matches could be the result of three possibilities:

1. She gathered some information paranormally. That is, she can see into people's bodies, but imperfectly.
2. She gathered information by deliberately exploiting available clues such as outward appearances and behavior of the subjects.
3. She obtained information unconsciously from available clues. To me, this is the most likely explanation, other than chance or in addition to chance. Much recent work in psychology demonstrates implicit learning: how people unconsciously learn to exploit a variety of clues, often subtle ones.

Both inherent and unforeseen limitations of our test provided possible clues to the target conditions for some subjects. I already discussed the daunting task of finding seven appropriate subjects. We had to settle for a less than optimal set of subjects. These subjects differed sufficiently in outward appearance to provide possible clues about their conditions. Another problem occurred through two violations of the test protocol. Together these problems created the possibility for identifying the target conditions-by external, normal means-for the following four subjects:

1. The "control" subject, the one who had no internal medical condition, was obviously the youngest of the group. He also looked in good physical condition and appeared much healthier. He was a good candidate for the person with no defects.
2. The subject with the staples in his chest (because of major heart surgery) was male, the oldest of the group and looked the least healthy. He was an obvious choice for the person with the staples in his chest.
3. A breach of protocol occurred on the first trial. Natasha posed a question and her interpreter translated it aloud in front of the subjects. The question, contrary to our protocol, allowed the subjects to know that Natasha was looking for the subject with part of her lung removed. Here it was possible that, knowing which condition Natasha was looking for, the subject with the missing lung might have given herself away through bodily reaction.
4. After the test was over, I learned that Natasha and her companions, because of an apparent misunderstanding, had arrived at the test site before we had expected them. They waited outside the test building where they reportedly observed at least two of the test subjects climb the long flight of stairs and enter the test building. This breach of protocol may have provided them clues about which subjects did or did not have the artificial hip.

We do not know if Natasha took advantage of the clues I've described in the previous four paragraphs. However, it is suggestive that these were just the four subjects for whom Natasha achieved her correct matches. The probability that she was relying upon nonparanomal clues increases when we consider her misses. She wrongly picked the subject who was wearing a baseball cap as the one who had the metal plate in his head. Conceivably, she picked this subject because one might assume (falsely in this case) that the subject was trying to cover a scar on his head. We should also emphasize that her failure to correctly match the subject with the metal plate in his head further argues against any fledgling paranormal powers. If she truly can see into bodies, she should have easily detected the large area of missing skull along with the metal plate covering the hole.

Our test included five subjects for whom external clues were available concerning their internal condition. The clues correctly pointed to the true target condition for four subjects. The external clue for the fifth subject falsely pointed to the hole in the skull. In each of these five cases Natasha made her choice consistent with how the external clue was pointing.

Because a single test, even one done under ideal conditions, cannot settle a paranormal claim, we conceived our test as the first stage of a potential series. The first stage would not necessarily rule out nonparanormal alternatives. If Natasha could pass the first stage, this would justify continuing onto the next stage. If she passed that stage, then we would continue studying her claim. On the other hand, if she failed at any of the early stages, this would end our interest in her claim.

Keep in mind that the burden of proof belongs to the parties making an extraordinary claim. Extraordinary claims require extraordinary proof. Our test had its limitations. None of these limitations, however, worked against Natasha's claim. If anything, they may have artificially enhanced her score. Our task was not to prove that Natasha does not have X-ray vision. Rather, Natasha and her supporters had the responsibility to show us that she could perform well enough to deserve further scientific investigation. This they failed to do.

Acknowledgments

Notes

1. We debated about how to refer to the seven volunteers who had conditions which Natasha had to detect. Each of the candidate terms such as volunteer, participant, patient, or client seemed ambiguous or not quite correct. Although not completely satisfactory, we decided to refer to these individuals as subjects.
2. Pfungst, O. 1911. Clever Hans. New York: Henry Holt & Co. Also see Vogt, E.Z., and Hyman, R. 2000. Water Witching U.S.A. Chicago: University of Chicago Press.
3. Here is another compromise we had to make in the test. Ideally, everyone in the test situation should be blind as to the true target condition for each subject. In our case, the subjects were not blind to their own conditions. Because the subjects had to be in the test room and Natasha had to study them visually, the test lacked this blindness. The use of the opaque sunglasses hopefully kept the subjects blind as to which target condition Natasha was looking for on a given trial, but this is not completely satisfactory.
4. At the start of the test some initial confusion existed as to who would be allowed into the test and briefing rooms. This was quickly corrected and Natasha's mother and Will Stewart were given the option of staying in one of these rooms.
   

Feedback | Reverse links for this page | Translate this page