Stats Articles

test

2010-08-29T18:42:00.001-07:00

Read and summarize the following article. Pay careful attention to HOW the data was collected.

Suffering Adds Up in a Hurry In Survey of Tsunami Survivors

By David Brown

Washington Post Staff Writer

Saturday, January 15, 2005; Page A01

CALANG, Indonesia -- For two days this week, an Australian doctor and his Acehnese assistant knocked on every 10th door in Calang, a seaside town on Sumatra island that was decimated by the Dec. 26 tsunami. At each house, they asked the same brief questions, thanked the residents and departed after about 10 minutes.

What they learned in their bare-bones, random statistical survey, conducted for the International Rescue Committee, was chilling.

Before the tsunami, 8,700 people lived in Calang; now, that number is 2,500, and a third of those are displaced from other towns. Sixty-five percent of households have had a death in the immediate family. Twenty-two percent have taken in orphans, usually more than one. Only 8 percent of the population is younger than 5, and 85 percent of those children have had diarrhea in the past two weeks.

The survey, conducted by Richard Brennan and his assistant, Kamaruddin, has provided the most precise look to date at the tsunami's effects on the people living in the worst-hit part of the worst-hit country.

The survey produced more than numbers, however. It will help the International Rescue Committee, a humanitarian relief agency based in New York, plan how best to spend the $7 million it has budgeted to assist Indonesian survivors of the tsunami.

Relief organizations often use such systematic assessments during man-made disasters involving war, famine and forced dislocation, in which people's needs may not be obvious.

In natural disasters, relief experts said, the aid requirements are usually more clear-cut because the disruption tends to be short-lived and the sufferers start off healthy. But in the case of the recent tsunami, the magnitude of damage in Indonesia indicates that its effects will be severe and long-lasting.

Brennan's survey of Calang's households began Monday on one side of the tiny peninsula on the west coast of Sumatra. Two Indonesian navy ships were moored offshore.

The flat land beyond the beach where the tsunami came ashore was covered with fallen palm trees and trash. There were a dozen green military tents, and an orange one containing an Indonesian Red Cross clinic. A steady stream of refugees came and went, with people picking up boxes of flavored noodles and sorting through sodden piles of donated clothes.

The day before, Brennan had canvassed the residential area, built on slopes that drop toward the sea, and decided he needed to visit one in 10 homes to compile enough data to make the results of the survey valid.

To make it truly random, however, his starting point couldn't be the first house he encountered, but rather a point that could not be predicted. So his first scientific move in Calang was to ask someone for some paper money and then to read off the last digit in its serial number, which was a 2.

The scientific approach to disaster relief can be controversial. Some critics think it is foolish to prove the existence of "obvious" needs and cruel to ask for information before giving help. Brennan, 45, who heads the International Rescue Committee's health activities program, is not one of them.

"Okay," he said, "we start counting after the second house."

This meant that the third house near the beach became Household No. 1, the first to be surveyed. It was a one-room shack built from scavenged wood, corrugated metal sheets and dried palm branches.

Brennan approached a woman squatting inside who was surrounded by children. His first question was: "Where are you from?" The second was: "How long have you been here?" Then: "How many people slept here last night?"

The interpreter, whose only name is Kamaruddin and who uses the nickname Odon, took down the answers. At the third question, the woman seemed confused, and then Odon became engaged in an extended conversation with a man outside the shelter.

"Odon, don't get distracted, please," said Brennan, who hoped to complete the survey that day so he could move onto outlying villages.

"She can't count," the interpreter replied.

It turned out the answer was 13 -- one of the larger households the team would encounter. Brennan still needed the age and sex of each person. And he had a dozen more questions after that.

When the encounter was finally over, the visitors descended into a wide gully that had been made by the retreating water. It was full of tangled fronds, branches and uprooted trees. On the other side rose the hill where the next group of houses stood. Odon stopped in the middle of the gully.

"Rick, I think we need to ask fewer questions," he said. "They don't like just to give information and get nothing. If you go to someone's house three times and give them nothing, they will get angry."

Brennan said he was acutely conscious of the problem.

He told Odon he had used a satellite telephone that morning to try to reach the International Rescue Committee's emergency coordinator in Banda Aceh and request that a water and sanitation engineer be sent to Calang. Even without an assessment, he said, it was clear that people needed clean water and latrines. Unfortunately, he couldn't reach her.

"So what have you done?" demanded Odon, 29. Brennan said he had divided a large box of medicine and bandages between the Indonesian Red Cross clinic and a German clinic and promised that his office would send the engineer as soon as it could.

"Is that convincing?" Brennan asked with a doubtful smile. "I need to convince you before you can convince them."

But nobody in Calang refused to talk to the surveyors. Everyone was gracious. Two people asked for cigarettes, and two asked for rice. The team had only thanks to offer, though at some stops they let children peep at a digital photograph of themselves.

Up and down the hillsides the team went, stopping at every 10th house. If no one was home, they moved to the next house. They defined each "household" as a group of people who ate from one pot; some large structures contained more than one.

Each dwelling was a testament to human resourcefulness. The humblest was a lean-to where five people appeared to sleep on the ground, except for an infant in a hammock. The most elaborate was a wood-frame house with rafters made of galvanized pipe, with woven mats covering the floors. With rain almost constant, the settlement was a cat's cradle of lines hung with drying clothes.

Brennan's survey included questions about where each household obtained water and how much it used each day. The team inspected and estimated the capacity of buckets, jerrycans and former chemical containers used for storing water.

The visitors also made one measurement of each young child they encountered. The "mid-upper-arm circumference," known to epidemiologists as MUAC, changes little in children between 6 months and 5 years of age and is used as way of screening for malnutrition.

Odon, although not completely convinced of the survey's usefulness, assisted willingly. He kept the multicolored MUAC tape around his shoulder bag, and he and Brennan exchanged jokes about the adventures of "Mr. MUAC."

Over the course of two days, a solid statistical portrait emerged of the tsunami's effects on Calang.

There were 316 households, with an average size of 7.6 people. Most had lost at least one member, but the precise mortality rate remained unknown because some families had been wiped out and some survivors had left.

Children and the elderly seemed to have died at a slightly higher rate than adults -- although the town was filled with orphans. Nationwide, children younger than 5 constituted 10 percent of Indonesia's population in 2003; in Calang, the number is 8.2 percent. People older than 60 made up 7.9 percent of the nation's population before the tsunami; in Calang, the figure is 4.5 percent. The oldest survivor in the area was a woman of 80; a woman in the same household who was over 100 had died.

Yet considering the enormity of the disaster, the survivors of Calang seemed to be in generally good condition. The orphans were all being cared for, and there were no cases of acute malnutrition. Almost everyone defecated outdoors, nobody was drinking from a "protected" water source, and childhood diarrhea -- spread by fecal contamination of food and water -- was common. Yet no cases suggested cholera or dysentery. Sick people were seeing doctors, and most reported being happy with the treatment they received.

Brennan, who planned to share his findings with the World Health Organization's office in Banda Aceh, said the survey convinced him that the International Rescue Committee had made the right decision to concentrate on improving water quality and building latrines in Calang and to deploy its medical resources elsewhere.

In one house, the surveyors met a 32-year-old carpenter they had seen earlier at the Indonesian Red Cross clinic, where he had brought his infant daughter. The baby had a large, tender swelling below her jaw. Brennan and another doctor had concluded that it might be an infection of a salivary gland, and they prescribed an antibiotic.

When Odon reached the survey question about whether people were satisfied with medical treatment they had received in the last week, Brennan grinned at the carpenter and warned, "Careful what you say, mate."

The man replied, "It's hard to say. We'll have to see how the baby does when she takes more medicine."

Brennan rejoined: "Good answer."

Data plans

2010-07-01T09:05:00.000-07:00

Read this:

Data plans

Energy

2010-06-21T14:35:00.000-07:00

Articles to finish up Fall Semester 2009

2009-11-24T10:56:00.000-08:00

First article:

Read below about TV and happiness.

Second article (due the 4th):

Tufte. Either scroll down until you see Article #1, Tufte,

Or click on this: tufte

NOTE: Parts of this article are long! Spend about 5 minutes each on SparkNotes and the iPhone. You do not need to spend more time that on these long essays/videos.

Third articles (due the 11th):

Scroll down to right below the TV article and read the article on tsunami data collection. Just write a short, 1 paragraph summary on how the data was collected.

TV happiness

2009-02-17T13:25:00.000-08:00

This is a great first article to get us started. Read the article thoroughly enough to be able to discuss it in class. The questions are at the end for you to answer (one homework stamp). Due by Friday, November 20th.

What Happy People Don’t Do

By RONI CARYN RABIN

Published: November 19, 2008

Happy people spend a lot of time socializing, going to church and reading newspapers — but they don’t spend a lot of time watching television, a new study finds.

That’s what unhappy people do.

Although people who describe themselves as happy enjoy watching television, it turns out to be the single activity they engage in less often than unhappy people, said John Robinson, a professor of sociology at the University of Maryland and the author of the study, which appeared in the journal Social Indicators Research.

While most large studies on happiness have focused on the demographic characteristics of happy people — factors like age and marital status — Dr. Robinson and his colleagues tried to identify what activities happy people engage in. The study relied primarily on the responses of 45,000 Americans collected over 35 years by the University of Chicago’s General Social Survey, and on published “time diary” studies recording the daily activities of participants.

“We looked at 8 to 10 activities that happy people engage in, and for each one, the people who did the activities more — visiting others, going to church, all those things — were more happy,” Dr. Robinson said. “TV was the one activity that showed a negative relationship. Unhappy people did it more, and happy people did it less.”

But the researchers could not tell whether unhappy people watch more television or whether being glued to the set is what makes people unhappy. “I don’t know that turning off the TV will make you more happy,” Dr. Robinson said.

Still, he said, the data show that people who spend the most time watching television are least happy in the long run.

Since the major predictor of how much time is spent watching television is whether someone works or not, Dr. Robinson added, it’s possible that rising unemployment will lead to more TV time.

1. What is the name of this data collection method?

2. This is a classic example of the difference between association and causation. Explain:

3. Discuss what lurking variables might be influencing this study.

4. How might a researcher study causation for this scenario? What would make causation difficult to determine?

Article #4

2008-10-20T13:48:00.000-07:00

Read and summarize the following article. Pay careful attention to HOW the data was collected.

Suffering Adds Up in a Hurry In Survey of Tsunami Survivors

By David Brown

Washington Post Staff Writer

Saturday, January 15, 2005; Page A01

What they learned in their bare-bones, random statistical survey, conducted for the International Rescue Committee, was chilling.

Relief organizations often use such systematic assessments during man-made disasters involving war, famine and forced dislocation, in which people's needs may not be obvious.

Brennan's survey of Calang's households began Monday on one side of the tiny peninsula on the west coast of Sumatra. Two Indonesian navy ships were moored offshore.

"Okay," he said, "we start counting after the second house."

This meant that the third house near the beach became Household No. 1, the first to be surveyed. It was a one-room shack built from scavenged wood, corrugated metal sheets and dried palm branches.

"Odon, don't get distracted, please," said Brennan, who hoped to complete the survey that day so he could move onto outlying villages.

"She can't count," the interpreter replied.

It turned out the answer was 13 -- one of the larger households the team would encounter. Brennan still needed the age and sex of each person. And he had a dozen more questions after that.

Brennan said he was acutely conscious of the problem.

"Is that convincing?" Brennan asked with a doubtful smile. "I need to convince you before you can convince them."

Over the course of two days, a solid statistical portrait emerged of the tsunami's effects on Calang.

The man replied, "It's hard to say. We'll have to see how the baby does when she takes more medicine."

Brennan rejoined: "Good answer."

Article #3

2008-10-06T07:27:00.000-07:00

Just read it and write a short summary. Be prepared to discuss:

Shattering the Bell CurveThe power law rules.
by DAVID A. SHAYWITZ
Tuesday, April 24, 2007

Article:
Life isn't fair. Many of the most coveted spoils--wealth, fame, links on the Web--are concentrated among the few. If such a distribution doesn't sound like the familiar bell-shaped curve, you're right.
Along the hilly slopes of the bell curve, most values--the data points that track whatever is being measured--are clustered around the middle. The average value is also the most common value. The points along the far extremes of the curve contribute very little statistically. If 100 random people gather in a room and the world's tallest man walks in, the average height doesn't change much. But if Bill Gates walks in, the average net worth rises dramatically. Height follows the bell curve in its distribution. Wealth does not: It follows an asymmetric, L-shaped pattern known as a "power law," where most values are below average and a few far above. In the realm of the power law, rare and extreme events dominate the action.
For Nassim Taleb, irrepressible quant-jock and the author of "Fooled by Randomness" (2001), the contrast between the two distributions is not an amusing statistical exercise but something more profound: It highlights the fundamental difference between life as we imagine it and life as it really is. In "The Black Swan"--a kind of cri de coeur--Mr. Taleb struggles to free us from our misguided allegiance to the bell-curve mindset and awaken us to the dominance of the power law.
The attractiveness of the bell curve resides in its democratic distribution and its mathematical accessibility. Collect enough data and the pattern reveals itself, allowing both robust predictions of future data points (such as the height of the next five people to enter the room) and accurate estimations of the size and frequency of extreme values (anticipating the occasional giant or dwarf.
The power-law distribution, by contrast, would seem to have little to recommend it. Not only does it disproportionately reward the few, but it also turns out to be notoriously difficult to derive with precision. The most important events may occur so rarely that existing data points can never truly assure us that the future won't look very different from the present. We can be fairly certain that we will never meet anyone 14-feet tall, but it is entirely possible that, over time, we will hear of a man twice as rich as Bill Gates or witness a market crash twice as devastating as that of October 1987.
The problem, insists Mr. Taleb, is that most of the time we are in the land of the power law and don't know it. Our strategies for managing risk, for instance--including Modern Portfolio Theory and the Black-Scholes formula for pricing options--are likely to fail at the worst possible time, Mr. Taleb argues, because they are generally (and mistakenly) based on bell-curve assumptions. He gleefully cites the example of Long Term Capital Management (LTCM), an early hedge fund that blew up after its Nobel laureate founders "allowed themselves to take a monstrous amount of risk" because "their models ruled out the possibility of large deviations."

Mr. Taleb is fascinated by the rare but pivotal events that characterize life in the power-law world. He calls them Black Swans, after the philosopher Karl Popper's observation that only a single black swan is required to falsify the theory that "all swans are white" even when there are thousands of white swans in evidence. Provocatively, Mr. Taleb defines Black Swans as events (such as the rise of the Internet or the fall of LTCM) that are not only rare and consequential but also predictable only in retrospect. We never see them coming, but we have no trouble concocting post hoc explanations for why they should have been obvious. Surely, Mr. Taleb taunts, we won't get fooled again. But of course we will.
Writing in a style that owes as much to Stephen Colbert as it does to Michel de Montaigne, Mr. Taleb divides the world into those who "get it" and everyone else, a world partitioned into heroes (Popper, Hayek, Yogi Berra), those on notice (Harold Bloom, necktie wearers, personal-finance advisers) and entities that are dead to him (the bell curve, newspapers, the Nobel Prize in Economics).
A humanist at heart, Mr. Taleb ponders not only the effect of Black Swans but also the reason we have so much trouble acknowledging their existence. And this is where he hits his stride. We eagerly romp with him through the follies of confirmation bias (our tendency to reaffirm our beliefs rather than contradict them), narrative fallacy (our weakness for compelling stories), silent evidence (our failure to account for what we don't see), ludic fallacy (our willingness to oversimplify and take games or models too seriously), and epistemic arrogance (our habit of overestimating our knowledge and underestimating our ignorance).
For anyone who has been compelled to give a long-term vision or read a marketing forecast for the next decade, Mr. Taleb's chapter excoriating "The Scandal of Prediction" will ring painfully true. "What is surprising is not the magnitude of our forecast errors," observes Mr. Taleb, "but our absence of awareness of it." We tend to fail--miserably--at predicting the future, but such failure is little noted nor long remembered. It seems to be of remarkably little professional consequence.
I suspect that part of the explanation for this inconsistency may be found in a study of stock analysts that Mr. Taleb cites. Their predictions, while badly inaccurate, were not random but rather highly correlated with each other. The lesson, evidently, is that it's better to be wrong than alone.
If we accept Mr. Taleb's premise about power-law ascendancy, we are left with a troubling question: How do you function in a world where accurate prediction is rarely possible, where history isn't a reliable guide to the future and where the most important events cannot be anticipated?
Mr. Taleb presents a range of answers--be prepared for various outcomes, he says, and don't rush for buses--but it's clear that he remains slightly vexed by the world he describes so vividly. Then again, beatific serenity may not be the goal here. As Mr. Taleb warns, certitude is likely to be found only in a fool's (bell-curve) paradise, where we choose the comfort of the "precisely wrong" over the challenge of the "broadly correct." Beneath Mr. Taleb's blustery rhetoric lives a surprisingly humble soul who has chosen to follow a demanding and somewhat lonely path.
I wonder how many of us will have the courage to join him. Very few, I predict--unless, of course, something unexpected happens.

Article #2

2008-09-14T19:56:00.000-07:00

Due Friday, September 18th

Go to a NY Times link by clicking here.

Flip through all 9 slides.

Question 1: What is the name of this graph?

Q2: Is it better to be on the right or left of the graph?

Q3: What is graphed on the x-axis?

Q4: What is graphed on the y-axis?

Q5: What does it mean if one value has a very high stack?

Q6: What are the weaknesses of this display?

Q7: What are the strengths?

Q8: Do you think Tufte would be impressed with this display? Why or why not?

Article #1

2008-09-01T19:54:00.000-07:00

Edward Tufte is one of the most influential thinkers in the world of statistical graphs. He is actually a specialist and a formidable thinker about any and all issues that deal with displaying information of all sorts. This "article" will take you to various parts of his web site and ask you to think about some of the ideas he discusses.

Click on this link and then scroll down to see the acceptance letter:
Would you like to get a letter like this? Does it communicate the information it is trying to convey appropriately? Effectively?

Scroll down and read the first page or two about Sparklines:
What is a sparkline? Do you see any advantages to using them?

Scroll down to read about the misleading graphic:
Hopefully you can click on and see the graphic.
And read Tufte's response.
Our book talks about the "area priniciple".
What "graphic" does Tufte recommend the Times had used?
How might you, as a semi-literate statistician be useful to a graphic designer?

Finally, click here:
**Wow, the iPhone is better designed than you think!
**Do you like his museum guide?
**Note: this is a LONG essay. You don't have to read it all. But please learn a few things from the page and/or discussion.

Hope you were impressed by Tufte's thorough thinking! And I hope you read this stuff carefully enough to be able to contribute to a class discussion about these ideas. You don't have to sound like a genius, but you should have absorbed some information so that you can join in!

Article #10

2008-03-03T13:59:00.000-08:00

Questions:
1. Is this a study, survey or experiment? Why?
2. What is the factor and the levels?
3. Describe each treatment group:
4. The third group was not described. Could you guess at what they did? Why would you do this?
5. Describe the confounding variable that Dr. Keisler was worried about. Why is this confounding?
6. Design an experiment that would fix Dr. Keisler's concern.

Study finds dogs, robots cheer elderly

By CHERYL WITTENAUER, Associated Press Writer Sat Mar 1, 4:26 AM ET

ST. LOUIS - Dogs may have a hard time wrapping their paws around this one: Robotic competition is nipping at their heels in the man's-best-friend department. A study by Saint Louis University found that a lovable pooch named Sparky and a robotic dog, AIBO, were about equally effective at relieving the loneliness of nursing home residents and fostering attachments.
The study, which appears in the March issue of the Journal of The American Medical Directors Association, builds on previous findings by the researchers that frequent dog visits decreased loneliness of nursing home residents.

Andrew Ng, who leads Stanford University's team in building a home-assistance robot and was not involved in the study, said the strength of the research is very encouraging.

If humans can feel an emotional bond with robots, even fairly simple ones, some day they could "not just be our assistants, but also our companions," he said.

To test whether residents responded better to Sparky, a trained therapy dog, or the Sony-made robot dog, researchers divided 38 nursing home residents into three groups at a trio of long-term care facilities in St. Louis.

One group had weekly, 30-minute one-on-one visits with Sparky; another group had similar visits with AIBO; a control group did not visit with either dog. Their level of loneliness — determined by residents' answers to several questions — was tested at the beginning and near the end of eight weeks of visits.

Investigator Marian Banks delivered the dogs, but did not interact with the residents. In the end, both groups were less lonely and more attached.

Most of the elderly used Sparky, a 9-year-old, reddish-brown mutt with a white muzzle and floppy ears, as a confidant, telling him "their life story," Marian Banks said.

"He listened attentively, wagged his tail, and allowed them to pet him," said Banks, who adopted and trained Sparky after finding him in an alley behind her home seven years ago.

Those who visited with AIBO took a little longer — about a week — to warm up to the metallic creature. Over time, they grew more comfortable with AIBO, and petted and talked to him. He responded by wagging his tail, vocalizing and blinking his lights.

"AIBO is charismatic if you start to interact with him," said the study's author, Dr. William Banks, a professor of geriatric medicine at Saint Louis University. "He's an engaging sort of guy."
The research could mean that a world is possible where robots could substitute for living dogs and help people, William Banks said.

"They could be personal, not an intrusive crazy inanimate object," he said.

Sara Kiesler, professor of computer science and human-computer interaction at Carnegie Mellon University who was not involved in the study, said the results of the study are encouraging but not completely convincing.

The problem is inferring if it was the robotic dog that reduced the loneliness, and not the human who brought him into the room, she said. She said another study could compare a visit from AIBO with someone stopping by with a stuffed animal or even just a candy bar.

Article #9

2008-02-25T07:53:00.000-08:00

Read and write a short summary.

Word of fair warning: If I stamp you a full 4 points for the article, you are indicating that you read AND UNDERSTOOD the article and are ready to participate in the discussion about it. If you read the article, but are unable to discuss it, please ask for 2 points when I stamp.

Shattering the Bell CurveThe power law rules.
by DAVID A. SHAYWITZ
Tuesday, April 24, 2007

Article:
Life isn't fair. Many of the most coveted spoils--wealth, fame, links on the Web--are concentrated among the few. If such a distribution doesn't sound like the familiar bell-shaped curve, you're right.
Along the hilly slopes of the bell curve, most values--the data points that track whatever is being measured--are clustered around the middle. The average value is also the most common value. The points along the far extremes of the curve contribute very little statistically. If 100 random people gather in a room and the world's tallest man walks in, the average height doesn't change much. But if Bill Gates walks in, the average net worth rises dramatically. Height follows the bell curve in its distribution. Wealth does not: It follows an asymmetric, L-shaped pattern known as a "power law," where most values are below average and a few far above. In the realm of the power law, rare and extreme events dominate the action.
For Nassim Taleb, irrepressible quant-jock and the author of "Fooled by Randomness" (2001), the contrast between the two distributions is not an amusing statistical exercise but something more profound: It highlights the fundamental difference between life as we imagine it and life as it really is. In "The Black Swan"--a kind of cri de coeur--Mr. Taleb struggles to free us from our misguided allegiance to the bell-curve mindset and awaken us to the dominance of the power law.
The attractiveness of the bell curve resides in its democratic distribution and its mathematical accessibility. Collect enough data and the pattern reveals itself, allowing both robust predictions of future data points (such as the height of the next five people to enter the room) and accurate estimations of the size and frequency of extreme values (anticipating the occasional giant or dwarf.
The power-law distribution, by contrast, would seem to have little to recommend it. Not only does it disproportionately reward the few, but it also turns out to be notoriously difficult to derive with precision. The most important events may occur so rarely that existing data points can never truly assure us that the future won't look very different from the present. We can be fairly certain that we will never meet anyone 14-feet tall, but it is entirely possible that, over time, we will hear of a man twice as rich as Bill Gates or witness a market crash twice as devastating as that of October 1987.
The problem, insists Mr. Taleb, is that most of the time we are in the land of the power law and don't know it. Our strategies for managing risk, for instance--including Modern Portfolio Theory and the Black-Scholes formula for pricing options--are likely to fail at the worst possible time, Mr. Taleb argues, because they are generally (and mistakenly) based on bell-curve assumptions. He gleefully cites the example of Long Term Capital Management (LTCM), an early hedge fund that blew up after its Nobel laureate founders "allowed themselves to take a monstrous amount of risk" because "their models ruled out the possibility of large deviations."

Mr. Taleb is fascinated by the rare but pivotal events that characterize life in the power-law world. He calls them Black Swans, after the philosopher Karl Popper's observation that only a single black swan is required to falsify the theory that "all swans are white" even when there are thousands of white swans in evidence. Provocatively, Mr. Taleb defines Black Swans as events (such as the rise of the Internet or the fall of LTCM) that are not only rare and consequential but also predictable only in retrospect. We never see them coming, but we have no trouble concocting post hoc explanations for why they should have been obvious. Surely, Mr. Taleb taunts, we won't get fooled again. But of course we will.
Writing in a style that owes as much to Stephen Colbert as it does to Michel de Montaigne, Mr. Taleb divides the world into those who "get it" and everyone else, a world partitioned into heroes (Popper, Hayek, Yogi Berra), those on notice (Harold Bloom, necktie wearers, personal-finance advisers) and entities that are dead to him (the bell curve, newspapers, the Nobel Prize in Economics).
A humanist at heart, Mr. Taleb ponders not only the effect of Black Swans but also the reason we have so much trouble acknowledging their existence. And this is where he hits his stride. We eagerly romp with him through the follies of confirmation bias (our tendency to reaffirm our beliefs rather than contradict them), narrative fallacy (our weakness for compelling stories), silent evidence (our failure to account for what we don't see), ludic fallacy (our willingness to oversimplify and take games or models too seriously), and epistemic arrogance (our habit of overestimating our knowledge and underestimating our ignorance).
For anyone who has been compelled to give a long-term vision or read a marketing forecast for the next decade, Mr. Taleb's chapter excoriating "The Scandal of Prediction" will ring painfully true. "What is surprising is not the magnitude of our forecast errors," observes Mr. Taleb, "but our absence of awareness of it." We tend to fail--miserably--at predicting the future, but such failure is little noted nor long remembered. It seems to be of remarkably little professional consequence.
I suspect that part of the explanation for this inconsistency may be found in a study of stock analysts that Mr. Taleb cites. Their predictions, while badly inaccurate, were not random but rather highly correlated with each other. The lesson, evidently, is that it's better to be wrong than alone.
If we accept Mr. Taleb's premise about power-law ascendancy, we are left with a troubling question: How do you function in a world where accurate prediction is rarely possible, where history isn't a reliable guide to the future and where the most important events cannot be anticipated?
Mr. Taleb presents a range of answers--be prepared for various outcomes, he says, and don't rush for buses--but it's clear that he remains slightly vexed by the world he describes so vividly. Then again, beatific serenity may not be the goal here. As Mr. Taleb warns, certitude is likely to be found only in a fool's (bell-curve) paradise, where we choose the comfort of the "precisely wrong" over the challenge of the "broadly correct." Beneath Mr. Taleb's blustery rhetoric lives a surprisingly humble soul who has chosen to follow a demanding and somewhat lonely path.
I wonder how many of us will have the courage to join him. Very few, I predict--unless, of course, something unexpected happens.

Article #7

2007-12-05T08:56:00.000-08:00

Due Friday Dec. 7

PLEASE WRITE A SHORT ONE PARAGRAPH SUMMARY:

Gordon S. Black is Chairman and CEO of Harris Black International, Ltd., where George Terhanian is Director of Internet Research.

This article appeared in the Oct. 26, 1998, edition of The Polling Report.

Using the Internet for

Election Forecasting

by Gordon S. Black and George Terhanian

For more than 20 years, election polling in the United States and many other countries has been conducted by telephone. And why not? Telephone polls, on the whole, have proved to be remarkably accurate predictors of voter behavior -- the gold standard of all polling research. Telephone polls are also far less expensive to conduct than door-to-door interviews, the method they have largely replaced.

Although telephone polls possess many virtues, they are not without their imperfections -- ones we do not like to mention too often or too loudly to our clients. For instance, refusal rates today routinely exceed 40% of all households. Unreachable respondents (due to traveling, working, answering machines, the absence of telephones in college dorms, etc.) can regularly run another 30% of the sample. The cost of telephone research can also be prohibitive -- making it virtually impossible for one survey firm to cover every statewide race during an election year, not to mention congressional races.

A 16-State Test Drive

We at Harris Black International believe, however, that the strengths of the Internet as a platform for conducting research more than compensate for the documented limitations of the telephone, as well as the purported limitations of the Internet. This is one major reason why we are now testing a new methodology of forecasting statewide elections via the Internet.

Using an Internet database of more than one million cooperative respondents nationwide, we are inviting individuals from 16 states to complete three polls. The first is under way as this is being written, the week of Oct. 18. The second will be completed on the Sunday before the election. The third will be an exit poll, the day of the election.

This effort is a combined venture with Excite, Inc., one of the most popular Internet portals, and the political science faculty at the University of Rochester. Excite is driving Internet traffic to our web site, where respondents are invited to register to participate in the Harris Poll Online(sm) at www.harrispollonline.com. Excite will also display the results, as they become available.

The political science team from the University of Rochester, led by Dr. Richard Niemi, will receive the entire database after the experiment is complete. They will be able to use it for scientific evaluation and publication.

Accurate Election Forecasts

Conducting research on the Internet is by no means a simple or simple-minded exercise. In the past six months, for example, we have had to overcome a variety of complex problems, including software breakdowns and bugs, security problems, hardware failures, and so forth. We feel a little like Chilton must have felt when they introduced the first computerized telephone interviewing system back in the 1970s.

Our experience has convinced us, however, that properly weighted findings from Internet studies compare favorably to findings from telephone research. Nevertheless, we realize the validity of any polling method hinges on its ability to reliably and accurately forecast voter behavior.

We also realize that many of our colleagues in the polling community are rightfully skeptical of our efforts. To do what we are doing we have had to set aside the staple of our industry -- the simple random sample. Virtually every social scientist in America was educated on the power of random sampling.

Random sampling is a very powerful tool in every avenue of science and industry for increasing the accuracy of estimates while decreasing the cost of the process. It has been canonized by academics, and it is the standard by which we judge research in most scientific fields. To describe something as a "convenience sample" is to assign the research to the bottom of the ladder of scientific quality and value.

We are not challenging the validity of random sampling. In fact, we employ random sampling daily in our telephone polling and believe that it is a wonderful statistical tool for those applications for which it is designed. We are instead investigating whether findings from huge samples of Internet respondents, coupled with sophisticated weighting processes, are as accurate as anything done on the telephone or door-to-door.

We have decided to mount this investigation in public for myriad reasons, including our desire to advance the collective understanding of our colleagues in the polling and scientific communities who, at times, tend to dismiss this methodology without any empirical evidence.

Campaign 2000

We are also preparing for the future. In the year 2000 election, the first presidential election of the new millennium, we will possess a database of millions of cooperative respondents, and we plan to use that database to cover the entire campaign, from the primaries to the general election, beginning in January. Our coverage will include the presidency, state offices, the U.S. Senate, and all congressional districts for which we have adequate data. On election day 2000, we will conduct the largest -- by a factor of 15 -- "exit poll" ever conducted.

In a classic example of the impact of scientific advancement, we expect to improve the performance while dropping the prices. That is the vision that motivates this experiment in polling.

article #6

2007-11-14T18:56:00.000-08:00

What kind of survey is this? read and summarize the methods that were used to collect this data:

Suffering Adds Up in a Hurry In Survey of Tsunami Survivors

By David Brown

Washington Post Staff Writer

Saturday, January 15, 2005; Page A01

What they learned in their bare-bones, random statistical survey, conducted for the International Rescue Committee, was chilling.

Relief organizations often use such systematic assessments during man-made disasters involving war, famine and forced dislocation, in which people's needs may not be obvious.

Brennan's survey of Calang's households began Monday on one side of the tiny peninsula on the west coast of Sumatra. Two Indonesian navy ships were moored offshore.

"Okay," he said, "we start counting after the second house."

This meant that the third house near the beach became Household No. 1, the first to be surveyed. It was a one-room shack built from scavenged wood, corrugated metal sheets and dried palm branches.

"Odon, don't get distracted, please," said Brennan, who hoped to complete the survey that day so he could move onto outlying villages.

"She can't count," the interpreter replied.

It turned out the answer was 13 -- one of the larger households the team would encounter. Brennan still needed the age and sex of each person. And he had a dozen more questions after that.

Brennan said he was acutely conscious of the problem.

"Is that convincing?" Brennan asked with a doubtful smile. "I need to convince you before you can convince them."

Over the course of two days, a solid statistical portrait emerged of the tsunami's effects on Calang.

The man replied, "It's hard to say. We'll have to see how the baby does when she takes more medicine."

Brennan rejoined: "Good answer."