Conditional Probability

In my last entry on probability theory, I promised to have a more detailed look at conditional probability. We need this for solving the disease test problem and for solving the Monty Hall problem mathematically.

P(A | B) reads "the probability of A given B", and this is referred to as conditional probability. The formula for calculating this probability is

P(A | B) = P(A AND B) / P(B)

Why?

We're trying to figure out the probability that the event A is also true, given that B is true. Sometimes A may be true even though B is not, but we're not interested in those instances.

Independent events

Now, sometimes the probability of A is independent of B. For example, if we flip two coins [A, B], and each event A and B is true if the corresponding coin comes up heads. If B is true (that is, comes up heads) the probability of A is still 50% (1). This means that

P(A | B) = P(A)

where P(A) is the a-priori probability and the conditional probability is unchanged due to the information that we gained from flipping the coin B.

Dependent events

But what if

P(A | B) != P(A)

("!=" reads "does not equal") In this case, A is dependent on B. What this means is that, as we gain information about B, the probability of A changes from the a-priori probability. In this case, we need to consider all the cases when B is true:

P(A AND B) + P(NOT A AND B) = P(B)

Those are all of the instances where B is true. So we know that B is true. Out of all the instances where B is true [P(B)], some of them are instances where A is also true [P(A AND B)]:

P(A | B) = P(A AND B) / P(B)

An example

Let's say we have a drawn three cards from a deck: An ace, a king and a queen [A, K, Q]. We shuffle those three cards and draw two of them, trying to draw an ace. Let's say that the first card is not an A. What's the probability that the second one is?

First, let's define two events:

Card1 is the event that the first card is an A
Card2 is the event that the second card is an A

P(Card1) = 1/3
P(NOT Card1) = 2/3
P(Card2 AND NOT Card1) = 1/3

The last probability is easy to see from an a-priori standpoint - The probability that any one of the drawn cards will be an A is 1/3. So the probability of that one card to be an A and the other one not to be an A is also 1/3.

P(Card2 | NOT Card1) = P(Card2 AND NOT Card1) / P(NOT Card1)
P(Card2 | NOT Card1) = (1/3) / (2/3) = 1/2

More intuitively, this can be illustrated as follows:

		Card 2
		A	K	Q
Card 1	A	0	1/6	1/6
	K	1/6	0	1/6
	Q	1/6	1/6	0

As we can see in this matrix, there are six possible combinations of two cards. [AA, KK, QQ] are not possible, since there's only one card of each rank. Each possible combination has a 1/6 probability of occuring. [AK, AQ] are the possible combinations where the first card is an A. In the matrix, we can find the probabilities stated earlier:

P(Card1) = 2 * 1/6 = 1/3
P(NOT Card1) = 4 * 1/6 = 2/3
P(Card2 AND NOT Card1) = 2 * 1/6 = 1/3

Asking what P(Card2 | NOT Card1) is, is the same as asking "how big a fraction of the times that we don't pick an A as our first card do we pick an ace as our second card?". We can easily see in the matrix that there are 4 cases (marked as green) where we don't pick an A as our first card. In two of those cases our second card is an A. 2/4 = 1/2. But also, 2*(1/6) / 4*(1/6) = 1/2.

2*(1/6) / 4*(1/6) = P(Card2 AND NOT Card1) / P(NOT Card1)

Stated in words, in 2 out of 4 cases when the first card is not an A (out of a total of 6 possible cases, which includes draws where the first card is an A), the second card is an A. So, knowing that the first card is not an A, we can narrow the situation down to those 4 cases, giving us a probability of 2/4 = 1/2.

I hope this entry has helped your understanding of how conditional probability works. It's not very formal, and it's not very extensive, but hopefully it's quite intuitive and at least free of too big gaps in it's logic. However, it's late now, and I kind of just threw this one out there, because I haven't posted anything for a while.

__________
Notes:
(1) If you think otherwise, you're subjected to the gamblers fallacy, which we'll have a closer look at in a future post.

Quiz: The Probability of Having a Disease

A new super-resistent virus has emerged in Farawayistan. The Ministry of Health estimate 1 out of 1000 to be infected. After immense efforts, a very accurate test has been developed. If the test subject is infected, 99.9% of the time the test will come out positive. But if the subject is not infected, 1 in 1000 times the test will yield a false positive result.

A random person is tested, and the test comes out positive. What's the probability that he's infected?

Please, answer in the comments.

Probability Theory for Dummies

Before we start examining the Monty Hall problem in any kind of detail, let's look at some probability notation and theory. This entry is intended for those readers who aren't already altogether familiar with basic probability theory. Those who are may read this as a review, or skip reading today's post.

I believe that about 50% of my readers are going to read this specific entry. The others may not be too interested in this particular topic, they may be deterred by the mathematical notation (don't let that deter you, though), or they may already be very familiar with the topics in this entry. However, taking a random reader, the probability that he'll read this entry, given that my assumption is correct, is 50%. Put another way:

P(Read) = 0.5

"P" is the notation for probability. "Read" is the event that he reads the entry, and 0.5 is the decimal form of the probability of that event. In English, P(Read) = 0.5 reads "the probability of the event that he reads this entry equals 50%".

Conditional probability (2)

However, I feel that at least 75% of my math-inclined readers probably are familiar with those topics and, thus, will skip reading this entry. So,

P(NOT Read | Math-inclined) = 0.75

"P" and "Read" are the same as above."NOT" is an operator stating that the "Read" event is not true. "|" is the the operator of conditional probability, and it reads "given". "Math-inclined" is the event that the chosen reader is one of those math-inclined readers. So this whole statement reads "The probability of the event that he does not read this entry, given the event that he's math-inclined, equals 75%".

The NOT operator

The chances that an event takes place, plus the chance that it does NOT, always adds up to 100%. Put differently, in 100% of cases, either the event takes place or it doesn't. So,

P(A) + P(NOT A) = 1.

With some very basic algebra, this can be turned into

P(NOT A) = 1 - P(A)

The probability that the math-inclined reader will read this entry equals 1 minus the probability that he will NOT read it, so

P(Read | Math-inclined) = 1 - P(NOT Read | Math-inclined) = 1 - 0.75 = 0.25

Joint probability

I believe that about 75% of my readers are math-inclined, so

P(Math-inclined AND Read) = 0.75 * 0.25 = 0.1875

There's an 18.75% chance that a randomly chosen reader is math-inclined but still reads this entry. "AND" is the operator of joint probability. This means that, for the statement "Math inclined AND Read" to be true, both events have to be true. The reason why we multiply the probabilities of both events can be illustrated as follows:

P(Math-inclined) = 0.75 = 3/4
P(Read | Math-inclined) = 0.25 = 1/4

as stated above. Out of 16 randomly picked readers, on average 12 readers will be math-inclined (3/4 * 16). Out of those 12 math-inclined readers, 3 will read this entry (1/4 * 12). So out of 16 initially chosen readers, 3 are math-inclined AND will read this entry. 3 out of 16 are 18.75% (3/16 = 0.1875). This is consistent with multiplying the probabilities of both required events: 3/4 * 1/4 = 3/16.

Put differently, in 75% of cases the randomly chosen reader is math-inclined. In 25% of those cases, he'll still read this entry. As we all know, a certain percentage of a quantity equals that quantity multiplied by the decimal form of the percentage. So, 25% of 75% is 0.25 * 75%, which equals 18.75% (0.1875).

The OR operator and the Sieve principle

P(Math-inclined OR NOT Math-inclined) = 0.75 + 0.25 = 1

Being put this way, this one seems pretty obvious. The probability that a reader is either math-inclined or not is obviously 1. But it may not seem equally obvious if we put it in more abstract terms. Formally,

P(A OR B) = P(A) + P(B) - P(A AND B)

Why is this? Well, let's resort to an intuitive explanation. Let's say we have six two-letter combinations, for example [AB, AC, AD, BC, BD, CD]. If we choose one of those combinations at random,

P(A) = 3/6 (the probability of the event that the letter "A" is represented in the combination)
P(B) = 3/6 (the probability of the event that the letter "B" is represented in the combination)

What's the probability that either A or B are represented in the combination? Well, from a quick look, we can see that it's 5/6, since only 1 in 6 combinations include neither A or B. But how do we arrive at this mathematically?

Adding the number of combinations where A is represented to the number of combinations where B is represented seems like a good first step. But this leaves us with 6/6, which is clearly wrong. This is because, in this way, we've double-counted the instance where both A and B are represented. So, having counted that instance twice, we need to subtract it once:

P(A OR B) = P(A) + P(B) - P(A AND B) = 3/6 + 3/6 - 1/6 = 5/6

This is called the Sieve principle, or the inclusion-exclusion principle. With more events, such as P(A OR B OR C) it gets a little bit trickier, but let's not bother with that right now.

Returning to

P(Math-inclined OR NOT Math-inclined) = 0.75 + 0.25 = 1

didn't I forget to subtract the double-counted instances? No. Double counting isn't possible in this case, since one reader can't be both math-inclined and non-math-inclined. So

P(Math-inclined AND NOT Math-inclined) = 0

and so, actually,

P(Math-inclined OR NOT Math-inclined) = 0.75 + 0.25 - 0 = 1

but we can just skip that step.

That's a very brief look at probability theory. We'll expand on it later on, and we'll use it when examining the Monty Hall problem further. But first, we'll have a more intuitive look at the Monty Hall problem tomorrow.

Next entry on this topic: Conditional Probability

__________
Notes:
(1) This, and all of the other figures concerning the inclinations and tendencies of my readers, are completely made up. They are probably not even nearly accurate, and I use them solely for illustration purposes.
(2) In a future entry, we'll have a more detailed look at conditional probability

The Monty Hall Problem, Continued

I suppose one comment for my very first blog entry is to be considered good. To recap, I gave a new wording to the famous Monty Hall problem. Three cups were placed upside down on a bar desk, with a $100 bill under one of them. You were told to chose one of the cups, and if that particular cup contained the $100 bill, the bill would be yours. I lifted one of the other cups after watching you choose one, revealing that there was no bill under that one. You were offered an opportunity to change your mind and go for the other unrevealed cup instead of the one you initially chose. The question is: Should you switch or should you stay with your initial choice?

Anonymous commented as follows:

You should switch. The probability that you picked the right cup first is 1/3, so its 2/3 that the bill is in one of the other cups. Knowing that the cup is not in one of them doesnt change that. If you switch the probability is 2/3.

This may seem counter-intuitive to many readers. But from a probabilistic point of view, it seems to make sense. This is the answer that I expected from most readers who are already familiar with the Monty Hall problem, since this is the correct answer to the problem in its basic formulation. However, remember that I hinted that there is a twist to this one? There is, and this answer is, in fact, wrong. You should stay with your initial choice.

Now, I expect to be ridiculed by some of my readers. Please, shoot. I'm very confident that my solution is correct, and I will explain why in a later entry. In the meantime, can anyone come up with an explanaition? Why should you stay? Or, if you don't agree with me, why am I wrong?

Who is the Economatheek?

I suppose you've had some trouble interpreting my nom de plume, The Economatheek. It's simple, though. It's a makeup word composed by the three parts economics, mathematics and geek. It says, if not all, at least some of it.

Being a math-inclined law-and-business student among the polar bears and huskies of Sweden, I spend the long winters studying a wide range of sources, only connected by the fact that they're not actually on the curriculus. I've developed an interest in the shortcomings of mainstream economic theory and the fallacies that affect much of our thinking. I find the inherent uncertainty that impregnate every aspect of social and economic life, but which is widely assumed away in mainstream economic theory, quite intriguing and I feel that this is where focus needs to be set.

Besides from this, I also harbour a great interest in game theory, probability theory, statistics, and many more topics. There will be plenty of all of those topics here as time goes by.

In order not to scare our newcoming visitors off, let's start out slow:

The Monty Hall problem, popularized by columist and author Marilyn vos Savant, proves an intricate problem to most on their first acquaintance with it. Expecting at least some of my visitors to have already read, solved and understood the problem in it's basic formulation, I shall say that there is a twist to this one. So stay with me.

It's friday night, and you're standing by the bar in the local pub. From the shadows, a skinny, bearded man emerges, introducing himself as The Economatheek. Holding up a $100 bill, he asks the bartender for three coffee cups, and after receiving the cups, he asks you to look away. The bartender, who you trust as an honest man, serves as a witness as The Economatheek puts the three cups upside down with the $100 bill under one of them.

You are now told to chose one of the cups. If the bill is, indeed, to be found under that particular cup, it will be yours. You chose one of the cups, but just as you're about to lift it, the skinny man tells you to wait. He then lifts one of the other cups, to reveal that there is no bill under that particular cup. Thereafter, he asks you if you still want to go for the cup that you initially chose, or if you want to switch to the other one that has yet not been lifted.

So, what is your choice, and why? Please, share your thoughts in the comments.

Also, please excuse my English. Being brought up by polar bears, English was not the first language I learned. I will not take offence from remarks on my language, rather the opposite. After all, life as a geek is all about learning.

The next entry on this topic: The Monty Hall problem, continued
__________
External links in this post:
http://www.marilynvossavant.com/
Official website of columnist and author Marilyn vos Savant.