I learned quite a few things at SXSW. Many are interesting but potentially useless, such as how unexpectedly interesting the reviews for Tuscan Whole Milk, 1 Gallon, 128 fl oz are.
However the one that I found most fascinating, and is relevant to a lot of people, was from the panel that I was on. Kevin Hale, the CEO of Wufoo gave an example from their support form. In the process of trying to fill out a ticket you have the option of reporting your emotional state. Which can be anything from "Excited" to "Angry". This seems to be a very odd thing to do.
They did this to see whether they could get some useful tracking data which could be used to more directly address their corporate goal of making users happy. They found they could. But, very interestingly, they had an unexpected benefit. People who were asked their emotional state proceeded to calm down, write less emotional tickets, and then the support calls went more smoothly. Asking about emotional state, which has absolutely no functional impact on the operation of the website, is a social lubricant of immense value in customer support.
Does your website ask about people's emotional state? Should it? In what other ways do we address the technical interaction and forget about the emotions of the humans involved, to the detriment of everyone?
Wednesday, March 17, 2010
Serendipity at SXSW
This year I had the great fortune to be asked to be on a panel at SXSW. It was amazingly fun. However there was only one person I had ever met in person at the conference this year. So I was swimming in a sea of strangers.
But apparently there were a lot of people that I was tangentially connected to in some way.
I was commenting to one of my co-panelists, Victoria Ransom that a previous co-worker of mine looked somewhat similar to her, had a similar accent, and also had a Harvard MBA. Victoria correctly guessed the person I was talking about and had known her for longer than I had.
I was at the Google booth, relating an anecdote about a PDF that I had had trouble reading on a website, when I realized that the person from Australia who had uploaded said PDF was standing right there.
Another person had worked with the identical twin brother of Ian Siegel. Ian has been my boss for most of the last 7 years. (At 2 different companies.)
One of the last people I met was a fellow Google employee whose brother in law was Mark-Jason Dominus. I've known Mark through the Perl community for about a decade.
And these are just the people that I met and talked with long enough to find out how I was connected to them.
Other useful takeaways? Dan Roam is worth reading. Emergen-C before bed helps prevent hangovers. Kick-Ass is hilarious, you should catch it when it comes out next month. And if you're in the USA then Hubble 3D is coming to an IMAX near you this Friday. You want to see it. I'll be taking my kids.
And advice for anyone going to SXSW next year? Talk to everyone. If you're standing in line for a movie, talk to the random stranger behind you. Everyone is there to meet people. Most of the people there are interesting. You never know. Talking to the stranger behind you in line might lead to meeting an astronaut. (Yes, this happened to me.)
But apparently there were a lot of people that I was tangentially connected to in some way.
I was commenting to one of my co-panelists, Victoria Ransom that a previous co-worker of mine looked somewhat similar to her, had a similar accent, and also had a Harvard MBA. Victoria correctly guessed the person I was talking about and had known her for longer than I had.
I was at the Google booth, relating an anecdote about a PDF that I had had trouble reading on a website, when I realized that the person from Australia who had uploaded said PDF was standing right there.
Another person had worked with the identical twin brother of Ian Siegel. Ian has been my boss for most of the last 7 years. (At 2 different companies.)
One of the last people I met was a fellow Google employee whose brother in law was Mark-Jason Dominus. I've known Mark through the Perl community for about a decade.
And these are just the people that I met and talked with long enough to find out how I was connected to them.
Other useful takeaways? Dan Roam is worth reading. Emergen-C before bed helps prevent hangovers. Kick-Ass is hilarious, you should catch it when it comes out next month. And if you're in the USA then Hubble 3D is coming to an IMAX near you this Friday. You want to see it. I'll be taking my kids.
And advice for anyone going to SXSW next year? Talk to everyone. If you're standing in line for a movie, talk to the random stranger behind you. Everyone is there to meet people. Most of the people there are interesting. You never know. Talking to the stranger behind you in line might lead to meeting an astronaut. (Yes, this happened to me.)
Monday, March 8, 2010
Rogue Waves
Today I ran across an interesting essay on our changing understanding of scurvy. As often happens when you learn history better, the simple narratives turn out to be wrong. And you get strange things where as science progressed it discovered a good cure for scurvy, they lost the cure, they proved that their understanding was wrong, then wound up unable to provide any protection from the disease, and only accidentally eventually learned the real cause. The question was asked about how much else science has wrong.
This will be a shorter version of a cautionary tale about science getting things wrong. I thought of it because of a a hilarious comedy routine I saw today. (If you should stop reading here, do yourself a favor and watch that for 2 minutes. I guarantee laughter.) That is based on a major 1991 oil spill. There is no proof, but one possibility for the cause of that accident was a rogue wave. (Rogue waves are also called freak waves.) If so then, comedy notwithstanding, the ship owners could in no way be blamed for the ship falling apart. Because the best science of the day said that such waves were impossible.
Here is some background on that. The details of ocean waves are very complex. However if you look at the ratio between the height of waves and the average height of waves around it you get something very close to a Rayleigh distribution, which is what would be predicted based on a Gaussian random model. And indeed if you were patient enough to sit somewhere in the ocean and record waves for a month, the odds are good that you'd find a nice fit with theory. There was a lot of evidence in support of this theory. It was accepted science.
There were stories of bigger waves. Much bigger waves. There were strange disasters. But science discounted them all until New Years Day, 1995. That is when the Draupner platform recorded a wave that should only happen once in 10,000 years. Then in case there was any doubt that something odd was going on, later that year the RMS Queen Elizabeth II encountered another "impossible" wave.
Remember what I said about a month of data providing a good fit to theory? Well Julian Wolfram carried out the same experiment for 4 years. He found that the model fit observations for all but 24 waves. About once every other month there was a wave that was bigger than theory predicted. A lot bigger. If you got one that was 3x the sea height in a 5 foot sea, that was weird but not a problem. If it happened in a 30 foot sea, you had a monster previously thought to be impossible. One that would hit with many times the force that any ship was built to withstand. A wall of water that could easily sink ships.
Once the possibility was discovered, it was not hard to look through records of shipwrecks and damage to see that it had happened. When this was done it was quickly discovered that huge waves appeared to be much more common in areas where wind and wave travel opposite to an ocean current. This data had been littering insurance records and ship yards for decades. But until scientists saw direct proof that such large waves existed, it was discounted.
Unfortunately there were soon reports such as The Bremen and the Caledonian Star of rogue waves that didn't fit this simple theory. Then satellite observations of the open ocean over 3 weeks found about a dozen deadly giants in the open ocean. There was proof that rogue waves could happen anywhere.
Now the question of how rogue waves can form is an active research topic. Multiple possibilities are known, including things from reflections of wave focusing to the Nonlinear Schrödinger equation. While we know a lot more about them, we know we don't know the whole story. But now we know that we must design ships to handle this.
This leads to the question of how bad a 90 foot rogue wave is. Well it turns out that typical storm waves exert about 6 tonnes of pressure per square meter. Ships were designed to handle 15 tonnes of pressure per square meter without damage, and perhaps twice that with denting, etc. But due to their size and shape, rogue waves can hit with about 100 tonnes of pressure per square meter. Are you surprised that a major oil tanker could see its front fall off?
If you want to see what one looks like, see this video.
This will be a shorter version of a cautionary tale about science getting things wrong. I thought of it because of a a hilarious comedy routine I saw today. (If you should stop reading here, do yourself a favor and watch that for 2 minutes. I guarantee laughter.) That is based on a major 1991 oil spill. There is no proof, but one possibility for the cause of that accident was a rogue wave. (Rogue waves are also called freak waves.) If so then, comedy notwithstanding, the ship owners could in no way be blamed for the ship falling apart. Because the best science of the day said that such waves were impossible.
Here is some background on that. The details of ocean waves are very complex. However if you look at the ratio between the height of waves and the average height of waves around it you get something very close to a Rayleigh distribution, which is what would be predicted based on a Gaussian random model. And indeed if you were patient enough to sit somewhere in the ocean and record waves for a month, the odds are good that you'd find a nice fit with theory. There was a lot of evidence in support of this theory. It was accepted science.
There were stories of bigger waves. Much bigger waves. There were strange disasters. But science discounted them all until New Years Day, 1995. That is when the Draupner platform recorded a wave that should only happen once in 10,000 years. Then in case there was any doubt that something odd was going on, later that year the RMS Queen Elizabeth II encountered another "impossible" wave.
Remember what I said about a month of data providing a good fit to theory? Well Julian Wolfram carried out the same experiment for 4 years. He found that the model fit observations for all but 24 waves. About once every other month there was a wave that was bigger than theory predicted. A lot bigger. If you got one that was 3x the sea height in a 5 foot sea, that was weird but not a problem. If it happened in a 30 foot sea, you had a monster previously thought to be impossible. One that would hit with many times the force that any ship was built to withstand. A wall of water that could easily sink ships.
Once the possibility was discovered, it was not hard to look through records of shipwrecks and damage to see that it had happened. When this was done it was quickly discovered that huge waves appeared to be much more common in areas where wind and wave travel opposite to an ocean current. This data had been littering insurance records and ship yards for decades. But until scientists saw direct proof that such large waves existed, it was discounted.
Unfortunately there were soon reports such as The Bremen and the Caledonian Star of rogue waves that didn't fit this simple theory. Then satellite observations of the open ocean over 3 weeks found about a dozen deadly giants in the open ocean. There was proof that rogue waves could happen anywhere.
Now the question of how rogue waves can form is an active research topic. Multiple possibilities are known, including things from reflections of wave focusing to the Nonlinear Schrödinger equation. While we know a lot more about them, we know we don't know the whole story. But now we know that we must design ships to handle this.
This leads to the question of how bad a 90 foot rogue wave is. Well it turns out that typical storm waves exert about 6 tonnes of pressure per square meter. Ships were designed to handle 15 tonnes of pressure per square meter without damage, and perhaps twice that with denting, etc. But due to their size and shape, rogue waves can hit with about 100 tonnes of pressure per square meter. Are you surprised that a major oil tanker could see its front fall off?
If you want to see what one looks like, see this video.
Monday, March 1, 2010
Fun with Large Numbers
I haven't been blogging much. In part that is because I've been using buzz instead. (Mostly to tell a joke a day.) However I've got a topic of interest to blog about this time. Namely large numbers.
Be warned. If thinking about how big numbers like 9999 really are hurts your head, you may not want to read on.
It isn't hard to find lots of interesting discussion of large numbers. See Who can name the bigger number? for an example. However when math people go for big numbers they tend to go for things like the Busy Beaver problem. However there are a lot of epistemological issues involved with that, for instance there is a school of mathematical philosophy called constructivism which denies that the Busy Beaver problem is well-formulated or that that sequence is well-defined. I may discuss mathematical philosophy at some future point, but that is definitely for another day.
So I will stick to something simpler. Many years ago in sci.math we had a discussion that saw several of us attempt to produce the largest number we could following a few simple ground rules. The rules were that we could use the symbols 0 through 9, variables, functions (using f(x, y) notation), +, *, the logical operators & (and), ^ (or) ! (not), and => (implies). All numbers are non-negative integers. The goal was to use at most 100 non-whitespace characters and finish off with Z = (the biggest number we can put here). (A computer science person might note that line endings express syntactic intent and should be counted. We did not so count.)
A non-mathematician's first approach would likely be to write down Z = 999...9 for a 98 digit number. Of course 9999 is much larger - you would need an 8 digit number just to write out how many digits it has. But unfortunately we have not defined exponentiation. However that is easily fixed:
We now have used up 25 characters and have enough room to pile up a tower of exponents 6 deep.
Of course you can do better than that. Anyone with a CS background will start looking for the Ackermann function.
That's 49 characters. Incidentally there are many variants of the Ackermann function out there. This one is sometimes called the Ackermann–Péter function in the interest of pedantry. But it was actually first written down by Raphael M. Robinson.
(A random note. When mathematicians define rapidly recursing functions they often deliberately pick ones with rules involving +1, -1. This is not done out of some desire to get a lot done with a little. It is done so that they can try to understand the pattern of recursion without being distracted by overly rapid initial growth.)
However the one thing that all variants on the Ackermann function share is an insane growth rate. Don't let the little +1s fool you - what really matters to growth is the pattern of recursion, and this function has that in spades. As it recurses into itself, its growth keeps on speeding up. Here is its growth pattern for small n. (The n+3/-3 meme makes the general form easier to recognize.)
There is no straightforward way to describe A(5, n). Basically it takes the stacked exponent that came up with A(4, n) and iterates that operation n+3 times. Then subtract 3. Which is the starting point for the next term. And so on.
By most people's standards, A(9, 9) would be a large number. We've got about 50 characters left to express something large with this function. :-)
It is worth noting that historically the importance of the Ackermann function was not just to make people's heads hurt, but to demonstrate that there are functions that can be expressed with recursion that grow too quickly to fall into a simpler class of primitive recursive functions. In CS terms you can't express the Ackermann function with just nested loops with variable iteration counts. You need a while loop, recursion, goto, or some other more flexible programming construct to generate it.
There is a reason for some of the odd details of this dream. You'll soon see b and b-1 come into things. But for now notice that the pattern with a0 and a1 is somewhat similar to m and n in the Ackermann function. Details differ, but recursive patterns similar to ones that crop up in the Ackermann function crop up here.
And if a1 is 2, then you get something like a stacked tower of exponentials (going 2,3,2,3,... with some complex junk). And you continue on through various such growth patterns.
Since all numbers are non-negative integers the second rule leads to an unambiguous result. The first and second rules can both apply when the third argument is 0, but that is OK since they lead to the same answer. And so far we've used 40 symbols (remember that => counts as 1 in our special rules).
This leads us to be able to finish off defining our dream function with:
This took another 42 characters.
This leaves us 18 characters left, two of which have to be Z=. So we get
So our next entry is
We're nearly done. The only thing I know to improve is one minor tweak:
Here I changed B into T, and made the 0 case be something that had some growth. This starts us off with the slowest growth being T(b, i) being around b2i, and then everything else gets sped up from there. This is a trivial improvement in overall growth - adding a couple more to the second parameter would be a much bigger win. But if you're looking for largest, every small bit helps. And modulo a minor reformatting and a slight change in the counting, this is where the conversation ended.
Is this the end of our ability to discuss large numbers? Of course not. As impressive as the function that I provided may be, there are other functions that grow faster. For instance consider Goodstein's function. All of the growth patterns in the function that I described are realized there before you get to bb. In a very real sense the growth of that function is as far beyond the one that I described as the one that I described is beyond the Ackermann function.
If anyone is still reading and wants to learn more about attempts by mathematicians to discuss large (but finite) numbers in a useful way, I recommend Large Numbers at MRROB.
Be warned. If thinking about how big numbers like 9999 really are hurts your head, you may not want to read on.
It isn't hard to find lots of interesting discussion of large numbers. See Who can name the bigger number? for an example. However when math people go for big numbers they tend to go for things like the Busy Beaver problem. However there are a lot of epistemological issues involved with that, for instance there is a school of mathematical philosophy called constructivism which denies that the Busy Beaver problem is well-formulated or that that sequence is well-defined. I may discuss mathematical philosophy at some future point, but that is definitely for another day.
So I will stick to something simpler. Many years ago in sci.math we had a discussion that saw several of us attempt to produce the largest number we could following a few simple ground rules. The rules were that we could use the symbols 0 through 9, variables, functions (using f(x, y) notation), +, *, the logical operators & (and), ^ (or) ! (not), and => (implies). All numbers are non-negative integers. The goal was to use at most 100 non-whitespace characters and finish off with Z = (the biggest number we can put here). (A computer science person might note that line endings express syntactic intent and should be counted. We did not so count.)
A non-mathematician's first approach would likely be to write down Z = 999...9 for a 98 digit number. Of course 9999 is much larger - you would need an 8 digit number just to write out how many digits it has. But unfortunately we have not defined exponentiation. However that is easily fixed:
p(n,0) = 1
p(n, m+1) = n * p(n, m)
We now have used up 25 characters and have enough room to pile up a tower of exponents 6 deep.
Of course you can do better than that. Anyone with a CS background will start looking for the Ackermann function.
A(0, n) = n+1
A(m+1, 0) = A(m, 1)
A(m+1, n+1) = A(m, A(m+1, n))
That's 49 characters. Incidentally there are many variants of the Ackermann function out there. This one is sometimes called the Ackermann–Péter function in the interest of pedantry. But it was actually first written down by Raphael M. Robinson.
(A random note. When mathematicians define rapidly recursing functions they often deliberately pick ones with rules involving +1, -1. This is not done out of some desire to get a lot done with a little. It is done so that they can try to understand the pattern of recursion without being distracted by overly rapid initial growth.)
However the one thing that all variants on the Ackermann function share is an insane growth rate. Don't let the little +1s fool you - what really matters to growth is the pattern of recursion, and this function has that in spades. As it recurses into itself, its growth keeps on speeding up. Here is its growth pattern for small n. (The n+3/-3 meme makes the general form easier to recognize.)
A(1, n) = 2 + (n+3) - 3
A(2, n) = 2 * (n+3) - 3
A(3, n) = 2n+3 - 3
A(4, n) = 22…2 - 3 (the tower is n+3 high)
There is no straightforward way to describe A(5, n). Basically it takes the stacked exponent that came up with A(4, n) and iterates that operation n+3 times. Then subtract 3. Which is the starting point for the next term. And so on.
By most people's standards, A(9, 9) would be a large number. We've got about 50 characters left to express something large with this function. :-)
It is worth noting that historically the importance of the Ackermann function was not just to make people's heads hurt, but to demonstrate that there are functions that can be expressed with recursion that grow too quickly to fall into a simpler class of primitive recursive functions. In CS terms you can't express the Ackermann function with just nested loops with variable iteration counts. You need a while loop, recursion, goto, or some other more flexible programming construct to generate it.
Of course with that many characters to work with, we can't be expected to be satisfied with the paltry Ackermann function. No, no, no. We're much more clever than that! But getting to our next entry takes some background.
Let us forget the rules of the contest so far, and try to dream up a function that in some way generalizes the Ackermann function's approach to iteration. Except we'll use more variables to express ever more intense levels of recursion. Let's use an unbounded number of variables. I will call the function D for Dream function because we're just dreaming at this point. Let's give it these properties:
D(b, 0, ...) = b + 1
D(b, a0 + 1, a1, a2, ..., an, 0, ...)
= D(D(b, a0, a1, ..., an, 0, ...), a0, a1, ..., an, 0, ...)
D(b, 0, ..., 0, ai+1, ai+1, ai+2, ..., an, 0, ...)
= D(
D(b, a0, a1, ..., an, 0, ...),
b-1, b-1, ..., b-1,
ai, ai+1, ..., an, 0, ...
)
There is a reason for some of the odd details of this dream. You'll soon see b and b-1 come into things. But for now notice that the pattern with a0 and a1 is somewhat similar to m and n in the Ackermann function. Details differ, but recursive patterns similar to ones that crop up in the Ackermann function crop up here.
D(b, a0, 0, ...) = b+a0+1
D(b, a0, 1, 0, ...) ≈ 32a0 b
And if a1 is 2, then you get something like a stacked tower of exponentials (going 2,3,2,3,... with some complex junk). And you continue on through various such growth patterns.
But then we hit D(b, a0, a1, 1, 0, ...). That is kind of like calling the Ackermann function to decide how many times we will iterate calling the Ackermann function against itself. In the mathematical literature this process is called diagonalization. And it grows much, much faster than the Ackermann function. With each increment of a2 we grow much faster. And each higher variable folds in on itself to speed up even more. The result is that we get a crazy hierarchy of insane growth functions that grow much, much, much faster. Don't bother thinking too hard about how much faster, our brains aren't wired to really appreciate it.
Now we've dreamed up an insanely fast function, but isn't it too bad that we need an unbounded number of variables to write this down? Well actually, if we are clever, we don't. Suppose that b is greater than a0, a1, ..., an. Then we can represent that whole set of variables with a single number, namely m = a0 + a1 b + ... + an bn. Our dream function can be recognized to be the result of calculating D(b, m+1) by subtracting the then replacing the base with D(b, m) (but leaving all of the coefficients alone. So this explains why I introduced b, and all of the details about the -1s in the dream function I wrote.
Now can we encode this using addition, multiplication, non-negative integers, functions and logic? With some minor trickiness we can write the base rewriting operation:
B(b, c, 0) = 0
i < b => B(b, c, i + j*b) = i + B(b, c, j) * c
Since all numbers are non-negative integers the second rule leads to an unambiguous result. The first and second rules can both apply when the third argument is 0, but that is OK since they lead to the same answer. And so far we've used 40 symbols (remember that => counts as 1 in our special rules).
This leads us to be able to finish off defining our dream function with:
D(b, 0) = b + 1
D(b, n+1) = D(D(b, n), B(b, D(b, n), n))
This took another 42 characters.
This leaves us 18 characters left, two of which have to be Z=. So we get
Z = D(2, D(2, D(2, 9)))
So our next entry is
B(b, c, 0) = 0
i < b => B(b, c, i + j*b) = i + B(b, c, j) * c
D(b, 0) = b + 1
D(b, n+1) = D(D(b, n), B(b, D(b, n), n))
Z = D(2, D(2, D(2, 9)))
We're nearly done. The only thing I know to improve is one minor tweak:
B(b, c, 0) = 0
i < b => B(b, c, i + j*b) = i + B(b, c, j) * c
T(b, 0) = b * b
T(b, n+1) = T(T(b, n), B(b, T(b, n), n))
Z = T(2, T(2, T(2, 9)))
Here I changed B into T, and made the 0 case be something that had some growth. This starts us off with the slowest growth being T(b, i) being around b2i, and then everything else gets sped up from there. This is a trivial improvement in overall growth - adding a couple more to the second parameter would be a much bigger win. But if you're looking for largest, every small bit helps. And modulo a minor reformatting and a slight change in the counting, this is where the conversation ended.
Is this the end of our ability to discuss large numbers? Of course not. As impressive as the function that I provided may be, there are other functions that grow faster. For instance consider Goodstein's function. All of the growth patterns in the function that I described are realized there before you get to bb. In a very real sense the growth of that function is as far beyond the one that I described as the one that I described is beyond the Ackermann function.
If anyone is still reading and wants to learn more about attempts by mathematicians to discuss large (but finite) numbers in a useful way, I recommend Large Numbers at MRROB.
Friday, February 5, 2010
Developing on HEAD scales to Google
I have a simple rule of thumb for what I am and am not allowed to say about Google. If I can find it said in some official-looking place from Google, then I think I'm allowed to say it. Otherwise not.
I was therefore very glad to run across this talk by Guido Van Rossum describing his code review tool named Mondrian, put on youtube by Google Tech Talks, and judging from the knowledge level of the audience, delivered to a non-Google audience. Confirming that this was not, in fact, accidentally open, I found an O'Reilly article confirming that it was public.
I therefore feel comfortable talking about anything and everything said in that video. So you can view this as my spin on the material in that video, and if you want the full version without the editorializing, feel free to take an hour and watch the video for the original source.
These days everyone who is competent has agreed that source control is a Good Thing to use. However opinions on how to use it vary widely. One argument in particular that I've seen in multiple organizations is on the value of using multiple branches versus having everyone develop in one branch, which is often called something like MAIN or HEAD. The issues involved are somewhat complex, and not obvious. So let me give a quick overview.
The primary argument for branching is that people can work on different features on different schedules without getting in each other's way. The primary argument for all developing on the same branch is that you discover conflicts immediately when they are easiest to resolve, rather than months later when people have moved on to different things and have lost context for the pieces of code that conflicts. The primary argument against branching is the pain of merging later. (Granted, distributed version control systems like git have done a lot to reduce this pain. But they have not eliminated it.) The primary argument against developing on HEAD is that it requires a constant level of diligence on the part of developers. When any developer can break all developers, you need to be careful in what you check in.
I've just kept this down to the primary arguments because the secondary back and forth arguments get long, involved, and heated. Also what I've described as two ways of working is really two families of working. There are a lot of ways to organizing multiple branches. And there are quite a few useful uses for branches in a software project even if everyone is developing on the same branch. None of this is made easier by the fact that (as with many religious wars) the people involved have imprinted on one way of doing things. That makes them hyper aware of the problems in other approaches, and they don't even notice potential pain points in their own.
Until I came to Google my personal position was that everyone working on HEAD was the best approach as long as your team was small enough that you could make it work. And I vaguely accepted that the pain of branching was a necessary evil on large software projects. Even when the pain reached the point of craziness, I was mostly willing to accept unsupported claims that it is necessary.
Then I came to Google. There are a lot of groups at Google, and they are free to do their own thing. Many do. However most groups develop out of one giant code base on HEAD. And it works as a development model. Google has made it scale.
Guido's talk describes many elements of what makes it scale. The first piece is having good developers. The second piece is an enforced policy of having every single patch go through a code review process before you check anything in. The third piece is a lot of proprietary infrastructure that Google has built up to make things work. And beyond that you have people paying attention to best practices such as consistent style, good unit testing, so on and so forth. (All of which are reinforced in the code review process.)
My opinion after seeing Google has changed. I freely admit that there are real process and tool challenges to making it possible for large teams to develop everything on HEAD and have it scale smoothly. However it is possible. I've seen it work. And, speaking personally, this is my preferred way to work.
Different organizations are different, have different capabilities, different needs, and different goals. Sight unseen I'm not going to tell anyone else that their organization should try to work like Google does. I simply don't have the facts about your situation. But if, like me not that long ago, you've accepted the claim that large development teams have no choice, you now know better.
Now admittedly most people can't go and see this first hand for themselves at Google. But if you want to watch a large successful project developing code to a high standard in this way, I recommend watching clang. And if you want to know more about what kinds of tool support you need to make this work, go listen to Guido. He's smarter than I am, has been doing it longer, and actually built some of the basic tool support for it.
I was therefore very glad to run across this talk by Guido Van Rossum describing his code review tool named Mondrian, put on youtube by Google Tech Talks, and judging from the knowledge level of the audience, delivered to a non-Google audience. Confirming that this was not, in fact, accidentally open, I found an O'Reilly article confirming that it was public.
I therefore feel comfortable talking about anything and everything said in that video. So you can view this as my spin on the material in that video, and if you want the full version without the editorializing, feel free to take an hour and watch the video for the original source.
These days everyone who is competent has agreed that source control is a Good Thing to use. However opinions on how to use it vary widely. One argument in particular that I've seen in multiple organizations is on the value of using multiple branches versus having everyone develop in one branch, which is often called something like MAIN or HEAD. The issues involved are somewhat complex, and not obvious. So let me give a quick overview.
The primary argument for branching is that people can work on different features on different schedules without getting in each other's way. The primary argument for all developing on the same branch is that you discover conflicts immediately when they are easiest to resolve, rather than months later when people have moved on to different things and have lost context for the pieces of code that conflicts. The primary argument against branching is the pain of merging later. (Granted, distributed version control systems like git have done a lot to reduce this pain. But they have not eliminated it.) The primary argument against developing on HEAD is that it requires a constant level of diligence on the part of developers. When any developer can break all developers, you need to be careful in what you check in.
I've just kept this down to the primary arguments because the secondary back and forth arguments get long, involved, and heated. Also what I've described as two ways of working is really two families of working. There are a lot of ways to organizing multiple branches. And there are quite a few useful uses for branches in a software project even if everyone is developing on the same branch. None of this is made easier by the fact that (as with many religious wars) the people involved have imprinted on one way of doing things. That makes them hyper aware of the problems in other approaches, and they don't even notice potential pain points in their own.
Until I came to Google my personal position was that everyone working on HEAD was the best approach as long as your team was small enough that you could make it work. And I vaguely accepted that the pain of branching was a necessary evil on large software projects. Even when the pain reached the point of craziness, I was mostly willing to accept unsupported claims that it is necessary.
Then I came to Google. There are a lot of groups at Google, and they are free to do their own thing. Many do. However most groups develop out of one giant code base on HEAD. And it works as a development model. Google has made it scale.
Guido's talk describes many elements of what makes it scale. The first piece is having good developers. The second piece is an enforced policy of having every single patch go through a code review process before you check anything in. The third piece is a lot of proprietary infrastructure that Google has built up to make things work. And beyond that you have people paying attention to best practices such as consistent style, good unit testing, so on and so forth. (All of which are reinforced in the code review process.)
My opinion after seeing Google has changed. I freely admit that there are real process and tool challenges to making it possible for large teams to develop everything on HEAD and have it scale smoothly. However it is possible. I've seen it work. And, speaking personally, this is my preferred way to work.
Different organizations are different, have different capabilities, different needs, and different goals. Sight unseen I'm not going to tell anyone else that their organization should try to work like Google does. I simply don't have the facts about your situation. But if, like me not that long ago, you've accepted the claim that large development teams have no choice, you now know better.
Now admittedly most people can't go and see this first hand for themselves at Google. But if you want to watch a large successful project developing code to a high standard in this way, I recommend watching clang. And if you want to know more about what kinds of tool support you need to make this work, go listen to Guido. He's smarter than I am, has been doing it longer, and actually built some of the basic tool support for it.
Monday, February 1, 2010
What is intelligence?
Last September I posted on why hard sciences are different from soft ones. The subject of this blog post is a perfect illustration of the critical difference. In the hard sciences people mostly agree on basic definitions, broad agreement exists on key problems to solve, and in general there is agreement on a basic paradigm. Through no fault of their own, in the soft sciences, none of this exists.
Few things illustrate this better than getting a handle on what intelligence is. Here we have a subject that has been seriously studied for over a century. Yet researchers have not achieved agreement on whether they should focus on a single measure of intelligence called g, or multiple measures, or if multiple measures then how many there should be and how they should be divided. Note here that if you talk to individual researchers this fact does not worry them, because each researcher works in a community of other researchers who have achieved agreement on this. Each school of thought therefore churns out a stream of research, but they have failed to convert each other. And after a century the debate shows no sign of ending any time soon.
Please note that I am far from expert on any of this, so I'll try to summarize in broad terms my understanding, but my understanding could be incorrect. Also be aware that a lot of experts disagree with each other strongly, and therefore it is easy to find experts who will disagree with anything interesting that I can say. Please keep those caveats in mind as I try to flounder my way through the complications.
Let's start with the simplest approach, namely Spearman's g. What Spearman did in 1904 is take a number of things that one would think correlate to intelligence, such as grades, and found that they seemed correlated well with each other. He then looked for some linear combination which correlated better with each of the initial factors than any initial factor did. He found a very combination that did so, and found that it successfully predicted a very large part of the variation. In modern mathematical terms he did something called a Principal Component Analysis and looked for the most significant component. He then found that the most significant component really did capture a lot of the variability.
Spearman called this factor g for general intelligence.
Not long after the IQ test was invented. Before long people added IQ tests to the mix of factors, and found that a well-designed IQ test served as a fairly good predictor of g. Then g in turn served as a decent predictor for many things, including academic performance, and success in the workplace. Since IQ tests were fast and easy to administer, IQ tests and ability tests were on. (Ability tests such as the SAT and GRE exams actually are fairly directly comparable to IQ tests, and high-IQ societies such as MENSA are happy to accept them as proof that your IQ is high enough to belong.)
All of this begs many important questions. We got g out of a mathematical construct. But is it real? Is there really such a thing as general intelligence? Also even if there is such a thing as general intelligence, is it a stable attribute of a person? Does your general intelligence change over time? And the whole principal components analysis admits of the discovery of multiple components, how important are the lesser components? (The existence of a split between mathematical and verbal ability is obvious enough that most ability tests split those into separate sections.) Are there important factors which have been missed? Also how much of this is affected by heredity versus environment? How does race play into this? Can training change how you score on the test? Can it change your actual intelligence?
Believe you me, those questions have been researched. Extensively. And argued. More extensively. The only ones that there seems to be a decent consensus on is that to the extent that general intelligence makes sense, it seems to be fairly stable, and both hereditary and environmental factors affect it. Every other one is still under debate, though it is easy to find experts that make definitive claims that the matter has been completely settled one way or the other.
As one example of the ongoing debate, ETS, which puts out various tests including the SAT exams, claims that training cannot improve your performance. At the same time Kaplan makes money selling the claim that they can significantly improve your performance. Kaplan has enough evidence of effectiveness that I accept that they are right, but the debate continues. I suspect not the least because ability tests are just IQ tests under a different name, and so acceptance that Kaplan can teach an ability test is evidence that a key tool underlying a lot of research into psychology is fundamentally flawed, which opens up a can of worms that many don't want to look into.
For another example, I strongly suggest that mathematically inclined readers read g a Statistical Myth. It presents a toy statistical model which demonstrates how readily a large number of independent random factors with a small amount of interaction could give rise to all of the statistical evidence which underlies claims that such a thing as general intelligence exists. Given the fact that any hereditary effect on intelligence is the result of a large number of fairly independent random factors called genes, I think the point is an important one. The idea that we each have a huge number of different intellectual abilities at different degrees of competence accords very well with my understanding of how the brain works, and my experience of interacting with others.
However it is also true that a small number of factors have a major effect on many different kinds of mental activities. Thus some combination of the amount of things you can keep in your mind at the same time and speed of reasoning may be influential enough on enough things to deserve the name "general intelligence".
At this point I'd say that the jury is not just out; they are deadlocked and randomly fighting each other. And have been in this state for the better part of a century.
So what do I have to contribute to the discussion? I'd like to throw out a toy mathematical model for how IQ and intelligence relate. Please do not take this too seriously. After all it is not even agreed on at this point that general intelligence is a particularly well-defined concept. But what I'd like to show is that even if you do accept that general intelligence exists, and that IQ is reasonably well correlated with it, that IQ tests are not a particularly effective way to find the smartest people. (Whether or not any better way exists is a question I'd like to avoid. In fact, please forget this parenthetical point.)
Now suppose you sit down to take a test. You come prepared with many relevant abilities. You have your vocabulary, your polished reasoning ability, and a body of basic knowledge you're expected to have acquired about the world. We are generally prepared to accept these as components of intelligence, and the test will try to measure those. Let's call that INT. However you come prepared with other relevant abilities. There is how well you are feeling, how much sleep you got, any tricks you know about how to handle the types of questions that are likely to be asked, cultural cues you share with the test makers and how stressed you are. We are generally not prepared to accept these as components of intelligence, yet they affect how we do. Let us call these TST.
For my toy model let's say that INT and TST are independent random normals with a mean of 0 and a standard deviation of 10. That's math talk describing a particular bell curve. If you want the odds of it being less than a particular number you just have to divide the number by 10 to get the number of standard deviations you're out, and then look at a standard table and read off the probability.
Now let's say that the measured IQ will be 100+INT+TST. If you're a mathematician you'll know that IQ will have a normal distribution with mean 100 and a standard deviation of 10*sqrt(2) which is about 14.14. The correlation between IQ and INT turns out to be 70.7%. By comparison the Stanford-Binet is designed to have a mean of 100 and a standard deviation of 16, and its correlation with other measures of intelligence like academic performance is (depending on who measures it and how it is done) in the range of 70-80%. So while the model is pretty simple, it is not too far off from how a real IQ test behaves. And it captures the obvious observation that there are factors that affect test performance which we don't think of as intelligence.
Now the fun thing about a model like this is that we can play with it since we know exactly how it is set up. And are free to add any internal details we want. And while we can't necessarily believe the answers we get, we can gain valuable intuition about what kinds of answers we can expect to see if we understood a far more complex system. Such as real people with real brains.
To analyze this I'm going to complicate the model one more time by introducing two more random variables. Let's make them independent normal variables with mean 0 and standard deviation 10/sqrt(2), and call them AVG and DIF. It turns out that (AVG+DIF) and (AVG-DIF) are independent normal variables with mean 0 and standard deviation 10. So let's make (AVG+DIF) be INT, and (AVG-DIF) be TST. High-school algebra shows that AVG = (INT+TST)/2 and DIF = (INT-TST)/2. (Hence the names.)
What was the point of that? Well it is this. At this point IQ = 100+2*AVG. And INT = AVG+DIF. Which means that if you give me an IQ, I can calculate AVG then use the known distribution of DIF to calculate confidence intervals on INT.
For example, in this model how intelligent is a typical member of MENSA? Well MENSA accepts anyone whose IQ is in the top 2% of the population. So a typical member might have an IQ at the boundary of the top 1%. (If you haven't taken statistics these calculations may make you eyes gloss over. You'll have to just trust that I'm coming up with correct numbers.) That IQ is about 2.326 standard deviations out, which after multiplying by the 10*sqrt(2) and adding 100 is an IQ of 132.90. (Remember that our standard deviation is lower than the regular IQ test, so our purported IQ scores will be lower than a regular test.) Which means that AVG is 16.45. On average our INT will be the same as the AVG, which puts you 1.645 standard distributions out, which is right around the 95th percentile.
We can go farther and calculate a 95% confidence interval for what the real intelligence of this individual is. 95% of the time a normally distributed variable will be within 1.960 standard deviations of 0, so that means that DIF will be in the range +- 13.86. Which means that INT will be (2.59, 30.31). Which means that INT is anywhere from 0.0259 to 3.031 standard deviations out, which would put INT anywhere from the 51st percentile to almost the 99.9th percentile. Hmmm, we really haven't nailed down INT very closely, have we? A similar calculation shows that with 80% odds INT is between the 77th and 99.5th percentiles. And you can even do fun things like show that with about 78% odds, this individual is not in the top 2% of the population on INT, and therefore would not qualify for MENSA if they were able to measure INT directly instead of the more complicated IQ.
Let's reverse the question. Let's take someone at the boundary of the 99th percentile in INT, and ask what IQ we should expect on average. Well that INT is 2.326 standard deviations out, which is an INT of 16.44, for an IQ of 116.44, which is 1.1625 standard deviations out, which is at the 87.77th percentile. So sorry, you're probably not going to test as smart enough for MENSA even though you're probably smarter than most of the people in MENSA. How likely is this person to fail to meet MENSA's admission standards? Well the cutoff for MENSA is top 2% which is 2.0537 standard deviations out, which is an IQ of 129.04 (again remember that the toy model's IQs are not as spread out as real IQ tests), so we need TST to be 12.6, which is 1.26 standard deviations out, which gives a probability of 89.6% of not qualifying for MENSA.
The upshot? This model predicts that most members of MENSA are not in the top 2% on intelligence, and most people in the top 2% of intelligence would not qualify for membership in MENSA.
Now let's apply this to a question of more personal interest. According to this model, how intelligent am I likely to be? I bring this up because in my last blog post I used my GRE to discuss my likely IQ, and therefore likely intelligence, then used it as a benchmark to compare with the people I've met at Google. So if I believe that high IQs systemically overestimate INT, how badly is mine likely to be overestimated according to my model?
Well my scores were V: 760, Q: 780 and A: 800. Going to a handy IQ calculator I put in 1540 for my GRE V+Q and find that I am 3.625 standard deviations out and only 1/7143 people have an IQ that high. So on my toy model that is an IQ of 151.265... Which implies that AVG is 25.633. So on average you'd expect my INT to be 25.633, which is 2.5633 standard deviations out, which puts me around the 99.5th percentile, or about 1/200 on INT. (Incidentally the average person with an INT that high would fall in the 96% percentile on IQ, and therefore would not qualify for MENSA.) But let's go farther. 50% of the time a normal distribution falls within 0.67449 standard distributions of the mean, which means that DIF is half the time within the range -4.769 to 4.769. That makes a 50% confidence interval for my INT be the range (20.864, 30.402). Which would put me anywhere from the 98th percentile to the 99.9th percentile in INT. Which is in line with my claim in the previous blog post that I wouldn't be entirely surprised to find that I am at the 99.9th percentile in intelligence. According to this model the odds are that I'm not that smart, but it is possible.
Interestingly there is a symmetry in this model which means that all calculations that I've done for INT apply for TST as well. Which means that I am likely to have a number of characteristics that make me good at taking tests but which are not reflected in my general intelligence. This is actually true.
The vast majority of people when they are faced with a major test or interview get stressed. Now what does stress do to you? Well stress is caused by adrenaline. Adrenaline prepares you for a flight-or-fight reaction. This means decreased blood flow to things like your stomach and neocortex (which is where most of your higher reasoning happens), and increased blood flow to areas that are likely to see action, like your muscles. The result? Improved vision, hearing, faster reflexes, and increased strength. Also significant damage to your ability to carry out complex cognitive tasks. Needless to say this is about the worst possible physiological response to sitting in place for several hours and completing a series of complex cognitive tasks.
Apparently evolution didn't anticipate that millions of us would have our life courses depend on how we do on multiple choice exams. :-)
Very oddly, many members of my family, including me, have the exact opposite response. When I walk into a test the thought process I have is basically, "Well this is it. I've prepared all I'm going to, and now I've got a multiple-guess test which will overstate my abilities. Let's see how I do." And I relax. Comparing me to a normal person based on the resulting test score is like starting with two runners, taking one out back and beating on him for a while, then expecting them to run a fair race. No matter how objectively you measure the result, I have an unfair advantage.
Random note. I originally intended to just present the toy model and use it to provide mathematical support for my impression that a really high IQ score is decent evidence of intelligence, but the true intelligence of most intelligent people isn't actually reflected in their IQ. But to sanity test the model I needed to find the estimated correlation between a standard IQ test and g, the general intelligence factor. And reading up on that I realized how much more complicated the subject was than I realized. I also found myself convinced by g a Statistical Myth that g was probably a statistical artifact with no real meaning. Which is how this turned into 2 long and only tangentially related blog posts mushed into 1. I apologize for the length. I didn't intend it to turn out this way, but I hope it was an interesting read.
Few things illustrate this better than getting a handle on what intelligence is. Here we have a subject that has been seriously studied for over a century. Yet researchers have not achieved agreement on whether they should focus on a single measure of intelligence called g, or multiple measures, or if multiple measures then how many there should be and how they should be divided. Note here that if you talk to individual researchers this fact does not worry them, because each researcher works in a community of other researchers who have achieved agreement on this. Each school of thought therefore churns out a stream of research, but they have failed to convert each other. And after a century the debate shows no sign of ending any time soon.
Please note that I am far from expert on any of this, so I'll try to summarize in broad terms my understanding, but my understanding could be incorrect. Also be aware that a lot of experts disagree with each other strongly, and therefore it is easy to find experts who will disagree with anything interesting that I can say. Please keep those caveats in mind as I try to flounder my way through the complications.
Let's start with the simplest approach, namely Spearman's g. What Spearman did in 1904 is take a number of things that one would think correlate to intelligence, such as grades, and found that they seemed correlated well with each other. He then looked for some linear combination which correlated better with each of the initial factors than any initial factor did. He found a very combination that did so, and found that it successfully predicted a very large part of the variation. In modern mathematical terms he did something called a Principal Component Analysis and looked for the most significant component. He then found that the most significant component really did capture a lot of the variability.
Spearman called this factor g for general intelligence.
Not long after the IQ test was invented. Before long people added IQ tests to the mix of factors, and found that a well-designed IQ test served as a fairly good predictor of g. Then g in turn served as a decent predictor for many things, including academic performance, and success in the workplace. Since IQ tests were fast and easy to administer, IQ tests and ability tests were on. (Ability tests such as the SAT and GRE exams actually are fairly directly comparable to IQ tests, and high-IQ societies such as MENSA are happy to accept them as proof that your IQ is high enough to belong.)
All of this begs many important questions. We got g out of a mathematical construct. But is it real? Is there really such a thing as general intelligence? Also even if there is such a thing as general intelligence, is it a stable attribute of a person? Does your general intelligence change over time? And the whole principal components analysis admits of the discovery of multiple components, how important are the lesser components? (The existence of a split between mathematical and verbal ability is obvious enough that most ability tests split those into separate sections.) Are there important factors which have been missed? Also how much of this is affected by heredity versus environment? How does race play into this? Can training change how you score on the test? Can it change your actual intelligence?
Believe you me, those questions have been researched. Extensively. And argued. More extensively. The only ones that there seems to be a decent consensus on is that to the extent that general intelligence makes sense, it seems to be fairly stable, and both hereditary and environmental factors affect it. Every other one is still under debate, though it is easy to find experts that make definitive claims that the matter has been completely settled one way or the other.
As one example of the ongoing debate, ETS, which puts out various tests including the SAT exams, claims that training cannot improve your performance. At the same time Kaplan makes money selling the claim that they can significantly improve your performance. Kaplan has enough evidence of effectiveness that I accept that they are right, but the debate continues. I suspect not the least because ability tests are just IQ tests under a different name, and so acceptance that Kaplan can teach an ability test is evidence that a key tool underlying a lot of research into psychology is fundamentally flawed, which opens up a can of worms that many don't want to look into.
For another example, I strongly suggest that mathematically inclined readers read g a Statistical Myth. It presents a toy statistical model which demonstrates how readily a large number of independent random factors with a small amount of interaction could give rise to all of the statistical evidence which underlies claims that such a thing as general intelligence exists. Given the fact that any hereditary effect on intelligence is the result of a large number of fairly independent random factors called genes, I think the point is an important one. The idea that we each have a huge number of different intellectual abilities at different degrees of competence accords very well with my understanding of how the brain works, and my experience of interacting with others.
However it is also true that a small number of factors have a major effect on many different kinds of mental activities. Thus some combination of the amount of things you can keep in your mind at the same time and speed of reasoning may be influential enough on enough things to deserve the name "general intelligence".
At this point I'd say that the jury is not just out; they are deadlocked and randomly fighting each other. And have been in this state for the better part of a century.
So what do I have to contribute to the discussion? I'd like to throw out a toy mathematical model for how IQ and intelligence relate. Please do not take this too seriously. After all it is not even agreed on at this point that general intelligence is a particularly well-defined concept. But what I'd like to show is that even if you do accept that general intelligence exists, and that IQ is reasonably well correlated with it, that IQ tests are not a particularly effective way to find the smartest people. (Whether or not any better way exists is a question I'd like to avoid. In fact, please forget this parenthetical point.)
Now suppose you sit down to take a test. You come prepared with many relevant abilities. You have your vocabulary, your polished reasoning ability, and a body of basic knowledge you're expected to have acquired about the world. We are generally prepared to accept these as components of intelligence, and the test will try to measure those. Let's call that INT. However you come prepared with other relevant abilities. There is how well you are feeling, how much sleep you got, any tricks you know about how to handle the types of questions that are likely to be asked, cultural cues you share with the test makers and how stressed you are. We are generally not prepared to accept these as components of intelligence, yet they affect how we do. Let us call these TST.
For my toy model let's say that INT and TST are independent random normals with a mean of 0 and a standard deviation of 10. That's math talk describing a particular bell curve. If you want the odds of it being less than a particular number you just have to divide the number by 10 to get the number of standard deviations you're out, and then look at a standard table and read off the probability.
Now let's say that the measured IQ will be 100+INT+TST. If you're a mathematician you'll know that IQ will have a normal distribution with mean 100 and a standard deviation of 10*sqrt(2) which is about 14.14. The correlation between IQ and INT turns out to be 70.7%. By comparison the Stanford-Binet is designed to have a mean of 100 and a standard deviation of 16, and its correlation with other measures of intelligence like academic performance is (depending on who measures it and how it is done) in the range of 70-80%. So while the model is pretty simple, it is not too far off from how a real IQ test behaves. And it captures the obvious observation that there are factors that affect test performance which we don't think of as intelligence.
Now the fun thing about a model like this is that we can play with it since we know exactly how it is set up. And are free to add any internal details we want. And while we can't necessarily believe the answers we get, we can gain valuable intuition about what kinds of answers we can expect to see if we understood a far more complex system. Such as real people with real brains.
To analyze this I'm going to complicate the model one more time by introducing two more random variables. Let's make them independent normal variables with mean 0 and standard deviation 10/sqrt(2), and call them AVG and DIF. It turns out that (AVG+DIF) and (AVG-DIF) are independent normal variables with mean 0 and standard deviation 10. So let's make (AVG+DIF) be INT, and (AVG-DIF) be TST. High-school algebra shows that AVG = (INT+TST)/2 and DIF = (INT-TST)/2. (Hence the names.)
What was the point of that? Well it is this. At this point IQ = 100+2*AVG. And INT = AVG+DIF. Which means that if you give me an IQ, I can calculate AVG then use the known distribution of DIF to calculate confidence intervals on INT.
For example, in this model how intelligent is a typical member of MENSA? Well MENSA accepts anyone whose IQ is in the top 2% of the population. So a typical member might have an IQ at the boundary of the top 1%. (If you haven't taken statistics these calculations may make you eyes gloss over. You'll have to just trust that I'm coming up with correct numbers.) That IQ is about 2.326 standard deviations out, which after multiplying by the 10*sqrt(2) and adding 100 is an IQ of 132.90. (Remember that our standard deviation is lower than the regular IQ test, so our purported IQ scores will be lower than a regular test.) Which means that AVG is 16.45. On average our INT will be the same as the AVG, which puts you 1.645 standard distributions out, which is right around the 95th percentile.
We can go farther and calculate a 95% confidence interval for what the real intelligence of this individual is. 95% of the time a normally distributed variable will be within 1.960 standard deviations of 0, so that means that DIF will be in the range +- 13.86. Which means that INT will be (2.59, 30.31). Which means that INT is anywhere from 0.0259 to 3.031 standard deviations out, which would put INT anywhere from the 51st percentile to almost the 99.9th percentile. Hmmm, we really haven't nailed down INT very closely, have we? A similar calculation shows that with 80% odds INT is between the 77th and 99.5th percentiles. And you can even do fun things like show that with about 78% odds, this individual is not in the top 2% of the population on INT, and therefore would not qualify for MENSA if they were able to measure INT directly instead of the more complicated IQ.
Let's reverse the question. Let's take someone at the boundary of the 99th percentile in INT, and ask what IQ we should expect on average. Well that INT is 2.326 standard deviations out, which is an INT of 16.44, for an IQ of 116.44, which is 1.1625 standard deviations out, which is at the 87.77th percentile. So sorry, you're probably not going to test as smart enough for MENSA even though you're probably smarter than most of the people in MENSA. How likely is this person to fail to meet MENSA's admission standards? Well the cutoff for MENSA is top 2% which is 2.0537 standard deviations out, which is an IQ of 129.04 (again remember that the toy model's IQs are not as spread out as real IQ tests), so we need TST to be 12.6, which is 1.26 standard deviations out, which gives a probability of 89.6% of not qualifying for MENSA.
The upshot? This model predicts that most members of MENSA are not in the top 2% on intelligence, and most people in the top 2% of intelligence would not qualify for membership in MENSA.
Now let's apply this to a question of more personal interest. According to this model, how intelligent am I likely to be? I bring this up because in my last blog post I used my GRE to discuss my likely IQ, and therefore likely intelligence, then used it as a benchmark to compare with the people I've met at Google. So if I believe that high IQs systemically overestimate INT, how badly is mine likely to be overestimated according to my model?
Well my scores were V: 760, Q: 780 and A: 800. Going to a handy IQ calculator I put in 1540 for my GRE V+Q and find that I am 3.625 standard deviations out and only 1/7143 people have an IQ that high. So on my toy model that is an IQ of 151.265... Which implies that AVG is 25.633. So on average you'd expect my INT to be 25.633, which is 2.5633 standard deviations out, which puts me around the 99.5th percentile, or about 1/200 on INT. (Incidentally the average person with an INT that high would fall in the 96% percentile on IQ, and therefore would not qualify for MENSA.) But let's go farther. 50% of the time a normal distribution falls within 0.67449 standard distributions of the mean, which means that DIF is half the time within the range -4.769 to 4.769. That makes a 50% confidence interval for my INT be the range (20.864, 30.402). Which would put me anywhere from the 98th percentile to the 99.9th percentile in INT. Which is in line with my claim in the previous blog post that I wouldn't be entirely surprised to find that I am at the 99.9th percentile in intelligence. According to this model the odds are that I'm not that smart, but it is possible.
Interestingly there is a symmetry in this model which means that all calculations that I've done for INT apply for TST as well. Which means that I am likely to have a number of characteristics that make me good at taking tests but which are not reflected in my general intelligence. This is actually true.
The vast majority of people when they are faced with a major test or interview get stressed. Now what does stress do to you? Well stress is caused by adrenaline. Adrenaline prepares you for a flight-or-fight reaction. This means decreased blood flow to things like your stomach and neocortex (which is where most of your higher reasoning happens), and increased blood flow to areas that are likely to see action, like your muscles. The result? Improved vision, hearing, faster reflexes, and increased strength. Also significant damage to your ability to carry out complex cognitive tasks. Needless to say this is about the worst possible physiological response to sitting in place for several hours and completing a series of complex cognitive tasks.
Apparently evolution didn't anticipate that millions of us would have our life courses depend on how we do on multiple choice exams. :-)
Very oddly, many members of my family, including me, have the exact opposite response. When I walk into a test the thought process I have is basically, "Well this is it. I've prepared all I'm going to, and now I've got a multiple-guess test which will overstate my abilities. Let's see how I do." And I relax. Comparing me to a normal person based on the resulting test score is like starting with two runners, taking one out back and beating on him for a while, then expecting them to run a fair race. No matter how objectively you measure the result, I have an unfair advantage.
Random note. I originally intended to just present the toy model and use it to provide mathematical support for my impression that a really high IQ score is decent evidence of intelligence, but the true intelligence of most intelligent people isn't actually reflected in their IQ. But to sanity test the model I needed to find the estimated correlation between a standard IQ test and g, the general intelligence factor. And reading up on that I realized how much more complicated the subject was than I realized. I also found myself convinced by g a Statistical Myth that g was probably a statistical artifact with no real meaning. Which is how this turned into 2 long and only tangentially related blog posts mushed into 1. I apologize for the length. I didn't intend it to turn out this way, but I hope it was an interesting read.
Sunday, January 31, 2010
Things I've learned at Google so far
Well I've been an employee at Google for about a month. So this seems like as good a place to reflect for a moment.
The first thing that I've learned is that internally Google is incredibly open, but externally there is a lot we can't say. I understand and support a lot of the reasons why it is so, but it can be frustrating. There is a lot of really cool technology at Google that people never hear about. The statistics of what Google deals with are astounding. The technology we use to deal with it is amazing. The way we scale is unbelievable. (I really wish I could go back and have a few discussions on software development methodology raising points about what has proven to scale at Google...) One random fact that I know I can say is that computations happen in our data centers with about half the power drawn for what is industry standard. I'm not allowed to say how we do it, but it is a rather amazing testimony to what smart people can accomplish when we put our minds to it.
Moving on, what about Google's culture? I would describe Google's culture as "creative chaos". There was some confusion about where I was supposed to be when I started. This resulted in the following phone call, "Hello?", "Hello Ben, this is Conner (that's my new manager), where are you?" "Mountain View." "Why are you there?" "Because this is where the recruiter said to go." "Good answer! Nice of them to tell me. Enjoy your week!" This caused me to ask an experienced Googler, "Is it always this chaotic?" The response I got was, "Yes! Isn't it wonderful?" That response sums up a lot about Google's culture. If you're unable to enjoy that kind of environment, then Google isn't the place for you.
Seriously, the corporate culture is based on hiring really smart people, giving them responsibilities, letting them know what problems the company thinks it should focus on, then letting them figure out how to tackle it. What management hierarchy there is is very flat. And people pay little attention to it unless there is a problem. You are expected to be a self-directed person, who solves problems by reaching out to whomever you need to and talking directly. Usually by email. The result is an organization which is in a constant state of flux as things are changing around you, usually for the better. With a permanent level of chaos and very large volumes of email. It is as if an entire company intuitively understood that defect rates are tied to distance on the corporate org-chart, and tried to solve it by eliminating all barriers to people communicating directly with whoever they need to communicate with. (Incidentally the point about defect rates and org charts is actually true, see Facts and Fallacies of Software Engineering for a citation.)
Speaking of email, working at Google you learn really fast how gmail is meant to be used. If you want to deal with a lot of email in gmail, here is what you need to do. Go into settings and turn keyboard shortcuts on. The ones you'll use a lot are j/k to move through email threads, n to skip to the next message, and the space bar to page through text. And m to hide any active thread that you're not interested in (direct emails to you will still show up). There are other shortcuts, but this is enough to let you skim through a lot of email fairly quickly without touching the mouse too much. Next go into labels and choose to show all labels. Your labels are basically what you'd call folders in another email client. (Unfortunately they are not hierarchical, but they do work.) Next as you get email, you need to be aggressive about deciding what you need to see, versus what is context specific. Anything that is context specific you should add a filter for, that adds a label, and skips the inbox. Nothing is lost, you can get to the emails through the list of labels on the left-hand side of your screen in gmail. But now various kinds of automated emails, lower priority mailing lists, and so on won't distract you from your main email until you go looking for them.
When you combine all of these options with gmail's auto-threading features, it is amazing how much more efficiently you can handle email. In fact this is exactly the problem that gmail was invented to handle. Because this was the problem that Paul Buchheit was trying to solve for himself when he started gmail. It is worth pointing out that Paul Buchheit was a software engineer at Google. He didn't need permission to write something like gmail. Corporate culture says that if you need something like that, you just go ahead and do it. In fact this is enshrined as an official corporate policy - engineers get 20% of their time to do with pretty much as they please, and are judged in part on how they use that time. I found a speech claiming that over half of Google's applications started as a 20% project. (I'm surprised that the figure is so low.) To get a sense of how much stuff people just do, visit Google Labs. No corporate decision. No central planning. People just do things like start putting up solar panels in the parking lot, and the next thing you know Google has one of the largest solar panel installations in the world and has decided to go carbon neutral. And the attitude that this is how you should operate is enshrined as official corporate policy!
You've got to love corporate policies like that. Speaking of nice corporate policies, Google has quite a few surprising ones. For instance they have benefits like heavily subsidized massage on site (I've still got to take my free hour massage for joining), free gym membership, and the like. Or take their attitude on dogs. Policy says that if your immediate co-workers don't object, you can bring your dog to work. Cats are different, however. Nothing against cats, but Google is a dog place and cats wouldn't be comfortable. (Yes, there are lots of dogs around the offices, and I've even seen people randomly wander over to find out if they can borrow someone else's dog for a while.) Hmmm. Sick day policy. Don't show up when you're sick and tell people why you're not showing up. Note what's missing. There is no limit to how much sick time you get if you need it. Oh, and food. Official Google policy is that at all times there shall be good, free food within 150 feet of every Googler. OK, admittedly the food quality does vary. That in Mountain View is better than anywhere else (the larger clientele base lets them have a much more varied selection). But you quickly learn why it is common for new Googlers to put on 15-20 pounds in their first year. (I'm trying to avoid that. We'll see if I succeed...)
But, you say, isn't this crazy? Doesn't it cost a fortune? The answer is that of course it does. But it provides value. People bond over food. Even if you're not bonding, having food close by makes short meals easier. And the temptation to continue working until dinnertime is very real. (Particularly if, as with me, you'd like to wait until rush hour is over before going home.) Obviously no normal CFO would crunch numbers and see things that way. But Google stands behind that decision, and the people who work there treasure the company for it.
Speaking of the people who work there, Google has amazing people. It is often said that engineers find working at Google a humbling experience. This is absolutely true. It took me less than a day to realize that the guy sitting next to me is clearly much smarter than I am, and he's nowhere near the top of the range of talent at Google. In fact, as best as I can tell, I'm pretty much average, though I'm trying hard to hold out a ray of hope that I'm slightly better than average.
Let me put that in context. The closest thing that I have to an estimate for my IQ is scoring 2340 on the GRE exam in 1991. Based on conversions that I've seen, that puts me at about the top 0.01% in IQ. Now I was really "on" that day, happen to believe that there are problems with the measurement of intelligence by an IQ test (a subject which I may devote a future blog post to), but without false modesty I wouldn't be surprised to find that I'm as high as being in the top 0.1% in general intelligence (however that could be measured). Which in most organizations means that I get thought of as being very smart.
However software development is a profession that selects for intelligence. By and large only good software developers bother applying to Google. And Google rejects the vast majority of their applicants. Granted the filtering process is far from perfect, but by the time you get through that many filters, someone like me is just average.
This leads to another point of interest. How astoundingly complex the company is. I believe that organizations naturally evolve until they are as complex as the people in them can handle. Well Google is tackling really big, complex problems, and is full of people who can handle a lot of complexity. The result? I've been told that I should expect that after 2 months I'll only be marginally useful. My initial learning curve should start to smooth out after about 6 months. And every year I should expect half of what I've learned to become obsolete. (Remember what I said about Google having a certain level of permanent chaos? If you're like me, it is exhilarating. But sometimes the line between exhilarating and terrifying can be hard to find...)
Oh, and what else did I learn? That we're hiring more people this year. :-)
The first thing that I've learned is that internally Google is incredibly open, but externally there is a lot we can't say. I understand and support a lot of the reasons why it is so, but it can be frustrating. There is a lot of really cool technology at Google that people never hear about. The statistics of what Google deals with are astounding. The technology we use to deal with it is amazing. The way we scale is unbelievable. (I really wish I could go back and have a few discussions on software development methodology raising points about what has proven to scale at Google...) One random fact that I know I can say is that computations happen in our data centers with about half the power drawn for what is industry standard. I'm not allowed to say how we do it, but it is a rather amazing testimony to what smart people can accomplish when we put our minds to it.
Moving on, what about Google's culture? I would describe Google's culture as "creative chaos". There was some confusion about where I was supposed to be when I started. This resulted in the following phone call, "Hello?", "Hello Ben, this is Conner (that's my new manager), where are you?" "Mountain View." "Why are you there?" "Because this is where the recruiter said to go." "Good answer! Nice of them to tell me. Enjoy your week!" This caused me to ask an experienced Googler, "Is it always this chaotic?" The response I got was, "Yes! Isn't it wonderful?" That response sums up a lot about Google's culture. If you're unable to enjoy that kind of environment, then Google isn't the place for you.
Seriously, the corporate culture is based on hiring really smart people, giving them responsibilities, letting them know what problems the company thinks it should focus on, then letting them figure out how to tackle it. What management hierarchy there is is very flat. And people pay little attention to it unless there is a problem. You are expected to be a self-directed person, who solves problems by reaching out to whomever you need to and talking directly. Usually by email. The result is an organization which is in a constant state of flux as things are changing around you, usually for the better. With a permanent level of chaos and very large volumes of email. It is as if an entire company intuitively understood that defect rates are tied to distance on the corporate org-chart, and tried to solve it by eliminating all barriers to people communicating directly with whoever they need to communicate with. (Incidentally the point about defect rates and org charts is actually true, see Facts and Fallacies of Software Engineering for a citation.)
Speaking of email, working at Google you learn really fast how gmail is meant to be used. If you want to deal with a lot of email in gmail, here is what you need to do. Go into settings and turn keyboard shortcuts on. The ones you'll use a lot are j/k to move through email threads, n to skip to the next message, and the space bar to page through text. And m to hide any active thread that you're not interested in (direct emails to you will still show up). There are other shortcuts, but this is enough to let you skim through a lot of email fairly quickly without touching the mouse too much. Next go into labels and choose to show all labels. Your labels are basically what you'd call folders in another email client. (Unfortunately they are not hierarchical, but they do work.) Next as you get email, you need to be aggressive about deciding what you need to see, versus what is context specific. Anything that is context specific you should add a filter for, that adds a label, and skips the inbox. Nothing is lost, you can get to the emails through the list of labels on the left-hand side of your screen in gmail. But now various kinds of automated emails, lower priority mailing lists, and so on won't distract you from your main email until you go looking for them.
When you combine all of these options with gmail's auto-threading features, it is amazing how much more efficiently you can handle email. In fact this is exactly the problem that gmail was invented to handle. Because this was the problem that Paul Buchheit was trying to solve for himself when he started gmail. It is worth pointing out that Paul Buchheit was a software engineer at Google. He didn't need permission to write something like gmail. Corporate culture says that if you need something like that, you just go ahead and do it. In fact this is enshrined as an official corporate policy - engineers get 20% of their time to do with pretty much as they please, and are judged in part on how they use that time. I found a speech claiming that over half of Google's applications started as a 20% project. (I'm surprised that the figure is so low.) To get a sense of how much stuff people just do, visit Google Labs. No corporate decision. No central planning. People just do things like start putting up solar panels in the parking lot, and the next thing you know Google has one of the largest solar panel installations in the world and has decided to go carbon neutral. And the attitude that this is how you should operate is enshrined as official corporate policy!
You've got to love corporate policies like that. Speaking of nice corporate policies, Google has quite a few surprising ones. For instance they have benefits like heavily subsidized massage on site (I've still got to take my free hour massage for joining), free gym membership, and the like. Or take their attitude on dogs. Policy says that if your immediate co-workers don't object, you can bring your dog to work. Cats are different, however. Nothing against cats, but Google is a dog place and cats wouldn't be comfortable. (Yes, there are lots of dogs around the offices, and I've even seen people randomly wander over to find out if they can borrow someone else's dog for a while.) Hmmm. Sick day policy. Don't show up when you're sick and tell people why you're not showing up. Note what's missing. There is no limit to how much sick time you get if you need it. Oh, and food. Official Google policy is that at all times there shall be good, free food within 150 feet of every Googler. OK, admittedly the food quality does vary. That in Mountain View is better than anywhere else (the larger clientele base lets them have a much more varied selection). But you quickly learn why it is common for new Googlers to put on 15-20 pounds in their first year. (I'm trying to avoid that. We'll see if I succeed...)
But, you say, isn't this crazy? Doesn't it cost a fortune? The answer is that of course it does. But it provides value. People bond over food. Even if you're not bonding, having food close by makes short meals easier. And the temptation to continue working until dinnertime is very real. (Particularly if, as with me, you'd like to wait until rush hour is over before going home.) Obviously no normal CFO would crunch numbers and see things that way. But Google stands behind that decision, and the people who work there treasure the company for it.
Speaking of the people who work there, Google has amazing people. It is often said that engineers find working at Google a humbling experience. This is absolutely true. It took me less than a day to realize that the guy sitting next to me is clearly much smarter than I am, and he's nowhere near the top of the range of talent at Google. In fact, as best as I can tell, I'm pretty much average, though I'm trying hard to hold out a ray of hope that I'm slightly better than average.
Let me put that in context. The closest thing that I have to an estimate for my IQ is scoring 2340 on the GRE exam in 1991. Based on conversions that I've seen, that puts me at about the top 0.01% in IQ. Now I was really "on" that day, happen to believe that there are problems with the measurement of intelligence by an IQ test (a subject which I may devote a future blog post to), but without false modesty I wouldn't be surprised to find that I'm as high as being in the top 0.1% in general intelligence (however that could be measured). Which in most organizations means that I get thought of as being very smart.
However software development is a profession that selects for intelligence. By and large only good software developers bother applying to Google. And Google rejects the vast majority of their applicants. Granted the filtering process is far from perfect, but by the time you get through that many filters, someone like me is just average.
This leads to another point of interest. How astoundingly complex the company is. I believe that organizations naturally evolve until they are as complex as the people in them can handle. Well Google is tackling really big, complex problems, and is full of people who can handle a lot of complexity. The result? I've been told that I should expect that after 2 months I'll only be marginally useful. My initial learning curve should start to smooth out after about 6 months. And every year I should expect half of what I've learned to become obsolete. (Remember what I said about Google having a certain level of permanent chaos? If you're like me, it is exhilarating. But sometimes the line between exhilarating and terrifying can be hard to find...)
Oh, and what else did I learn? That we're hiring more people this year. :-)
Subscribe to:
Posts (Atom)