Random Observations

Financial stress on the system

2011-08-08T13:22:00.001-07:00

I posted this last week on Google+ at https://plus.google.com/u/0/114613808538621741268/posts/13tBzre1nRa. Given market drops since then, I'm reposting it in a more public forum.

How could our next financial crisis start? I was in an interesting discussion about this last night. I'd give better than even odds that it will be ugly before Christmas. Here are some of the possibilities to watch.

- General economic slowdown. The economy seems to be slowing. Plus there are all of the other stresses I'll discuss, all of which are likely to make it slow more. But it seems implausible that a general slowdown will happen suddenly enough to precipitate a full-blown crisis. Though it sure provides stress that could cause something else to happen.

- European sovereign debt issues. We all have heard about Greece. Italy is in the news. But the really scary quantities of debt are in Spain. And Europe has proven to have no political appetite for having meaningful bailouts of one country by another. Either the prospect of the EMU (European Monetary Union) dissolving, or a liquidity crisis caused by a large default (even if it doesn't get called a default) is a likely bet.

- US sovereign debt issue. We dodged the debt ceiling debate. 2 of the 3 rating agencies have left the USA at AAA, but on a negative watch. (Meaning better than even odds of a downgrade within a year.) The third, S&P, based on past guidance is likely to downgrade the USA this month. Though they could do what the other two did. The next test is what the super committee comes up with for the second half of the cuts that are in the debt ceiling deal. That's expected around Thanksgiving. Whether that triggers a crisis depends on what is cut. But a crisis is unlikely. The bigger problem is that people have been backed into a political corner where the US government will be unable to intervene if something else goes south.

- Residential mortgages. The current expectation in the industry is a 3-6% drop next year, with 15% as a worst case scenario. However if the mortgage tax deduction were to be yanked (for instance as part of deficit reduction), much worse changes are possible. (And, of course, another financial crisis would take all bets off of the table.)

- Commercial mortgages. Everyone is aware that commercial mortgages are in a world of pain. However servicers have been very forgiving. They really don't want to force bankruptcies which are in nobody's best interest, so they restructure deals, collect as much blood as they can, then limp along. (A clear case of, "If you owe $100,000, you have a problem. If you owe $100,000,000 the bank has a problem.") However the sector is still under stress. And last month the delinquency rate spiked to the highest rate on record. There could still be trouble there. Particularly if a slowing economy reduces how much money is available to pay back debt.

- Private equity. There is a world of pain coming here as deals struck in 2006 and 2007 (2007 had a lot more) see principal payments come due this year and next. Based on the experience with commercial mortgages, I would expect the banks to be unusually forgiving. But the question is whether they have sufficient liquidity to be forgiving enough. (As much as the banks may want to make deals, they have a real problem if nobody pays them back.) Borders just went bankrupt, how many more companies will follow in the next year?

- Municipal debt. Meredith Whitney has been crying wolf on this for some time, but she has a real point. There are trillions of dollars of debt issued by municipal governments who are under stress. In the first half of this year they got unexpected revenue increases. However a slowing economy could cause them to get back in trouble. And their budget cuts to deal with financial pain could slow the economy. To add salt to the wound, most have large investments in pension plans that is budgeted to get 8% annual growth. Those rates are hard to reach in today's economy, so a lot of that money is in risky asset classes like investments in private equity. It is very likely that those assets will under perform, adding to budget holes. Central Falls, Rhode Island just filed for bankruptcy. How many more will follow in the next year?

- Chinese asset bubble pops. I know little about what is going on there. They do have a big asset bubble. History says that it will pop eventually. Nobody knows when, or what the fallout will be. However my impression is that when it happens, the pain will be mostly felt in China.

- External shocks. You name it. Successful terrorist attack. (The 10'th anniversary of 9/11 is coming up...) Unrest in the Middle East triggers oil spike. An unexpected major earthquake. (For instance the Cascadia fault slips, leaving over 100K dead in Seattle, Vancouver and Victoria.) These are all unlikely, but possible.

A worst case for the possible US government default

2011-07-26T04:33:00.001-07:00

(This is crossposted from Google+. If you use Google+, you scan find the original here.)

Lots of people are scared about the possibility of a US government default. But I have an unusual reason that few share. And many might think me crazy for worrying about one of the things that I worry about. So I'd like to share why I think what I do before I confess to my worry.

Close to a decade ago I read a fascinating book, Wealth and Democracy by Kevin Phillips. SIt is a large book. It is a detailed book. Most of it is a detailed history of when and where great wealth was created in US history, and its effects on the political process. (In a nutshell the conclusion is that periods of great wealth are periods where the wealthy wield great political influence, and conversely periods where the wealthy have less political control saw great declines of wealth disparities.) Don't try reading it unless you can get into the idea of reading close to 500 pages from a policy wonk going on about things that happened a hundred years ago.

Of course if you are able to read it, there is a lot of interesting material there. Kevin Phillips put a lot of effort in, and knows his stuff. He also has a history of making absurd predictions that work out. For instance in 1970 he wrote The Emerging Republican Majority which made the (then) absurd prediction that white Southerners who reflexively voted against the Republicans because of their role in the Civil War, were about to switch to the Republican party. Nixon followed the strategy that Phillips outlined, and they eventually did. In 1990, when Bush Sr had the highest approval rating of any president ever, Phillips wrote The Politics of Rich and Poor: Wealth and Electorate in the Reagan Aftermath which argued that Bush was vulnerable. Clinton later said that he used it as his election plan.

So what absurd prediction did he make in Wealth and Democracy? Well here is it as best as I remember it. In the last 400 years there have been three other examples of a country that was the sole global superpower. (Spain, the Netherlands, and England.) All three followed a remarkably similar trajectory. All had a major boom that was more based on financial speculation than real production. During this bubble, actual production was outsourced to other countries. This bubble crashed. A couple of years later all three got into a foreign adventure that was supposed to be quick and easy. Said foreign adventure turned out to be much longer and more costly than predicted. Parallel to this there was a "recovery" in which average people didn't really get much ahead. During this period there was even more outsourcing of real production. That ended in a credit crisis some 7-8 years later. Following the credit crisis there was a paper recovery whose effects weren't widely felt. That was not many years later followed by a second financial crisis. After the second financial crisis there was growing outrage in the population. Followed, almost exactly 15 years after the peak, by civil war in 2 of the cases. In the case of England there was civil unrest, but it was overwhelmed by WW I. The outcome was the establishment of a third major political party, Labour.

Incidentally in all three cases one of the countries that production and industrial knowledge was transferred to eventually wound up as the next superpower. Currently the best two candidates for that next superpower role are India and China.

Remember. This was written after the dot com bust, and before the Iraq invasion. Now compare with events of the last decade.

At first I thought, "He did a lot of work, and has a good track record, so I'll not discount it out of hand but I'll keep a lot of skepticism." Ever since then things have played out as predicted, almost like clockwork. I currently see lots of potential triggers for the second financial crisis. So I think that part is likely to play out, and my main hope is that we're more like England with civil unrest than Spain and the Netherlands with an actual civil war. However I have to say that the increasing polarization of the country over the last decade has made civil war much less unthinkable to me as a possibility than it was when I first read the book.

If the next financial crisis is triggered by a political stalemate, that makes me fear that civil war is more likely. Because the event will be one which arrives with already polarized constituencies that each has has a well-articulated story for the crisis that blames the other for not being willing to (cut government|consider any kind of tax). As financial disaster is felt by all, how can tensions fail to escalate along already clearly drawn boundaries? As they escalate, where will it end?

Please, Congress, find a way to compromise. If you fail, trillions of dollars in excess interest payments is the best possible outcome we can hope for. None of us wants to see the worst. Really.

I've switched to Google+

2011-07-20T23:47:00.000-07:00

I've been interacting on Google+. You can see my new posts there. The content is different, but includes some content which previously would have appeared here, such as this explanation of homophobia..

Therefore if you were following me here, you may want to try to follow me there instead.

An elementary explanation of the Riemann Hypothesis

2011-03-09T22:50:00.000-08:00

Elementary here is relative, but this explanation of what the Riemann Hypothesis means should be understandable to people who have no idea about complex analysis, zeta functions, and other usual things that you see in explanations of the Riemann Hypothesis. However I will invoke a substantial amount of probability theory.

There is a famous hypothesis in number theory. It is the basis of something known as Probabilistic Number Theory. It is trivially false. And yet so tantalizingly accurate. Here is the first form.

Every integer n has independent probability 1/log(n) of being prime. (Note that log(n) here means the natural log of n. Mathematicians always use that form, and never the log base 10.)

This is obviously false. We can improve it. The primes are 2, and then every odd integer n has independent probability 2/log(n) of being prime.

This is obviously false. We can improve it again. The primes are 2 and 3, and every number n not divisible by 3 with independent random probability (2/1)*(3/2)*(1/log(n)) of being prime.

Obviously we can improve this set of "random" conjectures infinitely often. And eventually will arrive at the right answer.

Now this is obviously false. Everyone knows that the primes are not, actually, random. Why did I say that it was tantalizingly accurate? Well because of the following.

All evidence available to us suggests that every statement that you would guess from the above sequence of conjectures is, in fact, true.

Let us take a simple example. Suppose that every integer n has independent probability 1/log(n) of being prime. Then on average how many primes would you expect to find up to N? It would be 1/log(2) + 1/log(3) + ... + 1/log(N). Let's call that function li(N). Let's call the count of primes up to N pi(n). (That is what number theorists call it. I wouldn't have called it that, but I'll follow tradition.) Here are some statements that are known to be true about pi(n) and li(n).

First of all li(n) is the integral, from 2 to N, of 1/log(x) plus O(1). That integral is n/log(n) + o(n/log(n)). You'd expect pi(n) to follow similar behavior. And, in fact, the prime number theorem says, pi(n) = n/log(n) + o(n/log(n)).

But we can do better than that. We expect pi(n) - li(n) to be some kind of weird random walk, centered around 0. If you know about random walks, you'd expect that expression to switch signs, infinitely often. Surprisingly, Littlewood proved this true in 1914. But the switches should be very, very rare. And, in fact, the first switch is believed to be around 1.39822×10³¹⁶.

Can we go on? What is the density of twin primes? Naively you might expect it to be (1/log(n))². But the version with 3 and 2 struck out tells you that in every 6 primes you have 2 numbers with probability (3/log(n))² of being prime. You can iteratively improve this estimate with 5, 7, 11, etc. This converges to a special case of the Hardy–Littlewood conjecture which would prove the twin prime conjecture. All existing numerical evidence points to this being true.

So from this probabilistic hypothesis we can come up with many, many statements. Some of which are known to be true. Some of which are not. But I have not yet connected this to the Riemann Hypothesis. Let me correct that.

Both the conjecture and Littlewood's theorem suggests that pi(N) - li(N) behaves a lot like a random walk. If that is so, what can we say about the errors? Well from the central limit theorem, we should expect pi(N) - li(N) to have something like a normal distribution. In fact there are many forms of the central limit theorem. The right one to invoke is the Lyapunov condition, from which pi(N) - li(N) can be shown to have average 0, and variance equal to the sum of the variances for each number being prime. If you look at the formula for a biased coin flip, the variance for n being prime turns out to be 1/log(n)-1/log(n)². If you sum up the variances up to N, the sum of the 1/log(n) terms dominates, and the variance turns out to be li(N) + o(li(N)) = N/log(N) + o(N/log(N)). From which the expected error is O(sqrt(variance)) = O(sqrt(N/log(N)).

Now the expected error may be that, but the error will occasionally take very unexpected values. However the law of the iterated logarithm says that the maximum deviation should be O(sqrt(N/log(N)) * log(log(sqrt(N/log(N))))) < O(sqrt(N)).

Now let me get in to the Riemann Hypothesis proper.

Usually when you see the Riemann Hypothesis it usually talks about nontrivial zeros of the Riemann Zeta function. Thanks to a bunch of complex analysis that I won't get in to, it turns out that all of the non-trivial zeros of the Riemann zeta function have real part < p if and only if pi(N) - li(N) = o(N^p). Well from the above analysis one would expect that for every p > 1/2, that there would be no non-trivial zeros of the Riemann zeta functions which are p or greater. It is also known that if non-trivial zeros are under 1/2, then some must be over 1/2, and therefore the above analysis (plus these facts I am giving you from complex analysis that I have not described) say that all of the non-trivial zeros should have real part exactly 1/2. Conversely if any non-trivial zero has real part greater than 1/2, then pi(N)-li(N) cannot obey the rules one would expect from probabilistic number theory.

Currently all existing evidence suggests that the Riemann hypothesis is true. It is known that the first 250 * 10⁹ roots of the Riemann zeta function have real part 1/2. It it known that infinitely many roots have real part 1/2. All existing numerical evidence we have suggests that the result is true. All that we are missing at this point is a pesky detail called a proof. :-)

Finding related items

2011-02-13T22:54:00.000-08:00

I've mentioned in the past (not in this blog) that C++ programs can beat databases by many, many orders of magnitude. I learned this while doing a specific fun project. I have permission from the company to discuss that project, and here is a description of that project.

I was working for a contract for ThisNext. Their database has a large number of records of different items, which a large number of people have tagged with arbitrary tags. Think millions of items, millions of distinct tags, and hundreds of millions of times that tags were applied to items. (Often the same tag has been applied to the same item multiple times.) Their idea was that this data could be used to discover which other items are somehow "related" to the current one, and possibly of interest if you find the current one interesting. This is used to populate related items, which is intentionally somewhat whimsical. For instance if you are interested in a pair of silk hotpants then perhaps you might be interested in velvet shorts, lumberjack hotpants or a Bo Peep Jumpsuit.

The suggestions are intentionally somewhat whimsical, and are generated by user activity. The idea behind them is that two items with tags in comment are likely to be related. The more common the tag, the less related they are. For every item they wanted the 30 most related items, sorted in the order of how related they are.

There is an obvious computational problem that you will face. Suppose a popular tag was applied to 30,000 different items. Then for 30,000 items you're going to have to compare to 30,000 other items, which gives you 900,000,000 relationships. This clearly is a scalability problem. So their initial version had implemented this as they just counted the number of tags in common between items, and they skipped any tag that was used over 500 times.

This originally worked fairly well, but as time went by it slowly got worse. The first problem is that their program slowed down as they got more data. When I arrived it was taking a couple of hours to run and was affecting when they could start running backups. The second problem was that if you had an item that had only been tagged once with a popular tag, such as "sunglasses", then no suggestions would be generated for it. They wanted me to try to solve one or both, preferably both, problems.

My first step was to try to figure out the ideal result. And then I was going to try to optimize that. So my first step was to try to find a smooth function that worked over multiple orders of magnitude, that captured the idea that a common tag was less of a relationship than an uncommon tag. The phrase "multiple orders of magnitude" made me think of logarithms, and so I decided that if a tag had been used n times for one item, m times for the other, and s times all told, then log(n + m)/log(s) would be the strength of that relation. The total weight of the connection between two items, I decided, should be the sum of the weights from the shared tags. I decided to break ties in favor of the most tagged other items. (Thinking that this was evidence of popularity, and popular items were probably better.) And any remaining ties in favor of the most recently created item. (Being tagged a lot in a short time is also evidence of popularity.)

Note that this formula is reasonable, but arbitrary. It was chosen not out of any knowledge of what the "right" answer is, but rather as the simplest reasonable formula that looked like it reflected all of the ideas that they wanted to capture.

Next came finding a potentially efficient way to implement it. My idea was that I could do it with two passes. In the first pass I'd compare each item with up to the top 100 items withing a tag. Then I'd sort and find the top 100 related items. In the second pass I'd figure out the full relationship between those items, and sort in the final order. This would prevent the quadratic performance problem that kept popular tags from being analyzed in the first implementation.

If you are familiar with databases then you may wonder how I was planning to pick the top 100 items from each tag, for all tags at once. If I was using Oracle I could just use analytic queries. That feature made it into the SQL 2003 standard, and so support for windowed functions is showing up in other major databases. But I wasn't using a database with support for that. I was using MySQL. However MySQL has an efficient hack for this problem which is described in http://www.xaprb.com/blog/2006/12/02/how-to-number-rows-in-mysql/. (If you're doing a lot of work with MySQL, that hack is worth knowing about.) With that hack it is easy to have a query that will return the top 100 items for each tag.

So I coded up my program as a series of MySQL queries. When it came to actually processing the items there was no way that there was space for the database to handle it in one big query, so I opted for having a temp table into which I stuck ranges of items, which I'd then process in batches. The program ran..but it took 5 days to finish. This obviously was not going to be good enough. I thought about it long and hard, and decided that the best way to optimize would be to try to rewrite it in something other than SQL.

Normally I'd use Perl, but Perl wastes memory fairly liberally, so I was sure that that it couldn't store this dataset in RAM. I am not a C++ programmer, but I noticed is that C++ has vectors. Vectors support the same basic operations that I am used to in Perl. But (unlike Perl arrays) they are memory efficient and very fast. So I talked to the company, got permission to try rewriting this in C++, then went to work.

According to my memory, I used 4 data structures.

item: A vector of structs. Each struct said how often the item had been tagged, where the item_tags started, and the number of item_tags there were. (The item_id was your position in the vector.)
item_tag: A vector of structs. Each struct knew the item, the tag, and how many times that tag had been applied to that item. It was sorted by item_id then tag_id.
tag: A vector of structs. Each struct knew the number of times that tag had been tagged, where the tag_items started, and the number of tag_items there were.
tag_item: A vector of structs which is exactly like item_tag but this one was sorted by tag and then item.

Again I wrote my program in 2 passes exactly like before.

For each item the first pass was meant to identify which other items might possibly be the most related. Here it is in pseudocode:


foreach tag for that item (found in item_tag, the range is from item_tag):
    from tag figure out the range of tag_items I was interested in
    (the size of the range was a constant, other_items_limit)
    foreach other item found in tag_items:
        if item and the other item differ:
            save the existence of that relationship in relationships
            (a relationship contains the other item id, the weight, and
            the popularity of the other tag = how many times it was tagged)

    sort relationships by other item id

    aggregate relationships by other item id (aggregating weights across tags)

    sort relationships by weight, times other item tagged, and other item id
    (all descending)

    pick top other_items_limit items for the second pass

The purpose of the second pass is to find the final list of other items. Here it is in pseudocode:


foreach other item found in the first pass:
    weight := 0
    traverse item_tag in parallel for both:
        if a common tag is found:
            add to the calculated weight
    save the relationship

sort relationships by weight, times other item tagged, and other item id
(all descending)
for the top 30 relationships:
    print this relationship to a file

Remember that the original SQL version took 5 days. The C++ program took under 20 minutes. I got rid of a details file that wasn't actually needed except for debugging, and I stored all of the logarithms that I needed in a vector so I could do an index lookup rather than an expensive floating point calculation. Total run time dropped to under 5 minutes. I left in various assertions that had been a development aid, and called it a job well done. Together with the steps to run queries, extract a dataset, run the program, and do the final upload, end to end I was generating the full result set in something like 15 minutes.

This astounded me. I had good reason to believe that C++ was going to be faster than the database. But 3 orders of magnitude faster? That I did not expect!

I believe that the biggest single difference is the work you need to do to access data.

What is the fastest way to get at an isolated piece of information in the database? You do an index lookup. Meaning that you do a binary search through a tree that then tells you what database row has your information and what page it is to be found on. (Don't forget that walking a binary tree means multiple comparisons, if/then checks, etc.) You then read that page of memory, parse through it to find the row you want, parse the information you want out of that row and return it. Note that the instructions for what exactly to do is likely represented in some sort of byte code that has to be interpreted on the fly.

In my C++ program the equivalent operation is, "Access vector by index." Is there really any surprise that this is several orders of magnitude faster?

There are other differences as well. In the database I was working hard to keep batches small enough that the temporary tables I was throwing around would fit in RAM. In C++ a good deal of the time the next piece of information I wanted would be in L1 or L2 cache inside of the CPU. (The fact that I was walking through data linearly helps here.) MySQL likes to represent numeric types in inefficient ways, while the C++ program uses faster native data types. MySQL had to calculate a lot of logarithms, which was work I mostly avoided in C++. These things all helped.

It is also possible that I had some fixable performance problem in my SQL solution. But in discussing this project I have encountered a number of other people who have had similar experiences. And the general order of magnitude of speedup that I encountered is apparently not particularly rare when going from SQL to C++.

However despite this experience, I remain a big proponent of pushing work to the database and not worrying about performance unless you need to. Why? Because I've pushed databases much harder than most people do. Only a handful of times have I hit their limits. And until I hit those limits, they do a lot to simplify my life, and speed development.

But I am now aware exactly how many orders of magnitude faster I can make things by rewriting in C++. But with a giant corresponding increase in development time and required testing before you know you have correct results.

A final note. This project was fun for me, but what did ThisNext think? They were delighted! I solved their batch processing performance problem. In testing the suggestions produced were clearly better. And based on traffic and revenue figures following the release of my change, I was told that I had increased revenue by at least 10%.

SQL formatting style

2011-02-03T14:12:00.000-08:00

I have written considerably more SQL than most programmers have. And after you've written and maintained several hundred line queries, it becomes obvious that SQL is a programming language like any other. Which means that indentation helps comprehension for all the same reasons that it would in any other language. (I'm continually puzzled by programmers who fail to grasp this, and try to put SQL statements on one line.) But what indentation style should you use?

Here are the principles that lead to my current chosen style.

Indentation should be in the range of 2-4 spaces. Research indicates that people can read and understand code better with an indent in that range than one which is larger or smaller than that.
When one thing depends on another, indentation should be increased. This makes the core statements obvious.
When easy, line things up.
In any list of things, it should be possible to extend the list by pasting new lines on without changing others. In a language like SQL where you can't have trailing commas, that means putting the comma before the line.
If you wrap, then operators belong at the front of lines. This makes the logical flow easy to see at a glance. (I got this one from Perl Best Practices.)
SQL is usually case insensitive, so use underscores in field names to avoid ambiguous parses of things like ExpertsExchange and ExpertSexChange.

And here is an example of what this looks like in practice.


  SELECT s.foo
    , t.bar
    , x.baz
    , CASE
        WHEN s.blat = 'Hello'
        THEN 'Greeting'
        WHEN s.blat = 'Goodbye'
        THEN 'Farewell'
        ELSE 'Unknown'
      END as blat_type
  FROM table1 s
    JOIN table2 t
      ON s.some_id = t.some_id
        AND s.some_type = 'wanted type here'
    JOIN (
        SELECT *
        FROM yet_another_table yat
        WHERE yat.reason = 'Subquery demonstration'
      ) AS x
      ON t.another_id = x.another_id
  WHERE s.blat = 'some'
    AND t.blot = 'condition'
  ;

If you've never thought about how to format SQL this style is likely to have a few surprises. But try maintaining a lot of SQL that has been formatted in this way, and you should find that it works out quite well.

How I do Proofs

2011-01-03T21:46:00.000-08:00

I was looking through some old papers, and decided to put online some advice that I gave many years ago about how I tackle mathematical proofs. The advice is at How I Do Proofs. This is only lightly edited from the original that I handed out to a course I was teaching in the late 90s.

The course is the one I described at Teaching Linear Algebra. I never got any feedback on my request for improvements, but judging from how well that class did at proofs I believe that the advice was helpful. As one student I asked put it, "Doing proofs is like filling out a shopping list." That is how it should be in a first course where you are learning how to do proofs.

The class where I introduced this advice was fun. I started by handing out printouts of the advice. I then went through the flowchart at a high level. And then I put up a routine theorem from the text, which was the next thing I was supposed to do in the course. And I said, "OK, now that you've all learned how to prove things, you're going to prove this."

You should have seen the shock on their faces! Some started complaining. So I said, "No, seriously. You will all prove this. Just wait and see. It will work."

Then I began. I asked the one person to say what the first step was. A second person to do it. A third person what the next step was. A fourth to do it. And so on. And every time they did something that could be written down, I wrote down exactly what they said. Before long they had proven a result that none of them thought they could prove!

The third of the homework for that night which was on that day's material (if this statement puzzles you, go back to Teaching Linear Algebra and read about the homework strategy in that course) was very heavy on doing proofs. And I can tell that they referred to the handout by one fact. The next day the grader came to me, very puzzled, and asked me to explain this strange piece of reasoning that everyone had used. It was the contrapositive, which is an interesting alternative to proof by contradiction. One of the problems could be solved either with contradiction or the contrapositive, and they all chose the contrapositive because I'd listed it before contradiction.

I considered this a very good thing. One of the major problems that students have with proof by contradiction is that it works too often. Proofs that require a bit of routine algebra or computation can always be rewritten as proof by contradiction. The rewritten proof is correct, but it is unnecessarily confusing and it is better not to have used contradiction. However because contradiction seems to always work, it becomes a sledgehammer that the student always uses. Without noticing that it is sometimes the wrong tool.

The contrapositive doesn't have this drawback. It can solve the same problems as those that "really" needed proof by contradiction. But it doesn't lend itself to solving problems that never needed proof by contradiction. And so students who have been trained to try the contrapositive do not tend to make their simpler proofs over-complicated.

In fact when I drew up the advice I nearly left out contradiction. But it is so widely used that I figured I had to include it. However I tried to subtly discourage its use. And to the best of my knowledge I succeeded, I'm not aware that any students in my class ever used it in any of their proofs.

Bluer than blue

2010-11-15T21:57:00.000-08:00

On my profile I have answered the question Something I can't find using Google with The lyrics to the version of "Bluer than blue" we sang in highschool jazz. It isn't that you can't find lyrics to the song. The problem is that there are many songs out there with that name, that have unrelated lyrics.

Well my question has been answered. Someone who was in my high school jazz choir found my profile, emailed me, and emailed her memory of the lyrics. Courtesy of Lisa Schmidt, here they are:

Bluer than Blue

Once I thought the world was ours to share
Always knew there'd be someone to care
Now I realize
Promises were lies
Wish I'd find the one who really loves me

I'm feelin' bluer than blue
I'm all alone and feeling bluer than blue
My baby left me, now I'm cryin' like a fool
I want somebody who will love me
Oh well I'm bluer than blue
I'm all alone and need somebody that's true
Looking around to see if you'll come back to me
I want somebody who cares

Guess I've learned my lesson / Broken all the rules
Well, you know I try to recall
One thing I am certain / Love's a game of fools
You're never ready when loneliness calls

Bluer than blue
I'm all alone and need somebody that's true
Hanging around and hoping you'll come back to me
I want somebody who cares

[Here there was a vocal interlude, where we all went "doo, doo do dooo, doo wop etc."]

Guess I learned my lesson / When we started kissin'
Well, you know I try to recall
One thing I am certain / love's a word for hurtin'
You're never ready when loneliness starts calling [key change]

Bluer than blue
I'm all alone and need somebody that's true
Looking around to see if you'll come back to me
I want somebody who will love me, come on baby love me
Can't you see I'm lonely?
I want somebody who's gonna care [she's bluer than bluuuuuuuue]

Thank you Lisa!

Now a bit of reminiscing. When I was in choir, Canada did something really good. What they did is arrange for a variety of regional competitions, some of which would qualify you for the nationals. In those competitions each choir had 15 minutes to do a prepared set. Then one of the judges would come up and work with the choir for 15 minutes. Then the choir would go back stage and be given a workshop. I have no idea if they still do it, but I can't praise it highly enough.

The point of this was not the competition, though that was serious, but rather that it taught the teachers how to teach better. This quickly made everyone better. A lot of the choirs were simply amazing. I had the fortune to be at Esquimalt when the choir teacher, Eileen Cooper, began really going through this system and got mentored by the choir teacher at Argyle. In the time we were there, we went from a mediocre choir using instrumental mikes and with no idea how to really hold them to being among the top high school jazz choirs in the country. To this day I believe that I was only tolerated because it was so hard to find baritones who were willing to be part of the choir.

But be that as it may, most of my positive memories of high school involve choir. And as a parent it has been wonderful that I am able and willing to sing for my kids.

Practice poker online for free

2010-10-26T22:00:00.001-07:00

I enjoy playing poker. I'm not that good, but I do OK in small home games and I enjoy myself.

Recently I've been having a lot of fun playing online for free. How, you ask? Well go to http://www.fulltiltpoker.com/, sign up for free, get a free account, and start playing with play chips. Work your way up to the level you've got skills for, then have fun.

Now everyone's first reaction when I tell them this is that people play really stupidly when they are playing online for free. It isn't worth trying.

But this is not necessarily so! When you play on Full Tilt you can reload to 1000 play chips every 5 minutes. And at the bottom tables people play like complete donks because they can with impunity. However there are a lot of different buy-ins. As you move up, the skill level increases. So you get a gradation of skill from ridiculously soft at 1/2 up to a boss level of 50K/100K. At the latter level typical buy-ins are 4,000,000. The game I just looked at had 2 players, each with about 25,000,000 in front of them. You can't play that game unless you've managed to earn your way to that level. Anyone who does that has legitimately beaten a *lot* of other players, and put a lot of time in. Do you think those people play poorly?

If you've got basic skills (easily acquired), it isn't hard to beat the easy "donk" levels of free poker online. Then you can move up to more fun levels that require more skill. And once you can play poker well? Well, then you can buy in with some confidence about your skills (at least for low level play) and enjoy both the game and the money you're making at it.

Here is a data point. My sister read a couple of books, went online, got a free account, worked her bankroll up to a few hundred thousand and was playing at the 100/200 level. Then she began playing for money, entered the WPS, and look what happened.

So if you find poker fun, and want to practice, you can online for free. The only thing you'll lose is your time. Which isn't a cost if it is fun for you.

Piaget Water Level Test

2010-08-25T23:43:00.000-07:00

Many years ago in a dentist's office I read an interesting article in a magazine. It talked about how there were questions that were specific to certain adult genders. In particular until puberty there was no measurable difference in performance on the question, but after puberty there was a large difference. As examples they offered a verbal question that women do well on, and a question where they drew two cups, one tilted, and asked people to fill in the water level if both were half full. (Artistry not required.) Men do well on this. Most women don't. And education doesn't matter, women who graduate college do worse than men who drop out of high-school.

I botched the verbal one due to something that looked silly to me, got the male one, and dismissed the article as garbage. Then a month later it came up in a conversation with my girlfriend, and she got the men's question wrong. I couldn't remember the one that women do well on. This made me curious, so I asked my mother the same question and she got it wrong. I can call my mother many things, but unintelligent is very much not among them. As an example, when she was at Stanford in the 50s they gave her a battery of ability tests. The only one she was not in the top 1% on was manual dexterity.

(Side note for married men. Do not rush to give your wife this test. I've started many marital fights that way, and I have never tracked down a question that women do better than men on. I know there is one, and I know that at 19 I couldn't do it, but I don't remember what it was and have never encountered another.)

Since then I've learned that this question is called the Piaget Water Level Test. The background on this is that Piaget found that children typically gain specific mental abilities at specific ages that are tied to specific growth spurts. And so he collected examples. For instance there is a specific age at which children learn that people who were not present would not have seen what they did, and another at which children learn that when you pour water into a tall thin glass you don't have more than you used to. And as he collected examples, he eventually offered the water level test. Which, oddly, men gain the ability to do during our last growth spurt, puberty, and most women never do.

I've seen various estimates for how well people do on it. One was that 90% of men can do the task, and only 30% of women. That seems to be a good fit with my experience.

An interesting side note. It is widely noted that there is a large gender imbalance within programming. But I've found through experience that programmers I know, whether male or female, have a 100% success rate on this question. I have no idea what to make of this tidbit, but I find it interesting.

Incidentally for those wondering what the answer to the question is, the water level is horizontal to the ground. The most common answer among women I've asked is to draw the water level parallel to the bottom of the cup. The second most common answer from women is to realize that there is a trick, and to draw the water level tilted twice as much as the cup. When the correct answer is pointed out, women recognize it as very obvious. I imagine that their feeling is much like how I felt after botching the verbal question that I blanked out of my memory.

BTW if anyone knows of any question with the reverse gender characteristics, I've been looking for it for over 20 years. It is frustrating - I know that at least one such question exists, but I've never found it.

Analysis vs Algebra predicts eating corn?

2010-08-19T23:14:00.000-07:00

I like learning about odd connections between disparate things. This probably is the oddest example that I know.

Broadly speaking, mathematicians can be divided into those who like analysis, and those who like algebra. The distinction between the two types runs throughout math. Even those who work in areas that are far from analysis or algebra are very aware of the difference between them, and usually are very clear on which their preference is. I'll delve into this in more depth soon, but for now let's just take it for granted that this is a well-known distinction, and it has meaning for mathematicians.

Back when I was in grad school there was a department lunch with corn on the cob. Partway through the meal one of the analysts looked around the room and remarked, "That's odd, all of the analysts are eating corn one way and the algebraists are eating corn another!" Everyone looked around. In fact everyone was eating the corn in one of two ways. One way was to munch over the length of the corn in a straight line, back up, turn slightly, and do another row across. Kind of like how an old typewriter goes. The other way was to go around in a spiral. All of the analysts were eating in spirals, and the algebraists in rows.

There were a number of mathematicians present whose fields of study didn't make it clear whether they were on the analysis or algebra side of things. We went around and asked, and in every case the way they ate corn matched their preference. Since then I've made a point of amusing myself by asking mathematicians I meet whether they prefer algebra or analysis, and then predicting which way they will eat corn. I'm probably up to 40 or so by now, and in every case but one I've been able to correctly predict how they eat corn. The one exception was a logician who claimed to be exactly on the fence between the two. When I explained the corn thing to him he looked surprised, and said that he had an unusual way of eating corn. He went in loose spirals! In other words he truly was a perfect combination of algebra and analysis!

If you have even a passing familiarity of probability, it is clear that despite how unbelievable it initially is that the type of mathematics you prefer is connected to how you eat corn, it is pretty much certain that there actually is a very strong connection. If you believe, as I do, that this difference is connected to how we think about other things, then there must be some odd connection between how we like to understand the world and how we eat corn. Why is another matter.

How do I explain the distinction between algebra and analysis? Well the best way to understand it is to ask you to study advanced mathematics. You will have to take many courses with the word "algebra" in the name, and others with "analysis" in the name. By the time you're done you'll have experienced the difference, and you'll be clear on which you prefer. Odds are you won't do that, but that is the most reliable way to come to understand it.

If I have to wave my hands and explain it, I would explain it like this. In algebra there are sequences of operations which have proven to be important and effective in one circumstance. Algebraists try to reuse these operations in different contexts in the hopes that what proved effective in one situation will be effective again. By contrast an analyst is likely to form an idiosyncratic mental model of specific problems. Based on that mental model you have intuitions that let you carry out long chains of calculations that are, in principle, obviously going to lead to the right thing. Typically your intuition is correct to within a constant factor, and you're only interested in some sort of limiting behavior so that is fine.

If you don't know any advanced math, the odds are about equal that my explanation is going to mislead you as to give you an idea what I am talking about. You'd be better off figuring out your preference by looking at how you eat corn. That said, the distinction carries through into other subjects that I've learned about. But not in a clear and obvious way.

For instance I've noticed the difference cropping up in programming. The distinction is often hard to explain. There are a wide variety of programming techniques, and most programmers have only really learned a few. Some of those techniques appeal to analysts, and others to algebraists. But if you've only been exposed to techniques that are a good fit for one, then how do you know which you'd prefer? Worse yet, when two programmers talk and have different experience bases, how can they tell whether their natural intellectual tastes are similar or different?

Let me give some examples. Upon my first encounter it was clear to me that object oriented programming is something that appeals to algebraists. So if you're a programmer and found Design Patterns: Elements of Reusable Object-Oriented Software to be a revelation, it is highly likely that you lean towards algebra and eat your corn in neat rows. Going the other way, if the techniques described in On Lisp appeal, then you might be on the analytic side of the fence and eat your corn in spirals. This is particularly true if you found yourself agreeing with Paul Graham's thoughts in Why Arc Isn't Especially Object-Oriented. There was a period that I thought that the programming division might be as simple as functional versus object oriented. Then I encountered monads, and I learned that there were functional programmers who clearly were algebraists. (I know someone who got his PhD studying Haskell's type system. My prediction that he ate corn in rows was correct.) Going the other way I wouldn't be surprised that people who love what they can do with template metaprogramming in C++ lean towards analysis and eating corn in spirals. (I haven't tested the last guess at all, so take it with a grain of salt.)

Going out on a limb, I wouldn't be surprised to find out that where people fall in the emacs/vi debate is correlated with how they eat corn. I wouldn't predict a very strong correlation, but I'd expect that emacs is likely to appeal to people who would like algebra, and vi to people who like analysis.

And now to wrap up, why would how we eat corn say something how we think? Here is what I think.

When you pick up a piece of corn on the cob, you have two cues for how to eat it. The first is that the corn is laid out in very nice rows. How can you not follow the lines that are laid out for you? The other is that as you eat, your teeth scrape down the corn. If you twist your wrist, you'll eat more efficiently. Why would someone want to eat inefficiently?

My best guess is that the cue you notice and follow reflects a natural tendency about how you tend to think in general. And this tendency is tied to such things as what kind of math you prefer or what programming techniques would prove interesting for you.

How did pterosaurs get so big?

2010-08-03T09:21:00.000-07:00

The pterosaurs got going something like 230 million years ago. They died out with the dinosaurs 65 million years ago. Over their history they came in all sizes, from Rhamphorhynchus who was the size of a sparrow to Quetzalcoatlus with a wingspan of variously estimated as being 30-40 feet. Spread out it was a similar size to a t-rex, and on the ground its estimated height was close to a giraffe's. The largest bird ever, Argentavis was much, much smaller than that.

However the birds arose about 150 million years ago. Feathers were a big advantage in flight, and over time the birds took over a lot of what the pterosaurs were doing. But the pterosaurs did not go away. Instead they wound up being in niches for very big flying animals. This is competition through specialization, which I talked about some time ago when I discussed the Neanderthals.

This coexistence provides evidence that birds were generally better fliers, but there was a niche for very big fliers that the pterosaurs were better at. The question I'm curious about is why the pterosaurs were better at being big fliers than birds were.

I have a theory. But before I can explain it I need to provide some background.

Wings have evolved in vertebrates three times in pterosaurs, birds, and bats. All three started with the basic vertebrate limb structure and found different ways of constructing a wing out of it. In both bats and birds the arm bones form part of the wing. Now an important fact about vertebrate bones is that different bones grow at different rates as you grow. In particular arm bones start off shorter and catch up later. The result is that in birds and bats, babies have the wrong proportions for their wings to be useful. Therefore baby birds and bats can't learn to fly until they have achieved a significant fraction of their full size.

Pterosaurs were different. Their wings were entirely constructed from wrist and hand bones. (Fully half the wing was supported by an elongated 4th finger.) Hand bones stay in proportion your whole life. Comparisons of fossils of pterosaurs at different ages in the same species verifies that their wings always had good proportions for flight. Furthermore we have fossils from baby pterosaurs that died miles out at sea, which is direct evidence that they flew young.

What does this have to do with the eventual size of the animals? Well birds cannot learn to fly until they are near full growth. Which means that they need intensive care from their parents until they reach that growth. This care is a significant fact of life for bird species, and is why most types of birds have both parents providing care. Unlike most mammals where the mother is generally capable of taking care of young on her own. The larger the bird is, the harder this care is to provide.

By contrast pterosaurs were probably able to take care of themselves at a much younger age, and smaller size. Which means that they were free to grow for a lot longer, to a lot larger size, without unduly taxing their parents. (In truth we don't have any data indicating how much or little parental care baby pterosaurs got. But I suspect it was less than birds get.) And, I believe, that is why they were able to get so much larger than birds.

Random trivia I came across in preparing this post. The reason bats can't fly during the day is that their wings are vulnerable to sunburn. There is evidence that pterosaurs had a protective layer so they didn't have this issue. Also birds have stiffer wings than bats do, which provides better lift and less maneuverability. Pterosaurs had more joints in their wings than birds do, but didn't have finger bones inside of the structure of their wings like bats, which suggests to me that their wings would have been somewhere between.

And my whole train of thought was started by watching National Geographic - Sky Monsters. If you're interested in pterosaurs, it is a worthwhile video.

Dozenal glyphs - a modest proposal

2010-07-06T15:09:00.000-07:00

A proposal that shows up every so often is to use dozenal arithmetic. The idea is simple, in base 12 more fractions work out very conveniently. You can easily divide things 3 and 4 ways. This is frequently convenient, and is why we often sell things in dozens. It is also why the much maligned Imperial system sneaks factors of 3 (3 tsp in a tbsp) and 12 (inches in a foot) in various places. And there are numerous minor benefits, such as the fact that the multiplication table becomes significantly simpler and therefore easier to learn.

When the French created the metric system, they based it on factors of 10 everywhere. They even went so far as to try to measure angles in gradians (a quarter circle had 100), and to use decimal time. The world as a whole rejected decimal angles and times, but has adopted decimal metric everywhere else. Which is very convenient for scientists, but is hard to divide into thirds and somewhat inconvenient for quarters. Furthermore the persistence of both systems results in occasional annoyances like the fact that in daily life we usually prefer to measure speed in km/h, but for energy and power calculations the units only work out properly if you measure in m/s. Resulting in an annoying factor of 3.6 that shows up converting between them.

The other day I read An Argument for Dozenalism that made many of these arguments. Nothing new. However it made the interesting point that ideally a dozenal arithmetic would have its own set of glyphs. It suggested that 0 and 1 could be kept, but everything else should be changed. In my opinion that argument is correct, it is very confusing if 21 sometimes is 5*5 and sometimes 3*7, which makes a mixed dozenal and decimal world harder than it needs to be. But this raises the interesting question of what a logical dozenal set of glyphs might look like.

I've amused myself with thinking about this, and I have a proposal for a set of glyphs that are (mostly) unused, easy to learn, quick to draw, and are much more logical than existing ones. All have a vertical line in the middle. At the top there is a choice of a hook starting on the left, a straight end, or a hook starting on the right. At the bottom there is a choice of a hook ending on the left, straight down, on the right, or bending right to cut across the vertical line. This gives 12 possibilities. By incrementing the bottom first, and the top when you get a carry, you get a sort of a 3,4 base. Which means that a glance at the glyph tells you immediately its sign mod 4. A glance at the top tells you whether it falls in the range 0-3, 4-7 or 8-11.

So instead of the current system of 10 random symbols to memorize, you get two simple rules. While at first it seems bizarre, it grows on you quickly. And it gives you a very quick way to tell whether a given number is written in decimal or dozenal. To get a sense what it looks like, take a look at the 12 times table (my apologies for the handwriting - my none too good penmanship gets worse when I'm using a mouse pad to draw with):

Note in particular how regular the patterns are for 2, 3, 4, and 6. Doesn't this look easier to memorize than the decimal times table? As an exercise try writing out your favorite sequences. Whether you're writing out squares, powers of 2, or primes you'll see that more patterns leap out at you in dozenal, making them easier to learn.

So there is my humble contribution to a dozenal future.

(In other news, I now have a patent to my name, though in fact it is owned by a previous employer. By the standards of the patent system, it is not a particularly bad patent. But if I had my druthers, it would have never been filed. Ah well.)

The stock market thinks Microsoft has just under 4 years left

2010-06-09T08:00:00.000-07:00

(I posted this basic analysis at http://news.ycombinator.com/item?id=1417156, which was a discussion of this Newsweek article, and then decided that it was interesting enough to call out for additional attention.)

Those of us who pay attention to finance know that the market tends to be more accurate than any individual person. So sometimes it is worth analyzing what the market is saying about different companies.

If you look at the stock market, it clearly saying that Microsoft's future doesn't look as bright as Google's or Apple's. That's why Microsoft is worth a P/E of about 13, while Google is worth one of 22 and Apple is worth one of 21.

The market projection gets substantially more stark when you subtract current book value to find how much the market values future revenue. (Book value is what all of the company assets would be worth if it was broken up and sold today. For Microsoft this is largely made up of their cash reserve.) Microsoft's market cap is 221 B, their book value is 46 B, and therefore 175 B of their market cap is projected future earnings. Their current profit is 46.28 B/year, and that works out to the market valuing them at their current earnings stream projected over a bit under 4 years.

For Google the equivalent exercise says a market cap of 154 B, and book value of 38 B so 116 B of market cap is projected future earnings. Their current profit is 14.81 B/year, which translates into the market valuing them at their current earnings stream projected over a bit over a decade. (10.4 years.)

For Apple the equivalent exercise says a market cap of 225 B, a book value of 39.4 B for 185.6 B of market cap due to projected future earnings. Their current profit is 17.22 B/year, which translates into their current earnings stream projected over a decade. (10.8 years.)

So the projection that Microsoft is walking over a cliff in a few years while both Google and Apple have a decent future. The market is perfectly aware that a lot changes in 10 years, and so they heavily discount any projections out that far. But the market is more likely to be correct for near events.

Now admittedly I've never liked Microsoft. But this isn't just claimed by some random haters on the Internet. This is the consensus of the stock market, which is based on a lot of informed people putting their money where their mouths are. This is worth thinking about.

(I took all figures for this from http://finance.yahoo.com/q/ks?s=msft, http://finance.yahoo.com/q/ks?s=goog and http://finance.yahoo.com/q/ks?s=aapl. I got book value by multiplying book value / share times shares outstanding.)

Le Châtelier's principle: not just for chemists

2010-05-16T23:01:00.000-07:00

I seem to be updating my blog once a month or so. I don't intend to do that, I'm just busy. This particular entry is one I've been meaning to write for months, but just haven't gotten around to.

But before I get to it, a brief digression. My sister would like to get into a poker tournament that she needs to be voted in to. If you could take a moment and vote for Jennifer Tilly, it would be most appreciated. Thank you.

Now on to the main subject. One of my favorite principles of chemistry is Le Châtelier's principle. It has several forms, but the most general (though admittedly not perfectly accurate) is Any change in status quo prompts an opposing reaction in the responding system.

What does this mean? Well let's take a simple example. Suppose we have air in a chamber, and apply pressure to compress the chamber. From Le Châtelier's principle we expect it to push back harder. In fact it does. From the ideal gas law, applied in a simplistic way, once the volume is cut in half the pressure will double, and it will indeed be pushing back harder.

However this understates the true effect. It turns out that the act of compressing the chamber heats up the gas, increasing the temperature, and this causes it to push back even harder than it would have otherwise. This rise in temperature when you compress is called adiabatic heating. The corresponding decrease in temperature when you decompress a gas is used inside of your refrigerator or AC system to move heat from a cool place to a warmer one. Similarly when warm air rises the pressure drops, which causes it to cool down. This is why air on mountain tops is colder than air at sea level. (The drop is about 10C per km of height.)

Anyways, the fine details notwithstanding, what we find is that when you push harder on this simple system, it winds up pushing back harder. And you eventually find yourself at another equilibrium. Furthermore what works for simple systems, works for many more complicated systems.

The big question that I had as a kid in chemistry class was why. Why does this always work out? I never got a good answer from my teacher, and my dissatisfaction with "it works because it always works" answers was one of the reasons why I chose to go on in math instead.

Interestingly, many years later in an advanced math course I learned the trivially simple answer. Which requires essentially no math to understand!

Here is that answer.

A system is at equilibrium when all forces on it are balanced, and it can rest in that state indefinitely. For instance take a pencil and lay it flat on a table. It is at equilibrium there. There is also another equilibrium where it is balanced on its tip, but it is very hard to put it in that equilibrium.

Now not all equilibria are created equal. A stable equilibrium is one where any perturbation of the system will cause it to head back towards that equilibrium. An unstable equilibrium is one in which some perturbation exists that causes it to head away from that equilibrium. In the example of the pencil, laying flat on its side is a stable equilibrium, while balancing on its tip is unstable. The key fact to remember about unstable equilibria is that they have a tendency to not stick around. No real system is perfectly balanced, and the imperfection will grow over time until equilibrium disappears on its own. This is is why we run into lots of pencils lying on their sides, and none balanced on their tips.

Now what does this have to do with Le Châtelier's principle? Well if we run across a system that has settled down to the point that it has a status quo we can notice, that system is extremely likely to be at some sort of equilibrium. Based on the point I made above, we can be pretty sure that it is a stable equilibrium. But Le Châtelier's principle is just a description of what it means to be at a stable equilibrium, so Le Châtelier's principle must be true of our system!

Now I should note that there is a world of difference between a local equilibrium and a global one. The pencil that is flat on the table, would be at an equal equilibrium on a different side, as a small push demonstrates. And at a better equilibrium flat on the floor, as a strong enough push will demonstrate. A mixture of hydrogen and oxygen that is heated at little bit will heat up, which increases the pressure, which causes the container it is in to expand, which cools it down, in accord with Le Châtelier's principle. But heat the same mixture enough and a chemical reaction will begin that results in it becoming hotter still, rather than cooler. (These examples are why the overly general formulation I quoted at the top is not perfectly accurate.)

It is clear that this is a very general argument, and makes it clear that the principle has nothing really to do with chemistry per se. In fact it applies to any sort of equilibrium. In chemistry or not. On the whole I find the examples outside of chemistry to be more interesting.

A case in point appears in the classic economics paper Cars, Cholera, and Cows: The Management of Risk and Uncertainty. One of the key themes is that our risk-taking behavior as a society winds up in an equilibrium. If it is at an equilibrium, then we should expect that any change which reduces risk will cause a some sort of compensatory increase in risk. Which will undo some of the positive benefits of the change. An example from that paper is that seat belt laws increase seat belt usage. But people using seat belts feel safer, and therefore drive more aggressively, resulting in more accidents. The net result? It appears that drivers are safer, pedestrians are less safe, and benefits to society are less clear than a naive analysis would predict.

Over time I've learned that equilibria are extremely common, and therefore the "unexpected consequence" is more often something to try to anticipate than something to be surprised at. For instance when a faster variation of cheetahs is bred by evolution, it creates pressure for antelope to become some combination of faster, better at spotting predators, and willing to bolt when predators are farther away. The net result is that the more effective predators wind up about equal versus their prey. (Look up the Red Queen Hypothesis for more on this.)

But how do you anticipate the unexpected? Well there are several ways.

If you know that it is common to find some sort of push-back, you know to be on the lookout for it, which will make it easier to spot. For instance suppose that you start injecting a stable compound in at a constant rate. Eventually it has to break down at the same rate, the only question is where. It was not until Sherry Rowland and Mario Molina applied this line of reasoning to CFCs that it was realized that one of the most innocuous and inert chemicals discovered by man was destroying the ozone layer.

There will be a set of related push-back phenomena associated with any stable equilibrium that you can find. However equilibria are very common. So look for potential equilibria, then actively ask how they are maintained. If you find them, you frequently learn something useful. For instance suppose there is an equilibrium level of major disasters in given area of human endeavor. By what means is this level maintained? I submit that it is maintained by memories of previous disaster, and desire to not experience that again. Which means that once memory fails and people become less careful, corners will be cut until disaster happens again. However memory fails on a time scale set by human lives, which tells me that an equilibrium rate for major disasters of any particular kind is never going to be more than a small number of human generations, no matter what the engineers promise us. For example it took just over 60 years to lose the regulations that were put in place to prevent another credit crisis like the one that started the Great Depression (and about 10 more years after that to experience a credit crisis - note that before the Depression credit crises arrived on average about once every 10 years), nuclear options are now getting a boost from the fact that memories of Three Mile Island are now fading, and I am willing to bet serious money that the surprisingly good record of safety devices for offshore drilling helped result in dangerous shortcuts that were key to causing the recent BP disaster.

Certain equilibria and their corresponding compensation mechanisms come up very frequently to explain otherwise puzzling events. My favorite example is that people like to maintain a positive self-impression. The result is that anything that challenges our good opinion of ourselves causes serious cognitive dissonance. People do the most amazing things to avoid this cognitive dissonance. For some of the negative results on people's ability to learn, see What you refuse to see, is your worst trap.

So, even if you're not a chemist, if you squint at the world in the right way you can see Le Châtelier's principle popping up in the most unexpected and interesting places.

Random grab bag

2010-04-18T20:49:00.000-07:00

I can't believe that it has been nearly a month since my last post.

If you've been following me on buzz you'll know that I've been doing a joke a day. I started Feb 18, so now I've gone for 2 months, with a new joke each day, without once having to look one up. I'm surprised by this. Today's joke had a musical component that needed to be heard, so it is on youtube.

Stories like this make me sad. Nothing will compensate that man's loss or replace what was stolen, but the county deserves to lose that case.

Moving on, on Hacker News a while ago I tried to explain why quantum mechanics and the general theory of relativity conflict with each other. That may be of interest. I sent that link to a well-respected researcher, John Baez, and he thought the explanation was reasonable in so far as it went. Then he recommended the far more detailed explanation at http://arxiv.org/abs/gr-qc/9310031. (Look for the Download box at the upper right.) Having read it, that explanation is far longer, but gives much more depth than mine. Be warned that my reaction was, "Now that I've read it I feel less educated on the subject than before. Not because I didn't learn anything, but because I was reminded of how much physics I never learned... I was reassured to learn that C.J. Isham has that effect on many physicists as well.

What else did I mean to talk about and didn't? Oh right. I read The Back of the Napkin: Solving Problems and Selling Ideas with Pictures. On the whole it was a very good book, but it had a spectacularly poorly chosen example at the end. The example was a hypothetical firm selling accounting software named SAX Inc. Despite being the market leader its revenue was not growing as the market expanded, and some low end open source platforms were growing rapidly. Economic projections showed that it was poised to lose huge amounts of market share to these new competitors in the near future, and the question is what it should do.

Now if you're ever in a situation that looks vaguely like this, I've got a recommend. Reach for The Innovator's Solution, read it, understand it, get every executive in your company to understand it, and follow its advice as best you can. Because you're on the wrong side of an disruptive innovation, so you want to see what has been tried in that situation, what worked and not (mostly not), and need to understand the organizational reasons for that.

If you don't know what a disruptive innovation is, that is the scenario where an established market with established companies has competition from a new kind of product which is not good enough for the market, but which will become so with projected technological improvement. In this situation what happens is that over time technology improves, the ecosystem of companies that grew up with the (initially) crappy technology take over, and the established companies see their market implode. For a fuller explanation and lots of examples, read The Innovator's Dilemma. That gives the theory, and then the follow-up, The Innovator's Solution, gives a lot more detail on why companies repeatedly make the same bad choices.

Suffice it to say that The Back of the Napkin comes up with a solution, and it is a solution that I guarantee will fail. And as part of the reasoning they manage to come up with a software development project that is much more ambitious than anything that the company has tried before, and gave some suspiciously precise estimates of how much time and money it will take. If you don't know what is wrong with that, go read Software Estimation by Steve McConnell. (If you are responsible for anything to do with scheduling software development, I highly recommend that book on general principle. Steve McConnell's books on software development range from good to classic, and that is on the higher end of the scale.)

None of this is to say that The Back of the Napkin is a bad book. It is not. Indeed if it were then I'd be less pained by the bad example. But it had the opportunity to explain a bunch of very important things to an audience that normally doesn't hear those things, and didn't. Not only did it fail to explain them, it proceeded to actively misinform.

On a happier note I've also read Accelerando by Charles Strauss. I have to say, without hesitation, that it is the strangest book I've read. Let me give an example. Near the beginning some lobsters that got uploaded managed to escape over the Internet, take over a computer network, and turned themselves sentient. They ask the protagonist for help in getting away from human civilization because they don't want to be near us when technology goes critical. He succeeds. Then the book gets weird.

Don't get me wrong. It is a very good book, and I enjoyed it very much. But the whole point of the book is that technology is accelerating, and the result will be continuous future shock. The first few iterations you are lead into trying to understand how the future rapidly got more bizarre. In later iterations you're just following the personal story of humans with unbelievable technology trying to survive in a world where they are obsolete and unequipped to understand the universe around them.

Touching Women

2010-03-20T15:03:00.000-07:00

Today I want to share two useful tidbits about touch and women that I think should be better known, but aren't because people get embarrassed to talk about this stuff.

The first is a pressure point to help menstrual cramps. Everyone knows about pinching next to the thumb to help with headaches. It doesn't take the pain away, but it dulls it and makes it more bearable. There is a spot that does about the same thing with menstrual cramps.

It is located just above your foot, between your tendon and your ankle. To get it properly you want to use a "fat pinch". You get this by folding your index finger over, putting that on one side of the ankle, and pinching with the thumb on the other. So you get a pinch spread out over the soft flesh between the bone and Achilles tendon. I've offered this advice to multiple women who suffer menstrual cramps. None have ever heard it before, but it has invariably helped.

The other is more *ahem* intimate. This would be a good time to stop reading if that bothers you.

There are various parts of your body where you have a lot of exposed nerves. A light brushing motion over them will set up a tingling/itching sensation. A good place to experience this is the palm of your hand. Gently stroke towards the wrist, then pay attention to how your hand feels. Yes, that. And thinking about it brings it back.

This happens anywhere where nerves get exposed. One place where that reliably happens is the inside of any joint. For instance the inside of your elbow. (Not as much is exposed there as the palm of the hand, but it is still exposed.)

The larger the joint, the more nerves, the more this effect exists. The largest joint, of course, is the hip. And the corresponding sensitive area is the crease between leg and crotch on each side. This works on both genders. But for various reasons is more interesting for women...

Enjoy. ;-)

Address emotions in your forms

2010-03-17T23:34:00.001-07:00

I learned quite a few things at SXSW. Many are interesting but potentially useless, such as how unexpectedly interesting the reviews for Tuscan Whole Milk, 1 Gallon, 128 fl oz are.

However the one that I found most fascinating, and is relevant to a lot of people, was from the panel that I was on. Kevin Hale, the CEO of Wufoo gave an example from their support form. In the process of trying to fill out a ticket you have the option of reporting your emotional state. Which can be anything from "Excited" to "Angry". This seems to be a very odd thing to do.

They did this to see whether they could get some useful tracking data which could be used to more directly address their corporate goal of making users happy. They found they could. But, very interestingly, they had an unexpected benefit. People who were asked their emotional state proceeded to calm down, write less emotional tickets, and then the support calls went more smoothly. Asking about emotional state, which has absolutely no functional impact on the operation of the website, is a social lubricant of immense value in customer support.

Does your website ask about people's emotional state? Should it? In what other ways do we address the technical interaction and forget about the emotions of the humans involved, to the detriment of everyone?

Serendipity at SXSW

2010-03-17T08:37:00.001-07:00

This year I had the great fortune to be asked to be on a panel at SXSW. It was amazingly fun. However there was only one person I had ever met in person at the conference this year. So I was swimming in a sea of strangers.

But apparently there were a lot of people that I was tangentially connected to in some way.

I was commenting to one of my co-panelists, Victoria Ransom that a previous co-worker of mine looked somewhat similar to her, had a similar accent, and also had a Harvard MBA. Victoria correctly guessed the person I was talking about and had known her for longer than I had.

I was at the Google booth, relating an anecdote about a PDF that I had had trouble reading on a website, when I realized that the person from Australia who had uploaded said PDF was standing right there.

Another person had worked with the identical twin brother of Ian Siegel. Ian has been my boss for most of the last 7 years. (At 2 different companies.)

One of the last people I met was a fellow Google employee whose brother in law was Mark-Jason Dominus. I've known Mark through the Perl community for about a decade.

And these are just the people that I met and talked with long enough to find out how I was connected to them.

Other useful takeaways? Dan Roam is worth reading. Emergen-C before bed helps prevent hangovers. Kick-Ass is hilarious, you should catch it when it comes out next month. And if you're in the USA then Hubble 3D is coming to an IMAX near you this Friday. You want to see it. I'll be taking my kids.

And advice for anyone going to SXSW next year? Talk to everyone. If you're standing in line for a movie, talk to the random stranger behind you. Everyone is there to meet people. Most of the people there are interesting. You never know. Talking to the stranger behind you in line might lead to meeting an astronaut. (Yes, this happened to me.)

Rogue Waves

2010-03-08T22:29:00.000-08:00

Today I ran across an interesting essay on our changing understanding of scurvy. As often happens when you learn history better, the simple narratives turn out to be wrong. And you get strange things where as science progressed it discovered a good cure for scurvy, they lost the cure, they proved that their understanding was wrong, then wound up unable to provide any protection from the disease, and only accidentally eventually learned the real cause. The question was asked about how much else science has wrong.

This will be a shorter version of a cautionary tale about science getting things wrong. I thought of it because of a a hilarious comedy routine I saw today. (If you should stop reading here, do yourself a favor and watch that for 2 minutes. I guarantee laughter.) That is based on a major 1991 oil spill. There is no proof, but one possibility for the cause of that accident was a rogue wave. (Rogue waves are also called freak waves.) If so then, comedy notwithstanding, the ship owners could in no way be blamed for the ship falling apart. Because the best science of the day said that such waves were impossible.

Here is some background on that. The details of ocean waves are very complex. However if you look at the ratio between the height of waves and the average height of waves around it you get something very close to a Rayleigh distribution, which is what would be predicted based on a Gaussian random model. And indeed if you were patient enough to sit somewhere in the ocean and record waves for a month, the odds are good that you'd find a nice fit with theory. There was a lot of evidence in support of this theory. It was accepted science.

There were stories of bigger waves. Much bigger waves. There were strange disasters. But science discounted them all until New Years Day, 1995. That is when the Draupner platform recorded a wave that should only happen once in 10,000 years. Then in case there was any doubt that something odd was going on, later that year the RMS Queen Elizabeth II encountered another "impossible" wave.

Remember what I said about a month of data providing a good fit to theory? Well Julian Wolfram carried out the same experiment for 4 years. He found that the model fit observations for all but 24 waves. About once every other month there was a wave that was bigger than theory predicted. A lot bigger. If you got one that was 3x the sea height in a 5 foot sea, that was weird but not a problem. If it happened in a 30 foot sea, you had a monster previously thought to be impossible. One that would hit with many times the force that any ship was built to withstand. A wall of water that could easily sink ships.

Once the possibility was discovered, it was not hard to look through records of shipwrecks and damage to see that it had happened. When this was done it was quickly discovered that huge waves appeared to be much more common in areas where wind and wave travel opposite to an ocean current. This data had been littering insurance records and ship yards for decades. But until scientists saw direct proof that such large waves existed, it was discounted.

Unfortunately there were soon reports such as The Bremen and the Caledonian Star of rogue waves that didn't fit this simple theory. Then satellite observations of the open ocean over 3 weeks found about a dozen deadly giants in the open ocean. There was proof that rogue waves could happen anywhere.

Now the question of how rogue waves can form is an active research topic. Multiple possibilities are known, including things from reflections of wave focusing to the Nonlinear Schrödinger equation. While we know a lot more about them, we know we don't know the whole story. But now we know that we must design ships to handle this.

This leads to the question of how bad a 90 foot rogue wave is. Well it turns out that typical storm waves exert about 6 tonnes of pressure per square meter. Ships were designed to handle 15 tonnes of pressure per square meter without damage, and perhaps twice that with denting, etc. But due to their size and shape, rogue waves can hit with about 100 tonnes of pressure per square meter. Are you surprised that a major oil tanker could see its front fall off?

If you want to see what one looks like, see this video.

Fun with Large Numbers

2010-03-01T09:02:00.001-08:00

I haven't been blogging much. In part that is because I've been using buzz instead. (Mostly to tell a joke a day.) However I've got a topic of interest to blog about this time. Namely large numbers.

Be warned. If thinking about how big numbers like 9^{9^9⁹} really are hurts your head, you may not want to read on.

It isn't hard to find lots of interesting discussion of large numbers. See Who can name the bigger number? for an example. However when math people go for big numbers they tend to go for things like the Busy Beaver problem. However there are a lot of epistemological issues involved with that, for instance there is a school of mathematical philosophy called constructivism which denies that the Busy Beaver problem is well-formulated or that that sequence is well-defined. I may discuss mathematical philosophy at some future point, but that is definitely for another day.

So I will stick to something simpler. Many years ago in sci.math we had a discussion that saw several of us attempt to produce the largest number we could following a few simple ground rules. The rules were that we could use the symbols 0 through 9, variables, functions (using f(x, y) notation), +, *, the logical operators & (and), ^ (or) ! (not), and => (implies). All numbers are non-negative integers. The goal was to use at most 100 non-whitespace characters and finish off with Z = (the biggest number we can put here). (A computer science person might note that line endings express syntactic intent and should be counted. We did not so count.)

A non-mathematician's first approach would likely be to write down Z = 999...9 for a 98 digit number. Of course 9^{9^9⁹} is much larger - you would need an 8 digit number just to write out how many digits it has. But unfortunately we have not defined exponentiation. However that is easily fixed:


p(n,0) = 1
p(n, m+1) = n * p(n, m)

We now have used up 25 characters and have enough room to pile up a tower of exponents 6 deep.

Of course you can do better than that. Anyone with a CS background will start looking for the Ackermann function.


A(0, n) = n+1
A(m+1, 0) = A(m, 1)
A(m+1, n+1) = A(m, A(m+1, n))

That's 49 characters. Incidentally there are many variants of the Ackermann function out there. This one is sometimes called the Ackermann–Péter function in the interest of pedantry. But it was actually first written down by Raphael M. Robinson.

(A random note. When mathematicians define rapidly recursing functions they often deliberately pick ones with rules involving +1, -1. This is not done out of some desire to get a lot done with a little. It is done so that they can try to understand the pattern of recursion without being distracted by overly rapid initial growth.)

However the one thing that all variants on the Ackermann function share is an insane growth rate. Don't let the little +1s fool you - what really matters to growth is the pattern of recursion, and this function has that in spades. As it recurses into itself, its growth keeps on speeding up. Here is its growth pattern for small n. (The n+3/-3 meme makes the general form easier to recognize.)


A(1, n) = 2 + (n+3) - 3
A(2, n) = 2 * (n+3) - 3
A(3, n) = 2ⁿ⁺³ - 3
A(4, n) = 2^{2^…²} - 3 (the tower is n+3 high)

There is no straightforward way to describe A(5, n). Basically it takes the stacked exponent that came up with A(4, n) and iterates that operation n+3 times. Then subtract 3. Which is the starting point for the next term. And so on.

By most people's standards, A(9, 9) would be a large number. We've got about 50 characters left to express something large with this function. :-)

It is worth noting that historically the importance of the Ackermann function was not just to make people's heads hurt, but to demonstrate that there are functions that can be expressed with recursion that grow too quickly to fall into a simpler class of primitive recursive functions. In CS terms you can't express the Ackermann function with just nested loops with variable iteration counts. You need a while loop, recursion, goto, or some other more flexible programming construct to generate it.

Of course with that many characters to work with, we can't be expected to be satisfied with the paltry Ackermann function. No, no, no. We're much more clever than that! But getting to our next entry takes some background.

Let us forget the rules of the contest so far, and try to dream up a function that in some way generalizes the Ackermann function's approach to iteration. Except we'll use more variables to express ever more intense levels of recursion. Let's use an unbounded number of variables. I will call the function D for Dream function because we're just dreaming at this point. Let's give it these properties:


D(b, 0, ...) = b + 1

D(b, a₀ + 1, a₁, a₂, ..., a_n, 0, ...)
    = D(D(b, a₀, a₁, ..., a_n, 0, ...), a₀, a₁, ..., a_n, 0, ...)

D(b, 0, ..., 0, a_i+1, a_i+1, a_i+2, ..., a_n, 0, ...)
    = D(
        D(b, a₀, a₁, ..., a_n, 0, ...),
        b-1, b-1, ..., b-1,
        a_i, a_i+1, ..., a_n, 0, ...
      )

There is a reason for some of the odd details of this dream. You'll soon see b and b-1 come into things. But for now notice that the pattern with a₀ and a₁ is somewhat similar to m and n in the Ackermann function. Details differ, but recursive patterns similar to ones that crop up in the Ackermann function crop up here.


D(b, a₀, 0, ...) = b+a₀+1
D(b, a₀, 1, 0, ...) ≈ 3^{2^a₀} b

And if a₁ is 2, then you get something like a stacked tower of exponentials (going 2,3,2,3,... with some complex junk). And you continue on through various such growth patterns.

But then we hit D(b, a₀, a₁, 1, 0, ...). That is kind of like calling the Ackermann function to decide how many times we will iterate calling the Ackermann function against itself. In the mathematical literature this process is called diagonalization. And it grows much, much faster than the Ackermann function. With each increment of a₂ we grow much faster. And each higher variable folds in on itself to speed up even more. The result is that we get a crazy hierarchy of insane growth functions that grow much, much, much faster. Don't bother thinking too hard about how much faster, our brains aren't wired to really appreciate it.

Now we've dreamed up an insanely fast function, but isn't it too bad that we need an unbounded number of variables to write this down? Well actually, if we are clever, we don't. Suppose that b is greater than a₀, a₁, ..., a_n. Then we can represent that whole set of variables with a single number, namely m = a₀ + a₁ b + ... + a_n bⁿ. Our dream function can be recognized to be the result of calculating D(b, m+1) by subtracting the then replacing the base with D(b, m) (but leaving all of the coefficients alone. So this explains why I introduced b, and all of the details about the -1s in the dream function I wrote.

Now can we encode this using addition, multiplication, non-negative integers, functions and logic? With some minor trickiness we can write the base rewriting operation:


B(b, c, 0) = 0
i < b => B(b, c, i + j*b) = i + B(b, c, j) * c

Since all numbers are non-negative integers the second rule leads to an unambiguous result. The first and second rules can both apply when the third argument is 0, but that is OK since they lead to the same answer. And so far we've used 40 symbols (remember that => counts as 1 in our special rules).

This leads us to be able to finish off defining our dream function with:


D(b, 0) = b + 1
D(b, n+1) = D(D(b, n), B(b, D(b, n), n))

This took another 42 characters.

This leaves us 18 characters left, two of which have to be Z=. So we get


Z = D(2, D(2, D(2, 9)))

So our next entry is


B(b, c, 0) = 0
i < b => B(b, c, i + j*b) = i + B(b, c, j) * c
D(b, 0) = b + 1
D(b, n+1) = D(D(b, n), B(b, D(b, n), n))
Z = D(2, D(2, D(2, 9)))

We're nearly done. The only thing I know to improve is one minor tweak:


B(b, c, 0) = 0
i < b => B(b, c, i + j*b) = i + B(b, c, j) * c
T(b, 0) = b * b
T(b, n+1) = T(T(b, n), B(b, T(b, n), n))
Z = T(2, T(2, T(2, 9)))

Here I changed B into T, and made the 0 case be something that had some growth. This starts us off with the slowest growth being T(b, i) being around b^2ⁱ, and then everything else gets sped up from there. This is a trivial improvement in overall growth - adding a couple more to the second parameter would be a much bigger win. But if you're looking for largest, every small bit helps. And modulo a minor reformatting and a slight change in the counting, this is where the conversation ended.

Is this the end of our ability to discuss large numbers? Of course not. As impressive as the function that I provided may be, there are other functions that grow faster. For instance consider Goodstein's function. All of the growth patterns in the function that I described are realized there before you get to b^b. In a very real sense the growth of that function is as far beyond the one that I described as the one that I described is beyond the Ackermann function.

If anyone is still reading and wants to learn more about attempts by mathematicians to discuss large (but finite) numbers in a useful way, I recommend Large Numbers at MRROB.

Developing on HEAD scales to Google

2010-02-05T08:25:00.000-08:00

I have a simple rule of thumb for what I am and am not allowed to say about Google. If I can find it said in some official-looking place from Google, then I think I'm allowed to say it. Otherwise not.

I was therefore very glad to run across this talk by Guido Van Rossum describing his code review tool named Mondrian, put on youtube by Google Tech Talks, and judging from the knowledge level of the audience, delivered to a non-Google audience. Confirming that this was not, in fact, accidentally open, I found an O'Reilly article confirming that it was public.

I therefore feel comfortable talking about anything and everything said in that video. So you can view this as my spin on the material in that video, and if you want the full version without the editorializing, feel free to take an hour and watch the video for the original source.

These days everyone who is competent has agreed that source control is a Good Thing to use. However opinions on how to use it vary widely. One argument in particular that I've seen in multiple organizations is on the value of using multiple branches versus having everyone develop in one branch, which is often called something like MAIN or HEAD. The issues involved are somewhat complex, and not obvious. So let me give a quick overview.

The primary argument for branching is that people can work on different features on different schedules without getting in each other's way. The primary argument for all developing on the same branch is that you discover conflicts immediately when they are easiest to resolve, rather than months later when people have moved on to different things and have lost context for the pieces of code that conflicts. The primary argument against branching is the pain of merging later. (Granted, distributed version control systems like git have done a lot to reduce this pain. But they have not eliminated it.) The primary argument against developing on HEAD is that it requires a constant level of diligence on the part of developers. When any developer can break all developers, you need to be careful in what you check in.

I've just kept this down to the primary arguments because the secondary back and forth arguments get long, involved, and heated. Also what I've described as two ways of working is really two families of working. There are a lot of ways to organizing multiple branches. And there are quite a few useful uses for branches in a software project even if everyone is developing on the same branch. None of this is made easier by the fact that (as with many religious wars) the people involved have imprinted on one way of doing things. That makes them hyper aware of the problems in other approaches, and they don't even notice potential pain points in their own.

Until I came to Google my personal position was that everyone working on HEAD was the best approach as long as your team was small enough that you could make it work. And I vaguely accepted that the pain of branching was a necessary evil on large software projects. Even when the pain reached the point of craziness, I was mostly willing to accept unsupported claims that it is necessary.

Then I came to Google. There are a lot of groups at Google, and they are free to do their own thing. Many do. However most groups develop out of one giant code base on HEAD. And it works as a development model. Google has made it scale.

Guido's talk describes many elements of what makes it scale. The first piece is having good developers. The second piece is an enforced policy of having every single patch go through a code review process before you check anything in. The third piece is a lot of proprietary infrastructure that Google has built up to make things work. And beyond that you have people paying attention to best practices such as consistent style, good unit testing, so on and so forth. (All of which are reinforced in the code review process.)

My opinion after seeing Google has changed. I freely admit that there are real process and tool challenges to making it possible for large teams to develop everything on HEAD and have it scale smoothly. However it is possible. I've seen it work. And, speaking personally, this is my preferred way to work.

Different organizations are different, have different capabilities, different needs, and different goals. Sight unseen I'm not going to tell anyone else that their organization should try to work like Google does. I simply don't have the facts about your situation. But if, like me not that long ago, you've accepted the claim that large development teams have no choice, you now know better.

Now admittedly most people can't go and see this first hand for themselves at Google. But if you want to watch a large successful project developing code to a high standard in this way, I recommend watching clang. And if you want to know more about what kinds of tool support you need to make this work, go listen to Guido. He's smarter than I am, has been doing it longer, and actually built some of the basic tool support for it.

What is intelligence?

2010-02-01T21:55:00.000-08:00

Last September I posted on why hard sciences are different from soft ones. The subject of this blog post is a perfect illustration of the critical difference. In the hard sciences people mostly agree on basic definitions, broad agreement exists on key problems to solve, and in general there is agreement on a basic paradigm. Through no fault of their own, in the soft sciences, none of this exists.

Few things illustrate this better than getting a handle on what intelligence is. Here we have a subject that has been seriously studied for over a century. Yet researchers have not achieved agreement on whether they should focus on a single measure of intelligence called g, or multiple measures, or if multiple measures then how many there should be and how they should be divided. Note here that if you talk to individual researchers this fact does not worry them, because each researcher works in a community of other researchers who have achieved agreement on this. Each school of thought therefore churns out a stream of research, but they have failed to convert each other. And after a century the debate shows no sign of ending any time soon.

Please note that I am far from expert on any of this, so I'll try to summarize in broad terms my understanding, but my understanding could be incorrect. Also be aware that a lot of experts disagree with each other strongly, and therefore it is easy to find experts who will disagree with anything interesting that I can say. Please keep those caveats in mind as I try to flounder my way through the complications.

Let's start with the simplest approach, namely Spearman's g. What Spearman did in 1904 is take a number of things that one would think correlate to intelligence, such as grades, and found that they seemed correlated well with each other. He then looked for some linear combination which correlated better with each of the initial factors than any initial factor did. He found a very combination that did so, and found that it successfully predicted a very large part of the variation. In modern mathematical terms he did something called a Principal Component Analysis and looked for the most significant component. He then found that the most significant component really did capture a lot of the variability.

Spearman called this factor g for general intelligence.

Not long after the IQ test was invented. Before long people added IQ tests to the mix of factors, and found that a well-designed IQ test served as a fairly good predictor of g. Then g in turn served as a decent predictor for many things, including academic performance, and success in the workplace. Since IQ tests were fast and easy to administer, IQ tests and ability tests were on. (Ability tests such as the SAT and GRE exams actually are fairly directly comparable to IQ tests, and high-IQ societies such as MENSA are happy to accept them as proof that your IQ is high enough to belong.)

All of this begs many important questions. We got g out of a mathematical construct. But is it real? Is there really such a thing as general intelligence? Also even if there is such a thing as general intelligence, is it a stable attribute of a person? Does your general intelligence change over time? And the whole principal components analysis admits of the discovery of multiple components, how important are the lesser components? (The existence of a split between mathematical and verbal ability is obvious enough that most ability tests split those into separate sections.) Are there important factors which have been missed? Also how much of this is affected by heredity versus environment? How does race play into this? Can training change how you score on the test? Can it change your actual intelligence?

Believe you me, those questions have been researched. Extensively. And argued. More extensively. The only ones that there seems to be a decent consensus on is that to the extent that general intelligence makes sense, it seems to be fairly stable, and both hereditary and environmental factors affect it. Every other one is still under debate, though it is easy to find experts that make definitive claims that the matter has been completely settled one way or the other.

As one example of the ongoing debate, ETS, which puts out various tests including the SAT exams, claims that training cannot improve your performance. At the same time Kaplan makes money selling the claim that they can significantly improve your performance. Kaplan has enough evidence of effectiveness that I accept that they are right, but the debate continues. I suspect not the least because ability tests are just IQ tests under a different name, and so acceptance that Kaplan can teach an ability test is evidence that a key tool underlying a lot of research into psychology is fundamentally flawed, which opens up a can of worms that many don't want to look into.

For another example, I strongly suggest that mathematically inclined readers read g a Statistical Myth. It presents a toy statistical model which demonstrates how readily a large number of independent random factors with a small amount of interaction could give rise to all of the statistical evidence which underlies claims that such a thing as general intelligence exists. Given the fact that any hereditary effect on intelligence is the result of a large number of fairly independent random factors called genes, I think the point is an important one. The idea that we each have a huge number of different intellectual abilities at different degrees of competence accords very well with my understanding of how the brain works, and my experience of interacting with others.

However it is also true that a small number of factors have a major effect on many different kinds of mental activities. Thus some combination of the amount of things you can keep in your mind at the same time and speed of reasoning may be influential enough on enough things to deserve the name "general intelligence".

At this point I'd say that the jury is not just out; they are deadlocked and randomly fighting each other. And have been in this state for the better part of a century.

So what do I have to contribute to the discussion? I'd like to throw out a toy mathematical model for how IQ and intelligence relate. Please do not take this too seriously. After all it is not even agreed on at this point that general intelligence is a particularly well-defined concept. But what I'd like to show is that even if you do accept that general intelligence exists, and that IQ is reasonably well correlated with it, that IQ tests are not a particularly effective way to find the smartest people. (Whether or not any better way exists is a question I'd like to avoid. In fact, please forget this parenthetical point.)

Now suppose you sit down to take a test. You come prepared with many relevant abilities. You have your vocabulary, your polished reasoning ability, and a body of basic knowledge you're expected to have acquired about the world. We are generally prepared to accept these as components of intelligence, and the test will try to measure those. Let's call that INT. However you come prepared with other relevant abilities. There is how well you are feeling, how much sleep you got, any tricks you know about how to handle the types of questions that are likely to be asked, cultural cues you share with the test makers and how stressed you are. We are generally not prepared to accept these as components of intelligence, yet they affect how we do. Let us call these TST.

For my toy model let's say that INT and TST are independent random normals with a mean of 0 and a standard deviation of 10. That's math talk describing a particular bell curve. If you want the odds of it being less than a particular number you just have to divide the number by 10 to get the number of standard deviations you're out, and then look at a standard table and read off the probability.

Now let's say that the measured IQ will be 100+INT+TST. If you're a mathematician you'll know that IQ will have a normal distribution with mean 100 and a standard deviation of 10*sqrt(2) which is about 14.14. The correlation between IQ and INT turns out to be 70.7%. By comparison the Stanford-Binet is designed to have a mean of 100 and a standard deviation of 16, and its correlation with other measures of intelligence like academic performance is (depending on who measures it and how it is done) in the range of 70-80%. So while the model is pretty simple, it is not too far off from how a real IQ test behaves. And it captures the obvious observation that there are factors that affect test performance which we don't think of as intelligence.

Now the fun thing about a model like this is that we can play with it since we know exactly how it is set up. And are free to add any internal details we want. And while we can't necessarily believe the answers we get, we can gain valuable intuition about what kinds of answers we can expect to see if we understood a far more complex system. Such as real people with real brains.

To analyze this I'm going to complicate the model one more time by introducing two more random variables. Let's make them independent normal variables with mean 0 and standard deviation 10/sqrt(2), and call them AVG and DIF. It turns out that (AVG+DIF) and (AVG-DIF) are independent normal variables with mean 0 and standard deviation 10. So let's make (AVG+DIF) be INT, and (AVG-DIF) be TST. High-school algebra shows that AVG = (INT+TST)/2 and DIF = (INT-TST)/2. (Hence the names.)

What was the point of that? Well it is this. At this point IQ = 100+2*AVG. And INT = AVG+DIF. Which means that if you give me an IQ, I can calculate AVG then use the known distribution of DIF to calculate confidence intervals on INT.

For example, in this model how intelligent is a typical member of MENSA? Well MENSA accepts anyone whose IQ is in the top 2% of the population. So a typical member might have an IQ at the boundary of the top 1%. (If you haven't taken statistics these calculations may make you eyes gloss over. You'll have to just trust that I'm coming up with correct numbers.) That IQ is about 2.326 standard deviations out, which after multiplying by the 10*sqrt(2) and adding 100 is an IQ of 132.90. (Remember that our standard deviation is lower than the regular IQ test, so our purported IQ scores will be lower than a regular test.) Which means that AVG is 16.45. On average our INT will be the same as the AVG, which puts you 1.645 standard distributions out, which is right around the 95th percentile.

We can go farther and calculate a 95% confidence interval for what the real intelligence of this individual is. 95% of the time a normally distributed variable will be within 1.960 standard deviations of 0, so that means that DIF will be in the range +- 13.86. Which means that INT will be (2.59, 30.31). Which means that INT is anywhere from 0.0259 to 3.031 standard deviations out, which would put INT anywhere from the 51st percentile to almost the 99.9th percentile. Hmmm, we really haven't nailed down INT very closely, have we? A similar calculation shows that with 80% odds INT is between the 77th and 99.5th percentiles. And you can even do fun things like show that with about 78% odds, this individual is not in the top 2% of the population on INT, and therefore would not qualify for MENSA if they were able to measure INT directly instead of the more complicated IQ.

Let's reverse the question. Let's take someone at the boundary of the 99th percentile in INT, and ask what IQ we should expect on average. Well that INT is 2.326 standard deviations out, which is an INT of 16.44, for an IQ of 116.44, which is 1.1625 standard deviations out, which is at the 87.77th percentile. So sorry, you're probably not going to test as smart enough for MENSA even though you're probably smarter than most of the people in MENSA. How likely is this person to fail to meet MENSA's admission standards? Well the cutoff for MENSA is top 2% which is 2.0537 standard deviations out, which is an IQ of 129.04 (again remember that the toy model's IQs are not as spread out as real IQ tests), so we need TST to be 12.6, which is 1.26 standard deviations out, which gives a probability of 89.6% of not qualifying for MENSA.

The upshot? This model predicts that most members of MENSA are not in the top 2% on intelligence, and most people in the top 2% of intelligence would not qualify for membership in MENSA.

Now let's apply this to a question of more personal interest. According to this model, how intelligent am I likely to be? I bring this up because in my last blog post I used my GRE to discuss my likely IQ, and therefore likely intelligence, then used it as a benchmark to compare with the people I've met at Google. So if I believe that high IQs systemically overestimate INT, how badly is mine likely to be overestimated according to my model?

Well my scores were V: 760, Q: 780 and A: 800. Going to a handy IQ calculator I put in 1540 for my GRE V+Q and find that I am 3.625 standard deviations out and only 1/7143 people have an IQ that high. So on my toy model that is an IQ of 151.265... Which implies that AVG is 25.633. So on average you'd expect my INT to be 25.633, which is 2.5633 standard deviations out, which puts me around the 99.5th percentile, or about 1/200 on INT. (Incidentally the average person with an INT that high would fall in the 96% percentile on IQ, and therefore would not qualify for MENSA.) But let's go farther. 50% of the time a normal distribution falls within 0.67449 standard distributions of the mean, which means that DIF is half the time within the range -4.769 to 4.769. That makes a 50% confidence interval for my INT be the range (20.864, 30.402). Which would put me anywhere from the 98th percentile to the 99.9th percentile in INT. Which is in line with my claim in the previous blog post that I wouldn't be entirely surprised to find that I am at the 99.9th percentile in intelligence. According to this model the odds are that I'm not that smart, but it is possible.

Interestingly there is a symmetry in this model which means that all calculations that I've done for INT apply for TST as well. Which means that I am likely to have a number of characteristics that make me good at taking tests but which are not reflected in my general intelligence. This is actually true.

The vast majority of people when they are faced with a major test or interview get stressed. Now what does stress do to you? Well stress is caused by adrenaline. Adrenaline prepares you for a flight-or-fight reaction. This means decreased blood flow to things like your stomach and neocortex (which is where most of your higher reasoning happens), and increased blood flow to areas that are likely to see action, like your muscles. The result? Improved vision, hearing, faster reflexes, and increased strength. Also significant damage to your ability to carry out complex cognitive tasks. Needless to say this is about the worst possible physiological response to sitting in place for several hours and completing a series of complex cognitive tasks.

Apparently evolution didn't anticipate that millions of us would have our life courses depend on how we do on multiple choice exams. :-)

Very oddly, many members of my family, including me, have the exact opposite response. When I walk into a test the thought process I have is basically, "Well this is it. I've prepared all I'm going to, and now I've got a multiple-guess test which will overstate my abilities. Let's see how I do." And I relax. Comparing me to a normal person based on the resulting test score is like starting with two runners, taking one out back and beating on him for a while, then expecting them to run a fair race. No matter how objectively you measure the result, I have an unfair advantage.

Random note. I originally intended to just present the toy model and use it to provide mathematical support for my impression that a really high IQ score is decent evidence of intelligence, but the true intelligence of most intelligent people isn't actually reflected in their IQ. But to sanity test the model I needed to find the estimated correlation between a standard IQ test and g, the general intelligence factor. And reading up on that I realized how much more complicated the subject was than I realized. I also found myself convinced by g a Statistical Myth that g was probably a statistical artifact with no real meaning. Which is how this turned into 2 long and only tangentially related blog posts mushed into 1. I apologize for the length. I didn't intend it to turn out this way, but I hope it was an interesting read.

Things I've learned at Google so far

2010-01-31T13:45:00.001-08:00

Well I've been an employee at Google for about a month. So this seems like as good a place to reflect for a moment.

The first thing that I've learned is that internally Google is incredibly open, but externally there is a lot we can't say. I understand and support a lot of the reasons why it is so, but it can be frustrating. There is a lot of really cool technology at Google that people never hear about. The statistics of what Google deals with are astounding. The technology we use to deal with it is amazing. The way we scale is unbelievable. (I really wish I could go back and have a few discussions on software development methodology raising points about what has proven to scale at Google...) One random fact that I know I can say is that computations happen in our data centers with about half the power drawn for what is industry standard. I'm not allowed to say how we do it, but it is a rather amazing testimony to what smart people can accomplish when we put our minds to it.

Moving on, what about Google's culture? I would describe Google's culture as "creative chaos". There was some confusion about where I was supposed to be when I started. This resulted in the following phone call, "Hello?", "Hello Ben, this is Conner (that's my new manager), where are you?" "Mountain View." "Why are you there?" "Because this is where the recruiter said to go." "Good answer! Nice of them to tell me. Enjoy your week!" This caused me to ask an experienced Googler, "Is it always this chaotic?" The response I got was, "Yes! Isn't it wonderful?" That response sums up a lot about Google's culture. If you're unable to enjoy that kind of environment, then Google isn't the place for you.

Seriously, the corporate culture is based on hiring really smart people, giving them responsibilities, letting them know what problems the company thinks it should focus on, then letting them figure out how to tackle it. What management hierarchy there is is very flat. And people pay little attention to it unless there is a problem. You are expected to be a self-directed person, who solves problems by reaching out to whomever you need to and talking directly. Usually by email. The result is an organization which is in a constant state of flux as things are changing around you, usually for the better. With a permanent level of chaos and very large volumes of email. It is as if an entire company intuitively understood that defect rates are tied to distance on the corporate org-chart, and tried to solve it by eliminating all barriers to people communicating directly with whoever they need to communicate with. (Incidentally the point about defect rates and org charts is actually true, see Facts and Fallacies of Software Engineering for a citation.)

Speaking of email, working at Google you learn really fast how gmail is meant to be used. If you want to deal with a lot of email in gmail, here is what you need to do. Go into settings and turn keyboard shortcuts on. The ones you'll use a lot are j/k to move through email threads, n to skip to the next message, and the space bar to page through text. And m to hide any active thread that you're not interested in (direct emails to you will still show up). There are other shortcuts, but this is enough to let you skim through a lot of email fairly quickly without touching the mouse too much. Next go into labels and choose to show all labels. Your labels are basically what you'd call folders in another email client. (Unfortunately they are not hierarchical, but they do work.) Next as you get email, you need to be aggressive about deciding what you need to see, versus what is context specific. Anything that is context specific you should add a filter for, that adds a label, and skips the inbox. Nothing is lost, you can get to the emails through the list of labels on the left-hand side of your screen in gmail. But now various kinds of automated emails, lower priority mailing lists, and so on won't distract you from your main email until you go looking for them.

When you combine all of these options with gmail's auto-threading features, it is amazing how much more efficiently you can handle email. In fact this is exactly the problem that gmail was invented to handle. Because this was the problem that Paul Buchheit was trying to solve for himself when he started gmail. It is worth pointing out that Paul Buchheit was a software engineer at Google. He didn't need permission to write something like gmail. Corporate culture says that if you need something like that, you just go ahead and do it. In fact this is enshrined as an official corporate policy - engineers get 20% of their time to do with pretty much as they please, and are judged in part on how they use that time. I found a speech claiming that over half of Google's applications started as a 20% project. (I'm surprised that the figure is so low.) To get a sense of how much stuff people just do, visit Google Labs. No corporate decision. No central planning. People just do things like start putting up solar panels in the parking lot, and the next thing you know Google has one of the largest solar panel installations in the world and has decided to go carbon neutral. And the attitude that this is how you should operate is enshrined as official corporate policy!

You've got to love corporate policies like that. Speaking of nice corporate policies, Google has quite a few surprising ones. For instance they have benefits like heavily subsidized massage on site (I've still got to take my free hour massage for joining), free gym membership, and the like. Or take their attitude on dogs. Policy says that if your immediate co-workers don't object, you can bring your dog to work. Cats are different, however. Nothing against cats, but Google is a dog place and cats wouldn't be comfortable. (Yes, there are lots of dogs around the offices, and I've even seen people randomly wander over to find out if they can borrow someone else's dog for a while.) Hmmm. Sick day policy. Don't show up when you're sick and tell people why you're not showing up. Note what's missing. There is no limit to how much sick time you get if you need it. Oh, and food. Official Google policy is that at all times there shall be good, free food within 150 feet of every Googler. OK, admittedly the food quality does vary. That in Mountain View is better than anywhere else (the larger clientele base lets them have a much more varied selection). But you quickly learn why it is common for new Googlers to put on 15-20 pounds in their first year. (I'm trying to avoid that. We'll see if I succeed...)

But, you say, isn't this crazy? Doesn't it cost a fortune? The answer is that of course it does. But it provides value. People bond over food. Even if you're not bonding, having food close by makes short meals easier. And the temptation to continue working until dinnertime is very real. (Particularly if, as with me, you'd like to wait until rush hour is over before going home.) Obviously no normal CFO would crunch numbers and see things that way. But Google stands behind that decision, and the people who work there treasure the company for it.

Speaking of the people who work there, Google has amazing people. It is often said that engineers find working at Google a humbling experience. This is absolutely true. It took me less than a day to realize that the guy sitting next to me is clearly much smarter than I am, and he's nowhere near the top of the range of talent at Google. In fact, as best as I can tell, I'm pretty much average, though I'm trying hard to hold out a ray of hope that I'm slightly better than average.

Let me put that in context. The closest thing that I have to an estimate for my IQ is scoring 2340 on the GRE exam in 1991. Based on conversions that I've seen, that puts me at about the top 0.01% in IQ. Now I was really "on" that day, happen to believe that there are problems with the measurement of intelligence by an IQ test (a subject which I may devote a future blog post to), but without false modesty I wouldn't be surprised to find that I'm as high as being in the top 0.1% in general intelligence (however that could be measured). Which in most organizations means that I get thought of as being very smart.

However software development is a profession that selects for intelligence. By and large only good software developers bother applying to Google. And Google rejects the vast majority of their applicants. Granted the filtering process is far from perfect, but by the time you get through that many filters, someone like me is just average.

This leads to another point of interest. How astoundingly complex the company is. I believe that organizations naturally evolve until they are as complex as the people in them can handle. Well Google is tackling really big, complex problems, and is full of people who can handle a lot of complexity. The result? I've been told that I should expect that after 2 months I'll only be marginally useful. My initial learning curve should start to smooth out after about 6 months. And every year I should expect half of what I've learned to become obsolete. (Remember what I said about Google having a certain level of permanent chaos? If you're like me, it is exhilarating. But sometimes the line between exhilarating and terrifying can be hard to find...)

Oh, and what else did I learn? That we're hiring more people this year. :-)

On jokes

2010-01-28T21:47:00.000-08:00

When I was in grad school I had the interesting problem of being a socially inept extrovert. Being an extrovert is OK. Being socially inept is survivable. But a socially inept extrovert is going to wind up being embarrassed. Repeatedly.

So I decided to do something about it. I decided that I'd learn to tell jokes. So I began following rec.humor.funny and would regularly inflict what I thought the funniest jokes were on people around me.

At first my delivery sucked and they wouldn't laugh, or would only laugh politely. Over time my delivery improved and people began laughing for real. Eventually I gained enough social skills to realize that telling bad jokes wasn't the best way towards a normal social life improve my social skills, but kept on doing it anyways because it was fun.

So why are jokes funny? Why do we laugh? When I reflect on it, I don't really know the answer. What I do know is that the structure of a joke inevitably sets up some tension, then relieves it in an unexpected way, and then people laugh. If you can't set up the tension, and then release it, people won't laugh. And once people have started laughing, it is easier to keep them laughing.

To see this in a simple example, consider the lowly pun:

Q: What is the difference between bird flu and swine flu?
A: One requires tweetment while the other requires oinkment!

As soon as you hear the question, you know what is coming. And you try to figure it out. Then you hear the answer, see how it connects, and (hopefully) laugh.

Many jokes rely on cultural references, often topical:

Q: What is the difference between Santa Claus and Tiger Woods?
A: Santa stopped at three hos.

Q: Why did the possum cross the road?
A: There was a car coming.

Q: Why are New Yorker's so cynical?
A: The light at the end of the tunnel is New Jersey!

Q: What is the difference between NY and LA?
A: In NY people say, "Fuck you!" and just mean, "Have a nice day!" In LA people say, "Have a nice day!" and mean "Fuck you!"

If I tell those jokes to the right people I am guaranteed a good laugh. If I tell them to the wrong person I'll get a blank look. Unless you're up on current scandals, have lived somewhere where opossums are common road kill, know that area of NJ or are familiar with stereotypes about NY and LA, those jokes won't cause a laugh.

Those jokes actually work with reasonably broad audiences. Some jokes only work with niche audiences. For example this only works with anglophones who know a little French. (I have great luck with Canadians on this one.)

Q: How do Australians say, "You're welcome" in French?
A: (in your best Australian accent) "Pas de probleme, mate!"

What is interesting about this is that I find it is the act of having to translate it that makes it funny. Living in Southern California I've tried to switch to Spanish, but it simply doesn't work. People hear "No problemo, mate!" as English instead of Spanish, so don't find it funny. But the French version forces the mental translation. For different reasons francophones who speak some English don't find it funny for the simple reason that they don't understand the cultural reference about Australians.

Now many people have told me that they can't remember jokes. Let me give a longish joke to illustrate the problems people encounter.

A little girl decided that she wanted to sell toothbrushes. So she went to a manager at a manufacturer and convinced him that, even though she was young, there was no harm in giving her a try. So he gave her her chance.

The first day she showed up, took her sample, went out, and came back. He asked, "How did you do today?" She said, "I thold ONE toothbwuth!" He said, "I'm sorry, but you're going to have to do better than that if this is going to work out."

The second day she showed up, took her sample, went out, and came back. He asked, "How did you do today?" She said, "I thold TWO toothbwutheth!" He said, "I'm sorry, but you're going to have to do a lot better than that if this is going to work out."

The third day she showed up, took her sample, went out, and came back. He asked, "How did you do today?" She said, "I thold THREE THOUTHAND toothbwutheth!" He said, "Three thousand? Wow! How did you do that?" She said, "Well, I went to the mall and offered people chipth and dip. They thaid, 'Thith tathtth like cwap!' I thaid, 'Thith ith cwap, want a toothbwuth?'"

(Incidentally that is one of my wife's favorites.)

Now the first problem people have in remembering a joke like this is that by the time they've heard the end of it, they've forgotten most of it. Memory is an odd thing. We don't realize it, but anything you remember 30 seconds is actually in long term memory. However we've got many levels of long term memory. So just because you've remembered something for 30 seconds doesn't mean that you'll remember it 5 minutes later. If you're going to remember something, you have to work for it. What you're going to need to do is remind yourself on a schedule until it really sticks. My minimal effective schedule is about 30 seconds later, 5 minutes later, again that day, then again the next day, then again sometime in the next week.

And that leads into the second problem. When people hear a joke like this, the most memorable thing is what sticks in your mind. So people remember, "Thith ith cwap, want a toothbwuth?" and then have no referent from which to remember how to start the joke. Or if they start the joke, they keep on jumping directly to the punch line. But it doesn't work without the build-up.

What you've therefore got to do is learn that, "Joke ___ starts with ___". And then you need to practice telling it. And every time you go off track, you've got to back up just slightly, and start telling it, correctly, from before you make the error. This is a subtle and important point that people miss. If you practice telling it, mess up, then start right at the error then you're practicing making the error. And you'll never stop making the error. You've got to practice leading up to the potential error without making the error to get it right. (This is something I learned in choir, but it works for jokes as well as it does for songs.)

And then the third problem is that no matter how many jokes you tell, and how well you tell them, you will forget jokes. I've told hundreds of different jokes. At no point do I remember most of them. Or even most of the good ones. But as long as you have some good ones, that doesn't matter.

Enough on telling jokes, what makes them funny?

As I said, I've noticed that the sudden release of tension is an element. The corollary of this is that it is easier for a joke to be funny when it's about something we have tension about. For instance if you want to make a pre-school child laugh, talk about poop.

Of course grown-ups don't have so much tension around potty training, but we do around sex. Which is why most of us remember and like dirty jokes. Like these.

Micky Mouse and his lawyer were talking. "I'm sorry Mr. Mouse, but the fact that your wife is crazy is not grounds for divorce." "I didn't say she was crazy, I said she was fucking Goofy!"

Q: What is the difference between good, very good, and showing off?
A: Spitting, swallowing, and gargling.

A little girl asked her dad for a GI Joe doll. He said, "Why do you want a GI Joe doll?" "For Barbie to play with." "I thought Barbie came with Ken?" "No, Barbie comes with GI Joe. She fakes it with Ken."

And to finish, here is the dirtiest joke that I am willing to repeat in public:

A businessman was going on a trip. So he decided to get his wife a special present to keep her happy while he was gone. Knowing her tastes, he decided to go to a sex shop.

Looking around the shop, nothing caught his eye. Sure they had a good selection of dildos, vibrators, and what not, but his wife was pretty experienced and nothing was really special. So he asked the woman who ran it.

She said, "Well I do have the voodoo penis, but I'm not sure I really want to sell it." "The what?" "Voodoo penis." Even though it was not really for sale, his curiosity was up and he convinced her to bring it out.

She went in back and came out with a weird wooden box. She opened it up and there was a wooden dildo with all sorts of strange occult symbols on it. He said, "Well that looks weird, but it's basically just a dildo." She said, "That's because you haven't seen what it can do."

Then she said, "Voodoo penis, the door!" And the voodoo penis flew out of the box, flew up to the door and began banging on it. Then she said, "Voodoo penis, the table!" It went over and began banging on the table. Then she said, "Voodoo penis, the box!" and it went back into the box.

When he saw this, he knew his wife had to have it. A thousand bucks later she was convinced to sell it, and he took it home. And he gave it to his wife with the instructions, "If you get bored while I'm gone, just lie back, get comfy, open the box and say, 'Voodoo penis, my crotch!' and you'll have an interesting experience."

She thought it was odd, but she accepted the present. And while he was away on his trip she decided that she would give it a try. Despite how silly she felt about it she took her clothing off, laid back, opened the box, and said, "Voodoo penis, my crotch!"

Of course the voodoo penis flew out and began fucking her! It was amazing! Soon she had a wonderful orgasm. But it wasn't all brute strength, because before long it had brought her back to a peak and she had a second! After her third, she decided that this was a good thing but she'd had enough for now. Besides, she was getting sore.

So she said, "Voodoo penis, stop!" It didn't. She tried grabbing it and pulling it out. She wasn't strong enough. Desperate she decided to throw on a dress and go to the hospital. Unfortunately, halfway to the hospital, she went off the road during yet another orgasm and hit a tree.

As she was lying there in her car the policeman came up and asked why she hit the tree. She said, "Because (pant) I'm being (ooh) fucked by (pant) a voodoo penis!" "You're what?" "Being (oooh) fucked (gasp) by a voodoo penis! (pant)" "Yeah, right. Voodoo penis, my ass!"

Pictures

2010-01-26T22:00:00.000-08:00

A couple of weeks ago my family had a good photographer come in for a photo shoot with me, my wife, her parents, and our children Sam and Claire.

Today you can see both photos my father-in-law took and the professional photographer's favorites.

This was money well spent. :-)

For anyone in Southern California, I would highly recommend Leo Flores as a photographer.

More random stuff on the sexes

2010-01-22T21:55:00.000-08:00

A couple of posts ago I discussed some complications in relations between the sexes. Nobody seemed too offended or disagreed too much, so I'll tackle the topic again, in a more haphazard way. (I touched on a somewhat related topic lst year when I claimed that males may have started as parasites.)

But as before, if you think you'll be offended you should stop reading now. Unlike before I'll actually touch on something interesting that applies to homosexuals at the end, though most of this is not about homosexuality.

My last blog reminded me of a book I read years ago that I wanted to re-read. So now I've bought it and am re-reading it. The book is Shadows of Forgotten Ancestors. It starts off slowly, with a basic overview of the history of the Earth, the history of life, how genes work, and so on. All pretty basic if you're conversant with science, though at some point it is likely to get detailed enough that you'll probably to find yourself in unfamiliar territory.

But what I like about it is the second half. This is where we encounter a discussion of the documented effects of testosterone on personality, an analysis of the social lives of our near primate relatives, and so on. I won't try to summarize, so much as to hit the some high points. Anyone who believes that women are basically men without the penis who can have babies should read through the effects of testosterone on personality. Anyone who has wondered why we say, "Fuck you!" when we're mad should read through accounts of how sex and dominance are intertwined in different primates. I'm not sure what lessons we should be drawn from a comparisons of militaristic chimpanzees with the free love bonobos, both of whom are very closely related to us, but I certainly saw strong parallels between both societies and human groups. All in all I rank this book as being one of of the strongest challenges to my preconceptions of how I want to see the world.

Moving on, a friend told me that a lot of what I said in my previous post is echoed in more detail, at longer length, and more amusingly in Sperm are from Men, Eggs are from Women. I haven't read the book, but the reviews look promising.

If I'm guessing right from the reviews it touches on one of my favorite pieces of evolutionary trivia: that there is good evidence that humans have significant sexual selection for large penises. That such selection exists is not surprising in light of all of the jokes and obvious sexual interest. (I will never forget having it pointed out to me 15 years ago women could estimate the size of a guy's penis from looking at the bulge in his pants. I was quickly assured that I had nothing to worry about in that department, which did absolutely nothing to lessen my self-awareness of people potentially looking at my crotch for the next couple of weeks...) However the actual concrete evidence is the simple fact that the human penis is about 3 times the size that would be expected based on estimates made from our size and the proportions of our close relatives the chimpanzees and gorillas.

Moving on, by coincidence this week on NPR I heard an interview with the authors of Why Women Have Sex: Understanding Sexual Motivations from Adventure to Revenge (and Everything in Between). It sounded interesting in the interview, and at some point I suspect I'll buy it as well.

I don't know if it is in the book, but there was an interesting piece of trivia that came up in the interview. It turns out that if you take a man and dress him up in a Burger King outfit, he's much less attractive to women than if you dress him up in a suit. By contrast men don't care about the clothing on a woman. In comparisons they'll judge her equally attractive in a Burger King outfit and in a nice cocktail dress. This ties very well with what I said in my last post about how women are looking for a good provider (status helps) while men are looking for evidence of fertility (status doesn't matter). Which helps explain why women remember what people wear while most men I know, including me, can't really remember what someone was wearing 30 seconds after they leave.

Moving on to men noticing signs of fertility, how sensitive are we? The answer is more than you'd think. For instance research has found that men react to how fertile a stripper is. It turns out that when you analyze how much strippers make in tips per night, the closer she is to ovulation, the more she makes, with income varying by a factor of 2. By contrast strippers who are on the pill permanently make about what strippers who are not on the pill would make during their period.

How do men notice? I don't know whether the full story is known, but at least some of it is behavioral. Women who are ovulating have been demonstrated to dress more provocatively, act more provocatively, put down the appearance of other women more, and show an increased attraction to men, particularly ones with masculine appearances. All of these behavioral cues reverse during her period. Even when strippers are trying to be provocative, perhaps the ones who are ovulating have a slight edge. Perhaps there are as yet unknown pheromones that men react to. Whatever the cause, men do react.

Moving on, the point about men with a masculine appearance is interesting to me. When researchers examined the subject, both men and women rank feminized faces as more attractive. But when push comes to shove, women are interested in other traits more than appearance, which gives a definite edge to men with masculine features. (Masculine is here defined by things such as a big jaw, square face, prow nose, and heavy brow. More on why it is defined that way shortly.)

Interestingly researchers in other studies have found that people form a variety of first impressions on men based on how masculine or feminine their faces are. For instance fathers are more likely to trust their teenage daughter's with a man whose face is feminine. But what can appearance tell you?

More than you'd think. It turns out that all of the characteristics that are linked with a masculine face are tied to testosterone exposure. Men who are judged as having masculine faces do turn out to have higher testosterone levels in their saliva. And the various personality judgments that are made based on this trait have a surprisingly good grounding in the known personality effects of testosterone.

Speaking personally, I am the kind of man who is more likely to be described as "beautiful" than "rugged". As much as I don't like it, this is concrete evidence of somewhat low testosterone levels when I was growing up, and it affects how people interact with me. In ways that most people don't even notice.

Moving on, the whole masculine/feminine thing leads to an odd piece of trivia that is not too widely known. Most people assume that gay men must be responsible for homosexual abuse. Not so fast. The little data that exists says that heterosexual men are far more likely to engage in homosexual abuse of boys than homosexual men! Now let's be clear, once a boy passes puberty, homosexual men may develop an interest. Legally an adult man and a 16 year old teenager might be child abuse. But that teen has sexually matured. Those examples notwithstanding, homosexual men do not seem to be attracted to sexually immature boys.

Why? Well when you ask adult heterosexuals who abuse boys why they are attracted to boys, all of the answers have to do with the feminine characteristics that boys have. By contrast when gay men are asked what they like about other men they cite masculine characteristics which boys don't have. It seems that attraction is tied to perceptions of masculinity/femininity in a very direct way that leaves little to no room for someone to be a gay pedophile. (I know of no research on the topic, but I have to wonder if this connects in any way to the fact that pedophilia appears to be very rare among women.)

As always I could ramble on, but this is enough for this post.

Solving Project Euler problems

2010-01-18T20:51:00.000-08:00

Late last year I got into Project Euler in my spare time for a couple of months. I solved over 150 of the problems, then I lost interest and have only sporadically solved any since. However I know I could do many more if I put some more time into it. I also contributed P240 which a number of people liked.

That's not a great track record, but it is not a bad one either.

This post is my advice for people who can solve basic Project Euler problems but want to get better at solving them. I happened to use Perl to solve problems, but the advice should work in a variety of languages. The techniques I used use closures a lot, but that's because that's the kind of guy I am.

First if you are unable to solve the first 5 then I would suggest that you're missing some basic background. I would suggest learning a programming language first, then some math. Only after that should you try Project Euler.

If you're going to tackle Project Euler problems, there are some things that come up over and over again. In particular I would recommend learning how to use dynamic programming to solve problems. That comes up a lot. Secondly I would recommend learning how to handle infinite streams of data. Thirdly do not be afraid of attempting brute force techniques with clever bucketing and pruning. If your program runs in a minute, you've found what they were looking for.

With those basic techniques out of the way, as you solve problems you'll develop a library of functions you use over and over again. Here are the functions in my library, in no particular order, with no particular attempt at good design (after all PE is just one throw away program after another...):

gcd Calculates the GCD using the Euclidean Algorithm

prime_iterator Returns a function which, every time it is called, will return the next prime. Internally I use the Sieve of Eratosthenes to generate this list. Note that in this code an "iterator" is always a function which iterates through some (possibly infinite) set as you call it.

is_prime Takes a number, makes sure I've generated all of the primes up to that number, then checks whether it is prime.

is_prime2 Takes a number, generates all of the primes up to the square root of that number, then uses trial division to check whether it is prime. This routine is slower than the previous if you're going to check a lot of primes, but uses much less memory. P118 is a good example of where you would use this version.

ith_prime Access the list of primes as if it were an array. (I had ith_prime(0) return 2, ith_prime(1) return 3, and so on. Remember what I said about not being carefully designed?)

factor Takes a number, returns a hash of prime factors and multiplicities. Trial division by primes up to the square root was efficient enough for all of the PE problems that I tackled.

factors Returns the prime factors of a number as an array with duplication to represent multiplicity.

n_pow_m_mod_k What the name says. I used it for P133 and stuck it in my library because I wanted to reuse it for P66 but it didn't work for the second one. (Search for Pell's Equation for how to handle that problem.)

minimal_divisible_repunit See P129 and P130 for what this is about. Internally this reuses the n_pow_m_mod_k function in its calculations.

pythagorean_triple_iterator Iterates through all Pythagorean triples ordered by the lengths of their sides. I'll have more to say on the algorithm in a bit.

generate_stream A promise is a function which, every time it is called, will return some set of future values you haven't reached yet, returning nothing when it runs out. This function takes a promise and lets you treat it like an infinite array, ask for the 1st, 5th or 500th element and it will generate the promise until it has enough stuff then return the value.

generate_memoryless_stream Like the previous but instead of accessing it like an array you access it like an iterator. This is useful for cases where it is convenient to calculate a block of values at a time. A perfect and pertinent example is the Sieve of Eratosthenes, which calculates blocks of primes at a time.

Now on to the Pythagorean triple iterator. How do I iterate through them? Well first I use Euler's formula to generate all primitive triples in the order that I want, and then I merge together all of the multiples of that stream to get all triples. To understand how that merging works, it would be good to go back to the infinite streams article and actually read it.

This sounds simple in theory. But in practice you have a problem. I'm merging a lot of data. I do that by maintaining a partially sorted pool of things I want to merge in, and then always taking the least next element. There is a data structure for that. It is called a heap. A heap is a tree, which has the heap property that each node is less than or equal to its children. This makes the top node at the root of the tree the smallest element. (Why do CS people always draw trees upside down so that the root is at the top? Really it would make more sense to call it a "root system" or else to draw it the other way around. Oh well, that's the convention.)

Luckily CPAN has multiple heap implementations. Unfortunately the one named Heap sucks, and I couldn't be bothered to look through the rest to find one that met my needs. So I took pointers from an implementation by Abigail, and implemented my own. The clever (and standard) trick in that implementation is that arrays can be used to represent a tree. The two nodes below the n'th element in the tree are at 2n+1 and 2n+2. So you can take an array of data, turn it into a heap, and then manipulate it as if it was a tree.

Writing your own heap implementation is a good exercise, so I'll just give the methods in my heap implementation for anyone who wants to duplicate it.

new Construct a new Heap. There is one optional argument, which is the comparison function. (Which is passed in as a closure, as you might expect.)

append Push a new element on the end, then let it "bubble up" to its natural position. Here is what I mean by "bubbling up". If it is currently at position $n > 0 then its parent is at position int(($n-1)/2). If it is greater than or equal to its parent you're done. Otherwise you need to swap with its parent and continue until it either has a smaller parent or reaches the root.

top Peek at the root element.

swap_top Take the root element out, replace it with something else, and let the something else "fall down" to its natural level. Returns the previous top.

extract First pop the last element off of the array, then call swap_top. This trick gives you the least element in the heap and makes everything else be represented in an array that is 1 smaller than the previous.

Hopefully this helps others who are trying to tackle Project Euler problems and haven't made much headway yet.

Why are relations between the sexes so complicated?

2010-01-15T20:55:00.000-08:00

Note: The subject of this blog post is likely to offend many people. If you think you're one of them the do us all a favor and don't read it.

Disclaimer. Everything in this essay is about my understanding of mainstream heterosexuality. If you disagree, remember that I'm talking about my opinions, not facts. Also, a certain proportion of people wind up homosexual, asexual, or otherwise don't fit the sexual norms. From the surveys I've seen, definitely over 90% and probably over 95% of people are, some experimentation notwithstanding, relatively normal heterosexuals. I have no idea how much or little any of this applies to the remaining minority.

You have been warned.

When I was young I wondered why relations between the sexes are so complicated. After all both genders desire the same thing, why not pair off in some orderly fashion with much less pain and complication? I could see that things didn't work that way, but it seemed to me to be a more sensible way for the world to work than how it actually does.

As I've grown older I've learned about complication after complication. As I learn about complications and see why they happen, the overall picture becomes clearer. Yes, the simpler world I imagined would be convenient, but it can't happen. Why not?

Well let's take my first assumption, that the sexes desire the same thing. We can actually tell what they desire from the questions they ask, and Google correlates those questions and summarizes what they find in autocompletes. When you compare what women and men want their boyfriends and girlfriends to do you get an interesting picture. Men and women want different things.

If we oversimplify, men want sex and women want commitment. Oversimplifying a tad less, men want sex and are willing to provide commitment as a way of getting it, while women want commitment and are willing to provide sex as a way of getting it. If you review the bf/gf comparison and pair the first few questions up, they answer each other in amusing ways. Partially that is coincidence. And partially it is that meeting of different goals.

Evolutionarily that picture makes a lot of sense. It takes little work for men to make children and a lot of work for women to. Therefore biologically men should be selected for a willingness to make children. However raising a human child takes enough effort to need multiple adults. Therefore it is to a woman's advantage to secure that commitment. A man who helps raise his own children benefits his genes, so he should be able to be convinced to do so. The point of lining up help for a woman is to make having children possible, so once the help is lined up, a willingness to have children should follow.

It should be noted that what I've described is not particular to humans. The same evolutionary tendencies and results can be seen in a wide variety of pair bonding species. Though when you apply it to humans people are more likely to be offended than when you describe some type of birds. Though for obvious reasons I find the human case of more personal interest.

It should also be noted very clearly that these strategies are not conscious on the part of the people involved. We have been genetically selected for traits that will cause us to act in these ways. And we directly react to those traits. For instance when a man meets an attractive woman, the parts of his brain that are not associated with libido are invited to take a short vacation. Sometimes with amusing results, such as a temporary inability to speak coherently. He certainly doesn't want this sudden burst of stupidity, but the immediate refocusing of his priorities aligns him with the evolutionarily successful agenda he is literally bred to follow.

Now let's introduce a complication. Sexual partners are not created equal. We have preferences about who of the opposite sex we'd like to have relations with. Preferences vary widely, but some interesting generalizations can be made.

First of all, what do men want? Well health is a good thing. Traits such as clear skin and symmetric faces are signs of a good immune system. Humans have established secondary sexual characteristics such as breasts as signs, and men's attraction to them is obvious. Signs of being good at childbearing as well. Cross-cultural studies have supported the importance of the waist-hip ratio, and even a brief acquaintance with the difficulties of human childbirth with large heads can illustrate why. And, of course, youth.

The subject of attraction to youth is an interesting one. I won't elaborate much on that topic, instead I'll direct your attention to Neotenty and Two-Way Sexual Selection in Human Evolution by David Brin, which is one of the most fascinating pieces of speculation on human evolution that I've read. I don't think he is entirely right, but I think he is right on several things I never thought of until I read that essay. The short summary that doesn't do justice to the essay is that men are attracted to youthful women, which poses the challenge of telling apart young mature women from as yet immature women. We do this through secondary sexual characteristics such as breasts. But this mechanism is fallible, and results in high incidences of pedophilia. (The essay has several parts, I would advise reading all of it.)

Since I have it, I'll give one random instance supporting Brin's thesis from the Google data that I have that he didn't. If you type into Google How do I get my ____ to with either girlfriend or wife in the blank, high among the suggestions will be shave. (Presumably the pubic area is referred to.) Why do so many men want this? Well general lack of hair is a sign of youth. But lack of pubic hair is a direct sign that a woman is not yet sexually mature. Isn't this awfully close to pedophilia?

Moving on, what do women want? Well, obviously, commitment from someone who will be a good care giver. Therefore it is no surprise that cross-cultural surveys (see Brin's essay for a reference) have found that top characteristics women say they desire are kindness, intelligence, and self-confidence. And other desired characteristics include accomplishment, reputation, health, vigor, reliability, and sense of humor. (Random note, the desire to prove accomplishment and reputation pushes men to create many social groups with their own hierarchies that men can try to get on top of.) All of these traits are associated with men who will have the ability and inclination to make good care givers. Note that physical handsomeness is not on the list. Women appreciate it but rate it more lowly, except when looking for extra-marital affairs. (A complication we haven't yet reached.)

Of course the above list isn't complete. For instance there is evidence that women react to pheromones, particularly ones that suggest that a man has a very different immune system. (When people with different immune systems mate, their children are likely to get protection from a broad spectrum of diseases.) Also what women react to and what they say they react to aren't always perfectly aligned, but that detail will have to wait.

OK, time for another complication that causes the pattern to fold in on itself into a pattern of fractal complexity. We have preferences in mates. We each would like to wind up with the best mate(s). But we can't all wind up with the best. Therefore each person tries to wind up with the best person they can, and the extent which we're willing to compromise to get a person depends on the person in question.

What do I mean by this? Well recall that men are aiming for sex, and are willing to provide commitment to get it. Women are aiming for commitment, and are willing to provide sex to get it. The million dollar question as a couple engages is who budges first. In general the higher status the other, the more willing we are to compromise. So the same man who may be willing to claim undying devotion for a chance at a gorgeous woman may be willing to sleep with an unattractive one, but won't commit to her. And a woman who may be willing to have sex with a high status man in hope of commitment may be only willing to have sex with a lower status man after commitment is established.

Tiger Woods provides a good illustration of the latter dynamic. Many women were interested enough in his wealth and talent to give him sexual access. However part of the dynamic was that he tried to make each believe that she was the other woman in his life, so there was a hope that at some point commitment could happen. When the news of his affairs broke and they reach realized how badly they'd been had, they got upset and one by one went public.

Now if you go back and look at the list of traits women want, you have a problem. The characteristics they are looking for are not immediately visible. Therefore women need to read between the lines to establish them. This pushes them towards judging based on secondary traits. For instance a man who is attractive to women who know him well is likely to have several positive characteristics, and therefore is judged as potentially more interesting. And anything that suggests such attractiveness will be seen as positive.

If you read a guide for pickup artists (I highly recommend that every woman should, just so you know what to watch out for), you'll find some odd consequences of this. For instance a man who demonstrates initial disinterest in a woman is demonstrating that he has access to more attractive women. This is evidence that he has what women wants, and therefore must be of more interest. The fastest way to demonstrate disinterest is to make an apparently thoughtless and somewhat insulting comment. If he follows up with nice comments then he has portrayed himself as being of interest but within reach. If the woman believes this message then she is more likely to offer sex without having required commitment first. Which is exactly what the pickup artist wants.

What woman would say that she wants a guy who subtly puts her down? None, obviously. Yet the technique is apparently effective on a lot of women! (Or at least on enough to make pickup artists happy.)

Moving on, what about the way that women tend to be insecure in relationships and the various games they play? I see at least two purposes of these. The first is to test the strength of a man's commitment. It is easy for men to say that they are committed, but if a guy stays through an emotional coaster ride and some demands then he's much more likely to actually be committed. But it isn't simple, because if she pushes it too far, then she has no relationship. How far she can push it will depend on the relationship, which varies widely.

The second purpose is to set clear expectations about how she wants him to behave. This pushes him towards accepting higher levels of commitment, and also makes it clear to him what she will and will not accept.

For an example of a book that helps women use these strategies, see The Rules. I would recommend that all single guys read it so that you can see what you need to defend yourself against. As you'll note from the reviews, it is very controversial, there is roughly an even divide between people who are offended and women who are happy with the men they landed using the strategy.

This brings us to an important point. I am trying to describe the natural dynamics as I see them. Those dynamics tend to be more clear when you look at people who consciously try to manipulate them. The two examples that I've used are pickup artists and The Rules. Most people's natural reaction to those strategies is to dislike how manipulative they are. A second reaction that follows closely is to point out that they result in significant power imbalances. Both reactions are perfectly accurate. They are manipulative and do cause power imbalances. However the manipulation mimics what happens naturally. Power imbalances are common within relationships. The point of the manipulation is to create a power imbalance that gets the manipulator's agenda to the fore. As much as we decry the means, most of us would be interested in the result.

Continuing, let's move on to a tricky complication. Cheating.

It is clear from the basic male agenda why men would cheat. Men cheat because more sexual encounters means more chances to have babies. But women don't want their men to cheat for fear that he'll wind up attached to someone else and leave. And men who are properly bonded will resist cheating, because that is part of what it means to be bonded. So men will not universally cheat, but will at least feel the temptation.

From the basic female agenda, cheating is not immediately obvious. After all it is not a good way to get commitment. But a woman who has a guy doesn't need commitment from anyone else. She has her caregiver, now she wants good genetic material for babies. Cheating offers her two advantages. The first is diversity of genetic material - you never know what combination of genes will work out best and experimentation has value. The second is that she can get superior genetic material than the guy she can get to commit. Of course the downside is that if she gets discovered, she may find herself out of a caregiver.

I've seen widely varying estimates on the levels of cheating in people. I know that some pair-bonded bird species have been found with up to 40% of the population born due to cheating. Humans are hopefully quite a bit lower than that. However one estimate I saw several years ago from England found that about 10% of children are not the children of the person who thought he was the father. That's pretty significant. And in any case it is high enough to contribute to high school biology teachers' reluctance to demonstrate the laws of genetics using the families of their pupils as examples. And is also high enough that a common first reaction among men to seeing his child is to look for a feature that verifies it is truly his. (I've experienced this reaction first hand, and felt silly but did it anyways. I also know someone who noticed that the child wasn't his, with appropriately bad results...)

At this point you get an interesting divergence between men and women. The reasons why men would want relationships and want to cheat are basically the same, so they are attracted to the same women for both purposes. (Though his standards are usually higher for commitment than cheating.) However women are different. A woman is looking for a relationship and affairs for different reasons. Therefore they are attracted to different men for those purposes. In particular for a relationship women want men who they think will make good caregivers. But for affairs women care much more about physical attractiveness. So attractiveness does matter to women, but not in as straightforward a way as it does for men. (Remember that when discussing Brin's essay I said I would come back to the survey result that women care more about appearance for extra-marital affairs than for relationships? Well I just did...)

I could go on but I've rambled quite enough and probably offended a good number of people in the process. Hopefully people found it interesting food for thought, whether or not you agree with all of it. I know that as recently as a few years ago I would have strongly disagreed with several items I stated, and wouldn't be surprised if I disagreed with several others in a few more years.

Our most popular drug is made of sugar, fat and salt

2009-12-28T23:33:00.000-08:00

On NPR not long ago I heard an interesting thought.

Suppose a hundred years ago that someone wanted to eat chicken. They had to go out, grab the chicken, cut its head off, pluck it, clean it, then cook it. That was just what it meant to eat chicken. Today that seems crazy. But in a hundred years that is how cooking will look.

Indeed, that is true for a lot of people. Just think how many people eat out more often than not. Or eat packaged meals. Cooking takes work, and most of us don't want to put out that effort if someone else is willing to do it for us.

On the surface that is quite reasonable. And I will confess to succumbing to the temptation quite a few times. I'm no food snob, and don't particularly enjoy cooking.

But all that notwithstanding I think that most of us should cook more. You'll be healthier. You'll eat better food. And it needn't take as much energy as you think.

Let me start with the health issues. My source here is The End of Overeating by David Kessler.

The problem that every commercial food distributer faces is getting people to buy their food. Preferably frequently and in quantity. People have a lot of reasons for buying food, but the three most important are named sugar, fat and salt. Individually we are attracted to each, however we reach a point where we saturate on any one of them. If you combine 2 of those, however, we are attracted to much higher concentrations than we are if you separate them. And combining all 3 hits a sweet spot where we want far higher concentrations than we would accept independently. Not only do we crave higher concentrations, but we experience a pleasurable rush of brain chemicals. In the pursuit of that pleasure we routinely overeat past the point of satiation. Then want to eat that way again when we recover. And this is why you're fat.

This principle is simple, effective, and well understood through the food industry. Whether you're a manufacturer designing a new kind of cheese puff or a restaurant trying to build a clientele you have to do the same thing. Deliver the magic combination of sugar, fat and salt and you can succeed. Fail to, and people will inevitably flock to the competitor who is willing to do it. The food industry has heard and obeyed. Whether it means giving us salads that are mainly a vehicle for delivering dressing (a base of oil with sugar and salt), or offering apparently healthy chickens that have been preprocessed with the injection of a syrup with sugar, fat and salt (this is why it falls apart so easily in your mouth) all of our processed food is a drug delivery mechanism.

The phrase "drug delivery mechanism" may seem overblown. It is not. The effects on our brain chemistry really do resemble an addict getting drugs. And measurements in animals of how motivated they are to get the reward show that modern processed food is only slightly less motivating than cocaine

Obviously there is some variation. But this is one of the prime reasons that obesity has expanded from being a fairly rare disorder to being about a third of the population. And today obesity is seen in ever younger people, and is more and more extreme. It would seem that the drug affects different people differently, and you can see from your waistline how much you are affected. (I'm hit about average, I weigh an extra 20 pounds or so.)

Of course this drug isn't exactly good for our bodies. It is no accident that the leading diseases tied to modern lifestyles are heart attack, stroke, diabetes and hypertension - all of which are tied to what we eat. Not that long ago I talked with someone who had a heart issue and just went to a cardiologist for the first time. The first lifestyle question he asked was, "How often do you eat out?" Note that where you eat out doesn't matter, because the entire industry has the message. They all know the game, and all are going to deliver the same drug. Ditto the manufacturers of processed food.

However if you go and cook food at home, you don't need to worry. You won't hit the sweet spot very easily because you lack the equipment to do it. You'll actually have to chew your chicken because before cooking you didn't break down the internal structure and fill it full of sugar, fat and salt.

The other interesting thing about cooking food at home is that it tastes better. Let's be clear, you won't crave it as much. If you've got one plate with fish you grilled at home, and another with potato chips, it will be the chips that you can't keep your hands off of. However the fish will be more enjoyable.

Let me give a concrete example. When I married my wife, her family was into pancakes made from mix. I've noticed two things about pancakes made from mix. The first is that they don't taste very good. The second is that they don't save you much work. Seriously you can't make the pancake until the pan is hot, right? Well it takes about as long to make the batter as to make the pan hot, and you do those at the same time. Therefore the mix didn't save you time!

Now when I tell people this, many claim that the pancakes they make from a mix taste great. And I tell them that they think that only because they've never had a decent pancake. Try that recipe, then come back and compare. (A funny story about that particular recipe. I had a co-worker who I gave that recipe to who it turned out had a friend who had been raving about my sister's pancakes for 30 years but didn't know how to make them! My friend was happy to be able to pass it along.)

All that said, when someone who doesn't cook faces cooking, it is scary. You don't know how anything works or what anything does. In reality all you need to do is get a recipe, follow directions, and it tends to work out. When you get brave you can experiment. Sometimes the experiments flop. But even that isn't a bid deal.

For instance yesterday I made turkey soup. It was going great until I added noodles. I envisioned firm noodles, and so it was at dinner because I put them in at the right time. But the noodles in the pot didn't stop cooking. So now instead of turkey soup I now have soggy turkey casserole. However it still tastes good and is still good for us. And I won't make that mistake again next year!

So stretch yourself. Look for an easy recipe and cook a meal. Even if you never come to enjoy cooking particularly much, you may find it worthwhile. And who knows, perhaps you'll turn out to be someone who enjoys cooking. Try it!

Work optimally, not long

2009-12-25T15:37:00.001-08:00

It has been a long time since I entered anything here. But I'm in the mood. I've cooked, eaten, and even done cleanup after my turkey dinner. If you're uninterested in hearing about cooking turkey, skip the next paragraph.

This time I tried dry brining the turkey. That was surprisingly easy. A few days ago I filled a small bowl with 1 tbsp of salt per 5 pounds of turkey (that's 6.5 ml salt per kg), added some spices for taste (I did sage and a ground up bay leaf), mix, then spread that over my freshly bought turkey, put a plastic bag over it, then let it thaw in the fridge. Today I just roasted it normally. Based on past experience I checked regularly to baste the turkey, but it really wasn't necessary since the juice stayed in the turkey. Tomorrow I'll get supplies, then make turkey soup, which for me is the best part of having turkey dinner.

Enough about food. What I want to write about today is something that we all know in theory, but far too few of us act on. That is that it is better to work at an optimal, sustainable level than it is to work long hours. Certainly this is true in software, and likely in many other fields as well.

More precisely I would give the following rule. If you see a developer working more than 40-45 hours per week, ask yourself two questions:

Will this effort be sustained less than 3 weeks?

When it is over, will this person get corresponding downtime?

If the answer to either question is "no", then I strongly recommend looking at the schedule and figuring out what deadlines to slip immediately to get people back to reasonable schedules. Because you're going to compromise something, and choosing up front is much better than making it by burning people out and getting sub par work.

The reason for this rule is that people have a maximum sustained throughput for creative work when they are working somewhere between 35 and 45 hours/week. (For a reference on that see Rapid Development by Steve McConnell.) You can temporarily raise your hours above that and get improved productivity. But after a few weeks it catches up to you, your productivity drops, and you'll get less done despite working harder. And once you're in that state you've got a vicious cycle. Your lack of productivity pushes you to work harder which makes you less productive until something gives. Quite possibly your sanity.

The one thing that seems odd about this to most people is that people don't realize that their brains aren't working. But the reason for this is simple. When you're impaired, the first thing that gets impaired is your ability to judge how impaired you are. Thus your self-monitoring for impairment is not trustworthy. This effect is seen with many kinds of impairments, including chronic sleep deprivation, alcohol, strokes, and so on.

The army once did a study that showed this in a striking way. They took soldiers and put them on different sleep regimens. Among other things they found that most soldiers could operate indefinitely on 6 hours of sleep per night and would claim to be fine. But their wives said they weren't fine. Comparing results on ability tests before and after the new schedule, the wives were right and the soldiers were wrong. Which shows how easy it is to think you're still functioning when you really aren't.

Now that's the theory. What is the practice? The practice is that most of us don't do this. The reason is that most organizations manage by schedule, putting pressure on us to work harder. We put the effort out, we try to make the deadline, and then we fail to be aware of our declining output. Possibly we make the deadline, but quality suffers, bug counts suffer, and development+maintenance comes out to a lot more than it would have if we had focused on development alone.

What keeps us from doing the smart thing? For many of us, fear. While intellectually we know that we should be able to do more if we keep to a peak level, as a practical matter we're scared of being called slackers for not visibly putting out effort. Particularly if everyone else is working overtime. This fear is particularly reasonable since management usually has no idea how much developers do or why things take us so long. And is doubly so if our direct managers are themselves working too hard and therefore not themselves in the best shape.

Another major thing that keeps us from doing the right thing is that nobody likes to deliver bad news. Nobody wants to tell their manager that the schedule will slip. So we put off delivering the bad news to anyone, and try to keep it from happening. Of course this doesn't keep the schedule from slipping. But there is always the hope that someone else will be forced to admit that the schedule is going to slip. And failing that some people will deliver an incomplete and non-working piece, declare it done, then push the admission of failure off even farther.

Delivering bad news gets harder still because everyone else has also avoided doing it. Everyone delivers their bad news as gently as they can, and the manager inevitably wants to hear it as better than it is. Therefore upper management is usually seriously out of touch. The result is How Shit Happens, which is only funny because it is true.

As an example of how extreme this effect is, The Mythical Man-Month points out that in the rare cases of early projects, the good news starts coming immediately. If the project is late, there is no sign that the project will slip until about 2 weeks before it is due. And this is true no matter how long the project is actually delayed. And the driving force behind this is that nobody wants to deliver bad news, and nobody wants to hear it.

So we have a real organizational problem. How should you solve it?

The easiest problem is to ignore the problem and try to survive. This is suboptimal in virtually every way. However it is what most of us do.

A better option is to try to find an organization without the problem. They do exist. I know because I've been lucky enough to work for some. (Something that I suspect is a key ingredient is working in short iterations. Reality and management dreams tend not to get too far out of sync on a 1 week project.) But those can be hard to find. And in the current economy most people have a hard enough time just finding a job that they can't worry about the quality of the employer.

The final option is to try to make your current organization into one that works better. If not better for everyone, then at least better for you. This is the most difficult path, and there is no simple recipe for how to solve the necessary interpersonal issues that you'll encounter. However I can give you some useful ingredients.

The first is education. Go and get Rapid Development, read it, and learn what it says. It is one thing to confront your manager with the message that you need saner work hours. It is quite another to share research about how to get more done and say that you really want to do it. This works even better if you've identified multiple things you'd like to change at once, so it doesn't come off as your cherry-picking the item that you most wanted to try.

The second ingredient is knowing that delivering negative messages often is not as bad as we think it will be. Admittedly it doesn't always go well. However any manager who has been around the block a couple of times knows that software projects tend to fail, so it won't be as much of a surprise as you might think. Of course your manager is usually under pressure from above to meet deadlines and so is likely to not want to face that bad news. But it won't be a surprise.

The third ingredient is being conscious of the fact that delivering bad news early, no matter how bad it may be immediately, usually works out better in the long run. Yes, it is unpleasant and we all want to procrastinate on unpleasant things. But delivering bad news up front then better news later makes people happier in the long run than letting people think they'll get good news and disappointing them with bad news. (Incidentally if you're going to deliver bad news, take a tip from Machiavelli's The Prince and deliver it all at once, then deliver good news afterwards.)

The fourth ingredient is to understand the person you are talking to, and deliver your message in a way that is likely to be heard. A lot goes into this, but a big factor that lots of people miss is cognitive dissonance. Let me wrap up this topic, then I'll go on to explain it shortly.

The last ingredient is being aware of what the status quo is likely to cost you. If you allow yourself to be aware of that, you'll have lots of motivation to avoid that fate. And in this case motivation is a good thing.

Now let me return to cognitive dissonance. I won't explain it at length. I've done that elsewhere and likely will do it again, but here are the basics.

The principle of cognitive dissonance says that people will go to any length to maintain their positive self-image of themselves. This means that messages which threaten your positive self-image trigger defensive reactions. The more powerfully the message is delivered, the worse the defensive reaction is likely to be. (When you see such a reaction the last thing you want to do is follow up with facts proving your position. That just increases the conflict and therefore increases the defensive reaction.) What this means for any given person will vary from person to person. However if you remain aware of this phenomenon, you can often figure out where the hidden landmines are, and have hope of communicating what would otherwise be impossible.

For example, suppose that your manager thinks he or she is a pretty good manager. (Most of us tend to think we're pretty good at our jobs. No matter how good or bad we may be.) Then at all costs you must avoid threatening that self-image. Therefore you would need to avoid saying anything that even looks like you're delivering the message that there is room to be a better manager. This is true, even if said manager regularly points out things that made them better managers. To avoid the possibility of trouble, it is best to make the case that what you are saying flows from what that manager says about management.

This makes it really bad to say, I was reading this excellent book by Steve McConnell and really liked some of what his advice about managing software teams... Doing that is explaining to your manager how to manage, and anything you say after that which your manager doesn't already know and accept is going to be really difficult to accept.

It is better to say, I've been reading a good book, and it has taught me a lot about myself. It is making me aware that I am not working as effectively as I could... Making the message be about you is a reasonably safe way of side stepping the question of whether this is something the manager should know to tell you.

But it may be possible to do better still. For example consider, You know how you always tell us that we should constantly been trying to improve ourselves? Well I've gotten some ideas from a software classic, and here are some things that I'd like to try... If this is true, then you've put cognitive dissonance in your corner. Your manager will now find that a "Yes" is an opportunity for confirmation of his or her good self-opinion. Better yet, your manager will then be inclined to interpret the outcome as success, giving an even stronger confirmation of your manager's ability as a manager.

Now all that said, there is no guarantee that you can keep a sane work schedule when the going gets tough. However if you can do it, the outcome will be better for everyone. That doesn't mean the outcome will be good, but it is better and that is really worth it.

The use and limitations of exceptions

2009-12-15T00:17:00.000-08:00

I'd have a longer blog but I just had a farewell party and so am do not have the most time to spend. So I'll just point to an email wrote to golang about what I see as the cases where exceptions prove useful, what the limitations of exceptions are, and some of the reasons why Go doesn't include them.

Design of a reporting system

2009-12-11T22:04:00.000-08:00

I've had to build a lot of reports, and in my last job I build a reporting system that I think worked quite well. This post explains some of the key ideas behind that system.

One of my best ideas was to use Excel, not compete with it. Users will come with detailed requests for the exact reports they want. You can try to provide what users ask for. Inevitably they will change their mind, or the report lovingly described by someone's manager won't be what that person wants, or a million variations on this theme. This will generate an endless stream of requests as people try to get you to tweak what is being provided into their exact desire. Alternately you can make the data from your system directly accessible in Excel, and encourage people create the exact reports they want themselves in an interface they are comfortable with. Now all that you need to do is provide raw data, appropriately aggregated, and you're no longer a bottleneck. Giving people what they want rather than what the initially ask for makes for happier users.

Luckily it is easy to integrate with Excel. What you need is a way for users to design a web-based report which provides the raw data they want, and make it available with a short URL (less than 50 characters). Now a user can open up a new Excel workbook, then choose Data, Import External Data, then New Web Query. This pops open a browser. The user puts in their short URL, and the browser loads it. The user can then click on the HTML table that they want, save the web query, and Excel will run it again then fill a spreadsheet with the data in that table. Now a complex set of spreadsheets can be built off of that data, and every time they refresh it will re-run the report and get fresh data.

That's the user experience. Behind the scenes the developer needs to do several things to make this work.

Provide users a way to save the parameters of a given report, and create a named report. The raw parameters will tend to be too long to be saved properly. When users access this named report you must not issue a redirect. If you do a redirect it will work initially, but Excel internally remembers the redirected location. Which means that if the user tweaks the saved report, the Excel spreadsheet won't notice the tweak.

In your HTML give an unambiguous ID attribute to the table with the data. Excel will use this ID to locate the data. Otherwise it will try to analyze the structure of the HTML, and that will break easily when the report changes.

Have a way to easily specify relative dates. People frequently want reports that run to the present, run for last month, and so on. If they only had fixed dropdowns when they set up the report then the Excel spreadsheets they get won't update to what they want it to be. I solved this problem by providing an optional text field that I passed to Perl's Date::Manip to parse. (I actually improved the date parsing slightly for my purposes, but that was the base.)

Moving on, my philosophy about reports is that 1 report that can be tweaked in 100 ways is more flexible and powerful than 10 reports that can be tweaked in 10 ways each. Similarly when given a one-off reporting request, it is better to find a way to add something to an existing report to make it capable of handling that request than to do the one-off request. That is because if someone wants that feature once, someone else is likely to want it again. And there is something really satisfying about receiving the second request and being able to say, "I've already built that, here let me show you..."

Of course the challenge is converting that philosophy into action. How exactly do you go about building a flexible report that can be tweaked in many ways?

My strategy was to look at each report as an array of SQL statements. All statements before the last would create temporary tables. The last query would return the data set that I would display. Thus if I needed to pull in extra information - for instance I needed to segment data by whether a given promotion was run - when building the array I could dynamically insert a query to fetch that particular piece of information, and then pass it through all of the tables.

This strategy naturally leads to two useful features. The first is that you can add the exact SQL statements used to generate the report in an HTML comment. This is very useful for auditing purposes. The second is that you can add the feature of having a dropdown box listing the temporary tables that will be created, and allow the report to run to that point then display those intermediate results. This is helpful when debugging. And not just for the reporting engineer - it was very useful for me when the accounting department was able to come to me and say, "This report, at this step, is missing these invoices."

This strategy, of course, requires being able to create definitions for temporary tables on the fly. Most databases provide that. Whether you are in MySQL, PostgreSQL, MS SQL, Sybase, etc you have exactly the right kind of temporary table available. The glaring exception is Oracle. There are a couple of workarounds available with Oracle. I've tried automatically building queries with large subqueries. It kind of works, however the result is unreadable by end users, you can't easily display intermediate results, and I miss the ability to control execution plans by creating temporary tables and adding appropriate indexes on the fly before moving on.

The other requirement of this strategy is that your SQL statements can be customized by the addition of an arbitrary list of extra fields, join conditions, etc. I wrote more reports than I care to admit before I realized that the best way to do this is to use a templating system to build your SQL. Now that I have used templates to dynamically generate SQL statements, I wouldn't want to go back. I used Perl's Template Toolkit for that purpose. Again I tweaked it substantially for my purposes. But many different templating systems could work well.

Moving on, let's look at how I organized things under the hood.

Obviously I needed complex forms with lots of checkboxes, input boxes, dropdowns, and the like. I chose to make each form control into an object that knew how to find its data out of the form parameters, do validation, write out its HTML, and pass information on itself to the templates. Some of the objects had SQL statements attached so that I didn't have to duplicate logic between reports.

There are endless arguments in software circles about separating presentation from content. In some circles mixing content and presentation as these objects did is anathema. In general web development I would not be inclined to move the HTML for form elements into what is basically a model object. However in this case it worked out very well. The form is so directly tied to the action of the report that it wouldn't make sense to have different people work on the two. These reports were for internal use, and so functionality mattered more than aesthetics. And as a practical matter this choice made the addition of options to forms extremely easy.

And, of course, I needed to have a way to coordinate everything. I did that with a report object. The report class had methods to create each kind of form object. It had a method to be displayed. When displayed it would show all of the form elements, check whether the report should be generated, if so it would run all of the queries, and display the report, and (of course), it would append necessary documentation.

With all of these pieces the code for a trivial report could look something like this:


use Report;

my $report = Report->new();

$report->add_textbox(
    name => "user",
    label => "Who is this for?",
);

$report->add_textbox_number(
    name => "age",
    label => "How old are you?",
);

my @queries;

push @queries, {
    temp_table => "results",
    sql => qq{SELECT :user as "User", [% age %] as "Age", current_date as "Today"},
    args => {
        user => $report->user,
    },
};

$report->display(
    title => "My Report",
    queries => \@queries,
    doc => qq{
        <ul>
            <li> <i>User:</i> The user this report was generated for.</li>
            <li> <i>Age:</i> Their reported age.</li>
            <li> <i>Today:&t;/i> The current date.</li>
        </ul>
    },
);

Now this report isn't very impressive. Also passing the age into the SQL as a template variable is poor style. (I did that just to show what the templating looked like.) But if you changed this to having a dozen form controls tweaking a data processing step with a half-dozen queries then you could have a very powerful and flexible report. Add in a few more reports, the ability to combine independent options, and the ability for users to build on top of this in Excel, and you wind up with a surprisingly powerful and flexible reporting system.

Learn to negotiate

2009-12-10T07:04:00.000-08:00

I can't believe it has been a month since I've added a post. A lot has been going on. The most important being, of course, that I've accepted a job at Google starting in the new year. I am convinced that this is a good move that will make me much happier in the long run. I'll be doing a more interesting job with a new set of tools at a company filled with great people, what's not to like?

However the process has reminded me that we can all benefit from learning to negotiate. Now I'll be the first to say that I don't enjoy negotiating. I prefer to find ways to grow the pot rather than trying to take a bigger share of it. But a little reading and practice with negotiation can pay off very well.

A few years ago I asked a friend who was a very good negotiator what book on negotiation he would recommend. I do this kind of thing every so often with different areas, and I usually learn something from the books I'm directed to. He recommended Start With No. Reading this book has easily been worth tens of thousands of dollars to me already. Reduced to a nutshell, the idea is that progress in negotiation comes when you ask questions that the other person can reasonably say no to. And your results in negotiations will be better when you don't need to hear a yes.

A second book that I recently read on negotiation was Bargaining for Advantage. From a theoretical perspective this book is clearly the stronger of the two. If you wish to understand the process of negotiation, learn to recognize different negotiation situations, and negotiate regularly, this is clearly the book for you. It is clearly best in class, and will work for a wide variety of negotiation styles. However it seems to me that if you don't already come with a wealth of knowledge and experience on negotiation, you can easily read this book and find yourself saying, "That's interesting, but what do I do now?"

For me personally, a lifetime of avoiding negotiation (and therefore doing it badly when I had to) left me without any negotiation skills to speak of. So Start With No was the better starting book.

With that said, how did my last negotiation go? From my point of view, very well. My compensation is structured differently from my last job, but overall is similar. From what I've heard, people moving to Google usually wind up accepting a pay cut. Due to personal circumstances I honestly couldn't do that. Without having learned some basics of negotiation I am sure that I couldn't have done that. Heck, without what I've learned about negotiation I doubt that I would have been able to structure my resume to make it clear to potential employers how valuable I could be. And without that, I wouldn't have even had the job negotiation in the first place!

My thoughts on Go (Google's new language)

2009-11-11T23:49:00.000-08:00

In the last couple of days I've been looking at Google's new language. Here is kind of a brain dump of my thinking on it.

First of all anything new produced by people of this caliber needs to be taken seriously. It is not done and may fizzle, but it shouldn't be dismissed out of hand. In that spirit here is a high level overview.

That said, it seems to be a mix of good things, interesting things, and things that are oddly missing. Looking at code examples it is in the C family. Since they want to appeal to systems programmers they are focused on simplicity, compilation time, and run-time performance. It has good stories to tell on all three, but I'm someone who finds scripting languages acceptable for the latter two, and that means I have a pretty high tolerance for complexity.

At a higher level they have garbage collection, first class functions and closures. Given the audience that they are aiming for they have downplayed the latter two (I spent the better part of an hour trying to verify that before finding it buried in the spec. Those are cool. They have goto, but looking through the spec I see that they have labeled loop control. There is an old theorem that any flow of control that can be achieved with goto can be achieved with equal efficiency with labeled loop control. Therefore I suspect that goto won't get used much. (Though it does make sense for state machines.)

Those are some basics, but they have three key features that they think set their language apart from most offerings out there.

The first is goroutines. You can take any function foo and type in go foo(arguments);. This basically means, "You go away and do foo while I do other stuff. When you return you exit." This is a very simple yet powerful concurrency model. The programmer does not know or care whether the goroutine executes in the current thread or another thread. In fact if the language had proper support for it then you'd be fine with it executing in another process on another machine.

Of course telling goroutines to do work in the background isn't very useful unless you can communicate between them. Their solution to that is their second key idea, channels. A channel is something like a Unix pipe. It is just a line of communication you can pass messages along, and data synchronizes by blocking until it is read. Channels may be one way or two way, and a given channel may only send specific types of data. They provide abstraction because you don't need to know the details of what is on the other end of the channel. This makes simple services very easy to write. For instance they offer this example of a simple RPC server:

  func startServer(op binOp) (service chan *request, quit chan bool) {
      service = make(chan *request);
      quit = make(chan bool);
      go server(op, service, quit);
      return service, quit;
  }

The first two ideas are not that surprising given the history of the people who came up with this. But the third is a little more surprising. The third idea they call interfaces. I actually dislike the name, I'd prefer to call it mixins instead. The idea is that an interface is defined to be any type that supports a given set of methods. You can then pass that object in anywhere where that interface is expected. You can also add methods to the interface, that are automatically available to objects that support that interface.

In short it is very similar to a Ruby mixin, except that you don't have to declare that you're importing the mixin, it autodetects that you fit the rules and does it for you.

OK, if that is what it has, what doesn't it have?

Well, libraries. Given that it was released yesterday, that is understandable. :-)

A bigger lack is exceptions. I understand this decision. The problem is that they envision writing servers with a micro-kernel design. But what happens when one of the services breaks down? Who can catch that error? The code that launched the service? The code that is communicating with it through channels? If there is no good answer, then what sense does it make to let a service just go down? If they do add exceptions they have hard design problems to solve which arise from the fact that you don't have a simple stack-based flow of control.

The much bigger omission is generics. The FAQ says that generics are missing because they introduce complexity in the type system and run time. Obviously so, but they are well worth it.

People who come from dynamic languages (eg Perl, JavaScript, Ruby, Python...) might well wonder what "generics" means. Here is a quick explanation. We have higher order functions. What if we wanted to implement grep? Well we could fairly easily write a function that takes a function mapping int to bool and returns a pair of channels where when you stick something in the one channel, it pops out the other if and only if the function returned true. We could write a similar one for Strings. And so on and so forth.

But you have to write one function per type you want to grep on! This is highly annoying. The functions all have identical form except different types. But you have to write the same function over and over again. As a result there is no way to avoid repetition of certain kinds of boring code. Which is why, when asked whether Go has libraries allowing functional constructs like map, reduce, scan, filter, etc the response was, The type system rather preludes them. They would either have to use reflection (slow) or be specialised for each type of slice. (See this thread for that exchange.)

If you wish to solve this you need to do one of three things. The first is to have a type system that is rich enough to handle meta-functions like this. (For instance Haskell.) The second is to have types be figured out and matched at run-time, which is what dynamic languages do. And the third is to implement generics.

-----

OK, enough of an overview. What are my thoughts?

Well first I'll watch it. It is young and nobody knows what it will do.

Moving on I hope the Go community that springs up start thinking about designing around the idea of capability based systems. That is a strategy of handling security/access by handing out opaque tokens that you need to make specific kinds of requests. If you have the token then you have access. If you don't, then you don't have permission and have no way to make the request. See this introduction for an explanation of how powerful this idea is. And in Go you'd realize it by having the "opaque tokens" be channels to services that do whatever you need done.

On a somewhat negative note I have mixed feelings about interfaces. My comment on mixins is, "It is bad unless you do it a lot, and then it is good." That is because a mixin involves magic at a distance that adds stuff to your class. So using a mixin requires learning exactly how it works. That mental overhead pays off if you get to reuse it over and over again. But if you don't use it heavily, it is a source of misunderstanding and bugs. The interface idea has the same issues, with more magic, but without the huge bug of overwriting methods you implemented in a class. (That said, I'm sure that more than a few people will be wondering why they didn't get the method they wanted from interface A when it conflicts with interface B that you also match.)

On a negative note I predict that there will be a lot of bugs in real Go systems around issues like deadlocks between services. Concurrency always opens up the potential for hard to diagnose bugs. And knowing that, I hope they develop good news for detecting/debugging those issues.

And finally I think that the omission of generics is a mistake. If they want the language to grow up, they'll need to fix that. Exceptions, if done well, would be a good addition, but there are reasons to leave it out. But generics are just too big an issue.

A proposal on patent reform

2009-11-09T22:24:00.000-08:00

In theory the purpose of patents is to increase the public domain by providing temporary incentives to spur innovation. In that spirit patents are supposed to be limited to things that would not be obvious to one versed in the art.

However patents have lead to many problems.

First is the question of which ideas are obvious, and which are not. This is not a simple question at all. The patent office has resolved this question by being fairly lenient about which patents are granted, thinking that the issue can always be resolved in litigation. However people are reluctant to litigate, so invalid patents are acquired in great number. The theory goes that any individual one is unlikely to stand up, but if I threaten you with a dozen patents at once, my odds of having at least one stand up are pretty good. This set of incentives leads to a lot of overly broad patents that are pretty obvious, which causes uncertainty for others.

Second you have the issue of how long patents are good for. When it comes to drugs, a multi-year search and FDA approval can only be recouped from a long patent. When it comes to software a 17 year patent warps the market for many generations of technologies. Thus there can be no simple balance between encouraging innovation and unnecessarily hobbling further advancement. (Many people in software think patents shouldn't apply there. Certainly patents do a lot of damage in the software world, but as Paul Graham points out, if you're against software patents then you're against patents in general.)

Third we have the challenge that areas that change rapidly have a lot of opportunity for being the first to face a particular problem. If you're first to face a problem then it is easy to get a patent. Given that the patent office is lenient on granting them, you're likely to get it. So we tend to see very intense patent activity in areas that change rapidly. However those are exactly the areas where patents do the most to inhibit further progress!

My proposal addresses these issues.

I propose that no patent shall be accepted unless the technologies that make the discovery feasible and commercially viable have been broadly available for at least a decade.

If the problem has been solvable and a solution desirable for at least a decade but nobody has done it, that is evidence that the solution is not obvious to one versed in the art. Conversely if the solution quickly occurs to someone fairly shortly after the problem has become both solvable and commercially interesting, that is evidence that it wasn't really that hard. Furthermore it is exactly advancements in fields that are rapidly changing where patent protection does the most economic harm.

Why did I choose a decade? I chose it as a round number that is fairly close to half the length that a patent lasts. My theory being that if nobody has discovered it in that length of time, then the odds are reasonably high that nobody would have discovered it in the time the patent restricted the use of the invention, and therefore we have evidence that the net economic result caused by granting the patent is positive.

Of course, like any attempt at compromise on a controversial issue, I am sure it will satisfy nobody. But I don't think it is that bad either.

What various scholars have changed their minds about

2009-11-07T00:13:00.000-08:00

The 2008 Edge questionnaire is one of my favorite collections of essays ever. What they did is asked many well-known thinkers the question, What have you changed your mind about? 165 of them answered. There is an index, but I find it better to just dive in.

Admittedly it is a lot of reading. What would I recommend? It depends on your interest. Personally I liked Freemon Dyson's explanation of how we know that dropping nuclear bombs did not cause the Japanese surrender, one of the founders of the inflation theory for how the early universe evolved saying why he thinks that the theory is wrong, a feminist explaining that the preponderance of men at the top echelons is because men vary more, a biologist discussing why experts often should be doubted, a psychologist discussing statistical illiteracy in medicine, and an information scientist talking about why the sample mean is a poor statistic to use. (The sample mean is where you add up all of the observations and divide by the count.)

Those are some of the ones that I liked, and can keep people occupied for a long time. But I'm sure everyone will have their personal favorites. And yours could easily not intersect mine.

Did males start as parasites?

2009-11-05T22:39:00.001-08:00

Many years ago I was curious about why sexual reproduction is so prevalent. This is a summary of what I found when I investigated.

There is a standard theory to explain sexual reproduction, but it only answers half the question. The standard theory is that a species with sexual reproduction has a pool of genetic diversity. Should the environment change suddenly (temperature changes, a new disease, more effective predators arrive, etc), any gene that helps will have an improved chance of survival and will quickly spread through the population. Without sexual reproduction any particular gene line would eventually go extinct unless it was lucky enough to have the beneficial mutation happen within it. Therefore sexual reproduction allows for much faster adaptation, and benefits the species that does it.

This is all well and good, and convinces me that sexual reproduction has advantages asexual reproduction (as well as being more fun), but it doesn't explain why we have distinct sexes. For instance look at the lowly earthworm. They are hermaphrodites. When they meet they impregnate each other, and both go off and have babies. Hermaphrodites therefore reproduce twice as quickly as a male/female version would. Why, then, aren't we all hermaphrodites?

It can't be a simple quirk of genetics. The genetics of sex reproduction is much more plastic than most people realize. Even if we stick to vertebrates you can find strange things such as hermaphrodites (mangrove killifish) and animals that change gender (African reed frogs, several kinds of fish). So gender is more complicated than the familiar XX vs XY rule for mammals that we learn in biology. If hermaphroditic reproduction was evolutionarily favored, there are enough ways to get there that you'd expect it to be more common than it is.

As it happens I have never encountered a biology text that attempts to tackle this part of the question. However many years ago I came up with a theory that seems reasonable to me. I've discussed this theory with a number of biologists who agree that it seems probable to them as well. So this is my theory, but it is at least plausible. And for all I know it is the standard theory in evolutionary biology, and I just never encountered any source that presented it. (I'm very much not a biologist.)

My theory is that males started as parasites. More precisely, cheaters in a sea of hermaphrodites. And once men were effective enough, the ability of hermaphrodites to impregnate each other atrophied and you got strict sexes.

Let me break this down in more detail.

Imagine a population of hermaphrodites. Now add individuals who can impregnate others, but can't themselves be impregnated. These are males. By refusing the be impregnated the males avoid the hardest part of reproduction. This gives them more energy to devote to impregnating others. As long as the cheater impregnates hermaphrodites twice as fast as hermaphrodites do, this strategy works out.

Evolutionary theory has studied cheating strategies like this. Theory says that in any given environment there will be a natural fraction of the population that is male. With more hermaphrodites/male than the ideal, the male strategy becomes more effective, so the male population will rise. With too few hermaphrodites/male the male strategy becomes less effective so the male population falls. The ideal mix is called the evolutionarily stable strategy. (For another example of an ESS, some salmon fight for the right to spawn, and some try to sneak in while spawning is happening. We can calculate the ideal ratio between how many males should try to fight vs cheat, and it is very close to the actual measured ratios in real salmon.)

If your imagination fails you, don't worry. This isn't a hypothetical possibility, we can find this exact kind of mix of hermaphrodites and males (with no females) in c. elegans. That is a nematode that a lot of biologists study. (Biologists often like to work with organisms that are heavily studied for the simple reason that when they need to know something tangential to their research about how that organism works, there is a good chance that someone else has already studied it.)

In this situation males are cheaters. Biologically in some ways they parallel a virus - they are unable to reproduce on their own but can reproduce by hijacking the reproductive system of their hosts. However biologists have found that symbiosis arises easily out of parasitic relationships. And that is what I believe happened with the genders.

Take our previous population. Imagine that the males are numerous and effective at their job. So effective that a hermaphrodite who attempts to find another hermaphrodite to impregnate will expend so much energy that she would have more children if she just let herself be impregnated more frequently by the very convenient males. In this situation a true female strategy makes sense. And once hermaphrodites take that step, it is only a question of time before the ability for a reproductive individual to play both the male and female parts atrophies, and strict genders emerge.

For more interesting thoughts on sexual differences in humans, here is a psychologist discussing the consequences of men being expendable. I don't agree with all of his conclusions, but he is starting with solid facts that most people aren't aware of. For instance here is confirmation that men are significantly over-represented both at the top and bottom of a lot of distributions.

And in a very different vein, I have long liked David Brin's Neoteny and Two-Way Sexual Selection in Human Evolution. A summary can't do it justice. It comes up with a plausible theory explaining topics as diverse as why women have breasts to why so many men are pedophiles. Even if, no especially if you think that it has to be BS, it is worth the read.

Is the financial crisis really over?

2009-11-05T08:09:00.001-08:00

I recently asked a friend who works on Wall St about whether these warnings about the municipal bond market were accurate. He gave me a detailed response, and gave me permission to repeat the comments but without his name or company attached. This is very reasonable given his current position, so I won't say anything more detailed than that he is in a good position to know what is happening in the credit markets, and I trust his opinion far more than any of the talking heads you see on TV.

Here is what he said:

Well, more than remotely accurate....

We have 3+1/2 more disasters to go in this crisis:
- Resi: this is the 1/2 disaster, as we're a little more than half way through the resi correction; unfortunately, we still have a long ways to go!

- Commercial real estate lending / CMBS: 4th Q retail will likely be a disaster, and commercial real estate has had as much or more price explosion as resi; already we are seeing soaring bankruptcies, etc. Many shopping center tenants have 'go dark' provisions (no rent due if >x% of the mall is dark), which are coming close to exercise. Banks and insurance companies are hugely exposed here; could make subprime look like a walk in the park.

- Credit card debt: even though the pace of job destruction has slowed, we are still in job destruction mode; consumers are increasingly falling behind on all debt payments. People are (wisely) looking to save rather than pay down debt (treating the debt as a lost cause)....

- Muni. As an example, tax revenue in your golden state is down 40%, and this story is repeated all over the country at all levels of government. So far, the Fed government has kicked a huge amount of money down the muni chain, which has kept the problems largely at bay; however, this process is nearing an end. The fact is that the entire country is massively leveraged: consumers, local government, state government, and the federal government. In the boom years, governments piled on debt and hugely increased their services, and most also hugely grew employment as well as entitlements (health care and pensions for muni employees). Now, they are facing revenue shortfalls but have great difficulties cutting services (often mandated by law), cutting staff (administration is opposed), cutting capital expenses (again, mandated by law). Default is in the cards for a lot of them, as federal money runs out.

Given the way the administration handled the automakers, I expect the muni bond holders to get hurt while the pension plans are made whole. I don't know what government does to avoid massive service cuts, though! It has seemed to me that one of the motivations for doing federal health care now is actually to ease up on state medicare spending (that's why your Governator has been making pro-health care reform statements).

So far, managing the crisis has relied on using the remaining credit of the borrower of last resort (the US). And while lots of reports have shown that govie debt is low relative to GDP on a historic max basis, these reports have completely overlooked the net debt position of both government and consumers. It is of course difficult to tease it all out (lots of munis fund housing projects), but personally I feel that we really can't have much debt ceiling left. Particularly if you consider that current/recent levels of GDP require a level of corp credit, but we now have vastly less corp credit available: without more corp credit availability, GDP must continue to slide, which implies debt-to-GDP continues to rise even without more borrowing.

An economist friend of mine (who lived through Argentina) and I both believe that the only real outcome of the whole crisis is that the Fed will manage a graceful and slow dollar dilution, so that we see a huge inflation but over a long enough stretch of time so as to not be unduly destabilizing. This process would basically inflate away our debt as it revives the domestic economy. Unfortunately, so far as I know, no country has ever achieved economic success via a weakening currency!

Anyway, yes, this is all really bad, and still has a ways to go!

Reading this I am strongly reminded of what I saw predicted in Wealth and Democracy several years ago. The book is a long read, but buried in it were a list of parallels between the USA circa 2000 and the last 3 great world empires shortly before their collapse. In all 3 cases shortly after a boom caused by financial speculation there was a financial crisis, which they recovered from fairly fast at the same time that they launched a military adventure that was expected to be a quick, successful war. The war dragged on and cost far more than expected. Then public mood turned against the war around the time that a more serious financial collapse hit. After a series of subsequent financial collapses there was political unrest leading to civil war 15 years after the peak in 2 out of 3 cases. (The exception was England, which got involved in WW I before the civil unrest could become worse.)

When I first read this I thought, "Interesting, and Kevin Phillips does have a track record of making apparently absurd predictions that came true, but I'm not overly concerned." I still believe that the prospect of expressing ourselves through democracy can head off the possibility of civil war, but it has been scary watching the timeline unfold like clockwork.

Why I left math

2009-11-02T21:20:00.000-08:00

Reading how shocked Doron Zeilberger is at the state of modern mathematics reminded me of why I left the subject.

Math departments regularly have visiting mathematicians come and give talks. Or at least the one I was at did. For the visiting professors these talks were a confirmation of success, all of these people came to hear about their research. So they would talk about their research and get quite excited about what they were describing.

As a grad student I attended. I quickly noticed that most of the professors in the math department went out of politeness. However they knew they wouldn't understand the talk, so they brought other things to do. If I looked around about 15 minutes into the talk, I'd see people reading books, grading homework, and otherwise not paying attention. At the end of the talk the speaker would ask whether there were questions. Inevitably the mathematician who invited the speaker would have some. Occasionally a second mathematician would have some. But the rest of the room wouldn't.

This was supposed to be the high point of the life of a mathematician? That's when I decided that, no matter how much I loved mathematics, I wanted a different career. Unfortunately my wife was in grad school as well, and we were in such a small town that I didn't have any immediate employment options. Therefore I remained a rather unmotivated grad student. In the end my wife switched to medical school just before I would have finished the PhD. I'm mildly disappointed that I didn't finish, but it really has been no loss.

Why do mathematicians put up with this? I'll need to describe a mathematical culture a little first. These days mathematicians are divided into little cliques of perhaps a dozen people who work on the same stuff. All of the papers you write get peer reviewed by your clique. You then make a point of reading what your clique produces and writing papers that cite theirs. Nobody outside the clique is likely to pay much attention to, or be able to easily understand, work done within the clique. Over time people do move between cliques, but this social structure is ubiquitous. Anyone who can't accept it doesn't remain in mathematics.

It is important for budding academics to understand this and get into a good clique. This is because your future career and possible tenure is based on your research. But the mathematicians making those decisions are unable to read your papers to judge your work. Therefore they base their decisions on the quality of journals you get your papers into, and the quality of people you get writing recommendations for your work. But both of those come down to getting into a group that includes some influential mathematicians who can get your papers accepted in good journals, and that can write strong letters of recommendation.

In fact if, like me, you are someone who likes to dabble in lots of things, you will be warned (as I was by multiple professors) about the dangers of not focusing on one small group. You will be told plenty of cautionary tales of mathematicians who published a number of good papers, but who didn't publish enough in any specific area to get good mathematicians to stand behind them. And therefore the unlucky generalist was unable to get tenure despite doing good work.

For a complete contrast, look at the situation in biology. A motivated advanced biology undergrad is both capable of, and expected to read current research papers. When biologists go to a talk they both expect to understand the talk. And biologists have no trouble making tenure decisions about colleagues based on reading their papers.

I subscribe to the belief that the difference is mainly cultural. Biology is fully as diverse and complex as mathematics. Furthermore what I have read about the history of mathematics suggests that the structure of the mathematical community was substantially different before WW II. For example David Hilbert was known for stopping speakers and forcing them to define anything he found unclear. (Amusingly he once had to ask Von Neumann what a "Hilbert Space" was.) But after WW II an explosion of university math departments and a focus on solving concrete problems lead to a fragmentation of mathematics. And once mathematicians came to accept that they couldn't be expected to understand each other, there was nothing to prevent mathematics from splintering into fairly small cliques. Which has happened, and this is unlikely to ever be reversed.

PS I'm amused at the fact that a number of comments at Y-combinator thought that the situation with programming was worse than mathematics. Yes, there are divisions within programming. But they are nothing compared to the fragmentation in mathematics. I've done both and there is simply no comparison.

Why write a blog?

2009-10-28T21:26:00.000-07:00

Before I had a blog, I wondered why people wrote them. Now that I have one, I wonder why other people write theirs. Because I know my reasons, and I am sure they are not shared by many other people.

I've always been the kind of person who liked learning and thinking about a lot of different things. Inevitably this means that I wind up having opinions on diverse topics. Which means that I regularly run into things that I have opinions about that I couldn't share because nobody in my immediate environment really cared. This blog has become a place for me to say those things. If nobody cares to listen, I still enjoyed getting my thoughts in electronic form.

Now to reasons I don't do the blog. I don't do it to share my personal life. My personal life is private, and the people I want to tell it to can hear about it directly from me. I don't do the blog for economic reasons. Admittedly I turned on ads out of curiosity for what it would be like. But to date I've made less than what I usually spend on lunch. I've made less than I get from 5 minutes of paid contract work, and have no shortage of potential contract work available right now. I don't do it out of desire for fame. I've seen enough fame in my life. And besides the built-in audience when I spent years posting on Perlmonks is much bigger than I'm going to get with a random blog.

So this blog is an outlet for expressing thoughts of a kind I've always had. Nothing more, and nothing less.

Now what triggered this particular post? It was a comment on my first entry that thought I was missing a big opportunity by not writing more about that topic and taking advantage of my current position as an expert. I simply don't see things that way.

Over my life I've been seen by people as an expert on a number of things. Lately I've been compensated fairly well for my expertise in Perl, A/B Testing, and general data manipulation. I don't think that being seen as an expert on spaced repetition learning would add much. I've also become moderately well-known in multiple communities. There is no need for me to try to become well-known for having taught a course. So I don't see the opportunity.

Furthermore there is a cost. I've always disliked people who took a good idea or story and expanded it with fluff to make the most of it. Why would I seek to be a kind of person I don't like? It makes no sense to me.

The same will go in the future. If I say my piece on a topic and don't feel I have more to say, I won't say more. If I say more it is because I feel I have more to say.

Stock buybacks shouldn't change stock prices

2009-10-27T22:52:00.000-07:00

Companies are owned by the shareholders. Companies try to make profits. And when they do, some combination of three things happens with those profits.

They get reinvested in the company.
They get paid out as a dividend.
Some shareholders are bought out in a stock buyback.

The mechanism of the first two is generally understood. But there is widespread misunderstanding of the third. Inevitably when a stock buyback is announced, you'll find articles that insert helpful explanations about how stock buybacks increase the stock price, helping investors.

Every time you see one of those explanations, I guarantee you that the person who wrote it is ignorant about basic financial matters. That appears to be most of the media.

Why, you ask? Because changing the stock price is not the purpose of a stock buyback. And to first order there is no effect on stock price. The reason for that is that the stock price is the value of the company divided by the number of shares. As the company buys back its stock, the value of the company drops by the money spent, and the amount of stock outstanding drops the same. Investors are rewarded by having those who wish to be bought out, bought out. And, unlike with a dividend, investors who didn't want to sell don't get taxable income.

So, for instance, if a company worth $50 billion with 10 billion outstanding shares of stock buys back $5 billion dollars, then the value of the company drops from $50 billion to $45 billion (because it spent $5 billion in cash), the amount of outstanding stock drops by 1 billion, and the remaining 9 billion shares are still valued at $5/share. This is finance 101 stuff. It falls right out of the CAPM.

Now I made a comment about "first order". Why? Well because there are several second order effects that realistically could cause stock prices to move. What sort of second order effects could those be?

It is always hard to buy large amounts of stock without increasing the price (and thereby cause inefficiency in your purchases). However buybacks are always announced in advance prominently enough to let interested shareholders know this is a good time to sell off large positions, so it works out.

Different investors have different opinions on the true value of the company. The ones you buy out tend to be the more pessimistic ones, leaving you with people who think the stock is worth more. This effect is generally small.

The market has many people who are confused about finance and who may expect prices to go up. If lots of people expect prices to go up, that's a self-fulfilling proposition. But it tends to be a temporary self-fulfilling proposition, and Wall St is very good at earning money from people predictably making such mistakes.

The company's cash holdings fluctuate less in value than the other assets that make up the company. Therefore as the relative balance of cash and other moves towards being more heavily weighted with "other", the volatility of the stock price increases. If you've studied options you'll know that increased volatility increases the returns for people holding options. Those returns come from somewhere, and the somewhere they come from is existing shareholders. Therefore the stock price should drop slightly. In theory. If investors are informed enough to pay attention.

I have no idea what the net result of these effects is on current stock holders. However all of the listed effects improve returns for option holders. Given that the people who make decisions about how to return money to shareholders (typically the CEO and board of directors) also tend to be the largest holders of long-term options, I have to wonder whether the ever-growing popularity of stock buybacks has more to do with avoiding giving investors unwanted ordinary income, or with improving the value of options held.

What are financial derivatives?

2009-10-26T00:04:00.001-07:00

This weekend my wife convinced me to see Michael Moore's movie about Capitalism. It had its good points, its bad points, and its silly points. One of the silly points was multiple purportedly smart people finding that they couldn't explain what a financial derivative was.

So I decided to see if I could come up with an easy to understand explanation. And then explain the risks and benefits in (hopefully) common sense terms.

A financial derivative is a contract that has value based on what some other thing might do.

Let me illustrate with a concrete example. A put is a binding contract saying that person A can sell person B something at a fixed price on a fixed date. There is no obligation to sell. But if A wants to sell, B has to buy at that price. No matter what the current price happens to be.

Why would someone buy or sell one of these? Well suppose I bought some stock whose price I think will fluctuate, but which I think will wind up higher. So I think I'll make money, but just in case I can buy puts at somewhat below the current stock price as insurance. If the stock goes down I can sell the stock at the pre-agreed price, and I limit my losses. Conversely the person who sells me the put doesn't think the price will go down either, which is why they are happy to sell me my insurance.

Please note this carefully. Neither the buyer nor seller here believes the put will be used. Therefore it can be purchased cheaply. (This is why it makes a good insurance policy.) Yet the put has value to the buyer and risk to the seller, and therefore money has to trade hands for the contract to be worthwhile to both.

The next thing to notice is that virtually any type of contract you can dream up can be made into a derivative. For instance if you want you can do the reverse of a put. Rather than saying that B has to buy from A at a fixed price, you can say that B has to sell to A at that price. Those actually more popular and are called calls.

Both puts and calls are called options because they give someone the option (but not obligation) to do something at some point in the future. In fact the options that are given to employees are almost always calls. (Almost, but not actually always. One job rewarded me with a contract that was carefully constructed to legally be a bond, and not a call. That was done for tax reasons.)

Now a warning about derivatives. In theory derivatives can be used to limit and manage risk. Supporters generally introduce derivatives by illustrating how all parties to them can use them to manage risks. However they can also be used to create and concentrate risk very rapidly. For instance if you sell many puts very cheaply and then the price of the stock falls, you can quickly be out an insane amount of money. This is not a hypothetical risk. Every so often a rogue trader will accidentally destroy a bank or other large financial institution by making big options trades that go sour. A good example is Barings Bank. Let me stress that this typically happens by accident, not intent. (Usually, as with that case, someone does something stupid, loses money, then keeps doubling up their bet hoping to get out of trouble. If the string of bad bets goes long enough, the company goes broke. Which makes me wonder how many mistakes are made then are successfully corrected without anyone being the wiser...)

So derivatives are contracts. People can buy and sell them. But how should you price them? Well the standard answer for puts and calls is to apply the Black-Scholes pricing model. I won't go into the math, but here is the idea. What they do is construct a portfolio that, managed very carefully, will in the end come to the same effect as a put or call. The price of this portfolio is theoretically the price of the option. Why? Well if the portfolio is less than the option, then you can buy the portfolio and sell the option, manage the portfolio, and you make a guaranteed profit. (Making some combination of trades with guaranteed profit is called arbitrage, and people on Wall St are very, very interested in finding opportunities to do this because arbitrage is free money.) Conversely if the portfolio costs more than the option you can buy the option, sell the portfolio, manage the portfolio and again make money. So in an efficient market the prices have to match.

OK, this is all well and good. Now let's move on to something more fun. Given that the complexity of derivatives are only limited by the imagination of the people writing the contract, here is no shortage of interesting possibilities. Let us focus on swaps.

A swap is just an agreement for two parties to trade things of equal, or close to equal, value at the time of the trade. There are many kinds of swaps traded today. Some are traded with insane volumes. For instance foreign exchange swaps trade money between currencies today in return for a reverse trade tomorrow. This is done to the estimated tune of $1.7 trillion (USD) per day! (The primary purpose is to transfer money at close of business from traders in New York to Tokyo, then from Tokyo to England, then back to New York. That way money can be used to make money 24 hours a day.) But let's take something different. I'll look at an example of two companies swapping mortgages.

Our scenario is that a US company is building a skyscraper in Japan. At the same time a Japanese company is building a skyscraper in the USA for approximately the same price. Neither has sufficient cash on hand to fund this, so each needs a mortgage. (This would likely be done with bonds, but let's not worry about that detail.) But rather than each taking out a mortgage in the other currency, let's say that they enter into a contract to swap the financial obligations - the US company pays the mortgage on the US building, while the Japanese company pays the mortgage on the Japanese building. That's a swap.

Well they certainly swapped something. But what is the point?

The point is that neither wants to have to worry about exchange rates. The mortgage on the Japanese building will be in yen. The mortgage on the US building will be in dollars. The US company has a reliable stream of dollars coming in in their future, but doesn't know how much to set aside to meet an obligation priced in yen. The Japanese company has the reverse situation. It is easier for each to budget and plan for an expense in their own currency, and so they'd each prefer to pay the other's mortgage.

It is important to stress that this is not just a convenience. When the companies go to the banks, the banks will recognize the exchange rate risk and that will affect the rates the companies can get. A US company can therefore get a loan in US dollars at a better interest rate than the Japanese company can. Conversely the Japanese company can get a loan in yen at a better interest rate than a US company can. So after the companies swap mortgages, each gets a better interest rate than they could otherwise get. And since both save on their interest payments, both write down immediate profits from making that swap.

This is one of the most important and counter-intuitive points to understand about financial derivatives. Two companies entered into a contract, and both wrote down an immediate profit. Nothing is faked here. Real money is saved in their monthly bills. That saving can be calculated, and as a result the company is better off. Over time the exchange rates will likely move and as a result one probably makes money and the other eventually loses on the deal. But on day 1, both marked down a profit and the profit is real.

Now if there is one thing that gets the attention of people in finance, it is the ability to manufacture money out of thin air. That apparently happened here. It is therefore worth repeating that this example has been studied to death and generally accepted accounting rules agree that both companies get to write down immediate profits in this case.

Of course you can only set this up if the conditions are just right. These profits aren't particularly easy to find. But once you've seen a way to do it, people can set them up over and over again. And over time the financial wizards found cases where companies could write private contracts and claim an immediate profit. As with the swap described there was inevitably long-term risk associated. But who wants to turn down free profits?

Now here we have a critical danger. With many financial derivatives people enter into complex contracts which, based on some complex theory, they think entitles them to write down a profit. However the investments take time to pan out, and come with significant risks. Worse yet there is the very real possibility that people will write down profits that aren't real due to honest mistakes. And if someone honestly overvalues the value of a particular kind of contract, they will enter into that kind of contract over and over again, writing down fake profits each time. (And getting paid correspondingly large bonuses for finding such great deals!) As a result in the end even the smartest people in the world can find themselves unable to truly measure the risks a given company is taking.

If you think I am exaggerating, go read what one of the greatest investors ever has to say about financial derivatives. There is a section about derivatives that starts on page 14 of this PDF from Berkshire-Hathaway. When Warren Buffett wanted to talk about financial derivatives in 2002, the first point he made was, Charlie and I are of one mind in how we feel about derivatives and the trading activities that go with them: We view them as time bombs, both for the parties that deal in them and the economic system. He then goes on to explain many of the same points that I have. His common sense comments are well worth reading by anyone who wishes to make sense of the recent financial crisis.

Now let's move on to securitization. Securitization starts with a bunch of things with promised future payments that may or may not materialize. The things could be loans on houses, credit card debt, student loans, risky bonds - as long as it is a stream of future payments it is meat for the grinder. Typically each thing is a stream of relatively small payments that lasts for a long time. An issuer takes a whole bunch of these, and puts them together in a deal. A deal likely will have a long stream of big payments. Technically a deal is a very carefully set up corporation whose assets are the things that go into the deal, and whose likely future revenue is used to issue a series of bonds. Each bond is a stream of payments that ends once a particular amount of money has been delivered. The first bonds issued are very likely to be worth full face value, and the last ones are unlikely to make much money. For technical reasons the different kinds of bonds are of interest to different investors.

A financial security is any financial instrument that can be traded. The loans etc that go into a deal can't be traded in any kind of open market. The bonds that come out of the deal can. Therefore this process creates securities out of debt, hence the name securitization.

When the deal is set up right, the loans, etc that go into the deal can be purchased for less than the bonds sell for, and the issuer gets to write down an immediate profit. If the deal is correctly modeled, every purchaser gets a known amount of possible future profit with a clearly understood associated risk. Everyone wins.

Given recent bad press it is worth pointing out that securitization has been around for a while and until recently it worked out well for most of us. Whether or not we were aware of it.

Securitization is really a mechanism for diversifying risk. Before securitization took hold, mortgages would be held by the local issuing bank. This meant that if a local area's economy took a nose dive, the local banks would get seriously hurt and sometimes got wiped out. This would restrict the availability of credit for a long time following, and made eventual economic recovery much, much slower.

Since the 80s things have been very different. Banks now sell their loans on to Wall St, which turns them into securities. This means that if a local economy collapses, like Pittsburgh's when the steel industry left or Michigan's with the auto industry troubles, local banks are much less likely to collapse with it. And they can continue to offer credit, which makes it much easier to recover down the road. Therefore securitization has made local economies much more resilient than they otherwise would be.

But, as we all know, there is a fly in the ointment. In the case of sub-prime mortgages we had bad risk models. Thanks to those risk models companies could write down easy profits while not understanding the size of the risks they were taking on. And in their pursuit of profit, they repeated the mistake over and over again. As a result widespread acceptance of securitization wound up replacing the risk of fairly frequent nasty local economic collapses with the risk of relatively rare nasty global collapses. Luckily with the last one having been averted. (So far, anyways.)

So there we have it. Financial derivatives are just contracts. People enter into them for all sorts of reasons. They are traded in large volumes. They can be arbitrarily complex. They can mitigate or create risk. We have theories on how to price them. When financial derivatives are set up correctly, apparently free money is created. This money comes with associated risk. Honest mistakes about the pricing or risks can lead to disaster. Warren Buffett has written on this topic with extreme warnings about this exact issue. Securitization is a way of bundling lots of little streams of cash into nice bonds. Securitization brought us increased local financial stability, which has been to our general benefit. But systemic mistakes in securitization also brought us a real risk of global collapse.

If you understood that then you likely understand financial derivatives better than half the talking heads you see on TV. And you definitely understand them better than Michael Moore appeared to in his movie.

Progress on Kelly materials

2009-10-24T00:24:00.000-07:00

Not done is nothing taught me a lesson. Publicly saying "I didn't complete this because I lost interest" doesn't feel good and provides motivation to finish. So I've returned to the Kelly Criterion explanation I was working on, added a little bit, and provided return calculator that optimizes.

That may not sound like much, but under the hood it is doing calculus in JavaScript. The generic parts of the implementation are in advanced-math.js. If you like the idea of doing math with closures, feel free to take a look.

Next I need to add some linear algebra, some multi-variable calculus, then implement something that finds maxima using Newton's method on a convex polytope defined in any number of dimensions using a set of linear inequalities. You can think of this as a non-linear version of the problem solved by the Simplex Algorithm. The fun part is that I need it to good enough to find the optimum point in an n-dimensional polytope sitting in n+1-dimensional space. (The actual polytope I will need is defined by 0 ≤ x_i &le 1 for i=0..n, and x₀ + ... + x_n = 1.) That means I have to deal with issues like finding my way back to the region after round-off error takes me away from it.

Oh. And of course I'll do all of this in JavaScript.

Perhaps that can be useful for others? Possibly. However JavaScript has no real equivalent to, say, CPAN, so the effort of putting it somewhere that someone in need can easily find it is beyond my motivation. That's why I did nothing with my port of Statistics::Distributions. But that is a thought for another day.

What granularity for MVC?

2009-10-20T21:21:00.000-07:00

Web developers often try to use an MVC architecture. This means that the business end of a website is divided into models, views, and controllers. Models encapsulate your data, and this is where you are supposed to put business rules like, "You can't take more money out of your account than you have in it." Views are about how to present the data. Ideally the model shouldn't know or care about whether the user is going to look at an edit page or a display page, or look at a page in English or Chinese, those things are presentation logic which belongs in the view. And finally the controller is responsible for the necessary glue between them, for instance to figure out what model and view to use for a given URL for a given person.

There is widespread agreement that this is a good thing in the principle, but things get messy when you go to the details. Why? Well in part because people often break the rules they agree to. Business logic creeps into views, models format text that is just passed through the view, and the controller gets into everyone else's business. After a while you're doing MVC in name only. But also there are honest disagreements about a myriad of issues including where exactly to draw the lines between classes, whether to add in helper classes and if so which ones, how complex views should be, and so on.

I'd like to draw people's attention to a design question that few in the web world think about, namely how granularly we should do MVC.

Traditionally in web development people assume that the right level of granularity is the web page. You get a URL, that goes into a controller, sometimes there can be a cascade of routing decisions, and eventually you get to the code that fetches an appropriate model and view, then puts them together to get a web page. Sounds pretty straightforward and flexible.

But what if we have a complex web page? What happens if we add independent sections on a web page that have little to do with each other but need different dynamic information? Alternately how do we handle displaying many related objects for editing? In my experience the natural tendency is for the controller to become a mess. And the view has to be synchronized with the controller, resulting in fragile dependencies.

For an alternative, let's look back at the history of MVC. Originally MVC arose in the Smalltalk world. And the appropriate granularity for MVC wasn't a whole page, it was a component. You'd write Smalltalk applications and every little dropdown could have its own model, view and controller. Which could all talk to each other, and permitted good decomposition of very complex behavior. This idea has continued in the world of GUI programming. For instance an object-oriented Flash program is likely to be developed on a similar paradigm.

What happens if we take this idea back to serving a web page? Well obviously you need a lot more models, views and controllers. However it isn't as bad as it seems because they also become much simpler. Plus it becomes easier to reuse components on multiple pages through a site. The whole page can still become complex, but MVC is a good way to manage it. After all our design is now closer to the original versions where MVC first proved itself useful.

And as a final thought, ever more complex and dynamic client side interfaces make web pages into something more like any other GUI. Which means that at some point there are likely to be real advantages to using component based MVC within the client interface. But this idea is more natural to try when there is no impedance mismatch between your server-side design and your client-side design. Which suggests that you should start with component-level MVC on the server side, and then "lift" components to dynamic AJAXified components one by one as appropriate. Therefore starting with a component-level MVC gives you more flexibility to later evolve a complex, dynamic client-side interface.

Obviously I wouldn't recommend that every developer rewrite their current websites based on this idea. However I think that the benefits are interesting enough that I'd recommend that developers play with this idea, so that they can make an informed decision about whether to try this approach on future websites.

Why did the Neanderthals die out?

2009-10-19T20:41:00.000-07:00

Yesterday's entry reminded me of a Scientific American article last summer about Neanderthals. Reading it I was reminded again of the curious resistance that anthropologists seem to have towards accepting lessons from biology.

But before I get into that, I need to give some background. The theory of punctuated equilibrium, put forth in the early 70s by Niles Eldredge and Stephen J. Gould. This theory held that new species to develop fairly quickly in small isolated regions, then if successful spread over a wide range and change only slightly until they are replaced by another species. Biologists long ago accepted this theory, and it is backed up with evidence ranging from observations of speciation in action in fish in Africa, to Eldredge and Gould's initial example of a type of trilobite that went through 2 speciation episodes in several million years, both of which we are lucky enough to have fossils from the speciation episode in a single mine each. So in general this is what we expect speciation to look like - something that happens in a small area somewhere which bursts out.

Anthropologists have traced a number of types of humans that existed in the past. Of particular interest are modern humans. Two theories existed on that. One is that we evolved in Africa, then spread. The other is the multi-regional hypothesis, which is that we evolved in several regions at once, and possibly mixed in other kinds of humans. Thus leading to side questions such as, "Did we interbreed with the Neanderthals?"

There is no question which version evolutionary theory supports. Punctuated equilibrium unambiguously says that we should expect our species to have started in a small area and then spread without significant (if any) mixing. I remember reading an essay by Gould many years ago that laid this out pretty bluntly. However anthropologists don't like hearing things like, "We can tell you what probably happened with humans based on general principles established with trilobites and fish." And so the active debate lasted for decades on this point.

Reading the Scientific American article I found that the question has now been tackled with the aid of DNA analysis. Based on genetics we now know that the evolutionary biologists were right, and the multi-regional hypothesis is wrong.

But now we have a new question that anthropologists are putting energy in. What killed the Neanderthals?

First let's list some facts. The Neanderthals survived for over 150,000 years. They lived through multiple ice ages. We have evidence of them hunting both large and small prey, with a preference for big game. We coexisted with Neanderthals for thousands of years. There are minor differences in technology (for instance our ancestors had better stitching on their clothing), but nothing specific. Our ancestors specialized in smaller prey (though we were happy to chase big game after the Neanderthals died out). Yet by the end of the ice age the Neanderthals had died out, both in places where our ancestors were found, and in places we weren't.

Hence the mystery. Why did the Neanderthals die? The long (apparently peaceful) coexistence suggests that we didn't directly kill them. They survived previous ice ages. We survived in the same places at the same time. And they should have been able to adapt and use our strategies - after all they did it in previous ice ages. With such small differences, why did they die out?

The article was very good. This was all laid out very well. Along with lots of detail about relative energy requirements, technology, and so on. But I found myself asking whether they had bothered asking the ecologists.

If you study ecology, one of the basic principles is that related species avoid directly competing. How so? Well when the related species is not there they broaden their niche, and when the related species is then each species defines its ecological niche closely enough that they don't directly compete.

Let me give an example. Originally I read this with 3 species of birds, but that was many years ago so I'll simplify to 2 species of birds that I'll call A and B. These birds all hunt insects, and have the choice of hunting them in the forest or by lakes and streams. When only one species is there, they can be found in both places. But if both species coexist in a region, species A does all of the hunting by lakes and streams, while B hunts only in the forest, and so they avoid directly competing.

Why do they do this? Well A is better than B at hunting in open water. B is better than A at hunting in the forest. When only A is around, therefore, the insect population in the forest rises until a given bird is equally well off in the forest and by the water, so you find birds in both places. With B the reverse happens. But when both species are present, then the A birds lower the population of insects near rivers and streams to the point where the B birds will go hungry if they try to feed there. And conversely the B birds keep the insect populations in the forest low enough that the A birds will go hungry if they try to feed there. So A and B coexist, and specialize.

This coexistence can last indefinitely. However there is very real competition here. The presence of the other species narrows the niche the birds live in, and result in fewer of A and B. Each population is now more precarious, and less able to adapt to environmental changes. Thus the general principle that related species tend to coexist and compete through specialization.

Now let us return to the mystery of the Neanderthal extinction. In the absence of our ancestors, the Neanderthals were willing and able to use a combination of food gathering strategies. However once our ancestors arrived, both groups specialized. The two had no difficulty coexisting for thousands of years, but both populations were more precarious than they had been. In particular the Neanderthals were no longer able to use our niche because our ancestors were occupying it more efficiently than they could. Not much more efficiently, perhaps, but even a small difference is the margin between survival and death.

And so it went. Where we were, our presence forced the Neanderthals to specialize. Where we weren't, our absence was evidence that our niche wasn't a good fit to that area, so the Neanderthals there would be unlikely to benefit much from being able to adapt freely to what worked best. Which left the Neanderthals unable to adapt to climate change, and eventually left them dead.

Based on the history of the multi-region hypothesis, I predict that anthropologists will eventually acquire overwhelming evidence for a detailed version of that scenario. But they are likely to take decades to get their. And when they do they won't notice that they could have gotten there faster if they were willing to accept general principles that are already well established in birds, salamanders and the like.

Thoughts on Youtube

2009-10-18T15:51:00.000-07:00

I think my son was 2 when I discovered this. To to youtube. Type in a random thing of interest, like "fire truck". And then watch videos about that topic.

Of course things have progressed since then. My son is almost 5 now. And he just discovered dinosaurs. So we watched some dinosaur videos yesterday and discovered the series of "Walking with ..." series. And as these things go, I wound up going to Amazon this morning and ordering several series. Walking with Monsters, Walking with Dinosaurs, Allosaurus, A Walking with Dinosaurs Special, Chased by Dinosaurs and Walking with Prehistoric Beasts. (Yeah, overboard. But I know my son and it is a good price.)

Yes, I know how much they made up. But there is a lot of good science in there as well. And I know how much my son will love it.

Now lots of people, including many copyright holders, think that youtube is a horrible violation of copyright. Technically they are right. But I believe that there are plenty of people like me out there, and the net effect is more people discovering then buying things they like. Which is good for copyright holders.

And it is a belief backed by some evidence as well. For instance look at the Baen Free Library. Put books online for free in multiple formats and what happens? Sales go up!

Literature not popular in the USA

2009-10-15T22:26:00.000-07:00

I grew up in Canada, but now live in the USA. In the grand scheme of things, Canada and the USA are very, very similar. Yet even so there are odd differences. One of them being that there are books which are popular around most of the English-speaking world (and often farther) that are popular in Canada which nobody in the USA has heard of.

In some cases it is obvious why. For instance consider Yes, Minister which is one of the most brilliant descriptions ever of how bureaucracies work. However it assumes you understand the British parliamentary system. This political system is used in most places that were part of the British empire, including Great Britain, England, India and Australia. Therefore there is little surprise that while I've met plenty of fans of the series from all of those countries, virtually nobody in the USA has even heard of it.

In other cases it is less obvious to me why it is so. Consider, for instance, the comic series Asterix and Obelix. Originally written in French, the series has been translated into many, many different languages and is loved by children around the world. I've talked with fans from France who didn't know it was translated, fans from Spain who didn't realize it was in English, fans from India who didn't know it wasn't originally written in English, and so on. Yet I think I've talked with a grand total of one person from the USA who had heard of it - and I believe he learned about it while traveling through Europe!

Why would this be so? I suspect that some marketer looked at the series and said, "A comic series set in ancient France shortly after the Roman invasion? No American kid will ever go for that!" And so it was never marketed here.

I think this was just a bad decision. Certainly no American that I've lent it to (including several children) had trouble with the material. They all loved it. Besides, the books are meant to be appreciated on many levels. I can read it to my 4 year old son and he laughs at how Obelix accidentally breaks doors when he knocks at them. I loved it at 9 even though I missed most of the jokes embedded in the names. And my kids' babysitter wants me to buy more so she can read them. It really is a series that grows with you as you learn enough to understand more of the jokes.

In short you'd think that any series that is popular around the world in multiple languages is worth trying in the USA as well. But apparently publishers don't think that way. And so American audiences miss out on some really great works.

But this makes me wonder. I'm aware of these works because I grew up in Canada. But the USA and Canada are very similar. What popular works would I love that I don't know because they were never marketed in either country?

Why do we use checks?

2009-10-14T22:14:00.000-07:00

Think about this scenario. I hand you a piece of paper. This piece of paper has all of the information you need to take any amount of money directly from my bank account. It has written on it the amount I wish to give you. You assume that I can indeed give you that amount, and I assume that you will not steal from me.

Does this seem like a sane thing to do? Just think of all that can go wrong here. You could steal from me. You might be honest, but someone else could take the information from you and steal from me. (This happened to Donald Knuth - people would get a check from him, scan them and post the proof, then scammers looking for pictures of checks online found them and forged checks. Knuth's reward checks are now no longer valid because of this.) I may not have money in my bank account. Perhaps I don't now, but I hope to. Banks actually give a little leeway called "float" to as a convenience to their customers. But this can backfire. If someone arranges things carefully they can bounce a series of payments between accounts, get the bank ready to say that all of the accounts have money, withdraw the money, and leave the bank holding the bag!

Obviously I've just describe a check. But, you ask, what is the alternative? Funny you should ask. Several months ago I went to the Netherlands. They don't have checks there. What do they do instead?

Well if you're my utility company and you want me to pay you, you send me the information I will need to deposit money in your bank account. I go to my bank and transfer money there. You get notified when it arrives. The technical name for this in English is giro transfer.

Now stop and think about how many problems this solves. I never hand out my bank details to anyone. You never have to deal with a bounced check. There is no possibility of anything like a kiting scheme. And the only practical change is that instead of my giving you information that can be used to draw from my account, you give me enough information to put money into your account.

The moral is that checking systems are fundamentally flawed. The design of a giro transfer system is fundamentally sound. Unfortunately tradition is set so that checks are here for a while to come. And people are honest enough that the problems don't generally rise to the point that would make people object. Sure, the security problems are obvious when you think about it. But as always when people aren't being bitten by the problems, people forget about the security implications.

Of course checks are losing popularity between credit cards, easy cash withdrawals, and automated payments. So there is hope that some day they will be seen to be superfluous and will eventually be abandoned. In the meantime checks serve as yet another example showing how little we care about security, even when it comes to our money.

Limitations of Capitalism

2009-10-13T21:07:00.000-07:00

I've commented that Capitalism is the most effective way known of getting people to do what it gets them to do. However you have no control of what that is. I thought I would expand on that. In that light, here are some of the major flaws of capitalism:

There are big problems we want solved that have no associated revenue. For example consider the problem of taking care of the million or so untreatable schizophrenics in the USA. (I estimated that number by knowing that the USA has 300 million people, about 1% of people have schizophrenia at some point in their life, and of those about 1/3 spontaneously recover, 1/3 respond to treatment, and 1/3 are not treatable.) Left to their own devices these people are unable to function. No matter how loving their families, family resources get severely strained supporting them. And there is no realistic hope of integrating them into society. As a society we do not wish to kill them and do not want them starving to death, so we need to take care of them. The magnitude of the problem is more than charity can support, so this is a valid role for the government. (Total charitable giving in the USA is about $300 billion/year. If we assume that institutionalizing a person in a place with medical care costs $30,000/year, then that would suck up 10% of all charitable giving. On just one cause.)

Capitalism ignores external costs. A rational profit seeking individual who can acquire revenue and leave costs for others, will. A classic example is pollution. Pollution is a diffuse cost that is shared by an entire community and is mostly not experienced by the polluter. Therefore polluters have little incentive to reduce pollution. Government regulation can solve the problem by artificially providing the incentive.

Capitalism ignores external benefits. What do universal education, basic research, and sound policing have in common? The providers of the benefit cannot readily recoup the benefit they provide. Poor kids who will do better with an education are poor right now, their parents can afford that education. It is the nature of research that it proceeds best by sharing ideas, but when ideas are shared then there is generally little or no connection between the people who did the basic research and the people who commercialize it. Law and order is great, but the people who enforce the law aren't generally the people who build businesses that can prosper because they exist within a well-regulated society. Private markets therefore are poor at allocating any of these things.

Capitalism has perverse incentives with asymmetric information. As an individual you are in an extremely poor position to judge whether cooks at a local restaurant wash their hands, the security of a safe, or how effective a medical treatment is. So people judge on the basis of things they can see that they hope are a good proxy for what they want to know. Such as the quality of the decor, how sturdy the safe looks, and the presentation of the person selling you the treatment. But you can fake those without providing the attribute people really want. As a result private markets left to their own devices will provide unhealthy restaurants, insecure products, and unreliable treatments. (Government is good about public health, only rarely intervenes in security, and frequently doesn't pay attention to effectiveness of treatments. If you know where to look, this shows.)

None of this is to say that we want or need government intervention in everything. As I said, capitalism is incredibly effective at generating economic activity, much of which benefits us all. Besides, government has its own characteristic failure modes, which are at least as bad, if not worse, than the ways that capitalism fails.

However when there is a solid justification for government intervention, usually one of these failure modes in free markets is behind it. And if you can identify no specific reason why free markets should do a bad job in that area, then odds are good that the effects of the government intervention is somewhere between ineffective and bad.

Not done is nothing

2009-10-05T21:50:00.000-07:00

I have a problem. I'll start personal projects, get to the point where I am satisfied that I could finish it, then don't. Sometimes it takes quite a bit of work to get there. For instance consider this explanation of the Kelly criterion for optimal betting patterns. I did quite a bit of work on it, created a calculator that could be useful. In addition I began a library to let me do basic differential calculus and linear algebra in JavaScript, and worked out exactly how to build numerical approximations of optimal betting patterns. Then I got distracted.

I began this blog. Then I got distracted. OK, it is the nature of blogs to never be done, and I certainly couldn't keep up the pace I started with, but I still could have done more.

The side project I now spend time thinking about is a design for a web development platform that I think would be really powerful. It could be a revolutionary way to work..if it ever gets completed.

I have no shortage of excuses. I have very little time for any side project once you subtract working full time, helping my ex-employer on the side, and taking care of my young children. I am interested in a large variety of things, and so have no shortage of distractions. Furthermore I really do constantly learn new stuff.

None of this is to say that I can't finish things. I am great at finishing things when I have an external reason to. I finish things for my employer all the time. I put a lot of work into my Effective A/B Testing tutorial, and a lot of people have been happy with the result.

However when I am doing something out of personal interest, I lose interest once I've completely learned how to do it. And as a result nobody else benefits from my efforts.

In short, I'm pretty much the opposite of being a member of The Cult of Done. :-(

What makes it science?

2009-09-29T20:02:00.000-07:00

Many people draw a division between the hard sciences and mathematics on the one hand, and everything else on the other. The implication being that one side is "really" science and the other is not. Which claim to upset members of "soft sciences" like psychology.

This post explores the question of how justified this division is.

The starting point for my thinking is something I learned from the wonderful essay In Oldenburg's Long Shadow about the serial pricing crisis.

To understand the serial pricing crisis you must first understand the science citation index. This is nothing more or less than an index of how many times a given paper has been cited by other published papers. When you look at it, you find that papers that appear in some journals are consistently cited more often than others. This is a direct measurement of how influential the journal is, and leads to the impact factor. When you look at journals across math and the hard sciences you find that there is a hierarchy of journals. At the bottom you have low impact journals that publish only for a small niche. But key papers from that niche are published in more prominent journals that are looked at by people in a wider range of subjects. And this goes all of the way to the highest impact journal of all, Nature, which is where people try to publish the absolute best work across all of math and the hard sciences. It doesn't matter whether you're a physicist or a biologist, the absolute best research goes to Nature.

So from the science citation index we can measure the impact factor of a journal, which in turn tells us its value to researchers. The value to researchers told publishers what universities were willing to pay, and so publishers have been steadily increasing the price of the most important journals. This costs universities more than they are happy with, and so is called a crisis. Librarians call journals "serials" (because you get a series of copies of a journal), hence this crisis is called the serial pricing crisis.

Now let's look at this in reverse. Across all of math and the hard sciences it is possible to make a somewhat reasonable comparison of how important any given paper is, and how good any given journal is. Furthermore there are small groups of editors whose judgment is regularly trusted to compare the best papers from different areas of science and select which are worthy to be in their journals. In the extreme example, the editors of Nature are trusted to draw comparison across all of the hard sciences. And for the most part, scientists agree with these decisions.

When you think about it, it is truly remarkable. It implies that there is a relatively well shared concept of relative value across all of the sciences. Of course people in the hard sciences seldom remark on it, it is just how things are.

To see how remarkable it is, compare with the humanities and social sciences. They have no such hierarchy. Instead of one grand hierarchy you get independent clumps of researchers who talk to each other but not the other groups. And they find this so natural that I have seen social scientists express disbelief that, say, a physicist in fluid mechanics can hear a key result in particle physics and will know that it is important. But it is true. If you ask one physicist, "What are the 10 most important results in physics in the last 30 years" and take that list to another, the other physicist will agree that those are all important. If you ask a psychologist for a similar list and take it to another, the other is likely to not even recognize many of the items.

What is going on here is a confirmation of one of Thomas Kuhn's key claims in The Structure of Scientific Revolutions. Which is that in a mature science (his term) researchers have come to share a paradigm about what would be progress. When a paradigm has become so compelling that virtually all researchers in the area accept it, then people who are not in that field can see the agreement that progress is happening. When no paradigm can compel general acceptance, then from a distance all that is visible is confusion.

Kuhn is careful to point out that within the field there will be groups of researchers who are doing good work and making progress. This is certainly is true. For instance I've brought up psychology. Yet if you read books like Parenting From the Inside Out you will find that solid research is being done, that comes up with valuable information. (I highly recommend this book to anyone, parent or not, who is willing to work through it carefully.) But the problem is that the case for this line of research is not compelling enough to convince other psychologists that this is the right way to try to understand the mind. So from a distance there isn't a clear impression of solid progress being made.

Therefore the hard/soft science division boils down to shared paradigms. In the hard sciences certain lines of research have become so compelling that everyone agrees that they are the right way to go. Because of this agreement, people in nearby fields get a clear picture of what progress looks like in that field. With clear pictures of what progress looks like in multiple fields, the ground is set for making comparisons between fields, which has evolved into a reasonably well shared value system across the entirety of the hard sciences.

The soft sciences share none of this structure. As a result there is no shared agreement within the soft science about what is important, let alone a shared agreement on the relative importance of different areas of science.

To close I would like to illustrate how much shared agreement there is within the hard science about what progress looks like. I'll do this by giving my personal top 10 lists of scientific advances in each century since science began to take off in the 1600s. I haven't tried to put them in any particular order. (They often are somewhat chronological.) While people may quibble with some of my specific choices, people who are well versed in the hard sciences will generally agree on the importance of these items.

1600s
1. Objects of different mass fall at the same rate (Galileo, physics)
2. Telescope used for astronomy (Galileo, astronomy)
3. Kepler's laws for planetary orbits (Kepler, astronomy)
4. Circulatory system accurately described (Harvey, biology)
5. Microbes discovered (Leeuwenhoek, biology)
6. Hooke's law of elasticity (Hooke, physics)
7. Newton's laws of motion (Newton, physics)
8. Newton's law of gravity (Newton, physics)
9. Speed of light first measured (Ole Römer, astronomy/physics)
10. Calculus (Newton/Leibniz, mathematics)

1700s
1. Lightning explained as static electricity (Ben Franklin, physics)
2. Fluid mechanics began to be analyzed (Bernoulli, physics)
3. Linnaean taxonomy system created (Linnaeus, biology)
4. Halley's comet's orbit predicted (Halley, astronomy)
5. Coulomb's law for attraction of electric charges (Coulomb, physics)
6. Oxygen discovered (Priestly/Scheele, chemistry) leading to the rejection of pholostigon (Lavoisier)
7. Uranus discovered (William Herschel, astronomy)
8. Conservation of mass demonstrated (Lavoisier, chemistry)
9. Stability of solar system confirmed (Laplace, astronomy)
10. Gravitational constant measured (Cavendish, physics)

1800s
1. Fourier series discovered, used to analyze heat transport (Joseph Fourier, mathematics/physics)
2. Ice ages discovered, theory of The Flood rejected (Louis Agassiz, geology)
3. Central Limit Theorem aka The Bell Curve (de Moivre/Laplace/Galton/Lyapunov etc, statistics) different versions were proven at different times, and Galton was making good use of it years before it was finally proven in generality by Lyapunov
4. Thermodynamics (many people starting with Carnot, physics)
5. Conservation of Energy (Joule/Mayer, physics)
6. Descent with Modification aka Evolution (Darwin, biology)
7. Germ theory (Pasteur, biology)
8. Atomic theory (Avagadro/Loschmidt etc, chemistry)
9. Maxwell's equations of electromagnetism (James Maxwell, physics)
10. Periodic table (Mendeleev, chemistry)

1900s
1. Relativity (Einstein, physics)
2. Radioactive Dating (Ernest Rutherford/Bertrand Boltwood, physics)
3. Quantum Mechanics (Heisenberg/Schrödinger, physics)
4. Gödel's Incompleteness Theorem (Kurt Gödel, mathematics)
5. Hypothesis Testing (Ronald Fisher/Jerzy Neyman/Karl Pearson/Egon Pearson, statistics) - Egon was Karl's son
6. The Structure of DNA (Watson/Crick/Franklin, biology)
7. Continental Drift (proposed Wegener and confirmed by lots of people at once, geology)
8. The Big Bang (Georges Lemaître/Edwin Hubble, astronomy) general acceptance followed the discovery of the CMBR by Arno Penzias and Robert Wilson
9. Synthesis of the Elements in Stars aka B²FH (Geoffrey Burbidge/Margaret Burbidge/William Fowler/Fred Hoyle, astronomy/physics)
10. Standard Model (Sheldon Glashow/Steven Weinberg/Abdus Salam, physics) tens of billions of dollars have been spent verifying this theory!
2000s - There is likely more disagreement over these

Teaching linear algebra

2009-09-28T23:01:00.000-07:00

In a recent Hacker News post I made reference to an interesting teaching experience I had in the mid-90s. This is a longer explanation of the same.

I was a graduate student in math at Dartmouth College. I wound up teaching an introduction to linear algebra course that was also the first course where students were asked to do proofs. The class was somewhere in the range of 15-20 students. If I remember correctly, this was in the fall of 1996.

In preparation for the class I set myself goals around how well the students would learn the material taught. After some thought I settled on four ideas that I would use:

Homework not present at the start of class would not be accepted. However students were only graded on the best 20 out of 27 possible homework sets.
All homework sets were cumulative. Generally 1/3 was the current day's material, 1/3 from the last week, and 1/3 from anywhere in the course. Those thirds were in increasing order of difficulty.
Every class would start with a question and answer session to last no less than 10 minutes.
Every student could expect to be asked at least one question every other class.

These ideas may seem odd, but there was a method to my madness. Here is each idea explained.

Homework not present at the start of class would not be accepted. However students were only graded on the best 20 out of 27 possible homework sets.

The point was to make sure that class started on time, with everyone ready to pay attention for question and answer time. I also didn't want to deal with people doing homework during lecture, evaluating sick excuses, etc. The leniency of not having to turn in 7 homework sets compensated for the rigidness of the policy. And cumulative homework sets meant that I didn't have to worry about students not practicing any given day's material.

This worked even better than I hoped. The downside was that I had an argument on the second day when someone came in 2 minutes late and was not allowed to turn in his homework. But the first complaint was the last, and the students liked the freedom to decide when something else took precedence over doing homework.
All homework sets were cumulative. Generally 1/3 was the current day's material, 1/3 from the last week, and 1/3 from anywhere in the course. Those thirds were in increasing order of difficulty.

This was the most important idea I wanted to try. I had long been aware that research on memory had demonstrated that when you're reminded of something as you're forgetting it, it goes into much longer term memory. As a result periodic review at lengthening intervals is very effective in increasing long term recall. A typical effective study schedule being to review after half an hour, the next day, the next week, then the next month.

Now of course you can tell students this until you're blue in the face - but they won't do it. However when the study schedule is disguised as homework, they don't have a choice.

This really seemed to work. What I noticed on tests is that students were noticeably shaky on material they had learned in the previous week, occasionally didn't remember stuff for a half-month before that, but absolutely nailed every concept that they'd first learned at least 3 weeks earlier. I credit the forced review schedule from cumulative homework sets for much of that.
Every class would start with a question and answer session to last no less than 10 minutes.

For me this was the most important part of the class. The questions that came up in this session were my opportunity to refresh people on what they were forgetting, and were how I kept track of what topics should come in for more review on future homework sessions. Given my knowledge of how critical review is to learning, I honestly felt that time spent answering questions was more valuable than lecture. As long as there were questions, there was no maximum on how much time I was willing to spend on this.

Of course the challenge is getting students to ask questions. My strategy was simple: I told them that someone will ask questions and someone will answer them, but they don't want me to be the one asking questions. On the second day nobody asked me any questions and I had to demonstrate. I picked a random person and asked her to explain a key point from the first day's lecture. She couldn't. I asked another student the same question. Again difficulty. I asked if everyone was sure that they had no questions. Someone asked me the question that I had been asking everyone else. I answered the question, answered the follow-up, and the point was made. I never again had to ask a question during question and answer period. :-)
Every student could expect to be asked at least one question every other class.

My goal here was to be sure that every student was awake and following the lecture. It was never my goal to embarrass anyone or put them on the spot. To that end I developed a rhythm. Every few minutes I'd stop, say, "Let's make that a question," ask the question, pause so everyone could think through the answer, then ask a random person the question. I made sure to rotate people around so that everyone got their turn fairly.

The questions I'd ask were always straightforward. They were things like, "What is the result of this calculation?" Or, "Why is this step OK?"

I treated failure to get the answer as my failures, not theirs. If they couldn't get the answers then they weren't following the lecture, and I needed to slow it down, figure out the rough spots, etc. It might seem that the constant interruptions were slow. But I found that having everyone pay attention more than made up for it. The class as a whole moved as fast as any other class - but with far greater comprehension. And the interactivity made the class become very open about asking questions.

As a bonus I managed to convince the entire class that taking notes was not worthwhile. I learned this lesson about math in first year undergrad. What you do is read ahead in the textbook. If you really want a set of notes, you can make them from the textbook before class. Then show up at class having read the day's material and ready to pay attention. Then if anything that the professor says doesn't make sense to you when you're paying attention and have already read the day's lesson, then ask the question then and there. If you don't understand it, then probably nobody else does either. Add to that periodic reviews, and you'll have a huge edge in any math courses.

Nobody ever believes that that works. But this class had no choice because there is simply no way to take notes and pay attention at the same time. Which meant that the note takers couldn't answer questions. But within a few days they learned to not take notes, and I believe did much better for it.

So how well did this package work? As far as my goals were concerned, much better than I had dreamed possible. What really brought this home was the final exam. Based on class performance I drew up a test that I though was a fair test of what I thought they understood. I showed it to some fellow graduate students. They thought I was crazy. They thought the class would bomb, and were willing to bet me on whether anyone would get the bonus question.

The class aced the test. That bonus question? 70% of the class got it. I don't remember what the bonus question was, but I do remember another one that I thought was cute. It went like this. Let V be the vector space of all polynomials of degree at most 2. a) Prove that d/dx is a linear operator on V. b) You can put a coordinate system on V by mapping p(x) to (p(0), p(1), p(2)). (Please imagine that flipped 90 degrees so it is a column.) Find the matrix that represents d/dx in this coordinate system. My fellow grad students got me worried that this might be too advanced for an introductory linear algebra courses. But I needn't have worried - the only significant errors were minor arithmetic mistakes in the calculation. And I think I dinged someone for not having enough detail in the proof.

Furthermore I was lucky enough to talk to some of my students about the experience a few months later. The general consensus was that the material really stuck. Furthermore nobody studied for the final. No joke. As one girl said, "I tried studying because I thought I should, but I gave up after a half-hour because I already knew it all." That is how I think it should be - if you study properly through the course, then you won't need to study for the final. Because you've already learned it. And you'll have a leg up on the next course because you still remember the material that everyone else has forgotten.

So were there any downsides? Unfortunately there were some big ones. I had set goals around learning. I failed to set any around happiness. Having to pay attention during class was hard on the class. Also it motivated them to work hard. Since everyone worked hard and they thought that I was going to grade them on a curve, there was a lot frustration that they wouldn't properly be recognized for their work. (In fact I gave half of them A's in the end.) This frustration showed up the teacher evaluations at the end of the course. :-(

Therefore if I had to do it over I'd ask somewhat fewer questions, hand out a lot more compliments, make it clear that I would not grade on a curve, and if they performed anything like that first class, I'd be even more liberal with good grades. Of course the point is moot since I've found myself profitably displaced from math to software development. But if anyone decides to replicate my experience, I'd recommend paying more attention than I did to those issues.