Our most popular drug is made of sugar, fat and salt

On NPR not long ago I heard an interesting thought.

Suppose a hundred years ago that someone wanted to eat chicken. They had to go out, grab the chicken, cut its head off, pluck it, clean it, then cook it. That was just what it meant to eat chicken. Today that seems crazy. But in a hundred years that is how cooking will look.

Indeed, that is true for a lot of people. Just think how many people eat out more often than not. Or eat packaged meals. Cooking takes work, and most of us don't want to put out that effort if someone else is willing to do it for us.

On the surface that is quite reasonable. And I will confess to succumbing to the temptation quite a few times. I'm no food snob, and don't particularly enjoy cooking.

But all that notwithstanding I think that most of us should cook more. You'll be healthier. You'll eat better food. And it needn't take as much energy as you think.

Let me start with the health issues. My source here is The End of Overeating by David Kessler.

The problem that every commercial food distributer faces is getting people to buy their food. Preferably frequently and in quantity. People have a lot of reasons for buying food, but the three most important are named sugar, fat and salt. Individually we are attracted to each, however we reach a point where we saturate on any one of them. If you combine 2 of those, however, we are attracted to much higher concentrations than we are if you separate them. And combining all 3 hits a sweet spot where we want far higher concentrations than we would accept independently. Not only do we crave higher concentrations, but we experience a pleasurable rush of brain chemicals. In the pursuit of that pleasure we routinely overeat past the point of satiation. Then want to eat that way again when we recover. And this is why you're fat.

This principle is simple, effective, and well understood through the food industry. Whether you're a manufacturer designing a new kind of cheese puff or a restaurant trying to build a clientele you have to do the same thing. Deliver the magic combination of sugar, fat and salt and you can succeed. Fail to, and people will inevitably flock to the competitor who is willing to do it. The food industry has heard and obeyed. Whether it means giving us salads that are mainly a vehicle for delivering dressing (a base of oil with sugar and salt), or offering apparently healthy chickens that have been preprocessed with the injection of a syrup with sugar, fat and salt (this is why it falls apart so easily in your mouth) all of our processed food is a drug delivery mechanism.

The phrase "drug delivery mechanism" may seem overblown. It is not. The effects on our brain chemistry really do resemble an addict getting drugs. And measurements in animals of how motivated they are to get the reward show that modern processed food is only slightly less motivating than cocaine

Obviously there is some variation. But this is one of the prime reasons that obesity has expanded from being a fairly rare disorder to being about a third of the population. And today obesity is seen in ever younger people, and is more and more extreme. It would seem that the drug affects different people differently, and you can see from your waistline how much you are affected. (I'm hit about average, I weigh an extra 20 pounds or so.)

Of course this drug isn't exactly good for our bodies. It is no accident that the leading diseases tied to modern lifestyles are heart attack, stroke, diabetes and hypertension - all of which are tied to what we eat. Not that long ago I talked with someone who had a heart issue and just went to a cardiologist for the first time. The first lifestyle question he asked was, "How often do you eat out?" Note that where you eat out doesn't matter, because the entire industry has the message. They all know the game, and all are going to deliver the same drug. Ditto the manufacturers of processed food.

However if you go and cook food at home, you don't need to worry. You won't hit the sweet spot very easily because you lack the equipment to do it. You'll actually have to chew your chicken because before cooking you didn't break down the internal structure and fill it full of sugar, fat and salt.

The other interesting thing about cooking food at home is that it tastes better. Let's be clear, you won't crave it as much. If you've got one plate with fish you grilled at home, and another with potato chips, it will be the chips that you can't keep your hands off of. However the fish will be more enjoyable.

Let me give a concrete example. When I married my wife, her family was into pancakes made from mix. I've noticed two things about pancakes made from mix. The first is that they don't taste very good. The second is that they don't save you much work. Seriously you can't make the pancake until the pan is hot, right? Well it takes about as long to make the batter as to make the pan hot, and you do those at the same time. Therefore the mix didn't save you time!

Now when I tell people this, many claim that the pancakes they make from a mix taste great. And I tell them that they think that only because they've never had a decent pancake. Try that recipe, then come back and compare. (A funny story about that particular recipe. I had a co-worker who I gave that recipe to who it turned out had a friend who had been raving about my sister's pancakes for 30 years but didn't know how to make them! My friend was happy to be able to pass it along.)

All that said, when someone who doesn't cook faces cooking, it is scary. You don't know how anything works or what anything does. In reality all you need to do is get a recipe, follow directions, and it tends to work out. When you get brave you can experiment. Sometimes the experiments flop. But even that isn't a bid deal.

For instance yesterday I made turkey soup. It was going great until I added noodles. I envisioned firm noodles, and so it was at dinner because I put them in at the right time. But the noodles in the pot didn't stop cooking. So now instead of turkey soup I now have soggy turkey casserole. However it still tastes good and is still good for us. And I won't make that mistake again next year!

So stretch yourself. Look for an easy recipe and cook a meal. Even if you never come to enjoy cooking particularly much, you may find it worthwhile. And who knows, perhaps you'll turn out to be someone who enjoys cooking. Try it!

Work optimally, not long

It has been a long time since I entered anything here. But I'm in the mood. I've cooked, eaten, and even done cleanup after my turkey dinner. If you're uninterested in hearing about cooking turkey, skip the next paragraph.

This time I tried dry brining the turkey. That was surprisingly easy. A few days ago I filled a small bowl with 1 tbsp of salt per 5 pounds of turkey (that's 6.5 ml salt per kg), added some spices for taste (I did sage and a ground up bay leaf), mix, then spread that over my freshly bought turkey, put a plastic bag over it, then let it thaw in the fridge. Today I just roasted it normally. Based on past experience I checked regularly to baste the turkey, but it really wasn't necessary since the juice stayed in the turkey. Tomorrow I'll get supplies, then make turkey soup, which for me is the best part of having turkey dinner.

---

Enough about food. What I want to write about today is something that we all know in theory, but far too few of us act on. That is that it is better to work at an optimal, sustainable level than it is to work long hours. Certainly this is true in software, and likely in many other fields as well.

More precisely I would give the following rule. If you see a developer working more than 40-45 hours per week, ask yourself two questions:

1. Will this effort be sustained less than 3 weeks?


2. When it is over, will this person get corresponding downtime?

If the answer to either question is "no", then I strongly recommend looking at the schedule and figuring out what deadlines to slip immediately to get people back to reasonable schedules. Because you're going to compromise something, and choosing up front is much better than making it by burning people out and getting sub par work.

The reason for this rule is that people have a maximum sustained throughput for creative work when they are working somewhere between 35 and 45 hours/week. (For a reference on that see Rapid Development by Steve McConnell.) You can temporarily raise your hours above that and get improved productivity. But after a few weeks it catches up to you, your productivity drops, and you'll get less done despite working harder. And once you're in that state you've got a vicious cycle. Your lack of productivity pushes you to work harder which makes you less productive until something gives. Quite possibly your sanity.

The one thing that seems odd about this to most people is that people don't realize that their brains aren't working. But the reason for this is simple. When you're impaired, the first thing that gets impaired is your ability to judge how impaired you are. Thus your self-monitoring for impairment is not trustworthy. This effect is seen with many kinds of impairments, including chronic sleep deprivation, alcohol, strokes, and so on.

The army once did a study that showed this in a striking way. They took soldiers and put them on different sleep regimens. Among other things they found that most soldiers could operate indefinitely on 6 hours of sleep per night and would claim to be fine. But their wives said they weren't fine. Comparing results on ability tests before and after the new schedule, the wives were right and the soldiers were wrong. Which shows how easy it is to think you're still functioning when you really aren't.

---

Now that's the theory. What is the practice? The practice is that most of us don't do this. The reason is that most organizations manage by schedule, putting pressure on us to work harder. We put the effort out, we try to make the deadline, and then we fail to be aware of our declining output. Possibly we make the deadline, but quality suffers, bug counts suffer, and development+maintenance comes out to a lot more than it would have if we had focused on development alone.

What keeps us from doing the smart thing? For many of us, fear. While intellectually we know that we should be able to do more if we keep to a peak level, as a practical matter we're scared of being called slackers for not visibly putting out effort. Particularly if everyone else is working overtime. This fear is particularly reasonable since management usually has no idea how much developers do or why things take us so long. And is doubly so if our direct managers are themselves working too hard and therefore not themselves in the best shape.

Another major thing that keeps us from doing the right thing is that nobody likes to deliver bad news. Nobody wants to tell their manager that the schedule will slip. So we put off delivering the bad news to anyone, and try to keep it from happening. Of course this doesn't keep the schedule from slipping. But there is always the hope that someone else will be forced to admit that the schedule is going to slip. And failing that some people will deliver an incomplete and non-working piece, declare it done, then push the admission of failure off even farther.

Delivering bad news gets harder still because everyone else has also avoided doing it. Everyone delivers their bad news as gently as they can, and the manager inevitably wants to hear it as better than it is. Therefore upper management is usually seriously out of touch. The result is How Shit Happens, which is only funny because it is true.

As an example of how extreme this effect is, The Mythical Man-Month points out that in the rare cases of early projects, the good news starts coming immediately. If the project is late, there is no sign that the project will slip until about 2 weeks before it is due. And this is true no matter how long the project is actually delayed. And the driving force behind this is that nobody wants to deliver bad news, and nobody wants to hear it.

---

So we have a real organizational problem. How should you solve it?

The easiest problem is to ignore the problem and try to survive. This is suboptimal in virtually every way. However it is what most of us do.

A better option is to try to find an organization without the problem. They do exist. I know because I've been lucky enough to work for some. (Something that I suspect is a key ingredient is working in short iterations. Reality and management dreams tend not to get too far out of sync on a 1 week project.) But those can be hard to find. And in the current economy most people have a hard enough time just finding a job that they can't worry about the quality of the employer.

The final option is to try to make your current organization into one that works better. If not better for everyone, then at least better for you. This is the most difficult path, and there is no simple recipe for how to solve the necessary interpersonal issues that you'll encounter. However I can give you some useful ingredients.

---

The first is education. Go and get Rapid Development, read it, and learn what it says. It is one thing to confront your manager with the message that you need saner work hours. It is quite another to share research about how to get more done and say that you really want to do it. This works even better if you've identified multiple things you'd like to change at once, so it doesn't come off as your cherry-picking the item that you most wanted to try.

The second ingredient is knowing that delivering negative messages often is not as bad as we think it will be. Admittedly it doesn't always go well. However any manager who has been around the block a couple of times knows that software projects tend to fail, so it won't be as much of a surprise as you might think. Of course your manager is usually under pressure from above to meet deadlines and so is likely to not want to face that bad news. But it won't be a surprise.

The third ingredient is being conscious of the fact that delivering bad news early, no matter how bad it may be immediately, usually works out better in the long run. Yes, it is unpleasant and we all want to procrastinate on unpleasant things. But delivering bad news up front then better news later makes people happier in the long run than letting people think they'll get good news and disappointing them with bad news. (Incidentally if you're going to deliver bad news, take a tip from Machiavelli's The Prince and deliver it all at once, then deliver good news afterwards.)

The fourth ingredient is to understand the person you are talking to, and deliver your message in a way that is likely to be heard. A lot goes into this, but a big factor that lots of people miss is cognitive dissonance. Let me wrap up this topic, then I'll go on to explain it shortly.

The last ingredient is being aware of what the status quo is likely to cost you. If you allow yourself to be aware of that, you'll have lots of motivation to avoid that fate. And in this case motivation is a good thing.

---

Now let me return to cognitive dissonance. I won't explain it at length. I've done that elsewhere and likely will do it again, but here are the basics.

The principle of cognitive dissonance says that people will go to any length to maintain their positive self-image of themselves. This means that messages which threaten your positive self-image trigger defensive reactions. The more powerfully the message is delivered, the worse the defensive reaction is likely to be. (When you see such a reaction the last thing you want to do is follow up with facts proving your position. That just increases the conflict and therefore increases the defensive reaction.) What this means for any given person will vary from person to person. However if you remain aware of this phenomenon, you can often figure out where the hidden landmines are, and have hope of communicating what would otherwise be impossible.

For example, suppose that your manager thinks he or she is a pretty good manager. (Most of us tend to think we're pretty good at our jobs. No matter how good or bad we may be.) Then at all costs you must avoid threatening that self-image. Therefore you would need to avoid saying anything that even looks like you're delivering the message that there is room to be a better manager. This is true, even if said manager regularly points out things that made them better managers. To avoid the possibility of trouble, it is best to make the case that what you are saying flows from what that manager says about management.

This makes it really bad to say, I was reading this excellent book by Steve McConnell and really liked some of what his advice about managing software teams... Doing that is explaining to your manager how to manage, and anything you say after that which your manager doesn't already know and accept is going to be really difficult to accept.

It is better to say, I've been reading a good book, and it has taught me a lot about myself. It is making me aware that I am not working as effectively as I could... Making the message be about you is a reasonably safe way of side stepping the question of whether this is something the manager should know to tell you.

But it may be possible to do better still. For example consider, You know how you always tell us that we should constantly been trying to improve ourselves? Well I've gotten some ideas from a software classic, and here are some things that I'd like to try... If this is true, then you've put cognitive dissonance in your corner. Your manager will now find that a "Yes" is an opportunity for confirmation of his or her good self-opinion. Better yet, your manager will then be inclined to interpret the outcome as success, giving an even stronger confirmation of your manager's ability as a manager.

---

Now all that said, there is no guarantee that you can keep a sane work schedule when the going gets tough. However if you can do it, the outcome will be better for everyone. That doesn't mean the outcome will be good, but it is better and that is really worth it.

The use and limitations of exceptions

I'd have a longer blog but I just had a farewell party and so am do not have the most time to spend. So I'll just point to an email wrote to golang about what I see as the cases where exceptions prove useful, what the limitations of exceptions are, and some of the reasons why Go doesn't include them.

Design of a reporting system

I've had to build a lot of reports, and in my last job I build a reporting system that I think worked quite well. This post explains some of the key ideas behind that system.

---

One of my best ideas was to use Excel, not compete with it. Users will come with detailed requests for the exact reports they want. You can try to provide what users ask for. Inevitably they will change their mind, or the report lovingly described by someone's manager won't be what that person wants, or a million variations on this theme. This will generate an endless stream of requests as people try to get you to tweak what is being provided into their exact desire. Alternately you can make the data from your system directly accessible in Excel, and encourage people create the exact reports they want themselves in an interface they are comfortable with. Now all that you need to do is provide raw data, appropriately aggregated, and you're no longer a bottleneck. Giving people what they want rather than what the initially ask for makes for happier users.

Luckily it is easy to integrate with Excel. What you need is a way for users to design a web-based report which provides the raw data they want, and make it available with a short URL (less than 50 characters). Now a user can open up a new Excel workbook, then choose Data, Import External Data, then New Web Query. This pops open a browser. The user puts in their short URL, and the browser loads it. The user can then click on the HTML table that they want, save the web query, and Excel will run it again then fill a spreadsheet with the data in that table. Now a complex set of spreadsheets can be built off of that data, and every time they refresh it will re-run the report and get fresh data.

That's the user experience. Behind the scenes the developer needs to do several things to make this work.

• Provide users a way to save the parameters of a given report, and create a named report. The raw parameters will tend to be too long to be saved properly. When users access this named report you must not issue a redirect. If you do a redirect it will work initially, but Excel internally remembers the redirected location. Which means that if the user tweaks the saved report, the Excel spreadsheet won't notice the tweak.


• In your HTML give an unambiguous ID attribute to the table with the data. Excel will use this ID to locate the data. Otherwise it will try to analyze the structure of the HTML, and that will break easily when the report changes.


• Have a way to easily specify relative dates. People frequently want reports that run to the present, run for last month, and so on. If they only had fixed dropdowns when they set up the report then the Excel spreadsheets they get won't update to what they want it to be. I solved this problem by providing an optional text field that I passed to Perl's Date::Manip to parse. (I actually improved the date parsing slightly for my purposes, but that was the base.)

---

Moving on, my philosophy about reports is that 1 report that can be tweaked in 100 ways is more flexible and powerful than 10 reports that can be tweaked in 10 ways each. Similarly when given a one-off reporting request, it is better to find a way to add something to an existing report to make it capable of handling that request than to do the one-off request. That is because if someone wants that feature once, someone else is likely to want it again. And there is something really satisfying about receiving the second request and being able to say, "I've already built that, here let me show you..."

Of course the challenge is converting that philosophy into action. How exactly do you go about building a flexible report that can be tweaked in many ways?

My strategy was to look at each report as an array of SQL statements. All statements before the last would create temporary tables. The last query would return the data set that I would display. Thus if I needed to pull in extra information - for instance I needed to segment data by whether a given promotion was run - when building the array I could dynamically insert a query to fetch that particular piece of information, and then pass it through all of the tables.

This strategy naturally leads to two useful features. The first is that you can add the exact SQL statements used to generate the report in an HTML comment. This is very useful for auditing purposes. The second is that you can add the feature of having a dropdown box listing the temporary tables that will be created, and allow the report to run to that point then display those intermediate results. This is helpful when debugging. And not just for the reporting engineer - it was very useful for me when the accounting department was able to come to me and say, "This report, at this step, is missing these invoices."

This strategy, of course, requires being able to create definitions for temporary tables on the fly. Most databases provide that. Whether you are in MySQL, PostgreSQL, MS SQL, Sybase, etc you have exactly the right kind of temporary table available. The glaring exception is Oracle. There are a couple of workarounds available with Oracle. I've tried automatically building queries with large subqueries. It kind of works, however the result is unreadable by end users, you can't easily display intermediate results, and I miss the ability to control execution plans by creating temporary tables and adding appropriate indexes on the fly before moving on.

The other requirement of this strategy is that your SQL statements can be customized by the addition of an arbitrary list of extra fields, join conditions, etc. I wrote more reports than I care to admit before I realized that the best way to do this is to use a templating system to build your SQL. Now that I have used templates to dynamically generate SQL statements, I wouldn't want to go back. I used Perl's Template Toolkit for that purpose. Again I tweaked it substantially for my purposes. But many different templating systems could work well.

---

Moving on, let's look at how I organized things under the hood.

Obviously I needed complex forms with lots of checkboxes, input boxes, dropdowns, and the like. I chose to make each form control into an object that knew how to find its data out of the form parameters, do validation, write out its HTML, and pass information on itself to the templates. Some of the objects had SQL statements attached so that I didn't have to duplicate logic between reports.

There are endless arguments in software circles about separating presentation from content. In some circles mixing content and presentation as these objects did is anathema. In general web development I would not be inclined to move the HTML for form elements into what is basically a model object. However in this case it worked out very well. The form is so directly tied to the action of the report that it wouldn't make sense to have different people work on the two. These reports were for internal use, and so functionality mattered more than aesthetics. And as a practical matter this choice made the addition of options to forms extremely easy.

And, of course, I needed to have a way to coordinate everything. I did that with a report object. The report class had methods to create each kind of form object. It had a method to be displayed. When displayed it would show all of the form elements, check whether the report should be generated, if so it would run all of the queries, and display the report, and (of course), it would append necessary documentation.

With all of these pieces the code for a trivial report could look something like this:

use Report;

my $report = Report->new();

$report->add_textbox(
    name => "user",
    label => "Who is this for?",
);

$report->add_textbox_number(
    name => "age",
    label => "How old are you?",
);

my @queries;

push @queries, {
    temp_table => "results",
    sql => qq{SELECT :user as "User", [% age %] as "Age", current_date as "Today"},
    args => {
        user => $report->user,
    },
};

$report->display(
    title => "My Report",
    queries => \@queries,
    doc => qq{
        <ul>
            <li> <i>User:</i> The user this report was generated for.</li>
            <li> <i>Age:</i> Their reported age.</li>
            <li> <i>Today:&t;/i> The current date.</li>
        </ul>
    },
);

---

Now this report isn't very impressive. Also passing the age into the SQL as a template variable is poor style. (I did that just to show what the templating looked like.) But if you changed this to having a dozen form controls tweaking a data processing step with a half-dozen queries then you could have a very powerful and flexible report. Add in a few more reports, the ability to combine independent options, and the ability for users to build on top of this in Excel, and you wind up with a surprisingly powerful and flexible reporting system.

Learn to negotiate

I can't believe it has been a month since I've added a post. A lot has been going on. The most important being, of course, that I've accepted a job at Google starting in the new year. I am convinced that this is a good move that will make me much happier in the long run. I'll be doing a more interesting job with a new set of tools at a company filled with great people, what's not to like?

However the process has reminded me that we can all benefit from learning to negotiate. Now I'll be the first to say that I don't enjoy negotiating. I prefer to find ways to grow the pot rather than trying to take a bigger share of it. But a little reading and practice with negotiation can pay off very well.

A few years ago I asked a friend who was a very good negotiator what book on negotiation he would recommend. I do this kind of thing every so often with different areas, and I usually learn something from the books I'm directed to. He recommended Start With No. Reading this book has easily been worth tens of thousands of dollars to me already. Reduced to a nutshell, the idea is that progress in negotiation comes when you ask questions that the other person can reasonably say no to. And your results in negotiations will be better when you don't need to hear a yes.

A second book that I recently read on negotiation was Bargaining for Advantage. From a theoretical perspective this book is clearly the stronger of the two. If you wish to understand the process of negotiation, learn to recognize different negotiation situations, and negotiate regularly, this is clearly the book for you. It is clearly best in class, and will work for a wide variety of negotiation styles. However it seems to me that if you don't already come with a wealth of knowledge and experience on negotiation, you can easily read this book and find yourself saying, "That's interesting, but what do I do now?"

For me personally, a lifetime of avoiding negotiation (and therefore doing it badly when I had to) left me without any negotiation skills to speak of. So Start With No was the better starting book.

With that said, how did my last negotiation go? From my point of view, very well. My compensation is structured differently from my last job, but overall is similar. From what I've heard, people moving to Google usually wind up accepting a pay cut. Due to personal circumstances I honestly couldn't do that. Without having learned some basics of negotiation I am sure that I couldn't have done that. Heck, without what I've learned about negotiation I doubt that I would have been able to structure my resume to make it clear to potential employers how valuable I could be. And without that, I wouldn't have even had the job negotiation in the first place!

My thoughts on Go (Google's new language)

In the last couple of days I've been looking at Google's new language. Here is kind of a brain dump of my thinking on it.

First of all anything new produced by people of this caliber needs to be taken seriously. It is not done and may fizzle, but it shouldn't be dismissed out of hand. In that spirit here is a high level overview.

That said, it seems to be a mix of good things, interesting things, and things that are oddly missing. Looking at code examples it is in the C family. Since they want to appeal to systems programmers they are focused on simplicity, compilation time, and run-time performance. It has good stories to tell on all three, but I'm someone who finds scripting languages acceptable for the latter two, and that means I have a pretty high tolerance for complexity.

At a higher level they have garbage collection, first class functions and closures. Given the audience that they are aiming for they have downplayed the latter two (I spent the better part of an hour trying to verify that before finding it buried in the spec. Those are cool. They have goto, but looking through the spec I see that they have labeled loop control. There is an old theorem that any flow of control that can be achieved with goto can be achieved with equal efficiency with labeled loop control. Therefore I suspect that goto won't get used much. (Though it does make sense for state machines.)

Those are some basics, but they have three key features that they think set their language apart from most offerings out there.

The first is goroutines. You can take any function foo and type in go foo(arguments);. This basically means, "You go away and do foo while I do other stuff. When you return you exit." This is a very simple yet powerful concurrency model. The programmer does not know or care whether the goroutine executes in the current thread or another thread. In fact if the language had proper support for it then you'd be fine with it executing in another process on another machine.

Of course telling goroutines to do work in the background isn't very useful unless you can communicate between them. Their solution to that is their second key idea, channels. A channel is something like a Unix pipe. It is just a line of communication you can pass messages along, and data synchronizes by blocking until it is read. Channels may be one way or two way, and a given channel may only send specific types of data. They provide abstraction because you don't need to know the details of what is on the other end of the channel. This makes simple services very easy to write. For instance they offer this example of a simple RPC server:

  func startServer(op binOp) (service chan *request, quit chan bool) {
      service = make(chan *request);
      quit = make(chan bool);
      go server(op, service, quit);
      return service, quit;
  }

The first two ideas are not that surprising given the history of the people who came up with this. But the third is a little more surprising. The third idea they call interfaces. I actually dislike the name, I'd prefer to call it mixins instead. The idea is that an interface is defined to be any type that supports a given set of methods. You can then pass that object in anywhere where that interface is expected. You can also add methods to the interface, that are automatically available to objects that support that interface.

In short it is very similar to a Ruby mixin, except that you don't have to declare that you're importing the mixin, it autodetects that you fit the rules and does it for you.

OK, if that is what it has, what doesn't it have?

Well, libraries. Given that it was released yesterday, that is understandable. :-)

A bigger lack is exceptions. I understand this decision. The problem is that they envision writing servers with a micro-kernel design. But what happens when one of the services breaks down? Who can catch that error? The code that launched the service? The code that is communicating with it through channels? If there is no good answer, then what sense does it make to let a service just go down? If they do add exceptions they have hard design problems to solve which arise from the fact that you don't have a simple stack-based flow of control.

The much bigger omission is generics. The FAQ says that generics are missing because they introduce complexity in the type system and run time. Obviously so, but they are well worth it.

People who come from dynamic languages (eg Perl, JavaScript, Ruby, Python...) might well wonder what "generics" means. Here is a quick explanation. We have higher order functions. What if we wanted to implement grep? Well we could fairly easily write a function that takes a function mapping int to bool and returns a pair of channels where when you stick something in the one channel, it pops out the other if and only if the function returned true. We could write a similar one for Strings. And so on and so forth.

But you have to write one function per type you want to grep on! This is highly annoying. The functions all have identical form except different types. But you have to write the same function over and over again. As a result there is no way to avoid repetition of certain kinds of boring code. Which is why, when asked whether Go has libraries allowing functional constructs like map, reduce, scan, filter, etc the response was, The type system rather preludes them. They would either have to use reflection (slow) or be specialised for each type of slice. (See this thread for that exchange.)

If you wish to solve this you need to do one of three things. The first is to have a type system that is rich enough to handle meta-functions like this. (For instance Haskell.) The second is to have types be figured out and matched at run-time, which is what dynamic languages do. And the third is to implement generics.

-----

OK, enough of an overview. What are my thoughts?

Well first I'll watch it. It is young and nobody knows what it will do.

Moving on I hope the Go community that springs up start thinking about designing around the idea of capability based systems. That is a strategy of handling security/access by handing out opaque tokens that you need to make specific kinds of requests. If you have the token then you have access. If you don't, then you don't have permission and have no way to make the request. See this introduction for an explanation of how powerful this idea is. And in Go you'd realize it by having the "opaque tokens" be channels to services that do whatever you need done.

On a somewhat negative note I have mixed feelings about interfaces. My comment on mixins is, "It is bad unless you do it a lot, and then it is good." That is because a mixin involves magic at a distance that adds stuff to your class. So using a mixin requires learning exactly how it works. That mental overhead pays off if you get to reuse it over and over again. But if you don't use it heavily, it is a source of misunderstanding and bugs. The interface idea has the same issues, with more magic, but without the huge bug of overwriting methods you implemented in a class. (That said, I'm sure that more than a few people will be wondering why they didn't get the method they wanted from interface A when it conflicts with interface B that you also match.)

On a negative note I predict that there will be a lot of bugs in real Go systems around issues like deadlocks between services. Concurrency always opens up the potential for hard to diagnose bugs. And knowing that, I hope they develop good news for detecting/debugging those issues.

And finally I think that the omission of generics is a mistake. If they want the language to grow up, they'll need to fix that. Exceptions, if done well, would be a good addition, but there are reasons to leave it out. But generics are just too big an issue.

A proposal on patent reform

In theory the purpose of patents is to increase the public domain by providing temporary incentives to spur innovation. In that spirit patents are supposed to be limited to things that would not be obvious to one versed in the art.

However patents have lead to many problems.

First is the question of which ideas are obvious, and which are not. This is not a simple question at all. The patent office has resolved this question by being fairly lenient about which patents are granted, thinking that the issue can always be resolved in litigation. However people are reluctant to litigate, so invalid patents are acquired in great number. The theory goes that any individual one is unlikely to stand up, but if I threaten you with a dozen patents at once, my odds of having at least one stand up are pretty good. This set of incentives leads to a lot of overly broad patents that are pretty obvious, which causes uncertainty for others.

Second you have the issue of how long patents are good for. When it comes to drugs, a multi-year search and FDA approval can only be recouped from a long patent. When it comes to software a 17 year patent warps the market for many generations of technologies. Thus there can be no simple balance between encouraging innovation and unnecessarily hobbling further advancement. (Many people in software think patents shouldn't apply there. Certainly patents do a lot of damage in the software world, but as Paul Graham points out, if you're against software patents then you're against patents in general.)

Third we have the challenge that areas that change rapidly have a lot of opportunity for being the first to face a particular problem. If you're first to face a problem then it is easy to get a patent. Given that the patent office is lenient on granting them, you're likely to get it. So we tend to see very intense patent activity in areas that change rapidly. However those are exactly the areas where patents do the most to inhibit further progress!

My proposal addresses these issues.

I propose that no patent shall be accepted unless the technologies that make the discovery feasible and commercially viable have been broadly available for at least a decade.

If the problem has been solvable and a solution desirable for at least a decade but nobody has done it, that is evidence that the solution is not obvious to one versed in the art. Conversely if the solution quickly occurs to someone fairly shortly after the problem has become both solvable and commercially interesting, that is evidence that it wasn't really that hard. Furthermore it is exactly advancements in fields that are rapidly changing where patent protection does the most economic harm.

Why did I choose a decade? I chose it as a round number that is fairly close to half the length that a patent lasts. My theory being that if nobody has discovered it in that length of time, then the odds are reasonably high that nobody would have discovered it in the time the patent restricted the use of the invention, and therefore we have evidence that the net economic result caused by granting the patent is positive.

Of course, like any attempt at compromise on a controversial issue, I am sure it will satisfy nobody. But I don't think it is that bad either.

What various scholars have changed their minds about

The 2008 Edge questionnaire is one of my favorite collections of essays ever. What they did is asked many well-known thinkers the question, What have you changed your mind about? 165 of them answered. There is an index, but I find it better to just dive in.

Admittedly it is a lot of reading. What would I recommend? It depends on your interest. Personally I liked Freemon Dyson's explanation of how we know that dropping nuclear bombs did not cause the Japanese surrender, one of the founders of the inflation theory for how the early universe evolved saying why he thinks that the theory is wrong, a feminist explaining that the preponderance of men at the top echelons is because men vary more, a biologist discussing why experts often should be doubted, a psychologist discussing statistical illiteracy in medicine, and an information scientist talking about why the sample mean is a poor statistic to use. (The sample mean is where you add up all of the observations and divide by the count.)

Those are some of the ones that I liked, and can keep people occupied for a long time. But I'm sure everyone will have their personal favorites. And yours could easily not intersect mine.

Did males start as parasites?

Many years ago I was curious about why sexual reproduction is so prevalent. This is a summary of what I found when I investigated.

There is a standard theory to explain sexual reproduction, but it only answers half the question. The standard theory is that a species with sexual reproduction has a pool of genetic diversity. Should the environment change suddenly (temperature changes, a new disease, more effective predators arrive, etc), any gene that helps will have an improved chance of survival and will quickly spread through the population. Without sexual reproduction any particular gene line would eventually go extinct unless it was lucky enough to have the beneficial mutation happen within it. Therefore sexual reproduction allows for much faster adaptation, and benefits the species that does it.

This is all well and good, and convinces me that sexual reproduction has advantages asexual reproduction (as well as being more fun), but it doesn't explain why we have distinct sexes. For instance look at the lowly earthworm. They are hermaphrodites. When they meet they impregnate each other, and both go off and have babies. Hermaphrodites therefore reproduce twice as quickly as a male/female version would. Why, then, aren't we all hermaphrodites?

It can't be a simple quirk of genetics. The genetics of sex reproduction is much more plastic than most people realize. Even if we stick to vertebrates you can find strange things such as hermaphrodites (mangrove killifish) and animals that change gender (African reed frogs, several kinds of fish). So gender is more complicated than the familiar XX vs XY rule for mammals that we learn in biology. If hermaphroditic reproduction was evolutionarily favored, there are enough ways to get there that you'd expect it to be more common than it is.

As it happens I have never encountered a biology text that attempts to tackle this part of the question. However many years ago I came up with a theory that seems reasonable to me. I've discussed this theory with a number of biologists who agree that it seems probable to them as well. So this is my theory, but it is at least plausible. And for all I know it is the standard theory in evolutionary biology, and I just never encountered any source that presented it. (I'm very much not a biologist.)

My theory is that males started as parasites. More precisely, cheaters in a sea of hermaphrodites. And once men were effective enough, the ability of hermaphrodites to impregnate each other atrophied and you got strict sexes.

Let me break this down in more detail.

Imagine a population of hermaphrodites. Now add individuals who can impregnate others, but can't themselves be impregnated. These are males. By refusing the be impregnated the males avoid the hardest part of reproduction. This gives them more energy to devote to impregnating others. As long as the cheater impregnates hermaphrodites twice as fast as hermaphrodites do, this strategy works out.

Evolutionary theory has studied cheating strategies like this. Theory says that in any given environment there will be a natural fraction of the population that is male. With more hermaphrodites/male than the ideal, the male strategy becomes more effective, so the male population will rise. With too few hermaphrodites/male the male strategy becomes less effective so the male population falls. The ideal mix is called the evolutionarily stable strategy. (For another example of an ESS, some salmon fight for the right to spawn, and some try to sneak in while spawning is happening. We can calculate the ideal ratio between how many males should try to fight vs cheat, and it is very close to the actual measured ratios in real salmon.)

If your imagination fails you, don't worry. This isn't a hypothetical possibility, we can find this exact kind of mix of hermaphrodites and males (with no females) in c. elegans. That is a nematode that a lot of biologists study. (Biologists often like to work with organisms that are heavily studied for the simple reason that when they need to know something tangential to their research about how that organism works, there is a good chance that someone else has already studied it.)

In this situation males are cheaters. Biologically in some ways they parallel a virus - they are unable to reproduce on their own but can reproduce by hijacking the reproductive system of their hosts. However biologists have found that symbiosis arises easily out of parasitic relationships. And that is what I believe happened with the genders.

Take our previous population. Imagine that the males are numerous and effective at their job. So effective that a hermaphrodite who attempts to find another hermaphrodite to impregnate will expend so much energy that she would have more children if she just let herself be impregnated more frequently by the very convenient males. In this situation a true female strategy makes sense. And once hermaphrodites take that step, it is only a question of time before the ability for a reproductive individual to play both the male and female parts atrophies, and strict genders emerge.

For more interesting thoughts on sexual differences in humans, here is a psychologist discussing the consequences of men being expendable. I don't agree with all of his conclusions, but he is starting with solid facts that most people aren't aware of. For instance here is confirmation that men are significantly over-represented both at the top and bottom of a lot of distributions.

And in a very different vein, I have long liked David Brin's Neoteny and Two-Way Sexual Selection in Human Evolution. A summary can't do it justice. It comes up with a plausible theory explaining topics as diverse as why women have breasts to why so many men are pedophiles. Even if, no especially if you think that it has to be BS, it is worth the read.

Is the financial crisis really over?

I recently asked a friend who works on Wall St about whether these warnings about the municipal bond market were accurate. He gave me a detailed response, and gave me permission to repeat the comments but without his name or company attached. This is very reasonable given his current position, so I won't say anything more detailed than that he is in a good position to know what is happening in the credit markets, and I trust his opinion far more than any of the talking heads you see on TV.

Here is what he said:

Well, more than remotely accurate....

We have 3+1/2 more disasters to go in this crisis:
- Resi: this is the 1/2 disaster, as we're a little more than half way through the resi correction; unfortunately, we still have a long ways to go!

- Commercial real estate lending / CMBS: 4th Q retail will likely be a disaster, and commercial real estate has had as much or more price explosion as resi; already we are seeing soaring bankruptcies, etc. Many shopping center tenants have 'go dark' provisions (no rent due if >x% of the mall is dark), which are coming close to exercise. Banks and insurance companies are hugely exposed here; could make subprime look like a walk in the park.

- Credit card debt: even though the pace of job destruction has slowed, we are still in job destruction mode; consumers are increasingly falling behind on all debt payments. People are (wisely) looking to save rather than pay down debt (treating the debt as a lost cause)....

- Muni. As an example, tax revenue in your golden state is down 40%, and this story is repeated all over the country at all levels of government. So far, the Fed government has kicked a huge amount of money down the muni chain, which has kept the problems largely at bay; however, this process is nearing an end. The fact is that the entire country is massively leveraged: consumers, local government, state government, and the federal government. In the boom years, governments piled on debt and hugely increased their services, and most also hugely grew employment as well as entitlements (health care and pensions for muni employees). Now, they are facing revenue shortfalls but have great difficulties cutting services (often mandated by law), cutting staff (administration is opposed), cutting capital expenses (again, mandated by law). Default is in the cards for a lot of them, as federal money runs out.

Given the way the administration handled the automakers, I expect the muni bond holders to get hurt while the pension plans are made whole. I don't know what government does to avoid massive service cuts, though! It has seemed to me that one of the motivations for doing federal health care now is actually to ease up on state medicare spending (that's why your Governator has been making pro-health care reform statements).

So far, managing the crisis has relied on using the remaining credit of the borrower of last resort (the US). And while lots of reports have shown that govie debt is low relative to GDP on a historic max basis, these reports have completely overlooked the net debt position of both government and consumers. It is of course difficult to tease it all out (lots of munis fund housing projects), but personally I feel that we really can't have much debt ceiling left. Particularly if you consider that current/recent levels of GDP require a level of corp credit, but we now have vastly less corp credit available: without more corp credit availability, GDP must continue to slide, which implies debt-to-GDP continues to rise even without more borrowing.

An economist friend of mine (who lived through Argentina) and I both believe that the only real outcome of the whole crisis is that the Fed will manage a graceful and slow dollar dilution, so that we see a huge inflation but over a long enough stretch of time so as to not be unduly destabilizing. This process would basically inflate away our debt as it revives the domestic economy. Unfortunately, so far as I know, no country has ever achieved economic success via a weakening currency!

Anyway, yes, this is all really bad, and still has a ways to go!

Reading this I am strongly reminded of what I saw predicted in Wealth and Democracy several years ago. The book is a long read, but buried in it were a list of parallels between the USA circa 2000 and the last 3 great world empires shortly before their collapse. In all 3 cases shortly after a boom caused by financial speculation there was a financial crisis, which they recovered from fairly fast at the same time that they launched a military adventure that was expected to be a quick, successful war. The war dragged on and cost far more than expected. Then public mood turned against the war around the time that a more serious financial collapse hit. After a series of subsequent financial collapses there was political unrest leading to civil war 15 years after the peak in 2 out of 3 cases. (The exception was England, which got involved in WW I before the civil unrest could become worse.)

When I first read this I thought, "Interesting, and Kevin Phillips does have a track record of making apparently absurd predictions that came true, but I'm not overly concerned." I still believe that the prospect of expressing ourselves through democracy can head off the possibility of civil war, but it has been scary watching the timeline unfold like clockwork.

Why I left math

Reading how shocked Doron Zeilberger is at the state of modern mathematics reminded me of why I left the subject.

Math departments regularly have visiting mathematicians come and give talks. Or at least the one I was at did. For the visiting professors these talks were a confirmation of success, all of these people came to hear about their research. So they would talk about their research and get quite excited about what they were describing.

As a grad student I attended. I quickly noticed that most of the professors in the math department went out of politeness. However they knew they wouldn't understand the talk, so they brought other things to do. If I looked around about 15 minutes into the talk, I'd see people reading books, grading homework, and otherwise not paying attention. At the end of the talk the speaker would ask whether there were questions. Inevitably the mathematician who invited the speaker would have some. Occasionally a second mathematician would have some. But the rest of the room wouldn't.

This was supposed to be the high point of the life of a mathematician? That's when I decided that, no matter how much I loved mathematics, I wanted a different career. Unfortunately my wife was in grad school as well, and we were in such a small town that I didn't have any immediate employment options. Therefore I remained a rather unmotivated grad student. In the end my wife switched to medical school just before I would have finished the PhD. I'm mildly disappointed that I didn't finish, but it really has been no loss.

Why do mathematicians put up with this? I'll need to describe a mathematical culture a little first. These days mathematicians are divided into little cliques of perhaps a dozen people who work on the same stuff. All of the papers you write get peer reviewed by your clique. You then make a point of reading what your clique produces and writing papers that cite theirs. Nobody outside the clique is likely to pay much attention to, or be able to easily understand, work done within the clique. Over time people do move between cliques, but this social structure is ubiquitous. Anyone who can't accept it doesn't remain in mathematics.

It is important for budding academics to understand this and get into a good clique. This is because your future career and possible tenure is based on your research. But the mathematicians making those decisions are unable to read your papers to judge your work. Therefore they base their decisions on the quality of journals you get your papers into, and the quality of people you get writing recommendations for your work. But both of those come down to getting into a group that includes some influential mathematicians who can get your papers accepted in good journals, and that can write strong letters of recommendation.

In fact if, like me, you are someone who likes to dabble in lots of things, you will be warned (as I was by multiple professors) about the dangers of not focusing on one small group. You will be told plenty of cautionary tales of mathematicians who published a number of good papers, but who didn't publish enough in any specific area to get good mathematicians to stand behind them. And therefore the unlucky generalist was unable to get tenure despite doing good work.

For a complete contrast, look at the situation in biology. A motivated advanced biology undergrad is both capable of, and expected to read current research papers. When biologists go to a talk they both expect to understand the talk. And biologists have no trouble making tenure decisions about colleagues based on reading their papers.

I subscribe to the belief that the difference is mainly cultural. Biology is fully as diverse and complex as mathematics. Furthermore what I have read about the history of mathematics suggests that the structure of the mathematical community was substantially different before WW II. For example David Hilbert was known for stopping speakers and forcing them to define anything he found unclear. (Amusingly he once had to ask Von Neumann what a "Hilbert Space" was.) But after WW II an explosion of university math departments and a focus on solving concrete problems lead to a fragmentation of mathematics. And once mathematicians came to accept that they couldn't be expected to understand each other, there was nothing to prevent mathematics from splintering into fairly small cliques. Which has happened, and this is unlikely to ever be reversed.

PS I'm amused at the fact that a number of comments at Y-combinator thought that the situation with programming was worse than mathematics. Yes, there are divisions within programming. But they are nothing compared to the fragmentation in mathematics. I've done both and there is simply no comparison.

Why write a blog?

Before I had a blog, I wondered why people wrote them. Now that I have one, I wonder why other people write theirs. Because I know my reasons, and I am sure they are not shared by many other people.

I've always been the kind of person who liked learning and thinking about a lot of different things. Inevitably this means that I wind up having opinions on diverse topics. Which means that I regularly run into things that I have opinions about that I couldn't share because nobody in my immediate environment really cared. This blog has become a place for me to say those things. If nobody cares to listen, I still enjoyed getting my thoughts in electronic form.

Now to reasons I don't do the blog. I don't do it to share my personal life. My personal life is private, and the people I want to tell it to can hear about it directly from me. I don't do the blog for economic reasons. Admittedly I turned on ads out of curiosity for what it would be like. But to date I've made less than what I usually spend on lunch. I've made less than I get from 5 minutes of paid contract work, and have no shortage of potential contract work available right now. I don't do it out of desire for fame. I've seen enough fame in my life. And besides the built-in audience when I spent years posting on Perlmonks is much bigger than I'm going to get with a random blog.

So this blog is an outlet for expressing thoughts of a kind I've always had. Nothing more, and nothing less.

Now what triggered this particular post? It was a comment on my first entry that thought I was missing a big opportunity by not writing more about that topic and taking advantage of my current position as an expert. I simply don't see things that way.

Over my life I've been seen by people as an expert on a number of things. Lately I've been compensated fairly well for my expertise in Perl, A/B Testing, and general data manipulation. I don't think that being seen as an expert on spaced repetition learning would add much. I've also become moderately well-known in multiple communities. There is no need for me to try to become well-known for having taught a course. So I don't see the opportunity.

Furthermore there is a cost. I've always disliked people who took a good idea or story and expanded it with fluff to make the most of it. Why would I seek to be a kind of person I don't like? It makes no sense to me.

The same will go in the future. If I say my piece on a topic and don't feel I have more to say, I won't say more. If I say more it is because I feel I have more to say.

Stock buybacks shouldn't change stock prices

Companies are owned by the shareholders. Companies try to make profits. And when they do, some combination of three things happens with those profits.

1. They get reinvested in the company.
2. They get paid out as a dividend.
3. Some shareholders are bought out in a stock buyback.

The mechanism of the first two is generally understood. But there is widespread misunderstanding of the third. Inevitably when a stock buyback is announced, you'll find articles that insert helpful explanations about how stock buybacks increase the stock price, helping investors.

Every time you see one of those explanations, I guarantee you that the person who wrote it is ignorant about basic financial matters. That appears to be most of the media.

Why, you ask? Because changing the stock price is not the purpose of a stock buyback. And to first order there is no effect on stock price. The reason for that is that the stock price is the value of the company divided by the number of shares. As the company buys back its stock, the value of the company drops by the money spent, and the amount of stock outstanding drops the same. Investors are rewarded by having those who wish to be bought out, bought out. And, unlike with a dividend, investors who didn't want to sell don't get taxable income.

So, for instance, if a company worth $50 billion with 10 billion outstanding shares of stock buys back $5 billion dollars, then the value of the company drops from $50 billion to $45 billion (because it spent $5 billion in cash), the amount of outstanding stock drops by 1 billion, and the remaining 9 billion shares are still valued at $5/share. This is finance 101 stuff. It falls right out of the CAPM.

Now I made a comment about "first order". Why? Well because there are several second order effects that realistically could cause stock prices to move. What sort of second order effects could those be?

• It is always hard to buy large amounts of stock without increasing the price (and thereby cause inefficiency in your purchases). However buybacks are always announced in advance prominently enough to let interested shareholders know this is a good time to sell off large positions, so it works out.


• Different investors have different opinions on the true value of the company. The ones you buy out tend to be the more pessimistic ones, leaving you with people who think the stock is worth more. This effect is generally small.


• The market has many people who are confused about finance and who may expect prices to go up. If lots of people expect prices to go up, that's a self-fulfilling proposition. But it tends to be a temporary self-fulfilling proposition, and Wall St is very good at earning money from people predictably making such mistakes.


• The company's cash holdings fluctuate less in value than the other assets that make up the company. Therefore as the relative balance of cash and other moves towards being more heavily weighted with "other", the volatility of the stock price increases. If you've studied options you'll know that increased volatility increases the returns for people holding options. Those returns come from somewhere, and the somewhere they come from is existing shareholders. Therefore the stock price should drop slightly. In theory. If investors are informed enough to pay attention.

I have no idea what the net result of these effects is on current stock holders. However all of the listed effects improve returns for option holders. Given that the people who make decisions about how to return money to shareholders (typically the CEO and board of directors) also tend to be the largest holders of long-term options, I have to wonder whether the ever-growing popularity of stock buybacks has more to do with avoiding giving investors unwanted ordinary income, or with improving the value of options held.

What are financial derivatives?

This weekend my wife convinced me to see Michael Moore's movie about Capitalism. It had its good points, its bad points, and its silly points. One of the silly points was multiple purportedly smart people finding that they couldn't explain what a financial derivative was.

So I decided to see if I could come up with an easy to understand explanation. And then explain the risks and benefits in (hopefully) common sense terms.

A financial derivative is a contract that has value based on what some other thing might do.

Let me illustrate with a concrete example. A put is a binding contract saying that person A can sell person B something at a fixed price on a fixed date. There is no obligation to sell. But if A wants to sell, B has to buy at that price. No matter what the current price happens to be.

Why would someone buy or sell one of these? Well suppose I bought some stock whose price I think will fluctuate, but which I think will wind up higher. So I think I'll make money, but just in case I can buy puts at somewhat below the current stock price as insurance. If the stock goes down I can sell the stock at the pre-agreed price, and I limit my losses. Conversely the person who sells me the put doesn't think the price will go down either, which is why they are happy to sell me my insurance.

Please note this carefully. Neither the buyer nor seller here believes the put will be used. Therefore it can be purchased cheaply. (This is why it makes a good insurance policy.) Yet the put has value to the buyer and risk to the seller, and therefore money has to trade hands for the contract to be worthwhile to both.

The next thing to notice is that virtually any type of contract you can dream up can be made into a derivative. For instance if you want you can do the reverse of a put. Rather than saying that B has to buy from A at a fixed price, you can say that B has to sell to A at that price. Those actually more popular and are called calls.

Both puts and calls are called options because they give someone the option (but not obligation) to do something at some point in the future. In fact the options that are given to employees are almost always calls. (Almost, but not actually always. One job rewarded me with a contract that was carefully constructed to legally be a bond, and not a call. That was done for tax reasons.)

Now a warning about derivatives. In theory derivatives can be used to limit and manage risk. Supporters generally introduce derivatives by illustrating how all parties to them can use them to manage risks. However they can also be used to create and concentrate risk very rapidly. For instance if you sell many puts very cheaply and then the price of the stock falls, you can quickly be out an insane amount of money. This is not a hypothetical risk. Every so often a rogue trader will accidentally destroy a bank or other large financial institution by making big options trades that go sour. A good example is Barings Bank. Let me stress that this typically happens by accident, not intent. (Usually, as with that case, someone does something stupid, loses money, then keeps doubling up their bet hoping to get out of trouble. If the string of bad bets goes long enough, the company goes broke. Which makes me wonder how many mistakes are made then are successfully corrected without anyone being the wiser...)

So derivatives are contracts. People can buy and sell them. But how should you price them? Well the standard answer for puts and calls is to apply the Black-Scholes pricing model. I won't go into the math, but here is the idea. What they do is construct a portfolio that, managed very carefully, will in the end come to the same effect as a put or call. The price of this portfolio is theoretically the price of the option. Why? Well if the portfolio is less than the option, then you can buy the portfolio and sell the option, manage the portfolio, and you make a guaranteed profit. (Making some combination of trades with guaranteed profit is called arbitrage, and people on Wall St are very, very interested in finding opportunities to do this because arbitrage is free money.) Conversely if the portfolio costs more than the option you can buy the option, sell the portfolio, manage the portfolio and again make money. So in an efficient market the prices have to match.

OK, this is all well and good. Now let's move on to something more fun. Given that the complexity of derivatives are only limited by the imagination of the people writing the contract, here is no shortage of interesting possibilities. Let us focus on swaps.

A swap is just an agreement for two parties to trade things of equal, or close to equal, value at the time of the trade. There are many kinds of swaps traded today. Some are traded with insane volumes. For instance foreign exchange swaps trade money between currencies today in return for a reverse trade tomorrow. This is done to the estimated tune of $1.7 trillion (USD) per day! (The primary purpose is to transfer money at close of business from traders in New York to Tokyo, then from Tokyo to England, then back to New York. That way money can be used to make money 24 hours a day.) But let's take something different. I'll look at an example of two companies swapping mortgages.

Our scenario is that a US company is building a skyscraper in Japan. At the same time a Japanese company is building a skyscraper in the USA for approximately the same price. Neither has sufficient cash on hand to fund this, so each needs a mortgage. (This would likely be done with bonds, but let's not worry about that detail.) But rather than each taking out a mortgage in the other currency, let's say that they enter into a contract to swap the financial obligations - the US company pays the mortgage on the US building, while the Japanese company pays the mortgage on the Japanese building. That's a swap.

Well they certainly swapped something. But what is the point?

The point is that neither wants to have to worry about exchange rates. The mortgage on the Japanese building will be in yen. The mortgage on the US building will be in dollars. The US company has a reliable stream of dollars coming in in their future, but doesn't know how much to set aside to meet an obligation priced in yen. The Japanese company has the reverse situation. It is easier for each to budget and plan for an expense in their own currency, and so they'd each prefer to pay the other's mortgage.

It is important to stress that this is not just a convenience. When the companies go to the banks, the banks will recognize the exchange rate risk and that will affect the rates the companies can get. A US company can therefore get a loan in US dollars at a better interest rate than the Japanese company can. Conversely the Japanese company can get a loan in yen at a better interest rate than a US company can. So after the companies swap mortgages, each gets a better interest rate than they could otherwise get. And since both save on their interest payments, both write down immediate profits from making that swap.

This is one of the most important and counter-intuitive points to understand about financial derivatives. Two companies entered into a contract, and both wrote down an immediate profit. Nothing is faked here. Real money is saved in their monthly bills. That saving can be calculated, and as a result the company is better off. Over time the exchange rates will likely move and as a result one probably makes money and the other eventually loses on the deal. But on day 1, both marked down a profit and the profit is real.

Now if there is one thing that gets the attention of people in finance, it is the ability to manufacture money out of thin air. That apparently happened here. It is therefore worth repeating that this example has been studied to death and generally accepted accounting rules agree that both companies get to write down immediate profits in this case.

Of course you can only set this up if the conditions are just right. These profits aren't particularly easy to find. But once you've seen a way to do it, people can set them up over and over again. And over time the financial wizards found cases where companies could write private contracts and claim an immediate profit. As with the swap described there was inevitably long-term risk associated. But who wants to turn down free profits?

Now here we have a critical danger. With many financial derivatives people enter into complex contracts which, based on some complex theory, they think entitles them to write down a profit. However the investments take time to pan out, and come with significant risks. Worse yet there is the very real possibility that people will write down profits that aren't real due to honest mistakes. And if someone honestly overvalues the value of a particular kind of contract, they will enter into that kind of contract over and over again, writing down fake profits each time. (And getting paid correspondingly large bonuses for finding such great deals!) As a result in the end even the smartest people in the world can find themselves unable to truly measure the risks a given company is taking.

If you think I am exaggerating, go read what one of the greatest investors ever has to say about financial derivatives. There is a section about derivatives that starts on page 14 of this PDF from Berkshire-Hathaway. When Warren Buffett wanted to talk about financial derivatives in 2002, the first point he made was, Charlie and I are of one mind in how we feel about derivatives and the trading activities that go with them: We view them as time bombs, both for the parties that deal in them and the economic system. He then goes on to explain many of the same points that I have. His common sense comments are well worth reading by anyone who wishes to make sense of the recent financial crisis.

Now let's move on to securitization. Securitization starts with a bunch of things with promised future payments that may or may not materialize. The things could be loans on houses, credit card debt, student loans, risky bonds - as long as it is a stream of future payments it is meat for the grinder. Typically each thing is a stream of relatively small payments that lasts for a long time. An issuer takes a whole bunch of these, and puts them together in a deal. A deal likely will have a long stream of big payments. Technically a deal is a very carefully set up corporation whose assets are the things that go into the deal, and whose likely future revenue is used to issue a series of bonds. Each bond is a stream of payments that ends once a particular amount of money has been delivered. The first bonds issued are very likely to be worth full face value, and the last ones are unlikely to make much money. For technical reasons the different kinds of bonds are of interest to different investors.

A financial security is any financial instrument that can be traded. The loans etc that go into a deal can't be traded in any kind of open market. The bonds that come out of the deal can. Therefore this process creates securities out of debt, hence the name securitization.

When the deal is set up right, the loans, etc that go into the deal can be purchased for less than the bonds sell for, and the issuer gets to write down an immediate profit. If the deal is correctly modeled, every purchaser gets a known amount of possible future profit with a clearly understood associated risk. Everyone wins.

Given recent bad press it is worth pointing out that securitization has been around for a while and until recently it worked out well for most of us. Whether or not we were aware of it.

Securitization is really a mechanism for diversifying risk. Before securitization took hold, mortgages would be held by the local issuing bank. This meant that if a local area's economy took a nose dive, the local banks would get seriously hurt and sometimes got wiped out. This would restrict the availability of credit for a long time following, and made eventual economic recovery much, much slower.

Since the 80s things have been very different. Banks now sell their loans on to Wall St, which turns them into securities. This means that if a local economy collapses, like Pittsburgh's when the steel industry left or Michigan's with the auto industry troubles, local banks are much less likely to collapse with it. And they can continue to offer credit, which makes it much easier to recover down the road. Therefore securitization has made local economies much more resilient than they otherwise would be.

But, as we all know, there is a fly in the ointment. In the case of sub-prime mortgages we had bad risk models. Thanks to those risk models companies could write down easy profits while not understanding the size of the risks they were taking on. And in their pursuit of profit, they repeated the mistake over and over again. As a result widespread acceptance of securitization wound up replacing the risk of fairly frequent nasty local economic collapses with the risk of relatively rare nasty global collapses. Luckily with the last one having been averted. (So far, anyways.)

So there we have it. Financial derivatives are just contracts. People enter into them for all sorts of reasons. They are traded in large volumes. They can be arbitrarily complex. They can mitigate or create risk. We have theories on how to price them. When financial derivatives are set up correctly, apparently free money is created. This money comes with associated risk. Honest mistakes about the pricing or risks can lead to disaster. Warren Buffett has written on this topic with extreme warnings about this exact issue. Securitization is a way of bundling lots of little streams of cash into nice bonds. Securitization brought us increased local financial stability, which has been to our general benefit. But systemic mistakes in securitization also brought us a real risk of global collapse.

If you understood that then you likely understand financial derivatives better than half the talking heads you see on TV. And you definitely understand them better than Michael Moore appeared to in his movie.

Progress on Kelly materials

Not done is nothing taught me a lesson. Publicly saying "I didn't complete this because I lost interest" doesn't feel good and provides motivation to finish. So I've returned to the Kelly Criterion explanation I was working on, added a little bit, and provided return calculator that optimizes.

That may not sound like much, but under the hood it is doing calculus in JavaScript. The generic parts of the implementation are in advanced-math.js. If you like the idea of doing math with closures, feel free to take a look.

Next I need to add some linear algebra, some multi-variable calculus, then implement something that finds maxima using Newton's method on a convex polytope defined in any number of dimensions using a set of linear inequalities. You can think of this as a non-linear version of the problem solved by the Simplex Algorithm. The fun part is that I need it to good enough to find the optimum point in an n-dimensional polytope sitting in n+1-dimensional space. (The actual polytope I will need is defined by 0 ≤ x_i &le 1 for i=0..n, and x₀ + ... + x_n = 1.) That means I have to deal with issues like finding my way back to the region after round-off error takes me away from it.

Oh. And of course I'll do all of this in JavaScript.

Perhaps that can be useful for others? Possibly. However JavaScript has no real equivalent to, say, CPAN, so the effort of putting it somewhere that someone in need can easily find it is beyond my motivation. That's why I did nothing with my port of Statistics::Distributions. But that is a thought for another day.

What granularity for MVC?

Web developers often try to use an MVC architecture. This means that the business end of a website is divided into models, views, and controllers. Models encapsulate your data, and this is where you are supposed to put business rules like, "You can't take more money out of your account than you have in it." Views are about how to present the data. Ideally the model shouldn't know or care about whether the user is going to look at an edit page or a display page, or look at a page in English or Chinese, those things are presentation logic which belongs in the view. And finally the controller is responsible for the necessary glue between them, for instance to figure out what model and view to use for a given URL for a given person.

There is widespread agreement that this is a good thing in the principle, but things get messy when you go to the details. Why? Well in part because people often break the rules they agree to. Business logic creeps into views, models format text that is just passed through the view, and the controller gets into everyone else's business. After a while you're doing MVC in name only. But also there are honest disagreements about a myriad of issues including where exactly to draw the lines between classes, whether to add in helper classes and if so which ones, how complex views should be, and so on.

I'd like to draw people's attention to a design question that few in the web world think about, namely how granularly we should do MVC.

Traditionally in web development people assume that the right level of granularity is the web page. You get a URL, that goes into a controller, sometimes there can be a cascade of routing decisions, and eventually you get to the code that fetches an appropriate model and view, then puts them together to get a web page. Sounds pretty straightforward and flexible.

But what if we have a complex web page? What happens if we add independent sections on a web page that have little to do with each other but need different dynamic information? Alternately how do we handle displaying many related objects for editing? In my experience the natural tendency is for the controller to become a mess. And the view has to be synchronized with the controller, resulting in fragile dependencies.

For an alternative, let's look back at the history of MVC. Originally MVC arose in the Smalltalk world. And the appropriate granularity for MVC wasn't a whole page, it was a component. You'd write Smalltalk applications and every little dropdown could have its own model, view and controller. Which could all talk to each other, and permitted good decomposition of very complex behavior. This idea has continued in the world of GUI programming. For instance an object-oriented Flash program is likely to be developed on a similar paradigm.

What happens if we take this idea back to serving a web page? Well obviously you need a lot more models, views and controllers. However it isn't as bad as it seems because they also become much simpler. Plus it becomes easier to reuse components on multiple pages through a site. The whole page can still become complex, but MVC is a good way to manage it. After all our design is now closer to the original versions where MVC first proved itself useful.

And as a final thought, ever more complex and dynamic client side interfaces make web pages into something more like any other GUI. Which means that at some point there are likely to be real advantages to using component based MVC within the client interface. But this idea is more natural to try when there is no impedance mismatch between your server-side design and your client-side design. Which suggests that you should start with component-level MVC on the server side, and then "lift" components to dynamic AJAXified components one by one as appropriate. Therefore starting with a component-level MVC gives you more flexibility to later evolve a complex, dynamic client-side interface.

Obviously I wouldn't recommend that every developer rewrite their current websites based on this idea. However I think that the benefits are interesting enough that I'd recommend that developers play with this idea, so that they can make an informed decision about whether to try this approach on future websites.

Why did the Neanderthals die out?

Yesterday's entry reminded me of a Scientific American article last summer about Neanderthals. Reading it I was reminded again of the curious resistance that anthropologists seem to have towards accepting lessons from biology.

But before I get into that, I need to give some background. The theory of punctuated equilibrium, put forth in the early 70s by Niles Eldredge and Stephen J. Gould. This theory held that new species to develop fairly quickly in small isolated regions, then if successful spread over a wide range and change only slightly until they are replaced by another species. Biologists long ago accepted this theory, and it is backed up with evidence ranging from observations of speciation in action in fish in Africa, to Eldredge and Gould's initial example of a type of trilobite that went through 2 speciation episodes in several million years, both of which we are lucky enough to have fossils from the speciation episode in a single mine each. So in general this is what we expect speciation to look like - something that happens in a small area somewhere which bursts out.

Anthropologists have traced a number of types of humans that existed in the past. Of particular interest are modern humans. Two theories existed on that. One is that we evolved in Africa, then spread. The other is the multi-regional hypothesis, which is that we evolved in several regions at once, and possibly mixed in other kinds of humans. Thus leading to side questions such as, "Did we interbreed with the Neanderthals?"

There is no question which version evolutionary theory supports. Punctuated equilibrium unambiguously says that we should expect our species to have started in a small area and then spread without significant (if any) mixing. I remember reading an essay by Gould many years ago that laid this out pretty bluntly. However anthropologists don't like hearing things like, "We can tell you what probably happened with humans based on general principles established with trilobites and fish." And so the active debate lasted for decades on this point.

Reading the Scientific American article I found that the question has now been tackled with the aid of DNA analysis. Based on genetics we now know that the evolutionary biologists were right, and the multi-regional hypothesis is wrong.

But now we have a new question that anthropologists are putting energy in. What killed the Neanderthals?

First let's list some facts. The Neanderthals survived for over 150,000 years. They lived through multiple ice ages. We have evidence of them hunting both large and small prey, with a preference for big game. We coexisted with Neanderthals for thousands of years. There are minor differences in technology (for instance our ancestors had better stitching on their clothing), but nothing specific. Our ancestors specialized in smaller prey (though we were happy to chase big game after the Neanderthals died out). Yet by the end of the ice age the Neanderthals had died out, both in places where our ancestors were found, and in places we weren't.

Hence the mystery. Why did the Neanderthals die? The long (apparently peaceful) coexistence suggests that we didn't directly kill them. They survived previous ice ages. We survived in the same places at the same time. And they should have been able to adapt and use our strategies - after all they did it in previous ice ages. With such small differences, why did they die out?

The article was very good. This was all laid out very well. Along with lots of detail about relative energy requirements, technology, and so on. But I found myself asking whether they had bothered asking the ecologists.

If you study ecology, one of the basic principles is that related species avoid directly competing. How so? Well when the related species is not there they broaden their niche, and when the related species is then each species defines its ecological niche closely enough that they don't directly compete.

Let me give an example. Originally I read this with 3 species of birds, but that was many years ago so I'll simplify to 2 species of birds that I'll call A and B. These birds all hunt insects, and have the choice of hunting them in the forest or by lakes and streams. When only one species is there, they can be found in both places. But if both species coexist in a region, species A does all of the hunting by lakes and streams, while B hunts only in the forest, and so they avoid directly competing.

Why do they do this? Well A is better than B at hunting in open water. B is better than A at hunting in the forest. When only A is around, therefore, the insect population in the forest rises until a given bird is equally well off in the forest and by the water, so you find birds in both places. With B the reverse happens. But when both species are present, then the A birds lower the population of insects near rivers and streams to the point where the B birds will go hungry if they try to feed there. And conversely the B birds keep the insect populations in the forest low enough that the A birds will go hungry if they try to feed there. So A and B coexist, and specialize.

This coexistence can last indefinitely. However there is very real competition here. The presence of the other species narrows the niche the birds live in, and result in fewer of A and B. Each population is now more precarious, and less able to adapt to environmental changes. Thus the general principle that related species tend to coexist and compete through specialization.

Now let us return to the mystery of the Neanderthal extinction. In the absence of our ancestors, the Neanderthals were willing and able to use a combination of food gathering strategies. However once our ancestors arrived, both groups specialized. The two had no difficulty coexisting for thousands of years, but both populations were more precarious than they had been. In particular the Neanderthals were no longer able to use our niche because our ancestors were occupying it more efficiently than they could. Not much more efficiently, perhaps, but even a small difference is the margin between survival and death.

And so it went. Where we were, our presence forced the Neanderthals to specialize. Where we weren't, our absence was evidence that our niche wasn't a good fit to that area, so the Neanderthals there would be unlikely to benefit much from being able to adapt freely to what worked best. Which left the Neanderthals unable to adapt to climate change, and eventually left them dead.

Based on the history of the multi-region hypothesis, I predict that anthropologists will eventually acquire overwhelming evidence for a detailed version of that scenario. But they are likely to take decades to get their. And when they do they won't notice that they could have gotten there faster if they were willing to accept general principles that are already well established in birds, salamanders and the like.

Thoughts on Youtube

I think my son was 2 when I discovered this. To to youtube. Type in a random thing of interest, like "fire truck". And then watch videos about that topic.

Of course things have progressed since then. My son is almost 5 now. And he just discovered dinosaurs. So we watched some dinosaur videos yesterday and discovered the series of "Walking with ..." series. And as these things go, I wound up going to Amazon this morning and ordering several series. Walking with Monsters, Walking with Dinosaurs, Allosaurus, A Walking with Dinosaurs Special, Chased by Dinosaurs and Walking with Prehistoric Beasts. (Yeah, overboard. But I know my son and it is a good price.)

Yes, I know how much they made up. But there is a lot of good science in there as well. And I know how much my son will love it.

Now lots of people, including many copyright holders, think that youtube is a horrible violation of copyright. Technically they are right. But I believe that there are plenty of people like me out there, and the net effect is more people discovering then buying things they like. Which is good for copyright holders.

And it is a belief backed by some evidence as well. For instance look at the Baen Free Library. Put books online for free in multiple formats and what happens? Sales go up!