Monday, December 28, 2009

Our most popular drug is made of sugar, fat and salt

On NPR not long ago I heard an interesting thought.

Suppose a hundred years ago that someone wanted to eat chicken. They had to go out, grab the chicken, cut its head off, pluck it, clean it, then cook it. That was just what it meant to eat chicken. Today that seems crazy. But in a hundred years that is how cooking will look.

Indeed, that is true for a lot of people. Just think how many people eat out more often than not. Or eat packaged meals. Cooking takes work, and most of us don't want to put out that effort if someone else is willing to do it for us.

On the surface that is quite reasonable. And I will confess to succumbing to the temptation quite a few times. I'm no food snob, and don't particularly enjoy cooking.

But all that notwithstanding I think that most of us should cook more. You'll be healthier. You'll eat better food. And it needn't take as much energy as you think.

Let me start with the health issues. My source here is The End of Overeating by David Kessler.

The problem that every commercial food distributer faces is getting people to buy their food. Preferably frequently and in quantity. People have a lot of reasons for buying food, but the three most important are named sugar, fat and salt. Individually we are attracted to each, however we reach a point where we saturate on any one of them. If you combine 2 of those, however, we are attracted to much higher concentrations than we are if you separate them. And combining all 3 hits a sweet spot where we want far higher concentrations than we would accept independently. Not only do we crave higher concentrations, but we experience a pleasurable rush of brain chemicals. In the pursuit of that pleasure we routinely overeat past the point of satiation. Then want to eat that way again when we recover. And this is why you're fat.

This principle is simple, effective, and well understood through the food industry. Whether you're a manufacturer designing a new kind of cheese puff or a restaurant trying to build a clientele you have to do the same thing. Deliver the magic combination of sugar, fat and salt and you can succeed. Fail to, and people will inevitably flock to the competitor who is willing to do it. The food industry has heard and obeyed. Whether it means giving us salads that are mainly a vehicle for delivering dressing (a base of oil with sugar and salt), or offering apparently healthy chickens that have been preprocessed with the injection of a syrup with sugar, fat and salt (this is why it falls apart so easily in your mouth) all of our processed food is a drug delivery mechanism.

The phrase "drug delivery mechanism" may seem overblown. It is not. The effects on our brain chemistry really do resemble an addict getting drugs. And measurements in animals of how motivated they are to get the reward show that modern processed food is only slightly less motivating than cocaine

Obviously there is some variation. But this is one of the prime reasons that obesity has expanded from being a fairly rare disorder to being about a third of the population. And today obesity is seen in ever younger people, and is more and more extreme. It would seem that the drug affects different people differently, and you can see from your waistline how much you are affected. (I'm hit about average, I weigh an extra 20 pounds or so.)

Of course this drug isn't exactly good for our bodies. It is no accident that the leading diseases tied to modern lifestyles are heart attack, stroke, diabetes and hypertension - all of which are tied to what we eat. Not that long ago I talked with someone who had a heart issue and just went to a cardiologist for the first time. The first lifestyle question he asked was, "How often do you eat out?" Note that where you eat out doesn't matter, because the entire industry has the message. They all know the game, and all are going to deliver the same drug. Ditto the manufacturers of processed food.

However if you go and cook food at home, you don't need to worry. You won't hit the sweet spot very easily because you lack the equipment to do it. You'll actually have to chew your chicken because before cooking you didn't break down the internal structure and fill it full of sugar, fat and salt.

The other interesting thing about cooking food at home is that it tastes better. Let's be clear, you won't crave it as much. If you've got one plate with fish you grilled at home, and another with potato chips, it will be the chips that you can't keep your hands off of. However the fish will be more enjoyable.

Let me give a concrete example. When I married my wife, her family was into pancakes made from mix. I've noticed two things about pancakes made from mix. The first is that they don't taste very good. The second is that they don't save you much work. Seriously you can't make the pancake until the pan is hot, right? Well it takes about as long to make the batter as to make the pan hot, and you do those at the same time. Therefore the mix didn't save you time!

Now when I tell people this, many claim that the pancakes they make from a mix taste great. And I tell them that they think that only because they've never had a decent pancake. Try that recipe, then come back and compare. (A funny story about that particular recipe. I had a co-worker who I gave that recipe to who it turned out had a friend who had been raving about my sister's pancakes for 30 years but didn't know how to make them! My friend was happy to be able to pass it along.)

All that said, when someone who doesn't cook faces cooking, it is scary. You don't know how anything works or what anything does. In reality all you need to do is get a recipe, follow directions, and it tends to work out. When you get brave you can experiment. Sometimes the experiments flop. But even that isn't a bid deal.

For instance yesterday I made turkey soup. It was going great until I added noodles. I envisioned firm noodles, and so it was at dinner because I put them in at the right time. But the noodles in the pot didn't stop cooking. So now instead of turkey soup I now have soggy turkey casserole. However it still tastes good and is still good for us. And I won't make that mistake again next year!

So stretch yourself. Look for an easy recipe and cook a meal. Even if you never come to enjoy cooking particularly much, you may find it worthwhile. And who knows, perhaps you'll turn out to be someone who enjoys cooking. Try it!

Friday, December 25, 2009

Work optimally, not long

It has been a long time since I entered anything here. But I'm in the mood. I've cooked, eaten, and even done cleanup after my turkey dinner. If you're uninterested in hearing about cooking turkey, skip the next paragraph.

This time I tried dry brining the turkey. That was surprisingly easy. A few days ago I filled a small bowl with 1 tbsp of salt per 5 pounds of turkey (that's 6.5 ml salt per kg), added some spices for taste (I did sage and a ground up bay leaf), mix, then spread that over my freshly bought turkey, put a plastic bag over it, then let it thaw in the fridge. Today I just roasted it normally. Based on past experience I checked regularly to baste the turkey, but it really wasn't necessary since the juice stayed in the turkey. Tomorrow I'll get supplies, then make turkey soup, which for me is the best part of having turkey dinner.

Enough about food. What I want to write about today is something that we all know in theory, but far too few of us act on. That is that it is better to work at an optimal, sustainable level than it is to work long hours. Certainly this is true in software, and likely in many other fields as well.

More precisely I would give the following rule. If you see a developer working more than 40-45 hours per week, ask yourself two questions:
  1. Will this effort be sustained less than 3 weeks?

  2. When it is over, will this person get corresponding downtime?

If the answer to either question is "no", then I strongly recommend looking at the schedule and figuring out what deadlines to slip immediately to get people back to reasonable schedules. Because you're going to compromise something, and choosing up front is much better than making it by burning people out and getting sub par work.

The reason for this rule is that people have a maximum sustained throughput for creative work when they are working somewhere between 35 and 45 hours/week. (For a reference on that see Rapid Development by Steve McConnell.) You can temporarily raise your hours above that and get improved productivity. But after a few weeks it catches up to you, your productivity drops, and you'll get less done despite working harder. And once you're in that state you've got a vicious cycle. Your lack of productivity pushes you to work harder which makes you less productive until something gives. Quite possibly your sanity.

The one thing that seems odd about this to most people is that people don't realize that their brains aren't working. But the reason for this is simple. When you're impaired, the first thing that gets impaired is your ability to judge how impaired you are. Thus your self-monitoring for impairment is not trustworthy. This effect is seen with many kinds of impairments, including chronic sleep deprivation, alcohol, strokes, and so on.

The army once did a study that showed this in a striking way. They took soldiers and put them on different sleep regimens. Among other things they found that most soldiers could operate indefinitely on 6 hours of sleep per night and would claim to be fine. But their wives said they weren't fine. Comparing results on ability tests before and after the new schedule, the wives were right and the soldiers were wrong. Which shows how easy it is to think you're still functioning when you really aren't.

Now that's the theory. What is the practice? The practice is that most of us don't do this. The reason is that most organizations manage by schedule, putting pressure on us to work harder. We put the effort out, we try to make the deadline, and then we fail to be aware of our declining output. Possibly we make the deadline, but quality suffers, bug counts suffer, and development+maintenance comes out to a lot more than it would have if we had focused on development alone.

What keeps us from doing the smart thing? For many of us, fear. While intellectually we know that we should be able to do more if we keep to a peak level, as a practical matter we're scared of being called slackers for not visibly putting out effort. Particularly if everyone else is working overtime. This fear is particularly reasonable since management usually has no idea how much developers do or why things take us so long. And is doubly so if our direct managers are themselves working too hard and therefore not themselves in the best shape.

Another major thing that keeps us from doing the right thing is that nobody likes to deliver bad news. Nobody wants to tell their manager that the schedule will slip. So we put off delivering the bad news to anyone, and try to keep it from happening. Of course this doesn't keep the schedule from slipping. But there is always the hope that someone else will be forced to admit that the schedule is going to slip. And failing that some people will deliver an incomplete and non-working piece, declare it done, then push the admission of failure off even farther.

Delivering bad news gets harder still because everyone else has also avoided doing it. Everyone delivers their bad news as gently as they can, and the manager inevitably wants to hear it as better than it is. Therefore upper management is usually seriously out of touch. The result is How Shit Happens, which is only funny because it is true.

As an example of how extreme this effect is, The Mythical Man-Month points out that in the rare cases of early projects, the good news starts coming immediately. If the project is late, there is no sign that the project will slip until about 2 weeks before it is due. And this is true no matter how long the project is actually delayed. And the driving force behind this is that nobody wants to deliver bad news, and nobody wants to hear it.

So we have a real organizational problem. How should you solve it?

The easiest problem is to ignore the problem and try to survive. This is suboptimal in virtually every way. However it is what most of us do.

A better option is to try to find an organization without the problem. They do exist. I know because I've been lucky enough to work for some. (Something that I suspect is a key ingredient is working in short iterations. Reality and management dreams tend not to get too far out of sync on a 1 week project.) But those can be hard to find. And in the current economy most people have a hard enough time just finding a job that they can't worry about the quality of the employer.

The final option is to try to make your current organization into one that works better. If not better for everyone, then at least better for you. This is the most difficult path, and there is no simple recipe for how to solve the necessary interpersonal issues that you'll encounter. However I can give you some useful ingredients.

The first is education. Go and get Rapid Development, read it, and learn what it says. It is one thing to confront your manager with the message that you need saner work hours. It is quite another to share research about how to get more done and say that you really want to do it. This works even better if you've identified multiple things you'd like to change at once, so it doesn't come off as your cherry-picking the item that you most wanted to try.

The second ingredient is knowing that delivering negative messages often is not as bad as we think it will be. Admittedly it doesn't always go well. However any manager who has been around the block a couple of times knows that software projects tend to fail, so it won't be as much of a surprise as you might think. Of course your manager is usually under pressure from above to meet deadlines and so is likely to not want to face that bad news. But it won't be a surprise.

The third ingredient is being conscious of the fact that delivering bad news early, no matter how bad it may be immediately, usually works out better in the long run. Yes, it is unpleasant and we all want to procrastinate on unpleasant things. But delivering bad news up front then better news later makes people happier in the long run than letting people think they'll get good news and disappointing them with bad news. (Incidentally if you're going to deliver bad news, take a tip from Machiavelli's The Prince and deliver it all at once, then deliver good news afterwards.)

The fourth ingredient is to understand the person you are talking to, and deliver your message in a way that is likely to be heard. A lot goes into this, but a big factor that lots of people miss is cognitive dissonance. Let me wrap up this topic, then I'll go on to explain it shortly.

The last ingredient is being aware of what the status quo is likely to cost you. If you allow yourself to be aware of that, you'll have lots of motivation to avoid that fate. And in this case motivation is a good thing.

Now let me return to cognitive dissonance. I won't explain it at length. I've done that elsewhere and likely will do it again, but here are the basics.

The principle of cognitive dissonance says that people will go to any length to maintain their positive self-image of themselves. This means that messages which threaten your positive self-image trigger defensive reactions. The more powerfully the message is delivered, the worse the defensive reaction is likely to be. (When you see such a reaction the last thing you want to do is follow up with facts proving your position. That just increases the conflict and therefore increases the defensive reaction.) What this means for any given person will vary from person to person. However if you remain aware of this phenomenon, you can often figure out where the hidden landmines are, and have hope of communicating what would otherwise be impossible.

For example, suppose that your manager thinks he or she is a pretty good manager. (Most of us tend to think we're pretty good at our jobs. No matter how good or bad we may be.) Then at all costs you must avoid threatening that self-image. Therefore you would need to avoid saying anything that even looks like you're delivering the message that there is room to be a better manager. This is true, even if said manager regularly points out things that made them better managers. To avoid the possibility of trouble, it is best to make the case that what you are saying flows from what that manager says about management.

This makes it really bad to say, I was reading this excellent book by Steve McConnell and really liked some of what his advice about managing software teams... Doing that is explaining to your manager how to manage, and anything you say after that which your manager doesn't already know and accept is going to be really difficult to accept.

It is better to say, I've been reading a good book, and it has taught me a lot about myself. It is making me aware that I am not working as effectively as I could... Making the message be about you is a reasonably safe way of side stepping the question of whether this is something the manager should know to tell you.

But it may be possible to do better still. For example consider, You know how you always tell us that we should constantly been trying to improve ourselves? Well I've gotten some ideas from a software classic, and here are some things that I'd like to try... If this is true, then you've put cognitive dissonance in your corner. Your manager will now find that a "Yes" is an opportunity for confirmation of his or her good self-opinion. Better yet, your manager will then be inclined to interpret the outcome as success, giving an even stronger confirmation of your manager's ability as a manager.

Now all that said, there is no guarantee that you can keep a sane work schedule when the going gets tough. However if you can do it, the outcome will be better for everyone. That doesn't mean the outcome will be good, but it is better and that is really worth it.

Tuesday, December 15, 2009

The use and limitations of exceptions

I'd have a longer blog but I just had a farewell party and so am do not have the most time to spend. So I'll just point to an email wrote to golang about what I see as the cases where exceptions prove useful, what the limitations of exceptions are, and some of the reasons why Go doesn't include them.

Friday, December 11, 2009

Design of a reporting system

I've had to build a lot of reports, and in my last job I build a reporting system that I think worked quite well. This post explains some of the key ideas behind that system.

One of my best ideas was to use Excel, not compete with it. Users will come with detailed requests for the exact reports they want. You can try to provide what users ask for. Inevitably they will change their mind, or the report lovingly described by someone's manager won't be what that person wants, or a million variations on this theme. This will generate an endless stream of requests as people try to get you to tweak what is being provided into their exact desire. Alternately you can make the data from your system directly accessible in Excel, and encourage people create the exact reports they want themselves in an interface they are comfortable with. Now all that you need to do is provide raw data, appropriately aggregated, and you're no longer a bottleneck. Giving people what they want rather than what the initially ask for makes for happier users.

Luckily it is easy to integrate with Excel. What you need is a way for users to design a web-based report which provides the raw data they want, and make it available with a short URL (less than 50 characters). Now a user can open up a new Excel workbook, then choose Data, Import External Data, then New Web Query. This pops open a browser. The user puts in their short URL, and the browser loads it. The user can then click on the HTML table that they want, save the web query, and Excel will run it again then fill a spreadsheet with the data in that table. Now a complex set of spreadsheets can be built off of that data, and every time they refresh it will re-run the report and get fresh data.

That's the user experience. Behind the scenes the developer needs to do several things to make this work.
  • Provide users a way to save the parameters of a given report, and create a named report. The raw parameters will tend to be too long to be saved properly. When users access this named report you must not issue a redirect. If you do a redirect it will work initially, but Excel internally remembers the redirected location. Which means that if the user tweaks the saved report, the Excel spreadsheet won't notice the tweak.

  • In your HTML give an unambiguous ID attribute to the table with the data. Excel will use this ID to locate the data. Otherwise it will try to analyze the structure of the HTML, and that will break easily when the report changes.

  • Have a way to easily specify relative dates. People frequently want reports that run to the present, run for last month, and so on. If they only had fixed dropdowns when they set up the report then the Excel spreadsheets they get won't update to what they want it to be. I solved this problem by providing an optional text field that I passed to Perl's Date::Manip to parse. (I actually improved the date parsing slightly for my purposes, but that was the base.)

Moving on, my philosophy about reports is that 1 report that can be tweaked in 100 ways is more flexible and powerful than 10 reports that can be tweaked in 10 ways each. Similarly when given a one-off reporting request, it is better to find a way to add something to an existing report to make it capable of handling that request than to do the one-off request. That is because if someone wants that feature once, someone else is likely to want it again. And there is something really satisfying about receiving the second request and being able to say, "I've already built that, here let me show you..."

Of course the challenge is converting that philosophy into action. How exactly do you go about building a flexible report that can be tweaked in many ways?

My strategy was to look at each report as an array of SQL statements. All statements before the last would create temporary tables. The last query would return the data set that I would display. Thus if I needed to pull in extra information - for instance I needed to segment data by whether a given promotion was run - when building the array I could dynamically insert a query to fetch that particular piece of information, and then pass it through all of the tables.

This strategy naturally leads to two useful features. The first is that you can add the exact SQL statements used to generate the report in an HTML comment. This is very useful for auditing purposes. The second is that you can add the feature of having a dropdown box listing the temporary tables that will be created, and allow the report to run to that point then display those intermediate results. This is helpful when debugging. And not just for the reporting engineer - it was very useful for me when the accounting department was able to come to me and say, "This report, at this step, is missing these invoices."

This strategy, of course, requires being able to create definitions for temporary tables on the fly. Most databases provide that. Whether you are in MySQL, PostgreSQL, MS SQL, Sybase, etc you have exactly the right kind of temporary table available. The glaring exception is Oracle. There are a couple of workarounds available with Oracle. I've tried automatically building queries with large subqueries. It kind of works, however the result is unreadable by end users, you can't easily display intermediate results, and I miss the ability to control execution plans by creating temporary tables and adding appropriate indexes on the fly before moving on.

The other requirement of this strategy is that your SQL statements can be customized by the addition of an arbitrary list of extra fields, join conditions, etc. I wrote more reports than I care to admit before I realized that the best way to do this is to use a templating system to build your SQL. Now that I have used templates to dynamically generate SQL statements, I wouldn't want to go back. I used Perl's Template Toolkit for that purpose. Again I tweaked it substantially for my purposes. But many different templating systems could work well.

Moving on, let's look at how I organized things under the hood.

Obviously I needed complex forms with lots of checkboxes, input boxes, dropdowns, and the like. I chose to make each form control into an object that knew how to find its data out of the form parameters, do validation, write out its HTML, and pass information on itself to the templates. Some of the objects had SQL statements attached so that I didn't have to duplicate logic between reports.

There are endless arguments in software circles about separating presentation from content. In some circles mixing content and presentation as these objects did is anathema. In general web development I would not be inclined to move the HTML for form elements into what is basically a model object. However in this case it worked out very well. The form is so directly tied to the action of the report that it wouldn't make sense to have different people work on the two. These reports were for internal use, and so functionality mattered more than aesthetics. And as a practical matter this choice made the addition of options to forms extremely easy.

And, of course, I needed to have a way to coordinate everything. I did that with a report object. The report class had methods to create each kind of form object. It had a method to be displayed. When displayed it would show all of the form elements, check whether the report should be generated, if so it would run all of the queries, and display the report, and (of course), it would append necessary documentation.

With all of these pieces the code for a trivial report could look something like this:

use Report;

my $report = Report->new();

name => "user",
label => "Who is this for?",

name => "age",
label => "How old are you?",

my @queries;

push @queries, {
temp_table => "results",
sql => qq{SELECT :user as "User", [% age %] as "Age", current_date as "Today"},
args => {
user => $report->user,

title => "My Report",
queries => \@queries,
doc => qq{
<li> <i>User:</i> The user this report was generated for.</li>
<li> <i>Age:</i> Their reported age.</li>
<li> <i>Today:&t;/i> The current date.</li>

Now this report isn't very impressive. Also passing the age into the SQL as a template variable is poor style. (I did that just to show what the templating looked like.) But if you changed this to having a dozen form controls tweaking a data processing step with a half-dozen queries then you could have a very powerful and flexible report. Add in a few more reports, the ability to combine independent options, and the ability for users to build on top of this in Excel, and you wind up with a surprisingly powerful and flexible reporting system.

Thursday, December 10, 2009

Learn to negotiate

I can't believe it has been a month since I've added a post. A lot has been going on. The most important being, of course, that I've accepted a job at Google starting in the new year. I am convinced that this is a good move that will make me much happier in the long run. I'll be doing a more interesting job with a new set of tools at a company filled with great people, what's not to like?

However the process has reminded me that we can all benefit from learning to negotiate. Now I'll be the first to say that I don't enjoy negotiating. I prefer to find ways to grow the pot rather than trying to take a bigger share of it. But a little reading and practice with negotiation can pay off very well.

A few years ago I asked a friend who was a very good negotiator what book on negotiation he would recommend. I do this kind of thing every so often with different areas, and I usually learn something from the books I'm directed to. He recommended Start With No. Reading this book has easily been worth tens of thousands of dollars to me already. Reduced to a nutshell, the idea is that progress in negotiation comes when you ask questions that the other person can reasonably say no to. And your results in negotiations will be better when you don't need to hear a yes.

A second book that I recently read on negotiation was Bargaining for Advantage. From a theoretical perspective this book is clearly the stronger of the two. If you wish to understand the process of negotiation, learn to recognize different negotiation situations, and negotiate regularly, this is clearly the book for you. It is clearly best in class, and will work for a wide variety of negotiation styles. However it seems to me that if you don't already come with a wealth of knowledge and experience on negotiation, you can easily read this book and find yourself saying, "That's interesting, but what do I do now?"

For me personally, a lifetime of avoiding negotiation (and therefore doing it badly when I had to) left me without any negotiation skills to speak of. So Start With No was the better starting book.

With that said, how did my last negotiation go? From my point of view, very well. My compensation is structured differently from my last job, but overall is similar. From what I've heard, people moving to Google usually wind up accepting a pay cut. Due to personal circumstances I honestly couldn't do that. Without having learned some basics of negotiation I am sure that I couldn't have done that. Heck, without what I've learned about negotiation I doubt that I would have been able to structure my resume to make it clear to potential employers how valuable I could be. And without that, I wouldn't have even had the job negotiation in the first place!