A couple of posts ago I discussed some complications in relations between the sexes. Nobody seemed too offended or disagreed too much, so I'll tackle the topic again, in a more haphazard way. (I touched on a somewhat related topic lst year when I claimed that males may have started as parasites.)
But as before, if you think you'll be offended you should stop reading now. Unlike before I'll actually touch on something interesting that applies to homosexuals at the end, though most of this is not about homosexuality.
My last blog reminded me of a book I read years ago that I wanted to re-read. So now I've bought it and am re-reading it. The book is Shadows of Forgotten Ancestors. It starts off slowly, with a basic overview of the history of the Earth, the history of life, how genes work, and so on. All pretty basic if you're conversant with science, though at some point it is likely to get detailed enough that you'll probably to find yourself in unfamiliar territory.
But what I like about it is the second half. This is where we encounter a discussion of the documented effects of testosterone on personality, an analysis of the social lives of our near primate relatives, and so on. I won't try to summarize, so much as to hit the some high points. Anyone who believes that women are basically men without the penis who can have babies should read through the effects of testosterone on personality. Anyone who has wondered why we say, "Fuck you!" when we're mad should read through accounts of how sex and dominance are intertwined in different primates. I'm not sure what lessons we should be drawn from a comparisons of militaristic chimpanzees with the free love bonobos, both of whom are very closely related to us, but I certainly saw strong parallels between both societies and human groups. All in all I rank this book as being one of of the strongest challenges to my preconceptions of how I want to see the world.
Moving on, a friend told me that a lot of what I said in my previous post is echoed in more detail, at longer length, and more amusingly in Sperm are from Men, Eggs are from Women. I haven't read the book, but the reviews look promising.
If I'm guessing right from the reviews it touches on one of my favorite pieces of evolutionary trivia: that there is good evidence that humans have significant sexual selection for large penises. That such selection exists is not surprising in light of all of the jokes and obvious sexual interest. (I will never forget having it pointed out to me 15 years ago women could estimate the size of a guy's penis from looking at the bulge in his pants. I was quickly assured that I had nothing to worry about in that department, which did absolutely nothing to lessen my self-awareness of people potentially looking at my crotch for the next couple of weeks...) However the actual concrete evidence is the simple fact that the human penis is about 3 times the size that would be expected based on estimates made from our size and the proportions of our close relatives the chimpanzees and gorillas.
Moving on, by coincidence this week on NPR I heard an interview with the authors of Why Women Have Sex: Understanding Sexual Motivations from Adventure to Revenge (and Everything in Between). It sounded interesting in the interview, and at some point I suspect I'll buy it as well.
I don't know if it is in the book, but there was an interesting piece of trivia that came up in the interview. It turns out that if you take a man and dress him up in a Burger King outfit, he's much less attractive to women than if you dress him up in a suit. By contrast men don't care about the clothing on a woman. In comparisons they'll judge her equally attractive in a Burger King outfit and in a nice cocktail dress. This ties very well with what I said in my last post about how women are looking for a good provider (status helps) while men are looking for evidence of fertility (status doesn't matter). Which helps explain why women remember what people wear while most men I know, including me, can't really remember what someone was wearing 30 seconds after they leave.
Moving on to men noticing signs of fertility, how sensitive are we? The answer is more than you'd think. For instance research has found that men react to how fertile a stripper is. It turns out that when you analyze how much strippers make in tips per night, the closer she is to ovulation, the more she makes, with income varying by a factor of 2. By contrast strippers who are on the pill permanently make about what strippers who are not on the pill would make during their period.
How do men notice? I don't know whether the full story is known, but at least some of it is behavioral. Women who are ovulating have been demonstrated to dress more provocatively, act more provocatively, put down the appearance of other women more, and show an increased attraction to men, particularly ones with masculine appearances. All of these behavioral cues reverse during her period. Even when strippers are trying to be provocative, perhaps the ones who are ovulating have a slight edge. Perhaps there are as yet unknown pheromones that men react to. Whatever the cause, men do react.
Moving on, the point about men with a masculine appearance is interesting to me. When researchers examined the subject, both men and women rank feminized faces as more attractive. But when push comes to shove, women are interested in other traits more than appearance, which gives a definite edge to men with masculine features. (Masculine is here defined by things such as a big jaw, square face, prow nose, and heavy brow. More on why it is defined that way shortly.)
Interestingly researchers in other studies have found that people form a variety of first impressions on men based on how masculine or feminine their faces are. For instance fathers are more likely to trust their teenage daughter's with a man whose face is feminine. But what can appearance tell you?
More than you'd think. It turns out that all of the characteristics that are linked with a masculine face are tied to testosterone exposure. Men who are judged as having masculine faces do turn out to have higher testosterone levels in their saliva. And the various personality judgments that are made based on this trait have a surprisingly good grounding in the known personality effects of testosterone.
Speaking personally, I am the kind of man who is more likely to be described as "beautiful" than "rugged". As much as I don't like it, this is concrete evidence of somewhat low testosterone levels when I was growing up, and it affects how people interact with me. In ways that most people don't even notice.
Moving on, the whole masculine/feminine thing leads to an odd piece of trivia that is not too widely known. Most people assume that gay men must be responsible for homosexual abuse. Not so fast. The little data that exists says that heterosexual men are far more likely to engage in homosexual abuse of boys than homosexual men! Now let's be clear, once a boy passes puberty, homosexual men may develop an interest. Legally an adult man and a 16 year old teenager might be child abuse. But that teen has sexually matured. Those examples notwithstanding, homosexual men do not seem to be attracted to sexually immature boys.
Why? Well when you ask adult heterosexuals who abuse boys why they are attracted to boys, all of the answers have to do with the feminine characteristics that boys have. By contrast when gay men are asked what they like about other men they cite masculine characteristics which boys don't have. It seems that attraction is tied to perceptions of masculinity/femininity in a very direct way that leaves little to no room for someone to be a gay pedophile. (I know of no research on the topic, but I have to wonder if this connects in any way to the fact that pedophilia appears to be very rare among women.)
As always I could ramble on, but this is enough for this post.
Friday, January 22, 2010
Monday, January 18, 2010
Solving Project Euler problems
Late last year I got into Project Euler in my spare time for a couple of months. I solved over 150 of the problems, then I lost interest and have only sporadically solved any since. However I know I could do many more if I put some more time into it. I also contributed P240 which a number of people liked.
That's not a great track record, but it is not a bad one either.
This post is my advice for people who can solve basic Project Euler problems but want to get better at solving them. I happened to use Perl to solve problems, but the advice should work in a variety of languages. The techniques I used use closures a lot, but that's because that's the kind of guy I am.
First if you are unable to solve the first 5 then I would suggest that you're missing some basic background. I would suggest learning a programming language first, then some math. Only after that should you try Project Euler.
If you're going to tackle Project Euler problems, there are some things that come up over and over again. In particular I would recommend learning how to use dynamic programming to solve problems. That comes up a lot. Secondly I would recommend learning how to handle infinite streams of data. Thirdly do not be afraid of attempting brute force techniques with clever bucketing and pruning. If your program runs in a minute, you've found what they were looking for.
With those basic techniques out of the way, as you solve problems you'll develop a library of functions you use over and over again. Here are the functions in my library, in no particular order, with no particular attempt at good design (after all PE is just one throw away program after another...):
Now on to the Pythagorean triple iterator. How do I iterate through them? Well first I use Euler's formula to generate all primitive triples in the order that I want, and then I merge together all of the multiples of that stream to get all triples. To understand how that merging works, it would be good to go back to the infinite streams article and actually read it.
This sounds simple in theory. But in practice you have a problem. I'm merging a lot of data. I do that by maintaining a partially sorted pool of things I want to merge in, and then always taking the least next element. There is a data structure for that. It is called a heap. A heap is a tree, which has the heap property that each node is less than or equal to its children. This makes the top node at the root of the tree the smallest element. (Why do CS people always draw trees upside down so that the root is at the top? Really it would make more sense to call it a "root system" or else to draw it the other way around. Oh well, that's the convention.)
Luckily CPAN has multiple heap implementations. Unfortunately the one named Heap sucks, and I couldn't be bothered to look through the rest to find one that met my needs. So I took pointers from an implementation by Abigail, and implemented my own. The clever (and standard) trick in that implementation is that arrays can be used to represent a tree. The two nodes below the n'th element in the tree are at 2n+1 and 2n+2. So you can take an array of data, turn it into a heap, and then manipulate it as if it was a tree.
Writing your own heap implementation is a good exercise, so I'll just give the methods in my heap implementation for anyone who wants to duplicate it.
Hopefully this helps others who are trying to tackle Project Euler problems and haven't made much headway yet.
That's not a great track record, but it is not a bad one either.
This post is my advice for people who can solve basic Project Euler problems but want to get better at solving them. I happened to use Perl to solve problems, but the advice should work in a variety of languages. The techniques I used use closures a lot, but that's because that's the kind of guy I am.
First if you are unable to solve the first 5 then I would suggest that you're missing some basic background. I would suggest learning a programming language first, then some math. Only after that should you try Project Euler.
If you're going to tackle Project Euler problems, there are some things that come up over and over again. In particular I would recommend learning how to use dynamic programming to solve problems. That comes up a lot. Secondly I would recommend learning how to handle infinite streams of data. Thirdly do not be afraid of attempting brute force techniques with clever bucketing and pruning. If your program runs in a minute, you've found what they were looking for.
With those basic techniques out of the way, as you solve problems you'll develop a library of functions you use over and over again. Here are the functions in my library, in no particular order, with no particular attempt at good design (after all PE is just one throw away program after another...):
- gcd Calculates the GCD using the Euclidean Algorithm
- prime_iterator Returns a function which, every time it is called, will return the next prime. Internally I use the Sieve of Eratosthenes to generate this list. Note that in this code an "iterator" is always a function which iterates through some (possibly infinite) set as you call it.
- is_prime Takes a number, makes sure I've generated all of the primes up to that number, then checks whether it is prime.
- is_prime2 Takes a number, generates all of the primes up to the square root of that number, then uses trial division to check whether it is prime. This routine is slower than the previous if you're going to check a lot of primes, but uses much less memory. P118 is a good example of where you would use this version.
- ith_prime Access the list of primes as if it were an array. (I had ith_prime(0) return 2, ith_prime(1) return 3, and so on. Remember what I said about not being carefully designed?)
- factor Takes a number, returns a hash of prime factors and multiplicities. Trial division by primes up to the square root was efficient enough for all of the PE problems that I tackled.
- factors Returns the prime factors of a number as an array with duplication to represent multiplicity.
- n_pow_m_mod_k What the name says. I used it for P133 and stuck it in my library because I wanted to reuse it for P66 but it didn't work for the second one. (Search for Pell's Equation for how to handle that problem.)
- minimal_divisible_repunit See P129 and P130 for what this is about. Internally this reuses the n_pow_m_mod_k function in its calculations.
- pythagorean_triple_iterator Iterates through all Pythagorean triples ordered by the lengths of their sides. I'll have more to say on the algorithm in a bit.
- generate_stream A promise is a function which, every time it is called, will return some set of future values you haven't reached yet, returning nothing when it runs out. This function takes a promise and lets you treat it like an infinite array, ask for the 1st, 5th or 500th element and it will generate the promise until it has enough stuff then return the value.
- generate_memoryless_stream Like the previous but instead of accessing it like an array you access it like an iterator. This is useful for cases where it is convenient to calculate a block of values at a time. A perfect and pertinent example is the Sieve of Eratosthenes, which calculates blocks of primes at a time.
Now on to the Pythagorean triple iterator. How do I iterate through them? Well first I use Euler's formula to generate all primitive triples in the order that I want, and then I merge together all of the multiples of that stream to get all triples. To understand how that merging works, it would be good to go back to the infinite streams article and actually read it.
This sounds simple in theory. But in practice you have a problem. I'm merging a lot of data. I do that by maintaining a partially sorted pool of things I want to merge in, and then always taking the least next element. There is a data structure for that. It is called a heap. A heap is a tree, which has the heap property that each node is less than or equal to its children. This makes the top node at the root of the tree the smallest element. (Why do CS people always draw trees upside down so that the root is at the top? Really it would make more sense to call it a "root system" or else to draw it the other way around. Oh well, that's the convention.)
Luckily CPAN has multiple heap implementations. Unfortunately the one named Heap sucks, and I couldn't be bothered to look through the rest to find one that met my needs. So I took pointers from an implementation by Abigail, and implemented my own. The clever (and standard) trick in that implementation is that arrays can be used to represent a tree. The two nodes below the n'th element in the tree are at 2n+1 and 2n+2. So you can take an array of data, turn it into a heap, and then manipulate it as if it was a tree.
Writing your own heap implementation is a good exercise, so I'll just give the methods in my heap implementation for anyone who wants to duplicate it.
- new Construct a new Heap. There is one optional argument, which is the comparison function. (Which is passed in as a closure, as you might expect.)
- append Push a new element on the end, then let it "bubble up" to its natural position. Here is what I mean by "bubbling up". If it is currently at position $n > 0 then its parent is at position int(($n-1)/2). If it is greater than or equal to its parent you're done. Otherwise you need to swap with its parent and continue until it either has a smaller parent or reaches the root.
- top Peek at the root element.
- swap_top Take the root element out, replace it with something else, and let the something else "fall down" to its natural level. Returns the previous top.
- extract First pop the last element off of the array, then call swap_top. This trick gives you the least element in the heap and makes everything else be represented in an array that is 1 smaller than the previous.
Hopefully this helps others who are trying to tackle Project Euler problems and haven't made much headway yet.
Friday, January 15, 2010
Why are relations between the sexes so complicated?
Note: The subject of this blog post is likely to offend many people. If you think you're one of them the do us all a favor and don't read it.
Disclaimer. Everything in this essay is about my understanding of mainstream heterosexuality. If you disagree, remember that I'm talking about my opinions, not facts. Also, a certain proportion of people wind up homosexual, asexual, or otherwise don't fit the sexual norms. From the surveys I've seen, definitely over 90% and probably over 95% of people are, some experimentation notwithstanding, relatively normal heterosexuals. I have no idea how much or little any of this applies to the remaining minority.
You have been warned.
When I was young I wondered why relations between the sexes are so complicated. After all both genders desire the same thing, why not pair off in some orderly fashion with much less pain and complication? I could see that things didn't work that way, but it seemed to me to be a more sensible way for the world to work than how it actually does.
As I've grown older I've learned about complication after complication. As I learn about complications and see why they happen, the overall picture becomes clearer. Yes, the simpler world I imagined would be convenient, but it can't happen. Why not?
Well let's take my first assumption, that the sexes desire the same thing. We can actually tell what they desire from the questions they ask, and Google correlates those questions and summarizes what they find in autocompletes. When you compare what women and men want their boyfriends and girlfriends to do you get an interesting picture. Men and women want different things.
If we oversimplify, men want sex and women want commitment. Oversimplifying a tad less, men want sex and are willing to provide commitment as a way of getting it, while women want commitment and are willing to provide sex as a way of getting it. If you review the bf/gf comparison and pair the first few questions up, they answer each other in amusing ways. Partially that is coincidence. And partially it is that meeting of different goals.
Evolutionarily that picture makes a lot of sense. It takes little work for men to make children and a lot of work for women to. Therefore biologically men should be selected for a willingness to make children. However raising a human child takes enough effort to need multiple adults. Therefore it is to a woman's advantage to secure that commitment. A man who helps raise his own children benefits his genes, so he should be able to be convinced to do so. The point of lining up help for a woman is to make having children possible, so once the help is lined up, a willingness to have children should follow.
It should be noted that what I've described is not particular to humans. The same evolutionary tendencies and results can be seen in a wide variety of pair bonding species. Though when you apply it to humans people are more likely to be offended than when you describe some type of birds. Though for obvious reasons I find the human case of more personal interest.
It should also be noted very clearly that these strategies are not conscious on the part of the people involved. We have been genetically selected for traits that will cause us to act in these ways. And we directly react to those traits. For instance when a man meets an attractive woman, the parts of his brain that are not associated with libido are invited to take a short vacation. Sometimes with amusing results, such as a temporary inability to speak coherently. He certainly doesn't want this sudden burst of stupidity, but the immediate refocusing of his priorities aligns him with the evolutionarily successful agenda he is literally bred to follow.
Now let's introduce a complication. Sexual partners are not created equal. We have preferences about who of the opposite sex we'd like to have relations with. Preferences vary widely, but some interesting generalizations can be made.
First of all, what do men want? Well health is a good thing. Traits such as clear skin and symmetric faces are signs of a good immune system. Humans have established secondary sexual characteristics such as breasts as signs, and men's attraction to them is obvious. Signs of being good at childbearing as well. Cross-cultural studies have supported the importance of the waist-hip ratio, and even a brief acquaintance with the difficulties of human childbirth with large heads can illustrate why. And, of course, youth.
The subject of attraction to youth is an interesting one. I won't elaborate much on that topic, instead I'll direct your attention to Neotenty and Two-Way Sexual Selection in Human Evolution by David Brin, which is one of the most fascinating pieces of speculation on human evolution that I've read. I don't think he is entirely right, but I think he is right on several things I never thought of until I read that essay. The short summary that doesn't do justice to the essay is that men are attracted to youthful women, which poses the challenge of telling apart young mature women from as yet immature women. We do this through secondary sexual characteristics such as breasts. But this mechanism is fallible, and results in high incidences of pedophilia. (The essay has several parts, I would advise reading all of it.)
Since I have it, I'll give one random instance supporting Brin's thesis from the Google data that I have that he didn't. If you type into Google How do I get my ____ to with either girlfriend or wife in the blank, high among the suggestions will be shave. (Presumably the pubic area is referred to.) Why do so many men want this? Well general lack of hair is a sign of youth. But lack of pubic hair is a direct sign that a woman is not yet sexually mature. Isn't this awfully close to pedophilia?
Moving on, what do women want? Well, obviously, commitment from someone who will be a good care giver. Therefore it is no surprise that cross-cultural surveys (see Brin's essay for a reference) have found that top characteristics women say they desire are kindness, intelligence, and self-confidence. And other desired characteristics include accomplishment, reputation, health, vigor, reliability, and sense of humor. (Random note, the desire to prove accomplishment and reputation pushes men to create many social groups with their own hierarchies that men can try to get on top of.) All of these traits are associated with men who will have the ability and inclination to make good care givers. Note that physical handsomeness is not on the list. Women appreciate it but rate it more lowly, except when looking for extra-marital affairs. (A complication we haven't yet reached.)
Of course the above list isn't complete. For instance there is evidence that women react to pheromones, particularly ones that suggest that a man has a very different immune system. (When people with different immune systems mate, their children are likely to get protection from a broad spectrum of diseases.) Also what women react to and what they say they react to aren't always perfectly aligned, but that detail will have to wait.
OK, time for another complication that causes the pattern to fold in on itself into a pattern of fractal complexity. We have preferences in mates. We each would like to wind up with the best mate(s). But we can't all wind up with the best. Therefore each person tries to wind up with the best person they can, and the extent which we're willing to compromise to get a person depends on the person in question.
What do I mean by this? Well recall that men are aiming for sex, and are willing to provide commitment to get it. Women are aiming for commitment, and are willing to provide sex to get it. The million dollar question as a couple engages is who budges first. In general the higher status the other, the more willing we are to compromise. So the same man who may be willing to claim undying devotion for a chance at a gorgeous woman may be willing to sleep with an unattractive one, but won't commit to her. And a woman who may be willing to have sex with a high status man in hope of commitment may be only willing to have sex with a lower status man after commitment is established.
Tiger Woods provides a good illustration of the latter dynamic. Many women were interested enough in his wealth and talent to give him sexual access. However part of the dynamic was that he tried to make each believe that she was the other woman in his life, so there was a hope that at some point commitment could happen. When the news of his affairs broke and they reach realized how badly they'd been had, they got upset and one by one went public.
Now if you go back and look at the list of traits women want, you have a problem. The characteristics they are looking for are not immediately visible. Therefore women need to read between the lines to establish them. This pushes them towards judging based on secondary traits. For instance a man who is attractive to women who know him well is likely to have several positive characteristics, and therefore is judged as potentially more interesting. And anything that suggests such attractiveness will be seen as positive.
If you read a guide for pickup artists (I highly recommend that every woman should, just so you know what to watch out for), you'll find some odd consequences of this. For instance a man who demonstrates initial disinterest in a woman is demonstrating that he has access to more attractive women. This is evidence that he has what women wants, and therefore must be of more interest. The fastest way to demonstrate disinterest is to make an apparently thoughtless and somewhat insulting comment. If he follows up with nice comments then he has portrayed himself as being of interest but within reach. If the woman believes this message then she is more likely to offer sex without having required commitment first. Which is exactly what the pickup artist wants.
What woman would say that she wants a guy who subtly puts her down? None, obviously. Yet the technique is apparently effective on a lot of women! (Or at least on enough to make pickup artists happy.)
Moving on, what about the way that women tend to be insecure in relationships and the various games they play? I see at least two purposes of these. The first is to test the strength of a man's commitment. It is easy for men to say that they are committed, but if a guy stays through an emotional coaster ride and some demands then he's much more likely to actually be committed. But it isn't simple, because if she pushes it too far, then she has no relationship. How far she can push it will depend on the relationship, which varies widely.
The second purpose is to set clear expectations about how she wants him to behave. This pushes him towards accepting higher levels of commitment, and also makes it clear to him what she will and will not accept.
For an example of a book that helps women use these strategies, see The Rules. I would recommend that all single guys read it so that you can see what you need to defend yourself against. As you'll note from the reviews, it is very controversial, there is roughly an even divide between people who are offended and women who are happy with the men they landed using the strategy.
This brings us to an important point. I am trying to describe the natural dynamics as I see them. Those dynamics tend to be more clear when you look at people who consciously try to manipulate them. The two examples that I've used are pickup artists and The Rules. Most people's natural reaction to those strategies is to dislike how manipulative they are. A second reaction that follows closely is to point out that they result in significant power imbalances. Both reactions are perfectly accurate. They are manipulative and do cause power imbalances. However the manipulation mimics what happens naturally. Power imbalances are common within relationships. The point of the manipulation is to create a power imbalance that gets the manipulator's agenda to the fore. As much as we decry the means, most of us would be interested in the result.
Continuing, let's move on to a tricky complication. Cheating.
It is clear from the basic male agenda why men would cheat. Men cheat because more sexual encounters means more chances to have babies. But women don't want their men to cheat for fear that he'll wind up attached to someone else and leave. And men who are properly bonded will resist cheating, because that is part of what it means to be bonded. So men will not universally cheat, but will at least feel the temptation.
From the basic female agenda, cheating is not immediately obvious. After all it is not a good way to get commitment. But a woman who has a guy doesn't need commitment from anyone else. She has her caregiver, now she wants good genetic material for babies. Cheating offers her two advantages. The first is diversity of genetic material - you never know what combination of genes will work out best and experimentation has value. The second is that she can get superior genetic material than the guy she can get to commit. Of course the downside is that if she gets discovered, she may find herself out of a caregiver.
I've seen widely varying estimates on the levels of cheating in people. I know that some pair-bonded bird species have been found with up to 40% of the population born due to cheating. Humans are hopefully quite a bit lower than that. However one estimate I saw several years ago from England found that about 10% of children are not the children of the person who thought he was the father. That's pretty significant. And in any case it is high enough to contribute to high school biology teachers' reluctance to demonstrate the laws of genetics using the families of their pupils as examples. And is also high enough that a common first reaction among men to seeing his child is to look for a feature that verifies it is truly his. (I've experienced this reaction first hand, and felt silly but did it anyways. I also know someone who noticed that the child wasn't his, with appropriately bad results...)
At this point you get an interesting divergence between men and women. The reasons why men would want relationships and want to cheat are basically the same, so they are attracted to the same women for both purposes. (Though his standards are usually higher for commitment than cheating.) However women are different. A woman is looking for a relationship and affairs for different reasons. Therefore they are attracted to different men for those purposes. In particular for a relationship women want men who they think will make good caregivers. But for affairs women care much more about physical attractiveness. So attractiveness does matter to women, but not in as straightforward a way as it does for men. (Remember that when discussing Brin's essay I said I would come back to the survey result that women care more about appearance for extra-marital affairs than for relationships? Well I just did...)
I could go on but I've rambled quite enough and probably offended a good number of people in the process. Hopefully people found it interesting food for thought, whether or not you agree with all of it. I know that as recently as a few years ago I would have strongly disagreed with several items I stated, and wouldn't be surprised if I disagreed with several others in a few more years.
Disclaimer. Everything in this essay is about my understanding of mainstream heterosexuality. If you disagree, remember that I'm talking about my opinions, not facts. Also, a certain proportion of people wind up homosexual, asexual, or otherwise don't fit the sexual norms. From the surveys I've seen, definitely over 90% and probably over 95% of people are, some experimentation notwithstanding, relatively normal heterosexuals. I have no idea how much or little any of this applies to the remaining minority.
You have been warned.
When I was young I wondered why relations between the sexes are so complicated. After all both genders desire the same thing, why not pair off in some orderly fashion with much less pain and complication? I could see that things didn't work that way, but it seemed to me to be a more sensible way for the world to work than how it actually does.
As I've grown older I've learned about complication after complication. As I learn about complications and see why they happen, the overall picture becomes clearer. Yes, the simpler world I imagined would be convenient, but it can't happen. Why not?
Well let's take my first assumption, that the sexes desire the same thing. We can actually tell what they desire from the questions they ask, and Google correlates those questions and summarizes what they find in autocompletes. When you compare what women and men want their boyfriends and girlfriends to do you get an interesting picture. Men and women want different things.
If we oversimplify, men want sex and women want commitment. Oversimplifying a tad less, men want sex and are willing to provide commitment as a way of getting it, while women want commitment and are willing to provide sex as a way of getting it. If you review the bf/gf comparison and pair the first few questions up, they answer each other in amusing ways. Partially that is coincidence. And partially it is that meeting of different goals.
Evolutionarily that picture makes a lot of sense. It takes little work for men to make children and a lot of work for women to. Therefore biologically men should be selected for a willingness to make children. However raising a human child takes enough effort to need multiple adults. Therefore it is to a woman's advantage to secure that commitment. A man who helps raise his own children benefits his genes, so he should be able to be convinced to do so. The point of lining up help for a woman is to make having children possible, so once the help is lined up, a willingness to have children should follow.
It should be noted that what I've described is not particular to humans. The same evolutionary tendencies and results can be seen in a wide variety of pair bonding species. Though when you apply it to humans people are more likely to be offended than when you describe some type of birds. Though for obvious reasons I find the human case of more personal interest.
It should also be noted very clearly that these strategies are not conscious on the part of the people involved. We have been genetically selected for traits that will cause us to act in these ways. And we directly react to those traits. For instance when a man meets an attractive woman, the parts of his brain that are not associated with libido are invited to take a short vacation. Sometimes with amusing results, such as a temporary inability to speak coherently. He certainly doesn't want this sudden burst of stupidity, but the immediate refocusing of his priorities aligns him with the evolutionarily successful agenda he is literally bred to follow.
Now let's introduce a complication. Sexual partners are not created equal. We have preferences about who of the opposite sex we'd like to have relations with. Preferences vary widely, but some interesting generalizations can be made.
First of all, what do men want? Well health is a good thing. Traits such as clear skin and symmetric faces are signs of a good immune system. Humans have established secondary sexual characteristics such as breasts as signs, and men's attraction to them is obvious. Signs of being good at childbearing as well. Cross-cultural studies have supported the importance of the waist-hip ratio, and even a brief acquaintance with the difficulties of human childbirth with large heads can illustrate why. And, of course, youth.
The subject of attraction to youth is an interesting one. I won't elaborate much on that topic, instead I'll direct your attention to Neotenty and Two-Way Sexual Selection in Human Evolution by David Brin, which is one of the most fascinating pieces of speculation on human evolution that I've read. I don't think he is entirely right, but I think he is right on several things I never thought of until I read that essay. The short summary that doesn't do justice to the essay is that men are attracted to youthful women, which poses the challenge of telling apart young mature women from as yet immature women. We do this through secondary sexual characteristics such as breasts. But this mechanism is fallible, and results in high incidences of pedophilia. (The essay has several parts, I would advise reading all of it.)
Since I have it, I'll give one random instance supporting Brin's thesis from the Google data that I have that he didn't. If you type into Google How do I get my ____ to with either girlfriend or wife in the blank, high among the suggestions will be shave. (Presumably the pubic area is referred to.) Why do so many men want this? Well general lack of hair is a sign of youth. But lack of pubic hair is a direct sign that a woman is not yet sexually mature. Isn't this awfully close to pedophilia?
Moving on, what do women want? Well, obviously, commitment from someone who will be a good care giver. Therefore it is no surprise that cross-cultural surveys (see Brin's essay for a reference) have found that top characteristics women say they desire are kindness, intelligence, and self-confidence. And other desired characteristics include accomplishment, reputation, health, vigor, reliability, and sense of humor. (Random note, the desire to prove accomplishment and reputation pushes men to create many social groups with their own hierarchies that men can try to get on top of.) All of these traits are associated with men who will have the ability and inclination to make good care givers. Note that physical handsomeness is not on the list. Women appreciate it but rate it more lowly, except when looking for extra-marital affairs. (A complication we haven't yet reached.)
Of course the above list isn't complete. For instance there is evidence that women react to pheromones, particularly ones that suggest that a man has a very different immune system. (When people with different immune systems mate, their children are likely to get protection from a broad spectrum of diseases.) Also what women react to and what they say they react to aren't always perfectly aligned, but that detail will have to wait.
OK, time for another complication that causes the pattern to fold in on itself into a pattern of fractal complexity. We have preferences in mates. We each would like to wind up with the best mate(s). But we can't all wind up with the best. Therefore each person tries to wind up with the best person they can, and the extent which we're willing to compromise to get a person depends on the person in question.
What do I mean by this? Well recall that men are aiming for sex, and are willing to provide commitment to get it. Women are aiming for commitment, and are willing to provide sex to get it. The million dollar question as a couple engages is who budges first. In general the higher status the other, the more willing we are to compromise. So the same man who may be willing to claim undying devotion for a chance at a gorgeous woman may be willing to sleep with an unattractive one, but won't commit to her. And a woman who may be willing to have sex with a high status man in hope of commitment may be only willing to have sex with a lower status man after commitment is established.
Tiger Woods provides a good illustration of the latter dynamic. Many women were interested enough in his wealth and talent to give him sexual access. However part of the dynamic was that he tried to make each believe that she was the other woman in his life, so there was a hope that at some point commitment could happen. When the news of his affairs broke and they reach realized how badly they'd been had, they got upset and one by one went public.
Now if you go back and look at the list of traits women want, you have a problem. The characteristics they are looking for are not immediately visible. Therefore women need to read between the lines to establish them. This pushes them towards judging based on secondary traits. For instance a man who is attractive to women who know him well is likely to have several positive characteristics, and therefore is judged as potentially more interesting. And anything that suggests such attractiveness will be seen as positive.
If you read a guide for pickup artists (I highly recommend that every woman should, just so you know what to watch out for), you'll find some odd consequences of this. For instance a man who demonstrates initial disinterest in a woman is demonstrating that he has access to more attractive women. This is evidence that he has what women wants, and therefore must be of more interest. The fastest way to demonstrate disinterest is to make an apparently thoughtless and somewhat insulting comment. If he follows up with nice comments then he has portrayed himself as being of interest but within reach. If the woman believes this message then she is more likely to offer sex without having required commitment first. Which is exactly what the pickup artist wants.
What woman would say that she wants a guy who subtly puts her down? None, obviously. Yet the technique is apparently effective on a lot of women! (Or at least on enough to make pickup artists happy.)
Moving on, what about the way that women tend to be insecure in relationships and the various games they play? I see at least two purposes of these. The first is to test the strength of a man's commitment. It is easy for men to say that they are committed, but if a guy stays through an emotional coaster ride and some demands then he's much more likely to actually be committed. But it isn't simple, because if she pushes it too far, then she has no relationship. How far she can push it will depend on the relationship, which varies widely.
The second purpose is to set clear expectations about how she wants him to behave. This pushes him towards accepting higher levels of commitment, and also makes it clear to him what she will and will not accept.
For an example of a book that helps women use these strategies, see The Rules. I would recommend that all single guys read it so that you can see what you need to defend yourself against. As you'll note from the reviews, it is very controversial, there is roughly an even divide between people who are offended and women who are happy with the men they landed using the strategy.
This brings us to an important point. I am trying to describe the natural dynamics as I see them. Those dynamics tend to be more clear when you look at people who consciously try to manipulate them. The two examples that I've used are pickup artists and The Rules. Most people's natural reaction to those strategies is to dislike how manipulative they are. A second reaction that follows closely is to point out that they result in significant power imbalances. Both reactions are perfectly accurate. They are manipulative and do cause power imbalances. However the manipulation mimics what happens naturally. Power imbalances are common within relationships. The point of the manipulation is to create a power imbalance that gets the manipulator's agenda to the fore. As much as we decry the means, most of us would be interested in the result.
Continuing, let's move on to a tricky complication. Cheating.
It is clear from the basic male agenda why men would cheat. Men cheat because more sexual encounters means more chances to have babies. But women don't want their men to cheat for fear that he'll wind up attached to someone else and leave. And men who are properly bonded will resist cheating, because that is part of what it means to be bonded. So men will not universally cheat, but will at least feel the temptation.
From the basic female agenda, cheating is not immediately obvious. After all it is not a good way to get commitment. But a woman who has a guy doesn't need commitment from anyone else. She has her caregiver, now she wants good genetic material for babies. Cheating offers her two advantages. The first is diversity of genetic material - you never know what combination of genes will work out best and experimentation has value. The second is that she can get superior genetic material than the guy she can get to commit. Of course the downside is that if she gets discovered, she may find herself out of a caregiver.
I've seen widely varying estimates on the levels of cheating in people. I know that some pair-bonded bird species have been found with up to 40% of the population born due to cheating. Humans are hopefully quite a bit lower than that. However one estimate I saw several years ago from England found that about 10% of children are not the children of the person who thought he was the father. That's pretty significant. And in any case it is high enough to contribute to high school biology teachers' reluctance to demonstrate the laws of genetics using the families of their pupils as examples. And is also high enough that a common first reaction among men to seeing his child is to look for a feature that verifies it is truly his. (I've experienced this reaction first hand, and felt silly but did it anyways. I also know someone who noticed that the child wasn't his, with appropriately bad results...)
At this point you get an interesting divergence between men and women. The reasons why men would want relationships and want to cheat are basically the same, so they are attracted to the same women for both purposes. (Though his standards are usually higher for commitment than cheating.) However women are different. A woman is looking for a relationship and affairs for different reasons. Therefore they are attracted to different men for those purposes. In particular for a relationship women want men who they think will make good caregivers. But for affairs women care much more about physical attractiveness. So attractiveness does matter to women, but not in as straightforward a way as it does for men. (Remember that when discussing Brin's essay I said I would come back to the survey result that women care more about appearance for extra-marital affairs than for relationships? Well I just did...)
I could go on but I've rambled quite enough and probably offended a good number of people in the process. Hopefully people found it interesting food for thought, whether or not you agree with all of it. I know that as recently as a few years ago I would have strongly disagreed with several items I stated, and wouldn't be surprised if I disagreed with several others in a few more years.
Monday, December 28, 2009
Our most popular drug is made of sugar, fat and salt
On NPR not long ago I heard an interesting thought.
Suppose a hundred years ago that someone wanted to eat chicken. They had to go out, grab the chicken, cut its head off, pluck it, clean it, then cook it. That was just what it meant to eat chicken. Today that seems crazy. But in a hundred years that is how cooking will look.
Indeed, that is true for a lot of people. Just think how many people eat out more often than not. Or eat packaged meals. Cooking takes work, and most of us don't want to put out that effort if someone else is willing to do it for us.
On the surface that is quite reasonable. And I will confess to succumbing to the temptation quite a few times. I'm no food snob, and don't particularly enjoy cooking.
But all that notwithstanding I think that most of us should cook more. You'll be healthier. You'll eat better food. And it needn't take as much energy as you think.
Let me start with the health issues. My source here is The End of Overeating by David Kessler.
The problem that every commercial food distributer faces is getting people to buy their food. Preferably frequently and in quantity. People have a lot of reasons for buying food, but the three most important are named sugar, fat and salt. Individually we are attracted to each, however we reach a point where we saturate on any one of them. If you combine 2 of those, however, we are attracted to much higher concentrations than we are if you separate them. And combining all 3 hits a sweet spot where we want far higher concentrations than we would accept independently. Not only do we crave higher concentrations, but we experience a pleasurable rush of brain chemicals. In the pursuit of that pleasure we routinely overeat past the point of satiation. Then want to eat that way again when we recover. And this is why you're fat.
This principle is simple, effective, and well understood through the food industry. Whether you're a manufacturer designing a new kind of cheese puff or a restaurant trying to build a clientele you have to do the same thing. Deliver the magic combination of sugar, fat and salt and you can succeed. Fail to, and people will inevitably flock to the competitor who is willing to do it. The food industry has heard and obeyed. Whether it means giving us salads that are mainly a vehicle for delivering dressing (a base of oil with sugar and salt), or offering apparently healthy chickens that have been preprocessed with the injection of a syrup with sugar, fat and salt (this is why it falls apart so easily in your mouth) all of our processed food is a drug delivery mechanism.
The phrase "drug delivery mechanism" may seem overblown. It is not. The effects on our brain chemistry really do resemble an addict getting drugs. And measurements in animals of how motivated they are to get the reward show that modern processed food is only slightly less motivating than cocaine
Obviously there is some variation. But this is one of the prime reasons that obesity has expanded from being a fairly rare disorder to being about a third of the population. And today obesity is seen in ever younger people, and is more and more extreme. It would seem that the drug affects different people differently, and you can see from your waistline how much you are affected. (I'm hit about average, I weigh an extra 20 pounds or so.)
Of course this drug isn't exactly good for our bodies. It is no accident that the leading diseases tied to modern lifestyles are heart attack, stroke, diabetes and hypertension - all of which are tied to what we eat. Not that long ago I talked with someone who had a heart issue and just went to a cardiologist for the first time. The first lifestyle question he asked was, "How often do you eat out?" Note that where you eat out doesn't matter, because the entire industry has the message. They all know the game, and all are going to deliver the same drug. Ditto the manufacturers of processed food.
However if you go and cook food at home, you don't need to worry. You won't hit the sweet spot very easily because you lack the equipment to do it. You'll actually have to chew your chicken because before cooking you didn't break down the internal structure and fill it full of sugar, fat and salt.
The other interesting thing about cooking food at home is that it tastes better. Let's be clear, you won't crave it as much. If you've got one plate with fish you grilled at home, and another with potato chips, it will be the chips that you can't keep your hands off of. However the fish will be more enjoyable.
Let me give a concrete example. When I married my wife, her family was into pancakes made from mix. I've noticed two things about pancakes made from mix. The first is that they don't taste very good. The second is that they don't save you much work. Seriously you can't make the pancake until the pan is hot, right? Well it takes about as long to make the batter as to make the pan hot, and you do those at the same time. Therefore the mix didn't save you time!
Now when I tell people this, many claim that the pancakes they make from a mix taste great. And I tell them that they think that only because they've never had a decent pancake. Try that recipe, then come back and compare. (A funny story about that particular recipe. I had a co-worker who I gave that recipe to who it turned out had a friend who had been raving about my sister's pancakes for 30 years but didn't know how to make them! My friend was happy to be able to pass it along.)
All that said, when someone who doesn't cook faces cooking, it is scary. You don't know how anything works or what anything does. In reality all you need to do is get a recipe, follow directions, and it tends to work out. When you get brave you can experiment. Sometimes the experiments flop. But even that isn't a bid deal.
For instance yesterday I made turkey soup. It was going great until I added noodles. I envisioned firm noodles, and so it was at dinner because I put them in at the right time. But the noodles in the pot didn't stop cooking. So now instead of turkey soup I now have soggy turkey casserole. However it still tastes good and is still good for us. And I won't make that mistake again next year!
So stretch yourself. Look for an easy recipe and cook a meal. Even if you never come to enjoy cooking particularly much, you may find it worthwhile. And who knows, perhaps you'll turn out to be someone who enjoys cooking. Try it!
Suppose a hundred years ago that someone wanted to eat chicken. They had to go out, grab the chicken, cut its head off, pluck it, clean it, then cook it. That was just what it meant to eat chicken. Today that seems crazy. But in a hundred years that is how cooking will look.
Indeed, that is true for a lot of people. Just think how many people eat out more often than not. Or eat packaged meals. Cooking takes work, and most of us don't want to put out that effort if someone else is willing to do it for us.
On the surface that is quite reasonable. And I will confess to succumbing to the temptation quite a few times. I'm no food snob, and don't particularly enjoy cooking.
But all that notwithstanding I think that most of us should cook more. You'll be healthier. You'll eat better food. And it needn't take as much energy as you think.
Let me start with the health issues. My source here is The End of Overeating by David Kessler.
The problem that every commercial food distributer faces is getting people to buy their food. Preferably frequently and in quantity. People have a lot of reasons for buying food, but the three most important are named sugar, fat and salt. Individually we are attracted to each, however we reach a point where we saturate on any one of them. If you combine 2 of those, however, we are attracted to much higher concentrations than we are if you separate them. And combining all 3 hits a sweet spot where we want far higher concentrations than we would accept independently. Not only do we crave higher concentrations, but we experience a pleasurable rush of brain chemicals. In the pursuit of that pleasure we routinely overeat past the point of satiation. Then want to eat that way again when we recover. And this is why you're fat.
This principle is simple, effective, and well understood through the food industry. Whether you're a manufacturer designing a new kind of cheese puff or a restaurant trying to build a clientele you have to do the same thing. Deliver the magic combination of sugar, fat and salt and you can succeed. Fail to, and people will inevitably flock to the competitor who is willing to do it. The food industry has heard and obeyed. Whether it means giving us salads that are mainly a vehicle for delivering dressing (a base of oil with sugar and salt), or offering apparently healthy chickens that have been preprocessed with the injection of a syrup with sugar, fat and salt (this is why it falls apart so easily in your mouth) all of our processed food is a drug delivery mechanism.
The phrase "drug delivery mechanism" may seem overblown. It is not. The effects on our brain chemistry really do resemble an addict getting drugs. And measurements in animals of how motivated they are to get the reward show that modern processed food is only slightly less motivating than cocaine
Obviously there is some variation. But this is one of the prime reasons that obesity has expanded from being a fairly rare disorder to being about a third of the population. And today obesity is seen in ever younger people, and is more and more extreme. It would seem that the drug affects different people differently, and you can see from your waistline how much you are affected. (I'm hit about average, I weigh an extra 20 pounds or so.)
Of course this drug isn't exactly good for our bodies. It is no accident that the leading diseases tied to modern lifestyles are heart attack, stroke, diabetes and hypertension - all of which are tied to what we eat. Not that long ago I talked with someone who had a heart issue and just went to a cardiologist for the first time. The first lifestyle question he asked was, "How often do you eat out?" Note that where you eat out doesn't matter, because the entire industry has the message. They all know the game, and all are going to deliver the same drug. Ditto the manufacturers of processed food.
However if you go and cook food at home, you don't need to worry. You won't hit the sweet spot very easily because you lack the equipment to do it. You'll actually have to chew your chicken because before cooking you didn't break down the internal structure and fill it full of sugar, fat and salt.
The other interesting thing about cooking food at home is that it tastes better. Let's be clear, you won't crave it as much. If you've got one plate with fish you grilled at home, and another with potato chips, it will be the chips that you can't keep your hands off of. However the fish will be more enjoyable.
Let me give a concrete example. When I married my wife, her family was into pancakes made from mix. I've noticed two things about pancakes made from mix. The first is that they don't taste very good. The second is that they don't save you much work. Seriously you can't make the pancake until the pan is hot, right? Well it takes about as long to make the batter as to make the pan hot, and you do those at the same time. Therefore the mix didn't save you time!
Now when I tell people this, many claim that the pancakes they make from a mix taste great. And I tell them that they think that only because they've never had a decent pancake. Try that recipe, then come back and compare. (A funny story about that particular recipe. I had a co-worker who I gave that recipe to who it turned out had a friend who had been raving about my sister's pancakes for 30 years but didn't know how to make them! My friend was happy to be able to pass it along.)
All that said, when someone who doesn't cook faces cooking, it is scary. You don't know how anything works or what anything does. In reality all you need to do is get a recipe, follow directions, and it tends to work out. When you get brave you can experiment. Sometimes the experiments flop. But even that isn't a bid deal.
For instance yesterday I made turkey soup. It was going great until I added noodles. I envisioned firm noodles, and so it was at dinner because I put them in at the right time. But the noodles in the pot didn't stop cooking. So now instead of turkey soup I now have soggy turkey casserole. However it still tastes good and is still good for us. And I won't make that mistake again next year!
So stretch yourself. Look for an easy recipe and cook a meal. Even if you never come to enjoy cooking particularly much, you may find it worthwhile. And who knows, perhaps you'll turn out to be someone who enjoys cooking. Try it!
Friday, December 25, 2009
Work optimally, not long
It has been a long time since I entered anything here. But I'm in the mood. I've cooked, eaten, and even done cleanup after my turkey dinner. If you're uninterested in hearing about cooking turkey, skip the next paragraph.
This time I tried dry brining the turkey. That was surprisingly easy. A few days ago I filled a small bowl with 1 tbsp of salt per 5 pounds of turkey (that's 6.5 ml salt per kg), added some spices for taste (I did sage and a ground up bay leaf), mix, then spread that over my freshly bought turkey, put a plastic bag over it, then let it thaw in the fridge. Today I just roasted it normally. Based on past experience I checked regularly to baste the turkey, but it really wasn't necessary since the juice stayed in the turkey. Tomorrow I'll get supplies, then make turkey soup, which for me is the best part of having turkey dinner.
Enough about food. What I want to write about today is something that we all know in theory, but far too few of us act on. That is that it is better to work at an optimal, sustainable level than it is to work long hours. Certainly this is true in software, and likely in many other fields as well.
More precisely I would give the following rule. If you see a developer working more than 40-45 hours per week, ask yourself two questions:
If the answer to either question is "no", then I strongly recommend looking at the schedule and figuring out what deadlines to slip immediately to get people back to reasonable schedules. Because you're going to compromise something, and choosing up front is much better than making it by burning people out and getting sub par work.
The reason for this rule is that people have a maximum sustained throughput for creative work when they are working somewhere between 35 and 45 hours/week. (For a reference on that see Rapid Development by Steve McConnell.) You can temporarily raise your hours above that and get improved productivity. But after a few weeks it catches up to you, your productivity drops, and you'll get less done despite working harder. And once you're in that state you've got a vicious cycle. Your lack of productivity pushes you to work harder which makes you less productive until something gives. Quite possibly your sanity.
The one thing that seems odd about this to most people is that people don't realize that their brains aren't working. But the reason for this is simple. When you're impaired, the first thing that gets impaired is your ability to judge how impaired you are. Thus your self-monitoring for impairment is not trustworthy. This effect is seen with many kinds of impairments, including chronic sleep deprivation, alcohol, strokes, and so on.
The army once did a study that showed this in a striking way. They took soldiers and put them on different sleep regimens. Among other things they found that most soldiers could operate indefinitely on 6 hours of sleep per night and would claim to be fine. But their wives said they weren't fine. Comparing results on ability tests before and after the new schedule, the wives were right and the soldiers were wrong. Which shows how easy it is to think you're still functioning when you really aren't.
Now that's the theory. What is the practice? The practice is that most of us don't do this. The reason is that most organizations manage by schedule, putting pressure on us to work harder. We put the effort out, we try to make the deadline, and then we fail to be aware of our declining output. Possibly we make the deadline, but quality suffers, bug counts suffer, and development+maintenance comes out to a lot more than it would have if we had focused on development alone.
What keeps us from doing the smart thing? For many of us, fear. While intellectually we know that we should be able to do more if we keep to a peak level, as a practical matter we're scared of being called slackers for not visibly putting out effort. Particularly if everyone else is working overtime. This fear is particularly reasonable since management usually has no idea how much developers do or why things take us so long. And is doubly so if our direct managers are themselves working too hard and therefore not themselves in the best shape.
Another major thing that keeps us from doing the right thing is that nobody likes to deliver bad news. Nobody wants to tell their manager that the schedule will slip. So we put off delivering the bad news to anyone, and try to keep it from happening. Of course this doesn't keep the schedule from slipping. But there is always the hope that someone else will be forced to admit that the schedule is going to slip. And failing that some people will deliver an incomplete and non-working piece, declare it done, then push the admission of failure off even farther.
Delivering bad news gets harder still because everyone else has also avoided doing it. Everyone delivers their bad news as gently as they can, and the manager inevitably wants to hear it as better than it is. Therefore upper management is usually seriously out of touch. The result is How Shit Happens, which is only funny because it is true.
As an example of how extreme this effect is, The Mythical Man-Month points out that in the rare cases of early projects, the good news starts coming immediately. If the project is late, there is no sign that the project will slip until about 2 weeks before it is due. And this is true no matter how long the project is actually delayed. And the driving force behind this is that nobody wants to deliver bad news, and nobody wants to hear it.
So we have a real organizational problem. How should you solve it?
The easiest problem is to ignore the problem and try to survive. This is suboptimal in virtually every way. However it is what most of us do.
A better option is to try to find an organization without the problem. They do exist. I know because I've been lucky enough to work for some. (Something that I suspect is a key ingredient is working in short iterations. Reality and management dreams tend not to get too far out of sync on a 1 week project.) But those can be hard to find. And in the current economy most people have a hard enough time just finding a job that they can't worry about the quality of the employer.
The final option is to try to make your current organization into one that works better. If not better for everyone, then at least better for you. This is the most difficult path, and there is no simple recipe for how to solve the necessary interpersonal issues that you'll encounter. However I can give you some useful ingredients.
The first is education. Go and get Rapid Development, read it, and learn what it says. It is one thing to confront your manager with the message that you need saner work hours. It is quite another to share research about how to get more done and say that you really want to do it. This works even better if you've identified multiple things you'd like to change at once, so it doesn't come off as your cherry-picking the item that you most wanted to try.
The second ingredient is knowing that delivering negative messages often is not as bad as we think it will be. Admittedly it doesn't always go well. However any manager who has been around the block a couple of times knows that software projects tend to fail, so it won't be as much of a surprise as you might think. Of course your manager is usually under pressure from above to meet deadlines and so is likely to not want to face that bad news. But it won't be a surprise.
The third ingredient is being conscious of the fact that delivering bad news early, no matter how bad it may be immediately, usually works out better in the long run. Yes, it is unpleasant and we all want to procrastinate on unpleasant things. But delivering bad news up front then better news later makes people happier in the long run than letting people think they'll get good news and disappointing them with bad news. (Incidentally if you're going to deliver bad news, take a tip from Machiavelli's The Prince and deliver it all at once, then deliver good news afterwards.)
The fourth ingredient is to understand the person you are talking to, and deliver your message in a way that is likely to be heard. A lot goes into this, but a big factor that lots of people miss is cognitive dissonance. Let me wrap up this topic, then I'll go on to explain it shortly.
The last ingredient is being aware of what the status quo is likely to cost you. If you allow yourself to be aware of that, you'll have lots of motivation to avoid that fate. And in this case motivation is a good thing.
Now let me return to cognitive dissonance. I won't explain it at length. I've done that elsewhere and likely will do it again, but here are the basics.
The principle of cognitive dissonance says that people will go to any length to maintain their positive self-image of themselves. This means that messages which threaten your positive self-image trigger defensive reactions. The more powerfully the message is delivered, the worse the defensive reaction is likely to be. (When you see such a reaction the last thing you want to do is follow up with facts proving your position. That just increases the conflict and therefore increases the defensive reaction.) What this means for any given person will vary from person to person. However if you remain aware of this phenomenon, you can often figure out where the hidden landmines are, and have hope of communicating what would otherwise be impossible.
For example, suppose that your manager thinks he or she is a pretty good manager. (Most of us tend to think we're pretty good at our jobs. No matter how good or bad we may be.) Then at all costs you must avoid threatening that self-image. Therefore you would need to avoid saying anything that even looks like you're delivering the message that there is room to be a better manager. This is true, even if said manager regularly points out things that made them better managers. To avoid the possibility of trouble, it is best to make the case that what you are saying flows from what that manager says about management.
This makes it really bad to say, I was reading this excellent book by Steve McConnell and really liked some of what his advice about managing software teams... Doing that is explaining to your manager how to manage, and anything you say after that which your manager doesn't already know and accept is going to be really difficult to accept.
It is better to say, I've been reading a good book, and it has taught me a lot about myself. It is making me aware that I am not working as effectively as I could... Making the message be about you is a reasonably safe way of side stepping the question of whether this is something the manager should know to tell you.
But it may be possible to do better still. For example consider, You know how you always tell us that we should constantly been trying to improve ourselves? Well I've gotten some ideas from a software classic, and here are some things that I'd like to try... If this is true, then you've put cognitive dissonance in your corner. Your manager will now find that a "Yes" is an opportunity for confirmation of his or her good self-opinion. Better yet, your manager will then be inclined to interpret the outcome as success, giving an even stronger confirmation of your manager's ability as a manager.
Now all that said, there is no guarantee that you can keep a sane work schedule when the going gets tough. However if you can do it, the outcome will be better for everyone. That doesn't mean the outcome will be good, but it is better and that is really worth it.
This time I tried dry brining the turkey. That was surprisingly easy. A few days ago I filled a small bowl with 1 tbsp of salt per 5 pounds of turkey (that's 6.5 ml salt per kg), added some spices for taste (I did sage and a ground up bay leaf), mix, then spread that over my freshly bought turkey, put a plastic bag over it, then let it thaw in the fridge. Today I just roasted it normally. Based on past experience I checked regularly to baste the turkey, but it really wasn't necessary since the juice stayed in the turkey. Tomorrow I'll get supplies, then make turkey soup, which for me is the best part of having turkey dinner.
Enough about food. What I want to write about today is something that we all know in theory, but far too few of us act on. That is that it is better to work at an optimal, sustainable level than it is to work long hours. Certainly this is true in software, and likely in many other fields as well.
More precisely I would give the following rule. If you see a developer working more than 40-45 hours per week, ask yourself two questions:
- Will this effort be sustained less than 3 weeks?
- When it is over, will this person get corresponding downtime?
If the answer to either question is "no", then I strongly recommend looking at the schedule and figuring out what deadlines to slip immediately to get people back to reasonable schedules. Because you're going to compromise something, and choosing up front is much better than making it by burning people out and getting sub par work.
The reason for this rule is that people have a maximum sustained throughput for creative work when they are working somewhere between 35 and 45 hours/week. (For a reference on that see Rapid Development by Steve McConnell.) You can temporarily raise your hours above that and get improved productivity. But after a few weeks it catches up to you, your productivity drops, and you'll get less done despite working harder. And once you're in that state you've got a vicious cycle. Your lack of productivity pushes you to work harder which makes you less productive until something gives. Quite possibly your sanity.
The one thing that seems odd about this to most people is that people don't realize that their brains aren't working. But the reason for this is simple. When you're impaired, the first thing that gets impaired is your ability to judge how impaired you are. Thus your self-monitoring for impairment is not trustworthy. This effect is seen with many kinds of impairments, including chronic sleep deprivation, alcohol, strokes, and so on.
The army once did a study that showed this in a striking way. They took soldiers and put them on different sleep regimens. Among other things they found that most soldiers could operate indefinitely on 6 hours of sleep per night and would claim to be fine. But their wives said they weren't fine. Comparing results on ability tests before and after the new schedule, the wives were right and the soldiers were wrong. Which shows how easy it is to think you're still functioning when you really aren't.
Now that's the theory. What is the practice? The practice is that most of us don't do this. The reason is that most organizations manage by schedule, putting pressure on us to work harder. We put the effort out, we try to make the deadline, and then we fail to be aware of our declining output. Possibly we make the deadline, but quality suffers, bug counts suffer, and development+maintenance comes out to a lot more than it would have if we had focused on development alone.
What keeps us from doing the smart thing? For many of us, fear. While intellectually we know that we should be able to do more if we keep to a peak level, as a practical matter we're scared of being called slackers for not visibly putting out effort. Particularly if everyone else is working overtime. This fear is particularly reasonable since management usually has no idea how much developers do or why things take us so long. And is doubly so if our direct managers are themselves working too hard and therefore not themselves in the best shape.
Another major thing that keeps us from doing the right thing is that nobody likes to deliver bad news. Nobody wants to tell their manager that the schedule will slip. So we put off delivering the bad news to anyone, and try to keep it from happening. Of course this doesn't keep the schedule from slipping. But there is always the hope that someone else will be forced to admit that the schedule is going to slip. And failing that some people will deliver an incomplete and non-working piece, declare it done, then push the admission of failure off even farther.
Delivering bad news gets harder still because everyone else has also avoided doing it. Everyone delivers their bad news as gently as they can, and the manager inevitably wants to hear it as better than it is. Therefore upper management is usually seriously out of touch. The result is How Shit Happens, which is only funny because it is true.
As an example of how extreme this effect is, The Mythical Man-Month points out that in the rare cases of early projects, the good news starts coming immediately. If the project is late, there is no sign that the project will slip until about 2 weeks before it is due. And this is true no matter how long the project is actually delayed. And the driving force behind this is that nobody wants to deliver bad news, and nobody wants to hear it.
So we have a real organizational problem. How should you solve it?
The easiest problem is to ignore the problem and try to survive. This is suboptimal in virtually every way. However it is what most of us do.
A better option is to try to find an organization without the problem. They do exist. I know because I've been lucky enough to work for some. (Something that I suspect is a key ingredient is working in short iterations. Reality and management dreams tend not to get too far out of sync on a 1 week project.) But those can be hard to find. And in the current economy most people have a hard enough time just finding a job that they can't worry about the quality of the employer.
The final option is to try to make your current organization into one that works better. If not better for everyone, then at least better for you. This is the most difficult path, and there is no simple recipe for how to solve the necessary interpersonal issues that you'll encounter. However I can give you some useful ingredients.
The first is education. Go and get Rapid Development, read it, and learn what it says. It is one thing to confront your manager with the message that you need saner work hours. It is quite another to share research about how to get more done and say that you really want to do it. This works even better if you've identified multiple things you'd like to change at once, so it doesn't come off as your cherry-picking the item that you most wanted to try.
The second ingredient is knowing that delivering negative messages often is not as bad as we think it will be. Admittedly it doesn't always go well. However any manager who has been around the block a couple of times knows that software projects tend to fail, so it won't be as much of a surprise as you might think. Of course your manager is usually under pressure from above to meet deadlines and so is likely to not want to face that bad news. But it won't be a surprise.
The third ingredient is being conscious of the fact that delivering bad news early, no matter how bad it may be immediately, usually works out better in the long run. Yes, it is unpleasant and we all want to procrastinate on unpleasant things. But delivering bad news up front then better news later makes people happier in the long run than letting people think they'll get good news and disappointing them with bad news. (Incidentally if you're going to deliver bad news, take a tip from Machiavelli's The Prince and deliver it all at once, then deliver good news afterwards.)
The fourth ingredient is to understand the person you are talking to, and deliver your message in a way that is likely to be heard. A lot goes into this, but a big factor that lots of people miss is cognitive dissonance. Let me wrap up this topic, then I'll go on to explain it shortly.
The last ingredient is being aware of what the status quo is likely to cost you. If you allow yourself to be aware of that, you'll have lots of motivation to avoid that fate. And in this case motivation is a good thing.
Now let me return to cognitive dissonance. I won't explain it at length. I've done that elsewhere and likely will do it again, but here are the basics.
The principle of cognitive dissonance says that people will go to any length to maintain their positive self-image of themselves. This means that messages which threaten your positive self-image trigger defensive reactions. The more powerfully the message is delivered, the worse the defensive reaction is likely to be. (When you see such a reaction the last thing you want to do is follow up with facts proving your position. That just increases the conflict and therefore increases the defensive reaction.) What this means for any given person will vary from person to person. However if you remain aware of this phenomenon, you can often figure out where the hidden landmines are, and have hope of communicating what would otherwise be impossible.
For example, suppose that your manager thinks he or she is a pretty good manager. (Most of us tend to think we're pretty good at our jobs. No matter how good or bad we may be.) Then at all costs you must avoid threatening that self-image. Therefore you would need to avoid saying anything that even looks like you're delivering the message that there is room to be a better manager. This is true, even if said manager regularly points out things that made them better managers. To avoid the possibility of trouble, it is best to make the case that what you are saying flows from what that manager says about management.
This makes it really bad to say, I was reading this excellent book by Steve McConnell and really liked some of what his advice about managing software teams... Doing that is explaining to your manager how to manage, and anything you say after that which your manager doesn't already know and accept is going to be really difficult to accept.
It is better to say, I've been reading a good book, and it has taught me a lot about myself. It is making me aware that I am not working as effectively as I could... Making the message be about you is a reasonably safe way of side stepping the question of whether this is something the manager should know to tell you.
But it may be possible to do better still. For example consider, You know how you always tell us that we should constantly been trying to improve ourselves? Well I've gotten some ideas from a software classic, and here are some things that I'd like to try... If this is true, then you've put cognitive dissonance in your corner. Your manager will now find that a "Yes" is an opportunity for confirmation of his or her good self-opinion. Better yet, your manager will then be inclined to interpret the outcome as success, giving an even stronger confirmation of your manager's ability as a manager.
Now all that said, there is no guarantee that you can keep a sane work schedule when the going gets tough. However if you can do it, the outcome will be better for everyone. That doesn't mean the outcome will be good, but it is better and that is really worth it.
Tuesday, December 15, 2009
The use and limitations of exceptions
I'd have a longer blog but I just had a farewell party and so am do not have the most time to spend. So I'll just point to an email wrote to golang about what I see as the cases where exceptions prove useful, what the limitations of exceptions are, and some of the reasons why Go doesn't include them.
Friday, December 11, 2009
Design of a reporting system
I've had to build a lot of reports, and in my last job I build a reporting system that I think worked quite well. This post explains some of the key ideas behind that system.
One of my best ideas was to use Excel, not compete with it. Users will come with detailed requests for the exact reports they want. You can try to provide what users ask for. Inevitably they will change their mind, or the report lovingly described by someone's manager won't be what that person wants, or a million variations on this theme. This will generate an endless stream of requests as people try to get you to tweak what is being provided into their exact desire. Alternately you can make the data from your system directly accessible in Excel, and encourage people create the exact reports they want themselves in an interface they are comfortable with. Now all that you need to do is provide raw data, appropriately aggregated, and you're no longer a bottleneck. Giving people what they want rather than what the initially ask for makes for happier users.
Luckily it is easy to integrate with Excel. What you need is a way for users to design a web-based report which provides the raw data they want, and make it available with a short URL (less than 50 characters). Now a user can open up a new Excel workbook, then choose Data, Import External Data, then New Web Query. This pops open a browser. The user puts in their short URL, and the browser loads it. The user can then click on the HTML table that they want, save the web query, and Excel will run it again then fill a spreadsheet with the data in that table. Now a complex set of spreadsheets can be built off of that data, and every time they refresh it will re-run the report and get fresh data.
That's the user experience. Behind the scenes the developer needs to do several things to make this work.
Moving on, my philosophy about reports is that 1 report that can be tweaked in 100 ways is more flexible and powerful than 10 reports that can be tweaked in 10 ways each. Similarly when given a one-off reporting request, it is better to find a way to add something to an existing report to make it capable of handling that request than to do the one-off request. That is because if someone wants that feature once, someone else is likely to want it again. And there is something really satisfying about receiving the second request and being able to say, "I've already built that, here let me show you..."
Of course the challenge is converting that philosophy into action. How exactly do you go about building a flexible report that can be tweaked in many ways?
My strategy was to look at each report as an array of SQL statements. All statements before the last would create temporary tables. The last query would return the data set that I would display. Thus if I needed to pull in extra information - for instance I needed to segment data by whether a given promotion was run - when building the array I could dynamically insert a query to fetch that particular piece of information, and then pass it through all of the tables.
This strategy naturally leads to two useful features. The first is that you can add the exact SQL statements used to generate the report in an HTML comment. This is very useful for auditing purposes. The second is that you can add the feature of having a dropdown box listing the temporary tables that will be created, and allow the report to run to that point then display those intermediate results. This is helpful when debugging. And not just for the reporting engineer - it was very useful for me when the accounting department was able to come to me and say, "This report, at this step, is missing these invoices."
This strategy, of course, requires being able to create definitions for temporary tables on the fly. Most databases provide that. Whether you are in MySQL, PostgreSQL, MS SQL, Sybase, etc you have exactly the right kind of temporary table available. The glaring exception is Oracle. There are a couple of workarounds available with Oracle. I've tried automatically building queries with large subqueries. It kind of works, however the result is unreadable by end users, you can't easily display intermediate results, and I miss the ability to control execution plans by creating temporary tables and adding appropriate indexes on the fly before moving on.
The other requirement of this strategy is that your SQL statements can be customized by the addition of an arbitrary list of extra fields, join conditions, etc. I wrote more reports than I care to admit before I realized that the best way to do this is to use a templating system to build your SQL. Now that I have used templates to dynamically generate SQL statements, I wouldn't want to go back. I used Perl's Template Toolkit for that purpose. Again I tweaked it substantially for my purposes. But many different templating systems could work well.
Moving on, let's look at how I organized things under the hood.
Obviously I needed complex forms with lots of checkboxes, input boxes, dropdowns, and the like. I chose to make each form control into an object that knew how to find its data out of the form parameters, do validation, write out its HTML, and pass information on itself to the templates. Some of the objects had SQL statements attached so that I didn't have to duplicate logic between reports.
There are endless arguments in software circles about separating presentation from content. In some circles mixing content and presentation as these objects did is anathema. In general web development I would not be inclined to move the HTML for form elements into what is basically a model object. However in this case it worked out very well. The form is so directly tied to the action of the report that it wouldn't make sense to have different people work on the two. These reports were for internal use, and so functionality mattered more than aesthetics. And as a practical matter this choice made the addition of options to forms extremely easy.
And, of course, I needed to have a way to coordinate everything. I did that with a report object. The report class had methods to create each kind of form object. It had a method to be displayed. When displayed it would show all of the form elements, check whether the report should be generated, if so it would run all of the queries, and display the report, and (of course), it would append necessary documentation.
With all of these pieces the code for a trivial report could look something like this:
Now this report isn't very impressive. Also passing the age into the SQL as a template variable is poor style. (I did that just to show what the templating looked like.) But if you changed this to having a dozen form controls tweaking a data processing step with a half-dozen queries then you could have a very powerful and flexible report. Add in a few more reports, the ability to combine independent options, and the ability for users to build on top of this in Excel, and you wind up with a surprisingly powerful and flexible reporting system.
One of my best ideas was to use Excel, not compete with it. Users will come with detailed requests for the exact reports they want. You can try to provide what users ask for. Inevitably they will change their mind, or the report lovingly described by someone's manager won't be what that person wants, or a million variations on this theme. This will generate an endless stream of requests as people try to get you to tweak what is being provided into their exact desire. Alternately you can make the data from your system directly accessible in Excel, and encourage people create the exact reports they want themselves in an interface they are comfortable with. Now all that you need to do is provide raw data, appropriately aggregated, and you're no longer a bottleneck. Giving people what they want rather than what the initially ask for makes for happier users.
Luckily it is easy to integrate with Excel. What you need is a way for users to design a web-based report which provides the raw data they want, and make it available with a short URL (less than 50 characters). Now a user can open up a new Excel workbook, then choose Data, Import External Data, then New Web Query. This pops open a browser. The user puts in their short URL, and the browser loads it. The user can then click on the HTML table that they want, save the web query, and Excel will run it again then fill a spreadsheet with the data in that table. Now a complex set of spreadsheets can be built off of that data, and every time they refresh it will re-run the report and get fresh data.
That's the user experience. Behind the scenes the developer needs to do several things to make this work.
- Provide users a way to save the parameters of a given report, and create a named report. The raw parameters will tend to be too long to be saved properly. When users access this named report you must not issue a redirect. If you do a redirect it will work initially, but Excel internally remembers the redirected location. Which means that if the user tweaks the saved report, the Excel spreadsheet won't notice the tweak.
- In your HTML give an unambiguous ID attribute to the table with the data. Excel will use this ID to locate the data. Otherwise it will try to analyze the structure of the HTML, and that will break easily when the report changes.
- Have a way to easily specify relative dates. People frequently want reports that run to the present, run for last month, and so on. If they only had fixed dropdowns when they set up the report then the Excel spreadsheets they get won't update to what they want it to be. I solved this problem by providing an optional text field that I passed to Perl's Date::Manip to parse. (I actually improved the date parsing slightly for my purposes, but that was the base.)
Moving on, my philosophy about reports is that 1 report that can be tweaked in 100 ways is more flexible and powerful than 10 reports that can be tweaked in 10 ways each. Similarly when given a one-off reporting request, it is better to find a way to add something to an existing report to make it capable of handling that request than to do the one-off request. That is because if someone wants that feature once, someone else is likely to want it again. And there is something really satisfying about receiving the second request and being able to say, "I've already built that, here let me show you..."
Of course the challenge is converting that philosophy into action. How exactly do you go about building a flexible report that can be tweaked in many ways?
My strategy was to look at each report as an array of SQL statements. All statements before the last would create temporary tables. The last query would return the data set that I would display. Thus if I needed to pull in extra information - for instance I needed to segment data by whether a given promotion was run - when building the array I could dynamically insert a query to fetch that particular piece of information, and then pass it through all of the tables.
This strategy naturally leads to two useful features. The first is that you can add the exact SQL statements used to generate the report in an HTML comment. This is very useful for auditing purposes. The second is that you can add the feature of having a dropdown box listing the temporary tables that will be created, and allow the report to run to that point then display those intermediate results. This is helpful when debugging. And not just for the reporting engineer - it was very useful for me when the accounting department was able to come to me and say, "This report, at this step, is missing these invoices."
This strategy, of course, requires being able to create definitions for temporary tables on the fly. Most databases provide that. Whether you are in MySQL, PostgreSQL, MS SQL, Sybase, etc you have exactly the right kind of temporary table available. The glaring exception is Oracle. There are a couple of workarounds available with Oracle. I've tried automatically building queries with large subqueries. It kind of works, however the result is unreadable by end users, you can't easily display intermediate results, and I miss the ability to control execution plans by creating temporary tables and adding appropriate indexes on the fly before moving on.
The other requirement of this strategy is that your SQL statements can be customized by the addition of an arbitrary list of extra fields, join conditions, etc. I wrote more reports than I care to admit before I realized that the best way to do this is to use a templating system to build your SQL. Now that I have used templates to dynamically generate SQL statements, I wouldn't want to go back. I used Perl's Template Toolkit for that purpose. Again I tweaked it substantially for my purposes. But many different templating systems could work well.
Moving on, let's look at how I organized things under the hood.
Obviously I needed complex forms with lots of checkboxes, input boxes, dropdowns, and the like. I chose to make each form control into an object that knew how to find its data out of the form parameters, do validation, write out its HTML, and pass information on itself to the templates. Some of the objects had SQL statements attached so that I didn't have to duplicate logic between reports.
There are endless arguments in software circles about separating presentation from content. In some circles mixing content and presentation as these objects did is anathema. In general web development I would not be inclined to move the HTML for form elements into what is basically a model object. However in this case it worked out very well. The form is so directly tied to the action of the report that it wouldn't make sense to have different people work on the two. These reports were for internal use, and so functionality mattered more than aesthetics. And as a practical matter this choice made the addition of options to forms extremely easy.
And, of course, I needed to have a way to coordinate everything. I did that with a report object. The report class had methods to create each kind of form object. It had a method to be displayed. When displayed it would show all of the form elements, check whether the report should be generated, if so it would run all of the queries, and display the report, and (of course), it would append necessary documentation.
With all of these pieces the code for a trivial report could look something like this:
use Report;
my $report = Report->new();
$report->add_textbox(
name => "user",
label => "Who is this for?",
);
$report->add_textbox_number(
name => "age",
label => "How old are you?",
);
my @queries;
push @queries, {
temp_table => "results",
sql => qq{SELECT :user as "User", [% age %] as "Age", current_date as "Today"},
args => {
user => $report->user,
},
};
$report->display(
title => "My Report",
queries => \@queries,
doc => qq{
<ul>
<li> <i>User:</i> The user this report was generated for.</li>
<li> <i>Age:</i> Their reported age.</li>
<li> <i>Today:&t;/i> The current date.</li>
</ul>
},
);
Now this report isn't very impressive. Also passing the age into the SQL as a template variable is poor style. (I did that just to show what the templating looked like.) But if you changed this to having a dozen form controls tweaking a data processing step with a half-dozen queries then you could have a very powerful and flexible report. Add in a few more reports, the ability to combine independent options, and the ability for users to build on top of this in Excel, and you wind up with a surprisingly powerful and flexible reporting system.
Subscribe to:
Posts (Atom)