It's ugly.
Ask yourself: Have you ever been turned on by anyone flipping the bird? Unlikely. That's because raising your middle finger also has disastrous consequences for your face -- scrunchy, scowly, middle-finger-specific consequences that are in no way attractive. And let's be honest: even if you're telling a real jerk to fuck off, you need all the help you can get in the looks department.
You look like a child.
It's hard to argue with the immaturity of the middle finger. After M.I.A. flipped off all of Footballwatchingdom, Madonna called the gesture a "teenager" move. Y'know what else teenagers are known for? Acne and body odor and Justin Bieber fandom. If you really want to piss a person off, just blast that "Baby" song at him until it's stuck in his head.