Average time to switch modes ranges in seconds, which is enough of a discontinuity to be noticed.
On the other hand, drawing with a mouse requires fine precision movement. Unfortunately, a mouse is not designed to be used with fine precision. Consider that a human hand has three modes of operation: precision, power, and paw. Precision primarily uses the index finger and thumb; power uses the forearm; and paw just, well, paws at things. Using a screwdriver as an example, precision movement is used to operate a jeweler's screwdriver, to carefully pry gears from a watch casing. Power is used to drive a screw into the wall. Pawing is used to grab the screwdriver off the workbench.
Unfortunately, the mouse operates in paw mode. Even worse is that it doesn't use the most dextrous nor powerful parts of the hand-arm to operate (fingers, forearm respectively), but instead relies on the relatively weak lateral motions of the wrist and rotations of the elbow and shoulder.
That being said, most people take several months to learn how to use the mouse precisely (and this is what the game Minesweeper is meant to teach). Even then, with the constant switching of modes, precise use of the mouse is unlikely. Therefore, the user interface should be forgiving with respect to precision. For example, if a user wants to select a line in a paint tool, she should still grab the line even if she "missed" by a few pixels. The "snap to" feature in many graphics apps does this.
Another consider with the mouse is repetitive stress injury. A right-handed individual (80-90% of the population) can easily rotate their wrist only down and to the left. A left-handed individual can only rotate their wrist down and to the right. If a particular action, say dragging objects to "the trash", must be repeated often, it should be activated from a HotZone down and to the left. (It may be preferable to flip orientation for left-handed users.)
This is also why horizontally-oriented toolbars, ala Windows, are less usable than vertically-oriented toolbars on the left, ala early Macintosh programs. A right-handed individual can easily rotate his wrist to the left, in a precise way (by using his fingers to dextrously position the cursor), to select an object. It's harder to select something vertically, with only the thumb and wrist to guide the mouse.
That being said, the five "magic" pixels that can be accessed quickly without fail are (in decreasing order of ease):
To avoid DominantHandBias, Microsoft has come to name the left mouse button the action button, and the right button the context button. Macintosh still calls their only mouse button, "the mouse." Of course, Macintosh users go through QuasiMode? acrobatics to use the mouse.
I hate to disagree, but those are typically the farthest pixels (or at least 3 of them are) from my pointer. And since I use a trackball, getting to them can be very slow (as many as 4 separate movements after I've positioned my hand on the trackball). On The Other Hand, anything within 5-600 pixels is quick and easy. Put an important UI element in a corner, and I will curse you for all eternity.
Actually, the problem comes with a large screen, and a trackball w/high resolution - you move the ball relatively large amounts to traverse the screen. I can hit the ball, have it spin and stop 4 times while crossing a 1600x1200 desktop diagonally. No problem once I hit the corner (mine stays in the corner, don't know where you've seen the crawl problem).
My point is merely that assuming all users use the same input devices is bad. Using techniques which are only beneficial to a single input device is bad. Taking things to extremes frequently causes more trouble than it's worth when your base assumptions turn out to be false (how about the X-Ray machine that lighted up the "on" and "off" buttons - so that "off" was lighted while it was turned on. German designers, U.S. users. Thank god for the safety interlocks on the door... and the best part was that doing it the other way would have caused problems in Germany.)