Another option might be to look around for some kind of small bag to put on the bike...on those middle bars between your legs, for example, or even on the handlebars themselves. Ideally it wouldn't have a zipper but would have some sort of shrouded opening where you could get into and out of quickly.
If you anticipate a problem, this enables you to make a much more natural movement than reaching under your shirt. It would also be much closer if located somewhere on the handlebars.
This probably wouldn't be ideal if you make any stops before returning home though.