I do it all the time (especially in class), and, believe it or not, it actually helps me concentrate on what the person is saying (if I look them in the eye, I get distracted by their face ; if I wait for their pauses to be over, I lose track of what they were saying ; to be drawing constantly somehow helps me keep my brain dynamic). But I understand it annoys some people (namely my dad), so I shall refrain from doing it in those occasions, even though it sometimes makes for some superb frescoes :).