When I ride a pannier-laden commuter bike, it is not uncommon for one side to be bulging while the other sits nearly empty. This is not because I can't be bothered to distribute the weight evenly, but because one of the panniers houses my enormous photo/laptop bag and I don't always have anything to put on the other side to compensate. I've cycled with this type of uneven rear load pretty much the entire time I've owned bicycles with rear racks. In the past, I've usually had a briefcase-type pannier clipped to one side of the rack, with nothing on the other, which is really no different from having unevenly loaded double panniers. But it's when I switched to the latter system that observers really began to notice. Over the past month in particular, I've received quite a few questions and concerned comments about the issue! These tend to fall into one of two categories: (1) Does the weight not pull to the side and cause handling issues? and (2) Isn't the uneven load bad for the bicycle frame?
As far as an uneven load being bad for the bike… I imagine that would depend on lots of factors, including what sort of weight the bicycle was built to accommodate in the rear, and how chronically, as well as for what distance and duration, the owner rides it unevenly loaded (assuming the weight is always on the same side). I have seen a few twisted vintage frames, where it had been suggested they got that way through chronic uneven load carry. But without knowing the owner, it is impossible to tell for sure. My own commuter bikes have all been fairly stiff laterally, and I've never feel any twist in the frames as a result of uneven weight at the rear. But these things are not always perceivable, and it hasn't been long enough to gauge long term effects. Just in case, I try to alternate the side of the load and never make it ridiculously heavy.
In general, I would say that the average utility bike can certainly handle uneven rear load distribution, and as long as you feel comfortable with the bicycle's handling there is no need to worry. As always, YMMV, and I welcome others to share their experiences.