There is no movement on the contact surface with the belt. When the pulley was stamped, the rounded edge on the front wasn't perfectly perpendicular with the pulley belt surface.
If there was movement between the belt and the pully, then you would have heat. The bearing in the idler will get warm from engine heat and from internal friction in the bearing. If it fails under warranty, the dealer will replace it.
I would worry more about the unnecessary clutch that Toyota saw fit to put on the alternator pulley going south and leaving you stranded.