How not to remove a pulley?

IMG_1559.JPGIMG_1564.JPGIMG_1566.JPGIMG_1567.JPG

So I purchased the Harbor Freight “3 Jaw Puller Set” (32184) today for $12 and proceeded to put together all the pullers.  I installed the correct puller then broke the pulley.

IMG_1569.JPGIMG_1512b.JPG

I guess I needed to remove the pen thing first (2nd picture with arrow)?  Not really sure how to remove it though.  I already removed the screw but not quite sure how to slide the piece out?