you'll notice the cold, but it won't hinder your riding.
my advice is to wear some gloves, because cold hands wear out long before cold limbs. also, a wearing a helmet prevents that ice cream headache feeling after you ride in cold weather.
I'll agree with Jon. Wear gloves. It may look lame, but you will feel so much cooler than everyone else when you can actually ride for more than 3 minutes.