The grow with me cardigans are available in a couple colors all made from waffle fabric. Waffle is very soft and has a stretch to it.
Since they are grow with me if you get it on the smaller size you can fold the sleeve cuffs to fit better. Then unfold as your child grows.