Should I use food based ingredients on my baby?

Please don't. 

Thats the short answer. 

Why?

Babies skin is thinner and has more openings (more permeable) than adult skin, which means it absorbs more of anything applied on the skin. Using food derivatives on baby skin can trigger an allergic response the next time it is given orally. For example - almond oil. If we apply almond oil on the skin, the immune system may not like it. The next time the baby is given almonds by the mouth (eats it) the immune system sees the almond in a different form and attacks it. This means the body attacks the almond thinking its an enemy causing the severe symptoms of food allergies.

This is why moisturizing skin with the RIGHT ingredients is so important.