The amount of caffeine in a standard cup of coffee is about 100 milligrams. It has the density of a bit more then water, so this translates to a volume of a tenth of a cubic centimeter (one cubic centimeter is about the volume of your pinky finger covered by fingernail). This is a macroscopic amount. You could easily see it. So caffeine isn't something which an invisible amount will effect a person.
Even if you were giving the baby a drop of coffee per morning, it is such a small amount, about 0.3 milligrams (one drop, about one milliliter out of a typical 300 milliliter cup) that it is unlikely to have any effect.
I guess the caffeine could percipitate out of solution and form a film which your brush could transfer. I think it unlikely.