Of course most of the people that start talking to you in the streets do so because they want money or your phone number. But it's worth the effort bypassing that first barrier every now and then. Once you have either bought that bracelet/bag of oranges/bottle of baobab honey/mentioned your boyfriend (or, even better, husband), or else made clear that you are not interested, I find most malagasies to be open for a chat. If you even dig out a couple words of malagasy (and it's always the same 5 words that you will need, so there is no excuse for not remembering them after a few weeks), chances are your counterpart's perception of you will change from somebody who merely has "money" written all over his white body, to an actual person.
While there are days when I am tired of all the whistles and "Bonjour madame"s, I am very aware that they are due to the fact that I really am unimaginably rich as compared to 90% of the people here. While I would not want to swich with, for example, a pousse-pousse (rickshaw) puller for even a day, not trying to shake them off like irritating inscets allows me to build tiny bridges over this huge gap between our worlds, at least on one or the other occasion.