I think so too. The label actually makes things worse, since one doesn't know if it indicates current state, or potential state as a result of "flipping" the switch. Every time I use one of those things I have to stop and think about it. The problem is that the label in the wrong place: if it were directly above the switch, it would make sense, but then of course it would take up more room.
Light switches in buildings are label-less, and this doesn't seem to confuse too many people. Perhaps toggle widgets should be the same, and follow convention: left off, right on.