In as much as I haven't come across a single case--not one--wherein the speaker has claimed he wasn't speaking a language, I'm curious as to how you came to the conclusion that the majority of them say they don't. Care to share?
It is primarily based on having been ordained in a non-denominational church (and later in an Assemblies of God) which practiced such. Part of the ordination process (I actually had to go through it twice because one didn't transfer to the other) was a study of this subject.
Yes, some will call it a language. And really, I see that out of ease (now there may be some denominations that are very conservative or hard headed on the matter, that would disagree, but I haven't met one myself). It is a lot easier to explain speaking in tongues as a language than something else.
However, when one really gets down to it, most adherents will admit that it is not a language in the typical usage of that word. It isn't a form of communication that is taught (I believe there are some who try to, but that is a minority, and really an extremist minority). It isn't a form of communication that has really any structure. There isn't and alphabet or actual words that show up. There isn't a consistency from one person to another as it is meant to be something personal. It isn't even meant to be understood by humans unless one is granted the gift of interpretation.
When it gets down to it, speaking in tongues simply does not fit a basic definition of language, and most adherents would agree with that. Calling it a language really is one of ease, even though the meaning of language in this context greatly changes.
Adherents, for the most part, do understand this. I have heard some say that it is the language of the angels, or of God, but like many heavenly things, it takes on a very different meaning.
So I guess I would agree that some call it a language. However, most don't mean it is a language in the typical fashion. It is something quite different.