Webmaster forum - User-agent string

Author

User-agent string - "compatibility flag"

Karl Groves

2007-02-15, 6:17 pm

I'm doing a little bit of research into user-agent strings and I keep
seeing reference to "compatibility flags".

This, for those who don't know would be where it says "compatible" in the
following UA String:
Mozilla/5.0 (compatible; U; Windows NT 5.1; en-US; rv:1.8.1.1)
Gecko/20061204 Firefox/2.0.0.1

In many UA strings, that "compatible" part is replaced with an actual
Operating System identifier such as "Windows" or "Macintosh".

My Googling for an answer on what this means has left me only with this:
"Indicates that this browser is compatible with a common set of features"

OK, so what the heck are those "common set of features"?

--
Karl Groves
http://karlcore.com
http://chevelle.karlcore.com

GreyWyvern

2007-02-15, 6:17 pm

And lo, Karl Groves didst speak in alt.www.webmaster:

> This, for those who don't know would be where it says "compatible" in the
> following UA String:
> Mozilla/5.0 (compatible; U; Windows NT 5.1; en-US; rv:1.8.1.1)
> Gecko/20061204 Firefox/2.0.0.1

> My Googling for an answer on what this means has left me only with this:
> "Indicates that this browser is compatible with a common set of features"
>
> OK, so what the heck are those "common set of features"?

It's legacy from those bygone days when Netscape first introduced
scriptable events. Soon everyone who was anyone was using them, and these
sites distinguished between technologically superior Netscape and poor IE2
by looking for "Mozilla" in the user agent.

When IE introduced the same scripting abilities as Mozilla in IE3, they
quickly realized that no one would see them flexing their muscles because
everyone was searching for Mozilla (or a "Mozilla" + a version number) in
the user agent string. Their solution was to "fudge" things by including
Mozilla in their user agent string as well, but suffixing it with
"compatible". So IE3 was "Mozilla compatible" !!!

Over time, it came to mean compatible with a certain Mozilla level of
development, based on the version number after the slash. When IE
released versions which equalled or superceded similar Mozilla versions,
they upped the version number they were displaying.

It doesn't mean anything more than that. Those things after the
"compatible" aren't flags of any kind. The brackets were just the default
OS ident area MSIE (and other browsers) always filled with misc junk, even
before IE3.

See the HTTP spec for more information on the standardized construction of
User Agent strings:
http://www.faqs.org/rfcs/rfc2068.html (sec 14.42 + 3.8)

Grey

--
The technical axiom that nothing is impossible sinisterly implies the
pitfall corollary that nothing is ridiculous.
- http://www.greywyvern.com/orca#search - Orca Search: Full-featured
spider and site-search engine

Karl Groves

2007-02-15, 6:17 pm

GreyWyvern <spam@greywyvern.com> wrote in
news:op.tns7i4uosl6xfd@news.nas.net:

> And lo, Karl Groves didst speak in alt.www.webmaster:
>
>
>
> It's legacy from those bygone days when Netscape first introduced
> scriptable events. Soon everyone who was anyone was using them, and
> these sites distinguished between technologically superior Netscape
> and poor IE2 by looking for "Mozilla" in the user agent.
>
> When IE introduced the same scripting abilities as Mozilla in IE3,
> they quickly realized that no one would see them flexing their
> muscles because everyone was searching for Mozilla (or a "Mozilla" +
> a version number) in the user agent string. Their solution was to
> "fudge" things by including Mozilla in their user agent string as
> well, but suffixing it with "compatible". So IE3 was "Mozilla
> compatible" !!!
>
> Over time, it came to mean compatible with a certain Mozilla level of
> development, based on the version number after the slash. When IE
> released versions which equalled or superceded similar Mozilla
> versions, they upped the version number they were displaying.
>
> It doesn't mean anything more than that. Those things after the
> "compatible" aren't flags of any kind. The brackets were just the
> default OS ident area MSIE (and other browsers) always filled with
> misc junk, even before IE3.
>
> See the HTTP spec for more information on the standardized
> construction of User Agent strings:
> http://www.faqs.org/rfcs/rfc2068.html (sec 14.42 + 3.8)
>

I'm reluctant to believe what you say is accurate, regarding what
"compatible" means.

I believe that what you've said is only relevant insofar as the
"Mozilla" part is concerned (at the very beginning of the string).

However, most UAs have some sort of OS-related name there:
Mozilla/5.0 (Macintosh; U; PPC Mac OS X; en-US; rv:1.0.1) Gecko/20021104
Chimera/0.6
Mozilla/5.0 (X11; U; Linux i686; en; rv:1.8.1) Gecko/20061203
Epiphany/2.16 Firefox/2.0
Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9a1) Gecko/20061204
GranParadiso/3.0a1

Here, we see three UA strings identifying themselves as "Macintosh",
"X11" and "Windows" respectively, in that area which is also used as a
"compatibility flag".

Given that, what's the qualitative difference between a listing that says
"compatible" and one that specifies an OS? Returning to the quote I saw
elsewhere which says "Indicates that this browser is compatible with a
common set of features", exactly *what* set of features? Are these OS-
related features? Hardware-related (doubtful), or something else?

--
Karl Groves
http://karlcore.com
http://chevelle.karlcore.com

GreyWyvern

2007-02-15, 6:17 pm

And lo, Karl Groves didst speak in alt.www.webmaster:

> GreyWyvern wrote:
>
>
> I'm reluctant to believe what you say is accurate, regarding what
> "compatible" means.

It means nothing more than "Mozilla compatible". A device designed to
bypass scripts which sniff for Mozilla without actually lying about the
agent's identity. Occam's razor.

Also, the original UA string you posted has been altered in some way
because Firefox never includes "Mozilla compatible" in their default user
agent, for *any* of their products. Why should they? They *are* Mozilla.

If you take the time to examine a swath of UA strings you'll find that
only MSIE - or browsers trying to spoof versions of MSIE - contain the
word "compatible".

Grey

--
The technical axiom that nothing is impossible sinisterly implies the
pitfall corollary that nothing is ridiculous.
- http://www.greywyvern.com/orca#search - Orca Search: Full-featured
spider and site-search engine

Karl Groves

2007-02-15, 6:17 pm

GreyWyvern <spam@greywyvern.com> wrote in
news:op.tns92hr8sl6xfd@news.nas.net:

> And lo, Karl Groves didst speak in alt.www.webmaster:
>
>
> It means nothing more than "Mozilla compatible". A device designed to
> bypass scripts which sniff for Mozilla without actually lying about
> the agent's identity. Occam's razor.
>
> Also, the original UA string you posted has been altered in some way
> because Firefox never includes "Mozilla compatible" in their default
> user agent, for *any* of their products. Why should they? They
> *are* Mozilla.
>
> If you take the time to examine a swath of UA strings you'll find that
> only MSIE - or browsers trying to spoof versions of MSIE - contain
> the word "compatible".

OK, I see where you're coming from. However, it does make me wonder why
these browsers do it:

ALL Konqueror versions, NetPositive, Netscape 8.0.1, Versions of iCab,
Avant (I know, built on IE), Maxthon (I think also built on IE), numerous
versions of Opera contain "compatible" in that spot:
http://www.useragentstring.com/pages/Browserlist/

(BTW you were right. I added "compatible" to the sample UA string just to
use it as an example)

--
Karl Groves
http://karlcore.com
http://chevelle.karlcore.com

Toby A Inkster

2007-02-15, 10:16 pm

Karl Groves wrote:

> I'm reluctant to believe what you say is accurate, regarding what
> "compatible" means.

What GreyWyvern said is accurate.

Microsoft added "compatible" into their user agent string to say that
although they weren't Mozilla, they were broadly compatible with it.
Similarly, Safari has always carried the string "like Gecko" in its
User-Agent headers. It's just to defeat stupid browser sniffing.

The User-Agent string is, in practise, a free-form text field. It usually
carries browser and version information, often including the operating
system and version, and sometimes includes the browser locale and a 'U' to
indicate Unicode support. These may be in any order, in any format, and
may or may not be false or misleading.

--
Toby A Inkster BSc (Hons) ARCS
Contact Me ~ http://tobyinkster.co.uk/contact
Geek of ~ HTML/SQL/Perl/PHP/Python*/Apache/Linux

* = I'm getting there!

Karl Groves

2007-02-15, 10:16 pm

Toby A Inkster <usenet200702@tobyinkster.co.uk> wrote in
news:ubbfa4-1l4.ln1@ophelia.g5n.co.uk:

> Karl Groves wrote:
>
>
> What GreyWyvern said is accurate.
>
> Microsoft added "compatible" into their user agent string to say that
> although they weren't Mozilla, they were broadly compatible with it.
> Similarly, Safari has always carried the string "like Gecko" in its
> User-Agent headers. It's just to defeat stupid browser sniffing.
>
> The User-Agent string is, in practise, a free-form text field. It
> usually carries browser and version information, often including the
> operating system and version, and sometimes includes the browser
> locale and a 'U' to indicate Unicode support. These may be in any
> order, in any format, and may or may not be false or misleading.
>

I think we're all clear in understanding what you've said above. What
I'm having a hard time with is this:
1. That this "compatibility flag" can be something (seemingly) legitimate
and clear, such as "Macintosh" or something vague like "compatible" and,
2. That "compatible" is meant to identify that it is "compatible with a
common set of features"

In other words, most browsers (esp. the Mozilla based ones) are placing
some real, honest identifying information there and that, in turn, is
supposed to mean something. While I have no doubt that Microsoft is
actively intending to do some flubbing somewhere, that "something" (in
the case of Mozilla) *has* to be there for a reason. Soo, what is it?

I know this is probably repetitive but here's what I'm getting at:
Functioning under these assumptions:
a. M$ is pretending to be Mozilla with the use of "compatible"
b. Mozilla doesn't need to pretend to be Mozilla, therefore they use
something legitimate in that place, such as "Macintosh"

Then what, exactly, is Mozilla intending to convey in that part of the UA
String?

--
Karl Groves
http://karlcore.com
http://chevelle.karlcore.com

Chuck Anderson

2007-02-16, 6:18 am

GreyWyvern wrote:
> And lo, Karl Groves didst speak in alt.www.webmaster:
>
>
>
> It means nothing more than "Mozilla compatible". A device designed to
> bypass scripts which sniff for Mozilla without actually lying about the
> agent's identity. Occam's razor.
>

.....

On a side note. My web hosting provider is using mod_security to block
requests. One of the keys to blocking is if the UA string contains
"compatible ;" (as opposed to compatible; - with no space before the
;). He says this indicates it is a comment spam bot.

Has anyone else heard of this - - - and does anyone understand why the
spam bot would include the extra space?

--
*****************************
Chuck Anderson • Boulder, CO
http://www.CycleTourist.com
*****************************

Rik

2007-02-16, 6:18 am

On Fri, 16 Feb 2007 05:01:59 +0100, Chuck Anderson
<websiteaddress@seemy.sig> wrote:

> On a side note. My web hosting provider is using mod_security to block
> requests. One of the keys to blocking is if the UA string contains
> "compatible ;" (as opposed to compatible; - with no space before the
> ;). He says this indicates it is a comment spam bot.
>
> Has anyone else heard of this - - - and does anyone understand why the
> spam bot would include the extra space?

Probably someone made a tiny mistake in creating the bot, which made it
possible to filter for it. Now the mistake is out in the open, I suspect
someone will fix this bot...
--
Rik Wasmus

Toby A Inkster

2007-02-16, 6:18 am

Karl Groves wrote:

> 1. That this "compatibility flag" can be something (seemingly) legitimate
> and clear, such as "Macintosh" or something vague like "compatible" and,
> 2. That "compatible" is meant to identify that it is "compatible with a
> common set of features"

Seriously, all "Mozilla/4.0 (compatible;" is indicating is:

1. Microsoft stuck "Mozilla/4.0" in there to trick browser
sniffers.
2. They added "compatible" to tell humans reading the log
files that they're not *really* Netscape 4, but are
broadly compatible with it. (Supporting Javascript,
Java and other flashy J-words that were so fashionable
at the time.)

There is not a "compatibility flag" that browsers either choose to include
or not. There is just a word "compatible" that Microsoft include in their
default User-Agent strings. It doesn't have any special technical meaning
beyond the normal definition of "compatible" which you can look up in your
nearest dictionary at your convenience.

> In other words, most browsers (esp. the Mozilla based ones) are placing
> some real, honest identifying information there and that, in turn, is
> supposed to mean something. While I have no doubt that Microsoft is
> actively intending to do some flubbing somewhere, that "something" (in
> the case of Mozilla) *has* to be there for a reason. Soo, what is it?

You're assuming that there is a "there" -- a particular location in the
User-Agent string what can either be "compatible" or something else. There
is no "there" -- the User-Agent string is not formatted as a set of
several fields, each signifying something: it is a single free-form text
field, which the browser maker can put whatever they like in.

Mozilla is not particularly honest though. When Mozilla 1.0 was released,
what was the User-Agent string? Well, it started "Mozilla/5.0", not
"Mozilla/1.0". The *real* Gecko version number is hidden away further along
the User-Agent string, prefixed by "rv:". It just used "Mozilla/5.0" to
trick browser sniffers into assuming it was the next version of Internet
Explorer. (IE 5 for Mac already used "Mozilla/5.0".)

Opera has been honest by default since version 8.0. It has always had an
honest user-agent string, but it was switched off by default from version
3.mumble IIRC. It could be enabled in the settings though.

Safari dishonestly has the magic strings "Mozilla/5.0" and "Gecko" in its
UA; Konqueror has both of those, and also "compatible".

> I know this is probably repetitive but here's what I'm getting at:
> Functioning under these assumptions:
> a. M$ is pretending to be Mozilla with the use of "compatible"
> b. Mozilla doesn't need to pretend to be Mozilla, therefore they use
> something legitimate in that place, such as "Macintosh"

You are then assuming that there is a "there" -- a particular slot where
MSIE puts "compatible" and Mozilla puts something else.

> Then what, exactly, is Mozilla intending to convey in that part of the UA
> String?

I would think that would be fairly clear: operating system.

You are really over-analysing the user-agent string. It is a free-form
text field. Browsers manufacturers tend to stuff it with certain keywords
to slip past browser sniffers.

It *can* be used to your advantage, but only if you keep up to date with
the latest new browsers coming out. Sniffing for "Gecko" in the UA string
was a reasonably useful way of detecting Gecko-based browsers... until
Safari was released. If you see "X11", "Linux", "BSD" or "Unix" in a
user-agent string, it would be a reasonable assumption that your visitor
is using a Linux or UNIX based platform. Similarly "Mac" and "Win".

But at its heart it is free-form text, and should be treated much like you
might treat the "<meta name=description>" tag used on some web pages:
search is for particular keywords which could give you a clue as to what
it's about, but don't try to formally parse it as you'll end up tearing
your hair out.

--
Toby A Inkster BSc (Hons) ARCS
Contact Me ~ http://tobyinkster.co.uk/contact
Geek of ~ HTML/SQL/Perl/PHP/Python*/Apache/Linux

* = I'm getting there!

Toby A Inkster

2007-02-16, 6:18 am

Toby A Inkster wrote:

> 'U' to indicate Unicode support

Apparently this actually indicates that it supports strong encryption. An
'I' for weak encryption and an 'N' for no encryption.

--
Toby A Inkster BSc (Hons) ARCS
Contact Me ~ http://tobyinkster.co.uk/contact
Geek of ~ HTML/SQL/Perl/PHP/Python*/Apache/Linux

* = I'm getting there!

Karl Groves

2007-02-16, 6:20 pm

<snip an excellent reply by Toby>

Having read the relevant portions of the RFCs for HTTP 1.0 and 1.1, I
realize that, not only is the content of the UA string not set in stone,
but neither is it even required. I understand that completely.

Given that, it would seem that what a browser manufacturer uses as its UA
string is driven by some conscious desire to convey *something*.

In Microsoft's case, it is clear that their goal was to deceptively
circumvent browser detection efforts of sites "optimized" for Netscape
Navigator.

In contrast, there are browsers (such as Mozilla and Opera) which are
"honest" in what they say in their UA string. In those cases, why would
the UA string mention the OS twice?

Let's take these three UA strings:
Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.7.5) Gecko/20060127
Netscape/8.1

Mozilla/5.0 (Macintosh; U; PPC Mac OS X; en) AppleWebKit/521.25 (KHTML,
like Gecko) Safari/521.24

Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.8.0.8) Gecko/20061029
SeaMonkey/1.0.6

In each case, some generic OS-related identifier is given in the same
place as "compatible" would be, were this browser IE.

This, despite the fact that there is a much more specific, kernel/ cpu
related OS identifier in the middle of the string.

IF this is optional
IF this means nothing
WHY is it there and WHAT does it mean?

And yes, I am overanalyzing it. That's what makes it so fun.

--
Karl Groves
http://karlcore.com
http://chevelle.karlcore.com

Steve Pugh

2007-02-16, 6:20 pm

On Feb 16, 2:43 pm, Karl Groves <k...@NOSPAMkarlcore.com> wrote:

> In contrast, there are browsers (such as Mozilla and Opera) which are
> "honest" in what they say in their UA string. In those cases, why would
> the UA string mention the OS twice?
>
> Let's take these three UA strings:
> Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.7.5) Gecko/20060127
> Netscape/8.1
>
> Mozilla/5.0 (Macintosh; U; PPC Mac OS X; en) AppleWebKit/521.25 (KHTML,
> like Gecko) Safari/521.24
>
> Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.8.0.8) Gecko/20061029
> SeaMonkey/1.0.6
>
> In each case, some generic OS-related identifier is given in the same
> place as "compatible" would be, were this browser IE.
>
> This, despite the fact that there is a much more specific, kernel/ cpu
> related OS identifier in the middle of the string.
>
> IF this is optional
> IF this means nothing
> WHY is it there and WHAT does it mean?

I happened to have some old UA strings in a file, and the listing of
the OS twice seems to have started with Gecko. Old Netscape didn't do
that.

My guess is that the first occurence is the platform the browser code
was compiled for and the second is the actual OS in use.

Steve

Dan

2007-02-17, 3:15 am

On Feb 16, 4:08 am, Toby A Inkster <usenet200...@tobyinkster.co.uk>
wrote:
> Mozilla is not particularly honest though. When Mozilla 1.0 was released,
> what was the User-Agent string? Well, it started "Mozilla/5.0", not
> "Mozilla/1.0". The *real* Gecko version number is hidden away further along
> the User-Agent string, prefixed by "rv:". It just used "Mozilla/5.0" to
> trick browser sniffers into assuming it was the next version of Internet
> Explorer. (IE 5 for Mac already used "Mozilla/5.0".)

The "Mozilla/5.0" part in Gecko-based browsers actually originally
derived from the fact that the original Netscape source code released
as open-source to start the Mozilla project was the code for a
partially-developed "Netscape 5.0", which used Mozilla/5.0 in its UA
string in accordance with Netscape tradition (e.g., Netscape 4.0 had
used Mozilla/4.0). However, most of the code was soon scrapped in
favor of developing a new rendering engine from scratch, and when a
new Netscape version finally did get derived from it, the marketing
types decided to skip 5.0 entirely and go to 6.0, but the UA string
had stuck at Mozilla/5.0 anyway, and they've been too cowardly to
change it since even though the number no longer stands for anything
coherent, out of fear that it will discombobulate the clueless
sniffers in Web sites.

--
Dan

GreyWyvern

2007-02-19, 6:18 pm

And lo, Karl Groves didst speak in alt.www.webmaster:

> In contrast, there are browsers (such as Mozilla and Opera) which are
> "honest" in what they say in their UA string. In those cases, why would
> the UA string mention the OS twice?
>
> Let's take these three UA strings:
> Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.7.5) Gecko/20060127
> Netscape/8.1
>
> Mozilla/5.0 (Macintosh; U; PPC Mac OS X; en) AppleWebKit/521.25 (KHTML,
> like Gecko) Safari/521.24
>
> Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.8.0.8) Gecko/20061029
> SeaMonkey/1.0.6
>
>
> In each case, some generic OS-related identifier is given in the same
> place as "compatible" would be, were this browser IE.
>
> This, despite the fact that there is a much more specific, kernel/ cpu
> related OS identifier in the middle of the string.
>
> IF this is optional
> IF this means nothing
> WHY is it there and WHAT does it mean?

I'm just guessing here, but even though there are no specifications
governing the order of tokens in the UA, most browsers have fallen
gracefully into a general consensus on what they include and in what
order. From this, I would assume that IE is trying to be nice to scripts
which split the contents of the parenthesized area on the semicolon. If
they included both "compatible" and "Windows" in their UA, then "U",
"Windows NT", "en-US" etc. would get pushed into the wrong array positions
wrt the same operation performed on a Mozilla UA.

In other words, it's just more fudging to appease old scripts.

> And yes, I am overanalyzing it. That's what makes it so fun.

Don't forget what's in the middle of overanalyzing...

Grey

--
The technical axiom that nothing is impossible sinisterly implies the
pitfall corollary that nothing is ridiculous.
- http://www.greywyvern.com/orca#search - Orca Search: Full-featured
spider and site-search engine