Adding "Content-encoding: gzip" to statcounter.js's headers ...?

[nickpelling]

Hi everyone,

The feature I'd really like is to have statcounter.js arrive gzip compressed: and I'm assured that it is indeed stored gzip compressed on statcounter's server. However, I just today noticed that the well-known webpage speed optimizer has a button for showing headers, and the headers my blog is getting for statcounter.js are:-

HTTP/1.1 200 OK
Date: Tue, 15 Jun 2010 23:23:11 GMT
Server: PWS/1.5.19.2
X-Px: ht ord-agg-n9.panthercdn.com
P3P: policyref="http://www.statcounter.com/w3c/p3p.xml", CP="ADMa OUR COM NAV NID DSP NOI COR"
ETag: "4948d1-1e50-4bb97f72"
Cache-Control: max-age=43200
Content-Length: 7760
Content-Type: application/x-javascript; charset=utf-8
Last-Modified: Mon, 05 Apr 2010 06:13:06 GMT
Connection: keep-alive

...whereas my (successfully) compressed css headers look like...

HTTP/1.1 200 OK
Date: Tue, 15 Jun 2010 23:32:06 GMT
Server: Apache
X-Powered-By: PHP/5.2.13
Content-Encoding: gzip
Vary: Accept-Encoding
Transfer-Encoding: chunked
Content-Type: text/css

Is there a good pragmatic way [ok, "hack"] to get "Content-Encoding: gzip Vary: Accept-Encoding" into the headers for statcounter.js?

Cheers, ....Nick Pelling....

[StatCounterJen]

Hi Nick,

As you mention, counter.js IS gzipped.

You can verify this here:
http://web-sniffer.net/?url=http%3A%...type=GET&uak=0

The encoded js file is only 2883 bytes.

Although our server supports gzip if your client does not then it will default to sending it out in its uncompressed form - this may explain the results you are seeing.

Every resource of the web has a "header" attached to it that contains extra info about the content it is delivering. Here is the header info for our javascript. You'll notice towards the bottom that the Content-Encoding is marked as gzip.

Date: Tue, 16 Mar 2010 13:38:39 GMT
Server: PWS/1.5.16.2
X-Px: ht fra-ti2-n91.panthercdn.com
Cache-Control: max-age=43200
ETag: "26803e-246e-4b9e0225"
P3P: policyref="http://www.statcounter.com/w3c/p3p.xml", CP="ADMa OUR COM NAV NID DSP NOI COR"
Content-Length: 2883
Content-Type: application/x-javascript; charset=utf-8
Content-Encoding: gzip
Vary: Accept-Encoding
Last-Modified: Mon, 15 Mar 2010 09:47:17 GMT

Furthermore, this page, which is specifically setup to test for gzip compression, confirms that our js is gzipped.

http://www.whatsmyip.org/http_compression/

Our JS address to test is : http://www.statcounter.com/counter/counter.js

[Arne]

Originally Posted by StatCounterJen

Furthermore, this page, which is specifically setup to test for gzip compression, confirms that our js is gzipped.

http://www.whatsmyip.org/http_compression/

Our JS address to test is : http://www.statcounter.com/counter/counter.js

And the WebSiteOptimization.com analyze says it's not zipped. As previously when this issue was discussed, I choose to not believe WebSiteOptimization in this case.

[nickpelling]

Hi Jen and Arne,

When a webserver serves a file (even a file that it doesn't itself host, such as StatCounter's "counter.js"), it wraps its own set of headers around it. This is quite distinct from the set of headers that StatCounter's own servers would wrap around the same file.

Never mind whether or not you happen to believe WebSiteOptimization.com, if you load the FireBug and PageSpeed plugins into Mozilla Firefox, record the activity for a typical StatCountered WordPress blog, and look at the headers of what is happening on your own machine, you'll see that these are simply not the same as the ones you get when loading counter.js directly as served from your own server. Sorry to say it, but there is a core difference between hosting and serving you don't quite seem to get.

This impacts other areas, too. For example, counter.js as served from an out-of-the-box WordPress blog will not have max-age set at all, so it will not be cached. Furthermore, if your web host stores counter.js gzip compressed but it is served uncompressed, your web host will (I'm pretty sure) be decompressing it before sending it out. Check this for yourself.

So, editing my Cipher Mysteries WordPress .htaccess file to say...

<FilesMatch "counter\.js$">
Header set Content-Encoding "gzip"
Header set Cache-Control "max-age=2592000, public"
</FilesMatch>

...does indeed cause a gzipped version of counter.js to be sent that is then cached on the user's machine. Hence I would simply point out that encouraging people to do this kind of thing for their own blogs could potentially reduce the load on your webhost by 90% (while also cutting the amount of bandwidth down) as well as making everyone's pages load noticeably faster.

There are still further improvements that can be made. PageSpeed notes that www.statcounter.com and c.statcounter.com require two separate DNS lookups rather than one: serving these from the same domain (www. and c. are different) would halve the number of DNS lookups being done each day, and hence speed up page loading.

These things aren't rocket science, but you have to understand the difference between hosting and serving - whereas the host merely stores the file, the server tells the client what to do with the file. And in the case of a blog, the server isn't your machine, it's the blog's own server.

Of course, don't just take my word for it - get in a £200/hour web guru to tell you the same thing, that'll work. Or you could install FireBug and PageSpeed and see for yourself. Actually, as I'm a professional programmer, you could pay me £200/hr to write you a report about this, that would work too.

Cheers, ....Nick Pelling.... //

[StatCounterJen]

Hi Nick,

Thanks for your post; we all want to make StatCounter into the best possible web stats service so we appreciate your input to improve it... however, I'm having some trouble following your train of thought...

In any event, I still stand by what I have said! If you believe firebug in preference to web-sniffer, then here are the results of loading www.statcounter.com/counter/counter.js in it.

First load with cache deleted. Note the use of gzip in the response and the content length is 2647. If it was not gzipped the content length would be 7,760.

Response Headersview source
Date	Mon, 21 Jun 2010 08:14:32 GMT
Server	PWS/1.5.19.2
X-Px	ht lhr-agg-n43.panthercdn.com
P3P	policyref="http://www.statcounter.com/w3c/p3p.xml", CP="ADMa OUR COM NAV NID DSP NOI COR"
Etag	"4948d1-1e50-4bb97f72"
Cache-Control	max-age=43200
Content-Length	2647
Content-Type	application/x-javascript; charset=utf-8
Content-Encoding	gzip
Vary	Accept-Encoding
Last-Modified	Mon, 05 Apr 2010 06:13:06 GMT
Connection	keep-alive
Request Headersview source
Host	www.statcounter.com
User-Agent	Mozilla/5.0 (Windows; U; Windows NT 5.1; en-GB; rv:1.9.1.9) Gecko/20100315 Firefox/3.5.9 ( .NET CLR 3.5.30729)
Accept	*/*
Accept-Language	en-GB,en;q=0.7,en-gb;q=0.3
Accept-Encoding	gzip,deflate
Accept-Charset	ISO-8859-1,utf-8;q=0.7,*;q=0.7
Keep-Alive	300
Connection	keep-alive
Referer	http://forum.statcounter.com/vb/showthread.php?p=241837
Cookie	landing=http%3A//www.statcounter.com/|||; is_unique=sc5765383.1272281776.0-4705966.1272358253.0-2453826.1277107773.29-5760181.1272475078.0-5726754.1272634858.0-2527102.1275040723.1-1521418.1272982439.0-5360627.1273058957.0-1072802.1273084318.0-3108683.1273485177.0-1533266.1273498088.0-4279096.1273501038.0-624054.1273570093.0-306741.1274871326.2-1367758.1273592298.0-1417324.1275992001.1-3873100.1273833851.0-477837.1273846328.0-4697194.1274869831.1-874220.1274180359.0-609958.1274269904.0-5124611.1274439508.0-255044.1274711512.0-5899082.1274726907.0-3984102.1274788895.0-3696903.1276076144.1-5906824.1274864376.0-2316699.1274867810.0-594085.1274869757.0-4906093.1274869774.0-2678513.1274869861.0-237649.1274869894.0-2026766.1274869981.0-3768968.1274870645.0-1352811.1274871609.0-5880594.1275040573.0-4937307.1275298022.0-3189873.1275307006.0-5136235.1275389876.0-2084610.1275404127.0-4053659.1275411619.0-4606630.1275478685.0-3877174.1275981682.0-4658975.1276245636.0-2226915.1276251594.0-1285416.1276279503.0-4447855.1276593639.0-1882249.1276590967.0-5759355.1276853339.0; vblastvisit=1272641498; vblastactivity=0; vbuserid=4; vbpassword=801b19148d81178753089d7e69a038bb; is_unique_1=sc5899082.1274727955.0-3730279.1274869774.0-1938274.1275307006.0; login=aodhan%26005d05de29487ec44cd07bd9d757d4e1; vbsessionhash=9f87d15a38feac58c0fffb952c3aa0a2; vbthread_lastview=799b165471a49f0d6bacf8e8dc70fcfc48696416a-1-%7Bi-38835_i-1276907199_%7D
Pragma	no-cache
Cache-Control	no-cache

Now if we reload www.statcounter.com/counter/counter.js (it will now be cached) and here is the resposne you will see in firebug. No mention of gzip or content length, as it was not downloaded this time. The browser reused its local cache.

Response Headersview source
Date	Mon, 21 Jun 2010 08:15:55 GMT
Server	PWS/1.5.19.2
X-Px	ht lhr-agg-n43.panthercdn.com
P3P	policyref="http://www.statcounter.com/w3c/p3p.xml", CP="ADMa OUR COM NAV NID DSP NOI COR"
Etag	"4948d1-1e50-4bb97f72"
Cache-Control	max-age=43200
Last-Modified	Mon, 05 Apr 2010 06:13:06 GMT
Connection	keep-alive
Request Headersview source
Host	www.statcounter.com
User-Agent	Mozilla/5.0 (Windows; U; Windows NT 5.1; en-GB; rv:1.9.1.9) Gecko/20100315 Firefox/3.5.9 ( .NET CLR 3.5.30729)
Accept	*/*
Accept-Language	en-GB,en;q=0.7,en-gb;q=0.3
Accept-Encoding	gzip,deflate
Accept-Charset	ISO-8859-1,utf-8;q=0.7,*;q=0.7
Keep-Alive	300
Connection	keep-alive
Referer	http://forum.statcounter.com/vb/showthread.php?p=241837
Cookie	landing=http%3A//www.statcounter.com/|||; is_unique=sc5765383.1272281776.0-4705966.1272358253.0-2453826.1277108074.29-5760181.1272475078.0-5726754.1272634858.0-2527102.1275040723.1-1521418.1272982439.0-5360627.1273058957.0-1072802.1273084318.0-3108683.1273485177.0-1533266.1273498088.0-4279096.1273501038.0-624054.1273570093.0-306741.1274871326.2-1367758.1273592298.0-1417324.1275992001.1-3873100.1273833851.0-477837.1273846328.0-4697194.1274869831.1-874220.1274180359.0-609958.1274269904.0-5124611.1274439508.0-255044.1274711512.0-5899082.1274726907.0-3984102.1274788895.0-3696903.1276076144.1-5906824.1274864376.0-2316699.1274867810.0-594085.1274869757.0-4906093.1274869774.0-2678513.1274869861.0-237649.1274869894.0-2026766.1274869981.0-3768968.1274870645.0-1352811.1274871609.0-5880594.1275040573.0-4937307.1275298022.0-3189873.1275307006.0-5136235.1275389876.0-2084610.1275404127.0-4053659.1275411619.0-4606630.1275478685.0-3877174.1275981682.0-4658975.1276245636.0-2226915.1276251594.0-1285416.1276279503.0-4447855.1276593639.0-1882249.1276590967.0-5759355.1276853339.0; vblastvisit=1272641498; vblastactivity=0; vbuserid=4; vbpassword=801b19148d81178753089d7e69a038bb; is_unique_1=sc5899082.1274727955.0-3730279.1274869774.0-1938274.1275307006.0; login=aodhan%26005d05de29487ec44cd07bd9d757d4e1; vbsessionhash=9f87d15a38feac58c0fffb952c3aa0a2; vbthread_lastview=799b165471a49f0d6bacf8e8dc70fcfc48696416a-1-%7Bi-38835_i-1276907199_%7D
If-Modified-Since	Mon, 05 Apr 2010 06:13:06 GMT
If-None-Match	"4948d1-1e50-4bb97f72"
Cache-Control	max-age=0

On another point, you say:

There are still further improvements that can be made. PageSpeed notes that http://www.statcounter.com and c.statcounter.com require two separate DNS lookups rather than one: serving these from the same domain (www. and c. are different) would halve the number of DNS lookups being done each day, and hence speed up page loading.

That's just not correct - www.statcounter.com serves a static file from a cdn for maximum speed and download whereas c.statcounter.com serve a dynamic request to our own pool of servers. Both dns look ups are required as they are going to different hosts to do different things.

Now - I note you say that examining a blog instead of counter.js is givng you different results - well if you have a blog that you want us to test for you so we can compare the results, just post it here.

[nickpelling]

Hi Jen,

When counter.js is served directly from your server, that server supplies both the file contents and the headers, and in that context the file is indeed supplied gzipped. So yes, I agree with your post as far as that goes.

However, when the HTML in a freshly-installed WordPress blog asks for counter.js to be sent, the headers it wraps around counter.js do not include the lines...

Cache-Control max-age=43200
Content-Encoding gzip
Vary Accept-Encoding

...that your server uses. This tiny bit of header magic is what (a) allows the file to be transmitted and received as gzipped data, and (b) allows the file to be cached (on a limited basis, anyway). Even this isn't quite perfect: lengthening the max-age and making the Cache-Control public would be an improvement, too.

So, as far as I can see, all you need to say to people installing a statcounter on their WordPress blog is that they should add a section onto the end of their main .htaccess file along the lines of:-

# StatCounter START
<FilesMatch "counter\.js$">
Header set Content-Encoding "gzip"
Header set Cache-Control "max-age=2592000, public"
</FilesMatch>
# StatCounter END

Oh, and as far as the DNS lookups go: of course I understand why using two domains would be convenient for StatCounter. However, all I'm pointing out is that this convenience comes at the cost of doubling the number of statcounter-related DNS lookups per page.

Overall, the point I'm trying to make is that the StatCounter and blog hosts both have very similar desires - minimizing DNS lookups, caching files, serving gzipped data, etc - and there are plenty of things StatCounter could do to try to do this better.

For example, has StatCounter thought of writing or commissioning a WordPress plugin that caches counter.js on the blog's own server and which silently checks for updates once in a while (say once a day)? Not exactly tricky, but it would surely save everyone a lot of DNS lookups, bandwidth, and time.

Cheers, ....Nick Pelling....