Hi, I want to share some findings from the www.opensuse.org / wiki and iChain setup. Especially, I wanted to find out some things about iChain which served mainly as a black box for me so far ;-) # telnet en.opensuse.org 80 GET /skins/opensuse/css/local/wiki.css HTTP/1.1 Host: en.opensuse.org Connection: keep-alive [...] Connection closed by foreign host. # => iChain doesn't use Keepalive to clients. Asking Matt about it, he said it is a global setting that had to be disabled for one of the 35 web sites hosted through iChain. However, he thought it might have been do to some problem that no longer exists, and he wanted to do testing and possibly reenable it. => UPDATE from last night: iChain now *does* use Keepalive to clients. Matt could switch them on, and it doubles the request throughput. Cool! Looking at Expire headers, and playing with forced reloads, iChain clearly caches, and does this correctly. The requests served by cache hits correctly never turn up on the backend server, so they are effectively offloaded. However only some parts of the web site sets Expires headers. And the Expires headers that are set are much too short. => iChain caches. => Expires headers are missing. (we need to add them at the web server.) Playing with requests on wiki pages, I initially thought iChain also would compress (gzip content encoding). Matt didn't know about it either ;) and it turns out that the wiki does this. However, there are two problems with that: It is a little inconsistent, because it only occurs for some Accept encoding headers or user agents. root@doozer ~ # curl -sv -o /dev/null 'http://en.opensuse.org/Mirror_Infrastructure' 2>&1 | grep Enc < Vary: Accept-Encoding,Cookie root@doozer ~ # curl -sv -o /dev/null -H "Accept-Encoding: gzip,deflate" 'http://en.opensuse.org/Mirror_Infrastructure' 2>&1 | grep Enc

Accept-Encoding: gzip,deflate < Vary: Accept-Encoding,Cookie < Content-Encoding: gzip

=> iChain doesn't compress, the wiki does, but Apache doesn't. The front page should *definitely* be served compressed, in the future. Possibly, the PHP/zlib compression can just be replaced by Apache-wide compression (and caching). The (www.o.o) web server is pretty fast for many things. 50 r/s for a PHP script (typical speed), thousands of r/s for static stuff. => web server is fast. (I didn't look much at the wiki yet. I didn't look for hot spots.) Through iChain, the speed is retained. With iChain's caching it actually increases a little. On the local network, 4000-6000 r/s are possible through iChain. => iChain can be very fast. HOWEVER, real-life performance for a web client sucks, from Europe. 2 (two!) requests/s is what I get from Germany. (UPDATE from last night: 8 r/s, with Keepalive and three further machines.) The other direction works significantly better. software.o.o or static.o.o deliver 25 r/s to an US client without keepalive. 50 r/s with keepalive. Thus, Internet is 10x faster when the server is in Europe. These numbers are reproducible and not susceptible to circadian variation. In addition, from a different US server, I can pull 5x as many requests to Europe as from iChain. Enter curl -I http://www.opensuse.org/ http://www.opensuse.org/en/ (as an Europaean...) and watch the *huge* latency between the two requests. Note that it doesn't matter if a request is tiny or "only" a if-modified-since request, as the latency is the same. => ? what is the conclusion... iChain is slow? No explanation for this. TCP/IP stack of Netware is not good for WANs? General conclusion: => overseas Internet is slow for web serving in general. It has a big pipe but large buffers suffers high latencies. => It would be much better to run a server locally at every continent. Anyway, regarding 11.1 release we can do quite some things still; and also, the last nights improvements were really significant. It is unclear whether the additional servers helped with that, because something still looks like some weird bottleneck around iChain, but we definitely have a quadrupled request throughput since last night, even though Keepalives only account for half of the improvement, and we don't know what caused the rest. Peter -- Contact: admin@opensuse.org (a.k.a. ftpadmin@suse.com) #opensuse-mirrors on freenode.net Info: http://en.opensuse.org/Mirror_Infrastructure SUSE LINUX Products GmbH Research & Development