...

I will see if I can keep working with Scott on this today. It's quite frustrating, because there is no conceivable reason for it to be slow... except that it is :-)

At http://www.alexa.com/siteinfo/opensuse.org# Traffic stats. Average Average Load Time for Opensuse.org Slow (2.326 Seconds), 73% of sites are faster

We were actually talking about the stage server, which was taking something like 30 seconds to load. Turns out that the memcached server had been turned off, so it was spending all that time to query something that wasn't running. We flipped it back on, and it's all fixed now! As for production, some of you may remember me talking about this a year or two ago. When I took over the wiki, I made a ton of performance enhancements. We now use almost every type of caching available (opcode, memcached, file caching, and an accelerator). I also did some Apache performance tuning, which was desperately needed. The wiki is much faster than it used to be, even as we have added a bunch of new features. I also mentioned that we still have a long way to go, but most of it should now be front end optimization. Things like combining javascript files, using sprites where it makes sense, and removing serializations as much as possible will give us the biggest wins at this point. If we can go from 40+ resources down to under 30, and get them parallelized and ordered as well as possible, we should expect to cut that loading time by at least a second. In addition to the front end optimizations, I have noticed that I'm not using gzip and browser caching as much as I could. I'm looking into that right now. It should help us out a bit, but not nearly as much as what I mentioned above.

Sometimes it is stuck at loading (nothing) and then after repeating request it is all loaded in a very short time.

I have noticed that too. Sometimes it gets stuck on the main page, sometimes on counter, and sometimes on beans. Just now, I noticed beans taking something like 30 seconds to come back. I can't speak for counter, beans, and static, as those are hosted elsewhere, but I have looked into why sometimes the main page doesn't come back. So far, the main culprit appears to be our ancient load balancer. The good news is that we are already looking at replacing it. Speaking of that, we are looking at some load balancer options that can do a lot for us. One vendor can implement SPDY and is capable of downsampling images on the fly for mobile devices. Another is capable of TLS false start, which can cut SSL negotiation time by 25% or more. All of them will give us TCP optimizations, TLS 1.1/1.2, and better IPv6 support, so good things are coming down the road. Stay tuned :) -Matt

...

...

...

Christian Boltz 8/21/2012 12:08 PM >>> Hello,

Matthew, can you please check the update on stage again?

Special:Version still doesn't list the ReplaceText extension, and the print.css also doesn't seem to be included. (In other words: it looks like you didn't deploy the latest code from the 1.19 git branch.)

My mistake. I didn't realize the new installation was put on different branch. I just now pulled the changes from there. I get a database error... was there some SQL that needed to be run with that?

Besides that, the test account opensuse_stage:opensuse_stage (created by Thomas) doesn't seem to work - I don't get an error message, but I'm not logged in after (trying to) login.

I'm not sure if this ever worked for me, and Thomas told me on IRC today <digitltom> cboltz: that seems to be a bug, the login works for the staging access manager

Any idea what could be wrong with the login?

It seems to work fine on the openSUSE staging blogs and on staging Novell. Because of the broken database query, I can't try it out on the wikis at the moment. I'll make sure to check as soon as it's fixed. My guess right now would be that the login extension is broken with the new version.

Am Dienstag, 21. August 2012 schrieb Matthew Ehle:

...

...

[performance issues] We were actually talking about the stage server, which was taking something like 30 seconds to load. Turns out that the memcached server had been turned off, so it was spending all that time to query something that wasn't running. We flipped it back on, and it's all fixed now!

That explains it ;-) Thanks for checking and fixing it!

No problem. It's a huge relief.

...

have looked into why sometimes the main page doesn't come back. So far, the main culprit appears to be our ancient load balancer.

Are you seriously telling us that the load balancer makes the wiki slower? ;-)

Not in general, but it looks like it can drop requests sometimes. We're still trying to track it down exactly, but that's the most likely candidate right now.

...

Speaking of that, we are looking at some load balancer options that can do a lot for us. One vendor can implement SPDY and is capable of downsampling images on the fly for mobile devices.

Does this really make sense, now that mobile devices start having high- resolution displays?

I think so. Even high-res phone displays are much lower than your typical desktop. For example, the 12.2 banner on the wiki home page alone takes up most of an iPhone 4 screen. Put in the context of the full wiki page, the image is rendered at maybe half that resolution. Apply that principle to all the page images, and that's a big difference if you are on a mobile network trying to bring up a page. Sure, some people might zoom in on the image and then notice a difference, but for 99% of users, it's going to be worth the difference in page loading time.

...

Another is capable of TLS false start, which can cut SSL negotiation time by 25% or more.

IIRC most of the wiki runs over http, not https - therefore I doubt SSL negotiation speed is a real problem ;-)

It's about half and half right now. For whatever reason, most of the static content is hardcoded to be pulled from a secure connection. In fact, I would propose that we go ahead and turn of SSL for the rest of the wiki. As it stands, people logged into the forums, wikis, and blogs are vulnerable to session hijacking, which is only really fixable by encrypting the entire session. SSL is much less of a performance problem than it used to be, and using things like SPDY and false start make it completely negligible.

...

...

...

Christian Boltz 8/21/2012 5:36 PM >>> Hello,

...

I just now pulled the changes from there. I get a database error... was there some SQL that needed to be run with that?

Did you run maintenance/update.php ?

Yes, that was done on the original upgrade to 1.19. I ran it again for good measure, on both the core installation and for SMW. No luck :-(

...

...

Besides that, the test account opensuse_stage:opensuse_stage (created by Thomas) doesn't seem to work -

...

It seems to work fine on the openSUSE staging blogs and on staging Novell.

Indeed - I can login on https://loginstage.provo.novell.com/ with this account.

...

Because of the broken database query, I can't try it out on the wikis at the moment. I'll make sure to check as soon as it's fixed. My guess right now would be that the login extension is broken with the new version.

I'm not even sure if logging in on enstage worked with the old version...

It did. I always make sure the login extension works before putting it in prod, so it definitely worked with 1.17. I also tested it more recently when making some Access Manager updates.

...

...

IIRC most of the wiki runs over http, not https - therefore I doubt SSL negotiation speed is a real problem ;-)

It's about half and half right now. For whatever reason, most of the static content is hardcoded to be pulled from a secure connection.

Maybe the author of the Bento theme wasn't aware of protocol-relative URLs? Changing "https://static.opensuse.org/..." to "//static/opensuse.org/..." shouldn't cause any problems.

Interesting that you bring that up. The www pages were the same way. In fact, I just changed all those references to be protocol relative a few days ago. That's why you see my name all over the landing page repository in Github. I had to touch a lot of files :-)

...

In fact, I would propose that we go ahead and turn of SSL for the rest of the wiki. As it stands, people logged into the forums, wikis, and blogs are vulnerable to session hijacking, which is only really fixable by encrypting the entire session. SSL is much less of a performance problem than it used to be, and using things like SPDY and false start make it completely negligible.

There's nothing wrong with SSL for logged in users, OTOH it's superfluous for other visitors.

I was reading some interesting insights from Google engineers the other week. It's their vision that someday, every site will be served over SSL/TLS, and HTTP will be then what RSH is today. They certainly have done a lot to make this more of a possibility (OpenSSL patches, TLS false start, and SPDY to name a few). In fact, you may notice that now Firefox defaults to using HTTPS for Google searches, even if you are not authenticated. Pretty interesting I think.

...

...

Matthew Ehle 8/20/2012 6:31 AM >>> Christian Boltz 8/20/2012 6:15 AM >>> Hello,

please update enstage.o.o to the latest wiki code in git / 1.19 branch.

I added the Bento print.css and the ReplaceText extension. Both changes are untested, but "should work"[tm] ;-)

I just put your changes on stage.

Can you also do something against the slowness of enstage.o.o? This would be very welcome ;-)

I will see if I can keep working with Scott on this today. It's quite frustrating, because there is no >conceivable reason for it to be slow... except that it is :-)

-Matt

The slowness has been resolved. It was assumed that memcached was already running on this server. Upon discovering that it was not, we enabled and started memcached. This immediately resolved the issue. -Scott