[opensuse] processwire, apache, 15000 stat(), open_basedir
I thought I'd create a new thread - I?ve had time to do a few more tests and have had one or two revelations. It seemed quite obvious that it was the php environment, specifically open_basedir, that was causing the problem. Those 15000 ca -- Per Jessen, Zürich (17.6°C) http://www.dns24.ch/ - free dynamic DNS, made in Switzerland. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
Per Jessen wrote:
I thought I'd create a new thread - I?ve had time to do a few more tests and have had one or two revelations.
It seemed quite obvious that it was the php environment, specifically open_basedir, that was causing the problem. Those 15000 ca
Sorry, this got sent way before I was done, I'll add the rest later today. -- Per Jessen, Zürich (19.6°C) http://www.dns24.ch/ - free dynamic DNS, made in Switzerland. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
Per Jessen wrote:
Per Jessen wrote:
I thought I'd create a new thread - I?ve had time to do a few more tests and have had one or two revelations.
It seemed quite obvious that it was the php environment, specifically open_basedir, that was causing the problem. Those 15000 ca
Sorry, this got sent way before I was done, I'll add the rest later today.
s/today// :-) It seemed quite obvious that it was the php environment, specifically open_basedir, that was causing the problem. Those 15000 lstat() calls had to have a performance impact. Well, that was easily tested. The 15000 calls were caused by open_basedir being enabled so just disable it and see what happens. No improvement. Far fewer lstat() calls, but no improvement. After a bit of trial&error, it turns out it is the opening/reading of 201 files that takes up the time. I have reproduced the numbers outside of the apache/php environment. Granted, I do find reading 200 files for a single request somewhat excessive, but with disk attached directly, it's done in 500ms, which is acceptable. One thing to look into is - why aren't those files being cached? That would appear to be because the device is network attached. I am still working on this. Interestingly, other apps such as wordpress and wikipedia do not appear to have this issue. Finally, I'm going to ask the developer to enable something called "processwire template caching". The website is largely static so it really should not to have to be dynamically generated all the time. -- Per Jessen, Zürich (13.8°C) http://www.hostsuisse.com/ - virtual servers, made in Switzerland. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 El 2016-06-16 a las 08:25 +0200, Per Jessen escribió:
After a bit of trial&error, it turns out it is the opening/reading of 201 files that takes up the time. I have reproduced the numbers outside of the apache/php environment. Granted, I do find reading 200 files for a single request somewhat excessive, but with disk attached directly, it's done in 500ms, which is acceptable.
IMHO, it is bad design for a web server having to open that many files for a single query. A compiler, that runs once, needs to read a ton of sources and includes, Ok. But something serving many clients and has to be fast, can not do that. IMO, it should work mostly on RAM. - -- Cheers Carlos E. R. (from 13.1 x86_64 "Bottle" (Minas Tirith)) -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.22 (GNU/Linux) iF4EAREIAAYFAldiYxYACgkQja8UbcUWM1w3AAD9EenpwcEW2GgkVY8CmjVpHLxh lPBef/gsYikUqF23/yAA/1it9y8iqJTCrWnOqFkc8AjwPIC6VnWxWncnslksucuq =e4P+ -----END PGP SIGNATURE-----
Carlos E. R. wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256
El 2016-06-16 a las 08:25 +0200, Per Jessen escribió:
After a bit of trial&error, it turns out it is the opening/reading of 201 files that takes up the time. I have reproduced the numbers outside of the apache/php environment. Granted, I do find reading 200 files for a single request somewhat excessive, but with disk attached directly, it's done in 500ms, which is acceptable.
IMHO, it is bad design for a web server having to open that many files for a single query.
I would tend to agree, but I want to look at wordpress and wikimedia to compare. Also, both filesystem caching and application caching should help alleviate the symptoms.
A compiler, that runs once, needs to read a ton of sources and includes, Ok. But something serving many clients and has to be fast, can not do that. IMO, it should work mostly on RAM.
With proper caching, it will. -- Per Jessen, Zürich (15.1°C) http://www.dns24.ch/ - your free DNS host, made in Switzerland. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
Am 16.06.2016 um 11:21 schrieb Per Jessen:
Carlos E. R. wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256
El 2016-06-16 a las 08:25 +0200, Per Jessen escribió:
After a bit of trial&error, it turns out it is the opening/reading of 201 files that takes up the time. I have reproduced the numbers outside of the apache/php environment. Granted, I do find reading 200 files for a single request somewhat excessive, but with disk attached directly, it's done in 500ms, which is acceptable.
IMHO, it is bad design for a web server having to open that many files for a single query.
I would tend to agree, but I want to look at wordpress and wikimedia to compare. Also, both filesystem caching and application caching should help alleviate the symptoms.
Did you try APC?
A compiler, that runs once, needs to read a ton of sources and includes, Ok. But something serving many clients and has to be fast, can not do that. IMO, it should work mostly on RAM.
With proper caching, it will.
I think, that the PHP7 will solve that situation as it's per default caching all the compiled sources in it's separate local cache. -- Johannes Weberhofer Weberhofer GmbH, Austria, Vienna -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
Johannes Weberhofer wrote:
El 2016-06-16 a las 08:25 +0200, Per Jessen escribió:
After a bit of trial&error, it turns out it is the opening/reading of 201 files that takes up the time. I have reproduced the numbers outside of the apache/php environment. Granted, I do find reading 200 files for a single request somewhat excessive, but with disk attached directly, it's done in 500ms, which is acceptable.
IMHO, it is bad design for a web server having to open that many files for a single query.
I would tend to agree, but I want to look at wordpress and wikimedia to compare. Also, both filesystem caching and application caching should help alleviate the symptoms.
Did you try APC?
Not yet, but it's my list of things to do. It looks interesting. For now, the developer is upgrading processwire and enabling template caching.
A compiler, that runs once, needs to read a ton of sources and includes, Ok. But something serving many clients and has to be fast, can not do that. IMO, it should work mostly on RAM.
With proper caching, it will.
I think, that the PHP7 will solve that situation as it's per default caching all the compiled sources in it's separate local cache.
It'll probably be quite some time before we move to PHP7 :-) -- Per Jessen, Zürich (15.5°C) http://www.dns24.ch/ - your free DNS host, made in Switzerland. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 06/16/2016 02:25 AM, Per Jessen wrote:
One thing to look into is - why aren't those files being cached? That would appear to be because the device is network attached. I am still working on this.
Cached where? I've mentioned the kernel caching inodes and pathname segments, a feature that has been around for yonks. But the file contents are accessed in the application. Now don't forget that web applications are not telnet session, the logic is that they are connectionless (unless you have, for example a java/JavaScript applet doing things). Open, download page, close. Client renders. Web front end fires up the application, it does its stuff and exits. That's the basic model. Yes there are ways to amplify that; using a sql database as a data store is one, the database is long lived, and does its own caching. There are tricks that can be performed on a dedicated server with lots of memory that keep not only the interpreter in core across transactions but increase the latency of the data files in the paging system. While this makes sense in heavily provisioned servers, its not practical in a desktop. Another optimization the BigName sites use is to have a separate server for images, CSS and JavaScript, and since those get used by all calls that machine can be set to leave these files in core and served up very fast across transactions from many clients. So comparing a single server with the BigName sites like Wikipedia isn't fair. And don't forget that some client applications also do file caching. There are also tricks a web application developer can play with the Apache (or other) server to keep a connection open for a few milliseconds. This makes sense, sometimes, if a single client is going to make another call, which is possible with some embedded apps/scriplets. But most of these optimizations are 'statistical', that is they make sense for a multi-user server being hit from many clients. The points I’m making here are - there are many types of caching going on and there’s nothing to say they aren't running interference with one another - when you say "caching", be more specific - the optimizations which make sense for a heavily traffics server don't apply in the small scale Finally, I'd point out that the MediaWiki code used by Wikipedia is available for inspection. If there's magic smoke in there you can see it. I've not looked at or for the Wordpress code but its there. As is, I should mention, an incredible amount of stuff on the optimization of web services and web servers, some of which I've used myself, some of which I've worked with for clients and lot of which you'll find in books on site development. Mine lead towards Ruby and erl with few Python ones. I don't have any texts on PHP. -- A: Yes. > Q: Are you sure? >> A: Because it reverses the logical flow of conversation. >>> Q: Why is top posting frowned upon? -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
Anton Aylward wrote:
On 06/16/2016 02:25 AM, Per Jessen wrote:
One thing to look into is - why aren't those files being cached? That would appear to be because the device is network attached. I am still working on this.
Cached where?
Locally, in RAM. Like any other filesystem. Afaict, filesystem cache doesn't work for filesystems on iSCSI, maybe due to the access semantics or some such. NFSv4 seems to be cached though. NFSv4 also seems to be quite a bit faster than iSCSI. Will have to investigate.
So comparing a single server with the BigName sites like Wikipedia isn't fair.
That was a typo, s/Wikipedia/wikimedia/. I wasn't comparing to other sites, just other apps running on the same server.
Finally, I'd point out that the MediaWiki code used by Wikipedia is available for inspection. If there's magic smoke in there you can see it. I've not looked at or for the Wordpress code but its there.
Sure, it's all there, also the processwire code, but there's little point in looking much at it when I want to fix the environment, not the application. For starters, the developer has now enabled caching in processwire, which has reduced processing time by 75%, that's good enough. Maybe they'll reduce the images too and the whole thing becomes quite workable :-) -- Per Jessen, Zürich (17.1°C) http://www.hostsuisse.com/ - virtual servers, made in Switzerland. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 06/16/2016 10:47 AM, Per Jessen wrote:
Anton Aylward wrote:
On 06/16/2016 02:25 AM, Per Jessen wrote:
One thing to look into is - why aren't those files being cached? That would appear to be because the device is network attached. I am still working on this.
Cached where?
Locally, in RAM. Like any other filesystem. Afaict, filesystem cache doesn't work for filesystems on iSCSI, maybe due to the access semantics or some such. NFSv4 seems to be cached though. NFSv4 also seems to be quite a bit faster than iSCSI. Will have to investigate.
Please do. It seems backwards to me. Yes, NFS and iSCSI have different semantics as far as 'sharing' goes. The kernel knows about when its using client-side NFS and that its not a file system like ext4/BtrFs/ReiserFS no matter what the server thinks is on its disk. Local filesystems like those have their own driver-level caching for inodes, never mind the system inode cache. How tunable they are is another matter :-) https://www.usenix.org/legacy/events/fast04/tech/full_papers/radkov/radkov_h... I've not used iSCSI and don't plan to.
So comparing a single server with the BigName sites like Wikipedia isn't fair.
That was a typo, s/Wikipedia/wikimedia/. I wasn't comparing to other sites, just other apps running on the same server.
Ah, right. Have you looked into such things as whether those two packages use things like "memcache()"?
Finally, I'd point out that the MediaWiki code used by Wikipedia is available for inspection. If there's magic smoke in there you can see it. I've not looked at or for the Wordpress code but its there.
Sure, it's all there, also the processwire code, but there's little point in looking much at it when I want to fix the environment, not the application.
For starters, the developer has now enabled caching in processwire,
What kind of caching is that?
which has reduced processing time by 75%, that's good enough. Maybe they'll reduce the images too and the whole thing becomes quite workable :-)
I not that at https://blog.crazyegg.com/2013/12/11/speed-up-your-website/ it mentions that the client can cache images, but its still worth while to re-size the images on the store so that the server sends images that the client doesn’t have to spend time resizing. There's another advantage to this: smaller images also take up less client-side cache.That means they are more likely to be retained. -- A: Yes. > Q: Are you sure? >> A: Because it reverses the logical flow of conversation. >>> Q: Why is top posting frowned upon? -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
Anton Aylward wrote:
On 06/16/2016 10:47 AM, Per Jessen wrote:
Anton Aylward wrote:
On 06/16/2016 02:25 AM, Per Jessen wrote:
One thing to look into is - why aren't those files being cached? That would appear to be because the device is network attached. I am still working on this.
Cached where?
Locally, in RAM. Like any other filesystem. Afaict, filesystem cache doesn't work for filesystems on iSCSI, maybe due to the access semantics or some such. NFSv4 seems to be cached though. NFSv4 also seems to be quite a bit faster than iSCSI. Will have to investigate.
Please do. It seems backwards to me.
Yeah, I'm puzzled too. There are studies out there that essentially say iSCSI and NFSv4 now perform roughly the same, but that's clearly not my experience.
So comparing a single server with the BigName sites like Wikipedia isn't fair.
That was a typo, s/Wikipedia/wikimedia/. I wasn't comparing to other sites, just other apps running on the same server.
Ah, right.
Have you looked into such things as whether those two packages use things like "memcache()"?
Not in this context, but afaik, the application needs to be aware, i.e. actively engage with memcached.
For starters, the developer has now enabled caching in processwire,
What kind of caching is that?
I have not looked at it in detail, my understanding is that it means generated pages are cached which makes a lot of sense. Seems to be the only sensible mode for production use :-)
which has reduced processing time by 75%, that's good enough. Maybe they'll reduce the images too and the whole thing becomes quite workable :-)
I not that at https://blog.crazyegg.com/2013/12/11/speed-up-your-website/ it mentions that the client can cache images, but its still worth while to re-size the images on the store so that the server sends images that the client doesn’t have to spend time resizing.
Some of the background jpegs on this site have been reduced in dimensions, but they're still 1Mb and more. By just increasing compression, the size could easily be reduced by 80-90%. -- Per Jessen, Zürich (15.5°C) http://www.hostsuisse.com/ - dedicated server rental in Switzerland. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
participants (4)
-
Anton Aylward -
Carlos E. R. -
Johannes Weberhofer -
Per Jessen