On 10/9/18 4:04 AM, Alberto Planas Dominguez wrote:

On Monday, October 8, 2018 7:17:08 PM CEST Robert Schweikert wrote:

...

On 10/8/18 12:07 PM, Alberto Planas Dominguez wrote:

...

[Dropping a very unproductive content]

<snip>

...

...

Is in my understanding that CPU cost is cheaper in relation with network transfer and storage. Can we measure the savings of network and storage here? Not in the Public Cloud, there is no data. Network data into the framework is always free and the size of the root volumes in our images is already 10GB (30GB in Azure) and thus offers up ample space for the pyc files. Meaning there is no gain if the actual disk space used by the packages we install is smaller and we have more empty space in the 10GB (30GB in Azure) image.

So uploading ISO / qcow2 images and storing them for a long period is free?

No, that is not free. However, I think there is some intermingling of concepts occurring. Let me try and explain by example. A user starts and instance from a SUSE provided image. The size of the image is 10GB in EC2 and GCE. The customer has no cost for storage of the root volume or any network data transferred into the instance during update, i.e. "zypper up" does not generate any cost for the customer even if the customer pulls packages from OBS or SCC. The cost is for the instance, i.e. CPU and memory cost. Also given that Public Cloud frameworks generally have a really good network connection and most instances have a decent network speed the size of the rpm that is being pulled in is not much of a concern. This is the case I had in mind when I provided the example calculation of how a "on first use pyc generation" strategy can cost a user money. In this second example the user generates their own image and uploads it. In this case the user is responsible for the cost of the root volume storage. In AWS for ssd based storage this amounts to $0.1 per GB per month. So if a user can save 1 GB of storage because there are no pyc files in their image and they make their image exactly the size they need they stand the potential of saving 10 cents a month or $1.2 over the course of a year. If I go back to the example with 2000 test instances, from my point of view, it is apparent that the cost savings potential by optimizing for size are significantly smaller when compared to the cost increase potential due to increased CPU use. So this scenario, IMHO, would also favor the use case where we can, in some way shape or form have images that contain the pyc files.

[...]

...

...

Or better, the user can add a `python -m compileall` in the kiwi config.sh, that will populate /var/cache for the cloud images only.

Well, I could turn around and state that this is a "hack" and "...we don't want any of your proposed hacks.." in our image creation process. Hopefully sounds familiar.... ;)

Indeed, the hack is breaking the content of the RPM removing files from the places where zypper put in (/usr). There is no hack making /var/cache hot before the service runs :-)

...

Anyway, on a more serious note, if we can resolve the security concerns and properly handle the upgrade mechanism while not generating multiple packages I am not categorically opposed to such an addition in our Public Cloud image builds.

Thanks!

I see the security problem, indeed. I will work to provide an answer to this problem and propose it here.

Meanwhile the image from [1] is providing some of the pieces that I talk in the first email. This is the image that I am using for another project related with Salt, but as today:

* Python 3.7 is installed (in TW we still have 3.6) * Python 3.7 contains the patch from 3.8 to enable storage of pycache in a different file system * I added two shim loaders (written in shell), that replace python3.7 and python3.7m, to enable the 3.8 feature * As a hack, I removed all the pycache from the ISO (I know, I know ...), so all is generated under demand on /var/cache

[1] https://build.opensuse.org/project/monitor/home:aplanas:Images? arch_x86_64=1&defaults=0&repo_images=1&succeeded=1

Over the next couple of days I will try to find the time to generate a SLES 15 image and clear the py{c,o} and __pycache__ and get it uploaded to EC2 so we can get a number for "cold start" for cloud-init. Later, Robert -- Robert Schweikert MAY THE SOURCE BE WITH YOU Distinguished Architect LINUX Team Lead Public Cloud rjschwei@suse.com IRC: robjo