[opensuse-buildservice] How to clean up /srv/obs/sources? - openSUSE Build Service - openSUSE Mailing Lists

newer
[opensuse-buildservice] deploying...

[opensuse-buildservice] How to clean up /srv/obs/sources?

older
[opensuse-buildservice] [PATCH...

Carsten Hoeger

30 Aug 2011 30 Aug '11

09:15

Hi, I just noticed that I have ~6GB in repositories (binary and source rpms/deps), but my obs uses ~600GB on disk... Then I found the culprit. It's the /srv/obs/sources folder: 6.4G repos 521G sources Okay, now how to get rid of the useless old stuff? My disk is again filling up... :-( -- With best regards, Carsten Hoeger

Attachments:

signature.asc (application/pgp-signature — 190 bytes)

Reply

Sign in to reply online Use email software

Show replies by date

doiggl＠velocitynet.com.au

30 Aug 30 Aug

03:45

On Tue, 30 Aug 2011 11:15:46 +0200, Carsten Hoeger wrote:

Hi,

I just noticed that I have ~6GB in repositories (binary and source rpms/deps), but my obs uses ~600GB on disk...

Then I found the culprit. It's the /srv/obs/sources folder:

6.4G repos 521G sources

Okay, now how to get rid of the useless old stuff? My disk is again filling up... :-(

Hello, I use a script called from crontab In this example I clean /tmp/ files greater than 7 days old. # cat delete-tmp-greater7days-old.sh #!/usr/bin/sh echo "remove /tmp/ files older than 7 days after in various locations" /bin/df -h # list files then remove find /tmp/. -mtime +7 -exec ls "{}" \; find /tmp/. -mtime +7 -exec rm -rf "{}" \; echo "used after :" /bin/df -h Note:your lines would read find /srv/obs/sources/. -mtime +7 -exec ls "{}" \; find /srv/obs/sources/. -mtime +7 -exec rm -rf "{}" \; Cheers Glenn -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-buildservice+help@opensuse.org

Reply

Sign in to reply online Use email software

Marcus Meissner

13:20

On Tue, Aug 30, 2011 at 01:45:42PM +1000, doiggl@velocitynet.com.au wrote:

On Tue, 30 Aug 2011 11:15:46 +0200, Carsten Hoeger wrote:

...
Hi,

I just noticed that I have ~6GB in repositories (binary and source rpms/deps), but my obs uses ~600GB on disk...

Then I found the culprit. It's the /srv/obs/sources folder:

6.4G repos 521G sources

Okay, now how to get rid of the useless old stuff? My disk is again filling up... :-(

Hello, I use a script called from crontab In this example I clean /tmp/ files greater than 7 days old.

# cat delete-tmp-greater7days-old.sh

#!/usr/bin/sh echo "remove /tmp/ files older than 7 days after in various locations" /bin/df -h

# list files then remove find /tmp/. -mtime +7 -exec ls "{}" \; find /tmp/. -mtime +7 -exec rm -rf "{}" \;

echo "used after :" /bin/df -h

Note:your lines would read find /srv/obs/sources/. -mtime +7 -exec ls "{}" \; find /srv/obs/sources/. -mtime +7 -exec rm -rf "{}" \;

This would delete all sources after 7 days of inactivity. Probably not what is wanted... ;) Ciao, Marcus -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-buildservice+help@opensuse.org

Reply

Sign in to reply online Use email software

Carsten Hoeger

13:30

On Tue, Aug 30, Marcus Meissner wrote:

...
...
I just noticed that I have ~6GB in repositories (binary and source rpms/deps), but my obs uses ~600GB on disk...

Then I found the culprit. It's the /srv/obs/sources folder:

6.4G repos 521G sources

Okay, now how to get rid of the useless old stuff? My disk is again filling up... :-(

Hello, I use a script called from crontab In this example I clean /tmp/ files greater than 7 days old.

# cat delete-tmp-greater7days-old.sh

#!/usr/bin/sh echo "remove /tmp/ files older than 7 days after in various locations" /bin/df -h

# list files then remove find /tmp/. -mtime +7 -exec ls "{}" \; find /tmp/. -mtime +7 -exec rm -rf "{}" \;

echo "used after :" /bin/df -h

Note:your lines would read find /srv/obs/sources/. -mtime +7 -exec ls "{}" \; find /srv/obs/sources/. -mtime +7 -exec rm -rf "{}" \;

This would delete all sources after 7 days of inactivity.

Probably not what is wanted... ;)

Hehe, no, not really. What I'd love to keep is: Just the files, that belong to what I still have in my published repos. Does anybody have some related sql queries which I could place into a script? Of course I will share the results. -- With best regards, Carsten Hoeger

Reply

Sign in to reply online Use email software

Adrian Schröter

13:42

Am Dienstag, 30. August 2011, 15:30:20 schrieb Carsten Hoeger:

On Tue, Aug 30, Marcus Meissner wrote:

...
...
...
I just noticed that I have ~6GB in repositories (binary and source rpms/deps), but my obs uses ~600GB on disk...

Then I found the culprit. It's the /srv/obs/sources folder:

6.4G repos 521G sources

Okay, now how to get rid of the useless old stuff? My disk is again filling up... :-(

Hello, I use a script called from crontab In this example I clean /tmp/ files greater than 7 days old.

# cat delete-tmp-greater7days-old.sh

#!/usr/bin/sh echo "remove /tmp/ files older than 7 days after in various locations" /bin/df -h

# list files then remove find /tmp/. -mtime +7 -exec ls "{}" \; find /tmp/. -mtime +7 -exec rm -rf "{}" \;

echo "used after :" /bin/df -h

Note:your lines would read find /srv/obs/sources/. -mtime +7 -exec ls "{}" \; find /srv/obs/sources/. -mtime +7 -exec rm -rf "{}" \;

This would delete all sources after 7 days of inactivity.

Probably not what is wanted... ;)

Hehe, no, not really.

What I'd love to keep is: Just the files, that belong to what I still have in my published repos.

Does anybody have some related sql queries which I could place into a script? Of course I will share the results.

You don't need SQL for that, best is to write some perl code parsing all .rev files and compare that with stored files. And in best case another tool, which allows you to short the history before. -- Adrian Schroeter SUSE Linux Products GmbH email: adrian@suse.de -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-buildservice+help@opensuse.org

Reply

Sign in to reply online Use email software

Jeff_Glanz＠Dell.com

14:01

I've attached a tarball of a source cleaner that reads rev files, marks containers & source files for deletion. It leverages OBS services. Jeff Glanz Dell | PG Release Engineering Team office + 1 512 724 9509 -----Original Message----- From: Adrian Schröter [mailto:adrian@suse.de] Sent: Tuesday, August 30, 2011 8:42 AM To: opensuse-buildservice@opensuse.org Cc: Carsten Hoeger Subject: Re: [opensuse-buildservice] How to clean up /srv/obs/sources? Am Dienstag, 30. August 2011, 15:30:20 schrieb Carsten Hoeger:

On Tue, Aug 30, Marcus Meissner wrote:

...
...
...
I just noticed that I have ~6GB in repositories (binary and source rpms/deps), but my obs uses ~600GB on disk...

Then I found the culprit. It's the /srv/obs/sources folder:

6.4G repos 521G sources

Okay, now how to get rid of the useless old stuff? My disk is again filling up... :-(

Hello, I use a script called from crontab In this example I clean /tmp/ files greater than 7 days old.

# cat delete-tmp-greater7days-old.sh

#!/usr/bin/sh echo "remove /tmp/ files older than 7 days after in various locations" /bin/df -h

# list files then remove find /tmp/. -mtime +7 -exec ls "{}" \; find /tmp/. -mtime +7 -exec rm -rf "{}" \;

echo "used after :" /bin/df -h

Note:your lines would read find /srv/obs/sources/. -mtime +7 -exec ls "{}" \; find /srv/obs/sources/. -mtime +7 -exec rm -rf "{}" \;

This would delete all sources after 7 days of inactivity.

Probably not what is wanted... ;)

Hehe, no, not really.

What I'd love to keep is: Just the files, that belong to what I still have in my published repos.

Does anybody have some related sql queries which I could place into a script? Of course I will share the results.

You don't need SQL for that, best is to write some perl code parsing all .rev files and compare that with stored files. And in best case another tool, which allows you to short the history before. -- Adrian Schroeter SUSE Linux Products GmbH email: adrian@suse.de -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-buildservice+help@opensuse.org

Reply

Sign in to reply online Use email software

Carsten Hoeger

14:45

On Tue, Aug 30, Jeff_Glanz@Dell.com wrote:

I've attached a tarball of a source cleaner that reads rev files, marks containers & source files for deletion. It leverages OBS services.

Great!!! Thank you very much!

Jeff Glanz Dell | PG Release Engineering Team office + 1 512 724 9509

-----Original Message----- From: Adrian Schröter [mailto:adrian@suse.de] Sent: Tuesday, August 30, 2011 8:42 AM To: opensuse-buildservice@opensuse.org Cc: Carsten Hoeger Subject: Re: [opensuse-buildservice] How to clean up /srv/obs/sources?

Am Dienstag, 30. August 2011, 15:30:20 schrieb Carsten Hoeger:

...
On Tue, Aug 30, Marcus Meissner wrote:

...
...
...
I just noticed that I have ~6GB in repositories (binary and source rpms/deps), but my obs uses ~600GB on disk...

Then I found the culprit. It's the /srv/obs/sources folder:

6.4G repos 521G sources

Okay, now how to get rid of the useless old stuff? My disk is again filling up... :-(

Hello, I use a script called from crontab In this example I clean /tmp/ files greater than 7 days old.

# cat delete-tmp-greater7days-old.sh

#!/usr/bin/sh echo "remove /tmp/ files older than 7 days after in various locations" /bin/df -h

# list files then remove find /tmp/. -mtime +7 -exec ls "{}" \; find /tmp/. -mtime +7 -exec rm -rf "{}" \;

echo "used after :" /bin/df -h

Note:your lines would read find /srv/obs/sources/. -mtime +7 -exec ls "{}" \; find /srv/obs/sources/. -mtime +7 -exec rm -rf "{}" \;

This would delete all sources after 7 days of inactivity.

Probably not what is wanted... ;)

Hehe, no, not really.

What I'd love to keep is: Just the files, that belong to what I still have in my published repos.

Does anybody have some related sql queries which I could place into a script? Of course I will share the results.

You don't need SQL for that, best is to write some perl code parsing all .rev files and compare that with stored files.

And in best case another tool, which allows you to short the history before.

-- Adrian Schroeter SUSE Linux Products GmbH email: adrian@suse.de

-- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-buildservice+help@opensuse.org

-- With best regards, Carsten Hoeger

Reply

Sign in to reply online Use email software

Carsten Hoeger

31 Aug 31 Aug

12:54

Hi Jeff,

I've attached a tarball of a source cleaner that reads rev files, marks containers & source files for deletion. It leverages OBS services.

It looks like it only deletes a very small fraction of stale files within the source folder (and took about half a day to figure that out ;-))... 129M from ~550G is not much. I started it using -preview and replaced the {$projectGlob} with "*" in the code. Does anybody have at least a basic documentation on how that .rev/.del/.mrev stuff works? I have the suspicion that it isn't possible at all to determine all unused sources. -- With best regards, Carsten Hoeger

Reply

Sign in to reply online Use email software

4633

Age (days ago)

4634

Last active (days ago)

Download

7 comments

5 participants

tags

participants (5)

Adrian Schröter
Carsten Hoeger
doiggl＠velocitynet.com.au
Jeff_Glanz＠Dell.com
Marcus Meissner