Mailinglist Archive: opensuse-buildservice (123 mails)

< Previous Next >
Re: [opensuse-buildservice] Proposal to use data_migrate gem for API
Adrian, thanks for raising these concerns. Let me just explain a use case of data migrations:

We have an event_payload column which is serialised in YAML in the notifications table, we need to change this to be serialised in JSON to fix issue #3638 <https://github.com/openSUSE/open-build-service/issues/3638> .

To do this we need a script to convert the serialisation of existing notifications from YAML to JSON, but we only want to run this once. We could create a rake task to do it, but then there is the problem that we need to add that rake task to the update guide so upgrading becomes more complicated and also all other developers will need to run this rake task to get their database up to date too.

The other option is to use a normal database migration to handle the conversion from yaml to json. However then we would have a database migration which does not change the database structure at all. Also it means that we have to include the database structure changes (which require downtime) with the data changes (which might not require downtime) so for some data changes which might take a long time (i.e. > 1hour) that would be difficult to deploy if there were also database migrations that needed running. (This is precisely the case when changing the serialisation of project_log_entires, the script to do that will take at least a couple hours because there are ~3million rows in the project_log_entries table).

See this stackoverflow: Rails migration: only for schema change or also for updating data? <https://stackoverflow.com/questions/19387440/rails-migration-only-for-schema-change-or-also-for-updating-data>

On 05/09/17 16:42, Adrian Schröter wrote:

I don't get this exactly, does this mean that there is no single
command to update all data to current state?
Please keep in mind the distinction between database and data, in this context by "database" i mean the database structure. And "data" means the content of the rows in tables. So the command to get the database up to date is (as it always has been) `rake db:migrate`. This gem now gives us a new command to get the data up to date (which we didn't have before) which is `rake data:migrate`.

You always have to know lot's of special commands when updating
from 2.8 to 2.9 documented in the README files?
No, thats one of the main reasons to use this gem, it gives you the `rake data:migrate` so that you don't have to know lots of special commands when updating. It even gives you this rake task which performs both the database and data migrate commands:

`rake db:migrate:with_data`

Does this also mean that it breaks our update tests in CI and auto
deployment?
It won't break any CI stuff because this is only for existing obs instances with populated databases. If you're creating a new obs instance from scratch then this rake task is not necessary. I'm not sure what "auto deployment" is but the deployment process in the wiki will need to be changed to make sure that we run `rake db:migrate:with_data` instead of just running `rake db:migrate`.

--
Evan Rolfe
Full Stack Web Developer
SUSE Linux GmbH, Maxfeldstr. 5, D-90409 Nürnberg
Tel: +49-911-74053-0; Fax: +49-911-7417755;https://www.suse.com/
SUSE Linux GmbH, GF: Felix Imendörffer, Jane Smithard,
Graham Norton, HRB 21284 (AG Nürnberg)

< Previous Next >
Follow Ups