Mailinglist Archive: opensuse-buildservice (123 mails)

< Previous Next >
Re: [opensuse-buildservice] Proposal to use data_migrate gem for API
On Mittwoch, 6. September 2017, 10:02:25 CEST wrote Evan Rolfe:
Adrian, thanks for raising these concerns. Let me just explain a use
case of data migrations:

We have an event_payload column which is serialised in YAML in the
notifications table, we need to change this to be serialised in JSON to
fix issue #3638
<https://github.com/openSUSE/open-build-service/issues/3638> .

To do this we need a script to convert the serialisation of existing
notifications from YAML to JSON, but we only want to run this once. We
could create a rake task to do it, but then there is the problem that we
need to add that rake task to the update guide so upgrading becomes more
complicated and also all other developers will need to run this rake
task to get their database up to date too.

The other option is to use a normal database migration to handle the
conversion from yaml to json. However then we would have a database
migration which does not change the database structure at all. Also it
means that we have to include the database structure changes (which
require downtime) with the data changes (which might not require
downtime) so for some data changes which might take a long time (i.e. >
1hour) that would be difficult to deploy if there were also database
migrations that needed running. (This is precisely the case when
changing the serialisation of project_log_entires, the script to do that
will take at least a couple hours because there are ~3million rows in
the project_log_entries table).

Okay, but I don't see this mixture as a big problem, because

* Updaters from OBS 2.8.x to 2.9.x need a down time anyway

* Updaters like us, who run on git master should follow the migrations
and understand the nature. We can still run this migration in parallel
and without downtime. We just need to ensure that there isn't another
migration afterwards, which requires a downtime, right?

Btw, we used to do even structural changes without downtime, if we know
that the old code won't cause problems (eg. when just adding a new column
which does not harm).

On the other side, I would like to keep to steps for updating as small
as possible to the user.

So I am still very much in favor to do this with our standard migrations,
if you don't see a problem in my points above.

good morning :)
adrian

See this stackoverflow: Rails migration: only for schema change or also
for updating data?
<https://stackoverflow.com/questions/19387440/rails-migration-only-for-schema-change-or-also-for-updating-data>

On 05/09/17 16:42, Adrian Schröter wrote:

I don't get this exactly, does this mean that there is no single
command to update all data to current state?
Please keep in mind the distinction between database and data, in this
context by "database" i mean the database structure. And "data" means
the content of the rows in tables. So the command to get the database up
to date is (as it always has been) `rake db:migrate`. This gem now gives
us a new command to get the data up to date (which we didn't have
before) which is `rake data:migrate`.

You always have to know lot's of special commands when updating
from 2.8 to 2.9 documented in the README files?
No, thats one of the main reasons to use this gem, it gives you the
`rake data:migrate` so that you don't have to know lots of special
commands when updating. It even gives you this rake task which performs
both the database and data migrate commands:

`rake db:migrate:with_data`

Does this also mean that it breaks our update tests in CI and auto
deployment?
It won't break any CI stuff because this is only for existing obs
instances with populated databases. If you're creating a new obs
instance from scratch then this rake task is not necessary. I'm not sure
what "auto deployment" is but the deployment process in the wiki will
need to be changed to make sure that we run `rake db:migrate:with_data`
instead of just running `rake db:migrate`.




--

Adrian Schroeter
email: adrian@xxxxxxx

SUSE Linux GmbH, GF: Felix Imendörffer, Jane Smithard, Graham Norton, HRB 21284
(AG Nürnberg)

Maxfeldstraße 5
90409 Nürnberg
Germany


--
To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@xxxxxxxxxxxx
To contact the owner, e-mail: opensuse-buildservice+owner@xxxxxxxxxxxx

< Previous Next >
Follow Ups