On Fri, Jun 22, Jiri Srain wrote:

Dne p??tek 22 ??erven 2007 14:30 Michael Andres napsal(a):

...

On Fri, Jun 22, Jiri Srain wrote:

...

Hi!

In order to allow workflows bound to patterns we need to extend the pattern definition on the media as well as the database and in-memory representation of a pattern. We need following tags:

"workflow_file" : string // file describing the workflow on the media "workflow_file_checksum" : the same as other checksums "workflow_order" : string // to allow ordering, string for better extendibility

(the names can be adapted to fit attribute naming schema)

The suggestions for the SuSEtags metadata format are

=Wkf =WkfCSum (if not specified in content file) =WkfOrd

ZYPP is not required to process this data anyhow (maybe except the checksum?), all that's needed is to deliver the data to YaST. The update of package bindings will be trivial.

- But why putting this into the pattern definition? Why not one (separate) file that maps pattern<->workflow, but spreading it across several locations?

It is an option as well. But thhere must be a binding between a pattern and itw workflow and it must be possible to define it in all our supported metadata formats. That's the only requirement.

...

- workflow_file and workflow_file_checksum must appear in the content file anyway.

Otherwise we had to parse all the patterns to see whether a workflow file changed and we must refresh. And changing the workflow would require reloading the pattern (because of the changed checksum)

Right, but if a new workflow is added, you have to modify the pattern in the database anyway, since the workflow is a part of a pattern the very same way as packages the pattern should drag in.

Only for YaST. Nobody else (e.g. zypper) associates such a workflow with the installation of a pattern. ( BTW: why do you limit the workflow extensions to be used with patterns only? Installation of a pattern is just the trigger to execute the workflow. So why not executing a workflow when a certain Package gets installed? Or a certain Language? Or something is updated, or gets deleted, or whatever event you can imagine... ) This would result in a kind of workflow index, that lists the available workflow, and defines the events that trigger it's execution. Installation of a certain patter could be such an event. Installation of a single package, or a language, or product, could be an event as well. Or even distinguish between install/update/delete. -- cu, Michael Andres +------------------------------------------------------------------+ Key fingerprint = 2DFA 5D73 18B1 E7EF A862 27AC 3FB8 9E3A 27C6 B0E4 +------------------------------------------------------------------+ Michael Andres YaST Development ma@novell.com SUSE LINUX Products GmbH, GF: Markus Rex, HRB 16746 (AG Nuernberg) Maxfeldstrasse 5, D-90409 Nuernberg, Germany, ++49 (0)911 - 740 53-0 +------------------------------------------------------------------+ -- To unsubscribe, e-mail: zypp-devel+unsubscribe@opensuse.org For additional commands, e-mail: zypp-devel+help@opensuse.org

Hi! Please, keep Lukas in CC, he's not on zypp-devel. Dne pátek 22 červen 2007 14:45 Duncan Mac-Vicar Prett napsal(a):

On Friday 22 June 2007 14:00:07 Jiri Srain wrote:

...

Hi! In order to allow workflows bound to patterns we need to extend the pattern definition on the media as well as the database and in-memory representation of a pattern. We need following tags:

I find the idea great, but the implementation propossal is really non-elegant (I don't have a better proposal)

...

"workflow_file" : string // file describing the workflow on the media "workflow_file_checksum" : the same as other checksums

what file is this? the workflow xml file?

Yes. The XML file describing the workflow the pattern brings.

...

"workflow_order" : string // to allow ordering, string for better extendibility

Yeah, I can see developers embeding YCP code in this string later. What is extendibility here then?

...

=Wkf =WkfCSum (if not specified in content file) =WkfOrd

What value does WkfOrd has? I still don't see YaST specific stuff being part of the pattern definition.

...

http://users.suse.cz/~locilka/PatternsAndAddOnsWithWorkflows/proposal-200 7- 06-04.txt

I have one question, I imagine the use case, you have a mail-server pattern, and if it installed you want to start the mail-server workflow.

How are you going to know either to start the postfix cfg workflow or the sendmail one, or [insert any supplied community module here]

The workflow must be defined the way that it will properly run which wizard is needed. It can check which dialogs to run depending on packages which are installed. In your case, if the pattern contains both sendmail and postfix, it should ask user in the first step. Obviously if we ship sendmail and postfix, user selects this pattern but installs eg. qmail, the pattern workflow cannot take care of it. Jiri -- Regards, Jiri Srain YaST Team Leader --------------------------------------------------------------------- SUSE LINUX, s.r.o. e-mail: jsrain@suse.cz Lihovarska 1060/12 tel: +420 284 028 959 190 00 Praha 9 fax: +420 284 028 951 Czech Republic http://www.suse.cz

* Lukas Ocilka [Jun 22. 2007 15:44]:

Jiri Srain wrote:

...

...

...

"workflow_order" : string // to allow ordering, string for better extendibility Yeah, I can see developers embeding YCP code in this string later. What is extendibility here then?

...

=Wkf =WkfCSum (if not specified in content file) =WkfOrd What value does WkfOrd has? I still don't see YaST specific stuff being part of the pattern definition.

First implementation expects only integer value but later one could define any possible dependency (after:pattern:some-other-pattern etc.). The initial proposal wanted just integer value but using a string we could express dependencies and orders across other Add-Ons and Patterns.

Klaus wanted this feature :) and it makes sense for future.

Well, I wanted some means to express dependencies between worflows. But I also expressively rejected(!) any kind of ordering based on numbers but pointed to the dependency mechanism implemented in the lsb start scripts. Klaus -- To unsubscribe, e-mail: zypp-devel+unsubscribe@opensuse.org For additional commands, e-mail: zypp-devel+help@opensuse.org

Duncan Mac-Vicar Prett wrote:

On Friday 22 June 2007 15:23:11 Jiri Srain wrote:

...

The workflow must be defined the way that it will properly run which wizard is needed. It can check which dialogs to run depending on packages which are installed. In your case, if the pattern contains both sendmail and postfix, it should ask user in the first step. Obviously if we ship sendmail and postfix, user selects this pattern but installs eg. qmail, the pattern workflow cannot take care of it.

That exactly my point. How are you going to know which packages a pattern "contains"?. Only the solver knows.

I'm afraid I didn't explain it in an understandable way. * Installation (as well as yast2-addon, sw_single...) will handle whether some patterns were selected to be installed. * Installation has own base-workflow, other workflows can be added by Add-On products and/or Patterns. * Installation doesn't handle the pattern-workflows, doesn't operate with information which packages are selected and what to do with it. It just runs the client defined in workflow, that's all. For instance, OES2 has own installation workflow - the one that extends the default SLES 10 SP1 workflow. Installation doesn't case what OES2's additional clients do, it just runs them.

You would need to do a 2nd stage kind of solving (looking for package names matching) after the solver runs (and you can still not be sure which pattern is the guilty one that selected the package. ), So you are putting more and more logic in the UI and breaking all the improvements from the old selections to patterns.

So as usual we are abusing the solver, the language and the UI/Controller, because at the end a workflow has _nothing_ to do with the pattern itself but with the pattern installation result (specially individual software)

Workflow is assigned to pattern, such as Add-On product can have its own workflow as well.

The correct way to do this would be using dependencies, so if a certain package gets installed, a worflow or special trigger file (part of the module package) is also grabed and installed. The installation detects this and starts the workflow. This only works if the workflow is run after the software is installed, but I guess you can't configure the software if it is not installed.

Workflow for packages was never wanted and might be added later. Currently we support only Add-Ons Products and Patterns. L.

Duncan Mac-Vicar Prett wrote:

On Friday 22 June 2007 15:54:27 Lukas Ocilka wrote:

...

Workflow for packages was never wanted and might be added later. Currently we support only Add-Ons Products and Patterns.

So we do it in a hacky way resolvable kind per resolvable kind. What will you do when we have to support it with packages?

It's actually not a "hacky" way, but the "way of minimal changes". What I wanted was a minimal change to libzypp and pkg-bindings just to have the wanted feature. The only changes I need are: * Support =Wkf: in libzypp and pkg-binding * Support =WkfOrd: in libzypp and pkg-bindings We wanted it in 10.3, a super-generic workflow handling might work nice too but that wouldn't be before 11.0.

It has to be a clean way to trigger the workflow after *any* event has happened. As I said the idea is great and can improve user experience, but you are relating workflow to patterns (which have no relation, because you never know what a pattern adds) and because you are doing a second "solving" at YaST level (very similar to how selections were handled before)

The wery same way as, e.g., running a 'yast2 http-server' does, checks whether required packages are installed.

For example, I gave the example of the mail server. The solution is to see what package did the pattern install, that is a second (simpler solve), instead of letting the package itself trigger a workflow.

Languages? If a special language is installed, you might want to trigger a workflow to configure fonts and input methods.

A Pattern can install another pattern, and another pattern, and so on. So a pattern by itself can't trigger the workflow (except if it pulls the workflow by itself).

I see your point, it's really interesting, but is libzypp-team able to do it now? It would also need implementing special callbacks in pkg-bindings. Please, talk to Klaus first. Lukas

Michael Andres wrote:

On Fri, Jun 22, Lukas Ocilka wrote:

...

...

On Friday 22 June 2007 15:54:27 Lukas Ocilka wrote:

...

Workflow for packages was never wanted and might be added later. Currently we support only Add-Ons Products and Patterns. So we do it in a hacky way resolvable kind per resolvable kind. What will you do when we have to support it with packages? It's actually not a "hacky" way, but the "way of minimal changes". What I wanted was a minimal change to libzypp and pkg-bindings just to have

Duncan Mac-Vicar Prett wrote: the wanted feature.

IMO "No": it's not the "way of minimal changes"

It's the way of minimal changes in libzypp. When I asked some libzypp developers how much work they had, they replied that I couldn't expect more changes than a minor ones because major parts of code were being rewritten.

You already have the workflow xml, the workflow is extensible, the only thing left is:

When to execute which workflow?

Workflows can be added and they can be, of course, removed later. Imagine five Add-On products with their own workflows which you add one by one and then remove the first, the third and the fifth ones. You can play with going back and forth through the installation from "Add-Ons" to "Installation Proposal" - it means that there is no "one particular" place when workflows are just merged. To present even a worse case: Add-On workflow is merged immediately on leaving the Add-Ons dialog, some workflow changes can be done just after that dialog (removing timezone, adding some other dialogs etc.). This all is already solved by a WorkflowManager module that stores the main and additional workflows and merges the additional workflows one by one from scratch when some workflow is removed. One of the main solutions of $this feature is the unification with Add-Ons and their workflows. Please, read the article mentioned in this thread first. There is only a little difference between Add-On and Pattern workflows.

- Provide an index file mapping 'workflow->selection'.

- Instead of looping through the patterns to be installed, and asking them for an associated workflow,

loop through the patterns to be installed, and look whether they appear in your workflow index.

Libzypp shouldn't decide where, when and how to merge a pattern workflow WorkflowManager needs to merge it. Partly because not only installation, but also "add-on" and "sw_single" on a running system should be supported. And sw_single, for instance, has no workflow.

That way, neither Pattern metadata generation (abuild), nor Pattern parsing (zypp), nor the cache database (zypp), are affected.

I can't see the point here. Patterns needs to be generated anyway, Pattern workflow as well. Add-On and Pattern workflows have the very same format as control.xml file - they can change every single feature that control.xml defines. Cache database is affected only when some workflow is added - not changed. Workflow files are downloaded using Pkg::SourceProvide(Optional)File().

And you can extend your feature anytime, without affecting abuild/zypp.

I've asked for use-cases many times, there is no single use-case even for this feature, I don't see any reason for extending it in a near future (but ordering).

Yes, you need one parser for your index file, but thas's IMO cheaper than the current plan.

Actually, I could really omit using libzypp and create a file that links patterns with a specific workflow and its presentation order ... but what's the difference between let's say pattern_order (useful for packagemenager) and pattern_workflow_order (for workflowmanager)? Moreover, the file would need to be cached because of CD/DVD-based Add-Ons, additionally a refresh in a WorkflowManager would need to be called for online repos. Where is the advantage of doing things twice when we have library, that already doe's the needed task? If it is too much work, I can, of course hack everything in YCP layer which is pretty slower than libzypp. Short story: One or two weeks ago, a new QA member came to you office and asked how to parse the rpm's command output effectively, how to find out am rpms that contain all the needed libraries etc. He just wanted to write libzypp solver in Bash. Isn't that funny? L. -- To unsubscribe, e-mail: zypp-devel+unsubscribe@opensuse.org For additional commands, e-mail: zypp-devel+help@opensuse.org

Duncan Mac-Vicar Prett wrote:

On Friday 22 June 2007 15:54:27 Lukas Ocilka wrote:

...

Workflow for packages was never wanted and might be added later. Currently we support only Add-Ons Products and Patterns.

So we do it in a hacky way resolvable kind per resolvable kind. What will you do when we have to support it with packages?

It has to be a clean way to trigger the workflow after *any* event has happened. As I said the idea is great and can improve user experience, but you are relating workflow to patterns (which have no relation, because you never know what a pattern adds) and because you are doing a second "solving" at YaST level (very similar to how selections were handled before) ...

Hi, I was in NBG for a short while during the HackWeek where I asked Duncan to clarify his (and others' ideas) about the proposed solution. Finally he persuaded me that his (their) solution is better. Anyway, it it more than I ever expected from libzypp to do, that's why my proposal was about just extending the .pat files. So, about the current solution: Every single resolvable can have its own workflow (pattern, language, product, package...). All these pieces of information also with the order of workflows are written into single XML file for one product (URL). Libzypp parses the file and if any resolvable having its own workflow is either selected or unselected to be installed or removed from the pool, libzypp useses a callback to YaST via package-bindings. YaST then operates with the workflow itself and libzypp doesn't care. What is it good for? - It solves all possible resolvables at once - Callbacks can be redefined without touching libzypp - YaST doesn't need to use any hacks for querying which resolvables were changes and which of them have a workflow etc. See the attached proposal (changed and old one) or see this link: http://users.suse.cz/~locilka/PatternsAndAddOnsWithWorkflows/ Anyway, we need a manager's blessing for such change of proposal :) because now more things would be implemented in libzypp then I first expected. Thanks & Bye Lukas -- Lukas Ocilka, YaST Developer (xn--luk-gla45d) ----------------------------------------------------------------- SUSE LINUX, s. r. o., Lihovarska 1060/12, Praha 9, Czech Republic FATE #129: Framework for pattern based Installation/Deployment ============================================================== About ----- This new feature is comparable to workflows defined by Add-On products. Currently, any Add-On used in installation can influence the installation workflow by adding some new installation workflow steps, removing old ones, or other modifications in installation settings (propose firewall to be enabled by default...). Moreover Add-On workflow can be defined for case when Add-On product is added later on a running system. The main difference of this feature to Add-Ons is that it is about patterns which can be also part of some Add-Ons. Patterns are sets of required and optional RPM packages with additional information such as supported architecture, dependencies on other resolvables. Products have usually some patterns defined (Add-Ons are also products). Notice: After discussing with Libzypps (mostly Duncan), the plan has changed in the way that not only patterns but also packages, languages, ... can have their own workflow through an unified API (callbacks from libzypp). Anyway, this document later on describes only patterns and their workflows but keep in mind that the other resolvables are covered as well. Expected Result --------------- * It should be possible to modify a workflow (installation/update/...) by patterns. * This new feature should not be in conflict with the current solution of Add-On workflows. Patterns vs. Add-Ons -------------------- * Patterns can influence the installation workflow the very same way as Add-On workflows do * that's why Patterns should also use the same workflow definition as Add-On Products (the very same XML format) * and also why Patterns and Add-Ons should use the very same functions for that (maximal unification) * Workflow of patterns, languages, packages ... will be adjusted via libzypp callbacks Involved Parts of the System ---------------------------- - LibZYPP (data) - YaST - Add-Ons (installation, running system) - Installation / Upgrade (workflow) - Packager (sw_single) - Pkg-Bindings (to get data from LibZYPP) Document mainly speaks of handling Patterns but Add-Ons will be handled the very same or similar way... Supported Scenarios ------------------- 1.) Product installation from media, network, etc... - Can use several Add-On product, also with own workflow and/or patterns - Can use base-product patterns 2.) Add-On Product installation on a running system (`yast2 add-on`) - Can use own workflow If hasn't own workflow, `yast2 sw_single` is called - Can use own patterns 3.) Common Software installation (`yast2 sw_single`) - Can use patterns Each installation type needs a special handling. Some scenarios will be unified at a high level of functional API (sw_single). All scenarios will be, at least, unified at low-level functional API. Patterns Scenarios (technical proposal) --------------------------------------- The symbol '$[x]' means: workflow definition including workflow order. 1.) Installation These two callbacks can be called from libzypp: * AddResolvableWorkflow (resolvable_type, resolvable_name, src_id, $[x]); * RemoveResolvableWorkflow (resolvable_type, resolvable_name, src_id, $[x]); At the end of the installation proposal, workflows are merged into one which is then executed. 2.) Software Management (Installation / Removing) This is `yast2 sw_single` on a running system. The same two callbacks can be called here as well: * AddResolvableWorkflow (...); * RemoveResolvableWorkflow (...); After the packages, patterns, ... are installed, sw_single checks whether there are some workflows to be executed and adjusts the workflow wizard if needed. 3.) Installation of an Add-On Product on a Running System 3.1.) Add-On Product Installation with Own Workflow Some Add-On products can have their own installation workflow defined in installation.xml (stage: normal, mode: installation). They are handled this way: * Add Add-On Product as a normal installation source * Call defined installation workflow Patterns can be selected during the Add-On installation workflow run, the solution is also simple: The same callbacks can be called here too. At the end, they will be merged into one workflow and the workflow executed. 3.2.) Add-On Product Installation without Own Workflow These Add-On products don't have any own workflow. They are handled this way: * Add Add-On Product as a normal installation source * Call sw_single That's why we can solve them the very same way as the case 2.) Software Management Sorting of Additional Workflows ------------------------------- Sorting means the order in which these patterns or add-ons workflows are presented to users. Some kind of sorting can be done even in these workflows themselves if one depends on another. This solution should cover both patterns and add-ons workflows * Add-Ons An add-on's workflow, which is added first, is also merged first. If some add-on is added by a dependency in the base-add-on, it's also added later which also means merged later. * Patterns The least difficult order method is an integer value for each pattern that has a workflow. At first, patterns would be sorted by the Add-On which they belong to (At first, all add-ons are merged, then all patterns from the first add-on, then all patterns from the second add-on...). Second, by that integer variable (the lowest number the earlier they appear). A missing order number would mean a random order (actually, an order how pkg-bindings return them). Example of Sorting ------------------ This is what we have: * Base Product - Base Workflow * 1st Add-On - Add-On 1 Workflow - Pattern 1A, Order 35 - Pattern 1C, Order 6 - Pattern 1R, Order 355 * 2nd Add-On - Add-On 2 Workflow - Pattern 2A, Order 6 - Pattern 2F, Order 211 - Pattern 2G, Order 2 Simple solution: (Patterns are sorted, firstlty by add-ons, secondly by pattern-order) * Base Workflow + Add-On 1 Workflow + Add-On 2 Workflow + Pattern 1C, Order 6 + Pattern 1A, Order 35 + Pattern 1R, Order 355 + Pattern 2G, Order 2 + Pattern 2A, Order 6 + Pattern 2F, Order 211 Problematic solution: (Patterns are sorted by pattern-order shared among all add-ons) * Base Workflow + Add-On 1 Workflow + Add-On 2 Workflow + Pattern 2G, Order 2 + Pattern 1C, Order 6 (conflict!) + Pattern 2A, Order 6 (conflict!) + Pattern 1A, Order 35 + Pattern 2F, Order 211 + Pattern 1R, Order 355 I'd rather have a simple solution extensible in the future. The second solution, that sorts all patterns at once, could make more errors. If some Add-On is dependent on another one (or product), it knowns that product's workflow then it can also provide an additional workflow that would update (add/remove/replace) the workflow on a proper position. The only thing we need to provide is a proper documentation with examples. Low Level API Proposal (technical) ---------------------------------- This API uses some new terms that need to be explained: * Workflow Store - Kind of database of installation or configuration workflows * Base Workflow - The initial workflow defined by the base product - In case of running system, this will be probably empty * Additional Workflow - Any workflow defined by Add-On or Pattern in installation or Pattern in running system * Final Workflow - Workflow that contains the base workflow modified by all additional workflows Proposed set of functions unified for Add-Ons and Patterns: * SetBaseWorkflow() - Stores the initial base-product workflow ProductControll::initial_workflows ProductControll::initial_proposals ProductControll::initial_inst_finish ProductControll::initial_clone_modules * AddWorkflow() - Adds a new workflow definition into the Workflow Store - Add-On workflows will be identified by source ID (unique) - Patterns workflows will be identified by source ID plus pattern name (unique) * RemoveWorkflow() - Removes an already stored workflow definition from the Workflow Store * MergeWorkflows() - Uses the Base workflow as the initial workflow and merges all additional workflows in use - Merging might need to follow some order in which these workflows are merged to avoid conflicts and undefined behavior * RedrawWizardSteps() - Intentionally redraws the installation wizard steps to match the current merged workflows. * HaveAdditionalWorkflows() - Returns whether some additional workflows were used or not And, of course, some other helpers that will be used by functions mentioned above. Workflow Callbacks (from LibZYPP) --------------------------------- * AddResolvableWorkflow (); - Called from libzypp when any resolvable (that has own workflow) is selected to be installed. WorkflowManager is then called to propagate that workflow to YaST. * RemoveResolvableWorkflow (); - Called from libzypp when any resolvable (that has own workflow) is removed from the pool. WorkflowManager is then called to remove the workflow from YaST. For YaST, it's better to have a path to some XML file instead of getting the XML as string data from libzypp. However YaST can store the data to some temporary file because it needs to do so anyway (just because of adding/removing workflows to/from the main workflow). Needed ZYPP Support for Pattern Workflows (technical) ----------------------------------------------------- There needs to be an additional XML file with workflows of resolvables on the media. /resolvables_workflows.xml This XML file would need to contain these pieces of information: XML file | +- Workflow definition item (1) | | | +- Resolvable name | | | +- Resolvable type | | | +- Workflow order | | | +- Workflow definition (added, removed dialogs, etc.) | (Or a link to file containing the XML definition) | +- Workflow definition item (2) | | | +- ... | +- Workflow definition item (n) | +- ... All this needs to be sent via the callback to YaST. "Workflow definition" and "Workflow order" doesn't need to be sent in case of removing the workflow. Needed Pkg-Binding Support -------------------------- Pkg bindings are requested to propagate libzypp bindings to YaST.

On Mon, Jul 02, Lukas Ocilka wrote:

So, about the current solution: Every single resolvable can have its own workflow (pattern, language, product, package...). All these pieces of information also with the order of workflows are written into single XML file for one product

(URL). Libzypp parses the file and if any resolvable having its own

IMO "No". - Libzypp can assist maintaining an uptodate copy of the workflow file on disk. - Libzypp provides information about items being installed/removed (we may enhance if the current state is not sufficient). But IMO libzypp should not process the xml. That's YaST (maybe pkg-bindings) stuff. -- cu, Michael Andres +------------------------------------------------------------------+ Key fingerprint = 2DFA 5D73 18B1 E7EF A862 27AC 3FB8 9E3A 27C6 B0E4 +------------------------------------------------------------------+ Michael Andres YaST Development ma@novell.com SUSE LINUX Products GmbH, GF: Markus Rex, HRB 16746 (AG Nuernberg) Maxfeldstrasse 5, D-90409 Nuernberg, Germany, ++49 (0)911 - 740 53-0 +------------------------------------------------------------------+ -- To unsubscribe, e-mail: zypp-devel+unsubscribe@opensuse.org For additional commands, e-mail: zypp-devel+help@opensuse.org