[opensuse-packaging] Automated binary java packaging using maven central
I've written a successful prototype that does the following: * generates a completely functional .spec file based on the metadata of java artifacts (POM) that is available at Maven Central (http://repo1.maven.org/maven2/) * does *NOT* build from source but, instead, simply takes the .jar file from Maven Central and installs it into %{_javadir} (including the usual symlinks) * optionally also creates a subpackage with the source jar from Maven Central (very useful for IDEs such as Eclipse) * all that including dependencies, as far as they're declared in the POM file * written in Perl (hey, it's a prototype ;)), just one file, no exotic dependencies The script is there: https://gitorious.org/pbleser/pbleser/blobs/master/pom2spec Here's an example on how to run it: 1) pull the script and chmod it: cd ~/bin wget https://gitorious.org/pbleser/pbleser/blobs/raw/master/pom2spec chmod +x pom2spec 2) go to your OBS sandbox and create a new package: osc mkpac log4j-log4j-1_2_15 3) generate the .spec file, passing groupId artifactId and version: pom2spec log4j log4j 1.2.15 (if you don't specify the version, pom2spec will give you a list of possible versions, i.e. those that are available in Maven Central) 4) well, obviously, build it ;) osc build ... ===== The longer story... First of all, I guess that everyone knows by now that the packaging situation around Java libraries (and, hence, applications (desktop and server) that depend on those libraries) is ... let's say... suboptimal. It is very complex to build those from source as the build systems that are typically used for Java projects (Ant, Maven, Gradle, Ant+Ivy, ...) have a dependency management of their own that is quite difficult to integrate with RPM packaging. Stephen Shaw has been putting a lot of effort into doing it properly since some time and while he's made a lot of progress already, I think we're still far away from having a useful set of common Java libraries (there are gazillions of open source Java libraries and frameworks out there). Hence, I believe that having them as "bad" RPMs (because they're not built from source but just install the already compiled .jar files from Maven Central (hence upstream)) is better than not having them _at all_. Of course, starting from there, we should still continue to improve our tool set and experience with building Java packages from source, in order to replace at least some of those "bad" RPMs with "good" ones. The drawback is that we cannot add our own patches to the upstream sources, for obvious reasons. But the advantage of this approach is that it is very easy to build the RPMs. And, to be honest, I don't think that, in the past or in the future, we often need to add patches to portable Java source code that haven't been included upstream already. Sure, that doesn't work well with the typical policy for the distribution updates (backport patches instead of version bumps) but hey... better than nothing, don't you think ? ===== Now the interactive part... While writing the pom2spec script, a few questions or, rather, a need for some decisions, appeared. We probably can all agree that we should add Provides in those Java library packages in the same fashion as with Perl packages, along the lines of this: Provides: java(${groupId}:${artifactId}) = ${version} e.g.: Provides: java(org.springframework:spring-beans) = 2.5.6.SEC02 (the g:a and g:a:v notation is very commonly used in Java/Maven/Gradle/...) That would make it easier, more portable and clearly more future-proof in dependencies (Requires). But what I'm not sure about, at this point, is how to name the packages themselves. I believe we should use fully qualified ${groupId}-${artifactId} names, e.g.: Name: org.springframework.spring-beans ^^^^^^^^^^^^^^^^^^^ ^^^^^^^^^^^^ ${groupId} ${artifactId} That (${ŋroupId}:${artifactId}) already is a unique and canonical name as of Maven Central, which is what 90%+ Java projects on the planet use (either through Maven, or Ant+Ivy, or Gradle, or ...). But that's too long for Joliet. Is that still an issue ? Another thing is that I believe that it would make our life easier if we also included the ${version} in the %{NAME} of the package, in order to be able to install several different versions of libraries side-by-side. That way we can always reflect upstream's dependencies as defined in their POM file (e.g. org.springframework:spring-beans:2.5.6.SEC02 requiring log4j:log4j:1.2.15). Example: Name: org.springframework.spring-beans-2_5_6_SEC02 Version: 2.5.6.SEC02 Yes, that's a pretty long package name alright. That would also mean dropping the symlink that drops the %{version} in %{_javadir}. Feedback please :) (And thanks for reading this far, I know this email is way too long, but I wanted to give enough background :)). cheers -- -o) Pascal Bleser /\\ http://opensuse.org -- we haz green _\_v http://fosdem.org -- we haz conf
Hi Pascal. Comments inline: I tested this and it mostly just works TM. Not bad for a prototype :) On 08/14/2011 04:13 PM, Pascal Bleser wrote:
I've written a successful prototype that does the following: * generates a completely functional .spec file based on the metadata of java artifacts (POM) that is available at Maven Central (http://repo1.maven.org/maven2/) * does *NOT* build from source but, instead, simply takes the .jar file from Maven Central and installs it into %{_javadir} (including the usual symlinks) * optionally also creates a subpackage with the source jar from Maven Central (very useful for IDEs such as Eclipse) * all that including dependencies, as far as they're declared in the POM file * written in Perl (hey, it's a prototype ;)), just one file, no exotic dependencies
The script is there: https://gitorious.org/pbleser/pbleser/blobs/master/pom2spec
Here's an example on how to run it: 1) pull the script and chmod it: cd ~/bin wget https://gitorious.org/pbleser/pbleser/blobs/raw/master/pom2spec chmod +x pom2spec
2) go to your OBS sandbox and create a new package: osc mkpac log4j-log4j-1_2_15
This part failed for me. But I am on a new laptop, so I needed to init my home repo locally.
3) generate the .spec file, passing groupId artifactId and version: pom2spec log4j log4j 1.2.15
(if you don't specify the version, pom2spec will give you a list of possible versions, i.e. those that are available in Maven Central)
4) well, obviously, build it ;) osc build ...
Missing step, move the spec into the project/package directory.
===== The longer story...
First of all, I guess that everyone knows by now that the packaging situation around Java libraries (and, hence, applications (desktop and server) that depend on those libraries) is ... let's say... suboptimal.
Very very true.
It is very complex to build those from source as the build systems that are typically used for Java projects (Ant, Maven, Gradle, Ant+Ivy, ...) have a dependency management of their own that is quite difficult to integrate with RPM packaging.
Sometimes face palming impossible.
Stephen Shaw has been putting a lot of effort into doing it properly since some time and while he's made a lot of progress already, I think we're still far away from having a useful set of common Java libraries (there are gazillions of open source Java libraries and frameworks out there).
Yes, and we are missing the boat there. The fewer rpms we have the messier it is to support Java.
Hence, I believe that having them as "bad" RPMs (because they're not built from source but just install the already compiled .jar files from Maven Central (hence upstream)) is better than not having them _at all_. Of course, starting from there, we should still continue to improve our tool set and experience with building Java packages from source, in order to replace at least some of those "bad" RPMs with "good" ones.
I see this as as tool and a repo to bootstrap building more java packages. We may need a separate repo just so the "bad" rpms intially.
The drawback is that we cannot add our own patches to the upstream sources, for obvious reasons. But the advantage of this approach is that it is very easy to build the RPMs. And, to be honest, I don't think that, in the past or in the future, we often need to add patches to portable Java source code that haven't been included upstream already.
99% of the patching I have seen to java packages are to the build.xml files for messing with class paths or install paths. I cannot recall ever seeing the sources themselves patched in rpm packages.
Sure, that doesn't work well with the typical policy for the distribution updates (backport patches instead of version bumps) but hey... better than nothing, don't you think ?
Yes and in the java world backports really do not exist afaik. Want a bug fixed ? Update..
===== Now the interactive part...
While writing the pom2spec script, a few questions or, rather, a need for some decisions, appeared.
We probably can all agree that we should add Provides in those Java library packages in the same fashion as with Perl packages, along the lines of this: Provides: java(${groupId}:${artifactId}) = ${version} e.g.: Provides: java(org.springframework:spring-beans) = 2.5.6.SEC02
(the g:a and g:a:v notation is very commonly used in Java/Maven/Gradle/...)
That would make it easier, more portable and clearly more future-proof in dependencies (Requires).
But what I'm not sure about, at this point, is how to name the packages themselves.
I believe we should use fully qualified ${groupId}-${artifactId} names, e.g.: Name: org.springframework.spring-beans ^^^^^^^^^^^^^^^^^^^ ^^^^^^^^^^^^ ${groupId} ${artifactId}
That (${ŋroupId}:${artifactId}) already is a unique and canonical name as of Maven Central, which is what 90%+ Java projects on the planet use (either through Maven, or Ant+Ivy, or Gradle, or ...).
But that's too long for Joliet. Is that still an issue ?
Works for me. I did not see an issue testing. Where I am more curious is how we map the provides auto generated by rpm vs build requires in the spec files generated from the pom file. That will need some attention it seems.
Another thing is that I believe that it would make our life easier if we also included the ${version} in the %{NAME} of the package, in order to be able to install several different versions of libraries side-by-side. That way we can always reflect upstream's dependencies as defined in their POM file (e.g. org.springframework:spring-beans:2.5.6.SEC02 requiring log4j:log4j:1.2.15).
Example: Name: org.springframework.spring-beans-2_5_6_SEC02 Version: 2.5.6.SEC02
Yes, that's a pretty long package name alright.
That would also mean dropping the symlink that drops the %{version} in %{_javadir}.
How will this affect runtimes ? The rest we can live with.
Feedback please :)
(And thanks for reading this far, I know this email is way too long, but I wanted to give enough background :)).
cheers
Thanks for taking this on.. Java packaging has become a pain point and I do not want to see either openSUSE nor SLES become less usable as a java platform. Thanks, Peter -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-packaging+help@opensuse.org
On 2011-08-15 00:19:38 (-0700), Peter Linnell
I tested this and it mostly just works TM. Not bad for a prototype :)
Yeah, I managed to package spring framework (spring-beans) and all its dependencies with it, but just locally, I didn't commit it to any repository (yet).
On 08/14/2011 04:13 PM, Pascal Bleser wrote:
I've written a successful prototype that does the following: * generates a completely functional .spec file based on the metadata of java artifacts (POM) that is available at Maven Central (http://repo1.maven.org/maven2/) * does *NOT* build from source but, instead, simply takes the .jar file from Maven Central and installs it into %{_javadir} (including the usual symlinks) * optionally also creates a subpackage with the source jar from Maven Central (very useful for IDEs such as Eclipse) * all that including dependencies, as far as they're declared in the POM file * written in Perl (hey, it's a prototype ;)), just one file, no exotic dependencies
The script is there: https://gitorious.org/pbleser/pbleser/blobs/master/pom2spec
Here's an example on how to run it: 1) pull the script and chmod it: cd ~/bin wget https://gitorious.org/pbleser/pbleser/blobs/raw/master/pom2spec chmod +x pom2spec
2) go to your OBS sandbox and create a new package: osc mkpac log4j-log4j-1_2_15
This part failed for me. But I am on a new laptop, so I needed to init my home repo locally.
That's supposed to be ran from your osc sandbox, from a checked out project directory.
3) generate the .spec file, passing groupId artifactId and version: pom2spec log4j log4j 1.2.15
(if you don't specify the version, pom2spec will give you a list of possible versions, i.e. those that are available in Maven Central)
4) well, obviously, build it ;) osc build ...
Missing step, move the spec into the project/package directory.
Yeah, I forgot to mention: 2b) cd log4j-log4j-1_2_15 ;)
===== The longer story... [...] Hence, I believe that having them as "bad" RPMs (because they're not built from source but just install the already compiled .jar files from Maven Central (hence upstream)) is better than not having them _at all_. Of course, starting from there, we should still continue to improve our tool set and experience with building Java packages from source, in order to replace at least some of those "bad" RPMs with "good" ones.
I see this as as tool and a repo to bootstrap building more java packages. We may need a separate repo just so the "bad" rpms intially.
Yes, that might be an option. But really, as long as we don't need to apply patches to the upstream sources (never seen the need for that), that situation is perfectly fine IMHO. *And* provided we don't do backport patches.
The drawback is that we cannot add our own patches to the upstream sources, for obvious reasons. But the advantage of this approach is that it is very easy to build the RPMs. And, to be honest, I don't think that, in the past or in the future, we often need to add patches to portable Java source code that haven't been included upstream already.
99% of the patching I have seen to java packages are to the build.xml files for messing with class paths or install paths. I cannot recall ever seeing the sources themselves patched in rpm packages.
Me neither. At worst, it means updating the package to a git/svn snapshot of upstream (e.g. log4j-log4j-1_2_15-1.2.15+r9876)
Sure, that doesn't work well with the typical policy for the distribution updates (backport patches instead of version bumps) but hey... better than nothing, don't you think ?
Yes and in the java world backports really do not exist afaik. Want a bug fixed ? Update..
Right. There are no SONAMEs in Java and while several projects are very careful with not breaking backwards API compatibility (there are well-defined rules for ABI compatibility on bytecode level too, not sure many projects pay attention to that though), most don't. Well. Pretty much the same as with C/C++ anyway, SONAMEs or not (at least from my experience). A consequence, I believe, is that we would be better off providing several versions, even minor, of the same libraries. For dependencies, we should just use the required versions of those dependencies as they are defined by the upstream projects in their POM file. Sure, we might (and would/will) end up with foo.bar -> org.spam-spam-1_0 = 1.0 and com.vikings -> org.spam-spam-1_1 = 1.1 (or whatever the package naming will be) But hey, that's what the upstream devs have defined as versions in the POM file. Might as well run it with that instead of the latest (which might not be compatible at all, neither on API nor ABI level). [...]
That (${ŋroupId}:${artifactId}) already is a unique and canonical name as of Maven Central, which is what 90%+ Java projects on the planet use (either through Maven, or Ant+Ivy, or Gradle, or ...).
But that's too long for Joliet. Is that still an issue ?
Works for me. I did not see an issue testing. Where I am more curious is how we map the provides auto generated by rpm vs build requires in the spec files generated from the pom file. That will need some attention it seems.
We could certainly try to add something to RPM in order to autoreq some Provides and Requires. The Provides and Requires I mentioned are generated by the pom2spec script itself. Quite some time ago, I wrote a prototype that can determine Requires: and Provides: automatically from analyzing the bytecode (in .jar or .class files). It uses jdepend [1]. Don't remember where it is, but if someone is interested, I can dig it out, must be somewhere on my harddisk. I even managed to compile it into a native binary with gcj ;) [1] http://www.clarkware.com/software/JDepend.html We'll end up with *very* long lists of dependencies though, and they wouldn't necessarily be correct -- as an example, Spring Framework is an integration layer with many technologies and libraries and for most of them, it does not require them to be present at runtime (e.g. if you don't use the JMS support, you don't need to have the JMS jars in the classpath at runtime) -- Java uses late binding at runtime, a .jar file is completely different from a .so. Having the pom2spec generate both the Requires and the Provides gives better results in my opinion: * sure, they *might* be wrong, as we don't analyze the bytecode to find *all* dependencies with 100% accuracy (jdepend can give us that) * the Requires are those that have been specified by upstream in the POM, and they ought to know which dependencies make sense, and which don't (see the spring framework example above) Given that an enormous amount of open source and non open source Java projects on the planet (and that's indeed quite a huge amount) use and depend on those POMs to be at least partially correct, and very much so in their dependency information, I don't doubt that cases where the POM is actually missing dependencies will be rare. And fixed by upstream if we file it as a bug.
Another thing is that I believe that it would make our life easier if we also included the ${version} in the %{NAME} of the package, in order to be able to install several different versions of libraries side-by-side. That way we can always reflect upstream's dependencies as defined in their POM file (e.g. org.springframework:spring-beans:2.5.6.SEC02 requiring log4j:log4j:1.2.15).
Example: Name: org.springframework.spring-beans-2_5_6_SEC02 Version: 2.5.6.SEC02
Yes, that's a pretty long package name alright.
That would also mean dropping the symlink that drops the %{version} in %{_javadir}.
How will this affect runtimes ? The rest we can live with.
It does not affect libraries at all. It only affects applications (server or desktop). The shell script wrapper that starts e.g. Netbeans, or Tomcat, or ... needs to build the CLASSPATH environment variable accordingly. For transitive dependencies, the pom2spec script also generates .classpath files (e.g. /usr/share/java/foo.org-spam-1.2.3.classpath) that can be aggregated. With what I've hacked in pom2spec, it could be something like this: ---8<----------------------------------------------------- #!/bin/sh CLASSPATH="" for d in \ org.springframework.spring-beans-2.5.6.SEC02 \ ... \ ... \ ; do CLASSPATH="$CLASSPATH:$(cat /usr/share/java/$d)" done CLASSPATH="${CLASSPATH#:}" export CLASSPATH exec java -jar \ /usr/share/java/my.awesome.application-1.2.jar ---8<----------------------------------------------------- But that's just an idea, if someone has a better one... ;) From my very long experience as a Java developer, I certainly find it much more deterministic to always run an application against the exact dependencies that upstream has defined (e.g. log4j 1.2.11 and not 1.2.15 because 1.2.15 happens to be the latest). [...]
Thanks for taking this on.. Java packaging has become a pain point and I do not want to see either openSUSE nor SLES become less usable as a java platform.
That would be extremely silly too. I'm quite aware that most SUSErs or even people in the community at large hate Java for whatever historic or religious reasons, but the fact is that there is a huge amount of open source projects out there in Java, and that's not going to go away any time soon. Especially if we want to also go into "the cloud" (blech, buzzword), that's really quite Java centric (Hadoop, Solr, Infinispan, OpenFire/Tigase, etherpad, .........). cheers -- -o) Pascal Bleser /\\ http://opensuse.org -- we haz green _\_v http://fosdem.org -- we haz conf
On 2011-08-15 22:18:51 (+0200), Pascal Bleser
We could certainly try to add something to RPM in order to autoreq some Provides and Requires.
The Provides and Requires I mentioned are generated by the pom2spec script itself.
Quite some time ago, I wrote a prototype that can determine Requires: and Provides: automatically from analyzing the bytecode (in .jar or .class files).
It uses jdepend [1]. Don't remember where it is, but if someone is interested, I can dig it out, must be somewhere on my harddisk. I even managed to compile it into a native binary with gcj ;)
Found it, and committed it to my git repo, here: https://gitorious.org/pbleser/pbleser/trees/master/jfindreqprov It's an Eclipse project, but there's also a "build.sh" to build it as a native ELF binary using gcj. It has no dependencies except for jdepend, which is included in a stripped-down jar file in lib/ [...] cheers -- -o) Pascal Bleser /\\ http://opensuse.org -- we haz green _\_v http://fosdem.org -- we haz conf
On 08/15/2011 01:13 AM, Pascal Bleser wrote:
I've written a successful prototype that does the following: * generates a completely functional .spec file based on the metadata of java artifacts (POM) that is available at Maven Central (http://repo1.maven.org/maven2/) * does *NOT* build from source but, instead, simply takes the .jar file from Maven Central and installs it into %{_javadir} (including the usual symlinks) * optionally also creates a subpackage with the source jar from Maven Central (very useful for IDEs such as Eclipse) * all that including dependencies, as far as they're declared in the POM file * written in Perl (hey, it's a prototype ;)), just one file, no exotic dependencies
The script is there: https://gitorious.org/pbleser/pbleser/blobs/master/pom2spec
Hi Pascal, I think is a good idea, I once wrote something similar but did not get that far... even the name was the same :-).
===== The longer story...
First of all, I guess that everyone knows by now that the packaging situation around Java libraries (and, hence, applications (desktop and server) that depend on those libraries) is ... let's say... suboptimal.
It is very complex to build those from source as the build systems that are typically used for Java projects (Ant, Maven, Gradle, Ant+Ivy, ...) have a dependency management of their own that is quite difficult to integrate with RPM packaging.
Stephen Shaw has been putting a lot of effort into doing it properly since some time and while he's made a lot of progress already, I think we're still far away from having a useful set of common Java libraries (there are gazillions of open source Java libraries and frameworks out there).
I agree with your view on the matter. I can speak with some experience here because we had to build a hundred of package in order to get SUSE Manager running. We are now in the process to reconcile those packages in a project called Java:base, merging our packages with stuff we rebuilt from Fedora, stuff from Java:packages and stuff from Stephen. The main issue for us (my team) with this approach is that we need to be able to patch sources to give L3 support (we can't wait for the next upstream release). When building Java:base, we gave up with maven. We have a maven3 package from binaries, but we are not using it. We continued packaging every maven dependent package using the maven ant plugin, generating build.xml and maven-build.xml and then using ant. Worked very well on most of the cases. However, we will never be able to package more complex stuff: think glassfish. Just imagine Redhat is still unable to build JBoss, which they own, and they are resorting to add mvn-build support directly to koji to handle the complexity. Still, I think your tool covers the 90% of the cases where people just want to run stuff, and this should not get lost. I propose to build a self-contained repository for Java based on this tool, it would be lot of value for the community and SUSE customers. The main goal of this repo would be more completeness and being vanilla instead of patches and bugfixes. Then we could work to make Java:base compatible with the "evil" packages, so that they can be mixed without much trouble: Provides, locations, etc. For the Provides part, we should give a look at what Fedora is doing as they are already adding provides for OSGi at least, and being compatible would be nice. Now that you are still playing with the technology, another project that looks interesting is Ivy Roundup, with is a self-contained and consistent repository of packager.xml metadata to be used with the Ivy packager resolver, this allows ivy to resolve an artifact directly from the upstream project (even unziping dist files) allowing the Roundup project itself to contain only metadata: http://code.google.com/p/ivyroundup/ cheers, and keep up this cool stuff. -- Duncan Mac-Vicar P. - http://www.suse.com/ SUSE LINUX Products GmbH, GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer, HRB 16746 (AG Nürnberg) Maxfeldstraße 5, 90409 Nürnberg, Germany -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-packaging+help@opensuse.org
Am Donnerstag, 18. August 2011, 11:58:51 schrieb Duncan Mac-Vicar P.:
On 08/15/2011 01:13 AM, Pascal Bleser wrote:
I've written a successful prototype that does the following: * generates a completely functional .spec file based on the
metadata of java artifacts (POM) that is available at Maven Central (http://repo1.maven.org/maven2/)
* does *NOT* build from source but, instead, simply takes the
.jar file from Maven Central and installs it into %{_javadir} (including the usual symlinks)
* optionally also creates a subpackage with the source jar from
Maven Central (very useful for IDEs such as Eclipse)
* all that including dependencies, as far as they're declared in
the POM file
* written in Perl (hey, it's a prototype ;)), just one file, no
exotic dependencies
The script is there: https://gitorious.org/pbleser/pbleser/blobs/master/pom2spec
Hi Pascal,
I think is a good idea, I once wrote something similar but did not get that far... even the name was the same :-).
The name has the problem that an extension for .dsc files (which should be easy, if it works for rpm) would conflict ... Btw, I turned this into an OBS source service and extended a bit the source system. It is possible now to generate files just by creating a package container in OBS (just uploading the package _meta file). The source service is project global in this example: osc co home:adrianSuSE:pom2spec You can find the generator in openSUSE:Tools:Unstable project. It is called obs-service-generator_pom and requires latest download_files from openSUSE:Tools. Also osc from openSUSE:Tools:Unstable if you want to run the service locally. -- Adrian Schroeter SUSE Linux Products GmbH email: adrian@suse.de -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-packaging+help@opensuse.org
On 2011-08-18 13:02:16 (+0200), Adrian Schröter
Am Donnerstag, 18. August 2011, 11:58:51 schrieb Duncan Mac-Vicar P.:
On 08/15/2011 01:13 AM, Pascal Bleser wrote: [...] I think is a good idea, I once wrote something similar but did not get that far... even the name was the same :-).
The name has the problem that an extension for .dsc files (which should be easy, if it works for rpm) would conflict ...
Ummm.. no idea what .dsc files are .. yast2 metadata ?
Btw, I turned this into an OBS source service and extended a bit the source system. It is possible now to generate files just by creating a package container in OBS (just uploading the package _meta file). The source service is project global in this example:
osc co home:adrianSuSE:pom2spec
You can find the generator in openSUSE:Tools:Unstable project. It is called obs-service-generator_pom and requires latest download_files from openSUSE:Tools.
Also osc from openSUSE:Tools:Unstable if you want to run the service locally.
Please don't! 1) pom2spec is highly experimental, and shouldn't be used except for fiddling around and making (constructive) remarks 2) the conventions for package and provides/requires naming aren't final in any way, needs some discussion 3) source services should be burned with fire, there's at most a handful who understand how they work, and they cause tons of issues when using osc -- -o) Pascal Bleser /\\ http://opensuse.org -- we haz green _\_v http://fosdem.org -- we haz conf
On 2011-08-18 11:58:51 (+0200), Duncan Mac-Vicar P.
On 08/15/2011 01:13 AM, Pascal Bleser wrote: [...]
Stephen Shaw has been putting a lot of effort into doing it properly since some time and while he's made a lot of progress already, I think we're still far away from having a useful set of common Java libraries (there are gazillions of open source Java libraries and frameworks out there).
I agree with your view on the matter.
I can speak with some experience here because we had to build a hundred of package in order to get SUSE Manager running. We are now in the process to reconcile those packages in a project called Java:base, merging our packages with stuff we rebuilt from Fedora, stuff from Java:packages and stuff from Stephen.
Ah, very good! First time I hear about it :)
The main issue for us (my team) with this approach is that we need to be able to patch sources to give L3 support (we can't wait for the next upstream release).
Understandable. I listed that as one of the drawbacks ;)
When building Java:base, we gave up with maven. We have a maven3 package from binaries, but we are not using it. We continued packaging every maven dependent package using the maven ant plugin, generating build.xml and maven-build.xml and then using ant. Worked very well on most of the cases. However, we will never be able to package more complex stuff: think glassfish. Just imagine Redhat is still unable to build JBoss, which they own, and they are resorting to add mvn-build support directly to koji to handle the complexity.
mmmmyeah, those are quite different beasts. Never tried to build them from source though, can't say much about it (except that no one should use JEE containers in the first place. I hope you don't use one for SUSE Manager. But that's another discussion ;))
Still, I think your tool covers the 90% of the cases where people just want to run stuff, and this should not get lost.
I propose to build a self-contained repository for Java based on this tool, it would be lot of value for the community and SUSE customers. The main goal of this repo would be more completeness and being vanilla instead of patches and bugfixes.
Yeah, had something like that in mind as well. At least to have something to start with. We (or I) should set a goal and see whether that approach works sufficiently well for that, e.g. packaging Glassfish or Tomcat.
Then we could work to make Java:base compatible with the "evil" packages, so that they can be mixed without much trouble: Provides, locations, etc.
And replace the "evil" packages by "proper" ones. IMHO what pom2spec brings to the table is an interim solution. It's far from perfect, but it's something.
For the Provides part, we should give a look at what Fedora is doing as they are already adding provides for OSGi at least, and being compatible would be nice.
Okay, didn't know about that. Yes, it would be utterly stupid to be incompatible, we have enough differences already.
Now that you are still playing with the technology, another project that looks interesting is Ivy Roundup, with is a self-contained and consistent repository of packager.xml metadata to be used with the Ivy packager resolver, this allows ivy to resolve an artifact directly from the upstream project (even unziping dist files) allowing the Roundup project itself to contain only metadata: http://code.google.com/p/ivyroundup/
Didn't have that one on my radar. I've pretty much given up on Ivy metadata, as what almost everyone is using is Maven Central. It's pretty much the authoritative source. And I know about the many issues with Maven Central, quite a lot of pom files there are b0rked, learned that the hard way. Most of them don't even include the license. But I do expect the dependency information to be rather dependable, and that's definitely the toughest part to get right (the license, summary or description can easily be looked up). But sure, pom2spec is simple and stupid enough to collect metadata from different sources, including Ivy RoundUp or other Maven repos (e.g. the one from JBoss). Having a look at the Ivy RoundUp metadata [1], they definitely seem to get a few things right, and have much more reliable metadata. But the <configurations/> thingy in Ivy is way too flexible (or, rather, loosely defined) to be useful. [1] curl \ https://ivyroundup.googlecode.com/svn/trunk/repo/modules/org.apache.wicket/w... \ | xmllint format -|view - Could be feasible to use the ivy.xml + packager.xml [2] to generate a spec file, within limits. We can't run ivyroundup itself, as it downloads the artifacts (jars, zips) by itself, which doesn't work in OBS. [2] curl \ https://ivyroundup.googlecode.com/svn/trunk/repo/modules/org.apache.wicket/w... \ | xmllint format -|view - I wonder whether we could find some time at the openSUSE conf to sit in a room for an hour and see whether we come up with a few ideas and gotchas. Right now, I'm stuck with pom2spec because I'm not sure how we should handle the versioning (see original post). * wicket = 1.4.3 * org.apache.wicket-wicket = 1.4.3 * wicket-1_4_3 = 1.4.3 * org.apache.wicket-wicket_1_4_3 = 1.4.3 * wicket-1_4 = 1.4.3 * org.apache.wicket-wicket-1_4 = 1.4.3 ? The same applies to the Provides, although we're a lot more flexible there as one package can make several provides. I would suggest something along the lines of Provides: java(org.apache.wicket:wicket) = 1.4.3 That way, packages that depend on it can choose the granularity: * Requires: java(org.apache.wicket:wicket) = 1.4.3 * Requires: java(org.apache.wicket:wicket) >= 1.4.1 Requires: java(org.apache.wicket:wicket) < 1.5 etc... That way the name of the package in itself doesn't matter too much, and it could be kept as simple as e.g. "wicket-1.4.3" (that's %{name}-%{version}), or "apache-wicket-1.4.3". Thoughts ? cheers -- -o) Pascal Bleser /\\ http://opensuse.org -- we haz green _\_v http://fosdem.org -- we haz conf
participants (4)
-
Adrian Schröter -
Duncan Mac-Vicar P. -
Pascal Bleser -
Peter Linnell