[opensuse-packaging] Introducing brand-new nodejs-packaging
Hi, Some of you may have seen this on Google Plus, so: Short version: The way to package Node.js modules for openSUSE ( I say "for openSUSE", _may_ not for self-use/happy-hacking, read below) will change from, "every module is a package" to "every logically reasonable bundle is a package" or "every simple application is a package", with the help of the "npkg" tool written by me here: https://github.com/marguerite/nodejs-packaging This will greatly reduce the amount of nodejs packages that need maintaining. Long version: (with "why" complaints and a simple howto) http://marguerite.su/introducing-the-brand-new-nodejs-packaging/ Impacts: Tumbleweed or any released product will _not_ get impacted, because there're no nodejs module packages in them at all :-) :-( This will affect those who package nodejs modules in their own home (devel:languages:nodejs will be cared by myself, sadly no one steps in for co-maintenance for almost 2 years), by providing an alternative way. The new tool is 99.99% compatible with the old Fedora one because I rewrote all existing functions. So most of you can just ignore the changes in toolkit (although something might change still). But if, if you're interested in the bundle method, and are tired of maintaining hundreds of packages, you can give it a try. I have strong evidence that the new tool can reduce the packages to maintain from 400+ to less than 10. Keep focus on things you really need, why not? Changes need for discussion: 1. The wiki needs to be updated to reflect the change in toolkit (the old way and the new way). This will get done by myself because I live long enough to know both methods well. 2, The nodejs naming scheme needs to be discussed. In the past we used "nodejs-xxx" as package names. Now I myself prefer to name bundles just as their upstream names(the old packages will proceed with the "nodejs-" prefix).eg: jshint, jslint, coffee-script, gulp...because in the new way, nodejs packages will be fully functional applications, no difference with "firefox", "chromium"... Shortcomings: 1. SLE_11_SP3 is currently not supported. I started the ruby 1.8 porting anyway. 2. Licenses may bother you. Because it's a bundle. it contains lots of .tgz sources from different nodejs upstreams. npkg will create a file like "gulp.license" to gather all licenses inside the bundle for you. But it's like this: "BSD and MIT and ISC and Apache-2.0 and BSD-3-Clause and Apache 2.0 and Apache" not fully compatible with openSUSE's license strings. In such case, you may need to download the built RPM, "unrpm" it, and use shell tools like "grep" to confirm eg Is "BSD" "BSD-3-Clause"? And one thing I still concern is the compatibility of the licenses. eg: Can MIT and ISC coexist together? 3. There might be bugs that have not found by me yet. And as I'm a newbie to ruby programming...I didn't write stuff like exceptions...so it might be a little hard for you to debug (but you have my words that I'll fix it ASAP if get told) 4. This tool will _not_ build native node modules (ones with .gyp files) for you. You need to build it yourself for now :-( Well, the old one has no such feature as well. So it might be easy to live with it. I may scan the source and give you some hints if there're .gyp files existing. Later I'll add such a feature to automatically build native modules. Test Coverage: I have tested it for almost a month, with 5 beta versions. I tested it with azure module: 1377:delayed-stream 1378:mime 1379:async 1380:validator 1381:mime 1382:node-uuid 1383:extend I think most of you will not meet such complex package...so I'd say my tool is usable for the public now. Glad for your feedback! Marguerite -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-packaging+owner@opensuse.org
Hi, On 02/12/2016 02:04 PM, Marguerite Su wrote:
Hi,
Some of you may have seen this on Google Plus, so:
Short version:
The way to package Node.js modules for openSUSE ( I say "for openSUSE", _may_ not for self-use/happy-hacking, read below) will change from, "every module is a package" to "every logically reasonable bundle is a package" or "every simple application is a package", with the help of the "npkg" tool written by me here: https://github.com/marguerite/nodejs-packaging
This will greatly reduce the amount of nodejs packages that need maintaining.
Long version: (with "why" complaints and a simple howto)
http://marguerite.su/introducing-the-brand-new-nodejs-packaging/
THANKS for the effort!
Impacts:
Tumbleweed or any released product will _not_ get impacted, because there're no nodejs module packages in them at all :-) :-(
This will affect those who package nodejs modules in their own home (devel:languages:nodejs will be cared by myself, sadly no one steps in for co-maintenance for almost 2 years), by providing an alternative way.
The new tool is 99.99% compatible with the old Fedora one because I rewrote all existing functions. So most of you can just ignore the changes in toolkit (although something might change still).
But if, if you're interested in the bundle method, and are tired of maintaining hundreds of packages, you can give it a try. I have strong evidence that the new tool can reduce the packages to maintain from 400+ to less than 10. Keep focus on things you really need, why not?
Changes need for discussion:
1. The wiki needs to be updated to reflect the change in toolkit (the old way and the new way). This will get done by myself because I live long enough to know both methods well.
2, The nodejs naming scheme needs to be discussed.
In the past we used "nodejs-xxx" as package names. Now I myself prefer to name bundles just as their upstream names(the old packages will proceed with the "nodejs-" prefix).eg: jshint, jslint, coffee-script, gulp...because in the new way, nodejs packages will be fully functional applications, no difference with "firefox", "chromium"...
Shortcomings:
1. SLE_11_SP3 is currently not supported. I started the ruby 1.8 porting anyway.
2. Licenses may bother you.
Because it's a bundle. it contains lots of .tgz sources from different nodejs upstreams.
npkg will create a file like "gulp.license" to gather all licenses inside the bundle for you. But it's like this:
"BSD and MIT and ISC and Apache-2.0 and BSD-3-Clause and Apache 2.0 and Apache"
not fully compatible with openSUSE's license strings. In such case, you may need to download the built RPM, "unrpm" it, and use shell tools like "grep" to confirm eg
Yes, so this is going to be one problem with this approach, but this is not really to your doing. Most problems with node.js are caused by the basic design decisions about handling dependencies. Unfortunately we will probably find that many upstream projects do not care about license compatibility and will just mix and match components as they see fit. The problem for us as a project maybe that we cannot distribute such a bundle, but IANL. We'll see how this fairs in legal review if such a package is ever proposed for inclusion in TW or Leap.
Is "BSD" "BSD-3-Clause"?
And one thing I still concern is the compatibility of the licenses. eg: Can MIT and ISC coexist together?
3. There might be bugs that have not found by me yet. And as I'm a newbie to ruby programming...I didn't write stuff like exceptions...so it might be a little hard for you to debug (but you have my words that I'll fix it ASAP if get told)
4. This tool will _not_ build native node modules (ones with .gyp files) for you. You need to build it yourself for now :-(
Well, the old one has no such feature as well. So it might be easy to live with it.
I may scan the source and give you some hints if there're .gyp files existing. Later I'll add such a feature to automatically build native modules.
Test Coverage:
I have tested it for almost a month, with 5 beta versions.
I tested it with azure module:
1377:delayed-stream 1378:mime 1379:async 1380:validator 1381:mime 1382:node-uuid 1383:extend
I think most of you will not meet such complex package...so I'd say my tool is usable for the public now.
The other problem, and this is what makes node.js so difficult to deal with at the distribution level, are security issues. But again this is a problem of the basic design decisions made within node.js and not a problem of your work. node.js basically provides the approach that one has the same module in a number of versions in various places on the system (in each bundle). If one version has a security issue it is not easy to determine if a.) one has the module with the security issue installed; and if one has it b.) which bundle provides the faulty module. Therefore, one is completely dependent on the person/team that develops the bundle to provide an update to the "application" that includes a fixed module, or I should say a pointer to a module without the problem such that it is pulled properly. Anyway, again this is nothing that is caused by your effort and is simply a problem of design decisions made for node.js. Whether or not such bundles can show up in TW or Leap is a different discussion. Later, Robert -- Robert Schweikert MAY THE SOURCE BE WITH YOU Public Cloud Architect LINUX rjschwei@suse.com IRC: robjo
Hi, Robert On Sat, Feb 13, 2016 at 4:07 AM, Robert Schweikert <rjschwei@suse.com> wrote:
Yes, so this is going to be one problem with this approach, but this is not really to your doing. Most problems with node.js are caused by the basic design decisions about handling dependencies. Unfortunately we will probably find that many upstream projects do not care about license compatibility and will just mix and match components as they see fit. The problem for us as a project maybe that we cannot distribute such a bundle, but IANL. We'll see how this fairs in legal review if such a package is ever proposed for inclusion in TW or Leap.
Luckily the modules are not compiled into each other 99% of the times. They are isolated enough (lives in their own directory) in the bundle. So the real challenge I think is whether our license digger has powerful enough tools to check all those pieces of files...I can also help by bundling one in the toolkit itself of course.
node.js basically provides the approach that one has the same module in a number of versions in various places on the system (in each bundle). If one version has a security issue it is not easy to determine if a.) one has the module with the security issue installed; and if one has it b.) which bundle provides the faulty module. Therefore, one is completely dependent on the person/team that develops the bundle to provide an update to the "application" that includes a fixed module, or I should say a pointer to a module without the problem such that it is pulled properly.
Actually it's easy to find the faulty module with specific version :-) npkg will generate a "<name>.json" with the same format. https://build.opensuse.org/package/view_file/devel:languages:nodejs/gulp/gul... And I have figured out the download URL for such json files from OBS. so finding a hash key with a specific value is really basic topic in Ruby... I even drafted a statistics tool (/usr/bin/npkg-mgmt-json-statistics) that gets how popular a specific module is (by counting its appearance across .json files) Well to make upstream to update the faulty module is another story... Marguerite -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-packaging+owner@opensuse.org
Hi Marguerite, Am Samstag, 13. Februar 2016, 03:04:53 schrieb Marguerite Su:
This will greatly reduce the amount of nodejs packages that need maintaining.
Long version: (with "why" complaints and a simple howto)
http://marguerite.su/introducing-the-brand-new-nodejs-packaging/
Thanks for your effort! Looks like the last chance to get nodejs handled. What I understand from the documentation: - you have to run it from a local machine with the nodejs-packaging installed - npkg will manage the download - existing .json files will be ignored Right so far? I run a test on tryton-sao: test@linux-vmko:~/tryton-sao> npkg tryton-sao 1:tryton-sao 2:bower test@linux-vmko:~/tryton-sao> --2016-02-14 21:05:25-- http://registry.npmjs.org/bower/-/bower-1.7.7.tgz Auflösen des Hostnamen »registry.npmjs.org (registry.npmjs.org)«... 23.235.43.162 Verbindungsaufbau zu registry.npmjs.org (registry.npmjs.org)| 23.235.43.162|:80... --2016-02-14 21:05:25-- http://registry.npmjs.org/tryton-sao/-/tryton-sao-3.8.3.tgz Auflösen des Hostnamen »registry.npmjs.org (registry.npmjs.org)«... 23.235.43.162 Verbindungsaufbau zu registry.npmjs.org (registry.npmjs.org)| 23.235.43.162|:80... verbunden. HTTP-Anforderung gesendet, warte auf Antwort... verbunden. HTTP-Anforderung gesendet, warte auf Antwort... 200 OK Länge: 3157637 (3,0M) [application/octet-stream] In »»bower-1.7.7.tgz«« speichern. 0% [ ] 0 --.-K/s 200 OK Länge: 498609 (487K) [application/octet-stream] In »»tryton-sao-3.8.3.tgz«« speichern. 100%[==============================================>] 498.609 43,8KB/s in 12s 2016-02-14 21:05:38 (41,6 KB/s) - »»tryton-sao-3.8.3.tgz«« gespeichert [498609/498609] 100%[==============================================>] 3.157.637 56,1KB/s in 49s 2016-02-14 21:06:16 (62,5 KB/s) - »»bower-1.7.7.tgz«« gespeichert [3157637/3157637] ---> here the npkg application got stuck. It does not resolve the stuff that bower usually downloads. How to continue from here? Thanks Axel -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-packaging+owner@opensuse.org
On Mon, Feb 15, 2016 at 4:52 AM, Axel Braun <axel.braun@gmx.de> wrote:
What I understand from the documentation: - you have to run it from a local machine with the nodejs-packaging installed - npkg will manage the download - existing .json files will be ignored
Right so far?
Right.
I run a test on tryton-sao:
test@linux-vmko:~/tryton-sao> npkg tryton-sao 1:tryton-sao 2:bower test@linux-vmko:~/tryton-sao> --2016-02-14 21:05:25-- http://registry.npmjs.org/bower/-/bower-1.7.7.tgz Auflösen des Hostnamen »registry.npmjs.org (registry.npmjs.org)«... 23.235.43.162 Verbindungsaufbau zu registry.npmjs.org (registry.npmjs.org)| 23.235.43.162|:80... --2016-02-14 21:05:25-- http://registry.npmjs.org/tryton-sao/-/tryton-sao-3.8.3.tgz Auflösen des Hostnamen »registry.npmjs.org (registry.npmjs.org)«... 23.235.43.162 Verbindungsaufbau zu registry.npmjs.org (registry.npmjs.org)| 23.235.43.162|:80... verbunden. HTTP-Anforderung gesendet, warte auf Antwort... verbunden. HTTP-Anforderung gesendet, warte auf Antwort... 200 OK Länge: 3157637 (3,0M) [application/octet-stream] In »»bower-1.7.7.tgz«« speichern.
0% [ ] 0 --.-K/s 200 OK Länge: 498609 (487K) [application/octet-stream] In »»tryton-sao-3.8.3.tgz«« speichern.
100%[==============================================>] 498.609 43,8KB/s in 12s
2016-02-14 21:05:38 (41,6 KB/s) - »»tryton-sao-3.8.3.tgz«« gespeichert [498609/498609]
100%[==============================================>] 3.157.637 56,1KB/s in 49s
2016-02-14 21:06:16 (62,5 KB/s) - »»bower-1.7.7.tgz«« gespeichert [3157637/3157637]
---> here the npkg application got stuck. It does not resolve the stuff that bower usually downloads.
How to continue from here?
Hit Enter key. -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-packaging+owner@opensuse.org
Am Montag, 15. Februar 2016, 18:38:17 schrieb Marguerite Su:
I run a test on tryton-sao:
test@linux-vmko:~/tryton-sao> npkg tryton-sao 1:tryton-sao 2:bower test@linux-vmko:~/tryton-sao> --2016-02-14 21:05:25-- http://registry.npmjs.org/bower/-/bower-1.7.7.tgz Auflösen des Hostnamen »registry.npmjs.org (registry.npmjs.org)«... 23.235.43.162 Verbindungsaufbau zu registry.npmjs.org (registry.npmjs.org)| 23.235.43.162|:80... --2016-02-14 21:05:25-- http://registry.npmjs.org/tryton-sao/-/tryton-sao-3.8.3.tgz Auflösen des Hostnamen »registry.npmjs.org (registry.npmjs.org)«... 23.235.43.162 Verbindungsaufbau zu registry.npmjs.org (registry.npmjs.org)| 23.235.43.162|:80... verbunden. HTTP-Anforderung gesendet, warte auf Antwort... verbunden. HTTP-Anforderung gesendet, warte auf Antwort... 200 OK Länge: 3157637 (3,0M) [application/octet-stream] In »»bower-1.7.7.tgz«« speichern.
0% [ ] 0 --.-K/s
200 OK Länge: 498609 (487K) [application/octet-stream] In »»tryton-sao-3.8.3.tgz«« speichern.
100%[==============================================>] 498.609 43,8KB/s in 12s
2016-02-14 21:05:38 (41,6 KB/s) - »»tryton-sao-3.8.3.tgz«« gespeichert [498609/498609]
100%[==============================================>] 3.157.637 56,1KB/s in 49s
2016-02-14 21:06:16 (62,5 KB/s) - »»bower-1.7.7.tgz«« gespeichert [3157637/3157637]
---> here the npkg application got stuck. It does not resolve the stuff that bower usually downloads.
How to continue from here?
Hit Enter key.
Then I return to the command prompt. No more activities inside the npkg /Ax -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-packaging+owner@opensuse.org
On Mon, Feb 15, 2016 at 7:23 PM, Axel Braun <axel.braun@gmx.de> wrote:
Then I return to the command prompt. No more activities inside the npkg
Then everything should be fine :-) If not, please open a bug report at github.com/marguerite/nodejs-packaging I'll see why Marguerite -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-packaging+owner@opensuse.org
participants (3)
-
Axel Braun
-
Marguerite Su
-
Robert Schweikert