Hi folks,
after some discussions and thinking I propose to clean up the obs
config setup as described below.
The configs are a little messy, each service doing his own stuff in
his own way (just like logging and monitoring, but that's another
story).
Now here's the challenge I have: The config is central to each
service, if I mess it up, it doesn't work any more, and it's easy to
mess it up because there is tiny bits of background knowledge you need
to have about e.g. rails, and how things are related. So I'm somewhat
reluctant to start patching things and messing around with them.
I'd like to propose a new scheme anyhow and get hints from you how to
migrate there.
Proposal:
Each service has two or more small config files below /etc/obs, a
'.public' one, an optional '.private' one, and if needed additional
'.whatsoever' ones.
/etc/obs/$SERVICE.{public,private,...}
where $SERVICE is one of
api
webclient
scheduler
srcserver
reposerver
worker
sign
dispatcher
publisher
hermes
The '.public' files are simply copied to all machines that need to
know about this specific service.
for example the api talks to a local database which belongs just to
this api server, so this is configured in /etc/obs/api.private.
However the api hostname is needed by several services, so that's in
/etc/obs/api.public.
Another example are the workers. They need to register with
dispatchers and need to talk to srcservers and reposervers, so they
read the /etc/obs/{dispatcher,srcserver,reposerver}.public and find
all they need in there.
I'm unbiassed wether to use yaml or ini style config files,
lightweight and full parsers for each exist in all scripting
languages,
The one thing I know for sure though is that I hardly can clean up the
phletora of hard coded URLs and addresses all over the place without
breaking everything :(
S.
===================================================================
I did a little search on candidates that need to use the config. I've
looked for the usual suspects as host names, plus for static IPv4
addresses and manually cleaned up the list a bit:
cd /space/SVN/opensuse-forge/trunk/buildservice/src
svngrep -nE "HOST|(opensuse|suse|novell)\.(de|com|org)|[0-9]\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}" .
./sign/signd:27:my $signhost = '127.0.0.1';
./sign/signd:212:die("will not proxy to myself\n") if $signhost eq '1270.0.1' && $port eq $proxyport && !$phrases;
./sign/signd:338: die("unknown key: $user\n") if $signhost eq '127.0.01' && $port eq $proxyport;
./sign/sign.conf:2:user: build@suse.de
./sign/sign.c:2104: host = strdup("127.0.0.1");
./clientlib/perl/lib/OpenSUSE/BuildServiceAPI.pm:17:use constant APIHOST => 'api.opensuse.org';
./clientlib/perl/lib/OpenSUSE/BuildServiceAPI.pm:37: $self->apihost( $param{apihost} || APIHOST );
./clientlib/python/osc/osc_expand_link.pl:86:$cfg->{apiurl} ||= 'https://api.opensuse.org';
./clientlib/python/osc/osc/core.py:2993: result.append('http://download.opensuse.org/repositories/%s/' % project.replace(':', ':/'))
./clientlib/python/osc/osc/conf.py:20:{'apisrv': 'https://api.opensuse.org/',
./clientlib/python/osc/osc/conf.py:22: 'auth_dict': {'api.opensuse.org': {'user': 'poeml', 'pass': 'secret'},
./clientlib/python/osc/osc/conf.py:23: 'apitest.opensuse.org': {'user': 'poeml', 'pass': 'secret'},
./clientlib/python/osc/osc/conf.py:24: 'foo.opensuse.org': {'user': 'foo', 'pass': 'foo'}},
./clientlib/python/osc/osc/conf.py:29: 'urllist': ['http://download.opensuse.org/repositories/%(project)s/%(repository)s/%(arch)...',
./clientlib/python/osc/osc/conf.py:30: 'http://api.opensuseorg/rpm/%(project)s/%(repository)s/_repository/%(buildarc...'],
./clientlib/python/osc/osc/conf.py:42:DEFAULTS = { 'apisrv': 'https://api.opensuse.org/',
./clientlib/python/osc/osc/conf.py:54: 'http://download.opensuse.org/repositories/%(project)s/%(repository)s/%(arch)...',
./clientlib/python/osc/osc/commandline.py:903: metadata) on download.opensuse.org.
./clientlib/python/osc/osc/commandline.py:914: url_tmpl = 'http://download.opensuse.org/repositories/%s/%s/%s.repo'
./clientlib/python/osc/osc/commandline.py:1742: sys.stderr.write('See http://download.opensuse.org/repositories/openSUSE:/Tools/\n')
./tools/upstream/import.rb:30:puts ">> Logging into https://api.opensuse.org"
./tools/upstream/oscupstream.rb:15:$api_url="https://api.opensuse.org/"
./tools/upstream/oscupstream.rb:62: # activeServer holds value of "apisrv" or default "api.opensuse.org"
./tools/upstream/config.rb:4: 'host' => 'http://127.0.01:3001'
./tools/eclipse-plugin/org.opensuse.bs.newWizard/src/org/opensuse/bs/newWizard/core/NewWizard.java:154: buf.add("# URL to access API server, e.g. https://api.opensuse.org/");
./tools/eclipse-plugin/org.opensuse.bs.newWizard/src/org/opensuse/bs/newWizard/core/NewWizard.java:155: buf.add("# you also need a section [https://api.opensuse.org/] with the credentials");
./tools/eclipse-plugin/org.opensuse.bs.newWizard/src/org/opensuse/bs/newWizard/core/NewWizard.java:156: buf.add("#apisrv = https://api.opensuse.org/"); buf.add("");
./tools/eclipse-plugin/org.opensuse.bs.newWizard/src/org/opensuse/bs/newWizard/core/BSPage.java:62: "\"api.opensuse.org\". " +
./tools/eclipse-plugin/org.opensuse.bs.ui/src/org/opensuse/bs/wizards/ConnectionWizard.java:133: buf.add("# URL to access API server, e.g. https://api.opensuse.org/");
./tools/eclipse-plugin/org.opensuse.bs.ui/src/org/opensuse/bs/wizards/ConnectionWizard.java:134: buf.add("# you also need a section [https://api.opensuse.org/] with the credentials");
./tools/eclipse-plugin/org.opensuse.bs.ui/src/org/opensuse/bs/wizards/ConnectionWizard.java:135: buf.add("#apisrv = https://api.opensuse.org/"); buf.add("");
./tools/eclipse-plugin/org.opensuse.bs.ui/src/org/opensuse/bs/wizards/BSPage.java:62: "\"api.opensuse.org\". " +
./template-system/apbd.gemspec:19:#spec.hompage = 'http://en.opensuseorg/Abstract_Package_Build_Description'
./obsc/setup.py:27: url='https://forgesvn1.novell.com/svn/opensuse/trunk/buildservice/src/obsc/',
./README.devel:66: server: buildservice.suse.de
./README.devel:72: server: buildservice.suse.de (internal), api.opensuse.org (external)
./README.devel:78: server: buildservice.suse.de (internal), build.opensuse.org (external)
./frontend/vendor/plugins/exception_notification/lib/exception_notifiable.rb:36: addresses = [IPAddr.new("127.0.0.1")]
./frontend/vendor/plugins/exception_notification/README:47: consider_local "64.72.18.143", "14.17.21.25"
./frontend/vendor/plugins/exception_notification/README:52: consider_local "64.72.18.143/24"
./frontend/vendor/plugins/exception_notification/README:54:The address "127.0.0.1" is always considered local. If you want to completely
./frontend/vendor/plugins/exception_notification/README:55:reset the list of all addresses (for instance, if you wanted "127.0.0.1" to
./frontend/config/environments/production_slave.rb:23:# ICHAIN_HOST = "212.12.12.12"
./frontend/config/environments/production_slave.rb:25:SOURCE_HOST = "localhost"
./frontend/config/environments/development.L12.rb:18:SOURCE_HOST = "buildservice.suse.de"
./frontend/config/environments/development.L12.rb:19:#SOURCE_HOST = "localhost"
./frontend/config/environments/production.rb:23:ICHAIN_HOST = "212.1212.12"
./frontend/config/environments/production.rb:25:SOURCE_HOST = "storage"
./frontend/config/environments/stage.rb:22:ICHAIN_HOST = "212.12.12.12"
./frontend/config/environments/stage.rb:25:SOURCE_HOST = "storage"
./frontend/config/environments/development_base.rb:18:SOURCE_HOST = "buildservice.suse.de"
./frontend/config/environments/development_base.rb:19:#SOURCE_HOST = "localhost"
./frontend/config/environments/development_base.rb:26:# If ICHAIN_HOST is set to "simulate", iChain is simulated in the
./frontend/config/environments/development_base.rb:31:# This will replace the old constant ICHAIN_HOST.
./frontend/config/environments/test.rb:22:SOURCE_HOST = "localhost"
./frontend/config/lighttpd.conf:4:server.bind = "0.0.0.0"
./frontend/config/database.yml:21: host: 127.0.0.1
./frontend/config/database.yml:34: host: 127.0.0.1
./frontend/README:52:should be on and ".cgi" should respond. All requests from 127.0.0.1 go
./frontend/README:62:on requests from 127.0.0.1.
./frontend/app/controllers/public_controller.rb:111: response = Net::HTTP.start(SOURCE_HOST,SOURCE_PORT) do |http|
./frontend/app/views/ichain_notifier/approval.rhtml:6:http://build.opensuse.org/project/show?project=home%3A<%= @user.login %>
./frontend/app/views/ichain_notifier/approval.rhtml:13: - Ask on the mailinglist (opensuse-buildservice@opensuse.org) if you want to
./frontend/app/views/layouts/rbac.rhtml:24: <a href="http://build.opensuse.org">Build Service Web Client</a> |
./frontend/app/views/layouts/rbac.rhtml:27: <a href="http://www.opensuse.org">openSUSE Home</a>
./frontend/app/models/ichain_notifier.rb:6: from 'admin@opensuse.org'
./frontend/app/models/ichain_notifier.rb:15: from 'admin@opensuse.org'
./webclient/config/environments/production_slave.rb:23:FRONTEND_HOST = "127.0.42.2"
./webclient/config/environments/production_slave.rb:27:BUGZILLA_HOST = "http://my.bugzilla/"
./webclient/config/environments/production_slave.rb:28:DOWNLOAD_URL = "http://127.0.42.3/"
./webclient/config/environments/production.rb:23:FRONTEND_HOST = "api-internal.opensuse.org"
./webclient/config/environments/production.rb:27:BUGZILLA_HOST = "https://bugzilla.novell.com"
./webclient/config/environments/production.rb:28:DOWNLOAD_URL = "http://download.opensuse.org/repositories"
./webclient/config/environments/production.rb:36:### ICHAIN_HOST = "212.12.12.12"
./webclient/config/environments/stage.rb:18:#config.action_controller.asset_host = "https://build.opensuse.org/stage"
./webclient/config/environments/stage.rb:23:FRONTEND_HOST = "api-internal.opensuse.org"
./webclient/config/environments/stage.rb:27:BUGZILLA_HOST = "http://bugzilla.novell.com"
./webclient/config/environments/stage.rb:34:### ICHAIN_HOST = "212.1212.12"
./webclient/config/environments/development_base.rb:20:FRONTEND_HOST = "api.opensuse.org"
./webclient/config/environments/development_base.rb:25:BUGZILLA_HOST = nil
./webclient/config/environments/development_base.rb:32:# If ICHAIN_HOST is set to "simulate", iChain is simulated in the
./webclient/config/environments/development_base.rb:46:### ICHAIN_HOST = "212.12.12.12"
./webclient/config/environments/test.rb:21:FRONTEND_HOST = "localhost"
./webclient/README:52:should be on and ".cgi" should respond. All requests from 127.0.0.1 go
./webclient/README:62:on requests from 127.0.0.1.
./webclient/public/javascripts/prototype.js:1:/* Prototype JavaScript framework, version 1.6.0.1
./webclient/public/javascripts/prototype.js:10: Version: '1.6.0.1',
./webclient/public/css/local/contentadd_style.css:21: list-style-image:url(http://static.opensuse.org/c/css/common/images/liDot_double_darkgreypng);
./webclient/public/css/local/contentadd_style.css:30: background: #fff url(http://static.opensuse.org/hosts/build.o.o/css/images/darkgrey_searchbox/top...) top left no-repeat;
./webclient/public/css/local/contentadd_style.css:33: background: url(http://static.opensuse.org/hosts/build.o.o/css/images/darkgrey_searchbox/top...) top right no-repeat;
./webclient/public/css/local/contentadd_style.css:36: background: url(http://static.opensuse.org/hosts/build.o.o/css/images/darkgrey_searchbox/bot...) bottom left no-repeat;
./webclient/public/css/local/contentadd_style.css:39: background: url(http://static.opensuse.org/hosts/build.o.o/css/images/darkgrey_searchbox/bot...) bottom right no-repeat;
./webclient/public/css/local/style.css:10:@import url(http://static.opensuse.org/c/css/yaml/core/base.css);
./webclient/public/css/local/style.css:11:@import url(http://static.opensuse.org/c/css/common/basemod_opensuse.css);
./webclient/public/css/local/style.css:12:@import url(http://static.opensuse.org/c/css/yaml/screen/content_default.css);
./webclient/public/css/local/style.css:13:@import url(http://static.opensuse.org/c/css/common/contentmod_opensuse.css);
./webclient/public/css/local/style.css:14:@import url(http://static.opensuse.org/hosts/build.o.o/css/contentadd_style.css);
./webclient/public/css/local/style.css:15:@import url(http://static.opensuse.org/c/css/common/opensuse_boxes.css);
./webclient/public/css/local/style.css:16:@import url(http://static.opensuse.org/c/css/common/print/print_003.css);
./webclient/public/css/local/style.css:17:@import url(http://static.opensuse.org/hosts/build.o.o/css/patch.css); /* overrides old styles */
./webclient/public/css/local/style.css:18:@import url(http://static.opensuse.org/hosts/build.o.o/css/dialog.css); /* style for dialog boxes */
./webclient/app/helpers/application_helper.rb:77: "#{BUGZILLA_HOST}/enter_bug.cgi?classification=7340&product=openSUSE.org&component=3rd%20party%20software&assigned_to=#{email}&short_desc=#{desc}"
./webclient/app/views/main/index.rhtml:14:<%= link_to 'openSUSE Build Service', 'http://en.opensuse.org/Build_Service',
./webclient/app/views/main/index.rhtml:16:It can build packages for <a href="http://en.opensuse.org/Build_Service/cross_distribution_package_how_to">various Linux distributions.</a><br/>
./webclient/app/views/main/index.rhtml:19:<p class="description">Everyone can download the resulting software packages via the <a href="http://software.opensuse.org/search">software search and browsing interface</a>.</p>
./webclient/app/views/main/index.rhtml:67:<a href="mailto:opensuse-buildservice+subscribe@opensuse.org">
./webclient/app/views/main/index.rhtml:68:opensuse-buildservice@opensuse.org</a> mailing list.<br/>
./webclient/app/views/main/index.rhtml:69:We have also an active IRC channel <a href="irc://irc.opensuse.org/opensuse-buildservice">#opensuse-buildservice</a> running on Freenode.<br/>
./webclient/app/views/main/about.rhtml:6:<p>Check the openSUSE <a href="http://en.opensuse.org/Build_Service">
./webclient/app/views/main/accountinfo.rhtml:9:to <a href="mailto:admin@opensuse.org">the OpenSUSE Admin</a>.
./webclient/app/views/main/accountinfo.rhtml:14: <a href="http://www.opensuse.org">www.opensuse.org</a> (e.g. for Bugzilla), please
./webclient/app/views/project/new.rhtml:19:Please request a new project via <a href="https://bugzilla.novell.com/enter_bug.cgi?classification=7340&product=openSUSE.org&submit=Use+This+Product&component=OBS%20Request">bugzilla</a>.
./webclient/app/views/project/new.rhtml:20:You can also discuss the requested project setup on <a href="mailto:opensuse-buildservice@opensuse.org">opensuse-buildservice@opensuse.org</a>
./webclient/app/views/user/edit.rhtml:33: <%= hidden_field_tag( "source_host", "buildservice.suse.de" ) %>
./webclient/app/views/user/edit.rhtml:35: <%= hidden_field_tag( "rpm_host", "buildservice.suse.de" ) %>
./webclient/app/views/user/edit.rhtml:42: <%= hidden_field_tag( "source_host", "buildservice.suse.de" ) %>
./webclient/app/views/user/edit.rhtml:44: <%= hidden_field_tag( "rpm_host", "buildservice.suse.de" ) %>
./webclient/app/views/user/request_ichain.rhtml:13: <a href="mailto:opensuse-buildservice+subscribe@opensuse.org">subscribe</a> to the
./webclient/app/views/user/request_ichain.rhtml:14: <a href="http://lists.opensuse.org/opensuse-buildservice/">buildservice mailing list</a>.</p>
./webclient/app/views/user/request_ichain.rhtml:18: <a href="http://en.opensuse.org/Application_Black_List">http://en.opensuse.org/Application_Black_List</a>
./webclient/app/views/user/request_ichain.rhtml:23: <a href="http://software.opensuse.org/search/">http://software.opensuse.org/search/</a>
./webclient/app/views/layouts/application.rhtml:23: <form id="loginfrm" action="https://build.opensuse.org/ICSLogin/auth-up" method="post" enctype="application/x-www-form-urlencoded">
./webclient/app/views/layouts/application.rhtml:24: <input type="hidden" name="url" value="http://build.opensuse.org/privacy/ichain_login" />
./webclient/app/views/layouts/application.rhtml:38: <td><a href="https://secure-www.novell.com/selfreg/jsp/forgotPassword.jsp" onclick="forgotPassword();return false;">forgot password?</a></td>
./webclient/app/views/layouts/application.rhtml:70: { 'The Build Service' => 'http://en.opensuse.org/Build_Service' },
./webclient/app/views/layouts/application.rhtml:71: { 'Building Packages' => 'http://en.opensuse.org/SUSE_Build_Tutorial' },
./webclient/app/views/layouts/application.rhtml:72: { 'Writing spec Files' => 'http://forgeftp.novell.com//library/SUSE%20Package%20Conventions/spc.html' },
./webclient/app/views/layouts/application.rhtml:73: { 'Reporting a Bug' => 'http://en.opensuse.org/Submit_a_Bug' }
./webclient/app/views/layouts/application.rhtml:115: <a href="http://en.opensuse.org/Legal">Legal Information</a> |
./webclient/app/views/layouts/application.rhtml:116: <a href="mailto:webmaster@opensuse.org">Feedback</a>
./webclient/app/views/layouts/application2.rhtml:11: <link href="http://static.opensuse.org/hosts/build.o.o/css/style.css" rel="stylesheet" type="text/css"/>
./webclient/app/views/layouts/application2.rhtml:12: <link href="http://static.opensuse.org/hosts/build.o.o/images/favicon.ico" rel="shortcut icon" />
./webclient/app/views/layouts/application2.rhtml:14: <link href="http://static.opensuse.org/c/css/common/iehacks_opensuse.css" rel="stylesheet" type="text/css"/>
./webclient/app/views/layouts/application2.rhtml:15: <link href="http://static.opensuse.org/hosts/build.o.o/css/iehacks.css" rel="stylesheet" type="text/css"/>
./webclient/app/views/layouts/application2.rhtml:33: <a href="http://build.opensuse.org"><img src="http://static.opensuse.org/c/images/common/geeko.jpg" alt="openSUSE" /></a>
./webclient/app/views/layouts/application2.rhtml:55: <li style="list-style-image: url(http://static.opensuse.org/c/css/common/images/liDot_download.png)"><a href="http://software.opensuse.org/">Get Software</a></li>
./webclient/app/views/layouts/application2.rhtml:56: <li style="list-style-image: url(http://static.opensuse.org/c/css/common/images/liDot_wiki.png)"><a href="http://en.opensuse.org/">Wiki</a></li>
./webclient/app/views/layouts/application2.rhtml:57: <li style="list-style-image: url(http://static.opensuse.org/c/css/common/images/liDot_build.png)"><a href="http://build.opensuse.org/">Build Software</a></li>
./webclient/app/views/layouts/application2.rhtml:58: <li style="list-style-image: url(http://static.opensuse.org/c/css/common/images/liDot_community.png)"><a href="http://users.opensuse.org/">User Directory</a></li>
./webclient/app/views/layouts/application2.rhtml:59: <li style="list-style-image: url(http://static.opensuse.org/c/css/common/images/liDot_wiki.png)"><a href="http://news.opensuse.org/">News</a></li>
./webclient/app/views/layouts/application2.rhtml:60: <li style="list-style-image: url(http://static.opensuse.org/c/css/common/images/liDot_wiki.png)"><a href="http://forums.opensuse.org">Forums</a></li>
./webclient/app/views/layouts/application2.rhtml:61: <li style="list-style-image: url(http://static.opensuse.org/c/css/common/images/liDot_wiki.png)"><a href="http://shop.opensuse.org/">Shop</a></li>
./webclient/app/views/layouts/application2.rhtml:146: { 'The Build Service' => 'http://en.opensuse.org/Build_Service' },
./webclient/app/views/layouts/application2.rhtml:147: { 'Building Packages' => 'http://en.opensuse.org/SUSE_Build_Tutorial' },
./webclient/app/views/layouts/application2.rhtml:148: { 'Writing spec Files' => 'http://en.opensuse.org/SUSE_Build_Tutorial#Specfiles' },
./webclient/app/views/layouts/application2.rhtml:149: { 'Reporting a Bug' => 'http://en.opensuse.org/Submit_a_Bug' }
./webclient/app/views/layouts/application2.rhtml:256: <option value="http://de.test.opensuse.org/Main_Page">Deutsch</option>
./webclient/app/views/layouts/application2.rhtml:257: <option value="http://fr.test.opensuse.org/Main_Page">Français</option>
./webclient/app/views/layouts/application2.rhtml:296: <a href="http://en.opensuse.org/openSUSE:About">About openSUSE</a> | <a href="http://en.opensuse.org/Legal">Legal information</a> | <a href="mailto:webmaster@opensuse.org">Feedback</a>
./webclient/app/views/layouts/application2.rhtml:299: <a href="http://www.novell.com/linux/"><img src="http://static.opensuse.org/c/images/common/founded_novell.gif" alt="Founded by Novell" /></a>
./webclient/app/views/layouts/application2.rhtml:326: <a href="http://en.opensuse.org/Sponsors"><img src="http://files.opensuse.org/opensuse/en/5/54/Amd.png" title="Sponsored by AMD" alt="Sponsored by AMD"/></a>
./webclient/app/views/layouts/application2.rhtml:327: <a href="http://en.opensuse.org/Sponsors"><img title="Sponsored by: IP Exchange" alt="Sponsored by: IP Exchange" src="http://files.opensuse.org/opensuse/en/b/b6/Ip-exchange.gif"/></a>
./tester/tests/svn_helper:4:svn_remote = "https://forgesvn1.novell.com/svn/opensuse/trunk/buildservice/"
./tester/tests/101.test:9:curl build.opensuse.org >/dev/null
./tester/tests/100.test:9:curl api.opensuse.org
./common/lib/switchtower/configuration.rb:10:set :repository, proc {"--username #{svnuser} https://forgesvn1.novell.com/svn/opensuse/trunk/buildservice/src/#{application}"}
./common/lib/switchtower/configuration.rb:15:set :cl_mail_from, "Automated Deploy Message