[opensuse-wiki] Lucene Search Update

Matthew Ehle

31 Aug 2010 31 Aug '10

15:53

Good Day, So here is an update on https://bugzilla.novell.com/show_bug.cgi?id=625677. After contacting the author about the spelling suggestion issue I was having, I finally got it to work. As an added bonus, I have discovered and enabled the related articles feature as well. With these features working, Lucene is definitely a better option than Sphinx now. All the "new" wikis on stage (en, dewiki, and languages) are now using Lucene search. The enstage wiki has the most content of the three, so you can get the best idea of how well it works by going there. However, those of you who are fluent in German should also try dewiki, so that you can see how well the search deals with German stemming and stopwords. The indexing takes 45-50 seconds on stage, so I'm expecting it to take about a minute to index the live English wiki, and probably another 30 seconds for the production dewiki and languages. Once everything is transitioned over to the new wiki structure, total indexing will probably be in the 10 minute range. I plan to put the indexer on a daily cron, so all changes should show up in the search within 24 hours. I am setting up a meeting with the data center to go over an OS upgrade plan so that we can get this going as soon as possible. -Matt

Attachments:

attachment.htm (text/html — 1.7 KB)

Show replies by date

pistazienfresser

13 Sep 13 Sep

18:14

-------- Original-Nachricht --------

...

Datum: Tue, 31 Aug 2010 09:53:48 -0600 Von: "Matthew Ehle" An: opensuse-wiki@opensuse.org Betreff: [opensuse-wiki] Lucene Search Update

...

Good Day,

So here is an update on https://bugzilla.novell.com/show_bug.cgi?id=625677. After contacting the author about the spelling suggestion issue I was having, I finally got it to work. As an added bonus, I have discovered and enabled the related articles feature as well. With these features working, Lucene is definitely a better option than Sphinx now.

All the "new" wikis on stage (en, dewiki, and languages) are now using Lucene search. The enstage wiki has the most content of the three, so you can get the best idea of how well it works by going there. However, those of you who are fluent in German should also try dewiki, so that you can see how well the search deals with German stemming and stopwords.

The indexing takes 45-50 seconds on stage, so I'm expecting it to take about a minute to index the live English wiki, and probably another 30 seconds for the production dewiki and languages. Once everything is transitioned over to the new wiki structure, total indexing will probably be in the 10 minute range. I plan to put the indexer on a daily cron, so all changes should show up in the search within 24 hours.

I am setting up a meeting with the data center to go over an OS upgrade plan so that we can get this going as soon as possible.

-Matt

http://en.opensuse.org/MediaWiki:Noarticletext Hello Matthew, hello all, could responding link on the default page that is shown as no article is there for the given URL be changed also from '''[[Special:Search/all:{{PAGENAME}}|search for this article]]''' to '''[[Special:Search/all:{{PAGENAME}}-|search for this article]]''' or something like that that makes a fuzzy search? Regards pistazienfresser (Martin) -- Achtung Sicherheitswarnung: GMX warnt vor Phishing-Attacken! http://portal.gmx.net/de/go/sicherheitspaket -- To unsubscribe, e-mail: opensuse-wiki+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-wiki+help@opensuse.org

pistazienfresser (see profile)

14 Sep 14 Sep

15:09

On 13/09/10 20:14, pistazienfresser wrote:

...

[...]

...

http://en.opensuse.org/MediaWiki:Noarticletext

Hello Matthew, hello all,

could responding link on the default page that is shown as no article is there for the given URL be changed also from '''[[Special:Search/all:{{PAGENAME}}|search for this article]]''' to '''[[Special:Search/all:{{PAGENAME}}-|search for this article]]'''

meant: "~" -> [[Special:Search/all:{{PAGENAME}}~|search for this article]] like http://en.opensuse.org/Special:Search/all:Nvidia~ pistazienfresser - openSUSE profile: https://users.opensuse.org/show/pistazienfresser -- To unsubscribe, e-mail: opensuse-wiki+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-wiki+help@opensuse.org

Tim Mohlmann

4 Oct 4 Oct

18:56

New subject: [opensuse-wiki] Search results layout (WAS:Lucene Search Update)

Guys, I just see that the layout of search results has changed. It is quite annoying I have to scroll down to see the first search result. My first impression was that there are no search results at all. In the old situation the additional search options are below the results, now they are on top. Can we go back to the old layout? http://en.opensuse.org/Portal_Talk:Wiki for screenshot ;) Greetings, Tim On Tuesday 14 September 2010 17:09:11 pistazienfresser (see profile) wrote:

...

On 13/09/10 20:14, pistazienfresser wrote:

[...]

...
http://en.opensuse.org/MediaWiki:Noarticletext

Hello Matthew, hello all,

could responding link on the default page that is shown as no article is there for the given URL be changed also from '''[[Special:Search/all:{{PAGENAME}}|search for this article]]''' to '''[[Special:Search/all:{{PAGENAME}}-|search for this article]]'''

meant: "~" -> [[Special:Search/all:{{PAGENAME}}~|search for this article]]

like

http://en.opensuse.org/Special:Search/all:Nvidia~

pistazienfresser

- openSUSE profile: https://users.opensuse.org/show/pistazienfresser -- To unsubscribe, e-mail: opensuse-wiki+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-wiki+help@opensuse.org

pistazienfresser (see profile)

6 Oct 6 Oct

12:13

New subject: [opensuse-wiki] Search results layout (WAS:Lucene Search Update)

Am 04/10/10 20:56, schrieb Tim Mohlmann:

...

Guys,

I just see that the layout of search results has changed. It is quite annoying I have to scroll down to see the first search result. My first impression was that there are no search results at all.

In the old situation the additional search options are below the results, now they are on top. Can we go back to the old layout?

http://en.opensuse.org/Portal_Talk:Wiki for screenshot ;)

Greetings, Tim

+1 Regards pistazienfresser -- - openSUSE profile: https://users.opensuse.org/show/pistazienfresser -- To unsubscribe, e-mail: opensuse-wiki+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-wiki+help@opensuse.org

Rajko M.

23:36

New subject: [opensuse-wiki] Search results layout (WAS:Lucene Search Update)

On Wednesday 06 October 2010 07:13:46 pistazienfresser (see profile) wrote:

...

Am 04/10/10 20:56, schrieb Tim Mohlmann:

...
Guys,

I just see that the layout of search results has changed. It is quite annoying I have to scroll down to see the first search result. My first impression was that there are no search results at all. ... http://en.opensuse.org/Portal_Talk:Wiki for screenshot ;)

Greetings, Tim

+1 Regards pistazienfresser

What about search like Wikipedia one? http://en.wikipedia.org/w/index.php?title=Special:Search&search= or http://en.wikipedia.org/w/index.php?title=Special:Search&fulltext=Search&advanced=1&ns0=1&redirs=0&search= It is just different order of elements. Although, it seems that is part of the vector skin that is not even enabled in our wiki. Matthew can you do something about ^^^^^^^^^^ :) -- Regards, Rajko -- To unsubscribe, e-mail: opensuse-wiki+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-wiki+help@opensuse.org

Matthew Ehle

7 Oct 7 Oct

00:03

New subject: [opensuse-wiki] Search results layout (WAS:Lucene Search Update)

...

...
...
...
"Rajko M." 10/6/2010 5:36 PM >>> On Wednesday 06 October 2010 07:13:46 pistazienfresser (see profile) wrote: Am 04/10/10 20:56, schrieb Tim Mohlmann: Guys,

I just see that the layout of search results has changed. It is quite annoying I have to scroll down to see the first search result. My first impression was that there are no search results at all. ... http://en.opensuse.org/Portal_Talk:Wiki for screenshot ;)

Greetings, Tim

+1 Regards pistazienfresser

What about search like Wikipedia one?

http://en.wikipedia.org/w/index.php?title=Special:Search&search= or title=Special:Search&fulltext=Search&advanced=1&ns0=1&redirs=0&http://en.wikipedia.org/w/index.php?>title=Special:Search&fulltext=Search&advanced=1&ns0=1&redirs=0&search= ( http://en.wikipedia.org/w/index.php? )

It is just different order of elements.

Although, it seems that is part of the vector skin that is not even enabled in our wiki. Matthew can you do something about ^^^^^^^^^^ :)

I'm a little confused about what's being requested. It sounds like an update needs to be made to the bento theme to handle styles the search result divs. Is there some specific setting that also needs to be enabled for MWSearch?

Rajko M.

02:01

New subject: [opensuse-wiki] Search results layout (WAS:Lucene Search Update)

On Wednesday 06 October 2010 19:03:33 Matthew Ehle wrote: ...

...

...
Although, it seems that is part of the vector skin that is not even enabled in our wiki.

Matthew can you do something about ^^^^^^^^^^ :)

I'm a little confused about what's being requested. It sounds like an update needs to be made to the bento theme to handle styles the search result divs. Is there some specific setting that also needs to be enabled for MWSearch?

The Wikipedia default skin now is Vector that is not included in our wiki. Whatever we do, we have to know what is done in Vector skin so that we can replicate that with bentofluid. The old default is MonoBook, which is enabled, but I can guess that it will be updated only when someone has nothing better to do. I can't see any change in http://svn.berlios.de/ The vector skin is not listed to be skipped, but it is not showing up as available skins, so it is skipped in actual settings used on the server :) -- Regards, Rajko -- To unsubscribe, e-mail: opensuse-wiki+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-wiki+help@opensuse.org

Thomas Schmidt

07:08

New subject: [opensuse-wiki] Search results layout (WAS:Lucene Search Update)

On 10/07/2010 04:01 AM, Rajko M. wrote:

...

On Wednesday 06 October 2010 19:03:33 Matthew Ehle wrote: ...

...
...
Although, it seems that is part of the vector skin that is not even enabled in our wiki.

Matthew can you do something about ^^^^^^^^^^ :)

I'm a little confused about what's being requested. It sounds like an update needs to be made to the bento theme to handle styles the search result divs. Is there some specific setting that also needs to be enabled for MWSearch?

The Wikipedia default skin now is Vector that is not included in our wiki. Whatever we do, we have to know what is done in Vector skin so that we can replicate that with bentofluid. The old default is MonoBook, which is enabled, but I can guess that it will be updated only when someone has nothing better to do.

I can't see any change in http://svn.berlios.de/ The vector skin is not listed to be skipped, but it is not showing up as available skins, so it is skipped in actual settings used on the server :)

We don't need the vector skin, we just need a better styling of the namespace list. I can have look into this. Greetings -- Thomas Schmidt (tom [at] opensuse.org) openSUSE Boosters Team "Don't Panic", Douglas Adams (1952 - 11.05.2001) -- To unsubscribe, e-mail: opensuse-wiki+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-wiki+help@opensuse.org

Thomas Schmidt

07:03

New subject: [opensuse-wiki] Search results layout (WAS:Lucene Search Update)

On 10/04/2010 08:56 PM, Tim Mohlmann wrote:

...

Guys,

I just see that the layout of search results has changed. It is quite annoying I have to scroll down to see the first search result. My first impression was that there are no search results at all.

In the old situation the additional search options are below the results, now they are on top. Can we go back to the old layout?

http://en.opensuse.org/Portal_Talk:Wiki for screenshot ;)

The options are only shown on top if you click on 'everything', aren't they? Greetings -- Thomas Schmidt (tom [at] opensuse.org) openSUSE Boosters Team "Don't Panic", Douglas Adams (1952 - 11.05.2001) -- To unsubscribe, e-mail: opensuse-wiki+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-wiki+help@opensuse.org

Thomas Schmidt

07:05

New subject: [opensuse-wiki] Search results layout (WAS:Lucene Search Update)

On 10/07/2010 09:03 AM, Thomas Schmidt wrote:

...

On 10/04/2010 08:56 PM, Tim Mohlmann wrote:

...
Guys,

I just see that the layout of search results has changed. It is quite annoying I have to scroll down to see the first search result. My first impression was that there are no search results at all.

In the old situation the additional search options are below the results, now they are on top. Can we go back to the old layout?

http://en.opensuse.org/Portal_Talk:Wiki for screenshot ;)

The options are only shown on top if you click on 'everything', aren't they?

Sorry, 'advanced' I meant. -- Thomas Schmidt (tom [at] opensuse.org) openSUSE Boosters Team "Don't Panic", Douglas Adams (1952 - 11.05.2001) -- To unsubscribe, e-mail: opensuse-wiki+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-wiki+help@opensuse.org

Rajko M.

10:47

New subject: [opensuse-wiki] Search results layout (WAS:Lucene Search Update)

On Thursday 07 October 2010 02:05:38 Thomas Schmidt wrote:

...

...
The options are only shown on top if you click on 'everything', aren't they?

Sorry, 'advanced' I meant.

It is shown any time advanced will be shown, no need to click advanced link. Search for Build_Service_KIWI and you will see them. That is the reason it is squeezed on Wikipedia as much as possible. BTW, I have Vector on my local instance and the layout is the same as what can be seen on Wikipedia, so it is just css issue. I would look at, if css would not have few layers each covering previous, so debugging is not really simple with Firefox extension Web Development. (Any better tools?) Just as reminder we have some paragraph rendering issues, table not rendering borders issue, and probably more that are not solved for quite some time. Which lead me to think, why we need skin that is so far from mainstream, that makes debugging possible only for highly trained people, that have no time to do that? -- Regards, Rajko -- To unsubscribe, e-mail: opensuse-wiki+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-wiki+help@opensuse.org

Thomas Schmidt

11:25

New subject: [opensuse-wiki] Search results layout (WAS:Lucene Search Update)

On 07.10.2010 12:47, Rajko M. wrote:

...

On Thursday 07 October 2010 02:05:38 Thomas Schmidt wrote:

...
...
The options are only shown on top if you click on 'everything', aren't they?

Sorry, 'advanced' I meant.

It is shown any time advanced will be shown, no need to click advanced link.

Search for Build_Service_KIWI and you will see them. That is the reason it is squeezed on Wikipedia as much as possible.

BTW, I have Vector on my local instance and the layout is the same as what can be seen on Wikipedia, so it is just css issue.

I would look at, if css would not have few layers each covering previous, so debugging is not really simple with Firefox extension Web Development. (Any better tools?)

I am using Firebug. It makes debugging multiple layers of css quite easy, and you can check your changes on the fly on the page.

...

Just as reminder we have some paragraph rendering issues, table not rendering borders issue, and probably more that are not solved for quite some time. Which lead me to think, why we need skin that is so far from mainstream, that makes debugging possible only for highly trained people, that have no time to do that?

We use the bento theme to have a consistant look and navigation for all opensuse pages. It would be cool if you could help enhancing the css layout of the advanced search page. Greetings -- Thomas Schmidt (tom [at] opensuse.org) openSUSE Boosters Team "Don't Panic", Douglas Adams (1952 - 11.05.2001) -- To unsubscribe, e-mail: opensuse-wiki+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-wiki+help@opensuse.org

Thomas Schmidt

09:46

On 14.09.2010 17:09, pistazienfresser (see profile) wrote:

...

On 13/09/10 20:14, pistazienfresser wrote:

...
[...]

...
http://en.opensuse.org/MediaWiki:Noarticletext

Hello Matthew, hello all,

could responding link on the default page that is shown as no article is there for the given URL be changed also from '''[[Special:Search/all:{{PAGENAME}}|search for this article]]''' to '''[[Special:Search/all:{{PAGENAME}}-|search for this article]]'''

meant: "~" -> [[Special:Search/all:{{PAGENAME}}~|search for this article]]

like

http://en.opensuse.org/Special:Search/all:Nvidia~

The ~ for fuzzy search is added now. Greetings -- Thomas Schmidt (tom [at] opensuse.org) openSUSE Boosters Team "Don't Panic", Douglas Adams (1952 - 11.05.2001) -- To unsubscribe, e-mail: opensuse-wiki+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-wiki+help@opensuse.org

4951

Age (days ago)

4988

Last active (days ago)

List overview

Download

13 comments

6 participants

participants (6)

Matthew Ehle
pistazienfresser
pistazienfresser (see profile)
Rajko M.
Thomas Schmidt
Tim Mohlmann