This forum is in READ-ONLY mode.
You can look around, but if you want to ask a new question, please use Stack Overflow.

Lucene Order By Score

Questions relating to sfLucenePlugin, sfSearch etc

Lucene Order By Score

by FeelLikeANut » Sat Apr 17, 2010 2:08 am

Hi. I was wondering if anyone knew a good trick for ordering doctrine results by order of a Lucene score.

Right now I have something like...

Code: Select all
    $pks = array();
    foreach ($this->getLuceneIndex()->find($query) as $hit)
    {
      $pks[] = $hit->pk;
    }

    return $this->createQuery('a')
           ->whereIn('a.id', $pks);


How can I get the results in the same order as the IDs in the $pks array?
FeelLikeANut
Member
 
Posts: 85
Joined: Tue Mar 02, 2010 6:48 pm

Re: Lucene Order By Score

by erisds » Thu May 13, 2010 12:29 pm

I am also looking for a way to do this.

The Symfony Jobeet tutorial provides code like what you have shown above, where you first get the Lucene hits, translate them to Symfony Primary Keys and use a WHERE IN sql query to get the results.

The problem, as you have noted is that WHERE IN does not honour the order of the PKs in the IN clause, and such the relevance information from Lucene is totally lost.

As I understand it, this is not how Lucene is intended to be used. This very old article is very similar to the Jobeet tutorial, but instead uses Lucene's stored values to generate the results list in the correct order.

So, instead of fetching the objects based on the PKs using a WHERE IN clause, you change your updateLuceneIndex() function to store the values you want to return in your search results like this:

Code: Select all
$doc->addField(Zend_Search_Lucene_Field::Text('title', $this->getTitle(), 'utf-8'));
$doc->addField(Zend_Search_Lucene_Field::Text('excerpt', $this->getExcerpt(), 'utf-8'));
$doc->addField(Zend_Search_Lucene_Field::Text('date', $this->getCreatedAt('l, jS F, Y'), 'utf-8'));
$doc->addField(Zend_Search_Lucene_Field::UnStored('body', $this->getBody(), 'utf-8'));


Here I am storing the title, excerpt and date of a news/blog item in the lucene index, but using UnStored for the body as I do not need this.

Then your getForLuceneQuery function becomes the following:

Code: Select all
static public function getForLuceneQuery($query)
{
  $hits = self::getLuceneIndex()->find($query);
 
  return $hits;
}


And in your template you can echo out the values direct from the Lucene hit object:

Code: Select all
<?php foreach($hits as $hit): ?>
<h2><?php echo $hit->title; ?></h2>
<p class="date><?php echo $hit->date; ?></p>
<p><?php echo $hit->excerpt; ?></p>
<?php endforeach; ?>


This works in the case I am using it at the moment, because I have a small data set and don't want to put too much info into the search result listing.

However I have another case with a much larger dataset and where I also want to display lots of information, including thumbnails in the search results list and so this is a less appropriate solution.

So if anyone knows of a clever way of fetching the Symfony objects in the correct order, I'd super appreciate some help!
erisds
Junior Member
 
Posts: 8
Joined: Fri Apr 16, 2010 4:02 pm

Re: Lucene Order By Score

by conrad » Tue Jul 20, 2010 10:16 pm

I'm having the same trouble with lucene and would like to use the doctrine collection for the output.. this is a little approach of an idea I had - definately not perfect and quite dowdy.. but it works, at least with an array holding doctrine objects

Code: Select all
    public function getForLuceneQuery($query)
    {
      // find hits
      $hits = self::getLuceneIndex()->find($query);
      // fetch primary keys of result for db query
      $pks    = array();
      $score  = array();
      foreach ($hits as  $i => $hit) {
        // pks for doctrine query
        $pks[]    = $hit->pk;
        // save the score temporarily
        // COMMENT: another Datatype than String?
        $score[]  = (String)$hit->score;
      }

      if (empty($pks)) {
        return array();
      }

      // get result
      $q = $this->createQuery('Seminare s')
        ->whereIn('s.id', $pks)
        ->leftJoin('s.Rubriken r');

      $resTmp = $q->execute();

      // run through hits again to add it to the final result
      $res = array();
      foreach($hits as $i => $hit) {
        // fetch doctrine object from collection
        $object = $resTmp->get($i);
        // store the lucene reference
        // (added it to the class, not neccessary)
        $object->lucene = $hit;
        // probably a lucene score exists twice or more often, so
        // add a new number to the end to avoid losing one entity
        // COMMENT: would decrementing be better?
        $resKey = $score[$i];
        $k = 0;
        do {
          $resKey .= $k;
          $k++;
        }while(array_key_exists($resKey, $res));
        // finally add the object to the array with the new key
        $res[$resKey] = $object;
      }
      // sort by keys ASC to have the appropriate order again
      krsort($res, SORT_NUMERIC);

      return $res;
    }


comments?
conrad
Junior Member
 
Posts: 17
Joined: Mon Jul 27, 2009 9:37 pm
Location: Düsseldorf, Germany

Re: Lucene Order By Score

by trontank » Tue Jul 27, 2010 11:13 am

trontank
Senior Member
 
Posts: 178
Joined: Thu Jul 31, 2008 2:18 pm
Location: Germany

Re: Lucene Order By Score

by conrad » Tue Jul 27, 2010 11:49 am

definitely a more sophisticated solution. thank you.
conrad
Junior Member
 
Posts: 17
Joined: Mon Jul 27, 2009 9:37 pm
Location: Düsseldorf, Germany