课外天地 李树青学习天地信息检索原理课件 → 程序代码——利用Lucene实现的空间向量查询方法


  共有23005人关注过本帖树形打印复制链接

主题:程序代码——利用Lucene实现的空间向量查询方法

帅哥哟,离线,有人找我吗?
admin
  1楼 博客 | 信息 | 搜索 | 邮箱 | 主页 | UC


加好友 发短信 管理员
等级:管理员 帖子:1940 积分:26616 威望:0 精华:34 注册:2003/12/30 16:34:32
程序代码——利用Lucene实现的空间向量查询方法  发帖心情 Post By:2011/3/28 18:54:38 [只看该作者]

1、查看向量
import java.io.IOException;

import org.apache.lucene.analysis.WhitespaceAnalyzer;
import org.apache.lucene.document.Document;
import org.apache.lucene.document.Field;
import org.apache.lucene.index.IndexReader;
import org.apache.lucene.index.IndexWriter;
import org.apache.lucene.index.TermFreqVector;
import org.apache.lucene.store.Directory;
import org.apache.lucene.store.RAMDirectory;

public class Exec {
        public static void main(String[] args) throws IOException {
                IndexReader reader;
                Directory directory = new RAMDirectory();
                IndexWriter writer = new IndexWriter(directory,
                                new WhitespaceAnalyzer(), true);
                writer.setUseCompoundFile(true);

                String[] titles = new String[] { "Modern Art of Education",
                                "Imperial Secrets of Health", "Tao Te Ching",
                                "G.del, Escher, Bach", "Mindstorms",
                                "Java Development with Ant", "JUnit in Action",
                                "Lucene in Action", "Tapestry in Action",
                                "Extreme Programming Explained", "The Pragmatic Programmer" };
                String[] pubmonths = new String[] { "198106", "199401", "198810",
                                "197903", "198001", "200208", "200310", "200406", "199910",
                                "200403", "199910" };
                String[] categories = new String[] { "/education/pedagogy",
                                "/health/alternative/chinese", "/philosophy/eastern",
                                "/technology/computers/ai",
                                "/technology/computers/programming/education",
                                "/technology/computers/programming",
                                "/technology/computers/programming",
                                "/technology/computers/programming",
                                "/technology/computers/programming",
                                "/technology/computers/programming/methodology",
                                "/technology/computers/programming" };
                String[] subjects = new String[] {
                                "education philosophy psychology practice Waldorf",
                                "diet chinese medicine qi gong health herbs",
                                "taoism",
                                "artificial intelligence number theory mathematics music",
                                "children computers powerful ideas LOGO education",
                                "apache jakarta ant build tool junit java development",
                                "junit unit testing mock objects",
                                "lucene search with lucene",
                                "tapestry web user interface components",
                                "extreme programming agile test driven development methodology",
                                "pragmatic agile methodology developer tools" };

                for (int i = 0; i < titles.length; i++) {
                        Document doc = new Document();
                        doc.add(Field.Keyword("titles", titles[i]));
                        doc.add(Field.Text("pubmonths", pubmonths[i]));
                        doc.add(Field.Text("categories", categories[i]));
                        doc.add(Field.Text("subjects", subjects[i], true));
                        writer.addDocument(doc);
                }
                writer.optimize();
                writer.close();

                // Reader
                reader = IndexReader.open(directory);
                int numDocs = reader.maxDoc();

                for (int i = 0; i < numDocs; i++) {
                        System.out.println();
                        Document doc = reader.document(i);
                        System.out.println(doc.get("titles"));
                        TermFreqVector vector = reader.getTermFreqVector(i, "subjects");
                        System.out.println(vector);
                }
        }
}


2、使用向量
import java.io.IOException;

import org.apache.lucene.analysis.WhitespaceAnalyzer;
import org.apache.lucene.document.Document;
import org.apache.lucene.document.Field;
import org.apache.lucene.index.IndexReader;
import org.apache.lucene.index.IndexWriter;
import org.apache.lucene.index.Term;
import org.apache.lucene.index.TermFreqVector;
import org.apache.lucene.search.BooleanQuery;
import org.apache.lucene.search.Hits;
import org.apache.lucene.search.IndexSearcher;
import org.apache.lucene.search.TermQuery;
import org.apache.lucene.store.Directory;
import org.apache.lucene.store.RAMDirectory;

public class Exec {
        public static Document[] docsLike(IndexReader reader,
                        IndexSearcher searcher, int id, int max) throws IOException {
                Document doc = reader.document(id);
                String[] categories = doc.getValues("categories");
                BooleanQuery categoriesQuery = new BooleanQuery();
                for (int i = 0; i < categories.length; i++) {
                        String category = categories[i];
                        categoriesQuery.add(
                                        new TermQuery(new Term("categories", category)), false,
                                        false);
                }
                categoriesQuery.setBoost(2.0f);

                TermFreqVector vector = reader.getTermFreqVector(id, "subjects");

                BooleanQuery subjectQuery = new BooleanQuery();
                for (int j = 0; j < vector.size(); j++) {
                        TermQuery tq = new TermQuery(new Term("subjects",
                                        vector.getTerms()[j]));
                        subjectQuery.add(tq, false, false);
                }

                BooleanQuery likeThisQuery = new BooleanQuery();
                likeThisQuery.add(categoriesQuery, false, false);
                likeThisQuery.add(subjectQuery, false, false);

                // exclude myself
                likeThisQuery.add(new TermQuery(new Term("isbns", doc.get("isbns"))),
                                false, true);

                // System.out.println(" Query: " + likeThisQuery.toString());
                Hits hits = searcher.search(likeThisQuery);
                int size = max;
                if (max > hits.length())
                        size = hits.length();

                Document[] docs = new Document[size];
                for (int i = 0; i < size; i++) {
                        docs[i] = hits.doc(i);
                }
                return docs;
        }

        public static void main(String[] args) throws IOException {
                IndexReader reader;
                IndexSearcher searcher;

                // Writer
                Directory directory = new RAMDirectory();
                IndexWriter writer = new IndexWriter(directory,
                                new WhitespaceAnalyzer(), true);
                writer.setUseCompoundFile(true);

                String[] isbns = new String[] { "1", "2", "3", "4", "5", "6", "7", "8",
                                "9", "10", "11" };

                String[] titles = new String[] { "Modern Art of Education",
                                "Imperial Secrets of Health", "Tao Te Ching",
                                "G.del, Escher, Bach", "Mindstorms",
                                "Java Development with Ant", "JUnit in Action",
                                "Lucene in Action", "Tapestry in Action",
                                "Extreme Programming Explained", "The Pragmatic Programmer" };
                String[] pubmonths = new String[] { "198106", "199401", "198810",
                                "197903", "198001", "200208", "200310", "200406", "199910",
                                "200403", "199910" };
                String[] categories = new String[] { "/education/pedagogy",
                                "/health/alternative/chinese", "/philosophy/eastern",
                                "/technology/computers/ai",
                                "/technology/computers/programming/education",
                                "/technology/computers/programming",
                                "/technology/computers/programming",
                                "/technology/computers/programming",
                                "/technology/computers/programming",
                                "/technology/computers/programming/methodology",
                                "/technology/computers/programming" };
                String[] subjects = new String[] {
                                "education philosophy psychology practice Waldorf",
                                "diet chinese medicine qi gong health herbs",
                                "taoism",
                                "artificial intelligence number theory mathematics music",
                                "children computers powerful ideas LOGO education",
                                "apache jakarta ant build tool junit java development",
                                "junit unit testing mock objects",
                                "lucene search",
                                "tapestry web user interface components",
                                "extreme programming agile test driven development methodology",
                                "pragmatic agile methodology developer tools" };

                for (int i = 0; i < titles.length; i++) {
                        Document doc = new Document();
                        doc.add(Field.Keyword("isbns", isbns[i]));
                        doc.add(Field.Text("titles", titles[i]));
                        doc.add(Field.Text("pubmonths", pubmonths[i]));
                        doc.add(Field.Text("categories", categories[i]));
                        doc.add(Field.Text("subjects", subjects[i], true));
                        writer.addDocument(doc);
                }
                writer.optimize();
                writer.close();

                // Reader
                reader = IndexReader.open(directory);
                searcher = new IndexSearcher(reader);
                int numDocs = reader.maxDoc();

                for (int i = 0; i < numDocs; i++) {
                        System.out.println();
                        Document doc = reader.document(i);
                        System.out.println(doc.get("titles"));

                        Document[] docs = docsLike(reader, searcher, i, 10);
                        if (docs.length == 0) {
                                System.out.println("  None like this");
                        }
                        for (int j = 0; j < docs.length; j++) {
                                Document likeThisDoc = docs[j];
                                System.out.println("  -> " + likeThisDoc.get("titles"));
                        }
                }
        }
}

3、使用向量结合余弦夹角计算方法
import java.io.IOException;

import org.apache.lucene.analysis.WhitespaceAnalyzer;
import org.apache.lucene.document.Document;
import org.apache.lucene.document.Field;
import org.apache.lucene.index.IndexReader;
import org.apache.lucene.index.IndexWriter;
import org.apache.lucene.search.IndexSearcher;
import org.apache.lucene.store.Directory;
import org.apache.lucene.store.RAMDirectory;

public class Exec {
        public static double computeAngle(String[] doc1, String[] doc2) {
                // assume words are unique and only occur once

                int dotProduct = 0;
                double denominator;

                for (int i = 0; i < doc1.length; i++) {
                        String word = doc1[i];
                        for (int j = 0; j < doc2.length; j++) {
                                if (word.equals(doc2[j])) {
                                        dotProduct += 1;
                                        break;
                                }
                        }
                }
                denominator = Math.sqrt(doc1.length) * Math.sqrt(doc2.length);
                double ratio = dotProduct / denominator;
                return Math.acos(ratio);
        }

        // For simple, only to get one document which must be belong to the matching
        // document
        public static Document docsLike(IndexReader reader, IndexSearcher searcher,
                        int id) throws IOException {
                Document doc = reader.document(id);
                String[] s1 = (doc.get("subjects")).split(" ");

                int matchid = 0;
                double bestAngle = Double.MAX_VALUE;

                for (int i = 0; i < reader.maxDoc(); i++) {
                        if (i != id) {
                                Document tempdoc = reader.document(i);
                                String[] s2 = (tempdoc.get("subjects")).split(" ");
                                double angle = computeAngle(s1, s2);
                                // System.out.print("***"+angle);
                                if (angle < bestAngle) {
                                        bestAngle = angle;
                                        matchid = i;
                                }
                        }
                }
                // System.out.println("\n***********"+bestAngle+"***********"+matchid);
                return reader.document(matchid);
        }

        public static void main(String[] args) throws IOException {

                IndexReader reader;
                IndexSearcher searcher;

                // Writer
                Directory directory = new RAMDirectory();
                IndexWriter writer = new IndexWriter(directory,
                                new WhitespaceAnalyzer(), true);
                writer.setUseCompoundFile(true);

                String[] isbns = new String[] { "1", "2", "3", "4", "5", "6", "7", "8",
                                "9", "10", "11" };

                String[] titles = new String[] { "Modern Art of Education",
                                "Imperial Secrets of Health", "Tao Te Ching",
                                "G.del, Escher, Bach", "Mindstorms",
                                "Java Development with Ant", "JUnit in Action",
                                "Lucene in Action", "Tapestry in Action",
                                "Extreme Programming Explained", "The Pragmatic Programmer" };
                String[] pubmonths = new String[] { "198106", "199401", "198810",
                                "197903", "198001", "200208", "200310", "200406", "199910",
                                "200403", "199910" };
                String[] categories = new String[] { "/education/pedagogy",
                                "/health/alternative/chinese", "/philosophy/eastern",
                                "/technology/computers/ai",
                                "/technology/computers/programming/education",
                                "/technology/computers/programming",
                                "/technology/computers/programming",
                                "/technology/computers/programming",
                                "/technology/computers/programming",
                                "/technology/computers/programming/methodology",
                                "/technology/computers/programming" };
                String[] subjects = new String[] {
                                "education philosophy psychology practice Waldorf",
                                "diet chinese medicine qi gong health herbs",
                                "taoism chinese ideas",
                                "artificial intelligence number theory mathematics music",
                                "children computers powerful ideas LOGO education",
                                "apache jakarta ant build tool junit java development",
                                "junit unit testing mock objects",
                                "lucene search programming",
                                "tapestry web user interface components",
                                "extreme programming agile test driven development methodology",
                                "pragmatic agile methodology developer tools" };

                for (int i = 0; i < titles.length; i++) {
                        Document doc = new Document();
                        doc.add(Field.Keyword("isbns", isbns[i]));
                        doc.add(Field.Text("titles", titles[i]));
                        doc.add(Field.Text("pubmonths", pubmonths[i]));
                        doc.add(Field.Text("categories", categories[i]));
                        doc.add(Field.Text("subjects", subjects[i], true));
                        writer.addDocument(doc);
                }
                writer.optimize();
                writer.close();

                // Reader
                reader = IndexReader.open(directory);
                searcher = new IndexSearcher(reader);
                int numDocs = reader.maxDoc();

                for (int i = 0; i < numDocs; i++) {
                        System.out.println();
                        Document doc = reader.document(i);
                        System.out.println(doc.get("titles"));
                        Document docs = docsLike(reader, searcher, i);
                        System.out.println("  -> " + docs.get("titles"));
                }
        }
}


 回到顶部