NAME
    Search::Fulltext::Tokenizer::Ngram - Character n-gram tokenizer for
    Search::Fulltext

VERSION
    version 0.01

SYNOPSIS
      use utf8;
      use Search::Fulltext;
      use Search::Fulltext::Tokenizer::Bigramm;
  
      my $searcher = Search::Fulltext->new(
          docs => [
              'ハンプティ・ダンプティ 塀の上',
              'ハンプティ・ダンプティ 落っこちた',
              '王様の馬みんなと 王様の家来みんなでも',
              'ハンプティを元に 戻せなかった',
          ],
          tokenizer => q/perl 'Search::Fulltext::Tokenizer::Bigram::get_tokenizer'/,
      );
      my $hit_document_ids = $searcher->search('ハンプティ');  # [0, 1, 3]

DESCRIPTION
    This module provides character N-gram tokenizers for Search::Fulltext.

    By default {1,2,3}-gram tokenzers are available.

CREATING A N(> 3)-GRAM TOKENIZER
    If you wish to use other N-grams where N > 3, you can create it by
    inheriting "Search::Fulltext::Tokenizer::Ngram":

      package My::Tokenizer::42gram;
  
      use parent qw/Search::Fulltext::Tokenizer::Ngram/;
  
      my $iterator_generator = __PACKAGE__->new(42);
  
      sub get_tokenizer {
          sub { $iterator_generator->create_token_iterator(@_) };
      }

SEE ALSO
    Search::Fulltext::Tokenizer::Unigram Search::Fulltext::Tokenizer::Bigram
    Search::Fulltext::Tokenizer::Trigram

AUTHOR
    Koichi SATOH <sekia@cpan.org>

COPYRIGHT AND LICENSE
    This software is Copyright (c) 2014 by Koichi SATOH.

    This is free software, licensed under:

      The MIT (X11) License

