@yozora/tokenizer-text

github flavor markdown spec

Any characters not given an interpretation by the above rules will be parsed as plain textual content.

• See github flavor markdown spec for details.
• See Live Examples for an intuitive impression.

Install

npm

npm install --save @yozora/tokenizer-text

Yarn

yarn add @yozora/tokenizer-text

pnpm

pnpm add @yozora/tokenizer-text

Usage

tip

@yozora/tokenizer-text has been integrated into @yozora/parser / @yozora/parser-gfm-ex / @yozora/parser-gfm, so you can use YozoraParser / GfmExParser / GfmParser directly.

Basic Usage

@yozora/tokenizer-text cannot be used alone, it needs to be registered in YastParser as a plugin-in before it can be used.

import { DefaultYastParser } from '@yozora/core-parser'
import ParagraphTokenizer from '@yozora/tokenizer-paragraph'
import TextTokenizer from '@yozora/tokenizer-text'
import TextTokenizer from '@yozora/tokenizer-text'

const parser = new DefaultYastParser()
  .useBlockFallbackTokenizer(new ParagraphTokenizer())
  .useInlineFallbackTokenizer(new TextTokenizer())
  .useTokenizer(new TextTokenizer())

// parse source markdown content
parser.parse("hello $.;'there")

YozoraParser

import YozoraParser from '@yozora/parser'

const parser = new YozoraParser()

// parse source markdown content
parser.parse("hello $.;'there")

GfmParser

import GfmParser from '@yozora/parser-gfm'

const parser = new GfmParser()

// parse source markdown content
parser.parse("hello $.;'there")

GfmExParser

import GfmExParser from '@yozora/parser-gfm-ex'

const parser = new GfmExParser()

// parse source markdown content
parser.parse("hello $.;'there")

Options

Name	Type	Required	Default
name	string	false	"@yozora/tokenizer-text"
priority	number	false	TokenizerPriority.FALLBACK

• name: The unique name of the tokenizer, used to bind the token it generates, to determine the tokenizer that should be called in each life cycle of the token in the entire matching / parsing phase.

• priority: Priority of the tokenizer, determine the order of processing, high priority priority execution. interruptable. In addition, in the match-block stage, a high-priority tokenizer can interrupt the matching process of a low-priority tokenizer.

  Exception: Delimiters of type full are always processed before other type delimiters.

Types

@yozora/tokenizer-text produce Text type nodes. See @yozora/ast for full base types.

import type { YastLiteral } from '@yozora/ast'

export const TextType = 'text'
export type TextType = typeof TextType

/**
 * Text represents everything that is just text.
 * @see https://github.com/syntax-tree/mdast#text
 * @see https://github.github.com/gfm/#textual-content
 */
export type Text = YastLiteral<TextType>

Live Examples

• Basic.

  #671

  yozora

    

  pretty-json

    

  hello $.;'there

  1
  hello $.;'there

• Internal spaces are preserved verbatim.

  #673

  yozora

    

  pretty-json

    

  Multiple spaces

  1
  Multiple     spaces

Related

• @yozora/ast
• @yozora/parser
• @yozora/parser-gfm
• @yozora/parser-gfm-ex
• @yozora/react-text
• @yozora/react-markdown
• Sourcecode
• Text | Yozora AST
• Textual content | Github Flavor Markdown Spec
• Text | Mdast