Skip to main content
Version: 3.x.x 🚧

@yozora/tokenizer-text

Npm VersionNpm DownloadNpm LicenseModule formats: cjs, esmNode.js VersionTested with JestCode Style: prettier

github flavor markdown spec

Any characters not given an interpretation by the above rules will be parsed as plain textual content.

Install

npm install --save @yozora/tokenizer-text

Usage

tip

@yozora/tokenizer-text has been integrated into @yozora/parser / @yozora/parser-gfm-ex / @yozora/parser-gfm, so you can use YozoraParser / GfmExParser / GfmParser directly.

import YozoraParser from '@yozora/parser'

const parser = new YozoraParser()

// parse source markdown content
parser.parse("hello $.;'there")

Options

NameTypeRequiredDefault
namestringfalse"@yozora/tokenizer-text"
prioritynumberfalseTokenizerPriority.FALLBACK
  • name: The unique name of the tokenizer, used to bind the token it generates, to determine the tokenizer that should be called in each life cycle of the token in the entire matching / parsing phase.

  • priority: Priority of the tokenizer, determine the order of processing, high priority priority execution. interruptable. In addition, in the match-block stage, a high-priority tokenizer can interrupt the matching process of a low-priority tokenizer.

    Exception: Delimiters of type full are always processed before other type delimiters.

Types

@yozora/tokenizer-text produce Text type nodes. See @yozora/ast for full base types.

import type { Literal } from '@yozora/ast'

export const TextType = 'text'
export type TextType = typeof TextType

/**
* Text represents everything that is just text.
* @see https://github.com/syntax-tree/mdast#text
* @see https://github.github.com/gfm/#textual-content
*/
export type Text = Literal<TextType>

Live Examples

  • Basic.

      
      
  • Internal spaces are preserved verbatim.

    #673