Version: 2.x.x

@yozora/core-parser

@yozora/core-parser provides a plug-in system that can load tokenizer for collaborative processing in the lifecycles.

Install

npm
Yarn
pnpm
Bun

npm install --save @yozora/core-parser

yarn add @yozora/core-parser

pnpm add @yozora/core-parser

bun add @yozora/core-parser

Usage

@yozora/core-parser provides a default Parser implementation DefaultParser.

import { DefaultParser } from '@yozora/core-parser'
import EmphasisTokenizer from '@yozora/tokenizer-emphasis'
import ParagraphTokenizer from '@yozora/tokenizer-paragraph'
import TextTokenizer from '@yozora/tokenizer-text'

const parser = new DefaultParser({ shouldReservePosition: true })
  .useFallbackTokenizer(new ParagraphTokenizer())
  .useFallbackTokenizer(new TextTokenizer())

  // add custom tokenizers through `useTokenizer()`
  .useTokenizer(new EmphasisTokenizer())

// Parse literal string
parser.parse('source content such as **emphasis**')

// Parse literal string lists
parser.parse(['source content', 'additional *content*'])

// Parse iterable string
// The Parser will processing at every time a line is read.
function* genContents () {
  yield 'source content1\n'
  yield 'source content2\n'
  return 'source content3\n'
}
parser.parse(genContents())

// Reserve the positions
parser.parse('source content', { shouldReservePosition: true })

DefaultParser Options

Constructor Options

Name Required Default Type
blockFallbackTokenizer false null BlockFallbackTokenizer | null
inlineFallbackTokenizer false null InlineFallbackTokenizer | null
defaultParseOptions false See below Parse Options
- blockFallbackTokenizer: Fallback tokenizer on processing block structure phase.
- inlineFallbackTokenizer: Fallback tokenizer on processing inline structure phase.
- defaultParseOptions: Default options for parse(). Default:
```
{
  presetDefinitions: [],
  presetFootnoteDefinitions: [],
  shouldReservePosition: false
}
```

Name	Required	Default	Type
`blockFallbackTokenizer`	`false`	`null`	BlockFallbackTokenizer \| `null`
`inlineFallbackTokenizer`	`false`	`null`	InlineFallbackTokenizer \| `null`
`defaultParseOptions`	`false`	See below	Parse Options

Parse Options

Name	Required	Default	Type
`presetDefinitions`	`false`	`defaultParserOptions.presetDefinitions`	[Association][src-Association] list.
`presetFootnoteDefinitions`	`false`	`defaultParserOptions.presetFootnoteDefinitions`	`boolean`
`shouldReservePosition`	`false`	`defaultParserOptions.shouldReservePosition`	`boolean`

presetDefinitions: Preset definition meta data list.
presetFootnoteDefinitions: Preset footnote definition meta data list.
shouldReservePosition: Whether it is necessary to reserve the position of the Node parsed.

Lifecycle

math-block

API: IMatchBlockPhaseApi

Name	Description
`extractPhrasingLines`	Extract phrasing content lines from block token.
`rollbackPhrasingLines`	Re-match token from phrasing content lines.
`registerDefinitionIdentifier`	Register a definition identifier.
`registerFootnoteDefinitionIdentifier`	Register a footnote definition identifier.

Hook: IMatchBlockHook

Name	Required	Description
`isContainingBlock`	`true`	Whether if it is a container block.
`eatOpener`	`true`	Try to match new block data.
`eatAndInterruptPreviousSibling`	`false`	Try to interrupt the eatContinuationText action of the last sibling node.
`eatContinuationText`	`false`	Try to eat the Continuation Text, and check if it is still satisfied to current opening MatchToken
`eatLazyContinuationText`	`false`	Try to eat the Laziness Continuation Text, and check if it is still satisfied to current opening MatchToken
`onClose`	`false`	Called when the token is saturated.

Example tokenizers:

parse-block

API: IParseBlockPhaseApi

Name	Description
`shouldReservePosition`	Whether it is necessary to reserve the position in the Node produced.
`processInlines`	Process node points into inline nodes.
`parseBlockTokens`	Parse block tokens to Yozora AST nodes.

Hook: IParseBlockHook

Name Required Description
parse true Processing a specified token list to Node list.
Example tokenizers:

Name	Required	Description
`parse`	`true`	Processing a specified token list to Node list.

match-inline

API: IMatchInlinePhaseApi

Name	Description
`hasDefinition`	Check if there is exists a definition with the given identifier.
`hasFootnoteDefinition`	Check if there is exists a footnote definition with the given identifier.
`getNodePoints`	Get the node points.
`getBlockStartIndex`	Start index of current block token.
`getBlockEndIndex`	End index of current block token.
`resolveFallbackTokens`	Resolve fallback inline tokens.
`resolveInternalTokens`	Resolve raw contents with the fallback inline tokenizer.

Hook: IMatchInlineHook

Name	Required	Description
`findDelimiter`	`true`	Find an inline token delimiter.
`isDelimiterPair`	`false`	Check if the given two delimiters can be combined into a pair.
`processDelimiterPair`	`false`	Process a pair of delimiters.
`processSingleDelimiter`	`false`	Process a single delimiter (its type should be one of 'both' and 'full').

Example tokenizers:

parse-inline

API: IParseInlinePhaseApi

Name	Description
`shouldReservePosition`	Whether it is necessary to reserve the position in the Node produced.
`getNodePoints`	Get the node points.
`calcPosition`	Calculate position of token.
`hasDefinition`	Check if there is exists a definition with the given identifier.
`hasFootnoteDefinition`	Check if there is exists a footnote definition with the given identifier.
`parseInlineTokens`	Parse inline tokens to Yozora AST nodes.

Hook: IParseInlineHook

Name Required Description
parse true Processing token list to Node list.
Example tokenizers:

Install​

Usage​

DefaultParser Options​

Lifecycle​

math-block​

parse-block​

match-inline​

parse-inline​

Related​