@yozora/parser

A markdown parser with rich built-in tokenizers.

github flavor markdown spec

• See github flavor markdown spec for details.
• See GFM Live Examples for an intuitive impression.

Install

npm

npm install --save @yozora/parser

Yarn

yarn add @yozora/parser

pnpm

pnpm add @yozora/parser

Usage

• Basic

  import YozoraParser from '@yozora/parser'
  
  const parser = new YozoraParser()
  
  // parse markdown source content.
  parser.parse('source markdown content')
  
  // parse markdown source content with custom options.
  parser.parse(
    'source markdown content',  // markdown source contents, `string|Iterable<string>`
    {},                         // ParseOptions, optional.
  )
  
  // parse multiple markdown source content pieces.
  parser.parse(['source', 'contents'])

• Use withing generator:

  import YozoraParser from '@yozora/parser'
  
  const parser = new YozoraParser()
  
  /**
   * String stream is supported through the iterator API.
   */
  function* source () {
    yield 'hello',
    yield 'world',
  }
  
  parser.parse(source())

Options

• Constructor Options

  Name	Type	Required	Description
  blockFallbackTokenizer	BlockFallbackTokenizer	false	Fallback tokenizer on processing block structure phase
  inlineFallbackTokenizer	InlineFallbackTokenizer	false	Fallback tokenizer on processing inline structure phase
  defaultParseOptions	ParseOptions	false	Default options for parse()

• ParseOptions

  Name	Type	Required	Description
  shouldReservePosition	boolean	false	Whether it is necessary to reserve the position in the YastNode produced
  presetDefinitions	Array<Omit<Definition, 'type'>	false	Preset definitions
  presetFootnoteDefinitions	Array<Omit<FootnoteDefinition, 'type'>	false	Preset footnote definition

Overview

• Built-in tokenizers

  • All tokenizers for processing tokens defined Github Flavor Markdown:

    Tokenizer	Description
    @yozora/tokenizer-autolink	Resolve GFM Autolinks
    @yozora/tokenizer-autolink-extension	Resolve GFM Autolinks (extension)
    @yozora/tokenizer-blockquote	Resolve GFM blockquotes
    @yozora/tokenizer-break	Resolve GFM hard line breaks and GFM soft line breaks
    @yozora/tokenizer-definition	Resolve GFM link reference definitions
    @yozora/tokenizer-delete	Resolve GFM strikethrough (extension)
    @yozora/tokenizer-emphasis	Resolve GFM emphasis and strong emphasis
    @yozora/tokenizer-fenced-code	Resolve GFM fenced code blocks
    @yozora/tokenizer-heading	Resolve GFM ATX headings
    @yozora/tokenizer-html-block	Resolve GFM HTML blocks
    @yozora/tokenizer-html-inline	Resolve GFM raw HTML
    @yozora/tokenizer-image	Resolve GFM images
    @yozora/tokenizer-image-reference	Resolve GFM reference images
    @yozora/tokenizer-indented-code	Resolve GFM indented code blocks
    @yozora/tokenizer-inline-code	Resolve GFM code spans
    @yozora/tokenizer-link	Resolve GFM links
    @yozora/tokenizer-link-reference	Resolve GFM reference links
    @yozora/tokenizer-list	Resolve GFM lists
    @yozora/tokenizer-list-item	Resolve GFM list items and GFM task list items
    @yozora/tokenizer-paragraph	Resolve GFM paragraphs
    @yozora/tokenizer-setext-heading	Resolve GFM setext headings
    @yozora/tokenizer-table	Resolve GFM tables
    @yozora/tokenizer-text	Resolve GFM textual contents
    @yozora/tokenizer-thematic-break	Resolve GFM thematic breaks

  • Additional tokenizers

    Tokenizer	Description
    @yozora/tokenizer-admonition	Resolve admonitions
    @yozora/tokenizer-footnote	Resolve footnotes
    @yozora/tokenizer-footnote-definition	Resolve footnote definitions
    @yozora/tokenizer-footnote-reference	Resolve footnote references
    @yozora/tokenizer-inline-math	Resolve inline formulas
    @yozora/tokenizer-math	Resolve block formulas

Live Examples

Thematic break

See @yozora/tokenizer-thematic-break for details.

#13

yozora

  

pretty-json

  

*** --- ___

1
2
3
***
---
___

ATX headings

See @yozora/tokenizer-heading for details.

#32

yozora

  

pretty-json

  

# foo ## foo ### foo #### foo ##### foo ###### foo

1
2
3
4
5
6
# foo
## foo
### foo
#### foo
##### foo
###### foo

Setext headings

See @yozora/tokenizer-setext-heading for details.

#50

yozora

  

pretty-json

  

Foo *bar* ========= Foo *bar* ---------

1
2
3
4
5
Foo *bar*
=========

Foo *bar*
---------

Indented code blocks

See @yozora/tokenizer-indented-code for details.

#77

yozora

  

pretty-json

  

a simple indented code block

1
2
    a simple
      indented code block

Fenced code blocks

See @yozora/tokenizer-fenced-code for details.

#89

yozora

  

pretty-json

  

``` < > ```

1
2
3
4
```
<
 >
```

HTML blocks

See @yozora/tokenizer-html-block for details.

#118

yozora

  

pretty-json

  

<table><tr><td> <pre> **Hello**, _world_. </pre> </td></tr></table>

1
2
3
4
5
6
7
<table><tr><td>
<pre>
**Hello**,

_world_.
</pre>
</td></tr></table>

Link reference definitions

See @yozora/tokenizer-definition for details.

#161

yozora

  

pretty-json

  

[foo]: /url "title" [foo]

1
2
3
[foo]: /url "title"

[foo]

Paragraphs

See @yozora/tokenizer-paragraph for details.

#189

yozora

  

pretty-json

  

aaa bbb

1
2
3
aaa

bbb

Tables (extension)

See @yozora/tokenizer-table for details.

#198

yozora

  

pretty-json

  

| foo | bar | | --- | --- | | baz | bim |

1
2
3
| foo | bar |
| --- | --- |
| baz | bim |

Block quotes

See @yozora/tokenizer-blockquote for details.

#206

yozora

  

pretty-json

  

> # Foo > bar > baz

1
2
3
> # Foo
> bar
> baz

List items

See @yozora/tokenizer-list-item for details.

#231

yozora

  

pretty-json

  

A paragraph with two lines. indented code > A block quote.

1
2
3
4
5
6
A paragraph
with two lines.

    indented code

> A block quote.

Task list items (extension)

See @yozora/tokenizer-list-item for details.

#279

yozora

  

pretty-json

  

- [ ] foo - [x] bar

1
2
- [ ] foo
- [x] bar

Lists

See @yozora/tokenizer-list for details.

#281

yozora

  

pretty-json

  

- foo - bar + baz

1
2
3
- foo
- bar
+ baz

Code spans

See @yozora/tokenizer-inline-code for details.

#338

yozora

  

pretty-json

  

`foo`

1
`foo`

Emphasis and strong emphasis

See @yozora/tokenizer-emphasis for details.

#360

yozora

  

pretty-json

  

*foo bar*

1
*foo bar*

Strikethrough (extension)

See @yozora/tokenizer-delete for details.

#491

yozora

  

pretty-json

  

~~Hi~~ Hello, world!

1
~~Hi~~ Hello, world!

Links

See @yozora/tokenizer-link for details.

#493

yozora

  

pretty-json

  

[link](/uri "title")

1
[link](/uri "title")

Reference links

See @yozora/tokenizer-link-reference for details.

#535

yozora

  

pretty-json

  

[foo][bar] [bar]: /url "title"

1
2
3
[foo][bar]

[bar]: /url "title"

Images

See @yozora/tokenizer-image for details.

#580

yozora

  

pretty-json

  


1
![foo](/url "title")

Reference images

See @yozora/tokenizer-image-reference for details.

#590

yozora

  

pretty-json

  

![foo][bar] [bar]: /url

1
2
3
![foo][bar]

[bar]: /url

Autolinks

See @yozora/tokenizer-autolink for details.

#602

yozora

  

pretty-json

  

<http://foo.bar.baz>

1
<http://foo.bar.baz>

Autolinks (extension)

See @yozora/tokenizer-autolink-extension for details.

#621

yozora

  

pretty-json

  

www.commonmark.org

1
www.commonmark.org

Raw HTML

See @yozora/tokenizer-html-inline for details.

#632

yozora

  

pretty-json

  

<a><bab><c2c>

1
<a><bab><c2c>

Hard line breaks

See @yozora/tokenizer-break for details.

#654

yozora

  

pretty-json

  

foo baz

1
2
foo  
baz

Soft line breaks

See @yozora/tokenizer-break for details.

#669

yozora

  

pretty-json

  

foo baz

1
2
foo
baz

Textual content

See @yozora/tokenizer-text for details.

#671

yozora

  

pretty-json

  

hello $.;'there

1
hello $.;'there

Related

• @yozora/ast
• @yozora/core-parser
• @yozora/parser-gfm
• @yozora/parser-gfm-ex
• Sourcecode
• Mdast
• Github Flavor Markdown Spec