Mutating relativeInput makes better-parse throw InvalidMutabilityException on iosX64 (probably on other Kotlin/Native platforms as well). This PR turns relativeInput into a local immutable object.

Since 0e6eb411b, Tokens defined by a Pattern or RegEx object are converted to a Token defined by a pattern String.

This breaks use-cases where the pattern object was created with flags set (e.g. token("^--".toRegex(RegexOption.MULTILINE)))

Currently, this can only be worked around by specifying the flags inside the pattern string (e.g. token("(?m)^--"))

Could you provide a version that compiles against kotlin 1.4.0 release? The 1.4-M2 version is rejected by the compiler as "compiled by a pre-release version". Thanks!

Is it possible to define the following grammar?

val singleton by lbrace cmd rbrace; // singleton -> cmd
val expr by number or funccall or ... or singleton; // expr -> singleton
val cmd by expr semicol; // cmd -> expr
// the dependency is a cycle: cmd -> expr -> singleton -> cmd

so that this grammar accepts

{ 1; }

as an expression. I can't write the grammar in the naive way, because Kotlin rejects this code complaining "Variable cmd must be initialized". I tried lazy but the properties seem to be NULL, not properly initialized.

edit: I added lazy to every parser but it ended up infinite recursion.

The goal

I have a file that looks like this:

Header
Line
Line
Line
Footer

And I would like to parse it into the following container:

data class File(
  val header: Header,
  val lines: List<Line>,
  val footer: Footer
)

The problem

I have written a grammar for each of the different variants, i.e Grammar<Header, Grammar<Line>, and Grammar<Footer>. There is some similarity in how these different lines look. How can I compose these grammars together? The variants are split on a newline \n character.

Any help would be much appreciated.

interface Item
class Number(val value: Int) : Item
class Variable(val name: String) : Item

object ItemsParser : Grammar<Item>() {
    val num by token("\\d+")
    val word by token("[A-Za-z]")
    val comma by token(",\\s+")

    val numParser = num use { Number(text.toInt()) }
    val varParser = word use { Variable(text) }

    override val <!PROPERTY_TYPE_MISMATCH_ON_OVERRIDE!>rootParser<!> = separatedTerms(numParser or varParser, comma)
}

val result: List<Item> = ItemsParser.<!TYPE_INFERENCE_EXPECTED_TYPE_MISMATCH!>parseToEnd("one, 2, three, 4, five")<!>

Some errors reported on code copied from README

I a trying to figure out how this library works but I simply can't get passed the simple literal token example. Here is my test code:

    @Test
    fun `should match foo`() {
        val literal = literalToken("foo")
        val tokenizer = DefaultTokenizer(listOf(literal))
        val sequence: TokenMatchesSequence = tokenizer.tokenize("my foo")

        assertThat(sequence.toList()).isNotEmpty() // success
        assertThat(sequence[0]!!.input).isEqualTo("my foo") // success
        assertThat(sequence[0]!!.type).isNotEqualTo(noneMatched) // failure
        assertThat(sequence[0]!!.text).isEqualTo("foo") // never reached
    }

This fails with assertThat(sequence[0]!!.type).isNotEqualTo(noneMatched) since there is no match for the simple input of my foo and the literal token looking for foo.

Any idea what I am doing wrong?

Infos:

implementation("com.github.h1tk3y.betterParse:better-parse:0.4.4")

Greetings!

Kotlins internal name for the nativePattern seems to have changed. This fix changes preprocessRegex to handle multiple names. It also replaces the js(...) call with regular Kotlin code

The range added to the SyntaxTree for an optional parser that is skipped, is reported as 0..0, as per this logic. This will affect the reported range of parent parsers as well, e.g. if the optional parser is the first in an "and combinator", the reported range of the "and combinator" as a whole will start at 0. This PR changes the fallback behavior to an empty range starting at the current token's offset. I think it can be debated if this is fully correct, but I like it because it's pragmatic.

Thanks for considering this PR. Let me know if you have any comments or things I should improve!

When an optional parser parses nothing, it's range becomes 0..0, because of the fallback here. This error will bubble up, e.g. when the optional is inside an "and" parser.

Example:

val integerExpression by (optional(MINUS) * INT)

If there is no - to parse, the start of the range of the SyntaxTree corresponding to a match of integerExpression will always start at 0, regardless of where the match happens. So if the input where Hello World! 5 (and imagine integerExpression is part of a larger parser that can also parse words), the actual range would be 0..14, while the expected range would be 13..14.

Do you have any pointers how this could be best resolved?

In grammars on the JVM, RegexTokens (at least) use stateful matchers, and therefore aren't thread-safe. It would be nice if the library either noted this, or was thread-safe (by creating a new matcher as needed- at a glance this doesn't seem to be an expensive JVM operation, I think that's the creation of the Pattern).

Hi there, I have a grammar with some negative lookahead rules. From what I can tell, some other PEG parsers have a ! operator that can help with this, but it appears this library doesn't. Is this something that would be easy to implement "in userspace"? I'm not a parsing expert. Thanks!

When the Kotlin/JS IR compiler is enabled in Kotlin 1.7.0, minification of member names is turned on by default which breaks preprocessRegex.

As a workaround, disabling minification as described in the 1.7.0 change notes fixes the issue.

Hi, I want to make a grammar with Python-style indentation. In Python:

def stuff():
    if True:
        if True:
            for i in range(2):
                print(i)
    print("YAY")

How would I make INDENT and DEDENT tokens using better-parse? I couldn't find any better-parse examples or tests showcasing how to do this.

Say I have a synthetic situation: number n says how many symbols x will be after this number. After that, I want to drop all other xs.

For example:

val digit by regexToken("\\d")
val n by literalToken("n")
val found by digit map {it.toInt()}
val result by found and (found times n) and -zeroOrMore(n)
//                       ^ can't do

Seeing that Grammar<T> extends Parser<T>, I figured I should be able to delegate to a Grammar<T>, such as:

val exp: Parser<Exp> by ExpGrammar() // where ExpGrammar is a Grammar<Exp>

However, it doesn't seem to behave as expected. The following is a small SSCCE to demonstrate:

package com.example

import com.github.h0tk3y.betterParse.combinators.and
import com.github.h0tk3y.betterParse.combinators.map
import com.github.h0tk3y.betterParse.combinators.separatedTerms
import com.github.h0tk3y.betterParse.combinators.skip
import com.github.h0tk3y.betterParse.grammar.Grammar
import com.github.h0tk3y.betterParse.grammar.parseToEnd
import com.github.h0tk3y.betterParse.lexer.TokenMatch
import com.github.h0tk3y.betterParse.lexer.literalToken
import com.github.h0tk3y.betterParse.lexer.regexToken
import com.github.h0tk3y.betterParse.parser.Parser

data class Inner(
    val names: List<String>,
)

data class Outer(
    val name: String,
    val inner: Inner,
)

abstract class TestGrammarBase<T> : Grammar<T>()
{
    val idToken by regexToken("\\w+")

    val spaceToken by regexToken("\\s*", true)

    val commaToken by literalToken(",")

    val lBraceToken by literalToken("{")

    val rBraceToken by literalToken("}")
}

class InnerTestGrammar : TestGrammarBase<Inner>()
{
    override val rootParser: Parser<Inner> by separatedTerms(idToken, commaToken, true) map inner@{ tokenMatches ->
        return@inner Inner(
            names = tokenMatches.map(TokenMatch::text),
        )
    }
}

class OuterTestGrammar : TestGrammarBase<Outer>()
{
    val innerTestParser by InnerTestGrammar()

    override val rootParser: Parser<Outer> by idToken and skip(lBraceToken) and innerTestParser and skip(rBraceToken) map outer@{ (tokenMatch, inner) ->
        return@outer Outer(
            name = tokenMatch.text,
            inner = inner,
        )
    }
}

fun main()
{
    val innerTest1 = "X, Y, Z"
    val outerTest1 = "A { }"
    val outerTest2 = "A { X, Y, Z }"

    val innerTestGrammar = InnerTestGrammar()
    val outerTestGrammar = OuterTestGrammar()

    innerTestGrammar.parseToEnd(innerTest1).also(::println)
    outerTestGrammar.parseToEnd(outerTest1).also(::println)
    outerTestGrammar.parseToEnd(outerTest2).also(::println)
}

And the output:

Inner(names=[X, Y, Z])
Outer(name=A, inner=Inner(names=[]))
Exception in thread "main" com.github.h0tk3y.betterParse.parser.ParseException: Could not parse input: MismatchedToken(expected=rBraceToken (}), found=idToken@5 for "X" at 4 (1:5))
	at com.github.h0tk3y.betterParse.parser.ParserKt.toParsedOrThrow(Parser.kt:92)
	at com.github.h0tk3y.betterParse.parser.ParserKt.parseToEnd(Parser.kt:29)
	at com.github.h0tk3y.betterParse.grammar.GrammarKt.parseToEnd(Grammar.kt:70)
	at com.example.__langKt.main(__lang.kt:68)
	at com.example.__langKt.main(__lang.kt)

As you can see, the third attempt to parse the grammar-combined input of A { X, Y, Z } errors out. InnerTestGrammar and OuterTestGrammar, having extended from TestGrammarBase can see the shared member tokens/parsers, but seem to get confused (or perhaps I'm confused).

Is this not an intended use of Grammar?