Guide to parsing with nom
- Introduction
- - Documentation
- Getting to know nom using a simple example
- Build a Markdown parser
+ - Other examples
+ - Related video
+ - References
@@ -340,70 +350,6 @@
object oriented), please take a look at this paper
by Will Crichton demonstrating Typed Design Patterns with Rust.
-
-
-
- Documentation #
-
-
-
-
-
-
-- - - Documentation # - - -
- -nom is a huge topic. This tutorial takes a hands on approach to learning nom. However, the resources -listed below are very useful for learning nom. Think of them as a reference guide and deep dive into -how the nom library works.
- --
-
- Useful:
-
-
-
- Source code examples (fantastic way to learn nom): - - -
- Videos:
-
-
-
- Intro from the author 7yrs old -
- Nom 7 deep dive videos: - - -
- Nom 6 videos (deep dive into how nom combinators themselves are constructed): - - -
- - Tutorials: - - -
- Reference docs: - - -
- - Less useful: - - -
@@ -414,22 +360,22 @@
-nom is a parser combinator library for Rust. You can write small +
nom is a parser combinator library for Rust. You can write small
functions that parse a specific part of your input, and then combine them to build a parser that
-parses the whole input. nom is very efficient and fast, it does not allocate memory when parsing if
-it doesnβt have to, and it makes it very easy for you to do the same. nom uses streaming mode or
+parses the whole input. nom is very efficient and fast, it does not allocate memory when parsing if
+it doesnβt have to, and it makes it very easy for you to do the same. nom uses streaming mode or
complete mode, and in this tutorial & code examples provided we will be using complete mode.
Roughly the way it works is that you tell nom how to parse a bunch of bytes in a way that matches +
Roughly the way it works is that you tell nom how to parse a bunch of bytes in a way that matches
some pattern that is valid for your data. It will try to parse as much as it can from the input, and
the rest of the input will be returned to you.
You express the pattern that youβre looking for by combining parsers. nom has a whole bunch of these -that come out of the box. And a huge part of learning nom is figuring out what these built in +
You express the pattern that youβre looking for by combining parsers. nom has a whole bunch of these
+that come out of the box. And a huge part of learning nom is figuring out what these built in
parsers are and how to combine them to build a parser that does what you want.
Errors are a key part of it being able to apply a variety of different parsers to the same input. If
-a parser fails, nom will return an error, and the rest of the input will be returned to you. This
+a parser fails, nom will return an error, and the rest of the input will be returned to you. This
allows you to combine parsers in a way that you can try to parse a bunch of different things, and if
one of them fails, you can try the next one. This is very useful when you are trying to parse a
bunch of different things, and you donβt know which one you are going to get.
-
Letβs dive into nom using a simple example of parsing +
++ +You can get the source code for the examples in this +repo.
+
Letβs dive into nom using a simple example of parsing
hex color codes.
//! This module contains a parser that parses a hex color string into a [Color] struct.
+//! This module contains a parser that parses a hex color
+//! string into a [Color] struct.
//! The hex color string can be in the following format `#RRGGBB`.
//! For example, `#FF0000` is red.
use std::num::ParseIntError;
-use nom::{bytes::complete::*, combinator::*, error::*, sequence::*, IResult, Parser};
+use nom::{
+ bytes::complete::*, combinator::*, error::*, sequence::*, IResult, Parser
+};
#[derive(Debug, PartialEq)]
pub struct Color {
@@ -466,18 +420,21 @@
}
}
-/// Helper functions to match and parse hex digits. These are not [Parser]
-/// implementations.
+/// Helper functions to match and parse hex digits. These are not
+/// [Parser] implementations.
mod helper_fns {
use super::*;
- /// This function is used by [map_res] and it returns a [Result], not [IResult].
- pub fn parse_str_to_hex_num(input: &str) -> Result<u8, std::num::ParseIntError> {
+ /// This function is used by [map_res] and it returns a [Result]
+ /// not [IResult].
+ pub fn parse_str_to_hex_num(input: &str) ->
+ Result<u8, std::num::ParseIntError>
+ {
u8::from_str_radix(input, 16)
}
- /// This function is used by [take_while_m_n] and as long as it returns `true`
- /// items will be taken from the input.
+ /// This function is used by [take_while_m_n] and as long as it
+ /// returns `true` items will be taken from the input.
pub fn match_is_hex_digit(c: char) -> bool {
c.is_ascii_hexdigit()
}
@@ -490,14 +447,17 @@
}
}
-/// These are [Parser] implementations that are used by [hex_color_no_alpha].
+/// These are [Parser] implementations that are used by
+/// [hex_color_no_alpha].
mod intermediate_parsers {
use super::*;
/// Call this to return function that implements the [Parser] trait.
- pub fn gen_hex_seg_parser_fn<'input, E>() -> impl Parser<&'input str, u8, E>
+ pub fn gen_hex_seg_parser_fn<'input, E>() ->
+ impl Parser<&'input str, u8, E>
where
- E: FromExternalError<&'input str, ParseIntError> + ParseError<&'input str>,
+ E: FromExternalError<&'input str, ParseIntError> +
+ ParseError<&'input str>,
{
map_res(
take_while_m_n(2, 2, helper_fns::match_is_hex_digit),
@@ -518,7 +478,8 @@
),
);
let (input, _) = tag("#")(input)?;
- let (input, (red, green, blue)) = tuple(it)(input)?; // same as `it.parse(input)?`
+ let (input, (red, green, blue)) =
+ tuple(it)(input)?; // same as `it.parse(input)?`
Ok((input, Color { red, green, blue }))
}
@@ -559,84 +520,122 @@
Please note that:
- - This string can be parsed:
#2F14DFπ
.
- - However, this string canβt
π
#2F14DF.
+ - This string can be parsed:
#2F14DFπ
β
.
+ - However, this string canβt
π
#2F14DF π€.
So what is going on in the source code above?
+
+
+
+ The Parser trait and IResult #
+
+
+
+
+
-
- -
-
The intermediate_parsers::hex_color_no_alpha() function is the main function that orchestrates
-all the other functions to parse an input: &str and turn it into a (&str, Color).
-
-
- - The
tag combinator function is used to match the # character. This means that if the input
-doesnβt start with #, the parser will fail (which is why π
#2F14DF fails). It returns the
-remaining input after #. And the output is # which we throw away.
- - A
tuple is created that takes 3 parsers, which all do the same exact thing, but are written
-in 3 different ways just to demonstrate how these can be written.
-
- - The
helper_fns::parse_hex_seg() function is added to a tuple.
- - The higher order function
intermediate_parsers::gen_hex_seg_parser_fn() is added to the
-tuple.
- - Finally, the
map_res combinator is directly added to the tuple.
-
-
- - An extension function on this tuple called
parse() is called w/ the input (thus far). This
-is used to parse the input hex number.
-
- - It returns the remaining input after the hex number which is why
#2F14DFπ
returns π
as
-the first item in the tuple.
- - The second item in the tuple is the parsed color string turned into a
Color struct.
-
-
-
-
- -
-
Letβs look at the helper_fns::parse_hex_seg (the other 2 ways shown above do the same exact
-thing). The signature of this function tells nom that you can call the function w/ input
-argument and it will return IResult<Input, Output, Error>. This signature is the pattern that
-we will end up using to figure out how to chain combinators together. Hereβs how the map_res
-combinator is used by parse_hex_seg() to actually do the parsing:
-
-
- take_while_m_n: This combinator takes a range of characters (2, 2) and applies the
-function match_is_hex_digit to determine whether the char is a hex digit (using
-is_ascii_hexdigit() on the char). This is used to match a valid hex digit. It returns a
-&str slice of the matched characters. Which is then passed to the next combinator.parse_str_to_hex_num: This parser is used on the string slice returned from above. It simply
-takes string slice and turns it into a Result<u8>, std::num::ParseIntError>. The error is
-important, since if the string slice is not a valid hex digit, then we want to return this
-error.
-
- -
-
The key concept in nom is the Parser trait which is implemented for any FnMut that accepts an
+
The key concept in nom is the Parser trait which is implemented for any FnMut that accepts an
input and returns an IResult<Input, Output, Error>.
-
- - If you write a simple function w/ the signature
+
+
+ - If you write a simple function w/ the signature
fn(input: Input) -> IResult<Input, Output, Error> then you are good to go! You just need to
call parse() on the Input type and this will kick off the parsing.
- - Alternatively, you can just call the nom
tuple function directly via
+ - Alternatively, you can just call the
nom tuple function directly via
nom::sequence::tuple(...)(input)?. Or you can just call the parse() method on the tuple
-since this is an extension function on tuples provided by nom.
- IResult is a very important type alias. It encapsulates 3 key types that are related to
+since this is an extension function on tuples provided by nom.IResult is a very important type alias. It encapsulates 3 key types that are related to
parsing:
-
- - The
Input type is the type of the input that is being parsed. For example, if you are
+
+ - The
Input type is the type of the input that is being parsed. For example, if you are
parsing a string, then the Input type is &str.
- - The
Output type is the type of the output that is returned by the parser. For example, if
+ - The
Output type is the type of the output that is returned by the parser. For example, if
you are parsing a string and you want to return a Color struct, then the Output type is
Color.
- - The
Error type is the type of the error that is returned by the parser. For example, if
+ - The
Error type is the type of the error that is returned by the parser. For example, if
you are parsing a string and you want to return a nom::Err::Error error, then the Error
type is nom::Err::Error. This is very useful when you are developing your parser
combinators and you run into errors and have to debug them.
-
+ -
+
Typically we are dealing with complete parsers which are character based. These are reflected
+in the functions that we import from nom. It is pretty common to see the 'input lifetime
+parameter used in functions that are parsers. This way slices of the input can be returned
+from the parser without having to Clone or allocate memory.
+
+ Hereβs an example of this:
+ pub fn parse_hex_seg<'input, E /* thread this generic type down */>(
+ input: &'input str,
+) -> IResult<&'input str, u8, E>
+where
+ E: ParseError<&'input str> + ContextError<&'input str>
+{ /* code */ }
+
+
+
+
+
+
+ Main parser function that calls all the other parsers #
+
+
+
+
+
+
+The intermediate_parsers::hex_color_no_alpha() function is the main function that
+orchestrates all the other functions to parse an input: &str and turn it into a
+(&str, Color).
+
+
+ - The
tag combinator function is used to match the # character. This means that if the input
+doesnβt start with #, the parser will fail (which is why π
#2F14DF fails). It returns the
+remaining input after #. And the output is # which we throw away.
+ - A
tuple is created that takes 3 parsers, which all do the same exact thing, but are written
+in 3 different ways just to demonstrate how these can be written.
+
+ - The
helper_fns::parse_hex_seg() function is added to a tuple.
+ - The higher order function
intermediate_parsers::gen_hex_seg_parser_fn() is added to the
+ tuple.
+ - Finally, the
map_res combinator is directly added to the tuple.
+
+
+ - An extension function on this tuple called
parse() is called w/ the input (thus far). This
+is used to parse the input hex number.
+
+ - It returns the remaining input after the hex number which is why
#2F14DFπ
returns π
as
+the first item in the tuple.
+ - The second item in the tuple is the parsed color string turned into a
Color struct.
+
+
+
+
+ The hex segment parser, comprised of nom combinator functions, and IResult #
+
+
+
+
+
+
+Letβs look at the helper_fns::parse_hex_seg (the other 2 ways shown above do the same exact
+thing). The signature of this function tells nom that you can call the function w/ input
+argument and it will return IResult<Input, Output, Error>. This signature is the pattern that
+we will end up using to figure out how to chain combinators together. Hereβs how the map_res
+combinator is used by parse_hex_seg() to actually do the parsing:
+
+
+ take_while_m_n: This combinator takes a range of characters (2, 2) and applies the
+ function match_is_hex_digit to determine whether the char is a hex digit (using
+ is_ascii_hexdigit() on the char). This is used to match a valid hex digit. It returns a
+ &str slice of the matched characters. Which is then passed to the next combinator.parse_str_to_hex_num: This parser is used on the string slice returned from above. It simply
+ takes string slice and turns it into a Result<u8>, std::num::ParseIntError>. The error is
+ important, since if the string slice is not a valid hex digit, then we want to return this
+ error.
@@ -686,6 +685,235 @@
+
+
+
+ Why canβt we parse βπ
#2F14DFβ? #
+
+
+
+
+
+
+The reason we canβt parse π
#2F14DF is because the tag("#") combinator is used to
+match the # character at the very start of our input. Remember that the parser will try
+to eat the bytes from the start of the input to the end. This means that if the input
+doesnβt start with #, the parser will fail.
+
+If we have the requirement to parse a hex color code that doesnβt start with #, then we
+can modify the parser to handle this case. Hereβs one way in which we can do this.
+
+/// This is the "main" function that is called by the tests.
+fn hex_color_no_alpha(
+ input: &str,
+) -> IResult<
+ (
+ /* start remainder */ &str,
+ /* end remainder */ &str,
+ ),
+ Color,
+> {
+ let mut root_fn = preceded(
+ /* throw away "#" */
+ context("remove #", tag("#")),
+ /* return color */
+ tuple((
+ context("first hex seg", helper_fns::parse_hex_seg),
+ context(
+ "second hex seg",
+ intermediate_parsers::gen_hex_seg_parser_fn(),
+ ),
+ context(
+ "third hex seg",
+ map_res(
+ take_while_m_n(2, 2, helper_fns::match_is_hex_digit),
+ helper_fns::parse_str_to_hex_num,
+ ),
+ ),
+ )),
+ );
+
+ // Get chars before "#".
+ let pre_root_fn = take_until::<
+ /* input after "#" */ &str,
+ /* start remainder */ &str,
+ nom::error::VerboseError<&str>,
+ >("#");
+
+ if let Ok((input_after_hash, start_remainder)) = pre_root_fn(input) {
+ if let Ok((end_remainder, (red, green, blue))) =
+ root_fn(input_after_hash)
+ {
+ Ok((
+ (start_remainder, end_remainder),
+ Color::new(red, green, blue),
+ ))
+ } else
+ {
+ Err(nom::Err::Error(Error::new(
+ (input_after_hash, ""),
+ ErrorKind::Fail,
+ )))
+ }
+ } else {
+ Err(nom::Err::Error(Error::new((input, ""), ErrorKind::Fail)))
+ }
+}
+
+
+And this is what the tests would look like:
+
+#[test]
+fn parse_valid_color() {
+ let input = "\nπ\n#2F14DF\nπ
\n";
+ let result = dbg!(hex_color_no_alpha(input));
+ let Ok((remainder, color)) = result else {
+ panic!();
+ };
+ assert_eq!(remainder, ("\nπ\n", "\nπ
\n"));
+ assert_eq!(color, Color::new(47, 20, 223));
+}
+
+
+
+
+ What if we wanted to get better error reporting on what is happening? #
+
+
+
+
+
+
+We can use the context combinator to provide better error reporting. This is a very useful
+combinator that you can use to provide better error messages when parsing fails. However, when
+using it, we need to:
+
+
+ - Be careful of expressing the
nom error types as generic arguments to the parser
+functions, by using the nom::error::VerboseError type to get more detailed error
+messages which are used by nom::error::convert_error.
+ - This type needs to be passed as a generic argument to each parser that uses the
+
context combinator.
+
+
+Hereβs an example of this.
+
+use nom::{
+ bytes::complete::{tag, take_while_m_n},
+ combinator::map_res,
+ error::{context, convert_error},
+ sequence::Tuple,
+ IResult, Parser,
+};
+
+/// `nom` is used to parse the hex digits from string. Then
+/// [u8::from_str_radix] is used to convert the hex string into a
+/// number. This can't fail, even though in the function signature,
+/// that may return a [core::num::ParseIntError], which never
+/// happens. Note the use of [nom::error::VerboseError] to get more
+/// detailed error messages that are passed to
+/// [nom::error::convert_error].
+pub fn parse_hex_seg(input: &str) -> IResult<
+ &str,
+ u8,
+ nom::error::VerboseError<&str>
+> {
+ map_res(
+ take_while_m_n::<_, &str, nom::error::VerboseError<_>>(
+ 2,
+ 2,
+ |it| { it.is_ascii_hexdigit() }),
+ |it| u8::from_str_radix(it, 16),
+ )
+ .parse(input)
+}
+
+/// Note the use of [nom::error::VerboseError] to get more detailed
+/// error messages that are passed to [nom::error::convert_error].
+pub fn root(input: &str) -> IResult<
+ &str,
+ (&str, u8, u8, u8),
+ nom::error::VerboseError<&str>
+> {
+ let (remainder, (_, red, green, blue)) = (
+ context("start of hex color", tag("#")),
+ context("hex seg 1", parse_hex_seg),
+ context("hex seg 2", parse_hex_seg),
+ context("hex seg 3", parse_hex_seg),
+ )
+ .parse(input)?;
+
+ Ok((remainder, ("", red, green, blue)))
+}
+
+
+This just sets up our code to use context, but we still have to format the output of the
+error in a human readable way to stdout. This is where convert_error comes in. Hereβs
+how you can use it.
+
+#[test]
+fn test_root_1() {
+ let input = "x#FF0000";
+ let result = root(input);
+ println!("{:?}", result);
+ assert!(result.is_err());
+
+ match result {
+ Err(nom::Err::Error(e)) | Err(nom::Err::Failure(e)) => {
+ println!(
+ "Could not parse because ... {}",
+ convert_error(input, e)
+ );
+ }
+ _ => { /* do nothing for nom::Err::Incomplete(_) */ }
+ }
+}
+
+
+Hereβs the output of the test.
+
+Err(Error(VerboseError { errors: [("x#FF0000", Nom(Tag)), ("x#FF0000", Context("start of hex color"))] }))
+Could not parse because ... 0: at line 1, in Tag:
+x#FF0000
+^
+
+1: at line 1, in start of hex color:
+x#FF0000
+^
+
+
+Hereβs another test to see even more detailed error messages.
+
+#[test]
+fn test_root_2() {
+ let input = "#FF_000";
+ let result = root(input);
+ println!("{:?}", result);
+ assert!(result.is_err());
+
+ match result {
+ Err(nom::Err::Error(e)) | Err(nom::Err::Failure(e)) => {
+ println!(
+ "Could not parse because ... {}",
+ convert_error(input, e)
+ );
+ }
+ _ => { /* do nothing for nom::Err::Incomplete(_) */ }
+ }
+}
+
+
+Hereβs the output of this test.
+
+Err(Error(VerboseError { errors: [("_000", Nom(TakeWhileMN)), ("_000", Context("hex seg 2"))] }))
+Could not parse because ... 0: at line 1, in TakeWhileMN:
+#FF_000
+ ^
+
+1: at line 1, in hex seg 2:
+#FF_000
+ ^
+
@@ -741,6 +969,100 @@
and all the other intermediate types (Fragment, Block, etc) & enums required for parsing.
+
+
+
+ Other examples #
+
+
+
+
+
+
+
+
+
+
+ Related video #
+
+
+
+
+
+
+
+ You can get the source code for the examples in this
+repo.
+
+
+TK: add video here
+
+
+
+ References #
+
+
+
+
+
+
+nom is a huge topic. This tutorial takes a hands on approach to learning nom. However, the resources
+listed below are very useful for learning nom. Think of them as a reference guide and deep dive into
+how the nom library works.
+
+
+ - Useful:
+
+ - Source code examples (fantastic way to learn
nom):
+
+
+ - Videos:
+
+ - Intro from the author 7yrs old
+ nom 7 deep dive videos:
+
+ nom 6 videos (deep dive into how nom combinators themselves are constructed):
+
+
+
+ - Tutorials:
+
+
+ - Reference docs:
+
+
+
+
+ - Less useful:
+
+
+
diff --git a/docs/authors/nazmulidris/index.html b/docs/authors/nazmulidris/index.html
index bb0261b5..e9ac330b 100644
--- a/docs/authors/nazmulidris/index.html
+++ b/docs/authors/nazmulidris/index.html
@@ -14,11 +14,11 @@
-
+
+{"@context":"https://schema.org","@type":"BlogPosting","author":{"@type":"Person","name":"Nazmul Idris"},"dateModified":"2024-06-18T14:20:49-05:00","datePublished":"2024-06-18T14:20:49-05:00","description":"Nazmul is a software engineer focused on Rust, TUI, Web, and Android technologies.","headline":"Nazmulidris","mainEntityOfPage":{"@type":"WebPage","@id":"http://developerlife.com/authors/nazmulidris/"},"url":"http://developerlife.com/authors/nazmulidris/"}
diff --git a/docs/feed.xml b/docs/feed.xml
index 1ad8c268..fbea6756 100644
--- a/docs/feed.xml
+++ b/docs/feed.xml
@@ -1,4 +1,4 @@
-Jekyll 2024-06-10T19:56:05-05:00 http://developerlife.com/feed.xml developerlife.com Rust, TUI, Android, Web, Desktop, Cloud technologies, and UX engineering and design tutorials. Nazmul Idris Rust error handling with miette 2024-06-10T10:00:00-05:00 2024-06-10T10:00:00-05:00 http://developerlife.com/2024/06/10/rust-miette-error-handling 
+Jekyll 2024-06-18T14:20:49-05:00 http://developerlife.com/feed.xml developerlife.com Rust, TUI, Android, Web, Desktop, Cloud technologies, and UX engineering and design tutorials. Nazmul Idris Rust error handling with miette 2024-06-10T10:00:00-05:00 2024-06-10T10:00:00-05:00 http://developerlife.com/2024/06/10/rust-miette-error-handling
diff --git a/docs/sitemap.xml b/docs/sitemap.xml
index dbc0898a..a25eb7bb 100644
--- a/docs/sitemap.xml
+++ b/docs/sitemap.xml
@@ -2,11 +2,11 @@
http://developerlife.com/authors/nadiaidris/
-2024-06-10T19:56:05-05:00
+2024-06-18T14:20:49-05:00
http://developerlife.com/authors/nazmulidris/
-2024-06-10T19:56:05-05:00
+2024-06-18T14:20:49-05:00
http://developerlife.com/1998/12/01/xml-and-java-tutorial-part-1/