eyecite available in rust

[jakeswenson]

I wanted to let everyone know that I have started work to port both eyecite and reporters-db into rust.

• eyecite crate in crates.io

• reporters-db create in crates.io

This work is because I would like to use eyecite in javascript, and so if I port it to rust, then I can (hopefully) use rust's support to build to wasm. I recently just got tokenization mostly working (there is still some cleanup to do with, and I need to unify the reporters/laws/journals) you can take a look at the test code here: https://github.com/jakeswenson/eyecite/blob/oxide/src/tokenizers.rs#L140

Secondly is that rust libraries can be easily exposed in python using PyO3 and maturin. I want to expose the rust libs to python so that I can to piggyback on the tests in this project. To be clear my intent is not replace the hard work down here in python, but there may be a possibility to merge in the future that is worth considering... 🤷🏻‍♂️

I would love to hear thoughts/feedback/concerns from the maintainers of this project on any of this work.

Thank you!

[mlissner]

Exciting! A few thoughts:

• How do you plan to keep up to date in your fork?
• Should we link to your fork from our readme and vice versa?
• I'm tempted to say that your fork should get merged back in here, but I don't know rust and don't know what our org would get from the extra work of maintaining it.
• Do you expect any other advantages (performance? security?) by running this in rust?

Finally, what are you working on, big picture?

[jakeswenson]

Hey, thanks for the quick response! Answers below.

Exciting! A few thoughts:

• How do you plan to keep up to date in your fork?

• Plan? what plan? :D Honestly I haven't thought about handling updates. If you have a suggestion I'm all ears.
  • That being said, for reporters-db JSON files it is currently relatively easy to stay in sync with those changes (just need to merge in changes from upstream.) I considered creating a new repo and including the JSON as a submodule/subtree but decided to just fork in the event we would like to merge this upstream in the future.

• Should we link to your fork from our readme and vice versa?

• Good idea. I am totally happy to make a clear link back to the main repo's in my forks. As for linking to my forks, right now this is still really nascent and there will probably be a lot of churn so I'd say probably not yet. I think it's a great idea in the future and I'll be happy to submit a PR to this repo then. If we make it clear that it's still very much work in progress maybe that's an option.

• I'm tempted to say that your fork should get merged back in here, but I don't know rust and don't know what our org would get from the extra work of maintaining it.

• I agree with the thought, and I also don't you know that it's worth it for you all to take on that burden yet. I think it's OK for us to feel this one out as it goes. As I add more functionality to expose the same functionality that is available in the python version it may become more interesting for you to leverage that work.

• Do you expect any other advantages (performance? security?) by running this in rust?

• Honestly perf isn't/wasn't my goal, however, yes I suspect (hope?) the native rust version to be faster. My goal was to make this great work available to be used in more languages. By porting it to rust I hope to make it available to any language that can compile to WASM (mainly javascript), to other rust apps natively, and potentially expose this rust core back to python so that you could leverage it too if we find its worth it. I considered exposing this python implementation through a docker container, but lately, I've been trying to avoid docker.

Finally, what are you working on, big picture?

:D I Love the question.
Big picture, I'm not sure how much I'm ready to share, but basically, I would like to explore legal documents with citations in them and rather than writing my own janky regexs to pull out citations I'd like to use the much more comprehensive and awesome work you've done here.

[mlissner]

This all sounds great. Sounds like from our end we should hang tight and see what develops.

I'm not sure you'll eke out too much more performance from this, actually, but we'll see. It's pretty heavily optimized and I think it even uses some CPU-level codes to be faster, but perhaps the Python overhead on that is greater than I think. We'll soon find out, I suppose!