Feedback: Confusing & Not intuitive API

[aaronclong]

I've been trying to follow your documentation (1, 2). However, I can't really make heads or tails of how to actually parse a document and its sections.

I can see the detail and care that when into designing the API, but I do think a simpler abstraction needs to be introduce. For instance, I've struggled for days to try and parse out a filing's sections and grab its text. While, I have gotten output it hasn't been usable, and I'm certain that text got truncated (all user error).

A big issue is just what method to call when since this works off inheritance. You have to do type checks all over the place. I think creating a higher level API or refactor might be needed to reduce the steep learning curve here.

I was kind of hoping this could just parse the tags automatically for me and put them into types.

[Elijas]

Hey @aaronclong, thanks for reaching out and trying out the package!

1. Could you link us to a document(s) you're trying to parse? And/or could you post some particular issues (screenshots/examples) that you're coming across?
2. As for this works off inheritance, could you provide a simple example of how the structure could be improved? I would love to hear your thoughts on it 🚀

I totally, understand the need for a simpler interface while also keeping it functional, if there's any way we could do it, it could help make the library more accessible for many

[agulli]

For instance, I've struggled for days to try and parse out a filing's sections and grab its text.

Probably, a good starting point is to have more examples. Starting from something simple like the request above

[Elijas]

Thanks for the suggestion! This makes sense. I added links to more examples to the README.md, you can find the full tutorial for extracting text from all of the document or specific sections, in this notebook

[aaronclong]

I don't know what a better abstraction would be for this. I have been mulling that over myself. I'll share my failed attempt with this library. In general, I prefer functional programming with composition over OPP with inheritance so admitting my own bias.

I was trying to parse out Google/Alphabet's 10Q

from sec_parser.processing_steps import AbstractProcessingStep
from sec_parser.semantic_elements import (
    TextElement,
    TableElement,
    TitleElement,
    TopSectionTitle,
    NotYetClassifiedElement,
    CompositeSemanticElement
)

# https://sec-parser.readthedocs.io/en/latest/notebooks/developer_guide.html#Handling-Multiple-Semantic-Elements-in-a-Single-HTML-Tag


class MyClassifier(AbstractProcessingStep):
    def __init__(self):
        super().__init__()
        # You can hold state in your processing steps
        self.processed_tags_count = 0
        self.unique_tag_names = set()
        

    # This method must be implemented when inheriting from AbstractProcessingStep
    def _process(self, elements):
        parsed = []
        for e in elements:
            self.processed_tags_count += 1
            self.unique_tag_names.add(e.html_tag.name)
            if e.html_tag.name == "b":
                parsed.append(TitleElement.create_from_element(e, ""))
            elif e.html_tag.name == "p":
                parsed.append(TextElement.create_from_element(e, ""))
            elif e.html_tag.name == "hr":
                continue
            elif e.html_tag.name == "div":
                children = e.html_tag.get_children()
            
                if len(children) > 0:
                    inner_elements = []
                    for c in children:
                        inner_elements.append(NotYetClassifiedElement(c))
                        print(c.name)
                        
                    parsed.append(CompositeSemanticElement.create_from_element(
                        e,
                        inner_elements=inner_elements,
                        log_origin="CompositeElementIdentificationStep",
                    ))
                
            else:
                parsed.append(e)
        # print(
        #     f"MyClassifier: Successfully processed {self.processed_tags_count} tags!\n"
        # )
        return parsed

classifier = MyClassifier()
def get_steps() -> list[AbstractProcessingStep]:
    return [MyClassifier()]

parsed = sp.Edgar10QParser(get_steps).parse(filings[0].html())
parsed[0].text

[aaronclong]

There were 20 to 30 other iterations that I tried that didn't work, but those are lost.