Issue 473 Add optional debug/info logging to extract_xri script

[rminsil]

This PR adds optional logging to the script.

Currently the script makes certain assumptions about the data which can make it a bit brittle causing it to crash mid-processing.

Some example assumptions:

• all columns exist with the expected schema
• no embedded newlines (e.g. each row is on one line)
• id's can all be parsed to int

Logging will help to quickly identify the row of data breaking it.

Currently the logging is off by default. Enabling with -log [PATH] sends the logs to the file specified.

Note this PR is built on top of another unmerged PR #473. I'll rebase this and target master when #473 is merged.

---

This change is 

[ddaspit]

Reviewed all commit messages.
Reviewable status: 0 of 1 files reviewed, 2 unresolved discussions (waiting on @rminsil)

---

silnlp/common/extract_xri.py line 52 at r1 (raw file):

logger = logging.getLogger(__name__)

You don't want to use __name__ here, because it is set to __main__. You will want to use the following:

LOGGER = logging.getLogger(__package__ + ".extract_xri")

---

silnlp/common/extract_xri.py line 197 at r1 (raw file):

        "-output", help="Optional path to the output directory where extract files are generated", type=str
    )
    parser.add_argument("-log", help="Optional path to enable logging. Logs are sent to the file passed", type=str)

In silnlp, we always have logging enabled and we output directly to stdout. I don't think there is a need for logging to a file.

[rminsil]

In silnlp, we always have logging enabled and we output directly to stdout. I don't think there is a need for logging to a file.

Each line of the file produces some logs and the files are around 4K entires, so you're typically getting about 8K-12K lines of logging, that's why I thought it better to have it off by default and send it to a file if it's enabled. Otherwise it's like a snowstorm in the terminal and it will usually overrun the terminal's buffer. We can pipe it to a file I guess but that's an extra step.

I was going to reserve stdout or stderr for the more meaningful output like statistics and summaries of data quality issues noticed.

Let me know what you think, I'm happy to do whatever.

UPDATE: Sorry I think I misunderstood some of what you're saying and was missing some context.

[rminsil]

@ddaspit I found the logging config in the parent __init.py__ and understand what you mean now.

I've removed the option to log to a file.

I've changed the logger name to be specific to the module. I'll make an analogous change to the repair logger used in #515.

I've added an optional flag to allow setting the debug level from the command line.

[ddaspit]

Reviewed 1 of 1 files at r2, all commit messages.
Reviewable status: complete! all files reviewed, all discussions resolved (waiting on @rminsil)