Add more info to each Chatbot Document

[AdamFipke]

Idea is to add total number of pages, whether or not parseAsSlides was true, and the size of the document to each document here:

Motivation is that it'd be nice to know what documents are really heavy that a course has uploaded.

For size of the document (in MB), that would be pretty easy to obtain since I think I save that on the chatbot-doc-pdf.entity.

For total number of pages... this might (?) be easy to do, but I think the main challenge here would be the backend since I don't think we collect that data in the chatbot document aggregates right now, so a funky migration could be required that estimates number of pages based on data from its chunks. Most of the work here would be backend work on the chatbot repo. @bhunt02 would know more

For parseAsSlides, I think we only use that during upload and then discard it and I don't think we save that information... but we should!

[bhunt02]

i'm absorbing this

[bhunt02]

so a couple things here:

• we don't track pages ... but we can (it will be nullable for the sake of reverse compatibility)
• parse as slides doesn't seem like something we need to store as it's never re-used but i'm eager to hear an argument as to why
• i feel uneasy about having to store the byte size in two places... maybe some stuff can be done on the helpme side when retrieving from CB to augment the response, but at most i'll create a generated column that reports the byte size from the chatbot

[AdamFipke]

• parse as slides doesn't seem like something we need to store as it's never re-used but i'm eager to hear an argument as to why

I guess when I'm viewing the chatbot documents of other courses, I would like to know if the user was properly using that feature. Like if they were making everything "parseAsPdf" maybe it's time to warn them. Or more likely they might've neglected the feature so their chunks suck

• i feel uneasy about having to store the byte size in two places... maybe some stuff can be done on the helpme side when retrieving from CB to augment the response, but at most i'll create a generated column that reports the byte size from the chatbot

Yeah, we can probably just augment the response with the data from the HMS. Each chatbot doc in HMS has the corresponding id from chatbot-side so we can use that. It's not the most efficient I guess but it's rare these queries are gonna pull more than 100 items

[bhunt02]

• parse as slides doesn't seem like something we need to store as it's never re-used but i'm eager to hear an argument as to why

I guess when I'm viewing the chatbot documents of other courses, I would like to know if the user was properly using that feature. Like if they were making everything "parseAsPdf" maybe it's time to warn them. Or more likely they might've neglected the feature so their chunks suck

• i feel uneasy about having to store the byte size in two places... maybe some stuff can be done on the helpme side when retrieving from CB to augment the response, but at most i'll create a generated column that reports the byte size from the chatbot

Yeah, we can probably just augment the response with the data from the HMS. Each chatbot doc in HMS has the corresponding id from chatbot-side so we can use that. It's not the most efficient I guess but it's rare these queries are gonna pull more than 100 items

they are paginated (as of my vector store refactor) so size of data retrieved is not a concern since we'll just filter by IDs

as for the parse as PDF thing: i'll probably add it then