Expose cloud and edge inference speed metrics #173
Conversation
There was a problem hiding this comment.
As a general comment, in the past, we have been asked by @bibireata and other people to remove these performance metrics from the response of cloud inference. I believe we should not expose this for cloud inference, maybe.
landingai/predict.py
Outdated
| # performance_metrics keeps performance metrics for the last call to _do_inference() | ||
| performance_metrics: Dict[str, int] = {} | ||
|
|
There was a problem hiding this comment.
Why store it as a global variable? Why not associate it with a Predictor instance?
There was a problem hiding this comment.
I guess my brain was running low on creativity 😞 ... I changed to a private variable to the class...thanks!
There was a problem hiding this comment.
Regarding the comment about performance metrics I think in the long term we need a tutorial on how to optimize models for speed and how to use metrics to profile and find problems. For now they are a bit obscure but at least now we expose them as part of the predict class 🤷♂️
This change helps users get a breakdown of the time spent during inference. This is useful to experiment with different inference pipelines.