Help with Adding Custom Models to Ecologits

[vchemla]

Hi all,

We are currently using Ecologits to monitor the energy consumption of cloud-based LLMs (Azure OpenAI), and it seems to be working well.

Now, we would like to run a model from Hugging Face Hub (phi2) on our local infrastructure.

However, we couldn’t find the option for a 'Custom' model as mentioned in the Ecologits calculator within the Ecologits packages.

Could you help us with this or perhaps create a new issue to add support for custom models in the list of Ecologits providers?

Thank you for your assistance!

[adrienbanse]

Hey @vchemla,

Thanks a lot for using EcoLogits. :-)

First of all, Ecologits really aims at approximating cloud-based requests impacts. If you run your model locally, I would advise to use other packages such as CodeCarbon (https://codecarbon.io/), that will lead to more precise results!

If however you want to approximate custom cloud-based models, as of today, the only way to do that is to directly call the compute_llm_impacts function from

ecologits/ecologits/impacts/llm.py

Line 424 in 0cd0df2

def compute_llm_impacts(

For each request, you'll have to provide

• The number of active parameters of the custom model
• The number of total parameters of the custom model
• The number of output token (that you need to find in the answer)
• The request latency (that you can measure with time for example)
• The electricity mix. For the zone zone, you can find it with our repository as follows

from ecologits.electricity_mix_repository import electricity_mixes
electricity_mix = electricity_mixes.find_electricity_mix(zone=zone)
if_electricity_mix_adpe=electricity_mix.adpe
if_electricity_mix_pe=electricity_mix.pe
if_electricity_mix_gwp=electricity_mix.gwp

This is the way it is coded right now in the calculator. Right now, it is not possible to wrap a custom request with an impact (and seems technically complex).

[vchemla]

Thank you for providing these specifications.

In fact, we are currently conducting a comparison between CodeCarbon and Ecologits.

[adrienbanse]

We would definitely be interested to see the results, if you agree!

[vchemla]

Yes, I will add our results here

[vchemla]

This is our result based on our experiment:

Hardware:

• CPU: 16 cores
• RAM: 32 GiB
• No GPU
• OS: Ubuntu 24.04.1 LTS - running in server mode. Only the experiment is running, with no additional processes (strict minimum).

Here, we compare Ecologits and Codecarbon on a small local LLM: phi2.

Important points to consider:

• Codecarbon does not account for the cooling energy used, while ecologits does (details here).
• Ecologits, by hypothesis, estimates energy usage as if a GPU (like an A100) is being used (details here).

Note

Please note that these values are experimental

Codecarbon seems not to follow the Ecologits curve.
We were wondering if the linear behaviour was a consequence of Ecologits modelisation ?

[adrienbanse]

@vchemla This is really interesting, thanks a lot! I'm quite happy the approximation in fact 👍 cc @samuelrince

The request energy is divided in two parts, the GPU energy and the server (without GPU) energy. The first one is linear with the number of tokens, where we approximate the energy by token with open source HF data (we fit a curve on the inverse of the "token/energy" column). The second one is linear with the delay, which is itself linear with the number of tokens (hence the linear orange curve).

If I'm not wrong, in Ecologits, the second term is very small compared to the first one. It could be that, in the case of your computer, this second term is larger and grows not linearly with the number of tokens... It is however surprising to me as the time should logically be linear with the number of output tokens, so I don't have any clear explanation right now. 😢