[BUG] - Update fails on RT-AX86S

[DrEthan77]

Describe the bug
Whenever the update script is run on my router (even manually), it fails due to not enough RAM.

To Reproduce
Steps to reproduce the behavior:

1. SSH into the router
2. Open AMTM
3. Open MerlinAU
4. Check for updates

Expected behavior
The update script to actually update the router

Screenshots

Logs
If needed, I can get the full log file. Basically, all it does is try to free up RAM 3 times, then says it doesn't have enough.

Router Information (please complete the following information):

• Router Model: RT-AX86S
• Firmware Version: 3004.388.9.0
• Browser: N/A (I'm using SSH)
• Other Scripts?: I do have quite a few other scripts active, such as Skynet, I posted a screenshot above.

Additional context
I do have a 2GB swap file configured, and it seems to see that, but it does not recognize it as free RAM space.

[ExtremeFiretop]

We cannot use swap space when the USB is going to be unmounted for the flash. The router itself must have enough ram for the actions of flashing.

If you reboot does it go through?

[DrEthan77]

Makes sense, unfortunately, no, it does not work after a reboot. I tried doing it manually twice, one after another, and the log I posted is from the second time, it's almost identical to the first log.

[ExtremeFiretop]

Makes sense, unfortunately, no, it does not work after a reboot. I tried doing it manually twice, one after another, and the log I posted is from the second time, it's almost identical to the first log.

I'll discuss with Martinski but we haven't run into this since the very early days of the project to be honest. I'll go back through some old PRs to see how we landed on this solution.

[ExtremeFiretop]

Makes sense, unfortunately, no, it does not work after a reboot. I tried doing it manually twice, one after another, and the log I posted is from the second time, it's almost identical to the first log.

@Martinski4GitHub

It's something about these routers I swear:
Find the PR: #167 where we had a user with the same issue.

At this moment the only thing I can think of doing is to lower the overhead required in this function

_GetRequiredRAM_KB_()
{
    local theURL="$1"
    local zip_file_size_bytes  zip_file_size_kb  overhead_kb
    local total_required_kb  overhead_percentage=50

    # Size of the ZIP file in bytes #
    zip_file_size_bytes="$(curl -LsI --retry 4 --retry-delay 5 "$theURL" | grep -i Content-Length | tail -1 | awk '{print $2}')"
    # Bytes to KBytes #
    zip_file_size_kb="$((zip_file_size_bytes / 1024))"

    # Calculate overhead based on the percentage #
    overhead_kb="$((zip_file_size_kb * overhead_percentage / 100))"

    total_required_kb="$((zip_file_size_kb + overhead_kb))"
    echo "$total_required_kb"
}

If we lowered the overhead requirement from 50% of the ZIP size, down to 20% the script would flash, but I don't know how safe that would be. Worst thing would be memory starvation while flashing

[ExtremeFiretop]

@Martinski4GitHub

Or we re-consider the minimum cushion solutuion which I see we removed here:

0993ac5

[ExtremeFiretop]

@Martinski4GitHub

I see in this release why we dropped that idea; we were having some misreporting memory issues potentially:
#174

Most of our long discussion on the matter can be found in this PR:
#169

[Martinski4GitHub]

@DrEthan77 & @ExtremeFiretop,

I know of at least 2 users (one of my cousins and a friend of mine) who have the RT-AX86S router, and they have reported no issues at all when the MerlinAU script performed the F/W Update from 3004.388.9.0 to 3004.388.9.2 versions some time within the last 3 weeks or so.

Here's an old screenshot from my cousin's router running the latest 3004.388.9.2 F/W version:

This evening, I connected remotely to my cousin's router and got the logfile of the F/W Update performed by the MerlinAU script:

As you can see, the router initially started off with plenty of RAM available (170357 KB =~ 166MB), and even after downloading the F/W image zip file and extracting its contents, there was still enough RAM available (130698 KB =~ 127MB) to proceed with the F/W Update. Note that, in this case, the initial amount of RAM available decreased by ~39MB before the actual F/W flash was even started, which is indicative of how much RAM is needed just as a "cushion" to even begin the actual F/W flash process, which loads the entire image into RAM.

IMO, you need to find out if one or more processes are running on the router, which are taking a lot of RAM and continue to hold on to it while MerlinAU is preparing the F/W Update. For example, is there a way to temporarily stop & disable Skynet before running MerlinAU? If so, you could try that to troubleshoot one possible scenario.

Bottom line is that the RT-AX86S router can be successfully updated via MerlinAU as long as enough RAM is available to proceed safely with the F/W flash. If, for some reason, RAM pages are not being freed by some processes, you need to find out what the problem is and why it is happening, because that's not normal behavior.

For more context, here's a screenshot showing the add-ons that are installed on the RT-AX86S:

My 2 cents.

[ExtremeFiretop]

My 2 cents.

I'm leaning that way too. Seeing the minimum cushion be around what we discussed the first time (we had landed on 40MB the first time) gives me some joy knowing that hasn't changed. But I also do know we've had multiple successful tests with this router... So it becomes a question why can't this one?

Ideally If @DrEthan77 can run "top" or some tool to identify the high memory offender, we might or might not be able to do something programmatically about it.

[Martinski4GitHub]

My 2 cents.

I'm leaning that way too. Seeing the minimum cushion be around what we discussed the first time (we had landed on 40MB the first time) gives me some joy knowing that hasn't changed. But I also do know we've had multiple successful tests with this router... So it becomes a question why can't this one?

Ideally If @DrEthan77 can run "top" or some tool to identify the high memory offender, we might or might not be able to do something programmatically about it.

Yes, exactly. Running the "top" utility before starting MerlinAU and while the script is running to perform the F/W Update may provide useful screenshots of what's going on. Another good utility is "htop" (install via the Entware package "htop") because it gives more information.

Again, we're looking for processes that have allocated MBytes of RAM for themselves and don't seem to relinquish that memory within a few seconds. IOW, it's OK for a process to acquire lots of RAM for a specific task/operation, but after a few seconds (perhaps a minute or two) that memory should be freed so it returns to the memory pool available for other processes; otherwise, that might indicate a memory leak or bad memory management within the process itself.

[DrEthan77]

Before manual f/w update

After

[ExtremeFiretop]

@DrEthan77

From the screenshow provided the network map appears to be the biggest culprit but that in itself is only about 9MB.
Pretending we clear that memory, you would still get stuck that only clears about ~8,704KB bringing your total up to 94,680Kb when you need 114,240Kb

Is the "before manual flash" a live example where we know MerlinAU was getting stuck?
Based on those screenshots I'm not sure what the next step would be...

@Martinski4GitHub did you want to consider re-vamping the solution or simply saying that this specific router for (x) reason does not have the memory to continue?

One thing I noticed is that the "MEMTotal" in his screenshots lists 410MB as the total but I thought it was a 512MB model router?
Can you confirm if you have similar output from the RT-AX86S router you have access too @Martinski4GitHub ?

[DrEthan77]

Is the "before manual flash" a live example where we know MerlinAU was getting stuck? Based on those screenshots I'm not sure what the next step would be...

Im not really sure what you mean by this, this is the Htop from right before I ran a manual f/w update check and when I ran it it got stuck.

Also, a really weird thing, I disabled Skynet, and the memory used didn't change at all, which I don't understand.