Memory analysis - 2023 edition

[JF002]

This post is a follow-up and an update to the previous memory analysis done back in 2020.

I'll focus on RAM usage (static allocation, heap and stack).

The goals of this new analysis are

• to have a better overview of the current RAM usage
• to find solutions to free RAM space so we get a bit of headroom to add new functionalities
• to find solutions that would allow us to more easily track and monitor memory usage

Current RAM usage

I took those measurement on tag 1.11.0 built with GCC 10:

Build info returned by the linker:

Memory region         Used Size  Region Size  %age Used
           FLASH:      417504 B       480 KB     84.94%
             RAM:       54376 B        64 KB     82.97%

Build info by arm-none-eabi-size :

   text	   data	    bss	    dec	    hex	filename
 417504	    940	  53432	 471876	  73344	pinetime-app-1.11.0.out

With more details:

./arm-none-eabi-size -A  /home/jf/git/PineTime/cmake-build-release-nrf52-gcc10/src/pinetime-app-1.11.0.out
/home/jf/git/PineTime/cmake-build-release-nrf52-gcc10/src/pinetime-app-1.11.0.out  :
section               size        addr
.text               417496           0
.ARM.exidx               8      417496
.data                  940   536870912
.bss                 53416   536871856
.noinit                 16   536925272
.heap                 4096   536925288
.stack_dummy          1024   536925288
.ARM.attributes         48           0
.comment                73           0
.debug_info        8337396           0
.debug_abbrev       492013           0
.debug_loc         1034813           0
.debug_aranges       36200           0
.debug_ranges       152304           0
.debug_macro        627226           0
.debug_line        1861998           0
.debug_str         1977356           0
.debug_frame        122920           0
.stab                  204           0
.stabstr               441           0
Total             15119988

Static allocation

The total available RAM is 65536B (64KB).

arm-none-eabi-size and linker do not take "global" stack (1024B) and heap (4096) into account.
Which mean that the reported "% used" might be a fix confusing : they assume that the whole 64KB are available for static allocation, which is not true. The memory available for allocation in in fact 65536B - 1024B - 4096B = **60416B**.

InfiniTime 1.11 uses 940B in section .data (initialized RAM variables) and 53416B in .bss (uninitialized variables). The total is 54356B.

--> From this, we can conclude that we currently have 60416B - 54356B = 6060B that are not allocated and that are available for future use.

Details per module (static RAM usage, does not take stack + heap into account):

• FreeRTOS heap : 17408
• LVGL heap : 14336
• LVGL frame buffer : 3952
• Nimble msys init 1 = 3504
• System task = 2852
• Nimble ble_att_svr_prep_entry_mem = 768
• Nimble ble_hci_ram_evt_hi_buf = 576
• Nimble ble_hci_ram_evt_lo_buf = 576
• Notifications = 556
• Nimble g_ble_ll_conn_sm = 392
• Nimble ble_sm_proc_mem = 384
• HR App = 304

Heaps

InfiniTime allocates 3 heaps of memory. Those heaps are used for dynamic memory allocation at runtime.

• The "global" heap initialized by the standard library : 4096B. malloc() and new() allocate memory from this heap.
• The FreeRTOS heap : 17KB. FreeRTOS uses this heap to allocate memory for OS objects (mutex, notifications,...) and also for the stack of the tasks.
• The LVGL heap : 14KB. It is obviously used by LVGL to allocate memory for the UI.

What's using malloc() and new()?

• MusicService for the std::string (they should really be replaced by const char *).
• WeatherService for the std::unique_ptr.
• LittleFS
  • Initialization : 48B
  • FileOpen() : 16B for each call
• NimBLE :
  • ble_att_svr_start - 2100B (proportional to the number of ble attributes; currently : 20B/attribute, and we have 105 of them)
  • ble_gatts_start - 72 + 112
  • ble_ll_task - 24
• LoadApp : depends on the app / clockface

NOTE : from those numbers, we can easily see that NimBLE allocates memory for each BLE attribute. We must keep that in mind when designing new BLE APIs!

Heaps unification

I have this idea in mind for quite some time now : let's try to unify all those 3 heaps into a single one. This should have many advantages:

• Less overhead since we won't have to "over allocate" the heap 3 times.
• Easier RAM usage tracking and monitoring.

Using the FreeRTOS heap as our unique heap looks like a good idea to me. It is design with constrained embedded use-cases in mind and it provides tools to monitor and track the memory usage.

On the other side, the LVGL one is dedicated to LVGL and cannot easily be used by other software module, and the "global" heap does not provide any way that I know to limit and monitor its usage.

LVGL supports custom heaps so replacing the LVGL one with the FreeRTOS implementation shouldn't be an issue.

It should be quite easy to specify a custom memory allocator for LoadApp() so that apps and watchfaces are allocated on the FreeRTOS heap as well.

Finally, Nimble and LittleFS call malloc() directly, and I couldn't find any way that would allow to easily switch to another memory heap allocator. I plan on checking multiple solutions (modify those library so they call our custom allocator, using the linker to replace calls to malloc() to a custom function,...).

[JF002]

I've finally had the time to make some progress on this topic.

Monitoring of the memory usage:

FreeRTOS provides a few functions that return the current heap usage and the all time highest heap usage.
Newer versions of FreeRTOS provide a bit more information with vPortGetHeapStats : number of free blocks, number of allocations/frees,... Upgrading to a newer version of FreeRTOS could be a good idea in the future, but not necessary right now.

InfiniSim

Using a unique heap from FreeRTOS should also help InfiniSim more accurate regarding memory usage : InfiniSim would be constrained with the same amount of memory than the PineTime (as opposed to now, where InfiniSim can use all the memory from the host computer).

Testing right now

I implemented the unique heap in InfiniTime and I'm currently testing it. Here's a summary of the changes:

In src/CMakeLists.txt :

• Set __HEAP_SIZE to 0, which removes the heap from the std library.
• Add -Wl,-wrap,malloc -Wl,-wrap,free to the linker flags so that I can override malloc and free.

In FreeRTOSConfig.h

• Set the heap size to 40KB
• Enable "check for stack overflow"
• Enable the malloc failed hook.

In lv_conf.h

• Set LV_MEM_CUSTOM to 1 and specify pvPortMalloc and vPortFree as alloc/free functions.

In main.cpp

• Provide an implementation for the allocation failed hook and the start overflow error (just log and increment a counter)
• Implement _wrap_malloc() and _wrap_free() (which call alloc/free functions from FreeRTOS).

In the SystemInfo App:

• Remove the information about the LVGL memory usage and replace them with the info from FreeRTOS heap. "Alloc / stack" display the number of allocation error and stack overflow error (should always display 0), "free heap" is the current amount of memory currently available, and minimum is the lowest available memory available in the heap since the beginning.
  

What's next ?

I'll test those changes in the next few days to see how it behaves. I'll use the casio watchface as this is the one that use the most memory (for the font).

I would also like to see how we could use the information during development to check for stack overflow/failed allocation, and integrate those changes in InfiniSim.

[JF002]

Using the linker to override malloc() and free() does not look such a good idea. And this would not allow InfiniSim to use the same memory manager than InfiniTime. But there's a much simpler solution : simply define those functions so that the linker will use them instead of the ones from libc.

According to the documentation, we have to override malloc(), free(), calloc() and realloc()

void* malloc(size_t size);
void free(void* ptr);
void* calloc(size_t num, size_t size);
void* realloc( void *ptr, size_t newSize):

At first, I was tempted to return NULL in calloc() and realloc() since they are not used in the code... except that NimBLE actually uses realloc() (https://github.com/apache/mynewt-nimble/blob/master/nimble/host/src/ble_gatts.c#L1965)

realloc() is supposed to either extend the current memory space to accommodate the new size or allocate a new memory space for new size and copy the data from the old to the new buffer.

FreeRTOS does not provide a function that maps to realloc. We have 2 options here: write a naive implement of the realloc function :

void* realloc( void *ptr, size_t newSize) {
  (void)(newSize);
  uint8_t* bytePtr = ptr;

  uint8_t* memory = pvPortMalloc(newSize);
  if(memory == NULL) {
    return NULL;
  }

  // This memcpy does InvalidReads because it reads outside of the memory allocated to ptr.
  memcpy(memory, bytePtr, newSize);

  vPortFree(ptr);

  return memory;
}

This is fine... but not completely : if newSize is greater than the older size, the call to memcpy() effectively access to memory that was not allocated to ptr since we do not know the size that was previously allocated. It should not trigger any issue since there's no MMU on the PineTime, but still, this is something we should avoid.

The other option consists in implementing our own memory manager (based on heap_4.c that implements a realloc() functionality.

Here's what an implement of pvPortRealloc() looks like:

void *pvPortRealloc( void *pv, size_t xWantedSize ) {
 // Local variable
 size_t move_size;
 size_t block_size;
 BlockLink_t* pxLink;
 void* pvReturn = NULL;
 uint8_t* puc = (uint8_t*) pv;

 if (xWantedSize > 0) {
   if (pv != NULL) {
     // The memory being freed will have an BlockLink_t structure immediately before it.
     puc -= xHeapStructSize;

     // This casting is to keep the compiler from issuing warnings.
     pxLink = (void*) puc;

     // Check allocate block
     if ((pxLink->xBlockSize & xBlockAllocatedBit) != 0) {
       // The block is being returned to the heap - it is no longer allocated.
       block_size = (pxLink->xBlockSize & ~xBlockAllocatedBit) - xHeapStructSize;

       // Allocate a new buffer
       pvReturn = pvPortMalloc(xWantedSize);

       // Check creation and determine the data size to be copied to the new buffer
       if (pvReturn != NULL) {
         if (block_size < xWantedSize) {
           move_size = block_size;
         } else {
           move_size = xWantedSize;
         }

         // Copy the data from the old buffer to the new one
         memcpy(pvReturn, pv, move_size);

         // Free the old buffer
         vPortFree(pv);
       }
     } else {
       // pv does not point to a valid memory buffer. Allocate a new one
       pvReturn = pvPortMalloc(xWantedSize);
     }
   } else {
     // pv points to NULL. Allocate a new buffer.
     pvReturn = pvPortMalloc(xWantedSize);
   }
 } else {
   // Zero bytes requested, do nothing (according to libc, this behavior implementation defined)
   pvReturn = NULL;
 }

 // Exit with memory block
 return pvReturn;
}