[HOW-TO] Fix MongoDB & tcmalloc MmapAligned Crash on RK3588
Posted: Fri Jul 18, 2025 6:33 am
### **[HOW-TO] Fix MongoDB & tcmalloc `MmapAligned` Crash on RK3588 (CM3588, Orange Pi 5) by Compiling Kernel with 48-bit VA**
#### **1. The Problem**
If you are using a Rockchip RK3588 based board (like the FriendlyELEC CM3588, Orange Pi 5/5 Plus, etc.) with a vendor-provided Linux image (e.g., Ubuntu 24.04, Debian 12), you may find that certain applications crash on startup. This is especially common with MongoDB (versions 8.0, and docker MongoDB) and other software that uses Google's `tcmalloc` library, like the Envoy Proxy.
I have already reported in this topic viewtopic.php?f=84&t=4553 about the problem with running MongoDB in docker, you can avoid it by running mongodb of older version DIRECTLY (non dockerized), say version 7.0 or if you need version 8.0 then see below.
The symptoms are clear and consistent:
* The `mongod` process fails to start, exiting with codes like `6 (SIGABRT)` or `62`.
* The logs show a fatal memory allocation error, even with plenty of free RAM. The key error messages are:
```
MmapAligned() failed - unable to allocate with tag (hint=..., size=1073741824, alignment=1073741824) - is something limiting address placement?
```
and
```
FATAL ERROR: Out of memory trying to allocate internal tcmalloc data (bytes=...)
```
This issue happens whether you install MongoDB directly or run it inside a Docker container, because the problem lies with the host system's kernel.
#### **2. The Root Cause**
The issue is the **Virtual Address (VA) space** provided by the Linux kernel.
* Applications like MongoDB (using the WiredTiger engine with `tcmalloc`) need to map very large (e.g., 1GB), contiguous blocks of virtual memory.
* Many default ARM64 kernels for these boards are compiled with a **39-bit virtual address space** (`CONFIG_PGTABLE_LEVELS=3`). This smaller address space makes it extremely difficult for the kernel to find a place for these large, aligned memory requests, causing the allocation to fail.
* The solution is to enable a **48-bit virtual address space** (`CONFIG_PGTABLE_LEVELS=4`), which gives the memory manager ample room to fulfill these requests.
#### **3. The Solution: Recompile the Kernel**
The definitive solution is to recompile the Linux kernel for your board with the 48-bit virtual address space enabled. The following guide provides the complete, corrected steps to do this directly on your device.
---
### **Step-by-Step Kernel Compilation Guide**
#### **Step 1: Setup the Build Environment**
First, install the necessary tools and compiler on your CM3588.
```bash
# Update your package list
sudo apt update
# Install required packages using FriendlyARM's recommended script
sudo apt install -y curl
sudo bash -c "$(curl -fsSL https://raw.githubusercontent.com/frien ... install.sh)"
# Activate the new environment settings for your current terminal session
# IMPORTANT: Note the space after the dot '.'
. ~/.bashrc
# Verify the compiler is working
aarch64-linux-gnu-gcc -v
```
#### **Step 2: Download the Kernel Source Code**
Download the kernel source provided by FriendlyELEC for your board.
```bash
# Create a workspace
mkdir -p ~/development
cd ~/development
# Clone the correct kernel version (6.1.y branch for RK3588)
git clone https://github.com/friendlyarm/kernel-rockchip --depth 1 -b nanopi6-v6.1.y kernel-rockchip
cd kernel-rockchip
```
#### **Step 3: Configure the Kernel for 48-bit VA**
This is the most important step. You will change the kernel's default configuration.
```bash
# Load the default configuration for the board
make ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- nanopi6_linux_defconfig
# Open the interactive configuration menu
make ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- menuconfig
```
Inside the text-based menu, use the arrow keys to navigate:
1. Select `Kernel Features` --->
2. Select `Virtual address space size` --->
3. Select **`48-bit`** and press `<Enter>`.
4. Navigate to `< Save >`, press `<Enter>`, and confirm the default filename `.config`.
5. Navigate to `< Exit >` and press `<Enter>` repeatedly to close the menu.
#### **Step 4: Compile the Kernel and Modules**
Now, build the kernel. This will take a significant amount of time. Grab a coffee.
```bash
# Compile the kernel image and device tree files
# -j$(nproc) uses all available CPU cores to speed up the process
make ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- nanopi6-images -j$(nproc)
# Compile the kernel modules
make ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- modules -j$(nproc)
```
#### **Step 5: Install the New Kernel and Modules (Corrected Method)**
With the compilation finished, you will now install the new components onto your live system.
```bash
# Backup your existing kernel modules folder (as a precaution)
sudo mv /lib/modules/$(uname -r) /lib/modules/$(uname -r).bak
# Install the new modules to the system's module directory
sudo make ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- INSTALL_MOD_PATH=/ modules_install
# Now, write the new kernel and resource images to the eMMC
# Note on File Paths: The 'nanopi6-images' target conveniently places the final
# kernel.img and resource.img files in the top-level directory of the source
# code (i.e., your current directory: ~/development/kernel-rockchip).
# Write resource.img to partition 4 of the eMMC
sudo dd if=resource.img of=/dev/mmcblk2p4 bs=1M
# Write kernel.img to partition 5 of the eMMC
sudo dd if=kernel.img of=/dev/mmcblk2p5 bs=1M
```
#### **Step 6: Reboot and Verify**
The new kernel is installed. Reboot to activate it.
```bash
sudo reboot
```
After the system restarts, log in and verify the change. (CHECK STEPS MAY NOT WORK - THE ULTIMATE CHECK STEP IS YOU RUN MONGO v8.0 - see Step 7). if you want proper "checks" then ask Gemini/ChatGPT, these check did not work on my board, i skipped to running MongoDB.
1. Check the new kernel version (it will likely have changed from the original).
```bash
uname -r
```
2. **Confirm the 48-bit VA is active.** This is the most reliable method:
```bash
zcat /proc/config.gz | grep CONFIG_PGTABLE_LEVELS
```
**The correct output must be:** `CONFIG_PGTABLE_LEVELS=4`
#### **Step 7: Start and Test MongoDB**
The underlying problem is now fixed. You can start your MongoDB instance.
```bash
# Start the mongod service
sudo systemctl start mongod
# Check its status
sudo systemctl status mongod
```
You should now see `Active: active (running)`. The `MmapAligned` error will be gone, and your MongoDB server will be fully functional.
#### **1. The Problem**
If you are using a Rockchip RK3588 based board (like the FriendlyELEC CM3588, Orange Pi 5/5 Plus, etc.) with a vendor-provided Linux image (e.g., Ubuntu 24.04, Debian 12), you may find that certain applications crash on startup. This is especially common with MongoDB (versions 8.0, and docker MongoDB) and other software that uses Google's `tcmalloc` library, like the Envoy Proxy.
I have already reported in this topic viewtopic.php?f=84&t=4553 about the problem with running MongoDB in docker, you can avoid it by running mongodb of older version DIRECTLY (non dockerized), say version 7.0 or if you need version 8.0 then see below.
The symptoms are clear and consistent:
* The `mongod` process fails to start, exiting with codes like `6 (SIGABRT)` or `62`.
* The logs show a fatal memory allocation error, even with plenty of free RAM. The key error messages are:
```
MmapAligned() failed - unable to allocate with tag (hint=..., size=1073741824, alignment=1073741824) - is something limiting address placement?
```
and
```
FATAL ERROR: Out of memory trying to allocate internal tcmalloc data (bytes=...)
```
This issue happens whether you install MongoDB directly or run it inside a Docker container, because the problem lies with the host system's kernel.
#### **2. The Root Cause**
The issue is the **Virtual Address (VA) space** provided by the Linux kernel.
* Applications like MongoDB (using the WiredTiger engine with `tcmalloc`) need to map very large (e.g., 1GB), contiguous blocks of virtual memory.
* Many default ARM64 kernels for these boards are compiled with a **39-bit virtual address space** (`CONFIG_PGTABLE_LEVELS=3`). This smaller address space makes it extremely difficult for the kernel to find a place for these large, aligned memory requests, causing the allocation to fail.
* The solution is to enable a **48-bit virtual address space** (`CONFIG_PGTABLE_LEVELS=4`), which gives the memory manager ample room to fulfill these requests.
#### **3. The Solution: Recompile the Kernel**
The definitive solution is to recompile the Linux kernel for your board with the 48-bit virtual address space enabled. The following guide provides the complete, corrected steps to do this directly on your device.
---
### **Step-by-Step Kernel Compilation Guide**
#### **Step 1: Setup the Build Environment**
First, install the necessary tools and compiler on your CM3588.
```bash
# Update your package list
sudo apt update
# Install required packages using FriendlyARM's recommended script
sudo apt install -y curl
sudo bash -c "$(curl -fsSL https://raw.githubusercontent.com/frien ... install.sh)"
# Activate the new environment settings for your current terminal session
# IMPORTANT: Note the space after the dot '.'
. ~/.bashrc
# Verify the compiler is working
aarch64-linux-gnu-gcc -v
```
#### **Step 2: Download the Kernel Source Code**
Download the kernel source provided by FriendlyELEC for your board.
```bash
# Create a workspace
mkdir -p ~/development
cd ~/development
# Clone the correct kernel version (6.1.y branch for RK3588)
git clone https://github.com/friendlyarm/kernel-rockchip --depth 1 -b nanopi6-v6.1.y kernel-rockchip
cd kernel-rockchip
```
#### **Step 3: Configure the Kernel for 48-bit VA**
This is the most important step. You will change the kernel's default configuration.
```bash
# Load the default configuration for the board
make ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- nanopi6_linux_defconfig
# Open the interactive configuration menu
make ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- menuconfig
```
Inside the text-based menu, use the arrow keys to navigate:
1. Select `Kernel Features` --->
2. Select `Virtual address space size` --->
3. Select **`48-bit`** and press `<Enter>`.
4. Navigate to `< Save >`, press `<Enter>`, and confirm the default filename `.config`.
5. Navigate to `< Exit >` and press `<Enter>` repeatedly to close the menu.
#### **Step 4: Compile the Kernel and Modules**
Now, build the kernel. This will take a significant amount of time. Grab a coffee.
```bash
# Compile the kernel image and device tree files
# -j$(nproc) uses all available CPU cores to speed up the process
make ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- nanopi6-images -j$(nproc)
# Compile the kernel modules
make ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- modules -j$(nproc)
```
#### **Step 5: Install the New Kernel and Modules (Corrected Method)**
With the compilation finished, you will now install the new components onto your live system.
```bash
# Backup your existing kernel modules folder (as a precaution)
sudo mv /lib/modules/$(uname -r) /lib/modules/$(uname -r).bak
# Install the new modules to the system's module directory
sudo make ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- INSTALL_MOD_PATH=/ modules_install
# Now, write the new kernel and resource images to the eMMC
# Note on File Paths: The 'nanopi6-images' target conveniently places the final
# kernel.img and resource.img files in the top-level directory of the source
# code (i.e., your current directory: ~/development/kernel-rockchip).
# Write resource.img to partition 4 of the eMMC
sudo dd if=resource.img of=/dev/mmcblk2p4 bs=1M
# Write kernel.img to partition 5 of the eMMC
sudo dd if=kernel.img of=/dev/mmcblk2p5 bs=1M
```
#### **Step 6: Reboot and Verify**
The new kernel is installed. Reboot to activate it.
```bash
sudo reboot
```
After the system restarts, log in and verify the change. (CHECK STEPS MAY NOT WORK - THE ULTIMATE CHECK STEP IS YOU RUN MONGO v8.0 - see Step 7). if you want proper "checks" then ask Gemini/ChatGPT, these check did not work on my board, i skipped to running MongoDB.
1. Check the new kernel version (it will likely have changed from the original).
```bash
uname -r
```
2. **Confirm the 48-bit VA is active.** This is the most reliable method:
```bash
zcat /proc/config.gz | grep CONFIG_PGTABLE_LEVELS
```
**The correct output must be:** `CONFIG_PGTABLE_LEVELS=4`
#### **Step 7: Start and Test MongoDB**
The underlying problem is now fixed. You can start your MongoDB instance.
```bash
# Start the mongod service
sudo systemctl start mongod
# Check its status
sudo systemctl status mongod
```
You should now see `Active: active (running)`. The `MmapAligned` error will be gone, and your MongoDB server will be fully functional.