#golang PSA: If you are shipping binaries built with Go 1.21.1 or newer to Linux systems with Transparent Huge Pages (THP) enabled (default in many cases), you either need to tweak the system THP settings via SysFS or upgrade to Go 1.21.6 AND set the workaround GODEBUG=disablethp environment variable.

If you don't, it can lead to what looks like a slow memory leak and eventually an out-of-memory condition. The issue doesn't affect every application (it depends on your memory use patterns), but when it does trigger, it's a pain to debug.

Go docs on THP: https://go.dev/doc/gc-guide#Linux_transparent_huge_pages

Github issue: https://github.com/golang/go/issues/64561

Original Linux kernel issue: https://bugzilla.kernel.org/show_bug.cgi?id=93111

Huge thanks to @TomSellers for tracking this down. The latest @runZeroInc build (4.0.240109.0) includes the fix for self-hosted customers.