#golang PSA: If you are shipping binaries built with Go 1.21.1 or newer to Linux systems with Transparent Huge Pages (THP) enabled (default in many cases), you either need to tweak the system THP settings via SysFS or upgrade to Go 1.21.6 AND set the workaround GODEBUG=disablethp environment variable.
If you don't, it can lead to what looks like a slow memory leak and eventually an out-of-memory condition. The issue doesn't affect every application (it depends on your memory use patterns), but when it does trigger, it's a pain to debug.
Go docs on THP: https://go.dev/doc/gc-guide#Linux_transparent_huge_pages
Github issue: https://github.com/golang/go/issues/64561
Original Linux kernel issue: https://bugzilla.kernel.org/show_bug.cgi?id=93111
Huge thanks to @TomSellers for tracking this down. The latest @runZeroInc build (4.0.240109.0) includes the fix for self-hosted customers.