Quick steps of how to create a flame graph using perf

perf-flame-graph-notes.md

The prep-script.sh will setup the latest Node and install the latest perf version on your Linux box.

When you want to generate the flame graph, run the following (folder locations taken from install script):

sudo sysctl kernel.kptr_restrict=0
# May also have to do the following:
# (additional reading http://unix.stackexchange.com/questions/14227/do-i-need-root-admin-permissions-to-run-userspace-perf-tool-perf-events-ar )
sudo sysctl kernel.perf_event_paranoid=0

perf record -i -g -e cycles:u -- ~/sources/node/node --perf-basic-prof script.js

perf script | egrep -v "( __libc_start| LazyCompile | v8::internal::| Builtin:| Stub:| LoadIC:|\[unknown\]| LoadPolymorphicIC:)" | sed 's/ LazyCompile:[*~]\?/ /' | ~/sources/FlameGraph/stackcollapse-perf.pl > out.perf-folded

~/sources/FlameGraph/flamegraph.pl out.perf-folded > node-flame.svg

The data munging is to help expose the most important bits. There is still some play that I'm working with, but right now it seems to be working.

Alternatives:

perf script | egrep -v "( __libc_start|node::Start\(| LazyCompile | Builtin:| Stub:| LoadIC:|\[unknown\]| LoadPolymorphicIC:)" | ../FlameGraph/stackcollapse-perf.pl | grep "uv_run" > out.perf-folded

perf script | egrep -v "( __libc_start|node::Start\(| LazyCompile | v8::internal::| Builtin:| Stub:| LoadIC:|\[unknown\]| LoadPolymorphicIC:)" | ../FlameGraph/stackcollapse-perf.pl | grep "uv_run" > out.perf-folded

prep-script.sh

#!/bin/bash

sh /etc/lsb-release

wget -O - http://llvm.org/apt/llvm-snapshot.gpg.key|sudo apt-key add -

apt-add-repository "deb http://llvm.org/apt/${DISTRIB_CODENAME}/ llvm-toolchain-${DISTRIB_CODENAME} main"

apt-get update

apt-get -y upgrade

apt-get -y install make build-essential linux-tools-common elfutils libelf-dev flex bison libunwind8 libunwind8-dev libaudit-dev libdw-dev binutils-dev libnuma-dev libslang2-dev asciidoc llvm-3.6 clang-3.6 lldb-3.6 libllvm3.6 subversion libc6-dev-i386 libgtk2.0-dev libperl-dev python-dev git tmux nginx gdb apache2-utils

ln -s /usr/bin/clang-3.6 /usr/bin/clang

ln -s /usr/bin/clang++-3.6 /usr/bin/clang++

ln -s /usr/bin/lldb-3.6 /usr/bin/lldb

echo 'export CC=clang' > ~/.bash_aliases
echo 'export CXX=clang++' >> ~/.bash_aliases
echo 'export GYP_DEFINES="clang=1"' >> ~/.bash_aliases

. ~/.bash_aliases

chmod 777 /usr/share/nginx/html
ln -s /usr/share/nginx/html www

mkdir sources

cd sources

wget https://github.com/joyent/node/archive/v0.11.13.tar.gz

tar xvf v0.11.13.tar.gz

rm v0.11.13.tar.gz

mv node-0.11.13 node

cd node

./configure --gdb

make

make install

cd ~/sources

wget https://github.com/brendangregg/FlameGraph/archive/master.tar.gz

tar xvf master.tar.gz

rm master.tar.gz

mv FlameGraph-master FlameGraph

#cd ~/sources

#wget https://www.kernel.org/pub/linux/kernel/v3.x/linux-3.15.6.tar.gz

#tar xvf linux-3.15.6.tar.gz

#rm linux-3.15.6.tar.gz

#cd linux-3.15.6/tools/perf/

#make -f Makefile.perf install prefix=/usr

I used this to get a html version (which is much more usable in the browser) of the graph using stackvis:

$ npm i -g stackvis
$ perf script | egrep -v "( __libc_start| LazyCompile | v8::internal::| Builtin:| Stub:| LoadIC:|\[unknown\]| LoadPolymorphicIC:)" | sed 's/ LazyCompile:[*~]\?/ /' > intermediate.out
$ stackvis perf < intermediate.out > flamegraph.htm

What's the use of nginx? to watch the dumped traces?
If so, using python -m SimpleHTTPServer

@trevnorris
Thanks, that's very helpful. I found two glitches on ubuntu:

• the lines that add the exports to .bash_aliases broke the build. I had to remove these exports.
• I got the following XML error in some of the generated SVGs:

<g class="func_g" onmouseover="s('RegExp:(^|[\r\n])([^"'\/]*)\/(?:\*[\s\S]*?\*\/|\/.*) (11 samples, 0.12%)')" onmouseout="c()">
-------------------------------------------------------^

I hacked around it that by inserting the following at line 460 of flamegraph.pl:

$info =~ s/\[\^\"\'/[^/g;

I don't know if the problem comes from flamegraph.pl or from one of the preprocessing steps.

Would be awesome to get a docker image that does this.

This is awesome. Any way to achieve this on Mac OS X?

@dignifiedquire Very nice. Thanks for sharing.

@arunoda The reason for nginx is because I found it was the easiest way to view the flamegraphs myself and share them with others (simple as a single URL). Instead of needing to scp/ftp the file down to my box to look at it.

@bjouhier Thanks for the info about the broken XML. Sorry about the .bash_aliases break. I used this script on fresh servers that hadn't been touched yet.

@trevnorris I just tried to do run perf after installing all the dependancies via the setup script. I've experienced the following error both in Ubuntu and Debian.

After running this step

$ perf record -i -g -e cycles:u -- ~/sources/node/node --perf-basic-prof script.js

I receive this

$ Perf session creation failed.

@aseemk you can do that with the flamegraph app I created.
Includes instructions on how to get things working on Linux and Mac.

Also removes the LazyCompile for you when you load a map file.

I encountered the following errors:

E: Unable to locate package llvm-3.6
E: Couldn't find any package by regex 'llvm-3.6'
E: Unable to locate package clang-3.6
E: Couldn't find any package by regex 'clang-3.6'
E: Unable to locate package lldb-3.6
E: Couldn't find any package by regex 'lldb-3.6'
E: Unable to locate package libllvm3.6
E: Couldn't find any package by regex 'libllvm3.6'

It seems that llvm/clang/lldb/libllvm 3.6 packages were removed. To fix this issue, I simply switched to 3.5 packages. Hopefully this will help some people.

$ perf script > perf.data.script
[kernel.kallsyms] with build id 9a87d34680d3efd31b5d57f56906d8c2fc38fc5e not found, continuing without symbols
Failed to open /tmp/perf-29438.map, continuing without symbols

What did I miss?

probably the node --perf-basic-prof part

I'm interested in the function calls. There's a lot of module._something, garbage collection and _ZN2v88internal5Scope11LookupLocalEPKNS0_12AstRawStringE stuff in the graph together with function calls. Can I remove all the other stuff and only have the function calls?

^^ +1 for the question above. Thx.