We talk with Kai Wern Choong about his experience with tracing performance problems in production. We talk about what “tracing” means in a BEAM system, available tools like recon, great resources for learning how the tools work and general approaches for troubleshooting live production Elixir systems. Kai also shares his Livebook notebook setup for demonstrating tracing techniques in an interactive way.

Show Notes online - http://podcast.thinkingelixir.com/66

Elixir Community News

https://twitter.com/hexpm/status/1439268469296570368 – New version of Hex released that fixes issues with version downgrades and slow version resolutions. Run
```
mix local.hex
```
to upgrade.
https://hexdocs.pm/credo/1.6.0-rc.0/changelog.html#1-6-0 – Credo 1.6.0 was released with new "First Run Mode"
https://twitter.com/elixirphoenix/status/1438222537301843970 – Removing Phoenix compiler config change can speed up compilation times when on up-to-date versions
https://github.com/Qqwy/elixir-type_check – New library called TypeCheck
https://blog.oestrich.org/2021/09/introducing-aino/ – Eric Osterich is experimenting with an alternative HTTP framework for Elixir
https://github.com/oestrich/aino – The Aino project
https://techcrunch.com/2021/09/14/logistics-startup-stord-raises-90m-in-kleiner-perkins-led-round-becomes-a-unicorn-and-acquires-another-company/ – Stord, an Elixir company, raised $90M and became a "unicorn"
https://spawnfest.org/ – SpawnFest 2021 happened
https://github.com/spawnfest/eflambe – SpawnFest submission - rapid generation of flamegraphs
https://github.com/spawnfest/Sketch – SpawnFest submission - library for creating generative art with Elixir
https://github.com/spawnfest/lotus – SpawnFest submission - Lotus is a Surface UI wrapper for UIKit
https://github.com/spawnfest/Discovery – SpawnFest submission - Platform for hosting realtime, stateful servers with zero downtime deployment and horizontal scaling on Kubernetes
https://github.com/spawnfest/crash – SpawnFest submission - Crash is a simple (distributed) docker-on-docker Continuous Integration system written in Elixir

Do you have some Elixir news to share? Tell us at @ThinkingElixir or email at show@thinkingelixir.com

Discussion Resources

https://kaiwern.com/posts/2021/06/27/debugging-with-tracing-in-elixir-with-recon_trace/
https://www.naluri.life/
https://ferd.github.io/recon/recon.html
https://hex.pm/packages/recon
https://twitter.com/kw7oe/status/1409126300636254219
https://github.com/kw7oe/livebook-notebooks/blob/main/debugging-with-tracing-in-elixir-with-recon_trace.livemd – Livebook notebook showing tracing examples
https://www.youtube.com/watch?v=sR9h3DZAA74 – Debugging Live Systems on the BEAM talk by Jeffery Utter
https://www.youtube.com/watch?v=OR2Gc6_Le2U – Operable Erlang and Elixir talk by Fred Hebert (skip to 25:51, where he started talking about tracing)
https://kaiwern.com/posts/2020/11/02/debugging-with-tracing-in-elixir/ – Tracing with dbg modules
https://en.wikipedia.org/wiki/Tracing_(software)
https://erlang.org/doc/man/dbg.html – dbg module in Erlang
:dbg.fun2ms
https://kubernetes.io/blog/2017/12/using-ebpf-in-kubernetes/
https://github.com/cilium/cilium
https://opentelemetry.io/docs/erlang/
https://www.brendangregg.com/blog/2019-01-01/learn-ebpf-tracing.html
https://www.erlang-solutions.com/capabilities/wombatoam/
https://hex.pm/packages/recon_ex
https://erlang-in-anger.com/ – Erlang in Anger (Chapter 9)
https://www.thegreatcodeadventure.com/testing-genservers-with-erlang-trace/ – Testing GenServers with Erlang Trace
http://blog.plataformatec.com.br/2016/04/how-to-trace-elixir-nodes-with-erlyberly/ – How to trace Elixir nodes with Erlyberly - Plataformatec Blog
http://erlang.org/doc/man/seq_trace.html#sequential-tracing – Seq_trace
http://erlang.org/doc/man/erlang.html#trace-3 – Erlang:trace
https://github.com/andytill/erlyberly – Erlang tracing for the masses
https://github.com/tatsuya6502/recon_ex – Elixir wrapper for Recon, tools to diagnose Erlang VM safely in production

Guest Information

Find us online

Message the show - @ThinkingElixir
Email the show - show@thinkingelixir.com
Mark Ericksen - @brainlid
David Bernheisel - @bernheisel
Cade Ward - @cadebward

66: Tracing Production with Kai Wern Choong

Thinking Elixir Podcast

66: Tracing Production with Kai Wern Choong