DevConf.US 2021

Developers use logs to diagnose performance problems in distributed applications. However, it is difficult to know a priori where logs are needed and what information is needed to help diagnose problems that may occur in the future. We present Variance-driven Automated Instrumentation Frameworks (VAIFs), which run alongside distributed applications. In response to newly-observed performance problems, VAIFs automatically search the space of possible instrumentation choices to enable the logs needed to help diagnose them. To work, VAIF combines distributed tracing (an enhanced form of logging) with insights about how response-time variance can be decomposed on the critical-path portions of requests' traces.