Show Your Working: Turning off PulseAudio "flat volumes"
23 July 2017
Show Your Working is a brand new segment where I write up things I have attempted to fix in open source software. Sometimes it's interesting to debug a problem yourself from first principles, even if the codebase is huge and you don't know anything going in. I will try and explain my thought process as I venture out into the weeds armed only with a butter knife.
The software
PulseAudio is the software audio mixer for most desktop Linux distributions. Let's not mince words; PulseAudio has an awful reputation, mostly earned from the botched rollout by Ubuntu and other distributions back in 2008. Long-time Linux users remember these as the dark years where games were unplayable, and everything was punctuated by a loud crackling from the constant buffer underruns. The most egregious problems were caused by the aforementioned shonky buffering, plus a reliance on timing-related ALSA features which had never been used/tested properly for many sound drivers. It didn't help that barely any applications had direct support for the PulseAudio API; there were compatibility shims for ALSA/OSS applications, but they were a crapshoot at best.
But that was a decade ago! We've moved from version 0.9 to version 11.1, a metric ass-ton of effort has gone into making the mixer first rate, most Linux sound libraries are fully Pulse compatible, etc. etc. Surely everything is perfect now, right?
The problem
Ahahahaha not quite. Five years ago PulseAudio added a feature called "flat volumes"; wherein by default, every per-application volume control is welded in some byzantine way to the master volume control. Completely ignoring that, you know, everyone assumed that per-application volume controls actually meant PER-APPLICATION, and quite a lot of apps had a knack for setting the volume to 100% on startup (or just at random intervals)! Cue newly-deafened users coming out in droves to complain, and package maintainers grumping about one more non-upstream setting they'll have to override.
Anyway. Disabling flat volumes reverts things back to per-application volumes scaled (post-fader) by an independent master volume, and all is right in the land once again. As of... recently? Disabling flat-volumes in /etc/pulse/daemon.conf stopped working on my system.
The preparation
The only software we need installed for tracking this down is the Git VCS and the standard Linux command line tools, plus whatever build dependencies PulseAudio has. Start by grabbing the PulseAudio source code of the release we know is bad.
$ git clone https://anongit.freedesktop.org/git/pulseaudio/pulseaudio
$ cd pulseaudio
# commit taken from https://git.archlinux.org/svntogit/packages.git/tree/trunk/PKGBUILD?h=packages/pulseaudio&id=f9dd2850e5eff2d2a7fde2af6e932d785a521912
$ git checkout 84952e6a092b6a0c5b153bd7a4f6e490810681c8
The diagnosis
Let's track down what went wrong. We know one thing for sure: the name of the config switch which is meant to fix this ("flat-volumes"). Let's find where it's used.
$ grep -rn flat-volumes *
man/pulse-daemon.conf.5.xml.in:220: <p><opt>flat-volumes=</opt> Enable 'flat' volumes, i.e. where
src/daemon/daemon-conf.c:531: { "flat-volumes", pa_config_parse_bool, &c->flat_volumes, NULL },
src/daemon/daemon-conf.c:740: pa_strbuf_printf(s, "flat-volumes = %s\n", pa_yes_no(c->flat_volumes));
src/daemon/daemon.conf.in:61:; flat-volumes = yes
If I had to pick the single most important tool for troubleshooting something you've never seen before, it would be grep. It does two things extremely well: search a bunch of files or piped input for a text/regex match, and adjust the amount of context you get back for the findings. grepping is the de-facto method to scrub through a ton of source code quickly to establish causality; the chain of events linking the thing we change (i.e. the config file) to the expected action (i.e. the volume adjusting as it damn well should).
Now we follow the trail of breadcrumbs and zero in on the code path which does the useful work. We can see above that the "flat-volumes" config parser sets a variable called flat_volumes, so we search for that.
$ grep -rn flat_volumes *
src/daemon/daemon-conf.c:71: .flat_volumes = true,
src/daemon/daemon-conf.c:531: { "flat-volumes", pa_config_parse_bool, &c->flat_volumes, NULL },
src/daemon/daemon-conf.c:740: pa_strbuf_printf(s, "flat-volumes = %s\n", pa_yes_no(c->flat_volumes));
src/daemon/daemon-conf.h:76: flat_volumes,
src/daemon/main.c:1044: c->flat_volumes = conf->flat_volumes;
src/pulsecore/core.c:138: c->flat_volumes = true;
src/pulsecore/core.h:197: bool flat_volumes:1;
src/pulsecore/sink.c:541: enable = enable && s->core->flat_volumes;
src/pulsecore/source.c:492: enable = enable && s->core->flat_volumes;
Okay, the only plased where flat_volumes is used to drive a decision is in src/pulseaudio/sink.c and src/pulseaudio/source.c. We'll focus in on sink.c as "sink" is PulseAudioese for "sound output".
$ grep -rn -B8 -A16 flat_volumes src/pulsecore/sink.c
533-}
534-
535-static void enable_flat_volume(pa_sink *s, bool enable) {
536- pa_sink_flags_t flags;
537-
538- pa_assert(s);
539-
540- /* Always follow the overall user preference here */
541: enable = enable && s->core->flat_volumes;
542-
543- /* Save the current flags so we can tell if they've changed */
544- flags = s->flags;
545-
546- if (enable)
547- s->flags |= PA_SINK_FLAT_VOLUME;
548- else
549- s->flags &= ~PA_SINK_FLAT_VOLUME;
550-
551- /* If the flags have changed after init, let any clients know via a change event */
552- if (s->state != PA_SINK_INIT && flags != s->flags)
553- pa_subscription_post(s->core, PA_SUBSCRIPTION_EVENT_SINK|PA_SUBSCRIPTION_EVENT_CHANGE, s->index);
554-}
555-
556-void pa_sink_enable_decibel_volume(pa_sink *s, bool enable) {
557- pa_sink_flags_t flags;
We can see that in terms of honouring the setting, nothing has changed. Enabling flat volume is always overridden by what the user has set, and the actual state is stored in a variable flags with the bitflag name PA_SINK_FLAT_VOLUME. Guess what we're searching for next???
$ grep -rn PA_SINK_FLAT_VOLUME *
src/modules/dbus/iface-device.c:461: has_flat_volume = (d->type == PA_DEVICE_TYPE_SINK) ? !!(d->sink->flags & PA_SINK_FLAT_VOLUME) : FALSE;
src/modules/dbus/iface-device.c:841: has_flat_volume = !!(d->sink->flags & PA_SINK_FLAT_VOLUME);
src/modules/module-virtual-sink.c:564: u->sink->flags |= PA_SINK_FLAT_VOLUME;
src/modules/module-virtual-surround-sink.c:702: u->sink->flags |= PA_SINK_FLAT_VOLUME;
src/pulse/def.h:801: PA_SINK_FLAT_VOLUME = 0x0040U,
src/pulse/def.h:838:#define PA_SINK_FLAT_VOLUME PA_SINK_FLAT_VOLUME
src/pulsecore/cli-text.c:286: sink->flags & PA_SINK_FLAT_VOLUME ? "FLAT_VOLUME " : "",
src/pulsecore/sink.c:547: s->flags |= PA_SINK_FLAT_VOLUME;
src/pulsecore/sink.c:549: s->flags &= ~PA_SINK_FLAT_VOLUME;
src/pulsecore/sink.c:1554: * When a sink uses volume sharing, it never has the PA_SINK_FLAT_VOLUME flag
src/pulsecore/sink.c:1563: return (s->flags & PA_SINK_FLAT_VOLUME);
src/pulsecore/sink.h:441:/* Use this instead of checking s->flags & PA_SINK_FLAT_VOLUME directly. */
Now THIS is a great example of why you want to grep as much source code as possible. We could have continued our investigation just in sink.c, but searching everywhere revealed something super useful! Namely src/pulsecore/cli-text.c; there is a command line thing somewhere which prints if PA_SINK_FLAT_VOLUME is set or not. This just potentially saved us having to make a debug build to find this out.
A little bit of poking around revealed that the string is printed when you invoke pacmd list-sinks (not pactl list sinks! that of course produces output that's near identical, but missing the one thing we want.), so we did that.
$ pacmd list-sinks | grep -B8 -A8 FLAT
6 sink(s) available.
index: 0
name: <alsa_output.pci-0000_2a_00.3.iec958-stereo>
driver: <module-alsa-card.c>
flags: HARDWARE HW_MUTE_CTRL DECIBEL_VOLUME LATENCY FLAT_VOLUME DYNAMIC_LATENCY
state: IDLE
suspend cause:
priority: 9058
volume: front-left: 0 / 0% / -inf dB, front-right: 0 / 0% / -inf dB
balance 0.00
base volume: 65536 / 100% / 0.00 dB
volume steps: 65537
muted: no
--
ports:
iec958-stereo-output: Digital Output (S/PDIF) (priority 0, latency offset 0 usec, available: unknown)
properties:
active port: <iec958-stereo-output>
* index: 1
name: <jack_out>
driver: <module-jack-sink.c>
flags: DECIBEL_VOLUME LATENCY FLAT_VOLUME
state: IDLE
suspend cause:
priority: 0
volume: front-left: 22925 / 35% / -27.37 dB, front-right: 22925 / 35% / -27.37 dB
balance 0.00
base volume: 65536 / 100% / 0.00 dB
volume steps: 65537
muted: no
Okay well there's the problem; the sink I'm using (JACK to my Scarlett box) has FLAT_VOLUME enabled. But why. Whyyyyyyyyy. I mean I told it not to!
No getting around it, we need to have a look at the program while it's running. Let's make a test build.
# copying the build steps from the PKGCONFIG
# by default CFLAGS should include -g, which adds debug symbols
$ ./bootstrap.sh
$ ./configure --prefix=/usr\
--sysconfdir=/etc \
--libexecdir=/usr/lib \
--localstatedir=/var \
--with-udev-rules-dir=/usr/lib/udev/rules.d \
--with-database=tdb \
--disable-tcpwrap \
--disable-bluez4 \
--disable-samplerate \
--disable-rpath \
--disable-default-build-tests \
DATADIRNAME=share
$ make
There's now a freshly built pulseaudio executable in the src directory. Except it's not, it's some libtool shell spew that passes enough lies to the built copy (in .libs) to make it run. Which is a problem, because you can't run gdb against shell spew.
Because they're not all heartless bastards, the libtool developers made a wrapper for the wrapper which allows pass-through execution. Before we start the session, make sure PulseAudio isn't set to autospawn, kill the current daemom, and then fire up the debugger.
$ pulseaudio -k
$ libtool --mode=execute gdb --args pulseaudio
GNU gdb (GDB) 8.0
Copyright (C) 2017 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
...
(gdb)
Smashing. All of the functions we want to test are in shared libraries, which we can only see after execution starts, so lets breakpoint immediately after starting.
(gdb) b main
Breakpoint 1 at 0x52e0: file daemon/main.c, line 369.
(gdb) r
Starting program: /home/scott/Development/pulseaudio/src/.libs/lt-pulseaudio
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/usr/lib/libthread_db.so.1".
Breakpoint 1, main (argc=1, argv=0x7fffffffdf68) at daemon/main.c:369
369 int main(int argc, char *argv[]) {
We'll start by adding a breakpoint in enable_flat_volume() and seeing what's going down there.
(gdb) b pulsecore/sink.c:536
Breakpoint 2 at 0x7ffff7b75690: pulsecore/sink.c:536. (2 locations)
(gdb) c
Continuing.
W: [lt-pulseaudio] main.c: /proc/self/exe does not point to /usr/bin/pulseaudio, cannot self execute. Are you playing games?
N: [lt-pulseaudio] daemon-conf.c: Detected that we are run from the build tree, fixing search path.
E: [lt-pulseaudio] module-alsa-card.c: Failed to find a working profile.
E: [lt-pulseaudio] module.c: Failed to load module "module-alsa-card" (argument: "device_id="3" name="usb-Clavia_DMI_AB_Nord_Electro_5-01" card_name="alsa_card.usb-Clavia_DMI_AB_Nord_Electro_5-01" namereg_fail=false tsched=yes fixed_latency_range=no ignore_dB=no deferred_volume=yes use_ucm=yes card_properties="module-udev-detect.discovered=1""): initialization failed.
Breakpoint 2, enable_flat_volume (s=0x555555877570, enable=true)
at pulsecore/sink.c:538
538 pa_assert(s);
Let's do a backtrace to see where the method got called from.
(gdb) bt
#0 enable_flat_volume (s=0x5555558734f0, enable=true) at pulsecore/sink.c:541
#1 0x00007ffff7b77110 in pa_sink_enable_decibel_volume (s=0x5555558734f0,
enable=<optimized out>) at pulsecore/sink.c:566
...
The default behaviour seems to be to enable flat volumes if the sink identifies itself as having decibel-based volume control? I... you know what I won't even pretend to understand why. So what happened to the config file? Isn't it meant to override this?
(gdb) p s->core->flat_volumes
$3 = true
I knew it! You betrayed me, config parser!
(gdb) b daemon/daemon-conf.c:607
Breakpoint 6 at 0x55555555f3a9: file daemon/daemon-conf.c, line 610.
(gdb) c
Continuing.
Breakpoint 6, pa_daemon_conf_load (c=0x55555578d990, filename=0x0)
at daemon/daemon-conf.c:607
...
612 pa_open_config_file(DEFAULT_CONFIG_FILE, DEFAULT_CONFIG_FILE_USER, ENV_CONFIG_FILE, &c->config_file);
...
(gdb) b pa_open_config_file
Breakpoint 7 at 0x7ffff6d5d580: file pulsecore/core-util.c, line 1993.
(gdb) c
Continuing.
Breakpoint 7, pa_open_config_file (
[email protected]=0x5555555638e1 "/etc/pulse/daemon.conf",
[email protected]=0x5555555638eb "/daemon.conf",
[email protected]=0x5555555638d4 "PULSE_CONFIG",
[email protected]=0x55555578d9e8) at pulsecore/core-util.c:1993
1993 FILE *pa_open_config_file(const char *global, const char *local, const char *env, char **result) {
Okay mea culpa time. I stupidly thought it would load /etc/pulse/daemon.conf first, which has the line 'flat-volumes = no' in it, then any changes made in the local config file (~/.pulse/daemon.conf) would override those base settings. We can see the truth by reading the code for pa_open_config_file; what ACTUALLY happens is that PulseAudio only reads the first config file it can find (first it tries the path in environment variable PULSE_CONFIG, then ~/.pulse/daemon.conf, then /etc/pulse/daemon.conf).
So because I had a near-empty ~/.pulse/daemon.conf which didn't define flat-volumes, the flat-volumes setting was forced on. I don't even know where the file came from, it has a last-modified date of two days ago. Urghhhh.
The solution
Remove ~/.pulse/daemon.conf. Restart PulseAudio.