FAudio - Accuracy-focused XAudio reimplementation for open platforms

This isn't quite ready to merge, but it's close. With this branch, I get good audio from Skyrim with gst-libav's WMA decoder. Further testing with games you were trying before is appreciated.

First commit is the WMA test I used to verify behavior with xaudio2. I don't think it's important to merge it, but I put it here if you want to run some tests. The important points I discovered are:

-WMA does allow skipping some of the start of the buffer (i.e. PlayBegin is respected).

-WMA does not allow looping arbitrarily. It always loops back to the start and plays to the end, no matter what you set the Loop variables to.

-Having incorrect pDecodedPacketCumulativeBytes values does not break decoding. It does limit the total decoded length to the last value, which is how these types of codecs often work (encode whole blocks and truncate the final block to get the right length).

The 2nd commit is your unchanged gstreamer commit. We can merge things down when it's ready to go.

The 3rd commit fixes the build system so it builds correctly in Proton. I was having issues with it finding the gstreamer installation. I'm not very good with CMake, so I probably did this wrong. In particular, the PkgConfig:: I think requires a later version of CMake than the CMakeList currently requires. Help here is appreciated.

The 4th commit implements the loop behavior I mentioned above.

The final commit rewrites much of the gstreamer algorithm. I won't recap the whole algorithm, but here are some important points:

-WMA is a stateful decoder, so we can't just hop around with random access. We must decode sequentially.

-But, because FAudio requires some amount of random access (mostly that 2 sample "fuck it" quirk), I store both the current and previous decoded chunks. I think this is sufficient.

-We now reset the decoder when looping. I'm not sure whether this is correct, but it makes sense to me.

-We're now using decodebin to let gstreamer select the codec for us.

-We limit the full amount of decoded samples to the last pDecodedPacketCumulativeBytes value, which I think matches native.

-Instead of using the pDecodedPacketCumulativeBytes values we get from the application, we depend on the decoder to give us the actual block sizes, and seek based on that.

There is one ugly hack that I wonder if you have a solution for. I store the actual decoded-samples values mentioned above in the blockSizes array, and use that for finding the current buffer requested by FAudio. But, we need to clear and re-size this array when we get a new buffer in. Currently I'm just using the value of the pointer to see if we get a new buffer incoming, but that feels super ugly. Is there some better way to tell this gstreamer code that we have a new buffer and should forget what we know about the old buffer?

tested with x86_64-mingw-w64 on Arch linux

but the Tools subdir won't build for me on mingw. And I don't know how to build the wine cpp wrappers under linux

All the matrix computation cases are handled, including multi-channel emitters, REDIRECT_TO_LFE / ZERO_CENTER, cones and custom curves. Done some manual testing and everything was giving good values (with differences due to the fact that I lerp everywhere) and out of the right speakers, so it's a good candidate for a field test. Non-implemented features (LPF / Reverb) have a minimal implementation that sets values, so an uninitialized input structure wouldn't come out with garbage. As such it can be tested with the COM wrapper as well and nothing should blow up.

I don't know how to debug this, but I thought I'd make an issue report. Not used to github, so please excuse me for the lack of information.

Using latest steam/steamplay on Arch (Fully updated). Used commit 879.

Built FAudio with FFMPEG support and installed using install_fake_dlls to proper directories in the fallout directory (pfx).

Sound works, music works, voices works. Everything works. There's a bit of audio clipping while entering terminal, but that's another issue. I'm not a proffessional, but I can't hear the difference using FAudio and XAudio, so good work!

Anyway, when using fast travel or entering a sanctuary (in my case, the Castle) the game will freeze. It's not reproducible with XAudio. This happens every time, no matter how I try to enter. It seems there's a specific line I cross from one region over to another (In this case, the castle), and the text should show up "Now entering The Castle". On the same spot, the same pixel evel, the game will crash.

Thunder, etc. works with FAudio though, did not with XAudio (sound would dissapear).

Shooting a missile at a group of raiders/mutants/etc will sometimes freeze the game using FAudio (when there's too many sounds on screen it seems. Same as XAudio, except that XAudio would just dissapear. Using FAudio and the game freezes). This happens frequently.

Fast traveling. Whenever I use fast travel, the loading screen comes up (I have just tested it on a few locations...sanctuaries. Might just be sanctuary related) and it completes the loading. Just when it's about to change from loading screen to game screen, the game freezes. This happens every time.

No output in terminal. Just game executable removed.

https://github.com/FNA-XNA/FAudio/blob/f7a0ce7e84933ea4913ca130d74cd39e5c080259/src/F3DAudio.c#L1127

The ComputeEmitterChannelCoefficients function does not take into acount LFEDistanceCurve. Maybe this line should look like: pMatrixCoefficients[ curConfig->LFSpeakerIdx * numSrcChannels + currentChannel ] += LFEattenuation;

The comment in CalculateMatrix function seems to be true /* TODO: this could be skipped if the destination has no LFE */ :+1: It really skipped for none multi-channel emitters..

I finally figured out what's causing all that racket with WMA/XMA streams. This one's for @aeikum (wrote the original FFmpeg work), @0x0ade (XMA), @GloriousEggroll (Warframe), and @JohanSmet (wrote the most recent FAudio FFmpeg work).

A quick overview of how the decoding system works: When a buffer is submitted we receive not just the data itself, but the amount of data we can get (measured in samples). So when we decode, we ask for exactly the number of samples we need based on the quantum size until the end, where the number will be a little bit smaller. In either case, the decoder always provides the exact number of samples we request because we always know how many samples are available without the decoder's assistance.

FFmpeg isn't playing by the rules. In particular, we're hitting this line which should never ever happen:

https://github.com/FNA-XNA/FAudio/blob/master/src/FAudio_ffmpeg.c#L382

Here's the quick version:

• Mixer asks for, say, 512 samples
• We read in from FFmpeg, filling the convert cache as needed
• FFmpeg gives us less than the total sample block size, but also claims we're out of samples
• available is 0, breaks
• todo is less than samples, resulting in uninitialized memory at the end
• R̫̘̜̕E̼͈̭̬̥A҉͔͙̪̞̘͈L̰ ̕G̜̳̱̯̱͈̰O̼̯͓̻͓͚O̢̼̥̮͍̝D̶ ͔̰̜͚͔S̭̗͚͈͞O͇̺̻̹̲͍U̝̦͕̺̳͍̺Ṋ̼̹̼̤D̫͈͉̹S̗̳̭͔̝̭̕

Eggroll has it in Warframe, I have it in Skyrim SE. We need to figure out how to make FFmpeg cooperate and actually give us all the samples. We can't leave without the full size, because then you just get skipping, which is more pleasant-sounding but still very wrong.

Anyone have any ideas? I'm not terribly familiar with FFmpeg so I don't know if there's a trick to making decoding consistent. Sadly it's not 100% consistent, but it's very easy to reproduce with those two games.

I think enough cases have come up to consider replacing the Makefile... Typically I just write a handwritten Makefile because there's not much you're meant to do with the library, but FAudio has run into WAY more use cases than anything I've ever built before, so CMake is starting to make a little more sense now. There are many reasons for this, including easy Debug/Release configuration, a clear options list (FFmpeg, XNA_Song, etc), improving the utils/ builds, and if we can, integrating the cpp/ folder's builds.

Note that this is replacing ONLY the Makefiles. The VS/Xcode projects will stay as-is. (Maik, I'll still take the Android.mk if that's still useful)

I wrote up a basic file that covers the core sources, with a bunch of TODOs to fill in:

https://github.com/FNA-XNA/FAudio/tree/cmake

A bunch of people are involved, so here's a list so everyone has an idea of what's going on:

• Myself and @aeikum need this to work not just with Proton's build system, but with the needs of the various Linux distributions. If we're going to integrate FAudio into Wine, we need to make it easy for distros to package FAudio first. @GloriousEggroll has been working on the Proton build with FAudio's Makefile and has been hacking my build files to death, so we have a decent idea of what we need to fix to make that happen.
• @JohanSmet wrote the build system for the COM wrapper and Wine DLL; thankfully the two aren't wildly different but they do have some specific needs that have to be introduced just for those files somehow. @GloriousEggroll was playing with this a lot for Warframe, before we started looking at the Proton build.
• @cybik and @NeroBurner actually wrote their own CMake projects already, we just need to write a new CMakeLists.txt that covers the whole repo, not just the core FAudio sources.

As far as mandatory requirements go, we're stuck supporting Debian's criminally old CMake version, 3.7. Yes, it really is that old. Aside from this limitation, there aren't really any hard rules I can think of.

I'm currently stuck on the code side of things, but I know a lot of people were getting frustrated with my crappy old build system so I thought I'd open this up for anyone who's ready to throw it in the dumpster.

Checklist:

• [x] libFAudio
• [x] utils
• [x] tests
• [x] cpp (MinGW only)
• [x] FFmpeg

OS: Slackware64-current FAudio: https://github.com/FNA-XNA/FAudio/commit/705f2cd5b399f1ed5e8cba310c999381705f3640 wine: 4.15 alsa-lib: 1.1.9 SDL: 2.0.10 game: Bloodstained-1.05 (GOG version)

After I upgraded FAudio to 19.09 from 19.08 I found that the game Bloodstained no longer starts. I next confirmed that its still broken in the current master (https://github.com/FNA-XNA/FAudio/commit/705f2cd5b399f1ed5e8cba310c999381705f3640) and then bisected the issue.

464050b54b977bfbc5b4004257c4d547de50e4bd is the first bad commit
commit 464050b54b977bfbc5b4004257c4d547de50e4bd
Author: Ethan Lee <[email protected]>
Date:   Fri Aug 2 01:24:28 2019 -0400

    Refactor PlatformInit/Quit to not use FAudio internals, add 1024_QUANTUM

 csharp/FAudio.cs           | 20 ++++++++++----------
 include/FAudio.h           | 18 +++++++++---------
 src/FAudio.c               | 24 ++++++++++++++++++++++--
 src/FAudio_internal.h      | 14 ++++++++++++--
 src/FAudio_platform_sdl2.c | 42 +++++++++++++++++++++++++++---------------
 5 files changed, 80 insertions(+), 38 deletions(-)

464050b54b977bfbc5b4004257c4d547de50e4bd

The game fails with this output, please let me know if there are any more info I can provide or patches I can try.

001b:err:plugplay:load_function_driver AddDevice failed for driver L"winebus", status 0xc0000002.
0038:fixme:ver:GetCurrentPackageId (0x5d5fda0 (nil)): stub
003c:err:winediag:SECUR32_initNTLMSP ntlm_auth was not found or is outdated. Make sure that ntlm_auth >= 3.0.25 is in your path. Usually, you can find it in the winbind package of your distribution.
003c:fixme:sync:SetWaitableTimerEx (00000000000001B4, 000000000671F160, 0, 0000000000000000, 0000000000000000, 0000000000000000, 32) semi-stub
002d:fixme:gameux:GameExplorerImpl_VerifyAccess (00000000000E3710, L"C:\\GOG Games\\Bloodstained Ritual of the Night\\BloodstainedRotN\\Binaries\\Win64\\BloodstainedRotN-Win64-Shipping.exe", 00000000005FEB58)
002d:fixme:win:DisableProcessWindowsGhosting : stub
002d:fixme:msctf:InputProcessorProfileMgr_GetActiveProfile (00000000000F9A70)->({34745c63-b2f0-4784-8b67-5e12c8701a31} 00000000005FF200)
002d:fixme:system:SystemParametersInfoW Unimplemented action: 59 (SPI_SETSTICKYKEYS)
002d:fixme:system:SystemParametersInfoW Unimplemented action: 53 (SPI_SETTOGGLEKEYS)
002d:fixme:system:SystemParametersInfoW Unimplemented action: 51 (SPI_SETFILTERKEYS)
002d:fixme:dxgi:DXGID3D10CreateDevice Ignoring flags 0x1.
002d:fixme:d3d11:d3d11_device_CheckFeatureSupport Returning fake threading support data.
002d:fixme:win:RegisterTouchWindow (0x4006c 00000000): stub
002d:fixme:vcruntime:__telemetry_main_invoke_trigger (0000000011C40000)
002d:fixme:vcruntime:__telemetry_main_invoke_trigger (000000001CB80000)
002d:fixme:vcruntime:__telemetry_main_invoke_trigger (0000000012830000)
INFO: OpenAudioDevice failed: ALSA: Couldn't open audio device: Device or resource busy
0065:fixme:win:DisableProcessWindowsGhosting : stub
0065:fixme:msctf:InputProcessorProfileMgr_GetActiveProfile (00000000000711D0)->({34745c63-b2f0-4784-8b67-5e12c8701a31} 00000000005FF2B0)
0065:fixme:system:SystemParametersInfoW Unimplemented action: 59 (SPI_SETSTICKYKEYS)
0065:fixme:system:SystemParametersInfoW Unimplemented action: 53 (SPI_SETTOGGLEKEYS)
0065:fixme:system:SystemParametersInfoW Unimplemented action: 51 (SPI_SETFILTERKEYS)
0065:fixme:dxgi:DXGID3D10CreateDevice Ignoring flags 0x1.
0065:err:d3dcompiler:compile_shader HLSL shader parsing failed.
0065:err:d3dcompiler:compile_shader HLSL shader parsing failed.
0065:fixme:dwmapi:DwmIsCompositionEnabled 00000000005FF078
0065:fixme:win:RegisterTouchWindow (0x2009e 00000000): stub
0065:fixme:dwmapi:DwmSetWindowAttribute (000000000002009E, 2, 00000000005FF0DC, 4) stub
0009:fixme:kernelbase:AppPolicyGetProcessTerminationMethod FFFFFFFFFFFFFFFA, 000000000060FCF0

This causes a SIGSEGV in Bleed 2 (armhf, mono git e1002857) trying to deref the aliased pointers to the ListenerVelocityComponent and EmitterVelocityComponent members of pDSPSettings in CalculateDoppler.

at <unknown> <0xffffffff>
at FAudio:FACT3DCalculate <0x00043>
at Microsoft.Xna.Framework.Audio.Cue:Apply3D <0x000db>
at LoopingCue:set_LeftToRightPan <0x0021f>
at LoopingCue:.ctor <0x0015f>
at Bleed_II.IJCAudioEngine:CreateLoopingCue <0x001cb>
<...>

I am unsure what mechanism causes this to happen, but a few things have successfuly worked around this crash:

• Setting Pack = 4 on related classes. (ABI breakage :/)
• Casting listenerVelocityComponent, emitterVelocityComponent and DopplerFactor to float __attribute__((aligned(1))). (Absolutely evil...)
• Adding [StructLayout(LayoutKind.Sequential, Pack = 4)] to SoundBank.

p.s.: The game still presents a ringing sound artifact during the first level - the same artifact is also reproduced on an unaltered build on AArch64 where the crash doesn't happen. Couldn't reproduce the artifact on amd64 tho, but also didn't investigate it yet.

@JohanSmet Duplicating report from Wine Bugzilla

diff --git a/src/FAudioFX_reverb.c b/src/FAudioFX_reverb.c
index bd58a06..b125b5f 100644
--- a/src/FAudioFX_reverb.c
+++ b/src/FAudioFX_reverb.c
@@ -101,7 +101,7 @@ static inline void DspDelay_Change(DspDelay *filter, float delay_ms)
 
 	/* Length */
 	filter->delay = MsToSamples(delay_ms, filter->sampleRate);
-	filter->read_idx = (filter->write_idx - filter->delay + filter->capacity) % filter->capacity;
+	filter->read_idx = (filter->write_idx + filter->delay + filter->capacity) % filter->capacity;
 }
 
 static inline float DspDelay_Read(DspDelay *filter)

Patch tested with Sniper Elite 3, Resident Evil 0 HD Remaster, Homesick. Behaviour with non-affected games needs to be checked but it shouldn't be worse.

Hi,

first of all thank you very much for bringing in a replacement for xact. I wanted to give it a try with Bulletstomr Full Clip Edition, as this game is having bad sound issue with the current wine builtin dll's and also using the native ones with wine does not work.

Using the mingw compiled Faudio dlls, i can get into the menu of the game and the sound is fine, i can lower/raise the different sound levels (music, environement and voices) in the sound menu of the game.

Trying to get into a mission i get the following error message:

Assertion failure at F3DAudioCheckCalculateParams (src/F3DAudio.c:342), triggered 1 time: 'pDSPSettings->DstChannelCount == ChannelCount && "Invalid channel count, DSP settings and speaker configuration must agree"'

After being able to klick on ignore, the following error message shows up:

Assertion failure at F3DAudioCalculate (src/F3DAudio.c:1502), triggered 1 time: '0 && "REVERB not implemented!"'

I can also press ignore here, but the game finally crashes after ignoring both issues 2 times

Many thanks ! Christian

Required:

• [x] Windows (MSVC)
• [x] Windows (MinGW)
• [x] Wine (MinGW with Win32 backend)
• [x] Linux (can just be ubuntu-latest)
• [x] macOS CMake (TODO: macOS Xcode)
• [ ] iOS/tvOS

Optional:

• [x] FreeBSD (take from SDL)

Assertion failed: voice->outputChannels > 0 && voice->outputChannels < 9, file FAudio\src\FAudio.c, line 1227 Running on Windows (v10.0.19044.0) X64 NetCore 6.0.5

FAudio built with the win32 (windows media feature pack) backend. User reports that disabling https://www.voicemod.net/ fixed the crashes. User experiences issue after random amount of playtime, not at startup.

I'm assuming that outputChannels == 0, will attempt to confirm.

FAudio should run silently if there are no output channels rather than crash, looking for some pointers where to start on this.

On a related note, FAudio currently doesn't recover from AUDCLNT_E_DEVICE_INVALIDATED which can be returned from most IAudioClient methods. Unfortunately the audio device is baked into the audio thread, so I'm not sure on a good place to implement the recovery process mentioned in https://docs.microsoft.com/en-us/windows/win32/coreaudio/recovering-from-an-invalid-device-error

New version dropped yesterday:

https://github.com/nothings/stb

This update's kind of messy, we have a couple changes properly marked as FAudio-unique changes, but there were also a handful of bugfixes that still haven't made it upstream, so it may require a bit of care to update to the latest version while also retaining the bugfixes yet to be mainlined. It's still a pretty small update though, so it'll just take slightly longer than a typical manual merge.

From MSDN: https://docs.microsoft.com/en-us/windows/desktop/api/xaudio2/ns-xaudio2-xaudio2_debug_configuration

In addition to filling in one more XAudio2 feature, this is a legitimately useful for general FAudio debugging without having to step through a whole lot. It can be extremely verbose, however, so this should only be enabled for non-release builds (i.e. !_DEBUG && !DEBUG && (GCC && !OPTIMIZED)). Additionally, I would like to add the ability to enable features with environment variables, not just SetDebugConfiguration. This is useful for projects like Wine that won't have access to these entry points without added hackery. Each flag/bool should get a variable in some way.

As for the implementation, this can be mapped to something like SDL_Log, though I don't know how complicated we'll need to get. I'm hoping we can make all these log types macros that are blanked in release builds, but that's totally speculation without having actually tried to write this at all.

It's a big ol checklist:

• [ ] TraceMask XAUDIO2_LOG_*
  • [x] ERRORS
  • [x] WARNINGS
  • [x] INFO
  • [x] DETAIL
  • [x] API_CALLS
  • [x] FUNC_CALLS
  • [ ] TIMING
  • [x] LOCKS
  • [ ] MEMORY
  • [ ] STREAMING
• [ ] BreakMask XAUDIO2_LOG_*
  • [ ] ERRORS
  • [ ] WARNINGS
• [x] LogThreadID
• [x] LogFileline
• [x] LogFunctionName
• [x] LogTiming

For the first release of FAudio we focused on what FNA and XACT needed, but now that this is looking to go into a whole lot more games we should look at the rest of the X3DAudio features.

The stubs are here:

https://github.com/FNA-XNA/FAudio/blob/master/src/F3DAudio.c#L1538

Don't expect a whole lot of information out there - the MSDN docs are completely useless and many months of looking for existing research on this turned up completely and entirely empty. If you're working on this, it's because you REALLY know what you're doing (I certainly don't).

Checklist:

• [ ] F3DAUDIO_CALCULATE_DELAY
• [x] F3DAUDIO_CALCULATE_LPF_DIRECT
• [x] F3DAUDIO_CALCULATE_LPF_REVERB
• [x] F3DAUDIO_CALCULATE_REVERB