Using the vs/lp64 branch of MKL.jl

julia> using LinearAlgebra

julia> x = rand(ComplexF32, 10); dot(x,x)
6.305651f0 + 0.0f0im

julia> using MKL

julia> x = rand(ComplexF32, 10); dot(x,x)
Error: no BLAS/LAPACK library loaded!
1.0f-45 + 4.5916f-41im

julia> BLAS.dotc(x,x)
Error: no BLAS/LAPACK library loaded!
2.5705f-41 + 0.0f0im

Fix #44. @nalimilan could you please test this out? I think this is now doing what you want:

% LANG=C make -C src CFLAGS="" LDFLAGS="" VERBOSE=1
make: Entering directory '/home/mose/repo/libblastrampoline/src'
cc -o build/libblastrampoline.o -fPIC -DF2C_AUTODETECTION -c libblastrampoline.c
cc -o build/dl_utils.o -fPIC -DF2C_AUTODETECTION -c dl_utils.c
cc -o build/config.o -fPIC -DF2C_AUTODETECTION -c config.c
cc -o build/autodetection.o -fPIC -DF2C_AUTODETECTION -c autodetection.c
cc -o build/threading.o -fPIC -DF2C_AUTODETECTION -c threading.c
cc -o build/deepbindless_surrogates.o -fPIC -DF2C_AUTODETECTION -c deepbindless_surrogates.c
cc -o build/trampolines/trampolines_x86_64.o -fPIC -DF2C_AUTODETECTION -c trampolines/trampolines_x86_64.S
cc -o build/f2c_adapters.o -fPIC -DF2C_AUTODETECTION -c f2c_adapters.c
cc -o build/libblastrampoline.so -fPIC -DF2C_AUTODETECTION build/libblastrampoline.o build/dl_utils.o build/config.o build/autodetection.o build/threading.o build/deepbindless_surrogates.o build/trampolines/trampolines_x86_64.o build/f2c_adapters.o -shared -ldl
make: Leaving directory '/home/mose/repo/libblastrampoline/src'

MacOS provides Accelerate.framework, which we can use to provide BLAS on MacOS and especially on apple silicon. Unfortunately there seems to be some ABI differences specifically related to return codes on these libraries. Since we want to provide the gfortran-compatible interface to all client code, we'll need to add some return code-altering shims and forward to those first, providing some small argument/return code massaging.

We should be able to auto-detect this similarly to how we auto-detect bitwidth; we can call an effected function (such as sdot) and look at the return code to see how the return codes are being passed back.

X-ref: https://github.com/tenomoto/dotwrp

https://www.intel.com/content/www/us/en/develop/documentation/onemkl-developer-reference-fortran/top/blas-and-sparse-blas-routines/blas-like-extensions/gemmt.html

-Werror should not be used in default build flags as it makes the compilation fail every time somebody tries to build with a new version of the compiler which triggers a new warning. It should only be set when running CI.

I saw this when trying to build the RPM for Julia 1.8.0-rc3 on Fedora rawhide (with GCC 12.1.1):

threading.c: In function 'lbt_register_thread_interface':
threading.c:44:5: error: 'strcpy' writing one too many bytes into a region of a size that depends on 'strlen' [-Werror=stringop-overflow=]
   44 |     strcpy(getter_names[idx], getter);
      |     ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
threading.c:43:34: note: destination object of size [0, 9223372036854775805] allocated by 'malloc'
   43 |     getter_names[idx] = (char *) malloc(strlen(getter));
      |                                  ^~~~~~~~~~~~~~~~~~~~~~
threading.c:46:5: error: 'strcpy' writing one too many bytes into a region of a size that depends on 'strlen' [-Werror=stringop-overflow=]
   46 |     strcpy(setter_names[idx], setter);
      |     ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
threading.c:45:34: note: destination object of size [0, 9223372036854775805] allocated by 'malloc'
   45 |     setter_names[idx] = (char *) malloc(strlen(setter));
      |                                  ^~~~~~~~~~~~~~~~~~~~~~
cc1: all warnings being treated as errors
make[1]: *** [Makefile:37: build/threading.o] Error 1
make[1]: *** Waiting for unfinished jobs....
make[1]: Leaving directory '/builddir/build/BUILD/julia-1.8.0-beta3/deps/scratch/blastrampoline-d32042273719672c6669f6442a0be5605d434b70/src'
make: *** [/builddir/build/BUILD/julia-1.8.0-beta3/deps/blastrampoline.mk:14: scratch/blastrampoline-d32042273719672c6669f6442a0be5605d434b70/build-compiled] Error 2
make: Leaving directory '/builddir/build/BUILD/julia-1.8.0-beta3/deps'

The warning probably deserves some attention but the problem isn't new I guess so there's no reason to worry the particular user who happened to build with a new compiler version.

It appears that the all-lowercase threading symbols are actually all FORTRAN ABI, as can be seen through dlsym():

julia> dlsym(libmkl_rt, "mkl_set_num_threads")
Ptr{Nothing} @0x00007fa64f24e3b0

julia> dlsym(libmkl_rt, "mkl_set_num_threads_")
Ptr{Nothing} @0x00007fa64f24e3b0

The C-ABI versions are not the non-underscore-suffixed ones, but the capitalized ones, e.g. MKL_Set_Num_Threads. The more you know.

Ref: https://github.com/JuliaLinearAlgebra/libblastrampoline/pull/59#discussion_r795090192

Also, use major version instead of full version number, ref: https://github.com/JuliaLinearAlgebra/libblastrampoline/pull/59#discussion_r795040249

MKL 2022.0.1 now has ILP64 symbols with a _64 suffix. I've verified this on Linux, and presumably this has been implemented on Mac and Windows too.

$ nm libmkl_intel_ilp64.so | grep -i _64 | grep -i dpotrf
000000000036f7c0 T dpotrf2_64
000000000036f7c0 T DPOTRF2_64
000000000035f900 T dpotrf_64
000000000035f900 T DPOTRF_64

blis_jll now has ILP64 the way we want it. But the library does not have enough symbols. I always believed it had a complete set of BLAS symbols. I haven't yet investigated if the issue is in BLIS, the way it is built, or LBT.

https://github.com/JuliaLinearAlgebra/BLIS.jl/issues/3 https://github.com/JuliaPackaging/Yggdrasil/pull/2666

julia> using LinearAlgebra, blis_jll
julia> BLAS.lbt_forward(blis_jll.blis, clear=true)
33

I tried to use FlexiBLAS as a BLAS/LAPACK backend for LBT. While lbt_forward seems to work (What does 2191 mean?) using it doesn't:

julia> libflexiblas = "/opt/software/pc2/EB-SW/software/FlexiBLAS/3.0.4-GCC-11.2.0/lib/libflexiblas.so"
"/opt/software/pc2/EB-SW/software/FlexiBLAS/3.0.4-GCC-11.2.0/lib/libflexiblas.so"

julia> using LinearAlgebra

julia> BLAS.lbt_forward(libflexiblas; clear=true)
2191

julia> BLAS.get_config()
LinearAlgebra.BLAS.LBTConfig
Libraries:
└ [ LP64] libflexiblas.so

julia> A = rand(1000,1000);

julia> A*A;
Error: no BLAS/LAPACK library loaded!

Why doesn't this work given that FlexiBLAS claims to have 100% compatibility with BLAS/LAPACK ABI/API?

Certain tools such as sanitizers don't like us loading libraries with RTLD_DEEPBIND, so since we already have the workarounds in place for systems that don't have RTLD_DEEPBIND at all, let's change these from using compile-time constants to instead use a runtime switch that can be overridden through setting the environment variable LBT_USE_RTLD_DEEPBIND=0 before running our program.

Keeping multiple copies of LBT on Windows is simply useless, wasteful, and error prone.

Refs

• https://github.com/JuliaLang/julia/issues/46872#issue-1383915227
• https://github.com/JuliaLang/julia/pull/47676#issuecomment-1367481852
• https://github.com/JuliaLinearAlgebra/libblastrampoline/pull/88#issuecomment-1367485884

I haven't tested whether it actually builds, I'm hoping CI will tell me.

I am attempting to port our BLAS/LAPACK shared libraries to work with the lbt_forward command. Our library is split into 4 shared libraries where we built them with (serial or SMP) and (32/64 or 64/64) options. However, our 64/64 libraries do not have any additional naming convention; both the 32/64 and 64/64 libraries use the same function names ( ie dgemm() ).

From my understanding, any pre-packaged Julia distribution forces users to use 64/64 BLAS. Since our 64/64 libraries do not have a suffix in any function, the lbt_forward command seems to call the 32/64 variant w/ 64-bit integers and causes a segfault at setup. In the ReadMe ABI section, it sounds like LBT require BLAS libraries to use the 64/64 OpenBLAS naming convention via the suffix. While possible on our end, is there any other way to resolve this?

Current version: 1.7.0-rc2 (last version prebuilt for powerpcle). Possible to move to 1.7.2 or higher if needed.

julia> if Base.USE_BLAS64
           # Load ILP64 forwards
           BLAS.lbt_forward(libkmlblas; clear=true, suffix_hint="64")
           # Load LP64 forward
           BLAS.lbt_forward(libkmlblas; suffix_hint="")
       else
           BLAS.lbt_forward(libkmlblas; clear=true, suffix_hint="")
       end
325

julia> BLAS.get_config()
LinearAlgebra.BLAS.LBTConfig
Libraries: 
└ [ LP64] libkblas.so

julia> Base.USE_BLAS64
true

julia> 

julia> 

julia> 

julia> aa * bb
Error: no BLAS/LAPACK library loaded!

Sorry to bother you, I'm trying to use BLAS.lbtforward to implement my own BLAS switching. I wrote a BLAS library and installed Julia on an arm-based machine. I didn't use BinaryBuilder to build my binary files. Instead, build the link library directly on this machine as shown in the code below, and successfully use the BLAS.lbtforward function to link to my link library, but when I call multiplication, the following error occurs, my library is based on MKL-BLAS The specification was written, and the switching method has also reached an agreement with MKL.jl. Please tell me where the problem lies.

Building Ipopt and MUMPS_seq with LBT using LP64 results in null pointer issues only on Windows (when using OpenBLAS32), both win64 and win32. Works fine on the other platforms.

Discussed in https://github.com/jump-dev/Ipopt.jl/pull/327

There are currently two orthogonal issues with the library on Windows (spotted in https://github.com/jump-dev/Ipopt.jl/pull/327#issuecomment-1233458190):

• the soversion of the library, fixed in src/Make.inc is stalling: https://github.com/JuliaLinearAlgebra/libblastrampoline/blob/b829b0e1583da58bc80acbcc0cfb83b482530b92/src/Make.inc#L26-L28 but the latest release at the moment is v5.1.1
• https://github.com/JuliaLinearAlgebra/libblastrampoline/blob/b829b0e1583da58bc80acbcc0cfb83b482530b92/src/Makefile#L45-L46 generates the import library for $@, which is LIB_FULL_VERSION, which is a bad idea because it means the ABI is broken in every single version (well, every single version where the soversion is actually bumped, see point above)

An overall solution is probably to stop generating the library with the full name on Windows, or at least generating the import library for LIB_MAJOR_VERSION instead of LIB_FULL_VERSION.

It remains the problem of actually remembering to update the soversion in src/Make.inc when doing a new release.

CC: @ViralBShah

All GitHub Actions builds for Windows x86 since July 2022 have failed, and the first one to fail was associated with the merge commit for #80, which literally only touched the README. All other platforms seem just fine.