You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
BLAS support for M1 ARM64 via Apple's Accelerate #869
So in theory, it should be possible to extend this so that depending on a given platform either OpenBLAS or other BLAS solutions are used transparently by default.
So this issue discusses what needs to be done to have Apple's Accelerate access to M1 hardware acceleration available by default in Julia
Reacted by Feraidoon Mehri, Remy Prechelt, Royi, Donglai Gong, cossio, Lamont Granquist, Zeyu Mao, Clemens Brunner and Louis Lac
Reacted by Feraidoon Mehri
Reacted by Feraidoon Mehri, Donglai Gong and C. Brenhin Keller
The plan was to have LBT as a way to pick a different BLAS than the default OpenBLAS for now. That requires you to load a package every time you start Julia to change the default. Eventually, once the Preferences mechanism becomes standard, we want to use that so that users can pick a different BLAS by default.
I don't think we want to depend on the Apple provided BLAS by default for the M1 for now.
Some (anecdotal) benchmark scenarios that might illustrate why Accelerate makes sense (at least for gemm type of operations). I've done a machine learning framework and benchmark suite, syncognite/bench written in C++ using eigen3 (so neither Julia nor OpenBLAS), but it might serve as a hint for what kind of performance improvements are possible.
The following table compare a benchmark run with native eigen3 on ARM64, and a second run using Apple's Accelerate within eigen (USE_SYSTEM_BLAS variable). The percentage columns show the speed-up for forward (fw) and backward (bw) path of neural network training when using accelerate, e.g:
Affine layers (basically a matrix mult): 480-600% faster
The first thing is to create an MKL.jl like package for Apple Accelerate. We already have some support in LBT for Accelerate - so this should be fairly quick.
A difficulty is that accelerate uses the 32 bit API, which is why AppleAccelerateLinAlgWrapper manually defines the methods it uses (and is based on Elliot's code).
(Also, AppleAccelerateLinAlgWrapper has a deliberately cumbersome name to avoid stealing/squatting on potentially valuable names for a future such package that supersedes it.)
Since Accelerate only has an LP64 BLAS and Julia wants ILP64, this is quite difficult, unless we can somehow make this choice dynamic as discussed in #891.
It should be possible to have a separate wrapper like @chriselrod discusses above that packages can directly invoke, but swapping it in as the default BLAS in Julia is fairly non-trivial.
Reacted by C. T. (Tim) Kelley, cossio and Tony Savostianov
I‘m no expert but wouldn‘t it be quite easy to write some kind of wrapper libblas which just redirects level 3 BLAS calls to the Apple accelerate BLAS and all other calls to OpenBLAS? I mean ILP64 does not really play a role for level 3 BLAS imho anyway. On the other hand, level 3 BLAS routines are probably the only routines which benefit from Apple‘s AMX extension…
Yes we could have a wrapper library that redirects all 64-bit ILP64 calls to a 32-bit BLAS. It seems like it would be easier to have Apple just provide ILP64 support with mangled names. Intel is doing that in MKL now.
Do we have a way to ask Apple to do this? @Keno@staticfloat Perhaps one of you had a contact at Apple?
Nice that you guys have the appropriate contacts ;-) However, what I heard from other discussions, Apple seams to assign currently very little resources to their BLAS/LAPACK development. So, I wouldn't bet on them... Nevertheless, I keep my fingers crossed ^^
The other proper solution would be to have AMX kernels implemented in OpenBLAS. It seems that (Apple M1) AMX has been decrypted now (inofficially) more or less completly: https://github.com/corsix/amx. I guess with this knowledge the guys from OpenBLAS should be able to do their job (if there are no legal restriction applying).
None of the problems we're talking about here should be impossible for Apple to fix
Most problems (aside from minor GFLOPS blips) probably are impossible to fix.
if they care.
I wouldn't phrase it like that. Apple engineers know OpenBLAS exists, and it already serves its purpose: high throughput, no matter the energy cost. Nvidia RTX 4090 exists, and it has the highest TFLOPS but zero regard for power efficiency. Apple's AMX hardware was tailored to be an "ML accelerator" (sic) and not a linear algebra accelerator. It performs real-valued convolutions or batched real-FP32 matrix multiplications. Modestly high-throughput real-FP64 was just a cool side effect.
Rather than reinvent the wheel, they gave developers something new - the ability to trade off between sheer performance and power efficiency. That is crucial for more apps than you think. They also did this with the GPU, sacrificing performance for the sake of efficiency.
I might be late to the party, but here are some additional results. I used a script based on @ctkelley 's script above. Notably, I added ComplexF64 calculations ("-64Z" columns below).
Tested on Mac mini M2 Pro (12/19), 32 GB RAM, with julia -t auto (giving 8 Julia threads and also 8 BLAS threads).
Finally, I wanted to see how Accelerate would perform in a parallel context. I prepared 800 pairs of matrices and multiplied them in a sequential loop and also in a Threads.@threads loop. When running a sequential loop I still allowed for Open BLAS parallellism to take place, but when running a threaded loop I used BLAS.set_num_threads(1).
For N = 32 I obtained the following:
In my experience, threading significantly degraded performance, at least for SGEMM and DGEMM when harnessing AMX. N=32 was just barely utilizing the AMX on my machine (>50 GFLOPS FP64, <100 GFLOPS FP32). With yours, both FP64 and FP32 would have nonphysical performance if using only single-core NEON.
On the M1 family with old Accelerate, complex multiplications had 2x less ALU utilization than real multiplications. That changed to 1x with M2, new Accelerate.
ZHEEV and DSYEV, k=unknown. I adjusted k for complex numbers to properly compare ALU % (k -> 0.25k). Accelerate has the same ALU utilization with complex, OpenBLAS has 2x more. I'll need to test whether ZHEEV_2STAGE and DSYEV_2STAGE are faster.
GFLOPS/k_adjusted
O64Z
A64Z
O64
A64
32
1.47
1.52
0.43
0.50
64
2.14
2.18
0.79
1.01
128
3.36
2.54
1.61
1.75
256
5.59
4.79
2.62
3.50
512
11.93
6.55
4.97
5.16
1024
20.45
8.10
8.26
7.16
2048
24.54
5.63
10.74
7.81
4096
28.63
5.98
14.62
6.25
8192
33.32
5.94
16.66
6.96
Sequential ZGEMM and DGEMM, k=2. My machine reached 78 GFLOPS for A64, N=32. Yours reached 109.2, maybe because M2 can issue more AMX instructions/cycle.
@philipturner Thanks for the in-depth benchmarking. I am doing the dead simple one with peakflops, in which Accelerate gives twice the FLOP rate of OpenBLAS. I suspect the first step here is to get a new version of LBT merged and tagged and included in Julia 1.9.
At that point AppleAccelerate.jl or a similar package can set up BLAS forwarding like we do with MKL.jl. That way, in Julia 1.9 (or 1.9.1), we can get the ability to use Accelerate with just a Pkg.add(). Later on, as we gain more experience, we can decide if we want it to be the default - but the evidence is unclear for that looking at your experiments.
Reacted by C. T. (Tim) Kelley, Dominik Schlösser and Zeyu Mao
Alright, I think the next step is to improve AppleAccelerate.jl to auto-load ILP64 Accelerate, when it is available. We may also want to give it the ability to load LAPACK_jll to provide LAPACK and only use Accelerate for the underlying BLAS calls. This simultaneously dodges the cholesky bug that we identified, and gives us the ability to see if vanilla LAPACK_jll provides any kind of performance improvement over Accelerate's LAPACK (doubtful). An even more interesting test would be to load OpenBLAS first, then load only Accelerate's libBLAS.dylib on top of that, so that we use OpenBLAS's threaded LAPACK routines, and Accelerate's BLAS routines.
I think the LAPACK in our OpenBLAS probably directly links to its own BLAS and not LBT. I think some work needs to be done to assemble Accelerate BLAS, openblas accelerated lapack routines and the rest of lapack.
The LAPACK in BB on the other hand does link to LBT. so I think the recipe would be to load accelerate first, then lapack from BB, and finally manually forward a few lapack routines to openblas.
My contact at Apple has asked for any examples of real-world usage that can be shown to work well on Accelerate; given that the LinearAlgebra test suite finishes with a small performance improvement over the OpenBLAS test suite, I'm inclined to think that small-scale LinearAlgebra problems may have a natural advantage here (as that is the majority of our test suite). They are looking for good examples that they can use to bolster the importance of developing good tools for people like us, so don't be shy!
For very small problems, which are too small for OpenBLAS to harness multiple cores, Accelerate should excel. It uses the AMX because it's going to help single-core performance. For example, iPhones have only 2 performance cores so the AMX doesn't have any less vector throughput than the NEON units. Unfortunately, this tactic doesn't scale when the amount of NEON compute increases.
Generally, one-sided factorizations like LU should be fast in Accelerate. They can use block size 32 on AMX single-core quite easily. So a good fraction of GEMM GFLOPS.
Two-sided factorizations like tridiagonalization are where Accelerate (and OpenBLAS) are painfully slow. I had to write custom kernels that convert these into panel (QR) factorizations with most computations batched into small GEMM calls. Then the final bulge chasing which still needs further optimization at larger block sizes.
Perf on M3 should be no different than M1 or M2. The hardware hasn’t changed much fundamentally, except for some new instruction issuing capabilities and lower precisions for AI stuff.
Activity
ViralBShah commentedon Sep 19, 2021
The plan was to have LBT as a way to pick a different BLAS than the default OpenBLAS for now. That requires you to load a package every time you start Julia to change the default. Eventually, once the Preferences mechanism becomes standard, we want to use that so that users can pick a different BLAS by default.
I don't think we want to depend on the Apple provided BLAS by default for the M1 for now.
domschl commentedon Sep 20, 2021
Some (anecdotal) benchmark scenarios that might illustrate why Accelerate makes sense (at least for
gemmtype of operations). I've done a machine learning framework and benchmark suite, syncognite/bench written in C++ using eigen3 (so neither Julia nor OpenBLAS), but it might serve as a hint for what kind of performance improvements are possible.The following table compare a benchmark run with native eigen3 on ARM64, and a second run using Apple's Accelerate within eigen (
USE_SYSTEM_BLASvariable). The percentage columns show the speed-up for forward (fw) and backward (bw) path of neural network training when using accelerate, e.g:all that with very low energy usage!
DilumAluthge commentedon Sep 20, 2021
I think @chriselrod has played around with this a little bit.
ViralBShah commentedon Sep 20, 2021
The first thing is to create an MKL.jl like package for Apple Accelerate. We already have some support in LBT for Accelerate - so this should be fairly quick.
chriselrod commentedon Sep 20, 2021
A difficulty is that accelerate uses the 32 bit API, which is why AppleAccelerateLinAlgWrapper manually defines the methods it uses (and is based on Elliot's code).
(Also, AppleAccelerateLinAlgWrapper has a deliberately cumbersome name to avoid stealing/squatting on potentially valuable names for a future such package that supersedes it.)
ViralBShah commentedon Sep 5, 2022
Since Accelerate only has an LP64 BLAS and Julia wants ILP64, this is quite difficult, unless we can somehow make this choice dynamic as discussed in #891.
It should be possible to have a separate wrapper like @chriselrod discusses above that packages can directly invoke, but swapping it in as the default BLAS in Julia is fairly non-trivial.
zinphi commentedon Oct 11, 2022
I‘m no expert but wouldn‘t it be quite easy to write some kind of wrapper libblas which just redirects level 3 BLAS calls to the Apple accelerate BLAS and all other calls to OpenBLAS? I mean ILP64 does not really play a role for level 3 BLAS imho anyway. On the other hand, level 3 BLAS routines are probably the only routines which benefit from Apple‘s AMX extension…
ViralBShah commentedon Oct 15, 2022
Yes we could have a wrapper library that redirects all 64-bit ILP64 calls to a 32-bit BLAS. It seems like it would be easier to have Apple just provide ILP64 support with mangled names. Intel is doing that in MKL now.
Do we have a way to ask Apple to do this? @Keno @staticfloat Perhaps one of you had a contact at Apple?
Keno commentedon Oct 15, 2022
We can ask.
zinphi commentedon Oct 17, 2022
Nice that you guys have the appropriate contacts ;-) However, what I heard from other discussions, Apple seams to assign currently very little resources to their BLAS/LAPACK development. So, I wouldn't bet on them... Nevertheless, I keep my fingers crossed ^^
The other proper solution would be to have AMX kernels implemented in OpenBLAS. It seems that (Apple M1) AMX has been decrypted now (inofficially) more or less completly: https://github.com/corsix/amx. I guess with this knowledge the guys from OpenBLAS should be able to do their job (if there are no legal restriction applying).
ViralBShah commentedon Oct 17, 2022
Please do file the pointer to Apple AMX kernels in an issue on the openblas github repo. Yes, it would be great for openblas to have those kernels.
zinphi commentedon Oct 17, 2022
I tried my best and opened a new issue there (see OpenMathLib/OpenBLAS#3789). Let's see what they think about that.
mzy2240 commentedon Nov 24, 2022
Might be relevant: mlpack/mlpack#3308
70 remaining items
philipturner commentedon Mar 31, 2023
Most problems (aside from minor GFLOPS blips) probably are impossible to fix.
I wouldn't phrase it like that. Apple engineers know OpenBLAS exists, and it already serves its purpose: high throughput, no matter the energy cost. Nvidia RTX 4090 exists, and it has the highest TFLOPS but zero regard for power efficiency. Apple's AMX hardware was tailored to be an "ML accelerator" (sic) and not a linear algebra accelerator. It performs real-valued convolutions or batched real-FP32 matrix multiplications. Modestly high-throughput real-FP64 was just a cool side effect.
Rather than reinvent the wheel, they gave developers something new - the ability to trade off between sheer performance and power efficiency. That is crucial for more apps than you think. They also did this with the GPU, sacrificing performance for the sake of efficiency.
yakovbraver commentedon Apr 1, 2023
I might be late to the party, but here are some additional results. I used a script based on @ctkelley 's script above. Notably, I added ComplexF64 calculations ("-64Z" columns below).
Tested on Mac mini M2 Pro (12/19), 32 GB RAM, with
julia -t auto(giving 8 Julia threads and also 8 BLAS threads).Dot product
Here, Accelerate gives a nice ~x2.5–x3 speed-up for ComplexF64 and Float64.
Matrix multiplication
These results are in line with @ctkelley 's. Accelerate shines in real Float calculations.
Eigenvalues and eigenvectors for complex Hermitian and real symmetric matrices
Here, Accelerate slows down the calculation.
Parallel matrix multiplication
Finally, I wanted to see how Accelerate would perform in a parallel context. I prepared 800 pairs of matrices and multiplied them in a sequential loop and also in a
Threads.@threadsloop. When running a sequential loop I still allowed for Open BLAS parallellism to take place, but when running a threaded loop I usedBLAS.set_num_threads(1).For N = 32 I obtained the following:
We get a considerable speed-up when parallelising the Accelerate calculations.
And now N = 512:
Perhaps running all 8 cores to get a x1.5 speed-up is not worth it, but at least it seems that threading Accelerate code does not degrade performance.
All in all, I think it is certainly beneficial to expose Accelerate to the general user.
philipturner commentedon Apr 1, 2023
In my experience, threading significantly degraded performance, at least for SGEMM and DGEMM when harnessing AMX. N=32 was just barely utilizing the AMX on my machine (>50 GFLOPS FP64, <100 GFLOPS FP32). With yours, both FP64 and FP32 would have nonphysical performance if using only single-core NEON.
Matrix Multiplication Performance.xlsx
On the M1 family with old Accelerate, complex multiplications had 2x less ALU utilization than real multiplications. That changed to 1x with M2, new Accelerate.
BLAS Performance.xlsx
https://github.com/danielchalef/openblas-benchmark-m1
ZHEEV and DSYEV, k=unknown. I adjusted k for complex numbers to properly compare ALU % (k -> 0.25k). Accelerate has the same ALU utilization with complex, OpenBLAS has 2x more. I'll need to test whether ZHEEV_2STAGE and DSYEV_2STAGE are faster.
Sequential ZGEMM and DGEMM, k=2. My machine reached 78 GFLOPS for A64, N=32. Yours reached 109.2, maybe because M2 can issue more AMX instructions/cycle.
ViralBShah commentedon Apr 2, 2023
@philipturner Thanks for the in-depth benchmarking. I am doing the dead simple one with
peakflops, in which Accelerate gives twice the FLOP rate of OpenBLAS. I suspect the first step here is to get a new version of LBT merged and tagged and included in Julia 1.9.At that point AppleAccelerate.jl or a similar package can set up BLAS forwarding like we do with MKL.jl. That way, in Julia 1.9 (or 1.9.1), we can get the ability to use Accelerate with just a
Pkg.add(). Later on, as we gain more experience, we can decide if we want it to be the default - but the evidence is unclear for that looking at your experiments.staticfloat commentedon Apr 6, 2023
Alright, I think the next step is to improve AppleAccelerate.jl to auto-load ILP64 Accelerate, when it is available. We may also want to give it the ability to load LAPACK_jll to provide LAPACK and only use Accelerate for the underlying BLAS calls. This simultaneously dodges the cholesky bug that we identified, and gives us the ability to see if vanilla LAPACK_jll provides any kind of performance improvement over Accelerate's LAPACK (doubtful). An even more interesting test would be to load OpenBLAS first, then load only Accelerate's
libBLAS.dylibon top of that, so that we use OpenBLAS's threaded LAPACK routines, and Accelerate's BLAS routines.ViralBShah commentedon Apr 6, 2023
I think the LAPACK in our OpenBLAS probably directly links to its own BLAS and not LBT. I think some work needs to be done to assemble Accelerate BLAS, openblas accelerated lapack routines and the rest of lapack.
The LAPACK in BB on the other hand does link to LBT. so I think the recipe would be to load accelerate first, then lapack from BB, and finally manually forward a few lapack routines to openblas.
staticfloat commentedon Apr 12, 2023
With JuliaLinearAlgebra/AppleAccelerate.jl#58 and incorporating this very minor fix to our LinearAlgebra test suite, we actually pass the entire Julia test suite while running on top of Accelerate.
My contact at Apple has asked for any examples of real-world usage that can be shown to work well on Accelerate; given that the LinearAlgebra test suite finishes with a small performance improvement over the OpenBLAS test suite, I'm inclined to think that small-scale LinearAlgebra problems may have a natural advantage here (as that is the majority of our test suite). They are looking for good examples that they can use to bolster the importance of developing good tools for people like us, so don't be shy!
philipturner commentedon Apr 12, 2023
For very small problems, which are too small for OpenBLAS to harness multiple cores, Accelerate should excel. It uses the AMX because it's going to help single-core performance. For example, iPhones have only 2 performance cores so the AMX doesn't have any less vector throughput than the NEON units. Unfortunately, this tactic doesn't scale when the amount of NEON compute increases.
ViralBShah commentedon May 22, 2023
Support is already on AppleAccelerate.jl master, and will be released when JuliaLinearAlgebra/AppleAccelerate.jl#62 merges.
Announcement in https://discourse.julialang.org/t/appleaccelerate-jl-v0-4-0/99351/3
ctkelley commentedon Sep 26, 2023
Sonoma does better than OS 13.x. Here are some revised lu! numbers
The discontinuity at higher dimensions seems to be gone. 8 core M2 Pro.
oscarbg commentedon Apr 13, 2024
Unrelated to the issue but can anyone share some m3 perf numbers like in lu for example..
philipturner commentedon Apr 14, 2024
Generally, one-sided factorizations like LU should be fast in Accelerate. They can use block size 32 on AMX single-core quite easily. So a good fraction of GEMM GFLOPS.
Two-sided factorizations like tridiagonalization are where Accelerate (and OpenBLAS) are painfully slow. I had to write custom kernels that convert these into panel (QR) factorizations with most computations batched into small GEMM calls. Then the final bulge chasing which still needs further optimization at larger block sizes.
Perf on M3 should be no different than M1 or M2. The hardware hasn’t changed much fundamentally, except for some new instruction issuing capabilities and lower precisions for AI stuff.
oscarbg commentedon Nov 27, 2024