Skip to content

BLAS support for M1 ARM64 via Apple's Accelerate #869

Closed

Description

@domschl

The default BLAS Julia uses is OpenBLAS. Apple's M1 has proprietary dedicated matrix hardware that is only accessible via Apple's Accelerate BLAS implementation. That proprietary interface can provide 2x to 4x speedups for some linear algebra use cases (see https://discourse.julialang.org/t/does-mac-m1-in-multithreads-is-slower-that-in-single-thread/61114/12?u=kristoffer.carlsson for some benchmarks and discussion.)

Since Julia 1.7 there's a BLAS multiplexer:

So in theory, it should be possible to extend this so that depending on a given platform either OpenBLAS or other BLAS solutions are used transparently by default.

So this issue discusses what needs to be done to have Apple's Accelerate access to M1 hardware acceleration available by default in Julia

Activity

  1. ViralBShah commented on Sep 19, 2021

    @ViralBShah
    SponsorMember

    The plan was to have LBT as a way to pick a different BLAS than the default OpenBLAS for now. That requires you to load a package every time you start Julia to change the default. Eventually, once the Preferences mechanism becomes standard, we want to use that so that users can pick a different BLAS by default.

    I don't think we want to depend on the Apple provided BLAS by default for the M1 for now.

  2. added
    needs decisionA decision on this change is needed
    system:apple siliconAffects Apple Silicon only (Darwin/ARM64) - e.g. M1 and other M-series chips
    on Sep 19, 2021
  3. domschl commented on Sep 20, 2021

    @domschl
    Author

    Some (anecdotal) benchmark scenarios that might illustrate why Accelerate makes sense (at least for gemm type of operations). I've done a machine learning framework and benchmark suite, syncognite/bench written in C++ using eigen3 (so neither Julia nor OpenBLAS), but it might serve as a hint for what kind of performance improvements are possible.

    The following table compare a benchmark run with native eigen3 on ARM64, and a second run using Apple's Accelerate within eigen (USE_SYSTEM_BLAS variable). The percentage columns show the speed-up for forward (fw) and backward (bw) path of neural network training when using accelerate, e.g:

    • Affine layers (basically a matrix mult): 480-600% faster
    • Affine layer with ReLU: 420-500% faster
    • LSTM recurrent networks: 200-300%

    all that with very low energy usage!

    Layer             fw(ms) fw/N(ms)   %  bw(ms) bw/N(ms)   %
    OneHot             1.068  0.0107  0.1   0.000  0.0000 66.6
    Affine             0.464  0.0050 620.1  1.210  0.0130 485.4
    AffineRelu         0.538  0.0056 506.4  1.294  0.0135 424.4
    Relu               0.023  0.0002  5.8   0.423  0.0042 -1.8
    Nonlin-Relu        0.064  0.0006 -1.9   0.067  0.0007  0.7
    Nonlin-Sgmd        0.126  0.0012 -1.2   0.067  0.0007 -1.8
    Nonlin-Tanh        0.099  0.0010 -0.0   0.070  0.0007 -0.5
    Nonlin-Selu        0.746  0.0075 -0.1   0.649  0.0065 -0.4
    Nonlin-Resilu      0.202  0.0020  0.2   0.243  0.0024 -0.6
    Dropout            0.713  0.0072 -0.8   0.030  0.0003  1.5
    BatchNorm          0.159  0.0016 -3.1   0.217  0.0022 -4.6
    SpatialBtchNrm     2.610  0.0260 -1.0   3.936  0.0393 -0.2
    Convolution       54.388  0.5470 20.3  58.559  0.5869 45.1
    Pooling            6.421  0.0643 -0.0  22.589  0.2262  1.7
    LSTM              27.616  0.2778 299.8 78.172  0.7677 198.0
    RNN               16.929  0.1660 110.4 33.648  0.3297 114.6
    TemporalAff        5.704  0.0569 105.7  5.545  0.0554 209.3
    WordEmbed          5.379  0.0541 129.1  4.227  0.0419 116.0
    SVM                0.229  0.0023  0.5   0.225  0.0022  0.9
    Softmax            0.228  0.0023  0.1   0.022  0.0002  1.5
    TemporalSM         4.356  0.0434 -0.8   1.401  0.0140  0.1
    TwoLayerNet        1.776  0.0177 300.5  2.555  0.0255 425.1
    
  4. DilumAluthge commented on Sep 20, 2021

    @DilumAluthge
    Member

    I think @chriselrod has played around with this a little bit.

  5. ViralBShah commented on Sep 20, 2021

    @ViralBShah
    SponsorMember

    The first thing is to create an MKL.jl like package for Apple Accelerate. We already have some support in LBT for Accelerate - so this should be fairly quick.

  6. chriselrod commented on Sep 20, 2021

    @chriselrod
    Contributor

    A difficulty is that accelerate uses the 32 bit API, which is why AppleAccelerateLinAlgWrapper manually defines the methods it uses (and is based on Elliot's code).

    (Also, AppleAccelerateLinAlgWrapper has a deliberately cumbersome name to avoid stealing/squatting on potentially valuable names for a future such package that supersedes it.)

  7. ViralBShah commented on Sep 5, 2022

    @ViralBShah
    SponsorMember

    Since Accelerate only has an LP64 BLAS and Julia wants ILP64, this is quite difficult, unless we can somehow make this choice dynamic as discussed in #891.

    It should be possible to have a separate wrapper like @chriselrod discusses above that packages can directly invoke, but swapping it in as the default BLAS in Julia is fairly non-trivial.

  8. zinphi commented on Oct 11, 2022

    @zinphi

    I‘m no expert but wouldn‘t it be quite easy to write some kind of wrapper libblas which just redirects level 3 BLAS calls to the Apple accelerate BLAS and all other calls to OpenBLAS? I mean ILP64 does not really play a role for level 3 BLAS imho anyway. On the other hand, level 3 BLAS routines are probably the only routines which benefit from Apple‘s AMX extension…

  9. ViralBShah commented on Oct 15, 2022

    @ViralBShah
    SponsorMember

    Yes we could have a wrapper library that redirects all 64-bit ILP64 calls to a 32-bit BLAS. It seems like it would be easier to have Apple just provide ILP64 support with mangled names. Intel is doing that in MKL now.

    Do we have a way to ask Apple to do this? @Keno @staticfloat Perhaps one of you had a contact at Apple?

  10. Keno commented on Oct 15, 2022

    @Keno
    Member

    We can ask.

  11. zinphi commented on Oct 17, 2022

    @zinphi

    Nice that you guys have the appropriate contacts ;-) However, what I heard from other discussions, Apple seams to assign currently very little resources to their BLAS/LAPACK development. So, I wouldn't bet on them... Nevertheless, I keep my fingers crossed ^^
    The other proper solution would be to have AMX kernels implemented in OpenBLAS. It seems that (Apple M1) AMX has been decrypted now (inofficially) more or less completly: https://github.com/corsix/amx. I guess with this knowledge the guys from OpenBLAS should be able to do their job (if there are no legal restriction applying).

  12. ViralBShah commented on Oct 17, 2022

    @ViralBShah
    SponsorMember

    Please do file the pointer to Apple AMX kernels in an issue on the openblas github repo. Yes, it would be great for openblas to have those kernels.

  13. zinphi commented on Oct 17, 2022

    @zinphi

    I tried my best and opened a new issue there (see OpenMathLib/OpenBLAS#3789). Let's see what they think about that.

  14. mzy2240 commented on Nov 24, 2022

    @mzy2240

    Might be relevant: mlpack/mlpack#3308

  15. 70 remaining items

  16. philipturner commented on Mar 31, 2023

    @philipturner

    None of the problems we're talking about here should be impossible for Apple to fix

    Most problems (aside from minor GFLOPS blips) probably are impossible to fix.

    if they care.

    I wouldn't phrase it like that. Apple engineers know OpenBLAS exists, and it already serves its purpose: high throughput, no matter the energy cost. Nvidia RTX 4090 exists, and it has the highest TFLOPS but zero regard for power efficiency. Apple's AMX hardware was tailored to be an "ML accelerator" (sic) and not a linear algebra accelerator. It performs real-valued convolutions or batched real-FP32 matrix multiplications. Modestly high-throughput real-FP64 was just a cool side effect.

    Rather than reinvent the wheel, they gave developers something new - the ability to trade off between sheer performance and power efficiency. That is crucial for more apps than you think. They also did this with the GPU, sacrificing performance for the sake of efficiency.

  17. yakovbraver commented on Apr 1, 2023

    @yakovbraver

    I might be late to the party, but here are some additional results. I used a script based on @ctkelley 's script above. Notably, I added ComplexF64 calculations ("-64Z" columns below).
    Tested on Mac mini M2 Pro (12/19), 32 GB RAM, with julia -t auto (giving 8 Julia threads and also 8 BLAS threads).

    Dot product

    Test of dot(A, B)
           N    O-64Z    A-64Z  R-64Z     O-64     A-64   R-64     O-32     A-32   R-32 
          32  1.9e-08  1.2e-08   0.67  1.0e-08  8.8e-09   0.87  7.0e-09  8.8e-09   1.24
          64  4.0e-08  2.0e-08   0.49  1.6e-08  1.1e-08   0.72  7.9e-09  9.9e-09   1.25 
         128  8.4e-08  3.8e-08   0.45  3.5e-08  1.8e-08   0.51  1.3e-08  1.2e-08   0.95 
         256  1.9e-07  7.2e-08   0.37  7.2e-08  3.1e-08   0.44  1.9e-08  1.8e-08   0.96 
         512  4.1e-07  1.5e-07   0.36  1.7e-07  6.1e-08   0.36  3.2e-08  3.2e-08   1.00 
        1024  8.5e-07  2.9e-07   0.35  3.9e-07  1.3e-07   0.34  6.2e-08  6.3e-08   1.02 
        2048  1.7e-06  5.9e-07   0.34  8.3e-07  2.8e-07   0.34  1.3e-07  1.4e-07   1.01 
        4096  3.5e-06  1.2e-06   0.34  1.7e-06  5.7e-07   0.34  2.8e-07  1.7e-07   0.62 
        8192  7.0e-06  2.9e-06   0.41  3.5e-06  1.2e-06   0.34  5.7e-07  2.9e-07   0.50 
       16384  1.4e-05  5.8e-06   0.41  7.0e-06  2.9e-06   0.41  1.2e-06  5.0e-07   0.43 
       32768  2.8e-05  1.1e-05   0.40  1.4e-05  5.7e-06   0.41  2.9e-06  9.4e-07   0.33
    

    Here, Accelerate gives a nice ~x2.5–x3 speed-up for ComplexF64 and Float64.

    Matrix multiplication

    Test of mul!(C, A, B)
           N    O-64Z    A-64Z  R-64Z     O-64     A-64   R-64     O-32     A-32   R-32 
          32  1.2e-05  4.1e-06   0.34  1.7e-06  3.4e-07   0.20  8.9e-07  2.3e-07   0.25 
          64  1.2e-05  1.5e-05   1.33  1.3e-05  1.8e-06   0.14  5.8e-06  6.1e-07   0.11 
         128  1.1e-04  7.8e-05   0.71  2.2e-05  1.2e-05   0.55  1.3e-05  3.5e-06   0.27 
         256  4.4e-04  6.1e-04   1.39  1.1e-04  9.0e-05   0.83  5.2e-05  2.4e-05   0.47 
         512  3.8e-03  3.2e-03   0.84  7.3e-04  3.8e-04   0.51  3.6e-04  1.1e-04   0.29 
        1024  2.7e-02  2.2e-02   0.80  5.7e-03  3.1e-03   0.54  2.9e-03  7.6e-04   0.26 
        2048  2.2e-01  1.5e-01   0.69  4.6e-02  3.0e-02   0.66  2.3e-02  8.3e-03   0.35 
        4096  1.8e+00  1.1e+00   0.60  3.6e-01  2.3e-01   0.63  1.9e-01  6.6e-02   0.35 
        8192  1.4e+01  7.8e+00   0.56  2.9e+00  1.8e+00   0.64  1.5e+00  5.0e-01   0.32 
    

    These results are in line with @ctkelley 's. Accelerate shines in real Float calculations.

    Eigenvalues and eigenvectors for complex Hermitian and real symmetric matrices

    Test of eigen(A)
           N    O-64Z    A-64Z  R-64Z     O-64     A-64   R-64     O-32     A-32   R-32 
          32  8.9e-05  8.6e-05   0.97  7.7e-05  6.6e-05   0.85  7.0e-05  5.3e-05   0.76 
          64  4.9e-04  4.8e-04   0.99  3.3e-04  2.6e-04   0.80  2.9e-04  2.3e-04   0.79 
         128  2.5e-03  3.3e-03   1.33  1.3e-03  1.2e-03   0.90  1.2e-03  9.7e-04   0.83 
         256  1.2e-02  1.4e-02   1.09  6.4e-03  4.8e-03   0.74  6.6e-03  5.0e-03   0.76 
         512  4.5e-02  8.2e-02   1.81  2.7e-02  2.6e-02   0.96  3.3e-02  3.1e-02   0.93 
        1024  2.1e-01  5.3e-01   2.57  1.3e-01  1.5e-01   1.09  1.4e-01  1.4e-01   1.04 
        2048  1.4e+00  6.1e+00   4.47  8.0e-01  1.1e+00   1.39  6.5e-01  7.8e-01   1.20 
        4096  9.6e+00  4.6e+01   4.83  4.7e+00  1.1e+01   2.22  2.0e+01  6.4e+00   0.31 
        8192  6.6e+01  3.7e+02   5.59  3.3e+01  7.9e+01   2.39  1.6e+02  1.4e+02   0.86
    

    Here, Accelerate slows down the calculation.

    Parallel matrix multiplication

    Finally, I wanted to see how Accelerate would perform in a parallel context. I prepared 800 pairs of matrices and multiplied them in a sequential loop and also in a Threads.@threads loop. When running a sequential loop I still allowed for Open BLAS parallellism to take place, but when running a threaded loop I used BLAS.set_num_threads(1).
    For N = 32 I obtained the following:

                   O-64Z    A-64Z  R-64Z     O-64     A-64   R-64     O-32     A-32   R-32 
     Sequential: 1.0e-02  3.7e-03   0.35  1.6e-03  4.8e-04   0.31  7.7e-04  2.2e-04   0.29 
     Parallel:   1.5e-03  8.9e-04   0.61  7.8e-04  1.5e-04   0.19  6.9e-04  7.9e-05   0.12
     Speed-up:       6.7      4.2             2.1      3.2             1.1      2.8
    

    We get a considerable speed-up when parallelising the Accelerate calculations.
    And now N = 512:

                   O-64Z    A-64Z  R-64Z     O-64     A-64   R-64     O-32     A-32   R-32 
     Sequential: 3.2e+00  2.7e+00   0.85  6.4e-01  5.5e-01   0.86  4.6e-01  1.7e-01   0.36 
     Parallel:   2.8e+00  1.6e+00   0.56  5.7e-01  3.7e-01   0.65  2.8e-01  1.1e-01   0.38
     Speed-up:       1.1      1.7             1.1      1.5             1.6      1.5
    

    Perhaps running all 8 cores to get a x1.5 speed-up is not worth it, but at least it seems that threading Accelerate code does not degrade performance.

    All in all, I think it is certainly beneficial to expose Accelerate to the general user.

  18. philipturner commented on Apr 1, 2023

    @philipturner

    In my experience, threading significantly degraded performance, at least for SGEMM and DGEMM when harnessing AMX. N=32 was just barely utilizing the AMX on my machine (>50 GFLOPS FP64, <100 GFLOPS FP32). With yours, both FP64 and FP32 would have nonphysical performance if using only single-core NEON.

    Matrix Multiplication Performance.xlsx

    On the M1 family with old Accelerate, complex multiplications had 2x less ALU utilization than real multiplications. That changed to 1x with M2, new Accelerate.

    BLAS Performance.xlsx
    https://github.com/danielchalef/openblas-benchmark-m1

    ZHEEV and DSYEV, k=unknown. I adjusted k for complex numbers to properly compare ALU % (k -> 0.25k). Accelerate has the same ALU utilization with complex, OpenBLAS has 2x more. I'll need to test whether ZHEEV_2STAGE and DSYEV_2STAGE are faster.

    GFLOPS/k_adjusted O64Z A64Z O64 A64
    32 1.47 1.52 0.43 0.50
    64 2.14 2.18 0.79 1.01
    128 3.36 2.54 1.61 1.75
    256 5.59 4.79 2.62 3.50
    512 11.93 6.55 4.97 5.16
    1024 20.45 8.10 8.26 7.16
    2048 24.54 5.63 10.74 7.81
    4096 28.63 5.98 14.62 6.25
    8192 33.32 5.94 16.66 6.96

    Sequential ZGEMM and DGEMM, k=2. My machine reached 78 GFLOPS for A64, N=32. Yours reached 109.2, maybe because M2 can issue more AMX instructions/cycle.

    GFLOPS/k_adjusted O64Z A64Z O64 A64
    32 10.5 28.3 16.4 54.6
    512 134.2 159.1 167.8 195.2
    GFLOPS O64Z A64Z O64 A64
    32 21.0 56.6 32.8 109.2
    512 268.4 318.2 335.6 390.4
  19. ViralBShah commented on Apr 2, 2023

    @ViralBShah
    SponsorMember

    @philipturner Thanks for the in-depth benchmarking. I am doing the dead simple one with peakflops, in which Accelerate gives twice the FLOP rate of OpenBLAS. I suspect the first step here is to get a new version of LBT merged and tagged and included in Julia 1.9.

    At that point AppleAccelerate.jl or a similar package can set up BLAS forwarding like we do with MKL.jl. That way, in Julia 1.9 (or 1.9.1), we can get the ability to use Accelerate with just a Pkg.add(). Later on, as we gain more experience, we can decide if we want it to be the default - but the evidence is unclear for that looking at your experiments.

  20. staticfloat commented on Apr 6, 2023

    @staticfloat
    SponsorMember

    Alright, I think the next step is to improve AppleAccelerate.jl to auto-load ILP64 Accelerate, when it is available. We may also want to give it the ability to load LAPACK_jll to provide LAPACK and only use Accelerate for the underlying BLAS calls. This simultaneously dodges the cholesky bug that we identified, and gives us the ability to see if vanilla LAPACK_jll provides any kind of performance improvement over Accelerate's LAPACK (doubtful). An even more interesting test would be to load OpenBLAS first, then load only Accelerate's libBLAS.dylib on top of that, so that we use OpenBLAS's threaded LAPACK routines, and Accelerate's BLAS routines.

  21. ViralBShah commented on Apr 6, 2023

    @ViralBShah
    SponsorMember

    I think the LAPACK in our OpenBLAS probably directly links to its own BLAS and not LBT. I think some work needs to be done to assemble Accelerate BLAS, openblas accelerated lapack routines and the rest of lapack.

    The LAPACK in BB on the other hand does link to LBT. so I think the recipe would be to load accelerate first, then lapack from BB, and finally manually forward a few lapack routines to openblas.

  22. staticfloat commented on Apr 12, 2023

    @staticfloat
    SponsorMember

    With JuliaLinearAlgebra/AppleAccelerate.jl#58 and incorporating this very minor fix to our LinearAlgebra test suite, we actually pass the entire Julia test suite while running on top of Accelerate.

    My contact at Apple has asked for any examples of real-world usage that can be shown to work well on Accelerate; given that the LinearAlgebra test suite finishes with a small performance improvement over the OpenBLAS test suite, I'm inclined to think that small-scale LinearAlgebra problems may have a natural advantage here (as that is the majority of our test suite). They are looking for good examples that they can use to bolster the importance of developing good tools for people like us, so don't be shy!

  23. philipturner commented on Apr 12, 2023

    @philipturner

    For very small problems, which are too small for OpenBLAS to harness multiple cores, Accelerate should excel. It uses the AMX because it's going to help single-core performance. For example, iPhones have only 2 performance cores so the AMX doesn't have any less vector throughput than the NEON units. Unfortunately, this tactic doesn't scale when the amount of NEON compute increases.

  24. ViralBShah commented on May 22, 2023

    @ViralBShah
    SponsorMember

    Support is already on AppleAccelerate.jl master, and will be released when JuliaLinearAlgebra/AppleAccelerate.jl#62 merges.

    Announcement in https://discourse.julialang.org/t/appleaccelerate-jl-v0-4-0/99351/3

  25. ctkelley commented on Sep 26, 2023

    @ctkelley

    Sonoma does better than OS 13.x. Here are some revised lu! numbers

            N      O-64            A-64         R-64        O-32        A-32        R-32 
          256  6.87e-04  1.46e-04  2.13e-01  6.66e-04  1.41e-04  2.11e-01
          512  1.22e-03   7.89e-04  6.48e-01  1.26e-03  6.07e-04  4.81e-01
         1024  3.95e-03  3.28e-03  8.32e-01  3.26e-03  2.45e-03  7.51e-01
         2048  2.05e-02  2.07e-02  1.01e+00  1.38e-02  1.55e-02  1.12e+00
         4096  1.49e-01  1.52e-01  1.02e+00  8.37e-02  8.61e-02  1.03e+00
         8192  1.14e+00  1.27e+00  1.11e+00  6.04e-01  6.52e-01  1.08e+00
    

    The discontinuity at higher dimensions seems to be gone. 8 core M2 Pro.

  26. oscarbg commented on Apr 13, 2024

    @oscarbg

    Unrelated to the issue but can anyone share some m3 perf numbers like in lu for example..

  27. philipturner commented on Apr 14, 2024

    @philipturner

    Generally, one-sided factorizations like LU should be fast in Accelerate. They can use block size 32 on AMX single-core quite easily. So a good fraction of GEMM GFLOPS.

    Two-sided factorizations like tridiagonalization are where Accelerate (and OpenBLAS) are painfully slow. I had to write custom kernels that convert these into panel (QR) factorizations with most computations batched into small GEMM calls. Then the final bulge chasing which still needs further optimization at larger block sizes.

    Perf on M3 should be no different than M1 or M2. The hardware hasn’t changed much fundamentally, except for some new instruction issuing capabilities and lower precisions for AI stuff.

  28. oscarbg commented on Nov 27, 2024

    @oscarbg
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    needs decisionA decision on this change is neededsystem:apple siliconAffects Apple Silicon only (Darwin/ARM64) - e.g. M1 and other M-series chips

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions