INSANE Machine Learning on Neural Engine | M2 Pro/Max

Just got an M1 Max 24c 64gb in Sept 2023 for video, feel like it’s still a good enough machine today.

the fact that you can only use the ANE with metal and swift is super annoying.
I wanna use tensor flow and python :|

asitop not working anymore.\!
any body had any luck?

great dude!

hey, i am student who is studying machine learning, can i buy m2 pro for coding and stuff?

Sorry so regarding the comparaison with PC laptop, is the extra $$$ of the M2 (max) worth it for data science work ?

hey Alex can you make a video on running Gpt4all falcon on m2 mac air

Meanwhile I’m trying to train a huge AI models using 8GB M2 Macbook Air on the GPU and give up afterwards.

I wish I’m rich enough to have 96Gb M1 Mac.

The ANE benchmark results for the M1 Ultra are astonishing. Especially where the M2 Pro was faster. Indeed, according to the specification, the M1 Ultra chip contains twice as many Neural Engine cores - 32, while the rest have only 16. The M1 Ultra was supposed to be faster than any M1 / M2 Max or Pro; in this case, it does not matter that the M2 has a slightly higher clock speed or more GPU cores. However, 32 ANE cores do not always give a performance boost. Very strange.

would be cool to revisit this, with LORA stable diffusion training comparison and other real life application of AI for average devs

Thanks Alex, would also want to see how these fruit machines compare with Nvidia GPU side by side

16GB is hardly sensible for anyone buying a laptop in 2023, let alone someone intending to do data intensive stuff

Very informative! Thank you!

The point about llm's not being able to be run is false. A friend of mine ran a huge vision transformer and a huge BERT model on her M1 Air with 16gb base config. The swap is insane on these machines.

Which sw used for screen recording, plz share.

you should be more accurate with the information, the pc isn't the one reducing the computing power when disconnected, is windows, if you install Linux it will work full power

Lmaoo

Great comparisons, I am planning to get 96gb RAM for my dL models as well.

Wow, Apple-porn, must be profitable.

Hi Alex, thanks for this video ! (and your interesting channel :) )
I vote for a DALL·E alternative to run on your M1 Ultra/MacBooks arsenal to generate funny images for the next video ! :)

interesting the processors never get to 90%+ usage... is that OS, micro-coding, hardware ?

Out of curiosity, why test the ANE when it's only used when developing on iOS? I mean you cannot use the ANE outside of iOS i.e. when you convert models from Tensorflow to CoreML. There is the Metal TensorFlow Plugin available which accelerates model training using the Mac's GPU, but it doesn't give us access to ANE for non CoreML processes. Any new developments on this front? Also, since Tim is talking about Tesla with say 40GB of RAM, how does a Mac with 64GB or 128GB of RAM compare?

Great! Thank you very much.

I'm interested in AUTOMATIC1111-API-SD-image-creation-differences between these devices. And there is also a question: Is there any way to bring stable diffusion into the neural engines?

thanks for you making the amazing video! one question: what's that chart? how do i visualize the cpu & gpu use on my macbook pro 14'' 2023?

3 months later, my mac mini with the Apple M2 Pro with 12‑core CPU, 19-core GPU, 16‑core Neural Engine
32GB unified memory, is coming this week, they say its good for AI stuff, anyone know what I can do with AI with it? in June/July of 2023, thanks

Hi Alex,

Do you think the mac studio m2 max with 30 core gpu would be enough for machine learning?

yougot to run the tests plugged in

More on the ANE please. If you can find a test made for PyTorch/TensorFlow and that also has a version for CoreML and compare Windows vs Mac.

i am play with stable diffusion recently, i first install it on my 1m pro Macbook pro, it works but slow. Now i am just trying to training some LoRAs, does it it working on M1 chips? and is anyone has experience on training stable diffusion models or LoRA on Mac, how it compare to RTX GPS???

At Alex I saw your prior video, I'm on the verge of purchasing a new laptop. I'm not a gamer, but rather an engineer who is focused on learning analyst tools along with ML tools. Currently have MacBook Pro 15 in from 2018, seeking to trade in and buy a new MacBook Pro or build my own PC. Been with Mac for a while, still love apple and MacBooks. Your videos have a been great help on determining which laptop to choose and understand fundamental applications based on specifications of laptop along with end application. Keep it up Alex!

I think you can utilise also tensorflow with ML cores. I tried it on M1 air and did some machine learning where CPU and GPU were basically idle.

hello

I don't understand what are you demonstrating. Practically users apps benefit how of those tests?

Because test sets are good for developpers but users what do they concretely get out of them except that "mine is bigger than yours you see this number, that's the proof". Mine being what you think. Anyway it's intellectual wanking: what counts is that when you use the chips it does the job fast because you have other things to do with it.

Let's say I have a pc eith an intel gpu and an i5 11th gen and 40GB ram and NVME ssd. Turning a 4.5 GB 30 mins video into a 750 MB video takes 30 min on the NVME and 22 min on ramdisk. How fast will it be on a Mac Mini M1 / M2 / M1 Max / M2 Pro?

And what others tasks can be done with the new chips which couldn't before? How better common tasks become using Full HD video, 2K videos, 4K videos? Etc... What happens from the user usage point of view? I buy a machine for what I can do with it, not running benchmarks which tells me nothing in the real. Real being when multitasking for example.

Amazing video! Can you do a comparison video running YOLO? I’m very curious to know how many fps these machines can pull up and it’s a more visual test. Thank you!

Why would people want to run these workload on laptop except for benchmark or for fun? Because it's far more efficient run on desktop or cloud. the RTX3070 on laptop is not the same as RTX3070 on desktop at all.

Hey Alex, It's nice to see that Tensorflow is working well in M2 chip, anyway,where do you live Alex?

One thing YT has taught me is that productivity is only about creating videos.

Thanks.

I'm still deciding between the 14" with 10cores and 32Gb over 12cores 16Gb. Is the SSD such a problem or could i get away with 512GB SSD?

Should I get 14 inch m2 max with 38 core gpu and 32 gb RAM or 30 core GPU and 64 gb of RAM. Does RAM really plays a big role in training?

Nice video Alex.

Thanks - great video and comparison between the different MBP models! Planning to ditch my Intel MBP and (maybe) my Linux RTX-PC. Apple Neural Engine has an insane potential for computation purposes on Apple Silicon macs. Is it really so that Apple Neural Engine does not expose any proper API to be used for generic (python) OSS tech-stack DL training purposes? Only Apple CreateML in training and CoreML inference?

Wow! My MacBook Air M2 maxed out (24Gb RAM), 4-efficiency & 4-performance CPU cores, and 10 GPU cores is faster on ANE and CPU than the M2 Max!

MLMacosPerf (showing ANE)

densenet121_keras_applications
Latency ANE : 0.0011653231354430318
RPS ANE : 858.1310793420579

densenet121_keras_applications
Latency GPU : 0.008527046729112043
RPS GPU : 117.27389702063168

densenet121_keras_applications
Latency CPU : 0.014966152854263783
RPS CPU : 66.81743863888875

Statistics look cool, but can someone explain which end user use cases can benefit from ANEs? e.g. if I own one of these MacBooks, in which scenario would I benefit the most of ANEs? Thanks.

This is the video I need. Thank you!!

Thanks for doing this test. It was quite eye opening. I am getting a M2 14" pro and was wondering about the requirements and how much upgrading the base models memory would help. Looks like I would have to upgrade the memory and go to Max to get a big performance increase. Since I have access to cloud based systems with GPUs and TPUs I think I will just go with the base system.

Thank you Peter and Alex for the video. Could you do this with pytorch as well ?

What I wonder is if the high RAM M2 Maxes (like 64GB or 96GB) can train significantly more complex models or use significantly bigger batches simply because they have more ram than most discrete GPUs.

Great video! Never thought ANE would be that powerful, Thank you for sharing your expertise and providing such valuable content. Keep up the good work!

Accoustic Windscreen (Whats ver that is) ... lol that killed me 🤣

Lovely video as always!
I've got a question, which Macbook would you consider 'worth the money' for a Data scientist and/or a ML/AI engineer ? Obviously it depends on the work one does, but it seems some tasks require terabytes of Ram, not GB's and so upgrading to 96 won't cut it anyway. On the other hand, going to low will enforce one to always use cloud services. At this point, I've tried a M2 macbook pro 16" base model (16 Gb ram), and I've run out of ram computing scattering transforms on a relatively small (2Gb) dataset. So the choice for me must be in the range of 32-96 I suppose.