There are some areas where Apple is prioritizing getting developers on board with their hardware, and the neural engine seems like one of them.
Over the past couple of years, coremltools [1], which is used to convert models from Tensorflow and other frameworks to run on Apple hardware (including the neural engine when available), has gone from a total joke to being quite good.
I had to get a Keras model running on iOS a few months ago, and I was expecting to spend days tracking down obscure errors and writing lots of custom code to get the conversion to work -- but instead it was literally 3 lines of code, and it worked on the first try.
You're earning money with a model deployed on an iOS device? Now that's an achievement. It's even rare to actually get productive models in the first place but then doubling down on less powerful hardware then you could get with aws is just mind-blowing to me in an production context
It's the age old thin client vs. fat client debate repeating itself again. It seems like as the chips & tools get more mature we'll see more and more model deployments on customer hardware. Transmitting gigabytes of sensor/input data to a nearby data center for real time result just isn't feasible for most applications.
There's probably lots of novel applications of AI/ML that remain to be built because of this limitation. Probably also good fodder for backing your way into a startup idea as a technologist.
Suppose you want to do object detection on a phone’s live camera stream. Running your model on aws is probably infeasible, because you’re killing the users data plan while streaming frames into your remote model, and network latency kills the user experience.
On-device detection (“edge ai”) is gaining steam. Apple recently purchased a company called xnor.ai which specialized in optimizing models for low power conditions.
Over the past couple of years, coremltools [1], which is used to convert models from Tensorflow and other frameworks to run on Apple hardware (including the neural engine when available), has gone from a total joke to being quite good.
I had to get a Keras model running on iOS a few months ago, and I was expecting to spend days tracking down obscure errors and writing lots of custom code to get the conversion to work -- but instead it was literally 3 lines of code, and it worked on the first try.
[1] https://github.com/apple/coremltools