Where do you host your NNs?
We've trained a few neural networks for our project and some of them are surprisingly fat (e.g. fine-tuned BERT). We have a beefy physical machine with a GPU for dev purposes and they run fine there, but when we put on our production (t2.medium/large/xlarge) instances on AWS, they become terribly slow. Buying GPU from AWS or TPU from GCP is, however, insanely expensive.
Question: do you know of any other hosting options where we can rent a GTX 1080 ti for a reasonable price?
#tech-questions
Hetzner, definitely. GTX 1080 for 94 euros per month with 64GB of RAM. https://www.hetzner.de/dedicated-rootserver/matrix-ex
I'd like to move there, but my project's traffic does not justify it yet. Hope the demand will make me move there soon.
I've heard of Hetzner too, they have a pretty sweet deal on those 1080s, but the last time I checked they were all out of stock :|
I've been fleshing out a small business related to deep learning desktop computers. One thing that came up was the NVIDIA license: you are forbidden to use GTX or RTX cards for 'datacenter use' (except crypto mining). There are a lot of angry and upset people.
Its a way to segment their customers and force you to pay more when you go to the cloud. There is no TOS-compliant way to rent a GTX 1080 for cloud purposes, period.
Oh wow, didn't know about that. Artificial restrictions are never good.
Basically, the thing is that we don't need that much power 100% of time, we just need our NNs to be online whenever our users make queries. This means 99.999% uptime, but like 1% GPU consumption. Buying a whole GPU from a cloud provider makes a lot of sense for training, when you need all of for a short time, but for running NNs some kind of shared environment (like good old virtual hosting) would make much more sense.
I'm currently looking at https://cloud.google.com/ml-engine/docs/tensorflow/prediction-overview . If I understand it correctly, they offer what we want but It's hard to tell how much will it actually cost and what would be the performance without actually trying it out.
Sounds like lambda, but for ML. You could run a performance test and track expenses. Could probably do the test for $30-$300 and be able to predict costs afterwards.
Will you use a lot of bandwidth? My last performance testing project cost as much in bandwidth as in compute haha (image servers though)
Yeah, we're working with text so bandwidth should spending should be minimal.
Definitely not my wheelhouse at all, but I know Linode recently announced GPU systems, may be worth a look!