Presented at the end of September, OpenAI's Quick Decision API, built on ChatGPT Luna, is now available for all customers, with response times reduced to 100-180 milliseconds.
OpenAI has made its Decisions API available to all of its customers. Announced at its developer day on September 29, it targets the niche opened up by Jev, the model of the start-up TypeSafe: models that, faced with a list of options, decide very quickly and indicate their degree of confidence, instead of producing text. Rather than training a dedicated model, the team of Nikunj Handa, product manager of APIs, took ChatGPT Luna and reconfigured it to speed up inference and lower its cost. The announced price is $0.10 per million input tokens, with no output billing.
From several seconds to a handful of milliseconds
According to OpenAI, response times increase from one to three seconds to 100-180 milliseconds. The public version adds voice and visual input and distributes inference geographically to reduce latency. Early adopters include Box, Datadog, Ramp, and Speak; the latter says it has reduced the delay between actions from 1.4 seconds to 0.13 seconds.
Competition is fierce for these "system 1" models: public rankings (AIM System One Index, Artificial Analysis Decision Index) place Liquid's D1 Ultra and Base Lab's Elephant-1 at the top, with Jev ranking seventh and third, and OpenAI's model eighth and fourteenth. OpenAI relies on its multimodal advantage, particularly for classifying images.
Read the full article on the original source
Go to the source →