Almost 80% of the 3D models generated by TRELLIS had manufacturing defects. With InstructMesh, novices corrected them in about 90% of cases.
3D generators know what a cup should look like, not what it is for: the result is often pretty but unusable once printed. Researchers at MIT's CSAIL, along with Google and Northeastern, propose InstructMesh, a tool that combines Microsoft's TRELLIS generator with GPT-4 to repair these defects. The user highlights an area of the object and describes the problem in natural language, or adjusts a cursor; the modification takes place directly in the latent space of the generative model, without retouching the mesh by hand.
In testing, the team found that almost 80 percent of the models produced by TRELLIS from Thingiverse objects had structural defects. Beginner users were able to identify and correct them in about 90 percent of cases, according to an evaluation by experts. Among the creations: a cup with a dragon's tail as a handle, a drink dispenser in the shape of an octopus or a small vibrating robot in the shape of a shrimp.
The work, led by Faraz Faruqi under the direction of Stefanie Mueller, will be presented in November at the ACM UIST symposium. The team plans to add physics simulations, material recommendations, and an augmented reality version.
Read the full article on the original source
Go to the source →