forked from cgvr/DeltaVR
73 lines
3.7 KiB
Markdown
73 lines
3.7 KiB
Markdown
|
|
|
|
# DeltaVR: Player-Prompted 3D Model Generation Game Mechanics
|
|
|
|
This is a fork of the [main DeltaVR project](https://cgvrgit.ulno.net/cgvr/DeltaVR). To learn more about DeltaVR itself, look at the main repository's README.
|
|
|
|
This project is about investigating the feasibility of player-prompted runtime 3D model generation game mechanics. These are game mechanics that allow the player to generate arbitrary 3D objects using natural language speech. This project includes three mini-games that use this speech-to-3D system as a core game mechanic.
|
|
|
|
## Mini-Games
|
|
|
|
### Delta Cafe
|
|
|
|
### Archery Range
|
|
|
|
### Shape Detection
|
|
|
|
Playthrough videos of each mini-game are available on YouTube: https://www.youtube.com/playlist?list=PLqIwPfWyxyaH-YHn64jW59aFhQ5OKf9CW
|
|
|
|
## Setup Instructions
|
|
|
|
These mini-games rely on a speech-to-3D pipeline to function. This pipeline needs to be set up locally and consists of two additional software applications in addition to the Unity game itself: InvokeAI and TRELLIS.
|
|
|
|
### InvokeAI
|
|
|
|
InvokeAI is used to generate images from text prompts. To install InvokeAI, follow the instructions in the official guide: https://invoke.ai/start-here/installation.
|
|
|
|
After installing it, an AI model needs to be set up in InvokeAI. This project has been tested with Stable Diffusion XL, but other models can also be used. Documentation on setting up models: https://invoke.ai/concepts/models.
|
|
|
|
Finally, to enable communication with InvokeAI across different computers, enable the Server Mode in InvokeAI settings.
|
|
|
|
### TRELLIS
|
|
|
|
TRELLIS is used to generate 3D models based on images. The simplest way to install TRELLIS on Windows is via the one-click installer provided in this fork of the original TRELLIS repo: https://github.com/IgorAherne/trellis-stable-projectorz. The download package includes instructions on how to launch TRELLIS as a local API server. For the default configuration, use the following command:
|
|
```
|
|
.\run.bat --ip 0.0.0.0
|
|
```
|
|
|
|
### Unity
|
|
|
|
In both editor mode and the built version, after first starting the game, the `config.json` file is created in the game's root directory. This file is used to persist configurable values between play sessions. The following values need to be set correctly for the game to be able to communicate with InvokeAI and TRELLIS:
|
|
|
|
* `invokeAiUrl` - the URL of InvokeAI (for example `http://192.168.0.53:9090`)
|
|
* `invokeAiModelKey` - the key of the specific model set up in InvokeAI to use. To see the keys of your installed models, send a GET request to `{INVOKEAI_URL}/api/v2/models/?model_type=main` (for example, `http://192.168.0.53:9090/api/v2/models/?model_type=main`)
|
|
* `trellisUrl` - the URL of TRELLIS (for example, `http://192.168.0.53:7960`)
|
|
|
|
## Credits
|
|
|
|
**Samuel Saar**<br/>
|
|
Creation of the following 3D models used in the mini-games:
|
|
* Delta Cafe: pen, notepad, serving tray
|
|
* Archery Range: capsule display, microphone stand, wires, table, screens
|
|
* Shape Detection: computer printer, 3D printer, radio transmitter, shape detector
|
|
|
|
**Timur Nizamov**<br/>
|
|
Sound effects and integrating voicelines into Unity
|
|
|
|
**Madis Vasser**<br/>
|
|
Voice acting for the Ufo and Professor characters
|
|
|
|
**Ulrich Norbisrath**<br/>
|
|
Voice acting for the Robert character
|
|
|
|
**Ranno Samuel Adson**<br/>
|
|
Technical help and tips for DeltaVR
|
|
|
|
(These credits are specifically for the work of this fork. Credits for the main DeltaVR project are shown in the main project's README.)
|
|
|
|
Developed in the [Computer Graphics and Virtual Reality Study Lab](https://cgvr.cs.ut.ee/) of the [Institute of Computer Science, University of Tartu](https://cs.ut.ee).
|
|
|
|
-----
|
|
|
|
|