forked from cgvr/DeltaVR
update main README
This commit is contained in:
@@ -13,7 +13,6 @@
|
||||
* prinditud objekti scale'imisele min ja max size limiit
|
||||
* 3d printerile soft particle'id, et ei oleks teravaid ääri seina sisse minnes
|
||||
* archery range:
|
||||
* archery targetite rework, et buildis ka töötaks
|
||||
* võiks jääda kordama viimast instruktsiooni, kui mängija ei progressi edasi
|
||||
* UFO möödalendamise animation/cutscene alguses kui ütleb "delta attacked by ufos" + panna juba alguses kapsli sisse ufo pöörlema
|
||||
* cafe:
|
||||
|
||||
@@ -14,7 +14,7 @@
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 5,
|
||||
"execution_count": 4,
|
||||
"id": "77b23cd8",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
@@ -22,29 +22,16 @@
|
||||
"# API endpoint\n",
|
||||
"BASE_URL = \"http://127.0.0.1:7960\"\n",
|
||||
"\n",
|
||||
"def generate_no_preview(image_base64: str):\n",
|
||||
"def generate_no_preview(data: dict):\n",
|
||||
" \"\"\"Generate 3D model from a single base64-encoded image without previews.\n",
|
||||
" \n",
|
||||
" Args:\n",
|
||||
" image_base64: Base64 string of the image (without 'data:image/...' prefix)\n",
|
||||
" \"\"\"\n",
|
||||
" try:\n",
|
||||
" # Set generation parameters\n",
|
||||
" params = {\n",
|
||||
" 'image_base64': image_base64,\n",
|
||||
" 'seed': 42,\n",
|
||||
" 'ss_guidance_strength': 7.5,\n",
|
||||
" 'ss_sampling_steps': 30,\n",
|
||||
" 'slat_guidance_strength': 7.5,\n",
|
||||
" 'slat_sampling_steps': 30,\n",
|
||||
" 'mesh_simplify_ratio': 0.95,\n",
|
||||
" 'texture_size': 1024,\n",
|
||||
" 'output_format': 'glb'\n",
|
||||
" }\n",
|
||||
" \n",
|
||||
" # Start generation\n",
|
||||
" print(\"Starting generation...\")\n",
|
||||
" response = requests.post(f\"{BASE_URL}/generate_no_preview\", data=params)\n",
|
||||
" response = requests.post(f\"{BASE_URL}/generate_no_preview\", data)\n",
|
||||
" print(\"Response status:\", response.status_code)\n",
|
||||
" response.raise_for_status()\n",
|
||||
" \n",
|
||||
@@ -75,7 +62,7 @@
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 6,
|
||||
"execution_count": 11,
|
||||
"id": "eb122295",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
@@ -85,7 +72,19 @@
|
||||
" image_data = image_file.read()\n",
|
||||
"\n",
|
||||
" base64_encoded = base64.b64encode(image_data).decode('utf-8')\n",
|
||||
" model = generate_no_preview(base64_encoded)\n",
|
||||
" # Set generation parameters\n",
|
||||
" data = {\n",
|
||||
" 'image_base64': base64_encoded,\n",
|
||||
" 'seed': 42,\n",
|
||||
" 'ss_guidance_strength': 8,\n",
|
||||
" 'ss_sampling_steps': 10,\n",
|
||||
" 'slat_guidance_strength': 6,\n",
|
||||
" 'slat_sampling_steps': 10,\n",
|
||||
" 'mesh_simplify_ratio': 0.9958,\n",
|
||||
" 'texture_size': 1024,\n",
|
||||
" 'output_format': 'glb'\n",
|
||||
" }\n",
|
||||
" model = generate_no_preview(data)\n",
|
||||
" \n",
|
||||
" with open(output_path, 'wb') as f:\n",
|
||||
" f.write(model)\n",
|
||||
@@ -94,7 +93,7 @@
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 9,
|
||||
"execution_count": 12,
|
||||
"id": "2ce7dfdf",
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
@@ -107,14 +106,14 @@
|
||||
"Progress: 100%\n",
|
||||
"Downloading model...\n",
|
||||
"Model downloaded.\n",
|
||||
"Model saved to test_resources/style_test_3_model.glb\n"
|
||||
"Model saved to test_resources/mesh_simplify_ratio_0.998.glb\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"\n",
|
||||
"image_path = 'test_resources/style_test_3.jpg'\n",
|
||||
"output_path = \"test_resources/style_test_3_model.glb\"\n",
|
||||
"image_path = 'test_resources/512x512.png'\n",
|
||||
"output_path = \"test_resources/mesh_simplify_ratio_0.998.glb\"\n",
|
||||
"\n",
|
||||
"generate_model(image_path, output_path)"
|
||||
]
|
||||
@@ -130,7 +129,7 @@
|
||||
],
|
||||
"metadata": {
|
||||
"kernelspec": {
|
||||
"display_name": ".venv",
|
||||
"display_name": ".venv (3.10.11)",
|
||||
"language": "python",
|
||||
"name": "python3"
|
||||
},
|
||||
|
||||
@@ -1,85 +1,71 @@
|
||||
|
||||
|
||||
DeltaVR is a virtual reality experience set in the [Delta Centre](https://delta.ut.ee/) of the [University of Tartu](https://ut.ee/). The virtual Delta Building includes several interactive scenes that demonstrate the teaching and research actively done in the Delta Centre.
|
||||
# DeltaVR: Player-Prompted 3D Model Generation Game Mechanics
|
||||
|
||||
The application works on PCVR, Meta Quest 2 and 3, HTC Vive, and regular Windows PC platforms. There is cross-platform multiplayer functionality that enables several users to be in the same virtual environment from both VR and PC platforms.
|
||||
This is a fork of the [main DeltaVR project](https://cgvrgit.ulno.net/cgvr/DeltaVR). To learn more about DeltaVR itself, look at the main repository's README.
|
||||
|
||||
This project is about investigating the feasibility of player-prompted runtime 3D model generation game mechanics. These are game mechanics that allow the player to generate arbitrary 3D objects using natural language speech. This project includes three mini-games that use this speech-to-3D system as a core game mechanic.
|
||||
|
||||
## Features
|
||||
## Mini-Games
|
||||
|
||||
### Exploration
|
||||
### Delta Cafe
|
||||
|
||||
DeltaVR features the first two floors of the Delta Educational Building for **exploration and discovery**. There are many diegetic elements representing the studies and research conducted at the Delta Building, such as robotics, the high-performance computing server room, video game development, and student life.
|
||||
### Archery Range
|
||||
|
||||
|
||||
### Shape Detection
|
||||
|
||||
### UFO Bow Game
|
||||
Playthrough videos of each mini-game are available on YouTube: https://www.youtube.com/playlist?list=PLqIwPfWyxyaH-YHn64jW59aFhQ5OKf9CW
|
||||
|
||||
At the terrace on the second floor, Delta explorers can defend the building from UFO-s using a bow and **achieve high scores**.
|
||||
## Setup Instructions
|
||||
|
||||
|
||||
These mini-games rely on a speech-to-3D pipeline to function. This pipeline needs to be set up locally and consists of two additional software applications in addition to the Unity game itself: InvokeAI and TRELLIS.
|
||||
|
||||
### Bolt Self-Driving Car
|
||||
### InvokeAI
|
||||
|
||||
The courtyard between the Educational and Entrepreneurial buildings of the Delta Centre, the explorers can see the Bolt Self-Driving Car. This car is developed by the [[http://adl.cs.ut.ee/|Autonomous Driving Lab]] of the [[https://cs.ut.ee|Institute of Computer Science]]. If one is brave enough, they can stop the car and catch a ride, simulating both the **feeling of being in a self-driving vehicle** as well as VR motion sickness.
|
||||
InvokeAI is used to generate images from text prompts. To install InvokeAI, follow the instructions in the official guide: https://invoke.ai/start-here/installation.
|
||||
|
||||
|
||||
After installing it, an AI model needs to be set up in InvokeAI. This project has been tested with Stable Diffusion XL, but other models can also be used. Documentation on setting up models: https://invoke.ai/concepts/models.
|
||||
|
||||
### Space Walk
|
||||
Finally, to enable communication with InvokeAI across different computers, enable the Server Mode in InvokeAI settings.
|
||||
|
||||
Where the actual Delta building has a set of skywalks connecting it with the entrepreneurship building, DeltaVR has a set of portals leading to the Space walk experience. In it, one can move in the **vastness of space** and experience **changes in gravity**. A fleet of UFO ships react to one's presence and come to investigate the arrival.
|
||||
### TRELLIS
|
||||
|
||||
|
||||
TRELLIS is used to generate 3D models based on images. The simplest way to install TRELLIS on Windows is via the one-click installer provided in this fork of the original TRELLIS repo: https://github.com/IgorAherne/trellis-stable-projectorz. The download package includes instructions on how to launch TRELLIS as a local API server. For the default configuration, use the following command:
|
||||
```
|
||||
.\run.bat --ip 0.0.0.0
|
||||
```
|
||||
|
||||
### Server Room
|
||||
### Unity
|
||||
|
||||
On the second floor, one can hear the humming of the servers. Should they investigate, they will find a room of server racks and a large red button. Should they push the button, they will trigger the **fire alarm** and have the server room fill with harmful invisible gas. This propms the player to escape the room. This largerly **auditory experience** is noted to be engaging and immersive. It represents the work of UT HPC in maintaining the servers of the University of Tartu.
|
||||
In both editor mode and the built version, after first starting the game, the `config.json` file is created in the game's root directory. This file is used to persist configurable values between play sessions. The following values need to be set correctly for the game to be able to communicate with InvokeAI and TRELLIS:
|
||||
|
||||
|
||||
|
||||
### Interactive Map
|
||||
|
||||
To navigate the two floors of the large Delta Educational Building, explorers have an interactive map. This provides a clear overview of where they currently are and what other interactions are located across the building. Explorers can teleport to a **select interactive experiences**, while others are left for them to discover based on the hints on the map.
|
||||
|
||||
|
||||
|
||||
### Whiteboard
|
||||
|
||||
In the virtual Computer Graphics and Virtual Reality Study Lab, explorers can use spray paint cans to draw on a whiteboard. Surprisingly, this is one of the **more popular interactive experiences** of DeltaVR.
|
||||
|
||||
|
||||
* `invokeAiUrl` - the URL of InvokeAI (for example `http://192.168.0.53:9090`)
|
||||
* `invokeAiModelKey` - the key of the specific model set up in InvokeAI to use. To see the keys of your installed models, send a GET request to `{INVOKEAI_URL}/api/v2/models/?model_type=main` (for example, `http://192.168.0.53:9090/api/v2/models/?model_type=main`)
|
||||
* `trellisUrl` - the URL of TRELLIS (for example, `http://192.168.0.53:7960`)
|
||||
|
||||
## Credits
|
||||
|
||||
**Ranno Samuel Adson**<br/>
|
||||
User experience design. Additional interactions. Interaction improvements.
|
||||
|
||||
**Toomas Tamm**<br/>
|
||||
Project architecture, model optimization, lighting. [Bachelor's Thesis](https://comserv.cs.ut.ee/ati_thesis/datasheet.php?id=71682) ([poster](https://courses.cs.ut.ee/student_projects/download/478.pdf)), [Master's Thesis](https://comserv.cs.ut.ee/ati_thesis/datasheet.php?id=77065&language=en).
|
||||
|
||||
**Joonas Püks**<br/>
|
||||
Multiplayer and cross-play functionality. [Bachelor's Thesis](https://comserv.cs.ut.ee/ati_thesis/datasheet.php?id=74390) ([poster](https://courses.cs.ut.ee/student_projects/download/534.pdf)).
|
||||
**Samuel Saar**<br/>
|
||||
Creation of the following 3D models used in the mini-games:
|
||||
* Delta Cafe: pen, notepad, serving tray
|
||||
* Archery Range: capsule display, microphone stand, wires, table, screens
|
||||
* Shape Detection: computer printer, 3D printer, radio transmitter, shape detector
|
||||
|
||||
**Timur Nizamov**<br/>
|
||||
Technical sound design.
|
||||
Sound effects and integrating voicelines into Unity
|
||||
|
||||
**Raimond Tunnel**<br/>
|
||||
Project management, visual design.
|
||||
**Madis Vasser**<br/>
|
||||
Voice acting for the Ufo and Professor characters
|
||||
|
||||
Developed in the [Computer Graphcis and Virtual Reality Study Lab](https://cgvr.cs.ut.ee/) of the [Institute of Computer Science, University of Tartu](https://cs.ut.ee).
|
||||
**Ulrich Norbisrath**<br/>
|
||||
Voice acting for the Robert character
|
||||
|
||||
### Used Attributions
|
||||
**Ranno Samuel Adson**<br/>
|
||||
Technical help and tips for DeltaVR
|
||||
|
||||
| Description | License | Source | Author |
|
||||
|-----------------------------------------------------|----------------------------------------------|---------------------------------------------------------------------------------------------|------------------|
|
||||
| Server rack model | Royalty Free, No AI License | [Link](https://www.cgtrader.com/free-3d-models/electronics/computer/simple-server-model) | anymelok |
|
||||
| Robot movement sound | Creative Commons 0 | [Link](https://freesound.org/people/Brazilio123/sounds/661435/) | Brazilio123 |
|
||||
| Spacewalk UFO sound | Attribution NonCommercial 4.0 | [Link](https://freesound.org/people/Speedenza/sounds/209366/) | Speedenza |
|
||||
| Keyboard icons | Creative Commons Attribution-NoDerivs 3.0 | [Link](https://icons8.com/) | icons8 |
|
||||
(These credits are specifically for the work of this fork. Credits for the main DeltaVR project are shown in the main project's README.)
|
||||
|
||||
-----
|
||||
|
||||
DeltaVR was moved to this repository in 2025. The previous repository is available here: [[https://gitlab.com/UT-CGVR/deltavr]]
|
||||
Developed in the [Computer Graphics and Virtual Reality Study Lab](https://cgvr.cs.ut.ee/) of the [Institute of Computer Science, University of Tartu](https://cs.ut.ee).
|
||||
|
||||
-----
|
||||
|
||||
|
||||
Reference in New Issue
Block a user