Update README.md

3d-generation-pipeline/README.md

@@ -1,38 +1,20 @@
-### TODO
+# Testing the pipeline
-* pidev krabamise heli, kui kahe käega korraga krabada: "(Un)requesting ownership of Cube"
-* FMOD ChannelControl errorid
-* keyboard numbrid ei tööta pärast minigame'i completemist?
-* speech-to-text:
-	* vahepeal lakkab töötamast lih??? tundub et siis kui pikalt tühjust salvestab? ei, vahepeal läheb kohe alguses ka
-	* sööta talle nulle kui on disabled, sest ikka kuidagi tuvastab eelnevat rääkimist
-	* uurida miks buildis Whisper halvemini töötab
-* npc character:
-	* klaas on näha temast eespool
-* shape detection:
-	* peab mängijale kuidagi selgitama, kuidas scale'ida prinditud objekte
-	* prinditud objekti scale'imisele min ja max size limiit
-	* 3d printerile soft particle'id, et ei oleks teravaid ääri seina sisse minnes
-* archery range:
-	* võiks jääda kordama viimast instruktsiooni, kui mängija ei progressi edasi
-	* UFO möödalendamise animation/cutscene alguses kui ütleb "delta attacked by ufos" + panna juba alguses kapsli sisse ufo pöörlema
-* cafe:
-	* võiks saada hinnata saadud tellimust
-	* "order again" nupp
 
-Can't/Won't Do:
+Install required dependency libraries using
-* glTF loading: vahetada ära shader Universal render pipeline Lit, mitte panna buildi kaasa glTf oma - **ei saa, objekt on siis ilma tekstuurita, lihtsalt hall**
+```
+pip install -r requirements.txt
+```
 
+Make a copy of `.env.example`, rename it as `.env`, and insert the necessary config values.
 
-### Notes
+## Python notebooks
-* Functionalities that were implemented using major help from AI:
-	* animating NPC mouth movement based on voice amplitude
-	* adjusting spawned gltf objects' scale
-	* porting InvokeAI client and TRELLIS client to Unity scripts
-	* Getting Whisper stream to work with FMOD instead of Unity default audio
-* TRELLIS: added functionality to specify texture baking optimisation total steps as an argument (`texture_opt_total_steps`), to replace the hardcoded 2500. But this is not tracked in Git (because modified this https://github.com/IgorAherne/trellis-stable-projectorz/releases/tag/latest)
-* Custom Shader Variant Collection to include glTF-pbrMetallicRoughness shader in build
 
-Katse tähelepanekud:
+* [`local_image_generation.ipynb`](notebooks/local_image_generation.ipynb)
-* peab väga proper english accenti tegema
+* [`local_model_generation.ipynb`](notebooks/local_model_generation.ipynb)
-* mõne inimese puhul kuuleb "cat" asemel "cut"
+* [`subprocess_test.ipynb`](notebooks/subprocess_test.ipynb)
-* kui helitugevus oli liiga suur siis cafe waiter kuulis kohviku taustaheli taldriku klirinat kui "piano music"
+
+## Python scripts
+
+* [`generate_image_local.py`](generate_image_local.py)
+* [`generate_model_local.py`](generate_model_local.py)
+* [`start_pipeline.py`](start_pipeline.py)

3d-generation-pipeline/line_rms.txt

@@ -1,49 +0,0 @@
-3.170447962475009263e-05
-2.923115389421582222e-04
-2.510738931596279144e-02
-1.111792679876089096e-02
-6.692767888307571411e-02
-1.006313711404800415e-01
-7.780694961547851562e-02
-5.686730891466140747e-02
-5.614304915070533752e-02
-4.554714635014533997e-02
-4.514135792851448059e-02
-5.479728057980537415e-02
-4.272024706006050110e-02
-3.989587724208831787e-02
-4.298635944724082947e-02
-4.074911773204803467e-02
-2.244980260729789734e-02
-1.105279754847288132e-02
-1.347438804805278778e-02
-1.654553040862083435e-02
-1.846965588629245758e-02
-2.045047841966152191e-02
-1.407719496637582779e-02
-6.578906439244747162e-03
-1.353173051029443741e-02
-1.625132374465465546e-02
-5.863697826862335205e-02
-1.110599413514137268e-01
-9.950184077024459839e-02
-1.184522062540054321e-01
-1.000181213021278381e-01
-6.772108376026153564e-02
-7.621638476848602295e-02
-3.018615581095218658e-02
-9.624224901199340820e-02
-1.259753555059432983e-01
-1.276500672101974487e-01
-1.206035763025283813e-01
-1.011835709214210510e-01
-6.155343726277351379e-02
-3.734333068132400513e-02
-2.485111355781555176e-02
-2.122259326279163361e-02
-1.139380130916833878e-02
-7.472451310604810715e-03
-5.807624198496341705e-03
-1.960268709808588028e-03
-8.761089411564171314e-04
-3.071058890782296658e-04

README.md

@@ -9,16 +9,21 @@ This project is about investigating the feasibility of player-prompted runtime 3
 ## Mini-Games
 
 ### Delta Cafe
+<img src="https://cgvrgit.ulno.net/henrisel/DeltaVR3DModelGeneration/media/branch/master/Doc/screenshots/delta_cafe_screenshot.png" width="800">
 
 ### Archery Range
+<img src="https://cgvrgit.ulno.net/henrisel/DeltaVR3DModelGeneration/media/branch/master/Doc/screenshots/archery_range_screenshot.png" width="800">
 
 ### Shape Detection
+<img src="https://cgvrgit.ulno.net/henrisel/DeltaVR3DModelGeneration/media/branch/master/Doc/screenshots/shape_detection_gameobjects.png" width="800">
 
 Playthrough videos of each mini-game are available on YouTube: https://www.youtube.com/playlist?list=PLqIwPfWyxyaH-YHn64jW59aFhQ5OKf9CW
 
 ## Setup Instructions
 
-These mini-games rely on a speech-to-3D pipeline to function. This pipeline needs to be set up locally and consists of two additional software applications in addition to the Unity game itself: InvokeAI and TRELLIS.
+The built version of the game is available here: https://owncloud.ut.ee/owncloud/s/tXBd4XGpbXTi7Hm.
+
+The build and the editor version both rely on a speech-to-3D pipeline to function. This pipeline needs to be set up locally and consists of two additional software applications in addition to the Unity game itself: InvokeAI and TRELLIS.
 
 ### InvokeAI
 
@@ -43,6 +48,27 @@ In both editor mode and the built version, after first starting the game, the `c
 * `invokeAiModelKey` - the key of the specific model set up in InvokeAI to use. To see the keys of your installed models, send a GET request to `{INVOKEAI_URL}/api/v2/models/?model_type=main` (for example, `http://192.168.0.53:9090/api/v2/models/?model_type=main`)
 * `trellisUrl` - the URL of TRELLIS (for example, `http://192.168.0.53:7960`)
 
+The [3d-generation-pipeline](3d-generation-pipeline/README.md) folder includes Python notebooks and scripts that can be used to test the pipeline working together.
+
+## Unresolved bugs and potential improvements for the future
+* continuous grabbing sound when grabbing with both hands at the same time: "(Un)requesting ownership of Cube"
+* FMOD ChannelControl errors
+* speech-to-text: investigate why Whisper still sometimes stops detecting speech
+* glTF loading: investigate how to use the Universal Render Pipeline Lit shader for the glTF-loaded objects.
+* NPC characters:
+	* glass wall is sometimes incorrectly visible in front of the NPC
+* Delta Cafe:
+	* ability for players to rate the orders
+	* "order again" button
+* Archery Range:
+	* NPC should repeat the last instruction when the player is not progressing
+	* reuse the UFO flying animation from Space Walk in the beginning, when NPC talks about looking out the window and the UFO attack + have the default UFO already in the model display capsule
+* Shape Detection:
+	* explain to the player how to scale printed objects
+	* set min and max scale limits on printed objects
+	* set 3d printer soft particles to avoid sharp edges when particles go inside the printer wall
+
+
 ## Credits
 
 **Samuel Saar**<br/>