1
0
forked from cgvr/DeltaVR

18 Commits

25 changed files with 261 additions and 586 deletions
+17 -30
View File
@@ -1,33 +1,20 @@
### TODO # Testing the pipeline
* FMOD ChannelControl errorid
* speech-to-text:
* uurida miks buildis Whisper halvemini töötab
* proovida suuremat Whisperi mudelit, äkki töötab mürases keskkonnas paremini
* npc character:
* klaas on näha temast eespool
* shape detection:
* peab mängijale kuidagi selgitama, kuidas scale'ida prinditud objekte
* prinditud objekti scale'imisele min ja max size limiit
* 3d printerile soft particle'id, et ei clipiks seina sisse
* archery range:
* archery targettide rework, et buildis ka töötaks
* võiks jääda kordama viimast instruktsiooni, kui mängija ei progressi edasi
* UFO möödalendamise animation/cutscene alguses kui ütleb "delta attacked by ufos" + panna juba alguses kapsli sisse ufo pöörlema
* cafe:
* audio detection sama heaks nagu ülejäänud 2 mängul!
* continuous krabamise heli miskipärast
* võiks saada hinnata saadud tellimust
* "order again" nupp
Can't/Won't Do: Install required dependency libraries using
* glTF loading: vahetada ära shader Universal render pipeline Lit, mitte panna buildi kaasa glTf oma - **ei saa, objekt on siis ilma tekstuurita, lihtsalt hall** ```
pip install -r requirements.txt
```
Make a copy of `.env.example`, rename it as `.env`, and insert the necessary config values.
### Notes ## Python notebooks
* Functionalities that were implemented using major help from AI:
* animating NPC mouth movement based on voice amplitude * [`local_image_generation.ipynb`](notebooks/local_image_generation.ipynb)
* adjusting spawned gltf objects' scale * [`local_model_generation.ipynb`](notebooks/local_model_generation.ipynb)
* porting InvokeAI client and TRELLIS client to Unity scripts * [`subprocess_test.ipynb`](notebooks/subprocess_test.ipynb)
* Getting Whisper stream to work with FMOD instead of Unity default audio
* TRELLIS: added functionality to specify texture baking optimisation total steps as an argument (`texture_opt_total_steps`), to replace the hardcoded 2500. But this is not tracked in Git (because modified this https://github.com/IgorAherne/trellis-stable-projectorz/releases/tag/latest) ## Python scripts
* Custom Shader Variant Collection to include glTF-pbrMetallicRoughness shader in build
* [`generate_image_local.py`](generate_image_local.py)
* [`generate_model_local.py`](generate_model_local.py)
* [`start_pipeline.py`](start_pipeline.py)
-49
View File
@@ -1,49 +0,0 @@
3.170447962475009263e-05
2.923115389421582222e-04
2.510738931596279144e-02
1.111792679876089096e-02
6.692767888307571411e-02
1.006313711404800415e-01
7.780694961547851562e-02
5.686730891466140747e-02
5.614304915070533752e-02
4.554714635014533997e-02
4.514135792851448059e-02
5.479728057980537415e-02
4.272024706006050110e-02
3.989587724208831787e-02
4.298635944724082947e-02
4.074911773204803467e-02
2.244980260729789734e-02
1.105279754847288132e-02
1.347438804805278778e-02
1.654553040862083435e-02
1.846965588629245758e-02
2.045047841966152191e-02
1.407719496637582779e-02
6.578906439244747162e-03
1.353173051029443741e-02
1.625132374465465546e-02
5.863697826862335205e-02
1.110599413514137268e-01
9.950184077024459839e-02
1.184522062540054321e-01
1.000181213021278381e-01
6.772108376026153564e-02
7.621638476848602295e-02
3.018615581095218658e-02
9.624224901199340820e-02
1.259753555059432983e-01
1.276500672101974487e-01
1.206035763025283813e-01
1.011835709214210510e-01
6.155343726277351379e-02
3.734333068132400513e-02
2.485111355781555176e-02
2.122259326279163361e-02
1.139380130916833878e-02
7.472451310604810715e-03
5.807624198496341705e-03
1.960268709808588028e-03
8.761089411564171314e-04
3.071058890782296658e-04
@@ -14,7 +14,7 @@
}, },
{ {
"cell_type": "code", "cell_type": "code",
"execution_count": 5, "execution_count": 4,
"id": "77b23cd8", "id": "77b23cd8",
"metadata": {}, "metadata": {},
"outputs": [], "outputs": [],
@@ -22,29 +22,16 @@
"# API endpoint\n", "# API endpoint\n",
"BASE_URL = \"http://127.0.0.1:7960\"\n", "BASE_URL = \"http://127.0.0.1:7960\"\n",
"\n", "\n",
"def generate_no_preview(image_base64: str):\n", "def generate_no_preview(data: dict):\n",
" \"\"\"Generate 3D model from a single base64-encoded image without previews.\n", " \"\"\"Generate 3D model from a single base64-encoded image without previews.\n",
" \n", " \n",
" Args:\n", " Args:\n",
" image_base64: Base64 string of the image (without 'data:image/...' prefix)\n", " image_base64: Base64 string of the image (without 'data:image/...' prefix)\n",
" \"\"\"\n", " \"\"\"\n",
" try:\n", " try:\n",
" # Set generation parameters\n",
" params = {\n",
" 'image_base64': image_base64,\n",
" 'seed': 42,\n",
" 'ss_guidance_strength': 7.5,\n",
" 'ss_sampling_steps': 30,\n",
" 'slat_guidance_strength': 7.5,\n",
" 'slat_sampling_steps': 30,\n",
" 'mesh_simplify_ratio': 0.95,\n",
" 'texture_size': 1024,\n",
" 'output_format': 'glb'\n",
" }\n",
" \n",
" # Start generation\n", " # Start generation\n",
" print(\"Starting generation...\")\n", " print(\"Starting generation...\")\n",
" response = requests.post(f\"{BASE_URL}/generate_no_preview\", data=params)\n", " response = requests.post(f\"{BASE_URL}/generate_no_preview\", data)\n",
" print(\"Response status:\", response.status_code)\n", " print(\"Response status:\", response.status_code)\n",
" response.raise_for_status()\n", " response.raise_for_status()\n",
" \n", " \n",
@@ -75,7 +62,7 @@
}, },
{ {
"cell_type": "code", "cell_type": "code",
"execution_count": 6, "execution_count": 11,
"id": "eb122295", "id": "eb122295",
"metadata": {}, "metadata": {},
"outputs": [], "outputs": [],
@@ -85,7 +72,19 @@
" image_data = image_file.read()\n", " image_data = image_file.read()\n",
"\n", "\n",
" base64_encoded = base64.b64encode(image_data).decode('utf-8')\n", " base64_encoded = base64.b64encode(image_data).decode('utf-8')\n",
" model = generate_no_preview(base64_encoded)\n", " # Set generation parameters\n",
" data = {\n",
" 'image_base64': base64_encoded,\n",
" 'seed': 42,\n",
" 'ss_guidance_strength': 8,\n",
" 'ss_sampling_steps': 10,\n",
" 'slat_guidance_strength': 6,\n",
" 'slat_sampling_steps': 10,\n",
" 'mesh_simplify_ratio': 0.9958,\n",
" 'texture_size': 1024,\n",
" 'output_format': 'glb'\n",
" }\n",
" model = generate_no_preview(data)\n",
" \n", " \n",
" with open(output_path, 'wb') as f:\n", " with open(output_path, 'wb') as f:\n",
" f.write(model)\n", " f.write(model)\n",
@@ -94,7 +93,7 @@
}, },
{ {
"cell_type": "code", "cell_type": "code",
"execution_count": 9, "execution_count": 12,
"id": "2ce7dfdf", "id": "2ce7dfdf",
"metadata": {}, "metadata": {},
"outputs": [ "outputs": [
@@ -107,14 +106,14 @@
"Progress: 100%\n", "Progress: 100%\n",
"Downloading model...\n", "Downloading model...\n",
"Model downloaded.\n", "Model downloaded.\n",
"Model saved to test_resources/style_test_3_model.glb\n" "Model saved to test_resources/mesh_simplify_ratio_0.998.glb\n"
] ]
} }
], ],
"source": [ "source": [
"\n", "\n",
"image_path = 'test_resources/style_test_3.jpg'\n", "image_path = 'test_resources/512x512.png'\n",
"output_path = \"test_resources/style_test_3_model.glb\"\n", "output_path = \"test_resources/mesh_simplify_ratio_0.998.glb\"\n",
"\n", "\n",
"generate_model(image_path, output_path)" "generate_model(image_path, output_path)"
] ]
@@ -130,7 +129,7 @@
], ],
"metadata": { "metadata": {
"kernelspec": { "kernelspec": {
"display_name": ".venv", "display_name": ".venv (3.10.11)",
"language": "python", "language": "python",
"name": "python3" "name": "python3"
}, },
Binary file not shown.
@@ -0,0 +1,7 @@
fileFormatVersion: 2
guid: 4fef61b9ad4f7eb488ea2657f1cc700e
DefaultImporter:
externalObjects: {}
userData:
assetBundleName:
assetBundleVariant:
Binary file not shown.
@@ -0,0 +1,7 @@
fileFormatVersion: 2
guid: 6f5a0c98d09a6dc4d9d570e23dc96ab9
DefaultImporter:
externalObjects: {}
userData:
assetBundleName:
assetBundleVariant:
@@ -87,8 +87,6 @@ public class ArcheryRange : NetworkBehaviour
if (exitedPlayer == _scoredPlayer) keyboardManager.DeActivate(); // If the player ´refuses to enter their name. if (exitedPlayer == _scoredPlayer) keyboardManager.DeActivate(); // If the player ´refuses to enter their name.
} }
public override void OnStopServer() public override void OnStopServer()
{ {
base.OnStopServer(); base.OnStopServer();
@@ -133,18 +131,18 @@ public class ArcheryRange : NetworkBehaviour
private ArcheryTarget SpawnTarget(Vector3 randomPos) private ArcheryTarget SpawnTarget(Vector3 randomPos)
{ {
GameObject targetObject; // Spawn default UFO
if (modelDisplay.Model == null) GameObject targetObject = Instantiate(targetPrefab, randomPos, targetStartPosition.rotation, null);
if (modelDisplay.Model != null)
{ {
// spawn default UFO // Spawn generated object directly as default UFO's child
targetObject = Instantiate(targetPrefab, randomPos, Quaternion.identity, null); GameObject childObject = Instantiate(modelDisplay.Model);
} else childObject.transform.SetParent(targetObject.transform, false);
{ childObject.transform.localScale *= targetSizeIncrease;
// spawn generated model
targetObject = Instantiate(modelDisplay.Model, randomPos, Quaternion.identity, null); // Disable UFO's default body
InitializeArcheryTargetObject(targetObject); targetObject.transform.Find("Body").gameObject.SetActive(false);
} }
targetObject.transform.rotation = targetStartPosition.rotation;
ArcheryTarget target = targetObject.GetComponent<ArcheryTarget>(); ArcheryTarget target = targetObject.GetComponent<ArcheryTarget>();
target.endPosition = targetEndPosition.position; target.endPosition = targetEndPosition.position;
@@ -153,22 +151,6 @@ public class ArcheryRange : NetworkBehaviour
return target; return target;
} }
private void InitializeArcheryTargetObject(GameObject targetObject)
{
ArcheryTarget archeryTarget = targetObject.AddComponent<ArcheryTarget>();
archeryTarget.pointsText = archeryTargetPointsText;
Rigidbody rigidbody = targetObject.AddComponent<Rigidbody>();
rigidbody.useGravity = false;
rigidbody.isKinematic = true;
targetObject.AddComponent<NetworkObject>();
targetObject.AddComponent<NetworkTransform>();
targetObject.transform.localScale *= targetSizeIncrease;
}
public void ResetRange() public void ResetRange()
{ {
if (!IsServer) return; if (!IsServer) return;
@@ -168,20 +168,14 @@ public class TutorialAudioListener : MonoBehaviour
private EventReference GetGrabEvent(GrabSoundType type) private EventReference GetGrabEvent(GrabSoundType type)
{ {
switch (type) return type switch
{ {
case GrabSoundType.Bow: GrabSoundType.Bow => FMODEvents.Instance.BowGrab,
return FMODEvents.Instance.BowGrab; GrabSoundType.Sprayer => FMODEvents.Instance.SprayerGrab,
case GrabSoundType.Sprayer:
return FMODEvents.Instance.SprayerGrab;
//Add more objects for grabbing here and do not forget to define them in FMODEvents.cs //Add more objects for grabbing here and do not forget to define them in FMODEvents.cs
//Add the GrabAudioProfile.cs component to an object instance //Add the GrabAudioProfile.cs component to an object instance
_ => FMODEvents.Instance.DefaultGrab,
default: };
return FMODEvents.Instance.DefaultGrab;
}
} }
private void HandleGrab(XRGrabInteractable grab) private void HandleGrab(XRGrabInteractable grab)
@@ -195,10 +189,6 @@ public class TutorialAudioListener : MonoBehaviour
: GrabSoundType.Default; : GrabSoundType.Default;
EventReference grabEvent = GetGrabEvent(type); EventReference grabEvent = GetGrabEvent(type);
Debug.Log(grabEvent);
AudioManager.Instance.PlayAttachedInstance(grabEvent, grab.gameObject); AudioManager.Instance.PlayAttachedInstance(grabEvent, grab.gameObject);
} }
} }
@@ -31,6 +31,7 @@ namespace _PROJECT.NewHandPresence
private XRControllerHintController _leftHintController; private XRControllerHintController _leftHintController;
private XRControllerHintController _rightHintController; private XRControllerHintController _rightHintController;
private XRBaseInteractor lastInteractor = null;
private SmartHandPresence _leftSmartHandPresence; private SmartHandPresence _leftSmartHandPresence;
private SmartHandPresence _rightSmartHandPresence; private SmartHandPresence _rightSmartHandPresence;
@@ -244,18 +245,24 @@ namespace _PROJECT.NewHandPresence
StopCoroutine(initializationInfoCoroutine); StopCoroutine(initializationInfoCoroutine);
} }
private void OnGripPerformed(SelectEnterEventArgs arg0) private void OnGripPerformed(SelectEnterEventArgs args)
{ {
var grab = arg0.interactableObject as XRGrabInteractable;
// Notify any listeners var interactor = args.interactorObject as XRBaseInteractor;
OnGrab?.Invoke(grab); var interactable = args.interactableObject as XRGrabInteractable;
if (_state != TutorialState.Grip) return; if (lastInteractor == interactor)
Debug.Log("Grip performed"); return; // same hand grabbing again
UpdateState(_state.Next());
lastInteractor = interactor;
OnGrab?.Invoke(interactable as XRGrabInteractable);
if (_state == TutorialState.Grip)
UpdateState(_state.Next());
} }
private void OnTeleportPerformed(LocomotionSystem obj) private void OnTeleportPerformed(LocomotionSystem obj)
{ {
if (_state != TutorialState.Teleport) return; if (_state != TutorialState.Teleport) return;
Binary file not shown.
Binary file not shown.
@@ -29,7 +29,7 @@ Transform:
m_GameObject: {fileID: 1659788510314838016} m_GameObject: {fileID: 1659788510314838016}
m_LocalRotation: {x: 0, y: 0, z: 0, w: 1} m_LocalRotation: {x: 0, y: 0, z: 0, w: 1}
m_LocalPosition: {x: 0, y: 0, z: 0} m_LocalPosition: {x: 0, y: 0, z: 0}
m_LocalScale: {x: 0.4, y: 0.35, z: 0.4} m_LocalScale: {x: 0.4, y: 0.38386, z: 0.4}
m_ConstrainProportionsScale: 0 m_ConstrainProportionsScale: 0
m_Children: m_Children:
- {fileID: 4211436824399037528} - {fileID: 4211436824399037528}
@@ -149,7 +149,7 @@ Transform:
m_PrefabAsset: {fileID: 0} m_PrefabAsset: {fileID: 0}
m_GameObject: {fileID: 4039766663692926199} m_GameObject: {fileID: 4039766663692926199}
m_LocalRotation: {x: -0, y: 0.000000014901158, z: -0, w: 1} m_LocalRotation: {x: -0, y: 0.000000014901158, z: -0, w: 1}
m_LocalPosition: {x: 0, y: 0, z: 0} m_LocalPosition: {x: 0, y: -0.36, z: 0}
m_LocalScale: {x: 1, y: 1, z: 1} m_LocalScale: {x: 1, y: 1, z: 1}
m_ConstrainProportionsScale: 0 m_ConstrainProportionsScale: 0
m_Children: [] m_Children: []
+54 -226
View File
@@ -9,9 +9,10 @@ GameObject:
serializedVersion: 6 serializedVersion: 6
m_Component: m_Component:
- component: {fileID: 3337587390139541606} - component: {fileID: 3337587390139541606}
- component: {fileID: 627565954663756710} - component: {fileID: 1081661767482416249}
- component: {fileID: 3779719438397915148}
m_Layer: 0 m_Layer: 0
m_Name: SportModel m_Name: Body
m_TagString: Untagged m_TagString: Untagged
m_Icon: {fileID: 0} m_Icon: {fileID: 0}
m_NavMeshLayer: 0 m_NavMeshLayer: 0
@@ -26,7 +27,7 @@ Transform:
m_GameObject: {fileID: 3508152469845169189} m_GameObject: {fileID: 3508152469845169189}
m_LocalRotation: {x: 0, y: 0, z: 0, w: 1} m_LocalRotation: {x: 0, y: 0, z: 0, w: 1}
m_LocalPosition: {x: 0, y: 0, z: 0} m_LocalPosition: {x: 0, y: 0, z: 0}
m_LocalScale: {x: 0.24, y: 0.2, z: 0.2} m_LocalScale: {x: 1, y: 1, z: 1}
m_ConstrainProportionsScale: 0 m_ConstrainProportionsScale: 0
m_Children: m_Children:
- {fileID: 4011537191296717654} - {fileID: 4011537191296717654}
@@ -35,47 +36,50 @@ Transform:
m_Father: {fileID: 7527508320363503049} m_Father: {fileID: 7527508320363503049}
m_RootOrder: 0 m_RootOrder: 0
m_LocalEulerAnglesHint: {x: 0, y: 0, z: 0} m_LocalEulerAnglesHint: {x: 0, y: 0, z: 0}
--- !u!114 &627565954663756710 --- !u!136 &1081661767482416249
MonoBehaviour: CapsuleCollider:
m_ObjectHideFlags: 0 m_ObjectHideFlags: 0
m_CorrespondingSourceObject: {fileID: 0} m_CorrespondingSourceObject: {fileID: 0}
m_PrefabInstance: {fileID: 0} m_PrefabInstance: {fileID: 0}
m_PrefabAsset: {fileID: 0} m_PrefabAsset: {fileID: 0}
m_GameObject: {fileID: 3508152469845169189} m_GameObject: {fileID: 3508152469845169189}
m_Material: {fileID: 0}
m_IncludeLayers:
serializedVersion: 2
m_Bits: 0
m_ExcludeLayers:
serializedVersion: 2
m_Bits: 0
m_LayerOverridePriority: 0
m_IsTrigger: 0
m_ProvidesContacts: 0
m_Enabled: 1 m_Enabled: 1
m_EditorHideFlags: 0 serializedVersion: 2
m_Script: {fileID: 11500000, guid: a2836e36774ca1c4bbbee976e17b649c, type: 3} m_Radius: 1.07
m_Name: m_Height: 0.97
m_EditorClassIdentifier: m_Direction: 1
_componentIndexCache: 2 m_Center: {x: 0.16, y: -0.51, z: 0}
_addedNetworkObject: {fileID: 2689384198849609103} --- !u!65 &3779719438397915148
_networkObjectCache: {fileID: 2689384198849609103} BoxCollider:
_synchronizeParent: 0 m_ObjectHideFlags: 0
_packing: m_CorrespondingSourceObject: {fileID: 0}
Position: 1 m_PrefabInstance: {fileID: 0}
Rotation: 1 m_PrefabAsset: {fileID: 0}
Scale: 0 m_GameObject: {fileID: 3508152469845169189}
_interpolation: 2 m_Material: {fileID: 0}
_extrapolation: 2 m_IncludeLayers:
_enableTeleport: 0 serializedVersion: 2
_teleportThreshold: 1 m_Bits: 0
_clientAuthoritative: 1 m_ExcludeLayers:
_sendToOwner: 1 serializedVersion: 2
_synchronizePosition: 1 m_Bits: 0
_positionSnapping: m_LayerOverridePriority: 0
X: 0 m_IsTrigger: 0
Y: 0 m_ProvidesContacts: 0
Z: 0 m_Enabled: 1
_synchronizeRotation: 1 serializedVersion: 3
_rotationSnapping: m_Size: {x: 4.5, y: 0.5, z: 3.77}
X: 0 m_Center: {x: 0.12, y: -0.24, z: 0.06}
Y: 0
Z: 0
_synchronizeScale: 1
_scaleSnapping:
X: 0
Y: 0
Z: 0
--- !u!1 &6744593628588765187 --- !u!1 &6744593628588765187
GameObject: GameObject:
m_ObjectHideFlags: 0 m_ObjectHideFlags: 0
@@ -87,7 +91,6 @@ GameObject:
- component: {fileID: 1251512755648678878} - component: {fileID: 1251512755648678878}
- component: {fileID: 4963625795242203983} - component: {fileID: 4963625795242203983}
- component: {fileID: 5457198189491340073} - component: {fileID: 5457198189491340073}
- component: {fileID: 8959643074287331097}
m_Layer: 0 m_Layer: 0
m_Name: BezierCurve.007 m_Name: BezierCurve.007
m_TagString: Untagged m_TagString: Untagged
@@ -102,13 +105,13 @@ Transform:
m_PrefabInstance: {fileID: 0} m_PrefabInstance: {fileID: 0}
m_PrefabAsset: {fileID: 0} m_PrefabAsset: {fileID: 0}
m_GameObject: {fileID: 6744593628588765187} m_GameObject: {fileID: 6744593628588765187}
m_LocalRotation: {x: 0.00000031987892, y: 0, z: -0, w: 1} m_LocalRotation: {x: 0.00000031987892, y: -0, z: -0, w: 1}
m_LocalPosition: {x: 0.5, y: -1.2, z: 0.3} m_LocalPosition: {x: 0.120000005, y: -0.24000001, z: 0.06000042}
m_LocalScale: {x: 1, y: 1, z: 1} m_LocalScale: {x: 0.24, y: 0.2, z: 0.2}
m_ConstrainProportionsScale: 0 m_ConstrainProportionsScale: 0
m_Children: [] m_Children: []
m_Father: {fileID: 3337587390139541606} m_Father: {fileID: 3337587390139541606}
m_RootOrder: 2 m_RootOrder: -1
m_LocalEulerAnglesHint: {x: 0, y: 0, z: 0} m_LocalEulerAnglesHint: {x: 0, y: 0, z: 0}
--- !u!33 &4963625795242203983 --- !u!33 &4963625795242203983
MeshFilter: MeshFilter:
@@ -161,47 +164,6 @@ MeshRenderer:
m_SortingLayer: 0 m_SortingLayer: 0
m_SortingOrder: 0 m_SortingOrder: 0
m_AdditionalVertexStreams: {fileID: 0} m_AdditionalVertexStreams: {fileID: 0}
--- !u!114 &8959643074287331097
MonoBehaviour:
m_ObjectHideFlags: 0
m_CorrespondingSourceObject: {fileID: 0}
m_PrefabInstance: {fileID: 0}
m_PrefabAsset: {fileID: 0}
m_GameObject: {fileID: 6744593628588765187}
m_Enabled: 1
m_EditorHideFlags: 0
m_Script: {fileID: 11500000, guid: a2836e36774ca1c4bbbee976e17b649c, type: 3}
m_Name:
m_EditorClassIdentifier:
_componentIndexCache: 5
_addedNetworkObject: {fileID: 2689384198849609103}
_networkObjectCache: {fileID: 2689384198849609103}
_synchronizeParent: 0
_packing:
Position: 1
Rotation: 1
Scale: 0
_interpolation: 2
_extrapolation: 2
_enableTeleport: 0
_teleportThreshold: 1
_clientAuthoritative: 1
_sendToOwner: 1
_synchronizePosition: 1
_positionSnapping:
X: 0
Y: 0
Z: 0
_synchronizeRotation: 1
_rotationSnapping:
X: 0
Y: 0
Z: 0
_synchronizeScale: 1
_scaleSnapping:
X: 0
Y: 0
Z: 0
--- !u!1 &7680366769166628765 --- !u!1 &7680366769166628765
GameObject: GameObject:
m_ObjectHideFlags: 0 m_ObjectHideFlags: 0
@@ -213,7 +175,6 @@ GameObject:
- component: {fileID: 4011537191296717654} - component: {fileID: 4011537191296717654}
- component: {fileID: 1910778184414920112} - component: {fileID: 1910778184414920112}
- component: {fileID: 363608006073787400} - component: {fileID: 363608006073787400}
- component: {fileID: 4439869206153374580}
m_Layer: 0 m_Layer: 0
m_Name: BezierCurve.005 m_Name: BezierCurve.005
m_TagString: Untagged m_TagString: Untagged
@@ -228,13 +189,13 @@ Transform:
m_PrefabInstance: {fileID: 0} m_PrefabInstance: {fileID: 0}
m_PrefabAsset: {fileID: 0} m_PrefabAsset: {fileID: 0}
m_GameObject: {fileID: 7680366769166628765} m_GameObject: {fileID: 7680366769166628765}
m_LocalRotation: {x: 0.00000031987892, y: 0, z: -0, w: 1} m_LocalRotation: {x: 0.00000031987892, y: -0, z: -0, w: 1}
m_LocalPosition: {x: 0.5, y: 0.3, z: 0.3} m_LocalPosition: {x: 0.120000005, y: 0.060000062, z: 0.06000042}
m_LocalScale: {x: 1, y: 1, z: 1} m_LocalScale: {x: 0.24, y: 0.2, z: 0.2}
m_ConstrainProportionsScale: 0 m_ConstrainProportionsScale: 0
m_Children: [] m_Children: []
m_Father: {fileID: 3337587390139541606} m_Father: {fileID: 3337587390139541606}
m_RootOrder: 0 m_RootOrder: -1
m_LocalEulerAnglesHint: {x: 0, y: 0, z: 0} m_LocalEulerAnglesHint: {x: 0, y: 0, z: 0}
--- !u!33 &1910778184414920112 --- !u!33 &1910778184414920112
MeshFilter: MeshFilter:
@@ -286,47 +247,6 @@ MeshRenderer:
m_SortingLayer: 0 m_SortingLayer: 0
m_SortingOrder: 0 m_SortingOrder: 0
m_AdditionalVertexStreams: {fileID: 0} m_AdditionalVertexStreams: {fileID: 0}
--- !u!114 &4439869206153374580
MonoBehaviour:
m_ObjectHideFlags: 0
m_CorrespondingSourceObject: {fileID: 0}
m_PrefabInstance: {fileID: 0}
m_PrefabAsset: {fileID: 0}
m_GameObject: {fileID: 7680366769166628765}
m_Enabled: 1
m_EditorHideFlags: 0
m_Script: {fileID: 11500000, guid: a2836e36774ca1c4bbbee976e17b649c, type: 3}
m_Name:
m_EditorClassIdentifier:
_componentIndexCache: 3
_addedNetworkObject: {fileID: 2689384198849609103}
_networkObjectCache: {fileID: 2689384198849609103}
_synchronizeParent: 0
_packing:
Position: 1
Rotation: 1
Scale: 0
_interpolation: 2
_extrapolation: 2
_enableTeleport: 0
_teleportThreshold: 1
_clientAuthoritative: 1
_sendToOwner: 1
_synchronizePosition: 1
_positionSnapping:
X: 0
Y: 0
Z: 0
_synchronizeRotation: 1
_rotationSnapping:
X: 0
Y: 0
Z: 0
_synchronizeScale: 1
_scaleSnapping:
X: 0
Y: 0
Z: 0
--- !u!1 &7983999977322064054 --- !u!1 &7983999977322064054
GameObject: GameObject:
m_ObjectHideFlags: 0 m_ObjectHideFlags: 0
@@ -340,8 +260,6 @@ GameObject:
- component: {fileID: 8954823527229761837} - component: {fileID: 8954823527229761837}
- component: {fileID: 2689384198849609103} - component: {fileID: 2689384198849609103}
- component: {fileID: 1843883317757509263} - component: {fileID: 1843883317757509263}
- component: {fileID: 7451921017238644471}
- component: {fileID: 2131608416831220487}
m_Layer: 0 m_Layer: 0
m_Name: TargetUFO m_Name: TargetUFO
m_TagString: Untagged m_TagString: Untagged
@@ -357,7 +275,7 @@ Transform:
m_PrefabAsset: {fileID: 0} m_PrefabAsset: {fileID: 0}
m_GameObject: {fileID: 7983999977322064054} m_GameObject: {fileID: 7983999977322064054}
m_LocalRotation: {x: 0, y: 0, z: 0, w: 1} m_LocalRotation: {x: 0, y: 0, z: 0, w: 1}
m_LocalPosition: {x: 1.398, y: 1.1, z: 8.292} m_LocalPosition: {x: 0, y: 0, z: 0}
m_LocalScale: {x: 1, y: 1, z: 1} m_LocalScale: {x: 1, y: 1, z: 1}
m_ConstrainProportionsScale: 0 m_ConstrainProportionsScale: 0
m_Children: m_Children:
@@ -429,10 +347,6 @@ MonoBehaviour:
_networkBehaviours: _networkBehaviours:
- {fileID: 4850307641519017041} - {fileID: 4850307641519017041}
- {fileID: 1843883317757509263} - {fileID: 1843883317757509263}
- {fileID: 627565954663756710}
- {fileID: 4439869206153374580}
- {fileID: 6540522042633253623}
- {fileID: 8959643074287331097}
<ParentNetworkObject>k__BackingField: {fileID: 0} <ParentNetworkObject>k__BackingField: {fileID: 0}
<ChildNetworkObjects>k__BackingField: [] <ChildNetworkObjects>k__BackingField: []
_isNetworked: 1 _isNetworked: 1
@@ -487,50 +401,6 @@ MonoBehaviour:
X: 0 X: 0
Y: 0 Y: 0
Z: 0 Z: 0
--- !u!136 &7451921017238644471
CapsuleCollider:
m_ObjectHideFlags: 0
m_CorrespondingSourceObject: {fileID: 0}
m_PrefabInstance: {fileID: 0}
m_PrefabAsset: {fileID: 0}
m_GameObject: {fileID: 7983999977322064054}
m_Material: {fileID: 0}
m_IncludeLayers:
serializedVersion: 2
m_Bits: 0
m_ExcludeLayers:
serializedVersion: 2
m_Bits: 0
m_LayerOverridePriority: 0
m_IsTrigger: 0
m_ProvidesContacts: 0
m_Enabled: 1
serializedVersion: 2
m_Radius: 1.07
m_Height: 0.97
m_Direction: 1
m_Center: {x: 0.16, y: -0.51, z: 0}
--- !u!65 &2131608416831220487
BoxCollider:
m_ObjectHideFlags: 0
m_CorrespondingSourceObject: {fileID: 0}
m_PrefabInstance: {fileID: 0}
m_PrefabAsset: {fileID: 0}
m_GameObject: {fileID: 7983999977322064054}
m_Material: {fileID: 0}
m_IncludeLayers:
serializedVersion: 2
m_Bits: 0
m_ExcludeLayers:
serializedVersion: 2
m_Bits: 0
m_LayerOverridePriority: 0
m_IsTrigger: 0
m_ProvidesContacts: 0
m_Enabled: 1
serializedVersion: 3
m_Size: {x: 4.5, y: 0.5, z: 3.77}
m_Center: {x: 0.12, y: -0.24, z: 0.06}
--- !u!1 &8462201784916155029 --- !u!1 &8462201784916155029
GameObject: GameObject:
m_ObjectHideFlags: 0 m_ObjectHideFlags: 0
@@ -542,7 +412,6 @@ GameObject:
- component: {fileID: 2406214921254263651} - component: {fileID: 2406214921254263651}
- component: {fileID: 6767210086105080178} - component: {fileID: 6767210086105080178}
- component: {fileID: 3766991064380355318} - component: {fileID: 3766991064380355318}
- component: {fileID: 6540522042633253623}
m_Layer: 0 m_Layer: 0
m_Name: BezierCurve.006 m_Name: BezierCurve.006
m_TagString: Untagged m_TagString: Untagged
@@ -558,12 +427,12 @@ Transform:
m_PrefabAsset: {fileID: 0} m_PrefabAsset: {fileID: 0}
m_GameObject: {fileID: 8462201784916155029} m_GameObject: {fileID: 8462201784916155029}
m_LocalRotation: {x: -0.00000015993946, y: 0.86602545, z: -0.00000027702328, w: -0.49999997} m_LocalRotation: {x: -0.00000015993946, y: 0.86602545, z: -0.00000027702328, w: -0.49999997}
m_LocalPosition: {x: 0.5, y: -3.4, z: 0.3} m_LocalPosition: {x: 0.121, y: -0.663, z: 0.072}
m_LocalScale: {x: 1, y: 1, z: 1} m_LocalScale: {x: 0.22, y: 0.2, z: 0.24}
m_ConstrainProportionsScale: 0 m_ConstrainProportionsScale: 0
m_Children: [] m_Children: []
m_Father: {fileID: 3337587390139541606} m_Father: {fileID: 3337587390139541606}
m_RootOrder: 1 m_RootOrder: -1
m_LocalEulerAnglesHint: {x: 0, y: 0, z: 0} m_LocalEulerAnglesHint: {x: 0, y: 0, z: 0}
--- !u!33 &6767210086105080178 --- !u!33 &6767210086105080178
MeshFilter: MeshFilter:
@@ -615,44 +484,3 @@ MeshRenderer:
m_SortingLayer: 0 m_SortingLayer: 0
m_SortingOrder: 0 m_SortingOrder: 0
m_AdditionalVertexStreams: {fileID: 0} m_AdditionalVertexStreams: {fileID: 0}
--- !u!114 &6540522042633253623
MonoBehaviour:
m_ObjectHideFlags: 0
m_CorrespondingSourceObject: {fileID: 0}
m_PrefabInstance: {fileID: 0}
m_PrefabAsset: {fileID: 0}
m_GameObject: {fileID: 8462201784916155029}
m_Enabled: 1
m_EditorHideFlags: 0
m_Script: {fileID: 11500000, guid: a2836e36774ca1c4bbbee976e17b649c, type: 3}
m_Name:
m_EditorClassIdentifier:
_componentIndexCache: 4
_addedNetworkObject: {fileID: 2689384198849609103}
_networkObjectCache: {fileID: 2689384198849609103}
_synchronizeParent: 0
_packing:
Position: 1
Rotation: 1
Scale: 0
_interpolation: 2
_extrapolation: 2
_enableTeleport: 0
_teleportThreshold: 1
_clientAuthoritative: 1
_sendToOwner: 1
_synchronizePosition: 1
_positionSnapping:
X: 0
Y: 0
Z: 0
_synchronizeRotation: 1
_rotationSnapping:
X: 0
Y: 0
Z: 0
_synchronizeScale: 1
_scaleSnapping:
X: 0
Y: 0
Z: 0
Binary file not shown.
@@ -4,9 +4,9 @@ using System.Runtime.InteropServices;
using UnityEngine; using UnityEngine;
using FMOD; using FMOD;
using FMODUnity; using FMODUnity;
using Whisper; // WhisperManager, WhisperStream, WhisperResult using Whisper;
using Whisper.Utils; using Whisper.Utils;
using Debug = UnityEngine.Debug; // AudioChunk using Debug = UnityEngine.Debug;
/// <summary> /// <summary>
/// FMOD mic is initialized once (Start) and runs continuously in a ring buffer. /// FMOD mic is initialized once (Start) and runs continuously in a ring buffer.
@@ -28,12 +28,6 @@ public class FMODWhisperBridge : MonoBehaviour
public int channels = 1; public int channels = 1;
[Range(1, 10)] public int bufferLengthSec = 5; [Range(1, 10)] public int bufferLengthSec = 5;
[Header("Loopback (monitor your voice)")]
public bool playLoopback = true;
[Tooltip("If true, loopback plays only while active; otherwise its always on.")]
public bool loopbackOnlyWhenActive = true;
[Range(0f, 2f)] public float loopbackVolume = 1.0f;
public delegate void OnWhisperSegmentUpdatedDelegate(string result); public delegate void OnWhisperSegmentUpdatedDelegate(string result);
public event OnWhisperSegmentUpdatedDelegate OnWhisperSegmentUpdated; public event OnWhisperSegmentUpdatedDelegate OnWhisperSegmentUpdated;
@@ -44,7 +38,6 @@ public class FMODWhisperBridge : MonoBehaviour
private FMOD.System _core; private FMOD.System _core;
private Sound _recSound; private Sound _recSound;
private Channel _playChannel; private Channel _playChannel;
private ChannelGroup _masterGroup;
private uint _soundPcmLength; // in samples private uint _soundPcmLength; // in samples
private int _nativeRate; private int _nativeRate;
private int _nativeChannels; private int _nativeChannels;
@@ -54,7 +47,6 @@ public class FMODWhisperBridge : MonoBehaviour
// Whisper // Whisper
private WhisperStream _stream; private WhisperStream _stream;
private bool _streamStarted;
// temp conversion buffer // temp conversion buffer
private float[] _floatTmp = new float[0]; private float[] _floatTmp = new float[0];
@@ -138,24 +130,6 @@ public class FMODWhisperBridge : MonoBehaviour
_core.getRecordPosition(recordDriverId, out _lastRecordPos); _core.getRecordPosition(recordDriverId, out _lastRecordPos);
Debug.Log("[FMOD→Whisper] Recording started."); Debug.Log("[FMOD→Whisper] Recording started.");
// Loopback channel (optional). Start once; pause when inactive if desired.
_core.getMasterChannelGroup(out _masterGroup);
if (playLoopback)
{
res = _core.playSound(_recSound, _masterGroup, false, out _playChannel);
if (res == RESULT.OK && _playChannel.hasHandle())
{
_playChannel.setMode(MODE._2D);
_playChannel.setVolume(loopbackVolume);
if (loopbackOnlyWhenActive) _playChannel.setPaused(true); // keep muted until Activate
Debug.Log("[FMOD→Whisper] Loopback playback ready.");
}
else
{
Debug.LogWarning($"[FMOD→Whisper] playSound failed or channel invalid: {res}");
}
}
// No Whisper stream here. It will be created on ActivateRecording(). // No Whisper stream here. It will be created on ActivateRecording().
await System.Threading.Tasks.Task.Yield(); await System.Threading.Tasks.Task.Yield();
} }
@@ -181,8 +155,7 @@ public class FMODWhisperBridge : MonoBehaviour
if (_stream != null) if (_stream != null)
{ {
try { _stream.StopStream(); } catch { } _stream.StopStream();
_streamStarted = false;
} }
try try
@@ -193,7 +166,6 @@ public class FMODWhisperBridge : MonoBehaviour
{ {
Debug.LogError($"[FMOD→Whisper] CreateStream exception: {e}"); Debug.LogError($"[FMOD→Whisper] CreateStream exception: {e}");
_stream = null; _stream = null;
_streamStarted = false;
return; return;
} }
@@ -214,16 +186,6 @@ public class FMODWhisperBridge : MonoBehaviour
whisper.useVad = useVadInStream; whisper.useVad = useVadInStream;
_stream.StartStream(); _stream.StartStream();
_streamStarted = true;
// --- NEW: Clear the ring buffer and reset read pointer ---
// Pause loopback while we clear (optional, but avoids clicks)
if (playLoopback && _playChannel.hasHandle())
_playChannel.setPaused(true);
// Clear buffer bytes
ClearRecordRingBuffer();
// Reset our read pointer to the current write head // Reset our read pointer to the current write head
_core.getRecordPosition(recordDriverId, out _lastRecordPos); _core.getRecordPosition(recordDriverId, out _lastRecordPos);
@@ -231,13 +193,8 @@ public class FMODWhisperBridge : MonoBehaviour
// Well skip feeding for one frame to guarantee a clean start // Well skip feeding for one frame to guarantee a clean start
_skipOneFeedFrame = true; _skipOneFeedFrame = true;
// Unpause loopback if we want it active during recording
if (playLoopback && _playChannel.hasHandle() && (!loopbackOnlyWhenActive || isRecordingActivated))
_playChannel.setPaused(loopbackOnlyWhenActive ? false : _playChannel.getPaused(out var paused) == FMOD.RESULT.OK && paused ? false : false);
isRecordingActivated = true; isRecordingActivated = true;
Debug.Log("[FMOD→Whisper] Stream activated (buffer cleared; reading from current head)."); Debug.Log("[FMOD→Whisper] Stream activated (buffer cleared; reading from current head).");
} }
/// <summary> /// <summary>
@@ -245,14 +202,10 @@ public class FMODWhisperBridge : MonoBehaviour
/// </summary> /// </summary>
public void DeactivateRecording() public void DeactivateRecording()
{ {
if (!isRecordingActivated && !_streamStarted) if (!isRecordingActivated)
return; return;
isRecordingActivated = false; isRecordingActivated = false;
// Pause loopback if it should only be active during recording
if (playLoopback && loopbackOnlyWhenActive && _playChannel.hasHandle())
_playChannel.setPaused(true);
} }
/// <summary> /// <summary>
@@ -277,7 +230,6 @@ public class FMODWhisperBridge : MonoBehaviour
return Mathf.Clamp01(Mathf.InverseLerp(-60f, -15f, db)); return Mathf.Clamp01(Mathf.InverseLerp(-60f, -15f, db));
} }
private void Update() private void Update()
{ {
// Always tick FMOD // Always tick FMOD
@@ -285,8 +237,7 @@ public class FMODWhisperBridge : MonoBehaviour
if (!_recSound.hasHandle()) return; if (!_recSound.hasHandle()) return;
// Compute how many samples recorded since last frame. // Compute how many samples recorded since last frame.
uint recPos; _core.getRecordPosition(recordDriverId, out uint recPos);
_core.getRecordPosition(recordDriverId, out recPos);
uint deltaSamples = (recPos >= _lastRecordPos) uint deltaSamples = (recPos >= _lastRecordPos)
? (recPos - _lastRecordPos) ? (recPos - _lastRecordPos)
@@ -337,7 +288,7 @@ public class FMODWhisperBridge : MonoBehaviour
} }
// 2) Feed audio to Whisper // 2) Feed audio to Whisper
if (_streamStarted && _stream != null) if (_stream != null)
{ {
if (isRecordingActivated && !_skipOneFeedFrame) if (isRecordingActivated && !_skipOneFeedFrame)
{ {
@@ -353,7 +304,6 @@ public class FMODWhisperBridge : MonoBehaviour
} }
} }
// If skipping, we just discard this frame to ensure no stale data leaks. // If skipping, we just discard this frame to ensure no stale data leaks.
} }
finally finally
@@ -364,7 +314,6 @@ public class FMODWhisperBridge : MonoBehaviour
if (_skipOneFeedFrame) _skipOneFeedFrame = false; if (_skipOneFeedFrame) _skipOneFeedFrame = false;
_lastRecordPos = recPos; _lastRecordPos = recPos;
} }
private string PostProcessInput(string input) private string PostProcessInput(string input)
@@ -450,56 +399,6 @@ public class FMODWhisperBridge : MonoBehaviour
} }
} }
private void ClearRecordRingBuffer()
{
if (!_recSound.hasHandle() || _soundPcmLength == 0) return;
uint totalBytes = _soundPcmLength * (uint)_nativeChannels * 2; // PCM16
IntPtr p1, p2;
uint len1, len2;
// Lock the whole buffer (start=0, length=totalBytes)
var r = _recSound.@lock(0, totalBytes, out p1, out p2, out len1, out len2);
if (r != FMOD.RESULT.OK)
{
Debug.LogWarning($"[FMOD→Whisper] Could not lock ring buffer to clear: {r}");
return;
}
try
{
if (len1 > 0)
{
// zero p1
// Well reuse a static zero array to avoid allocating huge buffers repeatedly
ZeroMem(p1, (int)len1);
}
if (len2 > 0)
{
ZeroMem(p2, (int)len2);
}
}
finally
{
_recSound.unlock(p1, p2, len1, len2);
}
}
// cheap zeroing helper (avoids allocating len-sized arrays each time)
private static readonly byte[] _zeroChunk = new byte[16 * 1024]; // 16 KB
private static void ZeroMem(IntPtr dst, int byteLen)
{
int offset = 0;
while (byteLen > 0)
{
int n = Math.Min(_zeroChunk.Length, byteLen);
Marshal.Copy(_zeroChunk, 0, dst + offset, n);
offset += n;
byteLen -= n;
}
}
/// <summary> /// <summary>
/// Computes RMS (root mean square) from a PCM16 block using only safe code. /// Computes RMS (root mean square) from a PCM16 block using only safe code.
/// Uses the shared _shortOverlay buffer (no allocations). /// Uses the shared _shortOverlay buffer (no allocations).
@@ -171,7 +171,7 @@ public class ShapeDetectionNPC : NPCController
state = 8; state = 8;
staticRadio.SetActive(false); staticRadio.SetActive(false);
questMarker.gameObject.SetActive(false); questMarker.gameObject.SetActive(false);
SpeakVoiceLine(7, radio.gameObject, 0); SpeakVoiceLine(7, gameObject, 0);
await Task.Delay(6500); await Task.Delay(6500);
MoveToNextPoint(0); MoveToNextPoint(0);
} }
BIN
View File
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
+60 -48
View File
@@ -1,85 +1,97 @@
![DeltaVR](https://cgvrgit.ulno.net/cgvr/DeltaVR/media/branch/master/Doc/Readme-Header.png) ![DeltaVR](https://cgvrgit.ulno.net/cgvr/DeltaVR/media/branch/master/Doc/Readme-Header.png)
DeltaVR is a virtual reality experience set in the [Delta Centre](https://delta.ut.ee/) of the [University of Tartu](https://ut.ee/). The virtual Delta Building includes several interactive scenes that demonstrate the teaching and research actively done in the Delta Centre. # DeltaVR: Player-Prompted 3D Model Generation Game Mechanics
The application works on PCVR, Meta Quest 2 and 3, HTC Vive, and regular Windows PC platforms. There is cross-platform multiplayer functionality that enables several users to be in the same virtual environment from both VR and PC platforms. This is a fork of the [main DeltaVR project](https://cgvrgit.ulno.net/cgvr/DeltaVR). To learn more about DeltaVR itself, look at the main repository's README.
This project is about investigating the feasibility of player-prompted runtime 3D model generation game mechanics. These are game mechanics that allow the player to generate arbitrary 3D objects using natural language speech. This project includes three mini-games that use this speech-to-3D system as a core game mechanic.
## Features ## Mini-Games
### Exploration ### Delta Cafe
<img src="https://cgvrgit.ulno.net/henrisel/DeltaVR3DModelGeneration/media/branch/master/Doc/screenshots/delta_cafe_screenshot.png" width="800">
DeltaVR features the first two floors of the Delta Educational Building for **exploration and discovery**. There are many diegetic elements representing the studies and research conducted at the Delta Building, such as robotics, the high-performance computing server room, video game development, and student life. ### Archery Range
<img src="https://cgvrgit.ulno.net/henrisel/DeltaVR3DModelGeneration/media/branch/master/Doc/screenshots/archery_range_screenshot.png" width="800">
![DeltaVR Exploration](https://cgvrgit.ulno.net/cgvr/DeltaVR/media/branch/master/Doc/clips/Explore-Clip.gif) ### Shape Detection
<img src="https://cgvrgit.ulno.net/henrisel/DeltaVR3DModelGeneration/media/branch/master/Doc/screenshots/shape_detection_gameobjects.png" width="800">
### UFO Bow Game Playthrough videos of each mini-game are available on YouTube: https://www.youtube.com/playlist?list=PLqIwPfWyxyaH-YHn64jW59aFhQ5OKf9CW
At the terrace on the second floor, Delta explorers can defend the building from UFO-s using a bow and **achieve high scores**. ## Setup Instructions
![DeltaVR UFO Bow Game](https://cgvrgit.ulno.net/cgvr/DeltaVR/media/branch/master/Doc/clips/UFO-Bow-Game-Clip.gif) The built version of the game is available here: https://owncloud.ut.ee/owncloud/s/tXBd4XGpbXTi7Hm.
### Bolt Self-Driving Car The build and the editor version both rely on a speech-to-3D pipeline to function. This pipeline needs to be set up locally and consists of two additional software applications in addition to the Unity game itself: InvokeAI and TRELLIS.
The courtyard between the Educational and Entrepreneurial buildings of the Delta Centre, the explorers can see the Bolt Self-Driving Car. This car is developed by the [[http://adl.cs.ut.ee/|Autonomous Driving Lab]] of the [[https://cs.ut.ee|Institute of Computer Science]]. If one is brave enough, they can stop the car and catch a ride, simulating both the **feeling of being in a self-driving vehicle** as well as VR motion sickness. ### InvokeAI
![DeltaVR Bolt Self-Driving Car](https://cgvrgit.ulno.net/cgvr/DeltaVR/media/branch/master/Doc/clips/Bolt-Self-Driving-Car-Clip.gif) InvokeAI is used to generate images from text prompts. To install InvokeAI, follow the instructions in the official guide: https://invoke.ai/start-here/installation.
### Space Walk After installing it, an AI model needs to be set up in InvokeAI. This project has been tested with Stable Diffusion XL, but other models can also be used. Documentation on setting up models: https://invoke.ai/concepts/models.
Where the actual Delta building has a set of skywalks connecting it with the entrepreneurship building, DeltaVR has a set of portals leading to the Space walk experience. In it, one can move in the **vastness of space** and experience **changes in gravity**. A fleet of UFO ships react to one's presence and come to investigate the arrival. Finally, to enable communication with InvokeAI across different computers, enable the Server Mode in InvokeAI settings.
![DeltaVR Spacewalk](https://cgvrgit.ulno.net/cgvr/DeltaVR/media/branch/master/Doc/clips/Skywalk-Clip.gif) ### TRELLIS
### Server Room TRELLIS is used to generate 3D models based on images. The simplest way to install TRELLIS on Windows is via the one-click installer provided in this fork of the original TRELLIS repo: https://github.com/IgorAherne/trellis-stable-projectorz. The download package includes instructions on how to launch TRELLIS as a local API server. For the default configuration, use the following command:
```
.\run.bat --ip 0.0.0.0
```
On the second floor, one can hear the humming of the servers. Should they investigate, they will find a room of server racks and a large red button. Should they push the button, they will trigger the **fire alarm** and have the server room fill with harmful invisible gas. This propms the player to escape the room. This largerly **auditory experience** is noted to be engaging and immersive. It represents the work of UT HPC in maintaining the servers of the University of Tartu. ### Unity
![DeltaVR Server Room](https://cgvrgit.ulno.net/cgvr/DeltaVR/media/branch/master/Doc/clips/Server-Room-Clip.gif) In both editor mode and the built version, after first starting the game, the `config.json` file is created in the game's root directory. This file is used to persist configurable values between play sessions. The following values need to be set correctly for the game to be able to communicate with InvokeAI and TRELLIS:
### Interactive Map * `invokeAiUrl` - the URL of InvokeAI (for example `http://192.168.0.53:9090`)
* `invokeAiModelKey` - the key of the specific model set up in InvokeAI to use. To see the keys of your installed models, send a GET request to `{INVOKEAI_URL}/api/v2/models/?model_type=main` (for example, `http://192.168.0.53:9090/api/v2/models/?model_type=main`)
* `trellisUrl` - the URL of TRELLIS (for example, `http://192.168.0.53:7960`)
To navigate the two floors of the large Delta Educational Building, explorers have an interactive map. This provides a clear overview of where they currently are and what other interactions are located across the building. Explorers can teleport to a **select interactive experiences**, while others are left for them to discover based on the hints on the map. The [3d-generation-pipeline](3d-generation-pipeline/README.md) folder includes Python notebooks and scripts that can be used to test the pipeline working together.
![DeltaVR Interactive Map](https://cgvrgit.ulno.net/cgvr/DeltaVR/media/branch/master/Doc/clips/Interactive-Map-Clip.gif) ## Unresolved bugs and potential improvements for the future
* continuous grabbing sound when grabbing with both hands at the same time: "(Un)requesting ownership of Cube"
* FMOD ChannelControl errors
* speech-to-text: investigate why Whisper still sometimes stops detecting speech
* glTF loading: investigate how to use the Universal Render Pipeline Lit shader for the glTF-loaded objects.
* NPC characters:
* glass wall is sometimes incorrectly visible in front of the NPC
* Delta Cafe:
* ability for players to rate the orders
* "order again" button
* Archery Range:
* NPC should repeat the last instruction when the player is not progressing
* reuse the UFO flying animation from Space Walk in the beginning, when NPC talks about looking out the window and the UFO attack + have the default UFO already in the model display capsule
* Shape Detection:
* explain to the player how to scale printed objects
* set min and max scale limits on printed objects
* set 3d printer soft particles to avoid sharp edges when particles go inside the printer wall
### Whiteboard
In the virtual Computer Graphics and Virtual Reality Study Lab, explorers can use spray paint cans to draw on a whiteboard. Surprisingly, this is one of the **more popular interactive experiences** of DeltaVR.
![DeltaVR Whiteboard](https://cgvrgit.ulno.net/cgvr/DeltaVR/media/branch/master/Doc/clips/Whiteboard-Clip.gif)
## Credits ## Credits
**Ranno Samuel Adson**<br/> **Samuel Saar**<br/>
User experience design. Additional interactions. Interaction improvements. Creation of the following 3D models used in the mini-games:
* Delta Cafe: pen, notepad, serving tray
**Toomas Tamm**<br/> * Archery Range: capsule display, microphone stand, wires, table, screens
Project architecture, model optimization, lighting. [Bachelor's Thesis](https://comserv.cs.ut.ee/ati_thesis/datasheet.php?id=71682) ([poster](https://courses.cs.ut.ee/student_projects/download/478.pdf)), [Master's Thesis](https://comserv.cs.ut.ee/ati_thesis/datasheet.php?id=77065&language=en). * Shape Detection: computer printer, 3D printer, radio transmitter, shape detector
**Joonas Püks**<br/>
Multiplayer and cross-play functionality. [Bachelor's Thesis](https://comserv.cs.ut.ee/ati_thesis/datasheet.php?id=74390) ([poster](https://courses.cs.ut.ee/student_projects/download/534.pdf)).
**Timur Nizamov**<br/> **Timur Nizamov**<br/>
Technical sound design. Sound effects and integrating voicelines into Unity
**Raimond Tunnel**<br/> **Madis Vasser**<br/>
Project management, visual design. Voice acting for the Ufo and Professor characters
Developed in the [Computer Graphcis and Virtual Reality Study Lab](https://cgvr.cs.ut.ee/) of the [Institute of Computer Science, University of Tartu](https://cs.ut.ee). **Ulrich Norbisrath**<br/>
Voice acting for the Robert character
### Used Attributions **Ranno Samuel Adson**<br/>
Technical help and tips for DeltaVR
| Description | License | Source | Author | (These credits are specifically for the work of this fork. Credits for the main DeltaVR project are shown in the main project's README.)
|-----------------------------------------------------|----------------------------------------------|---------------------------------------------------------------------------------------------|------------------|
| Server rack model | Royalty Free, No AI License | [Link](https://www.cgtrader.com/free-3d-models/electronics/computer/simple-server-model) | anymelok |
| Robot movement sound | Creative Commons 0 | [Link](https://freesound.org/people/Brazilio123/sounds/661435/) | Brazilio123 |
| Spacewalk UFO sound | Attribution NonCommercial 4.0 | [Link](https://freesound.org/people/Speedenza/sounds/209366/) | Speedenza |
| Keyboard icons | Creative Commons Attribution-NoDerivs 3.0 | [Link](https://icons8.com/) | icons8 |
----- Developed in the [Computer Graphics and Virtual Reality Study Lab](https://cgvr.cs.ut.ee/) of the [Institute of Computer Science, University of Tartu](https://cs.ut.ee).
DeltaVR was moved to this repository in 2025. The previous repository is available here: [[https://gitlab.com/UT-CGVR/deltavr]]
----- -----
+1 -1
View File
@@ -1,7 +1,7 @@
{ {
"isContinuousLocomotion": true, "isContinuousLocomotion": true,
"continuousLocomotionSpeed": 3.0, "continuousLocomotionSpeed": 3.0,
"volumeMaster": 0.5, "volumeMaster": 0.4399948716163635,
"volumeAmbient": 0.1889490932226181, "volumeAmbient": 0.1889490932226181,
"volumeMusic": 0.5, "volumeMusic": 0.5,
"volumeSFX": 0.5, "volumeSFX": 0.5,
+1 -1
View File
@@ -1 +1 @@
{"HighScore":416.0} {"HighScore":212.0}
+38 -38
View File
@@ -1,64 +1,64 @@
{ {
"entries": [ "entries": [
{ {
"name": "Perfoon", "name": "karlkolm",
"score": 416.0 "score": 241.0
},
{
"name": "jass",
"score": 212.0
}, },
{ {
"name": "andreas", "name": "andreas",
"score": 207.0 "score": 207.0
}, },
{
"name": "karlkaks",
"score": 198.0
},
{ {
"name": "rikkss", "name": "rikkss",
"score": 195.0 "score": 195.0
}, },
{
"name": "noob",
"score": 195.0
},
{ {
"name": "jjkujkkg", "name": "jjkujkkg",
"score": 194.0 "score": 194.0
}, },
{
"name": "kuues",
"score": 186.0
},
{
"name": "gert",
"score": 184.0
},
{
"name": "karl",
"score": 183.0
},
{ {
"name": "f", "name": "f",
"score": 181.0 "score": 181.0
}, },
{
"name": "kosta",
"score": 180.0
},
{
"name": "gg",
"score": 179.0
},
{
"name": "gert",
"score": 164.0
},
{ {
"name": "rikkss", "name": "rikkss",
"score": 163.0 "score": 163.0
},
{
"name": "andeas",
"score": 161.0
},
{
"name": "rikkss",
"score": 141.0
},
{
"name": "h",
"score": 138.0
},
{
"name": "raimps",
"score": 115.0
},
{
"name": "m",
"score": 109.0
},
{
"name": "andreas",
"score": 96.0
},
{
"name": "m",
"score": 76.0
},
{
"name": "k",
"score": 58.0
},
{
"name": "kr",
"score": 53.0
} }
] ]
} }