How To Use ChatGPT With Unity During Runtime ?
Hello everyone 👋
In today's video I like to show you a ChatGPT prototype that I recently created where I am able to provide ChatGPT with a prompt and rules which then generates valid Unity C# code that is executed from Unity at runtime by utilizing a Roslyn C# Runtime Compiler.
📣 Full video providing you with an overview of my early ChatGPT prototype with Unity can be watched here
💡 Few additional details about this ChatGPT prototype:
📌 This prototype uses a web service which I made with Python + Flask and it communicates with ChatGPT by using a headless browser (I do not recommend using this for production because ChatGPT API is currently not publicly available) but if interested you could do something similar by integrating the ChatGPT Python Wrapper which is currently available from GitHub.
📌 The code is generated by ChatGPT by an initial HTTP request sent from Unity, if ChatGPT generates the code successfully then a HTTP response is received and parsed in Unity which extracts the source code from the markdown. There is a ChatGPTClient which provide REST Client functionality and I will be covering next on my next ChatGPT video.
📌 The next step is to take the HTTP response and validate that it can be compiled, if so it gets compiled at runtime, however if there are issues during the compilation, I re-try (up to 3 times) and send a new prompt to ChatGPT which includes the exception message to avoid getting the same error again.
📌 To get 3D models in Unity at runtime I use Sketchfab REST APIs such as their Data API and Download API, these APIs allow me search for models and get a download URI from searched models results which I then use a glTF version which is downloaded to Unity and imported with glTFast.
ChatGPT Prototype Screenshots added below to show you Unity components including ChatGPTTester, ChatGPTQuestion scriptable object, UI components, and ChatGPT Python Flask service.
If you are interesting in learning more about ChatGPT with Unity be sure to check out the How To Use ChatGPT videos with Unity series and here’s a breakdown of what I am covering so far based on the ChatGPT Prototype diagram shown above on the screenshots.
How To Use ChatGPT With Unity: Python And API Setup #2
📚 This video covers how to integrate ChatGPT Python Wrapper and creating a Python Flask Service which will be the core ChatGPT component used for communicating with ChatGPT from Unity.
💡 What’s covered in this video?
Python Environment and ChatGPT Wrapper Setup
ChatGPT Wrapper (Setting Up Credentials / Communication With ChatGPT)
Creating A ChatGPT Web Service (API)
Testing ChatGPT Web Service (API) Routes
Displaying ChatGPT Questions And Answers with Debug Flag
How To Use ChatGPT With Unity: Creating A ChatGPT HTTP Client #3
📚 This video goes over creating a Web HTTP Client which will allow us to communicate with our API created previously, this new feature will be the beginning of getting ChatGPT HTTP responses into Unity and in future videos we will begin mapping the generated code to allow for Unity execution at runtime.
💡 What’s covered in this video?
What we will be implementing today with ChatGPT ?
Creating a ChatGPT Settings scriptable object
Creating a ChatGPT Request object
Creating a ChatGPT Response object
Creating a ChatGPT Client
Creating a ChatGPT Tester MonoBehaviour
Testing our ChatGPTClient
How To Use ChatGPT With Unity: Embed A Runtime Compiler With ChatGPT #4
📚This video covers how to embed Roslyn C# compiler in Unity with .NET Standard 2.1 and also how to integrate it to our ChatGPT prototype by adding a Roslyn Code Runner script which will be responsible for running ChatGPT generated code.
💡 What’s covered in this video?
Creating a Visual Studio Class Library project with .NET Standard 2.1
Adding Roslyn C# compiler nugget packages
Adding Runtime Loader nugget package
Modifying Project to allow to copy assemblies to bin directory
Adding a Roslyn Code Runner MonoBebaviour
Adding a ChatGPT Extensions static class to handle parsing ChatGPT generated code
ChatGPT And Roslyn Demos with a few prompts
ℹ️ If you like these videos and would like to see more detailed tutorials let me know in the comments below and I will be more than happy to do so.
I’m also planning on releasing to Patrons the GitHub repo with everything covered so far but currently I am trying to move from a paid runtime C# compiler to my own implementation so stay tuned as I will provide it via Patreon very soon.
💡 Also consider subscribing to my YouTube channel to get notified of new weekly videos !
Thank you !
