AI Agent Setup

Consumer hardware capability has reached the point where very sophisticated AI models can be run locally, allowing AI functionality to be added to the Galixsys Communicator platform without the need to use a 3rd party or external AI service (such as ChatGPT). GalixiHub has built-in AI agent capability, and when connected to an AI server, allows the creation of “AI members” to give AI capability to GalixiCom users. The following is an example of how to setup an AI server, integrate with GalixiHub, create an AI user, and then make use of that AI user in GalixiCom.

AI Server Setup Example

The following example is based on a Linux-based Nvidia DGX Spark “AI supercomputer”, but of course the GalixiHub AI agent works with a regular desktop computer with an appropriately powerful graphics card to run AI inference models, or other “AI desktop serer” devices from Asus (e.g. Ascent GX10), AMD (e.g. Strix Halo) and others. Most of the steps here will directly apply to any Linux-driven AI training and inference machine.

The GalixiHub AI agent support interfaces to a Large Language Model (LLM), image/video/giphy generation, modification and processing, image text reading, location-based prompts, and prompts based on GalixiCom usages (i.e., worlds joined, members, chats and messages). To support these, the following features, and server settings are required on the AI server:

  • ollama or vLLM as the LLM runtime running over network port 11434. Note that this port can be changed as it is set when configuring an AI member.

  • comfyUI for image detection and generation hard-coded (for now) at port 8188.

  • The following comfyUI plugins are used: CheckpointLoaderSimple, CLIPTextEncode, EmptyLatentImage, KSampler, VAEDecode, SaveImage, ADE_AnimateDiffLoaderWithContext, VHS_VideoCombine.

  • pillow + flash for image modification hard-coded (for now) at port 5555.

  • tesseract for image text reading hard-coded (for now) at port 5555 as well.

  • Overpass for local map information hard-coded (for now) at port 30000. If Overpass is not installed locally on the AI device, it will fallback to a public option on the Internet, but this may be unreliable.

  • SearXNG for local web search queries hard-coded (for now) at port 30001. if SearXNG is not installed locally on the AI device, it will fallback to using DuckDuckGo, but this would then be an Internet operation which may slow prompt replies even further.

Once ollama is installed, at least one model must be installed in order for it to be of use.

Installing each of the above services and models is beyond the scope of this document but is explained in the Nvidia DGX Spark users-guide.

AI Agent Capability

The GalixiHub AI agent can facilitate chatting to an AI ‘bot’ who is the AI member, image (including gyphy and even short video) generation, image recognition, image modification and image text reading. It is able to handle any location-based prompts if the location-based services permissions are granted to GalixiCom by the device. For GalixiCom specific related items, it can respond to queries about worlds, chat groups, members and messages, even across worlds (e.g., “Please show me all my unread messages”). It is also able to sent reminders, presence triggers (e.g., “tell me when Bob is on-line”) and digests (e.g., “Please give a summary of the unread messages”). Additional capability will be added as the local AI hardware continues to increase in capability, models improve and new use cases are considered.

Creating AI Members

Creating an AI member is done, on an Android instance of GalixiHub, directly in the GalixiHub app by using the menu AI Join pull down from the World Status screen. It is also available to GalixiCom users who have owner or administrator access also via from the the menu AI Join pull down from the World Status screen. Details on how to create the member is explained for Android users at Android Create AI Member. and for iOS users at iOS Create AI Member..

It is recommended to create a chat with only a single AI member to avoid confusing the AI server and agent. Prompting the AI agent is done via sending a text or media message like to any other ‘real’ user. Based on the AI sever hardware capability and its current load, responses may be (much) slower than using enterprise-grade AI services such as ChatGPT.