Documentation

Complete reference for Unreal AI Copilot v1.0 · Unreal Engine 5.4+

Installation

1 Install from the Fab Marketplace. The plugin installs to Plugins/UnrealAICopilot/ and is enabled by default.
2 Open your project in UE 5.4+ Editor — the plugin loads automatically.
3 Set your API key (see below), then open the chat panel via Tools → AI Copilot or the toolbar button.

API Key Setup

API keys are read from environment variables at runtime — they are never written to project config files.

Security: Because keys live in environment variables, they cannot be accidentally committed to source control.

Windows (recommended)

C:\> setx OPENAI_API_KEY "sk-..."

Then restart the editor for the variable to take effect. Use setx, not set — the latter only applies to the current terminal session.

Windows (GUI method)

Press Win + R, type sysdm.cpl, press Enter.
Go to Advanced → Environment Variables…
Under User variables, click New…
Variable name: OPENAI_API_KEY — Variable value: sk-...
Click OK, restart the editor.

Linux / macOS

          # Add to ~/.bashrc, ~/.zshrc, or ~/.profile:

          export OPENAI_API_KEY="sk-..."

Restart your terminal and editor.

Quick Start

Install

Install from Fab Marketplace. Enabled by default.

Set API Key

setx OPENAI_API_KEY "sk-..." then restart editor.

Open Chat Panel

Tools → AI Copilot, or click the toolbar button.

Start Chatting

Type a message, press Enter. AI responds in real-time.

Provider Settings

All settings: Project Settings → Plugins → AI Copilot

Setting	Default	Description
`Provider`	OpenAI	LLM backend: OpenAI, Anthropic, OpenRouter, GitHub Copilot, or Local
`Model Id`	gpt-4o	Model identifier sent to the provider
`Endpoint URL Override`	(empty)	Custom endpoint URL. Required for Local provider.
`API Key Env Var`	OPENAI_API_KEY	Environment variable name holding your API key

Generation Settings

Setting	Default	Range	Description
`Temperature`	0.3	0.0 – 2.0	Sampling temperature (lower = more deterministic)
`Max Tokens`	4,096	256 – 32,768	Maximum tokens per LLM response
`Max Tool Iterations`	50	1 – 100	Maximum tool-call rounds per message (legacy mode)
`Max Tool Result Chars`	16,000	0 – 500,000	Max characters per tool result. 0 = no limit.

Task Execution Limits

Setting	Default	Range	Description
`Max Discovery Iterations`	10	0 – 30	Read-only rounds during the planning phase
`Max Mutation Iterations Per Task`	15	1 – 100	Successful mutation rounds per task
`Max Total Round Trips Per Task`	30	1 – 200	Hard cap on total LLM calls per task
`Max Consecutive Tool Failures`	5	1 – 50	Auto-skip task after N consecutive failures
`Max Text Tool Auto-Parses Per Turn`	5	0 – 20	Text-format tool call auto-parse limit

Behavior

Setting	Default	Description
`Auto Include Editor State`	Enabled	Automatically include selected actors, level name, camera position
`Auto Include Viewport`	Disabled	Attach a viewport screenshot to every message
`Stream Responses`	Enabled	Show tokens as they arrive

Default Endpoints

When Endpoint URL Override is left empty:

Provider	Default Endpoint
OpenAI	`https://api.openai.com/v1/chat/completions`
Anthropic	`https://api.anthropic.com/v1/messages`
OpenRouter	`https://openrouter.ai/api/v1/chat/completions`
GitHub Copilot	`https://api.githubcopilot.com/chat/completions`
Local	`http://localhost:11434/v1/chat/completions`

Basic Chat

Open the chat panel (Tools → AI Copilot, or the toolbar button).
Type a message and press Enter to send (Shift+Enter for newline).
The AI responds in real-time with streaming text.
Click Clear to reset the conversation.
Click Cancel to abort an in-flight request.

Pinned Assets

Right-click any asset in the Content Browser and select "Add to AI Context". Pinned assets appear as chips above the chat input, giving the AI persistent awareness of the assets you're working with. Their metadata is injected into every LLM system prompt.

Viewport Screenshots

If Auto Include Viewport is enabled, a screenshot is attached to every message automatically. Otherwise, the AI captures the viewport on demand when it needs visual context.

Note: Requires a vision-capable model (e.g., GPT-4o, Claude Sonnet 4).

Conversation History

Click the history toggle button to open the side panel. Load or delete saved conversations. History is stored locally in Saved/AICopilot/Conversations/ as JSON files.

Switching Providers

Go to Project Settings → Plugins → AI Copilot.
Change Provider to the desired backend.
Update Model Id for that provider.
Set API Key Env Var to the matching environment variable.
For local models, set Endpoint URL Override.

Provider Setup Guides

OpenAI

1. Get an API key from platform.openai.com.

C:\> setx OPENAI_API_KEY "sk-..."

3. In settings: Provider: OpenAI · Model Id: gpt-4o · API Key Env Var: OPENAI_API_KEY

Anthropic

1. Get an API key from console.anthropic.com.

C:\> setx ANTHROPIC_API_KEY "sk-ant-..."

3. In settings: Provider: Anthropic · Model Id: claude-sonnet-4-20250514 · API Key Env Var: ANTHROPIC_API_KEY

OpenRouter

1. Get an API key from openrouter.ai.

C:\> setx OPENROUTER_API_KEY "sk-or-..."

3. In settings: Provider: OpenRouter · Model Id: e.g., anthropic/claude-sonnet-4-20250514 or openai/gpt-4o · API Key Env Var: OPENROUTER_API_KEY

GitHub Copilot

1. Create a GitHub Personal Access Token with the copilot scope at github.com/settings/tokens.

C:\> setx GITHUB_COPILOT_API_KEY "ghp_..."

3. In settings: Provider: GitHub Copilot · Model Id: gpt-4o · API Key Env Var: GITHUB_COPILOT_API_KEY

Requires an active Copilot Individual, Business, or Enterprise subscription.

Local Models (Ollama / LM Studio)

1. Start your local model server (e.g., ollama serve).

2. In settings:

Provider: Local
Model Id: e.g., llama3, codellama, mistral
Endpoint URL Override: http://localhost:11434/v1 (Ollama) or http://localhost:1234/v1 (LM Studio)
API Key Env Var: leave as default (not required)

Tool Reference — 23 Tools, ~177 Sub-actions

Tools are classified as Discovery (read-only, free iterations) or Mutating (counts toward task limit on success).

Category	Tools	Sub-actions
Editor	`capture_viewport`, `get_editor_state`, `get_selected_actor_details`, `editor_control`	16
Blueprint	`list_blueprints`, `read_blueprint_graph`, `manage_blueprint`, `manage_blueprint_graph`	30
Material	`list_materials`, `read_material`, `manage_material`	27
Actor & Level	`control_actor`, `manage_level`, `manage_lighting`, `access_property`	47
Geometry	`manage_geometry`	25
Environment	`manage_environment`	19
Splines	`manipulate_spline`	6
Asset & Project	`manage_asset`, `get_project_summary`, `read_source_file`, `list_source_files`	7+
Scripting	`execute_python`	—

Blueprint Tools

Tool	Type	Description
`list_blueprints`	Discovery	List Blueprint assets. Params: `path_filter`, `limit` (default 50).
`read_blueprint_graph`	Discovery	Read nodes, pins, connections, variables. Required: `asset_path`.
`manage_blueprint`	Mutating	Create/modify Blueprints. 18 sub-actions.
`manage_blueprint_graph`	Mutating	Create/wire/inspect graph nodes. 12 sub-actions + batch mode.

manage_blueprint Sub-actions (18)

Sub-action	Description
`create_blueprint`	Create from any parent class (Actor, Pawn, Character, Component, GameMode, etc.)
`compile`	Force compile a Blueprint
`add_variable`	Add a variable with type and metadata
`remove_variable`	Remove a variable by name
`rename_variable`	Rename an existing variable
`add_component`	Add a component to the SCS hierarchy
`remove_component`	Remove a component
`add_event_node`	Add an event node to the graph
`add_function`	Add a function graph
`add_interface`	Implement an interface
`add_event_dispatcher`	Add an event dispatcher
`set_default`	Set default property values
`remove_event`	Remove an event node
`set_variable_metadata`	Set metadata on a variable
`add_construction_script`	Add construction script logic
`set_scs_component_transform`	Set SCS component transform
`set_scs_component_property`	Set SCS component properties
`get_info`	Get Blueprint information

manage_blueprint_graph Sub-actions (12)

Sub-action	Key Params	Description
`create_node`	`nodeType`, `x`, `y`	Create a graph node
`connect_pins`	`fromNodeId`, `fromPinName`, `toNodeId`, `toPinName`	Wire two pins together
`get_nodes`	`graphName`	List all nodes with pins and connections
`delete_node`	`nodeId`	Remove a node
`break_pin_links`	`nodeId`, `pinName`	Disconnect a pin
`set_pin_default_value`	`nodeId`, `pinName`, `value`	Set a pin's default value
`set_node_property`	`nodeId`, `propertyName`, `value`	Set node properties
`create_reroute_node`	`x`, `y`	Create a reroute node
`get_node_details`	`nodeId`	Detailed info for one node
`get_graph_details`	`graphName`	Graph-level information
`get_pin_details`	`nodeId`, `pinName`	Pin-level information
`list_node_types`	—	Available node types

Batch Mode: Pass an actions array instead of subAction to execute multiple graph operations in a single call. Top-level blueprintPath and graphName apply to all actions unless overridden per-action.

Node Type Shortcuts

PrintString Branch Sequence Delay ForLoop Self Timeline SetActorLocation GetActorLocation DestroyActor SpawnActor GetPlayerPawn PlaySound IsValid MakeVector BreakVector Add_DoubleDouble VariableGet VariableSet CallFunction Event CustomEvent Cast InputAxisEvent

Material Tools

Tool	Type	Description
`list_materials`	Discovery	List Material/MaterialInstance assets.
`read_material`	Discovery	Read expressions, parameters, shading model.
`manage_material`	Mutating	Full material authoring. 27 sub-actions.

manage_material Sub-actions (27)

Sub-action	Key Params
`create_material`	`name`, `path`, `blendMode`, `shadingModel`, `materialDomain`, `twoSided`
`set_blend_mode`	`assetPath`, `blendMode` (Opaque/Masked/Translucent/Additive/Modulate)
`set_shading_model`	`assetPath`, `shadingModel` (Unlit/DefaultLit/Subsurface/ClearCoat/etc.)
`set_two_sided`	`assetPath`, `twoSided`
`set_material_domain`	`assetPath`, `materialDomain` (Surface/DeferredDecal/PostProcess/etc.)
`add_scalar_parameter`	`assetPath`, `parameterName`, `defaultValue`, `x`, `y`
`add_vector_parameter`	`assetPath`, `parameterName`, `r`, `g`, `b`, `a`
`add_texture_sample`	`assetPath`, `texturePath`, `parameterName`
`add_material_node`	`assetPath`, `nodeType`
`connect_nodes`	`sourceNodeId`, `targetNodeId`, `inputName`
`disconnect_nodes`	`sourceNodeId`, `targetNodeId`, `inputName`
`remove_material_node`	`assetPath`, `nodeId`
`get_material_node_details`	`assetPath`, `nodeId`
`compile_material`	`assetPath`
`create_material_instance`	`name`, `parentMaterial`
`set_scalar_param_value`	`assetPath`, `parameterName`, `value`
`set_vector_param_value`	`assetPath`, `parameterName`, `r`, `g`, `b`, `a`
`set_texture_param_value`	`assetPath`, `parameterName`, `texturePath`
`get_material_info`	`assetPath`
`assign_material`	`assetPath`, `actorName`, `slotIndex`
`create_material_function`	`name`, `path`, `description`
`add_function_input`	`assetPath`, `inputName`, `inputType`
`add_function_output`	`assetPath`, `outputName`, `outputType`
`use_material_function`	`assetPath`, `functionPath`
`create_landscape_material`	`name`, `path`, `layers`
`create_decal_material`	`name`, `path`
`create_post_process_material`	`name`, `path`

Supported Expression Node Types

Add Multiply Subtract Divide Power Lerp Clamp OneMinus Constant Constant3Vector ScalarParameter VectorParameter TextureSample TextureCoordinate Fresnel WorldPosition Noise Panner ComponentMask DotProduct Desaturation Append Frac If Custom

Actor & Level Tools

Tool	Type	Description
`control_actor`	Mutating	Spawn/delete/transform/query actors. 27 sub-actions.
`manage_level`	Mutating	Level management: save, create, open, list.
`manage_lighting`	Mutating	Spawn lights, sky lights, build lighting, set exposure.
`access_property`	Mutating	Get/set any UObject property by reflection. 9 sub-actions.

control_actor Sub-actions (27)

Sub-action	Key Params
`spawn_actor`	`className` or `meshPath`, `location`, `rotation`, `scale`, `name`
`spawn_blueprint`	`blueprintPath`, `location`, `rotation`, `scale`, `name`
`delete_actor`	`actorName`
`set_transform`	`actorName`, `location`, `rotation`, `scale`
`get_transform`	`actorName`
`set_visibility`	`actorName`, `visible`
`duplicate_actor`	`actorName`, `offset`, `count`
`list_actors`	`classFilter`, `nameFilter`
`find_by_name` / `find_by_class` / `find_by_tag`	`namePattern`, `className`, `tag`
`add_tag` / `remove_tag`	`actorName`, `tag`
`attach_actor` / `detach_actor`	`actorName`, `parentName`, `socketName`
`set_physics`	`actorName`, `simulate`, `gravityEnabled`
`set_collision`	`actorName`, `collisionEnabled`, `collisionProfile`
`set_mobility`	`actorName`, `mobility` (Static/Stationary/Movable)
`rename_actor`	`actorName`, `newName`
`focus_actor`	`actorName`
`get_actor_details`	`actorName`
`get_bounding_box`	`actorName`
`get_components`	`actorName`
`add_component_to_actor` / `remove_component_from_actor`	`actorName`, `componentClass`, `componentName`
`select_actors`	`actorNames`, `addToSelection`

access_property Sub-actions (9)

Sub-action	Key Params
`set_property` / `get_property`	`objectPath`, `propertyName`, `value`
`list_properties`	`objectPath`
`array_length` / `array_get` / `array_set`	`objectPath`, `propertyName`, `index`
`array_append` / `array_remove` / `array_clear`	`objectPath`, `propertyName`, `value`

Procedural Geometry — 25 Sub-actions

Sub-action	Key Params
`create_box` / `create_sphere` / `create_cylinder` / `create_cone`	`location`, `size`/`radius`/`height`, `name`
`create_capsule` / `create_torus` / `create_plane` / `create_disc`	`location`, `radius`/`width`/`height`, `name`
`create_stairs`	`location`, `stepCount`, `stepWidth`, `stepHeight`, `name`
`boolean_union` / `boolean_subtract` / `boolean_intersection`	`actorA`, `actorB`
`subdivide`	`actorName`, `levels`
`simplify`	`actorName`, `targetPercentage`
`smooth`	`actorName`, `iterations`
`flip_normals` / `recalculate_normals`	`actorName`
`weld_vertices`	`actorName`, `tolerance`
`fill_holes` / `mirror`	`actorName`, `axis`
`translate_mesh` / `scale_mesh`	`actorName`, `offset`/`scale`
`auto_uv`	`actorName`
`convert_to_static_mesh`	`actorName`, `assetPath`
`get_mesh_info`	`actorName`

Environment & Volumes — 19 Sub-actions

Sub-action	Key Params
`create_trigger_box` / `create_trigger_sphere`	`location`, `extent`/`radius`, `name`
`create_blocking_volume`	`location`, `extent`, `name`
`create_kill_z_volume`	`location`, `extent`
`create_post_process_volume`	`location`, `extent`, `infinite`
`create_audio_volume`	`location`, `extent`, `name`
`create_physics_volume` / `create_pain_causing_volume`	`location`, `extent`, `damage`
`create_nav_mesh_bounds` / `create_camera_blocking`	`location`, `extent`
`set_volume_extent` / `get_volumes_info` / `remove_volume`	`actorName`, `extent`
`spawn_sky_atmosphere` / `spawn_exponential_fog` / `spawn_volumetric_cloud`	`location`, `density`, `color`
`set_post_process` / `set_fog_settings`	`actorName`, settings params
`get_environment_info`	—

Spline Tools — 6 Sub-actions

Sub-action	Key Params
`get_spline_points`	`actorName`
`set_spline_points`	`actorName`, `points`
`add_spline_point`	`actorName`, `location`, `index`
`set_location_at_point`	`actorName`, `index`, `location`
`set_tangent_at_point`	`actorName`, `index`, `tangent`
`remove_spline_point`	`actorName`, `index`

Editor Control — 16 Sub-actions

Sub-action	Key Params
`play` / `stop` / `pause` / `eject`	—
`undo` / `redo`	—
`save_all`	—
`save_asset` / `open_asset` / `close_asset` / `reload_asset`	`assetPath`
`console_command`	`command`
`get_selection`	—
`get_viewport_info`	—
`set_viewport_location`	`location`, `rotation`
`set_realtime`	`enabled`

Asset & Project Tools

Tool	Description
`manage_asset`	Content browser operations: `duplicate`, `rename`, `delete`, `create_folder`, `list_assets`, `exists`, `save_all`.
`get_project_summary`	Asset counts by type, enabled plugins, source modules, folder structure overview.
`read_source_file`	Read project source files (sandboxed). Allowed: `.h`, `.cpp`, `.cs`, `.ini`, `.json`, `.py`, `.md`, etc.
`list_source_files`	List source files. Params: `subdirectory`, `extension`.

Python Scripting

Tool	Description
`execute_python`	Execute Python scripts via the `unreal` module. Required: `code`. Optional: `execution_mode` (`execute_file` / `execute_statement` / `evaluate_statement`). Last-resort tool — the AI prefers dedicated tools for standard operations.

Requires: The optional PythonScriptPlugin to be enabled.

Architecture

Module Structure

Module	Type	Loading Phase	Role
`UnrealAICopilotCore`	Editor	Default	LLM providers, orchestrator, conversation, settings, tool registry, types
`UnrealAICopilotTools`	Editor	Default	18 tool handler implementations
`UnrealAICopilotUI`	Editor	PostDefault	Dockable Slate chat panel, tab registration, toolbar

Key Classes

Class	Type	Role
`UAICopilotOrchestrator`	UEditorSubsystem	Agent loop: message → context → LLM → tools → response
`UAICopilotToolRegistry`	UObject (singleton)	O(1) tool dispatch via TMap
`UAICopilotSettings`	UDeveloperSettings	All plugin settings, exposed in Project Settings
`FAICopilotConversation`	Struct	Message history, token budgeting, context window management
`FAICopilotToolDescriptor`	Struct	Tool name, description, JSON Schema parameters, category
`SAICopilotChatPanel`	SCompoundWidget	Dockable Slate chat panel with streaming and task progress
`IAICopilotLLMProvider`	Interface	Abstract LLM backend — implemented by OpenAI, Anthropic, Local

Orchestrator Engine

The UAICopilotOrchestrator implements a two-phase agentic workflow:

Phase 1: Planning (Discovery)

User sends a message via SendMessage().
Automatic context is injected (editor state, viewport screenshot, pinned assets).
A planning-specific system prompt is sent with the discovery tool catalog.
Discovery phase — the LLM calls read-only tools to gather project state. Mutating calls are blocked.
The LLM outputs a JSON task plan with approach and tasks[] (2–8 tasks).
If no plan JSON is detected (simple question), the response is returned directly.

Phase 2: Task Execution

Tasks execute sequentially in plan order.
Each task gets a scoped conversation with full plan status markers ([DONE], [CURRENT], [PENDING], [FAILED]).
Full tool access — all 23 tools are available.
The loop runs until the LLM produces a text-only response (task complete) or limits are reached.

LLM Providers

Provider	Default Model	Streaming	Vision	Tool Calling
OpenAI	`gpt-4o`	Yes (SSE)	Model-dependent	OpenAI function calling
Anthropic	`claude-sonnet-4-20250514`	Yes (SSE)	All Claude 3+	Anthropic tool use
Local / OpenRouter / GitHub Copilot	`llama3`	Yes	Model-dependent	OpenAI-compatible

Iteration Counting & Safety

Smart Iteration Counting

The iteration counter only increments on successful mutating tool calls:

Read-only tools (prefixed read_, list_, get_, capture_) → always free.
Failed mutations → free, the AI gets another chance.
Success detection → parses JSON result for success, error, errors fields.

Consecutive Failure Bail-out

After N consecutive failures (default 5), the task is marked as Failed and execution skips to the next task. The counter resets at each new task.

Cancel Support

Clicking Cancel immediately cancels the HTTP request, marks remaining tasks as Failed, and resets execution state. Checked at 7 async re-entry points in the pipeline.

Troubleshooting

Problem	Solution
"API key not found" in chat	Use `setx VAR_NAME "key"` (not `set`) to persist, then restart the editor.
No response from AI	Check Output Log for `LogAICopilot`. Verify endpoint URL and model ID.
Local model not connecting	Confirm server is running and endpoint URL matches (e.g., `http://localhost:11434/v1`).
Viewport capture returns empty	Ensure an active Level Editor viewport is visible and not minimized.
AI keeps failing	After consecutive failures the task auto-skips. Check Output Log. May need an optional plugin.
Model selector is empty	Click dropdown to trigger async fetch. For local models, ensure `/models` endpoint is supported.
Tool results truncated	Increase Max Tool Result Chars in settings (default: 16,000). Set to 0 for no limit.
Conversation history not loading	Check that `Saved/AICopilot/Conversations/` exists and contains `.json` files.

FAQ

Does it send my project data to external servers?

Only to the LLM provider you configure. Use a local model (Ollama/LM Studio) to keep everything on your machine. No other server ever receives data.

Are my API keys stored safely?

Yes. Keys are read from environment variables at runtime and never written to project config files.

Which model should I use?

GPT-4o (OpenAI), Claude Sonnet 4 (Anthropic), or Kimi K2.5 (OpenRouter). Vision features require a vision-capable model.

Can I use this completely offline?

Yes. Use Ollama or LM Studio with a local model, set the Local provider, and no internet connection is required.

Does it work with C++ projects?

The AI reads and reasons about C++ source files. It modifies via the editor asset pipeline (Blueprints, materials, actors) — not by editing C++ directly.

Can I undo changes the AI makes?

Yes. All operations go through standard editor commands and support Ctrl+Z undo. The chat panel shows exactly which tools were called.

What happens if a task fails?

The AI retries with a different approach. After 5 consecutive failures (configurable), it skips to the next task. You can also click Cancel at any time.

Does it work in Play-in-Editor (PIE)?

The plugin operates in the editor, not during gameplay. You can chat while PIE is running, but tool operations target the editor world.

← Back to Home