CPX: CAOS Proxy For Cross-Language And Operating System Use

specification version number

	2212141732

license of specification

	Written starting in 2022 by contributors (see CREDITS.txt)
	To the extent possible under law, the author(s) have dedicated all copyright and related and neighboring rights to this software to the public domain worldwide. This software is distributed without any warranty.
	You should have received a copy of the CC0 Public Domain Dedication along with this software. If not, see <http://creativecommons.org/publicdomain/zero/1.0/>.

connection methods

	A. The client connects to localhost TCP port 19960 (or a host:port specified by the user).
	 Support for this method is MANDATORY.

	B. The client opens the (Windows-only) file "\\.\pipe\CAOSWorkaroundBecauseWindowsIsAFuckedUpPieceOfShit"
	 IMPORTANT: Yes, this is a named pipe, but it's accessible with file APIs too.
	 This is a workaround for Windows interfering with loopback TCP connections for non-Administrator tasks.
	 Support for this method is OPTIONAL.

	See the "standardized API design" section for selection method.

protocol

	1. The client connects via one of the given connection methods.
	   Client applications should try to AVOID concurrent connections to prevent spurious errors from overloading the recipient.
	   (In particular, keeping to a budget of one connection per process or application is advised.)
	2. The server will attempt to find an engine and lock the mutex.
	3. The server will then send the first 24 bytes of the shared memory (the header).
	   See http://double.nz/creatures/developer/sharedmemory.htm
	   The client is expected to read this before sending the request body, in order to check for errors and prevent TCP buffer deadlocking.
	   In particular, the client should check the "c2e@" signature (this is particularly helpful for cases where IO errors are silent).
	   The client need not check the capacity -- the server is expected to verify it in any case to prevent buffer overruns.
	4. It is legal for the client to close the connection here if the client has no request to make (i.e. if the client is simply querying the PID or connection state).
	5. The client will then send a little-endian 32-bit word indicating the length of the request data in bytes, and then send the request data.
	   The server must receive all data even on error, as otherwise the client can deadlock.
	6. The server will then perform the request (copy data, set size, signal request event and wait on response event).
	7. The server will send the entire response (the 24-byte header and then the amount of data bytes that header specifies).
	8. Finally, the server will close it's side of the socket and shutdown it's communications with the engine.

critical notes on the shared memory interface

	When sending a request, the C2E shared memory interface treats the request body as a null-terminated string.
	The null terminator is NOT automatically added by the CPX server.
	The C2E shared memory interface is designed as if it's sending back a null-terminated string, but it gets complicated in the presence of binary data.
	As said binary data may also contain zeroes. So basically it sends back whatever it wants and then appends a zero byte on the end.
	Finally:
		"c2e@" is at offset 0 in the header
		the 32-bit little-endian status code (1 for error text, which inhibits other output) is at offset 8
		and the response length (including null byte) is at offset 12

	All strings are theoretically in windows-1252 (Use latin1 if necessary but only as a consistent transformation).
	On Windows systems with alternative native configurations, this may not be the case.
	But languages with bad built-in encoding support such as Node.js complicate the situation.
	It is better to break in the presence of foreign characters than to break entirely if the configuration is foreign.

command set (standard)

	The normal C2E command handling essentially reads a null-terminated string, and that's the command.
	The command type is determined by the string-separated words of the first line.

	Commands are as follows:
	"execute\n" : Executes the (following) CAOS. Output is the result.
	"iscr\n" : Alias for execute
	"scrp X Y Z W\n" : Adds script with classifier X Y Z W, which follows. Don't add an "endm", it's implied.

command set (CPX extension)

	These commands are similar to engine commands, but they have stricter requirements on spacing as they're implemented in the CPX server.
	They are treated as "starts with this set of bytes", but will still expect null termination if they have a parameter.
	All CPX extended command start with the first four bytes "cpx-", and this is used to detect them.
	They are as follows:
	"cpx-fwd:" : This is used as an "escape command" to say you REALLY want to bypass this CPX server.
	             The rest of the command after the ":" is forwarded to the engine as if it were the whole command.
	             This doesn't quite apply if this server performs translation.
	             Then the command merely skips over the "CPX extension" part of the translator.
	             So "cpx-fwd:execute\n" will always have the same semantics as "execute\n".
	"cpx-ver\n" : Output is the CPX server version.
	"cpx-gamepath\n" : Output is the game path, i.e: C:\Program Files (x86)\Docking Station\
	                   This must be an absolute path.
	                   Applications using this are expected to provide an override mechanism due to Wine/virtual machines/remote configurations/etc.

	                   A suggestion for default behaviour if a Windows path (second character ":") is detected from an application running on a Unix-like:
	                   Convert "\" to "/" and lowercase the first character, then prepend "$HOME/.wine/dosdevices/".

	                   A suggestion for default behaviour if a Unixy path (no ":" in second character) is detected from an application running on a Windows-like:
	                   Convert "/" to "\" and prepend "Z:".

	                   Both of these behaviours should be overridable, they are heuristics only.

notes on extensions

	Extensions to this protocol, should they ever be required, should follow the format of the above extension commands.
	These pretend to be engine commands. It's not like the engine is ever going to change.
	This is why the standardized API design is careful about the raw request API.

	Do be aware extensions should not be considered a substitute for manual configuration.
	Returning the game path should not mean applications should only operate if the game path is correctly returned.
	***Extensions should absolutely not be abused to prevent manual configuration!!!***

standardized API design

	Following this part of the specification is not required but recommended.

	target selection

		A CPX library presumably has some way to select a specific CPX target (in the case of `libcpx.py` this is by passing a presumably already connected socket).
		A CPX library may have an *automatic* CPX target.
		Ideally, the automatic CPX target should be localhost port 19960, unless environment variables CPX_HOST / CPX_PORT are provided, which override these.
		However this is not required.

		The Windows named pipe file method is an additional method. It should be the preferred method if all of the following conditions are true:
		1. The application is running on Windows
		2. The application has not been given any host and port override
		3. The environment variable CPX_NO_PIPE is non-existent, empty, or "0"
		4. All IO operations on the named pipe file succeed (if any fail, retry the request the regular way)
		 THIS IS CRITICALLY IMPORTANT, AS NAMED PIPES DON'T HAVE A BACKLOG SYSTEM LIKE SOCKETS DO.
		 (caosprox spins up 32 threads to hopefully prevent problems, and applications are being adjusted.)

		On Windows, the TCP method seems to be failure-prone due to what appears to be Windows interference.
		Hence the recommendation of the named pipe file method.
		In the event that the named pipe file method cannot be supported on Windows for an application:
		Authors are advised to recommend Administrator permission for the application and CPX server.
		This seems to tend to resolve the problem.

	raw request

		A CPX client library's basic requirement to be a library that acts as a client for CPX implies a way of making a request.
		This is the primitive that most applications will care about.
		There are applications that need access to the full details of the protocol (i.e. pre-request SHM state, details like the process ID).
		But if your CPX library isn't intended for writing advanced applications (i.e. CPX servers or translators), you don't need to expose a header reader/writer.
		You just need to be capable of sending CPX requests.

		A function for a raw request should look like one of two forms.
		For languages without exceptions or where they'd be more trouble than they're worth:
			CPXRawRequest(CPXTarget, bytes) -> (int, bytes)
		For languages with exceptions where it's not troublesome:
			CPXRawRequest(CPXTarget, bytes) -> bytes throws CPXError

		The input bytes here are expected to have the null terminator already attached (as CPX expects).
		Where applicable, the length of the output bytes must be exactly equal to the response length returned in the shared memory header - they describe an equivalent range.
		This means no truncating based on the null terminator.
		However, where the API is using exceptions, errors can have their null terminator stripped and be decoded as Latin-1.

server error handling

	It's possible for an error to occur in the CPX Server, rather than in the engine (say, if the game hasn't been started yet)
	Clients do NOT have to care about this. The procedure described below looks almost identical to an engine-originated error to a client.

	If an error occurs in the CPX Server or associated components, the server should attempt to mimic a standard request error.

	That goes as follows:
	1. If the server has not sent the first 24-byte shared memory header, it will send a faked header twice.
	2. If it has sent the first 24-byte shared memory header, and needs to send a response, it will send a faked header once.
	3. If an error occurs half-way through a transmission, the server can't do much about it (since such errors usually come in the form of illegal access exceptions).

	The resulting error message should always start with "caosprox: " to indicate it's true origin.

recommendations

	Note that a CPX server should NOT support multiple connections at the same time, but should use OS "waiting socket queue" facilities.
	This allows a CPX server to implicitly mediate multiple accessors to the CAOS interface.

	A CPX server is allowed to host on :: / 0.0.0.0, use a custom port, or target a specific game name if the user requests it.
	A CPX client is allowed to only support connecting to the server on 0.0.0.0.

design rationale

	CPX-W32 uses similar logic to that which Creatures Labs uses, i.e. finding the "Software\Gameware Development\Creatures Engine\Default Game" key.
	This should ensure that CPX applications taking the "default option" act consistently with Creatures Labs applications.

	The CPX mechanism was designed to use the shared memory interface as the Linux "CAOS Console" interface is frankly crippled.
	A particular problem with the Linux "CAOS Console" interface is the use of newline-followed-by-rscr ("\nrscr") as a terminator.
	The translation mechanism seems to be stable for realistic cases.

	All SHM functionality was included (in particular it's still possible to get the process ID over CPX) to try and avoid losing anything in translation.
	Out of band data such as the game path registry keys are not included (yet? not absolutely necessary given some regediting).
	The ultimate goal with this approach is to make Creatures Labs software under Wine operate on the Linux Docking Station.
	(This goal has in fact been achieved for enough software for it to have been worth doing.)

	Worth noting is that CPX-W32 works under Wine (by design, and also it was tested on that platform).

	All software interacting with Creatures in general should avoid .NET and .NET Core, as they're dodgy under Wine (.NET Core crashes, try UndertaleModTool).
	One would think simply running the software natively is the solution, but then it can't access Wine functions to poke at things like shared memory interfaces.

	CPX-W32, and presumably any future replacement CPX servers designed properly, are expected to work in any environment in which the Windows version of the game runs.
	The choice of a TCP socket was intentional to allow CPX to act as a bridge outside of this "Wine/Windows world".
	A particular case I've heard about was the use of a virtual machine to play Creatures on Mac OS X.
	This is a good example of where CPX's choice of a TCP socket is helpful, as it allows supporting software to run outside of the VM.

	In addition, the choice to have the server close communications between connections was intentional.
	This allows CPX to co-exist with official Creatures Labs applications (co-existence between applications seems to be the purpose of the mutex).

	Error handling mimicking engine errors avoids the issues of a fancy bespoke solution that inevitably people have to write code for.
	People already have to write code to handle CAOS errors so they can simply treat this as a fancy new as-of-yet-unheard-of CAOS error.
	No reason to make things more complicated than they are.

	Since the initial creation of this specification, CPX has undergone testing under real conditions, and problems have arisen.
	Mainly, issues with the loopback TCP interface that CPX uses for communication.
	As such, the specification has been extended to include an alternate communication method.

implementation status (from most important to least)

	CPX Server Windows : OK
	Example CPX Client : OK
	Storyteller        : HANDLED BY UPSTREAM (but delayed on upstream's end)
	CPX Server Unix    : BETA (still no \nrscr fixups, but it doesn't seem possible for this to naturally occur in valid CAOS)
	CPX -> SHM         : BETA

possible topologies

	Standard

	+------------+      |       +------------+         +------------+
	|            |     CPX      |            |   SHM   |            |
	| CPX Client | ------------ | CPX Server | ------- | engine.exe |
	| Any        |      |       | C,C++      |         | C++/CAOS   |
	+------------+      |       +------------+         +------------+
	                    |
	 Native World       |                 Wine/Windows World


	Remote

	+---------------+   |   +------------+      |       +------------+         +------------+
	|               |   |   |            |     CPX      |            |   SHM   |            |
	|  BrainInAVat  |   |   | CPX Client | --+--------- | CPX Server | ------- | engine.exe |
	|  C++ probably |   |   | Any        |   |  |       | C,C++      |         | C++/CAOS   |
	+---------------+   |   +------------+   |  |       +------------+         +------------+
	       | SHM        |                    |  |
	+---------------+   |                    |  |
	|               |   |                    |  |
	|  CPX -> SHM   | - | -------------------+  |
	|  C,C++        |   |                       |
	+---------------+   |                       |
	                    |                       |
	Local Wine/Windows  |  Local Native World   |         Remote Host, Wine/Windows World


	Creatures Internet Edition, etc (native Linux/Mac ports)

	+---------------+       +------------+      |       +------------+         +------------+
	|               |       |            |     CPX      |            |   SHM   | BrainInVat |
	|  C.I.E        |       | CPX Client | --+--------- | CPX -> SHM | ------- | and etc.   |
	|  C++/CAOS     |       | Any        |   |  |       | C,C++      |         | C++ prob.  |
	+---------------+       +------------+   |  |       +------------+         +------------+
	       | CAOS Console/Files              |  |
	+---------------+                        |  |
	| CPX Server    |                        |  |
	| for CIE       | -----------------------+  |
	|  Any          |                           |
	+---------------+                           |
	                                            |
	                             Native World   |         Wine/Windows World

