So We Want to Write a Game Engine - Part 1 - Setting Expectations
As for any big project, the first thing to do when writing a game engine is to set the requirements. This will define the general direction of work and will guide all further decisions.
Right from the start we define compatibility and portability as the highest priorities for our project.
This also leads to an important decision: very strict limitations on data formats. While seemingly unfriendly, limiting support to a small number of the free and open formats with good cross-platform support (and some extra sane limitations) allows to ensure proper handling on all supported platforms.
Architecture
High-level architecture of our engine can be represented like this:
Let's dive deeper.
Graphics
The engine is made for 2D sprite-based games. So no per-pixel painting, no vector operations, no shading, no 3D transformations. Just painting bitmaps on the screen. This is already enough to make a huge variety of games, and most importantly -- this is enough to make (most of) the games we want to make.
Option: it may be cool to support arbitrary transformations on the sprites using matrices, but since this is not needed for our first game, I most likely will not implement it in the first version.
For our games we also need support for rich text. At the very least we need to be able to mix font families, sizes, weights and decorations.
The engine is going to handle anything painted on the screen as a layer. For initial release I plan these types of layers:
- Sprite -- a simple or animated sprite painted on the screen;
- Container -- a layer containing other layers; it's intended to be a convenient way for clipping and grouping layers and can be useful, for example in creating a GUI;
- Text -- a layer containing text; it will natively support everything necessary for rich text handling: formatting, wrapping, scrolling and clipping;
This approach will allow optimising rendering per layer type and renderer used. For the initial version I am also considering a special Tile layer. On one hand it will simplify making tile based games (including our games), but on the other it will make the engine more complex (i.e. harder to port).
Since our games will mostly be pixel-art, we will need some kind of scaling. Using layers will allow doing this transparently. Layers can be scaled independently, depending on their type.
I will likely hard limit resolution to something very large just to be safe. Minimum supported resolution will be 320x200.
For maximising compatibility, we are going to support both paletted and high/true color modes. We will allow using both color-key transparency (for all modes) and alpha transparency (for high/true color).
As I explained above, we are going to strictly limit the supported file formats. For graphical assets these will be PCX (compressed and uncompressed) and PNG (compressed) files. For our games we will likely use PNG in most cases, however uncompressed PCX will load noticeably faster on low-end computers.
Sound
With only 2D games in mind, we don't really need complex audio. No matter how cool it would be to support all kinds of advanced stuff like 3D positioning and environmental effects, at the first stage we can restrict ourselves to only support plain old mono or stereo samples and music.
Once again portability defines the need to limit file formats to the following:
- uncompressed PCM audio with 8 or 16-bit depth and 22050 or 44100Hz sampling frequency; container format is Wav only;
- compressed Vorbis audio with 8 or 16-bit depth and 22050 or 44100Hz sampling frequency; container format is Ogg only;
- For music we are also considering MIDI and tracker formats.
Uncompressed formats are a lot less CPU demanding, so can be used on slower machines; lower sound quality allows saving disk space and also reduces transcoding overhead on slower machines.
Input
This is the point where we cannot really afford any compromises. We have to support all mainstream types of input methods: keyboard, mouse and game controllers. Since input behaviour is highly game specific, we are not going to force any kind of input mapping on the engine level. Instead we are going to provide an optional and configurable library for this purpose. Obviously, we are going to optimise this library for our use-cases first of all.
Data
Usually game engines allow virtual file system approaches to allow seamlessly retrieving files from resource archives as well as local file system. This approach works well, but has an important caveat that file paths and file systems are in general not portable: some file systems are case-sensitive and some are not; file systems have different limits on length of the file name, number of files in the folder, allowed characters and so on. All of this not only forces the developer to constantly keep in mind the limitations of each of the supported systems, but also hinders porting to new systems (since their limitations may not be met by older games). More than that, it makes it extremely easy to make a game that only works in operating systems compatible with the one used by the developer.
It is also important to note difference between game resources, which should be read-only and user data which should be read/write. In general, writing files should be only possible within a specific "saved games" directory, but sometimes it is needed to make a file in an arbitrary place (for example if we are working with a map editor).
To resolve this I will:
- find or implement an archive format, suitable for both reading and writing data (alternatively, I consider using SQLite database for saving games. The downside of this is requiring understanding of SQL language for handling saved games and configuration, as well as an extra dependency);
- implement an extra API that will allow picking files from host file system. Obviously this will require some platform dependent UI, but most likely this will be necessary;
- implement a debug mode that will allow loading files directly from the host file system (this is necessary to avoid repacking step after every change).
Runtime
To make the game truly platform-independent it needs to be run within some kind of virtual machine. It can be an interpreted language, a byte-code VM or straight-away an emulator of a real CPU. I am heavily leaning towards Lua for this due to it's high portability and decent performance. Most of the engine's prototypes also use Lua. However I would like to at least mention some alternatives I am considering:
- Lua -- a big advantage will be runtime's native compatibility with INSTEAD, which will simplify porting of my older games;
- WASM -- a huge issue with it is a lot of different implementations and inconsistent tooling, as well as a compilation step required. Only worth it if performance is at least twice as good as Lua's;
- Pawn -- I know nothing about this language, except that it's compiled and it's designed for embedded systems. Once again, only worth it if performance is at least twice as good as Lua's.
Runtime choice is crucial since we will not be able to ever switch it. This means that I have to carefully consider upsides and downsides of each variant.
Target Systems
With the above said, we can set expectations for platform support. Realistically, using Lua as a runtime [1] will limit us to reasonably fast 32-bit machines. Of course a lot depends on the game itself. Ogg Vorbis music playback requires approximately Pentium MMX. High color graphics without acceleration is somewhere nearby as well (as a reference, A Dragon and the Tower with INSTEAD-9x can be checked).
For the initial release I plan to support:
- Windows 95-ME (build for 486 CPU with FPU)
- Windows 2000-11 (x86 builds only, running on x86 and x86_64 CPUs)
- GNU/Linux (Appimage builds for i686 and amd64, anything else will need to be built on-premise)
We are also considering (but probably at later points):
- Windows NT < 5.0 (we are not going to restrict the version, but we may use some APIs requiring a more recent OS)
- DOS (using HX-DOS; may just work right away, but I'm not sure if I have time to test everything properly for the first release)
- OS/2
- Atari TOS / MiNT (ARAnyM)
- AmigaOS (68K with RTG and PowerPC; will require quite fast machine) and AROS (x86-only)
- MacOS <= 9 (68K)
Obviously for some of the machines only enhanced variants or emulators will be suitable.
Summary
To summarise, this is what we have:
- I'm writing a game engine designed for 2D graphics with layer-based rendering
- a fully software renderer is required for maximum compatibility;
- an accelerated renderer is required (this will likely be OpenGL based, however I may also implement some platform specific renderers, for example DirectDraw on Windows);
- I will start with SDL 1.2 for it's wide platform support and once I implement what we need, I may add SDL 2 support if needed (it has some nice advantages, but has very limited platform support and also somewhat more resource intensive than 1.2);
- For audio I will use SDLMixer with the necessary limitations imposed
- for MIDI I may include some software synth when applicable, but I still need to find one; this will also require us to provide some bank, so I need to find something suitably licensed and sounding well;
- I will either find or implement an archive format that will be used for game data; it may also be used for user data or I will integrate SQLite for this purpose;
- For runtime I will evaluate a number of approaches and choose the one I find suitable
- most likely we'll just stay with Lua, but I'm not going to discard other options until I'm sure that they offer no advantage;
- runtime will be set in stone after release 1.0; while optimisations (custom or upstream) are possible, there will definitely be no changes to the language version ever (unless it's 100% compatible);
- For API a few things are important:
- anything unsupported should be explicitly forbidden [2];
- graphics API must obey the game's expectations (with reasonable tolerance) or explicitly fail;
- audio API must silently ignore errors and behave as if everything is ok, even if audio cannot be played [3];
- audio API must provide a list of supported audio formats, so that game can react if audio plays a significant part of gameplay;
- input API must provide at least one of the supported input methods; the engine startup must fail in case of no supported input available;
- I'll likely make a library to translate between input methods nicely; we are not going to force this in any way on the engine level, since handling of input is an important part of the game's UX;
- the engine should not provide any UI by itself, however a way is needed to transparently present platform-specific settings;
- Once the engine reaches release state, implementation becomes the source of truth, and instead of correcting the behaviour of the engine, we will correct the specification instead [4];
Further Steps
Currently I only have plans, a draft of the specifications and a prototype implementation using SDL and Lua.
As a next step I'm going to evaluate alternative runtimes and decide on storage API. After that we should be able to define the API specification and implement it enough to start working on a game.
We are going to open the source code as soon as we get the implementation to the Alpha stage, which is:
- API specification defines all necessary functionality;
- API is implemented according to the specification;
- we have one of the renderers (likely the software one) fully functional, however it does not have to be optimal;
- we will likely only support GNU/Linux and SDL 1.2 at this stage;
- we have some parts of our next game implemented in engine (tooling, gameplay mechanics, etc.);
Thank you for reading, as always I'm waiting for your feedback on email. Until we meet again!
| [1] | Assuming we use Lua, I have made a number of tests regarding Lua performance, and I'm not switching to anything slower. |
| [2] | This is required to prevent reliance on "undocumented behaviour". |
| [3] | Despite this contradicting the "explicitly fail" rule, in this case it's justified by requirement of portability and compatibility -- in most games audio is a nice to have rather than a hard requirement, so it's better to treat is as an option. |
| [4] | This decision is based on compatibility requirement; any games written with certain behaviour in mind will expect this behaviour, so we should not really change it; of course this does not correspond to obvious implementation errors (crashes, invalid rendering and so on). |