Thinking about localization/internationalization/globalization

June 4th, 2009

Currently I have been thinking very carefully about how to handle localization / internationalization / globalization (I’ll just call it localization for the rest of this post). Nereid (from irc.enterthegame.com; #fragbu) gave me a run down on his thoughts when it comes to dealing with localization. Originally, I had only considered localizing text, but he had considered localizing everything. Everything ranging from textures to sounds. When I considered that, I also agreed that this is a logical step in the way game engines should go perhaps. With the internet getting faster in more remote areas of the world, it is inevitable that people will be able to get their hands on your game. While supporting English is an obvious choice, it shouldn’t be the only choice. Certainly in countries like Japan, games that only support English don’t get very far!

However, there are two main problems that immediately come to mind if you are trying to localize everything. First of all, data look up can be a rather lengthy process and I had to put some limitations in the ways people could look up data. Secondly, because data look up is a lengthy process, physical data seeking could also be a problem.

The way Brute Engine handles data look up is by using URI (Uniform Resource Identifier) structure. URI’s have three strings in them, a package string, a group string and a resource string.

The package string represents the file name of the zip without the extension. All packages have the same extension (currently packages are just zips at this point, so for ease of development are just zip).

The group string represents a folder within a package. At first I would have liked to have an array of groups rather than just a singular one as this would have allowed developers to sub divide packages into clearer names. However for the sake of fast data look ups, I decided that only a single group would be best for now. Perhaps this may change in future, I’m not entirely too sure at this point since I have not actually bench marked the performance between having a single group and having multiple groups.

The resource string represents the file name within the package and folder. For now, the resource string does have an extension attached to it, although this may be dropped in favour of having a integer to represent the data that the URI. But at the moment, an extension is useful for development purposes.

Resources vary a lot depending on the resource type they are. Some resources may hold binary data, while some resources may hold human readable ASCII data. Others may be a combination of the two. Typically, the rule for this engine is that anything made by the engine & tools alone will be human readable XML data uncooked which can be cooked into a binary form (for faster loading and parsing) and everything else will be binary (since the data is created by another program, developers should have these stored in another file). More importantly, some resources are composed on multiple files.

A example of a complex internationalized resource, that I can think of right now, would be a sound resource. A sound file is composed of multiple files in order to provide internationalization.

  • One file stores the raw sound data as an array of 8-bit samples.
  • One file stores the sound sample rate and other sound related numbers.
  • One file stores the localized sub title text

In order to provide good localization, when Brute Engine goes to load this sound file, it must compose a virtual representation of the sound files that best represent the localization the user has chosen. This is only difficult because, not all localized versions of those sound files will contain those files. Since English is going to be the presumed fallback language, all English localized sound files would probably contain all three files. However, a French localized version of the sound file might only contain localized sub title text data … because the developer may not have hired someone to act the dialogue in French. So let’s say the user is French, and the sound is requested by a component to be played back. Thus when loading the sound file, Brute Engine will first load the French localized version, and if any other data is missing, it will then fall back through localized versions of the same sound file! (This is because Brute Engine doesn’t rely on just dual localization, it can handle variants of a language too, thus you may have French-Canadian localization which overloads a few things from the French localization which overloads many things [but not all] from the English localization).

Phew, that’s a bit of a mouthful and certainly a lot to keep in mind. In order to support this properly, I had to build some sort of a tree like structure to organise how this all works.

Lastly, to touch base on a subject that most people aren’t even concerned about is data seeking. Obviously with this sort of localization method, there is a lot of power and flexibility behind it. However, the data for a ’single’ sound file is distributed all over the place. Thus if the game was to be run off a CD/DVD/Blu-ray disc, loading times would be atrocious since the ROM drive would spend half the time just seeking to the right portion of the disc. However, since most consoles are region specific, it might be possible to simply cook the the fully localized development version into a single language specific version of the game. I certainly don’t know the dynamics of the cost involved with distributing language specific versions of the game … but it would solve this issue rather nicely! On the PC this problem isn’t really apparent since hard drives are pretty good at seeking for random data at random points on the drive, although loading performance is often increased if the data is all aligned by defragging the drive.

Protected structs idea.

June 3rd, 2009

During my phases of learning C++, I was quickly informed that the practise of passing pointers around is a relatively unsafe thing to do. No matter what constant modifiers you attach to a pointer, the programmer can always delete what the pointer is pointing to. I’m sure many other programmers would argue that passing around pointers is just as safe as any other practise and that my overall approach to that topic is rather unjustified.

“Of course you shouldn’t delete it, no one needs to delete except the person who originally newed the object into existence.”

The problem with that reasoning is that, what happens if you branch out into a group project or if you are picking up someone else’s code. Also what happens when the rules are slightly bent, in that the ownership of the newed objects aren’t the object’s or function’s. Then the above reasoning goes out the window.

So typically, what I do is just return constant references to an object. For example,

const Object& GetObject()
{
  if (!m_object_ptr)
    Assert("m_object_ptr is null.");

  return *m_object_ptr;
}

This ensures that this object is relatively safe from being deleted by other objects or programmers. Still, this is a problem when you need to get an array of objects. Certainly if you are using std::vector or any other types of arrays that can’t handle copying constant references or structures, then the whole idea does fall apart.

I was pondering exactly how to handle this situation for quite some time, and I thought of using a protected struct. What this means is that the struct would contain pointers to the objects I want to send to another class, but the other class can only access these pointers as a constant reference because the struct would hide these pointers from the other class. For example,

struct ProtectedObject
{
  ProtectedObject::ProtectedObject(Object *object_ptr)
  :
  m_object_ptr(object_ptr)
  { };

  const Object& GetObject()
  {
    if (!m_object_ptr)
      Assert("m_object_ptr is null."); return *m_object_ptr;
  };

  private:
    Object *m_object_ptr;
}

This sort of structure hides away the pointer from the other class, but still allows itself to be copied into a std::vector, and thus you can now pass an array of pointers around with the same safety net.  It’s hard to tell if this particular idea will slow things down tremendously because I have to create these structures, but hopefully it won’t and still allow me to keep everything within all my C++ projects safe. The ultimate benefit is to the users at the end of the day, as it means that my projects won’t crash and burn an unfortunate death … at least not intentionally anyways.

Researching deferred rendering again…

May 22nd, 2009

Now that I am very close to finishing my latest project which was Balding’s Quest Editor, I found out that I really enjoyed developing tool sets. From what people have been telling me, this is a rather rare facet of game development to actually enjoy. Apparently most people enjoy other areas such as game design, engine design and so forth. So with that in mind, I decided that perhaps it is best if I simply develop engines … and perhaps see where that leads me. Now that I’m very comfortable working two jobs (being a pharmacist here in New Zealand and developing software for Digital Confectioners), I suppose I can afford to simple develop an engine over a reasonably long period of time.

Because I have worked with Unreal Engine 3 for such a long time, I decided that I would try to emulate and expand on many of the tools they have created for their licensees. One of the things that really impressed me about Unreal Engine 3 was that it was always trying to focus on artists to get their content into the game as easily as possible. However, I considered that perhaps I should try to focus on game designers / story writers instead of artists. Perhaps Unreal Engine 3 was designed more for them … but it does feel very artist orientated to me.

So the first tool that I would like to develop is a similar material system that they have done. This could lead to some potentially very complex materials being run. So I went forth and decided to experiment with a deferred shading system in OpenGL. As some of you may know, I experimented with this earlier in DirectX quite a few months ago. The experiment was somewhat successful but it wasn’t as flexible as I would have liked.

So after a weekend of bashing out some code, I had my OpenGL deferred rendering system going. So far the only cool thing it can do is render a mesh with some normal mapping applied to it. But none the less, it’s a renderer that will allow me to start work on a material editor.

Deferred Rendering Immediate Output

This is the four rendering targets that I am using at the moment. The data held is diffuse, normal, position and depth. Nothing terribly complicated just yet.

Deferred Rendering Output

This is what the final lit rendering looks like (using a directional light). Note that this actually uses a diffuse and normal map to produce the rock like surface. It’s not fancy, but it’s enough to start work on a material editor.

Back to the drawing board for Hexlo’s component system

April 28th, 2009

About a week or so, I was discussing about how I was pretty happy with Hexlo’s new component system. Unfortunately, as most designs go … this design didn’t quite work out that well.

The problem came when I had to do rather complex systems such as compound meshes. A mesh contains vertex data which define position data, texture coordinate data and color data; triangle data which define which triplets of vertices form each triangle and a texture id. Compound meshes are just when you have multiple meshes together. The reason why you need compound meshes is when you have meshes that require more than a single texture.

Why this became a problem was because the amount of data that was having to be transferred between the component, then to the register became very large per frame. This reduced my frame rate to a crawl. The system works wonderfully when you have small amounts of data which was the case for components such as sprites, lines and points, but once the components became very complex it became exponentially harder.

I had already predicted that data duplication would happen with this design, but I did under estimate the magnitude of that.

The other problem with this component design had to do with caching. Because components had to stay very generic, there was no way for them to store specific data for OpenGL or DirectX. Because OpenGL and DirectX manage and store objects such as textures differently to each other, I had to write a generic system in which then both OpenGL and DirectX would then override to implement their own specific methods. While this worked reasonably well to an extent, the worst case scenario had never been simulated. Since I would need to do look ups per frame, I can imagine that once I start hitting a certain number of objects on the screen it would probably slow the game down quite considerably.

So with those in mind, it’s back to the drawing board I think. I know the component system is achievable, but perhaps I am looking at this the wrong way.

Rethinking the component system for Hexlo

April 19th, 2009

I don’t believe I have announced Hexlo on this blog yet, but never the less it is one of the games that I am writing at the moment. The important aspect about Hexlo is that it mainly serves as my highly experimental projects in which I inject a lot of my new experimental methods into. I find it a little easier to inject new methodologies into projects like this because of their scale. Anyways, in Hexlo I wanted to rethink about my component system again.

A component system is one where entities in the world has a list of components which then define how that entities looks like, sounds like, behaves like and so forth. For example, for a rocket entity you may have a mesh component which renders a rocket projectile mesh onto the screen, a sound component which plays back a “fssssssssh!” sound during the rocket’s flight and perhaps a script component which contains the logic for when the rocket contacts something. Component systems like this are designed for the end user in mind. It also helps code reusability and extending current classes rather than having to copy/paste sections of code for different classes that aren’t able to extend similar classes.

My first initial stab at a component system involved using a lot of RTTI (Run Time Type Information) which was functional at the time but certainly an unsafe method of doing things. It is unsafe to me, because you assume a few things. For example, consider:

if (component_ptr->GetType() == Enum_Mesh_Component)
  ((MeshComponent*)component_ptr)->RenderMesh();

You assume a few things with this sort of code. First, you assume that GetType() will always return the correct value for each class definition you write.

  • What happens if you forget to override the virtual function in your child class? Make it abstract you say?
  • What happens if you return a conflicting identifier to another class?
  • What happens if component_ptr is a child class of a child class of Component?

And lastly, your code will be littered with gigantic switch and if statements that may get out of control. Lastly, you simply assume that component_ptr is going to be MeshComponent. If it isn’t, then it is going to explode as you try to access a function from a class that doesn’t actually have it. There is simply no real fail-safe methods here. I suppose you may interject and suggest that I could have used dynamic_cast<> to provide fail safe methods. The main problem with using dynamic_cast<> is that it is rather slow in relative measures.

Certainly when everything is written correctly, everything would be fine. But that’s like saying that a program will never have bugs in it because everything will be written correctly.

My second attempt at a component system involved interfaces. This assumed that every component is just a component and that typing was unnecessary. The base class, Component, would essentially have a large list of virtual functions that were setters and getters for everything imaginable. While this was very fail-safe and anything error related was able to be caught during compile time, the base class became very bloated with one off virtual functions that would eventually create such a large virtual table for these classes that the amount of memory used per class would be huge. For example:

class Component
{
  virtual void Set(const unsigned int &id, const int &in)  { };
  virtual void Set(const unsigned int &id, const float &in) { };
  virtual void Set(const unsigned int &id, const short &in) { };
};

As you can imagine, not every component would require all of these virtual functions but they would still incur the penalty of having them within their virtual tables. I couldn’t make these into abstract functions because then every component would need to implement them anyways making every child class simple massive.

So with Hexlo, I wanted to avoid that sort of a setup because I knew it would be a horrible system to work with. While it was very safe, it was memory hogging and too bloated to be usable. So when I thought about it some more, I realized that most of the time it was the entities themselves that often needed to alter the properties of their own components. So with that in mind, since the owners of these class instances were the entities themselves, keeping a private pointer reference to the component would be fine. End users would also not have to care too much about the memory management of these pointers, since a pointer list would manage these class instances.

class Entity
{
public:
  void AddMesh()
  {
    m_mesh_component_ptr = new MeshComponent();
    m_components_ptr.push_back(m_mesh_component_ptr)
  };

private:
  std::vector<Component*> m_components;
  MeshComponent *m_mesh_component_ptr;
};

So, with that using m_mesh_component_ptr is used when the class itself wants to change any parameters within it, but I would suspect that the majority of the time this isn’t really required at all. So that’s nothing special just yet, but in this particular case, let’s use this within a rendering scenario.

class Viewport()
{
public:
  void Render(MeshComponent &mesh_component);
};

class Component()
{
public:
  void Render()
  {
    InternalRender();
  };

private:
  virtual void InternalRender() = 0;
};

class MeshComponent : public Component()
{
  virtual void InternalRender()
  {
    GetViewport().Render(*this);
  };
};

*GetViewport() returns the Viewport class instance as a reference

This sort of structure pretty much resolves most of the issues I came across with the top two methods. I don’t have huge banks of virtual functions that hardly ever get used by many child classes, I don’t require RTTI since C++ itself handles typing and it’s pretty darn fail safe at most areas. The only potential problem that could possible happen is that the amount of function calls that is required could be a little on the excessive side causing potential problems but I haven’t come across this yet in Hexlo. The only other ‘problem’ is that you have to write quite a lot of virtual functions within Viewport for all of the different component types you have. However, I don’t think that’s any different to having to write the actual rendering functions for handling the component anyways.

I’m feeling pretty good about this method right now. But perhaps in a few weeks, I might come up with an even better solution!

How I handled undo and redo in Balding’s Quest Editor

April 7th, 2009

Handling undo/redo sounds like a really trivial task for any application. In fact, I suppose because it is so common in all the applications we use these days that we simply just expect that every application we ever use simple have those features. However, upon further delving it actually turns out to be a rather difficult thing to implement.

The main constraint with any undo/redo systems is memory. For naive undo/redo systems they would just keep a copy of each modified object within memory. For example, if I was to do this, a copy of each state of the level prior to change would exist in memory. Since each level in Balding’s Quest is of relatively small size I could potentially get away with doing this. After all, if each level was to say only use 20 kilobytes of memory, then it would fifty one level states before I reached a megabyte of memory. While that sounds like a lot of states available, it really isn’t much to be honest.

The first problem is scalability. If Balding’s Quest level formats were going to become more complex and thus use more space in memory, even if it was another 10 kilobytes larger then the number of states I have available per megabyte then becomes thirty. This is a pretty big drop considering a small increase in the size.

The second problem is going to be assuming that most operations will encompass broad changes. But in most cases with the editor makes rather small changes to the level at each time. For example, when a user paints tiles onto the levels that is one tile at a tile that is being changed. So painting fifty one new tiles onto the map isn’t hard to do at all.

So, well with that in mind, the naive approach is obviously not going to work. So after much brain storming, I decided that a history event logger could work. The idea behind this is that each time the level changes, I would record it as an event detailing what happened there. If a tile got changed, I would write in memory that “Tile at x, y was changed from palette index 0 to 1″. To undo the event I would just interpret the event backwards and then to redo I interpret the event going forwards.

This seemed to work in the end, but the only ‘problem’ is that I have to write undo and redo functions for every action that a user can do. However, it does resolve both issues I stated above with the naive approach. Perhaps in time I’ll come up with a better method.

Latest Balding's Quest Editor screenshot

Announcing Balding’s Quest Editor

March 14th, 2009

Balding's Quest Editor dated 13/03/09

Well, I’ve been secretly working on this for quite some time now. I picked this project up because I really enjoyed playing Balding’s Quest, and while it had a perfectly good level editor included … I sort of wanted to make one as well. I also wanted to continue refining my skills in C++ as well as wxWidgets and this was certainly a good project to do. This is because the scope of the project is within reason, and there wasn’t a lot of things that I would have to do research in. So far, the majority of the functionality is there and you can now build levels for Balding’s Quest quite easily in it. There are a few more core functions that I need to add, and countless features as well, but all in all it is shaping up to be a very good project. From here on in, I shall post up weekly updates and how things are developing. Development builds are going to be released weekly so I can get bug reports and usability issues resolved as quickly as possible. If you have any questions about BqEd or would just like to have a chat with me, you can always email me. Thanks for staying tuned in!

Huh? What time is it?

February 3rd, 2009

Hmm, it certainly has been a long time since I last posted here. As the new year passed, I began to think about the previous year and all the things that could have gone better. There are a lot of them. I’ve been some what busy writing them all up and hopefully will get some of them done to post here.

I’ve decided to drop the Viva La Donuta from production as well. While I was ’somewhat’ close to finishing the game, after reading a post mortem on GameDev, I think that Viva La Donuta suffers from many of the same problems. It’s a little too obvious that I simply wanted to try to make a quick buck by doing something within the casual market. I feel that it would probably hurt the company more than actually benefitting. The project however may be released under freeware or donationware though, so I’ll probably just clean a few things up and put it up for download.

Thanks for continuing to check this blog out. We’ve been working on some other things in the mean time, but I’ll save those when I develop them further.

The eight chapters have finally been finished!

December 23rd, 2008

Yes! Viva La Donuta has now reached the forty level mark, and all of the chapters are now finished. There’s still quite a bit left to finish off, but it’s great that the major set of levels are now done, as well as the game mechanics! All eight game mechanics that I wanted in the game are now in, and are working. It’s a both a relief and a shock.

The relief comes from knowing that the game is very, very close to finishing and a shock is from how long it took over all! I knew game development was long, but I did not expect game development for a casual game to take this long. Certainly when I write the postmortem I’ll be able to look over every aspect of how this game was developed and see what things I could have done better … as well as what went horribly wrong and so forth.

Still, I’m going to celebrate a little bit to have thus far.

ScryEngine’s localization is finished

December 23rd, 2008

In my last blog post I mentioned how that I hadn’t quite resolved partial localization yet, but after sleeping on it … I did dream up of a way to do it. That, and I had some help from #gamedev. The way I was doing it before was just using helper classes that can read and write to the file directly. I could perhaps make an array of instances of these helper class’s but then I think the number of times I need to read and write would probably be annoying and could cause issues for people who try to run the engine on anything else other than a hard drive. I suppose one could say that I really should just read these files and store them into memory in the first place … and well, I didn’t quite think of doing that until someone told me that I should!

So that’s essentially what I did to achieve the end results. I first read the base localization file, that is INT and within Engine or anything considered more ‘base’ and then reading the other files replacing the entries within memory. Thus within memory, I should have a composite version of the selected localization file. Huzzah! Problem fixed.

The next two parts that are on the list for ScryEngine is the configuration module and the time module. More information on these as I write the systems.