Hello. I have made a GBA game before using butano and have general game dev experience with Godot. I want to learn to make a gba game engine but I don't know how to begin or how to go about it. I am not very good at cpp and had a tough time making the game with butano but I am looking to learn.

Hi! This is a kinda huge question but here's some tips I could recommend:

1. Get familiar with a systems programming language of choice (C and C++ are most common but there's also Rust, Zig, Nim, Ada, etc.)

2. Read Tonc to learn how to program the GBA hardware directly (IO registers, tiles, palettes, sprites, backgrounds, etc.) as well as general low-level embedded programming wisdom.

3. Read GBA Resource Management to learn how to make things like a palette allocator, object VRAM allocator, big map system with streaming tiles - if you don't yet know why this stuff is useful, try writing a game without them first.

4. Read Game Programming Patterns to learn about ways to structure your game code. Some things may be overkill (event queue) or not that useful on GBA (data locality) but in general this book is full of gold.

5. Have a game in mind, and try to develop the game + engine together. For example Butano Fighter was released alongside Butano, serving as a demonstration of everything the engine is capable of. One of the reasons why developers hate Unity is because Unity don't make games in their own engine! Instead of fixing the issues devs encounter, they're too busy building half-baked new features before abandoning them and moving onto the next shiny new feature.

6. Get familiar with a higher-level 'productivity' language of choice such as Python (but even compiled languages will do as long as you can wrangle files/strings/data easily in them) - this will be important for the next point.

7. Making an engine is also about making tools. In some cases (e.g. image converter, music converter) you can write a script that just reads a config file and calls an external tool (e.g. SuperFamiConv, mmutil). But then there's stuff like level design, for which there's no out-of-the-box GBA solution. A good level editor is Tiled, but you have to write your own converter from Tiled maps into level data (e.g. const C structs) that your game can understand. Not to say you need a level converter, for example Butano doesn't have one, but I mentioned it to illustrate how much non-GBA code might be involved in making an engine.

Thank you for your reply. I realized how mammoth of a task this is so I think I'll just try to get better at gba dev in general while making games with butano. But, I'll try to make my code modular and have my own tools to help me. Thanks.

Hi Bump!

Great topic and questions you raise here!

Allow me to share my two cents, for I'm new to the homebrew scene in the GBA as well.

I had that exact same question. And I'll share with you a thought that I read -- and I'm not sure, and forgive me for not searching, but I guess it was from Jonathan Harbour, in his excellent book that I now recommend to you as well: Nintendo: The Unoficial Guide -- and the gist was very simple:
-- You have a limited set of time to dedicate to your project: Where do you want to spend it? Executing your game idea, or creating a one shot game engine, that you and only you will actually care for it, even if you put the most intricate algorithms in it, you optimize it in 30% of the nanoseconds.. Well.. you spent all your time, and voila! you've the most spectacular game engine.. And your game is probably underdeveloped.

In the essence, the GBA is just a fine piece of electronic, and like any piece of electronic, it starts in the - of the battery, and ends in the + of the same battery; basically a computer, and like a computer, you need to understand it's interrupts, and control them, to control the flow of the electrons so the magic can happen.
What we all should be asking was how to I control those switches:
And to know all those switches and interrupts, or to know the important ones, read https://problemkaputt.de/gbatek.htm.

And finally use Assembler to do that. And in this forum you've got pretty good masters at it like @Velipso (if memory serves, his devlog to inky is an assembly primer)
Or you can soften the proccess, by using C, that just compiles everything to machine code and achieve the same goals in an easier way.
If you're into OOP (object oriented programming), go for C++, that is basically C on steroid, oriented to the OOP paradigm.

Allow me to go slightly off topic: To my students, I usually tell them that there's nothing that you can achieve with C++, that is not possible with C, except that magnificent OOP syntax :-) In the end, 90% of the programming languages you know, have libc underneath; so why not use C already? They save a lot of time, my students quickly reply. Do they, I snarkly get back. Return to me 5 years from now, when you're presented with a thing called library dependency hell, your project must go to production in two days, important unit tests are failing because of a bug in a dependency, so you must upgrade it, but your project depends on a bigger dependency that not compatible with the newest version of dependency B, but if you don't do, it'll break dependency C, etc. etc. etc.

These unproductive tasks are our day job. This is mainly the reason why I love GBA, retrotech, and basically, digital purism. Programming for GBA actually has a soothing effect on my day.

And now, returning to the GBA dev.
You've pretty good game engines and libraries that ease the tasks for you. Devkitpro is a complete solution; Butano is just wonderful, and is in C++. Personally, I'm using a library that eases a lot of processes and saves me a lot of time: libham, made by Emanuel “Tubooboo” Schleussinger. He implemented a very practical approach to the GBA game development process. It's not complete, and it doesn't seem that we're going to see more features than the already implemented, so it must be complemented by libhel (http://www.console-dev.de/project/hel-library-for-gba/), and other stuff custom made with the help of GBA Tech (I mean DMA programming)

For a more recent and complete approach, go for libtonc, or devkit pro.

@exelotl gave you a complete receipt of material for you to study, and must be in a folder in your hard drive.

Sorry for the lengthy response, only to demote you from going to a probably sterile adventure on reinventing the wheel.
I wish you the best of luck for your projects, and share your progresses! I too am starting to develop a project in GBA, and probably some more of the guys here; we'd love to know and learn how you're putting to use those GBA chips :-)