1 week ago
In this tutorial, we will try to understand the internal structure of the Arduino IDE and how the build process works. This is useful mainly for debugging, troubleshooting compilation issues and use the build tools with other programs such as VSCode.
This is really an attempt to document the structure of the IDE. As the whole structure is poorly documented (if you find some document that describes the internals of the IDE, please let me know!!), there is no guarantee that the information hereafter is fully correct. This is all based on my experience and observations, mainly under Windows, and a lot of googling. So please do not flood me with questions, I'm no specialist of the Arduino IDE, just a little curious.
This text is a technical article more than a real tutorial, but a couple of tweaks are given as well.
Here is a list of the software versions used in this tutorial:
Obviously, with other versions, the content of this tutorial may need to be adapted.
To start with, there are several ways to install the Arduino IDE:
1. Portable Gamebuino pre-configured Arduino IDE: simplest installation when starting with the Gamebuino, set up by decompressing the Gamebuino IDE archive into a folder. As of today, it is based on a portable installation of the Arduino IDE in version 1.8.5. A portable version of a more recent IDE could be setup quite easily. The advantage of such a setup is that everything is stored at the same place.
2. Arduino IDE installed as an application: classical installation when working with Arduino, set up by executing the official Arduino IDE installer. This kind of installation requires administrator rights and hence may not be possible.
3. Arduino IDE installed in a folder: classical installation without administrator rights under Windows, set up by choosing the ZIP file for non admin install on the official Arduino download page and decompressing it into a folder.
4. Web-based Arduino IDE: new style of IDE for Arduino, requiring the installation of a plugin and allowing to code in the web-browser. No description of this kind of installation in this tutorial, too far away from the other installations, and not likely to be useable with any other external tool.
Files are stored at different locations depending on the IDE installation. The location is difficult to follow, especially because several copies exist for some files and folders.
We will take the convention of using the slash
/ as path separator for all paths except the Windows-specific ones where the backslash
> is used.
Arduino IDE installation folder
In this section we look at the path where the Arduino IDE is installed. We will reference to it as
ARDUINO_IDE_PATH, located at:
C:\Program Files (x86)\Arduinofor a standard Windows installation or
Under this folder, we will find the default boards, libraries and tools:
ARDUINO_IDE_PATH/libraries: the built-in Arduino Standard libraries, either applicable to the Gamebuino board (e.g.
SD) or not (e.g.
ARDUINO_IDE_PATH/hardware: built-in Arduino boards and tools, more specifically:
ARDUINO_IDE_PATH/hardware/arduino/avr/cores/arduino: all AVR cores. A "core" contains the functions of the Arduino API for a specific group of chips, i.e. it provides the definition and implementation of classical functions such as
sin(). It contains, among others, the mandatory include file
ARDUINO_IDE_PATH/hardware/arduino/avr/libraries: some default built-in libraries for the AVR cores such as
ARDUINO_IDE_PATH/hardware/tools: the default tools including the AVR toolchain, where it is particularly difficult to understand what are the relevant files (e.g. many
includefolders in the tree).
ARDUINO_IDE_PATH/portable: only present for a portable installation, containing in turn the sub-folders
sketchbook, described in the next sections
Hardware packages folder in Arduino15
In this section we look at the folder named
packages. This folder contains the cores and tools from user-downloaded boards (installed with the Boards Manager in the IDE) and is either under
ARDUINO_IDE_PATH/portable for a portable installation, or under the mysterious
Arduino15 folder, even if normally hidden, may look familiar to you because it contains the preferences.txt file mentioned on the Preferences dialog of the Arduino IDE. The folder contains user preference files, the
staging folder (used when downloading a library) and the
Concatenated with the possible locations of its parent folder, the
packages folder, that we will call unsurprisingly
PACKAGES_PATH, should be located at:
~/Library/Arduino15/packageson Mac OS
ARDUINO_IDE_PATH/portable/packagesfor a portable installation
packages folder, we will find the user-downloaded boards, where each board is in a single sub-folder (
arduino for the Arduino SAMD 32-bits ARM Cortex-M0+, and
gamebuino for the Gambuino, both are necessary).
In each board sub-folder, we find again the same kind of files as in the Arduino IDE folder. The following files and folders are part of the Gamebuino META board:
PACKAGES_PATH/gamebuino/hardware/samd/1.2.1/platform.txt: definition of the tools and command options used in the build process
PACKAGES_PATH/gamebuino/hardware/samd/1.2.1/cores/arduino: main folder with the "core" definition of the Gamebuino, containing for example the applicable
PACKAGES_PATH/gamebuino/hardware/samd/1.2.1/libraries: low-level libraries adapted to the Gamebuino board:
SPI: the Serial Peripheral Interface, used to communicate with the TFT display
HID: USB HID library, allowing the board to become a USB Human Interface Device
Wire: the Wire library, used for communication with I2C / TWI devices
I2S: communication via the Inter-IC Sound (I2S) Bus, used for sound
USBHost: the USBHost library, allowing the board to become a USB Host accepting devices such as a mouse or a keyboard on the USB port
SAMD_AnalogCorrection: SAMD_AnalogCorrection library to set and enable the digital correction logic of the SAMD Analogic Digital Converter
The SAMD Board folder at
PACKAGES_PATH/arduino is apparently the basis of the Gamebuino board definition. It contains almost the same files with more board variants, and in addition it provides the tools, including:
PACKAGES_PATH/arduino/tools/bossac: Used for uploading the .bin to the board
PACKAGES_PATH/arduino/tools/openocd: Used for on-chip debug (I personally never used it)
PACKAGES_PATH/arduino/tools/CMSIS: Libraries of the vendor-unspecific Cortex Microcontroller Software Interface Standard (CMSIS) created by ARM (I have to try the fast DSP math functions)
PACKAGES_PATH/arduino/tools/CMSIS-Atmel: CMSIS vendor-specific files for ATMEL Smart-ARM (SAM) processors
PACKAGES_PATH/arduino/tools/arm-none-eabi-gcc/4.8.3-2014q1: the GNU ARM Embedded Toolchain with the GNU binutils plus GCC and many other libraries and include folders..
This folder requires some deeper analysis. In
PACKAGES_PATH/arduino/tools/arm-none-eabi-gcc/4.8.3-2014q1 you will find:
bin/arm-none-eabi-g++.exe: this is the compiler executable, part of the GNU ARM Embedded Toolchain. The strange name says that the tool generates object files compliant with the Embedded Application Binary Interface (ABI). The GCC compiler comes with a very long list of options on the command line.
lib/gcc/arm-none-eabi/4.8.3/include: basis include files containing, e.g.
stdbool.h, which may contain
#include_nextlines, saying to the pre-processor to seach the given file in the next include folder
arm-none-eabi/include: next folder with again system includes, e.g. another version of
arm-none-eabi\include\c++\4.8.3: C++ specific includes e.g. triggered by including
As a side note, to conclude this section, be aware that the
packages folder does include executables ! When installing a board, it is not clear that additional executables are downloaded. Under Windows, the location in
C:\Users[UserName]\AppData\Local may lead to strange effects (interferences with antivirus, difficulties with backup, etc). I would recommend to build a portable version of the Arduino IDE if issues arise.
In this section we look at the path where sketches are stored a.k.a. sketchbook, that we will name
SKETCHBOOK_PATH. This is the path set in the File > Preferences of the Arduino IDE in the field Sketchbook location. It is at
ARDUINO_IDE_PATH/portable/sketchbook for a portable installation, or by default in the
Arduino folder of the user documents folder, or somewhere else if the path was changed in the preferences.
In the sketchbook folder, in addition to the user sketches, we will find the Arduino libraries downloaded by the user, i.e. installed with the "Library Manager" in the IDE (menu Manage libraries..) at
Each library is in a single sub-folder (normally
Gamebuino_META for the Gamebuino), and in each library folder, the header files are in the
src sub-folder and examples, if any, are in the
Please note that the Gamebuino META library contains adaptations of several other libraries otherwise available with the Library Manager (e.g. Display ST7735 from Adafruit). These libraries are stored in
SKETCHBOOK_PATH/libraries/Gamebuino_META/src/utility. Looking carefully at these libraries, it can be noted that the main include files are always a little different in the original one and the one in
utility. For instance, the main include file of the Adafruit GFX Library is named originally
Adafruit_GFX.h, and renamed
Graphics.h in the Gamebuino folder. Similarly, the Adafruit ST7735 (TFT display of the Gamebuino) uses the
Adafruit_ST7735.h file and is renamed to
utility. As a result, no interference is expected by using both versions of a library.
ATTENTION: For some reasons, the folder of a library may be named
arduino_xxxxxx, where the
x are numbers, instead of the expected name e.g.
Gamebuino_META. The compilation will still work, but if an absolute path reference to the library is needed, then I would recommend deleting the
SKETCHBOOK_PATH/libraries folder and re-install all required libraries.
As we saw in the previous sections, there are plenty of possibilities to look for header files and tools, and multiple copies exist in the different folders.
The main selection principle is that user-downloaded tools/cores/libraries have priority over the default ones in the Arduino IDE installation folder.
For the Gamebuino, it means that the board definition is in
PACKAGES_PATH/gamebuino/hardware, the tools come from
PACKAGES_PATH/arduino/tools and the necessary libraries are in
SKETCHBOOK_PATH/libraries/Gamebuino_META. Hence, finally, except the
arduino-builder tool and the editor, nothing is used from the Arduino IDE installation, and it should not be necessary to download other libraries.
The arduino-builder does a pretty good job at identifying the files to include and chooses the correct tools transparently during the build process.
The best way to see how it works, and be sure about the applicable files, is to look at the verbose output of a compilation (see the next section). Alternatively, refer to the official Arduino wiki page.
In this section we look at some simple tweaks, that we will use later on in the tutorial.
Set the build path of the sketch
There is no way to set the path where temporay files are stored during compilation in the Arduino IDE. The path is chosen as a folder named
x are numbers) in the system-wide temporary folder (
C:\Temp for Windows), plus another folder named
arduino_cache_xxxxxx to store the compiled core.
The drawback with temporary folders is that the system may delete them, and the path changes all the time.
Instead of the temporary folder you can set a build sub-folder in your sketch folder in your preferences.txt file:
preferences.txtat the bottom of the Preferences dialog window of the Arduino IDE to open the folder.
preferences.txtfile (otherwise your changes will be overwritten when you exit the IDE)
buildas build path. Note that the build folder does not need to exist at that point, the build folder will be created at compilation time if not already existing except if you delete the folder while the IDE is started.
By the way, there is a foreseen variable to set the build path of the core, named
preferences.txt, but it seems to be buggy. The core is always built into a temporary folder.
Keep extra assembler and pre-processed source files after build
Normally the build process keeps only .o files (object files, i.e. compiled code before link pass) and .d files (text files listing the dependencies). Additional files can be saved as well, that can be important for debugging. To do so:
platform.txtfile, normally at
PACKAGES_PATH/gamebuino/hardware/samd/1.2.1. As explained in this file, you can either modify the
platform.txtfile directly or, if you prefer, create a
platform.local.txtfile in the same folder for some settings We choose here the first method for simplicity.
platform.txtfile and change the line with
For every compiled file, you will now find:
.iifile with the pre-processed code (
.ifor C), which can be used to debug complex issues with macros
.sfile with the assembler code, which can be used to check the result of optimization
Determine pre-defined macros and include files in the compilation of a sketch
In addition to the different
#define macros in the source code, some macros are pre-defined by GCC, depending on the environment and the language used. It may be necessary to know the pre-defined macros to better understand what is really applicable in the jungle of include files of the Arduino IDE.
It is possible to list the definition of such macros:
.cpp. For the rest of this section we will assume that this file is named
C:\Temp\empty.cpp, doing that under Windows.
arm-none-eabi-g++.exe -dM -E C:\Temp\empty.cppto get a list of the pre-defined macros in a non-specific environement. The
-Eoption tells GCC to use the pre-processor only, and the
-dMto display the macros. As explained on the list of pre-processor options, it generates a list of
#definedirectives for all the macros defined during the execution of the preprocessor. But because the file is empty, the output is limited to pre-defined macros.
Note that you can create a file with the output of the command (instead of just displaying it) by redirecting the standard output to a file, e.g. add
> output.txt 2>&1 at the end of the command. The
2>&1 is necessary to re-direct the standard error stream to the standard output (it works as well under Windows).
Now, it gives a long list of
#define instructions, but it does not reflect the complete set of options activated for the compilation of a sketch. The get a more precise view:
-o some_filenameoption (destination object file), remove the
-Isome_folderoptions (include files would pollute the output with additional macros), replace the string
-E -dM. At the end, under Windows for the user
bfx, the command will look like this (with some characters removed for readibility):
C:\Users\bfx\AppData\Local\Arduino15\packages\arduino\tools\arm-none-eabi-gcc\4.8.3-2014q1/bin/arm-none-eabi-g++ -mcpu=cortex-m0plus -mthumb -c -g -Os -w -std=gnu++11 -ffunction-sections -fdata-sections -fno-threadsafe-statics -nostdlib --param max-inline-insns-single=500 -fno-rtti -fno-exceptions -MMD -D__SKETCH_NAME__="Fractalino.ino" -DF_CPU=48000000L -DARDUINO=10808 -DARDUINO_SAMD_ZERO -DARDUINO_ARCH_SAMD -save-temps=obj -D__SAMD21G18A__ -DUSB_VID=0x2341 -DUSB_PID=0x804d -DUSBCON -DUSB_MANUFACTURER="Arduino LLC" -DUSB_PRODUCT="Arduino Zero" C:\Temp\empty.cpp -E -dM
Now the command shows the exhaustive list of macros that are not defined in the source code, composed of:
-DARDUINO=10808meaning that the Arduino IDE version 1.8.8 is used)
The same kind of trick can be used to list include files for a sketch:
-H(see previous paragraph)
The list of include files is displayed on the standard error stream, so if you want to re-direct the output to a file, make sure that you use the
2>&1 additional re-direction statement.
The output looks like this (absolute path and extra "
>" removed for readibility). The indentation with dots shows the include level:
. packages\gamebuino\hardware\samd\1.2.1\cores\arduino/Arduino.h .. packages\arduino\tools\arm-none-eabi-gcc\4.8.3-2014q1\lib\gcc\arm-none-eabi\4.8.3\include\stdbool.h .. packages\arduino\tools\arm-none-eabi-gcc\4.8.3-2014q1\lib\gcc\arm-none-eabi\4.8.3\include\stdint.h ... packages\arduino\tools\arm-none-eabi-gcc\4.8.3-2014q1\arm-none-eabi\include\stdint.h .... packages\arduino\tools\arm-none-eabi-gcc\4.8.3-2014q1\arm-none-eabi\include\machine_default_types.h ..... packages\arduino\tools\arm-none-eabi-gcc\4.8.3-2014q1\arm-none-eabi\include\sys\features.h etc
This is what I used to track the otherwise non-configurable includes of GCC. Alternatively, a less exhaustive list of include files can be deduced from
Optimize sketch compilation for speed instead of size
The standard flags for compilation include the
-Os option, which triggers optimization for size. This makes sense for Arduino in general,due to the limited flash size.
For games, it may be useful to optimize the compilation for speed. To do so:
- Edit the applicable
platform.txt file, normally at
- Change all occurrences of
If you are curious, the result of the optimization should be visible in the generated assembler code.
I hope you learned a couple of things in this tutorial. It may evolve as I discover new things, re-fine my understanding or feel the need to look at other parts of the Arduino IDE.
If you find inconsistencies (and I'm sure there are some) please do not hesitate to report it (here or on Discord)!
NEW 1 week ago
Very interesting stuff indeed. Thanks for the good work!
By the way, is there an actual reason for keeping the "-g" compile option? In practice, I am not aware of a way to exploit symbols for debugging in the Arduino world.
As a side note, I have installed a more recent gcc version on my main development machine (gcc 7.3 I believe), and I have noticed great improvements in code speed. Editing platform.txt allowed me to point to this version. The latest versions of Project88 were built with this compiler (which seems to break the emulation on the gamebuino web site).
You must be logged in in order to post a message on the forum