Add bootstrap gcc 4

1 files changed, 513 insertions, 0 deletions

diff --git a/content/bootstrapGcc/04_full_compiler.md b/content/bootstrapGcc/04_full_compiler.md
new file mode 100644
index 0000000..1c04e7a
--- /dev/null
+++ b/content/bootstrapGcc/04_full_compiler.md
@@ -0,0 +1,513 @@
+Title: Milestone — Source to Binary RISC-V support in GCC 4.6.4
+Date: 2022-06-20
+Category:
+Tags: Bootstrapping GCC in RISC-V
+Slug: bootstrapGcc4
+Lang: en
+Summary:
+    Description of the changes applied from a minimal compiler that runs and
+    generates assembly to something that is actually able to compile,
+    interacting with binutils and having a working libgcc.
+
+In the [series]({tag}Bootstrapping GCC in RISC-V) we already introduced GCC,
+and we already shared how I backported the RISC-V support from the GCC core to
+GCC-4.6.4. Now it's time to finish what we left half-done and actually
+introduce a *full* RISC-V compiler.
+
+### Where we left last time
+
+The Tuesday, 7th of April, I marked a commit with the `minimal-compiler` tag.
+That commit contains all the work we did until that time. In that tag we
+describe how we can build a compiler that is only able to assemble files to
+RISC-V.
+
+As we already explained around here, GCC is a driver program that calls other
+programs to do its work. The GCC core compiles the code to assembly language
+and then calls binutils to do the rest of the work: assembly and linking.
+
+At that point, we had to call binutils by hand.
+
+
+### The changes
+
+The changes applied at the time of writing are available in the
+[`working-compiler`][working-compiler] tag. As the tag message describes, they
+were split in two different branches: the `guix-package` branch and the `riscv`
+branch.
+
+[working-compiler]: https://github.com/ekaitz-zarraga/gcc/releases/tag/working-compiler
+
+The `guix_package` branch is merged in the `riscv` branch but this split lets
+us differentiate which changes are related with the compiler itself and which
+are related with the tooling around the compiler. That way we'll be able to
+choose what to do with the commits easily in the future. We'll probably need to
+rearrange some stuff.
+
+
+### The context is everything: Guix package part
+
+The `guix_package` branch contains all the commits that make the Guix tooling
+around the project work. This includes the compilation process definition in a
+reproducible way, the environment setup and all that.
+
+As the `working-compiler` tag message describes, this is the way you can
+currently make this compiler work and play with it:
+
+``` bash
+$ guix shell -m manifest.scm
+$ source PREPARE_FOR_COMPILATION.sh riscv64-linux-gnu
+ # This second command will prepare the PATH and other environment
+ # variables to make GCC find libraries and executables
+```
+
+> If you use this in the future and it fails, it might be because between the
+> time this blog post was written and you read it Guix made some changes in the
+> core packages that are used. You can always use the `time-machine` utility to
+> make sure you use everything like in the moment this post was written:  
+> `guix time-machine --channels=channels.scm -- shell -m manifest.scm`
+
+
+From this point you can directly run the compiler, it will need the `sysroot`
+option to be able to find the `crt*` files, but that's something I'm not
+worried about at this point, we'll fix that when we integrate this in the
+bootstrapping process.
+
+Run the compiler like this now:
+
+``` bash
+$ riscv64-linux-gnu-gcc --sysroot=$GUIX_ENVIRONMENT [-static]  ...
+```
+
+#### Notable changes in the Guix side
+
+The most notable change in the Guix side is the addition of the `manifest.scm`
+file and also the `PREPARE_FOR_COMPILATION.sh` file. With the help of my man
+Janneke, I realized the problems I had came from the fact that I was calling
+the compiler with the wrong environment and it was unable to find the linker
+and the assembler. Yes, this kind of things happen a lot in Guix if you are not
+careful (and I am *not* careful at all). Adding these tools let me prepare a
+working environment where the assembler and the compiler are found and called
+properly.
+
+This change also includes the some interesting extras: the GLibC added to the
+manifest also contains the static version so we can generate static binaries
+that are easier to test in an emulated environment without having to deal with
+the dynamic linker. Important stuff.
+
+Also, now the compilation process relies on a newer Guix version, which removed
+the `-unknown` part from the triplets (actually *quadruplets*), like
+`riscv64-unknown-linux-gnu`. That was a little bit of a pain, because I just
+tried to compile everything one day and failed, and in the end it was just that
+small change. I decided to update the Guix version needed to keep it up-to-date
+with the current Guix, so I didn't need to run `guix time-machine` each time.
+It's better like this.
+
+If you want to read more about the change and see how fast Guix
+people helped me understand what was going on, [see this mailing list
+thread][ml-unknown][^guix]. I have also to mention that I needed to add a small
+change to my GCC to be able to work in the case the `-unknown` part was not
+added to it: adding `riscv` to `config.sub` was enough for that.
+
+[^guix]: Some people also spent time with me in the IRC. Thanks to all that
+  helped!
+
+[ml-unknown]: https://lists.gnu.org/archive/html/bug-guix/2022-06/msg00092.html
+
+I also fixed a couple of extra things but they are not really relevant for
+this. Having a working environment preparation is a nice milestone by itself,
+but we did some things more on the GCC side!
+
+
+### Road to a working compiler: The GCC part
+
+The changes in the `riscv` branch contain some commits, most of them are small,
+but they are really important. I have to say this is full of details I don't
+really understand, so I'll try to focus on those I actually do. The rest of
+them are simply things that happened to work in the end. You know, this is
+pretty old software and the project is too complex to understand it all...
+
+#### Memory models and fences
+
+First, before doing anything else, we mentioned in the previous post that the
+memory models were something we needed to review. We knew this because the code
+related to memory models was used in a couple of parts of the RISC-V code we
+copied from the GCC 7.5 codebase, but it was not available in GCC 4.6.4. That
+API simply did not exist back then.
+
+The commit [`71dc25d`][memmodels] removes the memory models from the code
+(which were already commented out but not solved), taking in account the most
+conservative approach: always add the `.aq` flag and the `fence` instruction.
+This is not optimal, but the performance penalty is negligible and it's not
+affecting the functionality.
+
+[memmodels]: https://github.com/ekaitz-zarraga/gcc/commit/71dc25d08354dead26180bd552c0c3e299b012cb
+
+I did not come up with this myself, as I mentioned in the previous post, I
+asked the maintainer of the RISC-V support of GCC (who is also one of the big
+names of RISC-V) about this and he gave me this solution.
+
+I also had to change the optabs a little bit, using `memory_barrier` instead of
+one of the more recent optabs. For this I just compared the code from the MIPS
+architecture and checked how it changed from the 4.6.4 to the 7.5, as I did for
+many other parts of this work. Easy-peasy.
+
+#### Wrong arguments in the assembler call
+
+As I mentioned in the Guix part, we were unable to call the assembler. This
+means we didn't uncover the assembler call was broken until we actually put it
+in the `PATH` and tried to call it.
+
+The commit [`7030067`][as-call] shows how I needed to make small changes in the
+way the assembler is called by GCC to ensure that it was called correctly.
+
+[as-call]: https://github.com/ekaitz-zarraga/gcc/commit/7030067e6aa54b44a2f2447d4e706e76bc88f696
+
+This issue was easy to fix, but not that easy to catch. First I found the
+assembler was complaining because it didn't understand the `-k-march` option. I
+spent some time realizing the problem was that those were to options that were
+merged together due to a lack of a space. Yes, the space in the end of the line
+**is relevant**.
+
+I directly removed the `-k` option from the `ASM_SPEC` because my assembler was
+considering it ambiguous. I don't remember where I copied this from but it
+works and I don't want to think about it ever again.
+
+
+
+#### Libgcc: the core of this change
+
+The biggest thing in this set of changes was the addition of `libgcc`, which
+is mandatory if you want to link your programs compiled with GCC. `libgcc` is a
+library GCC uses for complex operations: instead of generating the assembly
+code directly, it generates calls to `libgcc`, where those complex operations
+are defined. You can read further about those operations but they are not
+really relevant for this post, the relevant part is we need to add `libgcc` in
+order to have a working compiler.
+
+The GCC codebase has different folders for its different blocks, so it's
+not surprising to see there's a folder called `gcc` for the core and a folder
+called `libgcc` for `libgcc`. Anyone would expect that just cherry picking the
+commit that added the `libgcc` support to GCC 7.5 would be enough to have the
+backport ready.
+
+Sadly, life is a little bit harder than that.
+
+##### Cherry picking the libgcc support
+
+The first and easiest thing to do is to cherry pick the commit
+[`72add2f`][libgcc-commit] and pray. It looked plausible to make it work,
+because, if you look at the changes it makes, it's pretty well contained in the
+`libgcc/config/riscv/` folder and adds just a couple of lines to the
+`libgcc/config.sub` to make it find the `riscv` folder.
+
+[libgcc-commit]: https://github.com/ekaitz-zarraga/gcc/commit/72add2fa4c354af4bf8db0b8dcb50c5b076b3ae5
+
+The contents of the commit are pretty clear:
+
+1. Some assembly files that implement some operations
+2. Some header files and C code that implement other things
+3. Some weird files called `t-something`
+
+The first two types of files we can understand as the body of the `libgcc`
+support: the juice. The `t-something` files are what are called Makefile
+Fragments.
+
+The Makefile Fragments are the basis of the GCC build system. The files like
+`config.host`, also part of the commit, sets a variable, `tmake_file`, where
+all the `t-something`s are added so the compiler generator framework knows how
+to build the things according to the rules described in them.
+
+That's how GCC buildsystem works. Now let's talk about the problems.
+
+##### LIB2ADD iteration is broken
+
+First thing I realized when I did the cherry pick of the `libgcc` support was
+the whole thing did not build anymore. There was a crazy issue here.
+
+We are not going to talk about `LIB2ADD` variable yet, but we can see this
+small change, [`b9c7f39`][lib2add], affects it. The main issue here was the
+whole makefile system (`*.mk` files in `libgcc`) was iterating over the values
+of the variable wrong, because `libgcc` support commit was appending values to
+`LIB2ADD` instead of setting it. The `LIB2ADD` variable was set empty from the
+main makefiles, and appending to it was leaving an empty entry, so the
+iteration process was trying to compile an empty value.
+
+[lib2add]: https://github.com/ekaitz-zarraga/gcc/commit/b9c7f394b33a60c1e64191b0e31f0cf98d6a5f93
+
+This was superhard to debug, but this small change just made the whole thing
+compile and now I was able to test the whole thing further.
+
+##### Still broken
+
+But it was still broken. GCC didn't want to compile. Some weird errors
+appeared, mentioning something like the `extra_parts` were not coherent between
+`gcc` and `libgcc`. Weird.
+
+Reading `gcc/config.gcc` and `libgcc/config.host` I realized the use of the
+`extra_parts` variable and how it was certainly incoherent between the two
+files. But why?
+
+This led me to analyze the whole build system, comparing the RISC-V support
+with others. I realized here that the buildsystem is mixed in `gcc` and
+`libgcc` folders and it's extremely difficult to know what's the line that
+separates one from another.
+
+Apart from that, the buildsystem was unable to compile the `crt*` files,
+because it didn't know how to do it... The recipes were missing.
+
+This made me go for the most aggressive change possible,
+[`9c0f736`][aggressive-fix]: just copy everything from the
+`libgcc/config/riscv/` to the `gcc/config/riscv`, add the rules for the `crt*`
+files and make the `extra_parts` coherent.
+
+[aggressive-fix]: https://github.com/ekaitz-zarraga/gcc/commit/9c0f7364b89acb38ea3af1cbe1884059671b3c04
+
+Of course, this is not a good change, but it lets us try if the generated
+compiler is able to compile anything. *"I'll have time to clean this up later"*
+I thought.
+
+
+##### The buildsystem is just a pain in the butt
+
+Now I was able to compile the GCC, so I could try it for some things.
+
+I build a RISC-V cross compiler and tried to statically compile a small Hello
+World program. Errors appeared:
+
+``` unknown
+/gnu/store/gbvg2msjz488d4s08p57mb1ajg48nxlj-profile/bin/riscv64-linux-gnu-ld: /gnu/store/gbvg2msjz488d4s08p57mb1ajg48nxlj-profile/lib/libc.a(printf_fp.o): in function `_nl_lookup':
+/tmp/guix-build-glibc-cross-riscv64-linux-gnu-2.33.drv-0/glibc-2.33/stdio-common/../include/../locale/localeinfo.h:315: undefined reference to `__unordtf2'
+/gnu/store/gbvg2msjz488d4s08p57mb1ajg48nxlj-profile/bin/riscv64-linux-gnu-ld: /gnu/store/gbvg2msjz488d4s08p57mb1ajg48nxlj-profile/lib/libc.a(printf_fp.o): in function `__printf_fp_l':
+/tmp/guix-build-glibc-cross-riscv64-linux-gnu-2.33.drv-0/glibc-2.33/stdio-common/printf_fp.c:394: undefined reference to `__unordtf2'
+/gnu/store/gbvg2msjz488d4s08p57mb1ajg48nxlj-profile/bin/riscv64-linux-gnu-ld: /tmp/guix-build-glibc-cross-riscv64-linux-gnu-2.33.drv-0/glibc-2.33/stdio-common/printf_fp.c:394: undefined reference to `__letf2'
+/gnu/store/gbvg2msjz488d4s08p57mb1ajg48nxlj-profile/bin/riscv64-linux-gnu-ld: /gnu/store/gbvg2msjz488d4s08p57mb1ajg48nxlj-profile/lib/libc.a(printf_fphex.o): in function `__printf_fphex':
+/tmp/guix-build-glibc-cross-riscv64-linux-gnu-2.33.drv-0/glibc-2.33/stdio-common/../stdio-common/printf_fphex.c:212: undefined reference to `__unordtf2'
+/gnu/store/gbvg2msjz488d4s08p57mb1ajg48nxlj-profile/bin/riscv64-linux-gnu-ld: /tmp/guix-build-glibc-cross-riscv64-linux-gnu-2.33.drv-0/glibc-2.33/stdio-common/../stdio-common/printf_fphex.c:212: undefined reference to `__unordtf2'
+/gnu/store/gbvg2msjz488d4s08p57mb1ajg48nxlj-profile/bin/riscv64-linux-gnu-ld: /tmp/guix-build-glibc-cross-riscv64-linux-gnu-2.33.drv-0/glibc-2.33/stdio-common/../stdio-common/printf_fphex.c:212: undefined reference to `__letf2'
+collect2: ld returned 1 exit status
+```
+
+The most logical thing to do was to build a MIPS cross compiler and check if
+the same issue appeared. Of course, it didn't.
+
+Researching a little bit in the old GCC internals documentation, I found a
+couple of interesting things:
+
+<https://gcc.gnu.org/onlinedocs/gcc-4.6.4/gccint/Target-Fragment.html#Target-Fragment>
+
+- The `LIB2FUNCS_EXTRA` variable is the one that contains what it should be
+  compiled and added to `libgcc`.
+- **Floating Point Emulation** support is added by generating a couple of files
+  with some macros on top: `fp-bit.c` and `dp-bit.c`.
+
+Neither of those were used in the `libgcc` support we backported because the
+GCC buildsystem changed a lot since 4.6.4. In fact, there is a commit[^commit],
+much later than the 4.6.4 release, that removes the need to generate those
+`fp-bit.c` thingies.
+
+[^commit]: `569dc494616700a3cf078da0cc631c36a4f15821`
+
+The `LIB2FUNCS_EXTRA` variable was not used either, but somewhere in the
+makefiles I found `LIB2ADD` was set from it. It looks like the whole
+buildsystem changed from `LIB2FUNCS_EXTRA` to `LIB2ADD`, which was an internal
+variable in the past. I don't know.
+
+I just moved the `LIB2ADD` to `LIB2FUNCS_EXTRA` and set the floating point
+emulation in the `t-riscv` makefile fragment and hoped my work was done there.
+
+##### A huge pain in the butt
+
+It still failed, but at least now the `__letf2` symbol was found. The only one
+I needed to fix now was `__unordtf2`.
+
+I was disheartened.
+
+The `__unordtf2` name did not appear anywhere in the code, but building
+`libgcc` for MIPS had the symbol inside (I checked it with `nm`!). I had no
+idea of what was going on.
+
+I asked all my peers about this, and I was sent a program that was actually
+compilable and runnable (Janneke is a genius, someone has to say it!):
+
+``` clike
+#include <stdio.h>
+
+int
+main ()
+{
+  return printf ("Hello, world!\n");
+}
+
+int
+__unordtf2 ()
+{
+  return 0;
+}
+```
+
+Hah! Still, no solution, but it was a little bit of hope.
+
+This gave me the energy I needed to research further. This `__unordtf2`
+function comes from software floating point support but the makefile fragments
+in the `libgcc` folder seem to be correctly set...
+
+##### Moxie for the rescue
+
+MIPS architecture was too complex to be understandable for this humble human
+being so I decided to go for Moxie this time.
+
+[Moxie](http://moxielogic.org/blog/pages/architecture.html) is a really
+interesting thing. But we are not going to spend time on it, but in its support
+in GCC 4.6.4. Take a look to the files on both parts of the Moxie support: the
+`libgcc` and `gcc`:
+
+``` unknown
+gcc/config/moxie
+├── constraints.md
+├── crti.asm
+├── crtn.asm
+├── moxie.c
+├── moxie.h
+├── moxie.md
+├── moxie-protos.h
+├── predicates.md
+├── rtems.h
+├── sfp-machine.h
+├── t-moxie
+├── t-moxie-softfp
+└── uclinux.h
+
+libgcc/config/moxie
+├── crti.asm
+├── crtn.asm
+├── sfp-machine.h
+├── t-moxie
+└── t-moxie-softfp
+```
+
+As you can see, some things are repeated, and most of the files are located in
+the `gcc` part, which was not the case in the backported commit. I used this as
+a reference for a massive cleanup of the previous aggressive duplication and I
+ended up with this commit: [`703efe3`][cleanup]
+
+[cleanup]: https://github.com/ekaitz-zarraga/gcc/commit/703efe3e86e68fe05380e996943c831e7ad9a541
+
+But that wasn't enough.
+
+I also found that the `soft-fp` support did not come from the `libgcc`
+directory, but from the `gcc` one, so I needed to fix some makefile fragments.
+The reference on how to do that was located in `gcc/config/soft-fp/t-softfp`.
+This file described all the variables that I needed to set up to make the whole
+process find the software floating point functions to add (see how the function
+names are built with the `$(m)` variable? That's why I couldn't find where did
+the `__unordtf2` came from...).
+
+Those variables were set in `libgcc/config/riscv/t-softp*` files. I replicated
+them in `gcc/config/riscv` as in the Moxie target and added referenced to them
+to the `gcc/config.gcc` file, copying the lines I had `libgcc/config.host`. The
+process was still failing, as the variables were not found by the main
+makefile. I decided to hardcode them and give it another go, this time it built
+and I was able to build files and the weird errors did not appear anymore.
+
+I realized in the end that the reason why the main makefile wasn't finding the
+variables was because I was referring to the `t-softfp*` files through the
+variable `host_address`, as it was done in the `libgcc/config.host`. The
+problem was that variable was not available in the main `gcc/config.gcc` file
+so I had to make a beautiful `switch-case` to deduce the wordsize.
+
+With all this knowledge and with the help from the Moxie support I finally
+arranged a new commit, where I duplicated the files that I needed to duplicate,
+added the correct references to the makefile fragments and I even fixed some of
+the variables in the makefiles: [`f42a214`][final-cleanup]
+
+[final-cleanup]: https://github.com/ekaitz-zarraga/gcc/commit/f42a21427361fb2d6d8481d143258af3237fd232
+
+Yeah, all this was hard to deduce, because this buildsystem is really complex
+and makefiles are really hard to debug[^debug-makefile]. Also the fact that I
+don't understand why I need to replicate the `t-softp*` files in both places
+drives me mad, but I have to learn to deal with the fact that I can't
+understand everything.
+
+[^debug-makefile]: Try to run `make --debug` in a project of the size of GCC
+  and laugh with me.
+
+In these commits you can see I deleted references to `extra_parts` and some
+other things, too. The reason is simple: if other architectures don't need
+to set those variables, me neither. In the end, the `crt*` files were generated
+anyway.
+
+
+#### Other changes
+
+I also removed `-latomic` from the calls to the linker because it looks like it
+didn't exist back then (we'll see how this explodes in my face in the future),
+and fixed a couple of things more, but that's not really interesting in my
+opinion[^interesting].
+
+[^interesting]: The rest of the post is not really interesting either, but I
+  need to report what I did. It's just me fighting against myself and a very
+  complex buildsystem that could've been simpler and/or better documented.
+
+
+### Missing things
+
+There are many things missing still, but this some I won't even try because
+they are out of the scope of the project. Remember: **we just need to be able
+to compile a more recent GCC**, not the rest of the world.
+
+Some of the things I left might become mandatory in the near future as we do
+proper testing of all this. My goal here was to provide something that can run,
+and then I'll collaborate with the different agents in this bootstrapping
+effort to fix anything we need to reach the full bootstrapping support.
+
+There are few obvious things missing:
+
+- **Big Endian support**: `riscv64be-linux-gnu` support, basically (note the
+  `be` in the target name). I won't add this until we are sure we need it. It
+  shouldn't be difficult, I already found some commits in the main GCC where
+  this was added and they were simple.
+- **Specific device support**: we didn't add support for any specific device
+  yet, that's something we'll need to think about in the future, but we
+  probably won't add because it will make us maintain more code, and I don't
+  think generic RISC-V code is going to have issues in the majority of the
+  devices.
+- There are also **many commits that came after** the main port that fix some
+  relocations and some other things. Many of them are not really relevant,
+  because most of them are related with bugs that were introduced later, fix
+  things that won't change anything in the only program we need to build (GCC)
+  and so on. In order to know which ones are relevant we need...
+- **Proper testing!** I didn't do this yet, and I'll probably need help with
+  it. Compile your RISC-V software with this and give it a try! Send me the
+  errors you get!
+- **Libatomic**: was directly removed from the calls to the linker, as I
+  mentioned before and we have to make sure it didn't exist back then and so
+  on. Boring things...
+- I didn't even bother to add the **testsuite support**, our only test has to
+  be if we are able to compile GCC with this, which I didn't really try yet
+  anyway (because it needs some extra things).
+
+### Conclusion
+
+This part of the project came in the worst moment. I wasn't really motivated
+and I had some personal things going on. It was difficult for me to do this.
+
+In contrast with what I did in the previous steps of the project, this part is
+really uninteresting because it doesn't give you a lot of chances for learning,
+which is the only thing that keeps me alive at this point.
+
+It's also pretty boring and exasperating to feel you'll never understand
+something and trying and trying almost in a *trial and error* way is really
+boring for someone like me.
+
+Sometimes, working like this makes you feel really alone. You have almost no
+people to help you, and the project needs a huge amount of context to be
+understood so you can't ask for help to *anyone*, and those who are supposed to
+know are really hard to reach. Or what it might be worse: maybe there's none
+that understands this thing well, because it's old, it changed a lot and
+probably just a handful of people do really took part in the development of the
+<del>fucking</del> buildsystem.
+
+In conclusion, this is boring and uninteresting job, but someone has to do
+this, and... It was my turn this time.
+
+You go next.