diff options
Diffstat (limited to 'content')
| -rw-r--r-- | content/windows2.md | 498 |
1 files changed, 498 insertions, 0 deletions
diff --git a/content/windows2.md b/content/windows2.md new file mode 100644 index 0000000..b151f04 --- /dev/null +++ b/content/windows2.md @@ -0,0 +1,498 @@ +Title: Guix + Zig + NSIS for the win...DOWS? +Date: 2023-12-15 +Category: +Tags: +Slug: windows2 +Lang: en +Summary: + How I made a program for Windows and GNU/Linux without touching any Windows + machine. The tools and the tricks to be effective (Zig and NSIS for the + win). + +Some months later, it's time to talk about [the post I made about writing +software for windows, without windows][win] because I put some of those things +in practice. + +I made a simple application with one external audio library (OpenAL) and simple +networking. No GUI this time, but some complexity was there. + +[win]: https://ekaitz.elenq.tech/windows.html + +### The program + +The program I'll discuss here is Karkarkar, a tool to read a Twitch chat out +loud in Basque. I made this for the Basque streaming community of gamers, which +is really cool but they had to rely on Text-To-Speech systems for other +languages as the main services for streamers don't include Basque in their +service, and the closest one, Spanish, is not good at some words' pronunciation +defer. + +Most of these people are gamers, and they use Windows, which I don't like, but +in the end they are also people and they deserve Free Software regardless of +the Operating System they decide to (or are forced to) use. + +I took this as a cool exercise to test all we talked about in the past, and as +a learning experience for the times I need to do something for a client that +requires Windows support or anything like that. + +#### The Text-To-Speech system + +I used a TTS library by Aholab, a research group from the Bilbao School of +Engineering, the university where I studied. They released AhoTTS (*ahots* +means *voice* in Basque) some years ago in Github, a working TTS library for +Basque and Spanish, with all the extra data it needs to work. + +The *only* problem it has is the AhoTTS codebase is a mess, with tons of +horrible decisions and undercover bugs. I had to fork the lib and make it work +more or less before adding it to the project (I won't discuss the issues here), +but I did. + +#### Connection to the Twitch chat + +This simply uses IRC to connect to Twitch chat. The protocol is simple, and I +only implemented an embarrassingly minimal amount of it. Enough to make it kind +of work. I hope to make more of this in the future. + +#### Playing the audio + +I started with `libao`, a simple audio library and then moved to OpenAL for +reasons I'll mention next. + +#### All together + +The program listens to IRC, when a message arrives it sends it to AhoTTS, +receives the samples to play and sends them to OpenAL which does its magic to +make it play out loud. + +Everything is the simplest thing possible as I didn't have a lot of time to +spend on this and also wanted to focus on the release process and being able to +make a package for windows, and some linux distributions. **Adding more code on +top of that is easy once the problem of the distribution is solved.** + + +### The tooling + +My strongest dependency[^1], AhoTTS, is written in C++, so I decided to go for +binary file distribution. And as [I discussed in aforementioned post][win-zig], +I was looking for an excuse to make a project in Zig, and its cross-compilation +capabilities could help me in this case, so I went for that. I made everything +with Zig 0.10.1, as that is the latest version packaged for Guix, but I was +forced to move to my Zig 0.11.0 package (see [Testing](#testing)). + +[^1]: There are not that many Basque TTS systems out there... You know? + +[win-zig]: https://ekaitz.elenq.tech/windows.html#zig + +For binary distribution, I didn't have many ideas when I wrote the post, but a +person in the Lisp Game Jam of that time suggested me to use NSIS for the +Windows installer. It was already packaged in Guix, my distro of choice, so I +just went for that, as it looked the simplest way to solve this. + +For testing all this, I relied in `wine`, what else could I do? + +So tl;dr: + +- I'm coding on [Guix](https://guix.gnu.org/). +- Programming in [Zig](https://ziglang.org/) 0.10.1 but then moved to + [Zig](https://ziglang.org/) 0.11.0. +- Installer using [NSIS](https://nsis.sourceforge.io/Main_Page). +- Testing done with [Wine64](https://www.winehq.org/). + + +### Keeping it small + +First of all, publishing software for Windows from Linux is painful if you have +to compile everything for it. Guix helps a little bit with that as you can use +`--target=` with mingw and have some luck. Sadly, many packages (most of them) +need `bash-minimal`, which [is not buildable for mingw at the +moment][issue-mingw]. + +[issue-mingw]: https://issues.guix.gnu.org/62226 + +In many software projects cross-compilation is not even possible. + +Knowing that I decided to keep my project as small as possible, because I +wanted to actually deliver something without losing my sanity. + +Each library you depend on you have to compile and deliver to your target, too. +It's not that common to have Windows users to install things by themselves as a +GNU/Linux person would. + +#### Audio library + +`libao` is the smallest audio library I could find, but I didn't manage to +build it for Windows myself, so I had to rely on something else. The +OpenAL-Soft maintainers have a binary distribution for windows, so I decided to +go for that one instead. It's way harder to use, and I had to do some weird +stuff to make it work, but it's easier for me: no need to build it myself. + +This is even more important in the case of having an audio library, interacting +with the system is always a pain in the ass, and trying to cross-compile a +library like this is not always easy, as their configure scripts are complex as +they need to check for many things. + +This is why I also avoided a GUI for the moment. Too much. + +#### AhoTTS + +AhoTTS is written in C++, it has zero dependencies and uses CMake as a build +system. + +At the beginning I compiled it using Guix to mingw, because it doesn't have +dependencies, it just worked. Later I encountered some issues though: +`libstdc++` and `libgcc` were not found at runtime and I had to statically link +them (`-static-libgcc -static-libstdc++` ftw). + +Later I started digging in its code, fixing some **horrible** things inside and +then I managed to add it as a submodule (yes, I hate that too) and compile it +directly with Zig. This is the best option so far: you can statically link the +library, it is built with the same process and it is cross-compiled by Zig. No +more missing library problems. + +> Extra: In order to interact with AhoTTS from Zig, I had to make [a small C++ +> to C bridge][c++-c], that I never did before. It's easy stuff, just convert +> the API a little bit, put some `extern "C"` on it and everything will go +> fine. `void *` is your friend. + +[c++-c]: https://github.com/ekaitz-zarraga/karkarkar/blob/master/ahotts_c/htts.cpp + +#### The rest of it + +That's just writing some actual code and making the program run. That's the +easiest part. + + +### Bringing it to the users + +One thing is making something that builds and runs in your computer and a very +different thing is making it work in other people's computers. And this is the +main problem that I wanted to discuss here. + +I have some requirements: + +- I made a terminal application, but I don't expect my users to know how to run + it. I need to make something that is click and run. +- I need it to have some icon in the desktop/startup-menu. +- The AhoTTS library needs some extra data files. These are searched by the + library and need to have some specific structure. These need to be installed + properly, too. +- I need to provide a simple way to uninstall the application. + +#### Windows + +The cool thing of this is I don't own a windows machine since ~2010 and I have +no access to any (living the good life!). I don't have any plans to change that +neither I have plans to really learn about Windows. So we have to be clever to +solve this. + +First things first: + +``` bash +zig build -Dcpu=baseline -Dtarget=x86_64-windows-gnu +``` + +Damn! That was actually very easy to do. Why isn't this the norm in other +languages? + +Of course, in order to do this I needed to provide a proper `build.zig` file +that was able to find the DLLs I was linking against and the header files. +That's not that difficult after all[^dll-names]. But has to be done. Still, +easy. Kudos for Zig. + +[^dll-names]: I still had weird issues with this. In Zig 0.10.1 DLLs where + found even if their name started with `lib`, but moving to Zig 0.11.0 + didn't find the libraries starting with `lib`. But Wine did find them so I + didn't know what to do. When I added the AhoTTS as a submodule the problem + disappeared, but not because it was solved, it was just avoided. + +Now, knowing where Windows searches the DLLs we depend on is important as our +installation process will depend on it. They have [some good documentation for +that][win-dll] which has a very interesting point: + +> Standard search order for unpackaged apps: +> ... +> 7. The folder from which the application loaded. + +This seems good enough for my purpose, so I can just install my binary and the +libraries in the very same folder. I thought I would need to install stuff +mixed somewhere else and learn a lot about windows, but I didn't have to. +Simple! + +[win-dll]: https://learn.microsoft.com/en-us/windows/win32/dlls/dynamic-link-library-search-order#standard-search-order-for-unpackaged-apps + +The extra data can be stored anywhere I want, because I am the one that +controls the search algorithm, but I still need to have easy access to it. I +decided to go for `LOCALAPPDATA`, but I'm thinking on putting it in the same +folder as the rest of the program. At the time of writing that's not done yet. + +##### NSIS + +Having clear where everything should be installed, it's time to make it +actually install it. + +NSIS is a great tool. The look of the vanilla installer is old-school but I +didn't even bother to use the modern interface because that required me to +think, activity I like to reserve to special occasions (like when I'm paid +extreme amounts of money for it or the time I'm in bed before falling asleep). + +NSIS in a nutshell: you write a script, run `makensis` with the script as an +input and, boom! You have a `.exe` that installs your stuff. + +It took me a little while to understand the structure of the script but once +you learn it is really easy (for the basic things, after that you can go as +hard as you want). It has two concepts you need to understand: Pages and +Sections. + +- Pages: define the different *pages* the user will navigate through during the + process. There are many pages pre-defined, and they cover all the basic + functionality: license agreement, component selection, installation directory + selection... +- Sections: define the different parts of your installation process. You can + mark some as optional or include them in different installation profiles + (all, minimal, recommended... You've seen this before). Then, if you use + `components` page, the sections will be listed to the user to choose which + one they want to install. You can also add sections for the uninstaller, + which will only be run when the user uninstalls the program. + +> EXAMPLE: I decided to include the sources also in the installer, but they are +> not required to run the program. It's as simple as adding a Section with the +> `/o` flag and they won't be automatically checked in the components step. +> Really cool stuff! + +The rest of the thing is just commands you can read in the documentation. You +can do many-many things with it. It has environment variables for almost +anything you'll need, so you don't need to hardcode things in the script. + +In my case I decided to ask the user for a configuration (the Twitch username) +during the installation using a custom page (this requires some digging in the +plugins' documentation, but it's not hard either), and created a launcher that +automagically inserts the username in the call to the program (not great for +later configuration, I know). This is done with a Batch file, which also makes +the dirty job of opening the terminal when it's double clicked. + +Here's the installer script I did, if you want to read it: + +<https://github.com/ekaitz-zarraga/karkarkar/blob/master/windows/installer.nsi> + + +#### GNU/Linux + +GNU/Linux world is really diverse, so it's not easy to know about every single +system's requirements. At the moment I stayed with Guix and Debian because they +are the only distros I use and I'm more familiar with them. + +In Windows I was asking to the user to add their username in the installation +process, but in Linux I don't have any simple (for the user) way to do it, so +for the moment I ask for the username when the program is called with no input +arguments. Ugly, but works for the moment. The goal was to deliver this thing, +not to make it perfect. I can do that later. + +The cool part is they provide different approaches for packaging: source vs +binary distribution. + +##### XDG standard: Desktop file and icons + +We need, of course, to arrange the same things we arranged in Windows: the +desktop icon and launching the terminal automatically. That's not hard to do +using XDG specification! + +Just add the `Terminal=true` line and it should open a terminal emulator when +clicked[^should]. The `Exec=` line in the desktop file has the program you want +to run, and it has to point to it correctly. + +[^should]: *"Should"* is the best word choice here, because it doesn't work for + some people, like in GNOME, as the terminals are hardcoded: + <https://gitlab.gnome.org/GNOME/glib/-/blob/main/gio/gdesktopappinfo.c#L2685> + +Once the `.desktop` file is done, we realize we need to deal with the icons. I +went just for a SVG icon, but I could add the rest. The only thing I needed to +do is put everything in the correct folder. Something like this: + +``` bash + usr + └── share + ├── applications + │ └── karkarkar.desktop + └── icons + └── hicolor + └── scalable + └── apps + ├── karkarkar.svg + └── karkarkar-symbolic.svg +``` + +Both Guix and Debian detect them properly once they are installed in the folder +where they expect them. + +##### Guix + +I wrote quite a few Guix packages lately so I'm pretty comfortable with this. + +I just need to tell Guix how it should build the program and put some files in +the proper directory. + +Also, I added the `zig-build-system` myself to Guix not that long ago. It's +pretty straightforward to use. + +The icons and the desktop file needs to put everything in the correct place, +`#$output/share/...`, and patch the Desktop file to point to the correct +binary, `#$output/bin/...` that is. For this, I kept a desktop file as a +template, with some reasonable defaults and just patched it in the Guix +package. That's easy. + +Not the best Guix package ever but it simply works, and that's everything I +want at this point. + +##### Debian + +Debian packages can be exported from Guix package definitions, but it exports +its file structure with every single dependency. In our case, that meant +hundreds of MegaBytes. That's too much so I did it manually, and the final size +was around 10 MegaBytes. Not bad. + +I never made a Debian package before, and I did the most minimalistic thing I +could think of. + +First, we need to build the thing: + +``` bash +zig build -Dcpu=baseline +``` + +And then just place everything in place in a folder, and call `dpkg-deb` on top +of it. In order to do that, I just wrote a bash script that did this whole +thing, it explains what I did way better than I can write in English: + +``` bash +# Run me in the linux/ folder +# I need the version in an argument, which should be Major.Minor-Revision +# for example 0.1-3. +version=$1 +outfolder="karkarkar-$version" +mkdir "$outfolder" +mkdir -p "$outfolder/usr/bin" +mkdir -p "$outfolder/usr/share/applications" +mkdir -p "$outfolder/usr/share/AhoTTS/" +mkdir -p "$outfolder/usr/share/icons/hicolor/scalable/apps" + +cp -r "../AhoTTS/data_tts" "$outfolder/usr/share/AhoTTS/" +cp "karkarkar.desktop" "$outfolder/usr/share/applications" +cp "../icons/karkarkar.svg" "$outfolder/usr/share/icons/hicolor/scalable/apps" +cp "../zig-out/bin/karkarkar" "$outfolder/usr/bin/" +patchelf "$outfolder/usr/bin/karkarkar" --set-interpreter "/lib64/ld-linux-x86-64.so.2" + +mkdir -p "$outfolder/DEBIAN" +cat > $outfolder/DEBIAN/control <<EOF +Package: karkarkar +Version: $version +Section: base +Priority: optional +Architecture: amd64 +Depends: libopenal1 (>=1.19.1) +Maintainer: Ekaitz Zarraga <blablablah> +Description: Karkarkar + Listen to a Twitch chat in Basque. +EOF + +dpkg-deb --root-owner-group --build "$outfolder" +rm -rf "$outfolder" +``` + +I need to highlight here the `zig build` I did automatically adds the Guix +dynamic linker to the binary, but that is not where the dynamic linker is in +Debian. I decided to patch the binary (`patchelf`) instead of trying to +configure the compilation process, I thought this would be easier. I don't know +if it was easier or not, but it was easy, so that's ok. + +Also note that the `DEBIAN/control` file has the bare minimum fields, but it's +enough to work. In Debian, everything is installed in `/usr/whatever`, but +that's the only detail I changed. + +Something I want to write down to remember later is the debian packages (`.deb` +files) can be extracted with `ar -x` and they have a couple of `tar.xz` files +inside. The `data.tar.xz` file has the file structure that will later installed +in the system. + + +### Testing {#testing} + +Yeah, you have to do it too. I tested it in Wine, but still needed to be tested +in Windows. I had a working version done in Zig 0.10.0 that was running well in +Wine, but it exploded in Windows because of [this issue][issue], that didn't +happen in Wine. I needed to use Zig 0.11 because of this error, which wasn't a +big deal anyway, because I already had it more or less packaged. + +So, yes, you have to test in Windows just in case, if you can. I have to thank +my friend (you know who you are!) for testing this program in his computer and +reporting the error. + +[issue]: https://github.com/ziglang/zig/issues/8943 + +### Conclusions + +The program itself it's really underdeveloped, but I actually made a lot of +progress understanding how to make a program reach users in different systems +easily. In the end, **being a solo developer forces me to be clever, and do +everything as simple as I can**. + +I don't normally have many dependencies because I believe software has to be +simple and tailor-made. This makes me control every aspect of the projects I do +but also greatly simplify the distribution. That's my context. + +I already talked about this in the [previous post on the subject][win], but in +the end, being able to do this relieves me from the "Web is Cross-platform" +mentality, which I don't think is a silver bullet, even though in some cases +might be a simple solution. + +I think developers today tend to avoid making native applications and software +quality suffers from that. There are many reasons for this (corporate control, +subscription systems, easy deployments, controlled environments...) but one +might be that the code people is used to write has too many dependencies, and +it is hard to package and distribute. I don't usually have that problem. I +already said I don't like having many dependencies. + +Now I have an easy way to do this, I can simply focus on writing the actual +code, which is, in the end, the most important part. + +Key points: + +- **Zig** happened to make this process really simple, just because the + developers decided to make it simple and they had the engineering skill to + back the decision. + +- **NSIS** gave me all I needed to put my application in a windows machine + without learning almost anything about Windows. I have the enough information + to make the thing work, not more. **NSIS** was the key I was missing. + +- **Guix** lets me cross-compile some of the dependencies to mingw, and I did + that at the beginning (and it worked!) so it's a powerful tool even for that. + +- **Wine** is a good way to see everything was working well, but I found some + discrepancies between it and Windows. + +The full problem is not completely solved yet. Finding a **GUI toolkit** I can +cross compile easily is still in the TODO list. But I'm pretty satisfied when +the result until now. + +> My colleague Andrius Štikonas talked to me about +> [MXE](https://github.com/mxe/mxe). I may try it in the future but I don't +> like the fact that it downloads things by itself. I leave it here just in the +> case I need it in the future. + +In fact, I think I can try other programming languages, even interpreted ones. +**NSIS** is surely capable of dealing with it. This opens a whole world of +possibilities for me. + +In the end, everything has been a very interesting process, making applications +targeted to the non-programmer (and non-gnu/linux) user has always been in my +mind. Now, I can say I almost solved the problem and I have the base to work +more on this in the future. + +Finally, the program is released in a very alpha stage in +[itch.io](https://ekaitz-zarraga.itch.io/karkarkar) where you can find the +installers and packages and the code is hosted in +[Github](https://github.com/ekaitz-zarraga/karkarkar/) until I get completely +mad and I delete my account entirely (which I have no doubt it will happen one +day or another). + +And that's mostly it, now I can focus on extending the behavior. |