diff options
Diffstat (limited to 'content/clopher')
| -rw-r--r-- | content/clopher/03-TUI.md | 207 | ||||
| -rw-r--r-- | content/clopher/04-Native_interface.md | 273 |
2 files changed, 480 insertions, 0 deletions
diff --git a/content/clopher/03-TUI.md b/content/clopher/03-TUI.md new file mode 100644 index 0000000..fb5ad02 --- /dev/null +++ b/content/clopher/03-TUI.md @@ -0,0 +1,207 @@ +Title: TUI: A look to the deep +Date: 2019-05-30 +Categories: Series +Tags: Clopher - TUI Gopher client +Slug: clopher03 +Lang: en +Summary: Research and work on the Terminal based User Interface of Clopher, the + gopher client. + +> Read all the posts in [Clopher series here]({tag}clopher-tui-gopher-client). + +This software have been introduced as a Gopher client but, as you can probably +deduce from the previous post, the Gopher part is probably the simplest one. +The complexity comes with user interaction. *People are hard*. That's why we +are going to delay that as much as possible, trying to cover all the points in +the middle before we *jump to the unknown*. + +Just joking. In fact, we have to shave many yaks before thinking about user +interaction anyway. This text talks about them. + +### Are you talking to me? + +Let's remember we can classify programs by two different categories like this: + +- **Non-interactive programs**, often called *scripts*, are programs that take + an input and return an output. There's no interaction with the user in + between. An example of this could be the command `ls`. + +- **Interactive programs** receive user input while they run and respond to the + user while they are running. An example of this could be the machine that + sells you the tickets for the subway, it asks you where are you going, then + tells you the price, take your money and so on. All of this with the program + constantly running. + +Remembering what we talked about Gopher: it's a *stateless* protocol. There's +no *state* stored in the server so all the queries *must* contain all the info +related to them. *Queries are independent*. + +This, somehow leaves the door open to two possible implementations of Clopher. +The *non-interactive* one would work like `curl`. Getting the IP, port, +selector string and an optional search string as input it would open the +connection retrieve the result and return it.[^1] + +But Clopher is designed as an **interactive program**. More like `lynx`, where +you interactively ask for the pages and have a *local state* that records your +history and other things. This is a decision, it's not imposed by the protocol. + +### Shell*f boycott* + +There are some different ways to handle user interaction in TUI based programs. +The simplest one is to read by line, waiting until the user hits `ENTER` to +read the result. That's the behaviour of the classic `scanf` function of C and +many others like `input` in Python, etc. + +In programs like Clopher, where the design is similar to `lynx` or `vi`, this +kind of input makes no sense at all. The program needs to be able to capture +every key pressed by the user and perform action in response to them. For +instance, in `vi` when the user hits `i` in normal mode it needs to change to +insert mode and when the user presses `i` in insert mode it needs to change the +contents of the buffer. + +The design of these kind of programs is simple to understand, it's an infinite +loop[^2] where key presses are captured and they change the *state* of the +program. When the user hits the key combination that halts the program the +loop is broken. + +In simple C code the program would look like this: + + ::clike + #include<stdio.h> + + int + main(int argc, char * argv[]){ + char c; + // Create some state + + while(1){ + c = getchar(); + if( c == 'q'){ // Exit if user pressed `q` + return 0; + } + // Update state here + putchar(c); // Show the character for debugging + } + } + +Or the simplified Clojure equivalent: + + ::clojure + (loop [c (char (.read *in*)) + state (->state)] ; Create some state + + (when (not= c \q) ; Exit if user pressed `q` + (print c) ; Show the character for debugging + (recur (char (.read *in*)) + (update-state c state))) ; Update state + + +Looks simple, right? + +Wait a second, there's a lot of stuff going on under the hood here. If you run +the code in any POSIX compatible operating system (I didn't test on others, +and I won't) you'll find the code might not be doing what we expected it to: +The `getchar` (or `.read`) calls will wait until `ENTER` is pressed in the +input buffer and then they'll get the characters one by one. But we want to get +them as they come! + +#### Saints and demons — canonical mode + +In POSIX operating systems, the input is buffered by default. But that behavior +can be configured following the POSIX terminal interface under the name +**canonical** mode or **non-canonical** mode. The mode we are looking for is +the non canonical mode. You can read more about it in [the Wikipedia][canon]. + +Choosing the non-canonical mode has some extra options: one controls the number +of minimum characters to have in the buffer to perform a `read` operation and +the other defines the amount of tenths of second to wait for that input[^3]. +Choosing the right value for those fields (`c_cc[MIN]` and `c_cc[TIME]`) +depends on the kind of interaction we are looking for. + +#### Make Dikembe smile — blocking + +Setting `c_cc[TIME]` field to `0` means the `read` operation will wait +indefinitely until the minimum amount of characters defined with `c_cc[MIN]` +are waiting in the buffer. Together with that, the `c_cc[MIN]` can be `0` that +means the read operations will wait until there are `0` characters in the +buffer, or, in other words, they won't wait. + +Be aware that both fields can provoke the read operations in the input buffer +be non-blocking operations and that will cause the read operation to return +with no value. + +In the case of Clopher, I decided to set the `c_cc[MIN]` to `1` so the read +operations block until there's at least one character in the buffer (that means +they will always return something) and the `c_cc[TIME]` to `0` so the read +operations have no timeout and will block until a character arrives. + +Depending on the application you are developing, you might choose other kind of +blocking configuration. For instance, setting a timeout can let you process +other parts of the system and wait for the input in the same thread. + +#### We're talking about practice? — termios + +So now we know where to find this theoretical configuration it's time to put it +in practice. In POSIX the standard way to access this is via `termios`[^4]. +It has some details that are not specified and depend on the implementation, so +it might have some differences from Linux to BSD or whatever. + +`tcsetattr` and `tcgetattr` calls can be used to set and read the terminal +configuration via termios. Check this example, compile it and compare it with +the C code of the previous example: + + ::clike + #include<stdio.h> + #include<termios.h> + + int + main(int argc, char* argv[]){ + // Get interface configuration to reset it later + struct termios term_old; + tcgetattr(0, &term_old); + + // Get interface configuration to edit + struct termios term; + tcgetattr(0, &term); + + // Set the new configuration + term.c_lflag &= ~(ECHO | ECHONL | ICANON | IEXTEN | ISIG); + term.c_cc[VMIN] = 1; // Wait until 1 character is in buffer + term.c_cc[VTIME] = 0; // Wait indifinitely + //TCSANOW makes the change occur immediately + tcsetattr(0, TCSANOW, &term); + + char ch; + while(1){ + if(ch == 'q'){ + // Set old configuration again and exit. + // If it's not set back the normal configuration of the + // terminal will be broken later! + tcsetattr(0, TCSANOW, &term_old); + return 0; + } + ch = getchar(); + putchar(ch); + } + } + +All the code has enough comments to be understood but there are some weird +flags it's better to check in termios documentation.[^4] + +### But this is C code and Clopher is written in Clojure! + +I know but this is becoming long and boring. Why not wait until I get some +spare time and write the next chapter? You have tons of information to check +until I write it so you won't be bored if you don't want to. + +See you next. + + + +[^1]: In fact, you can navigate the Gopherverse like this with `curl`. +[^2]: Unsurprisingly called *main loop*. Programmers are very creative. +[^3]: That `read` operation is what `getchar` is doing under the hood. +[^4]: `man termios` or visit [online `man` + pages](https://linux.die.net/man/3/termios) + +[canon]: https://en.wikipedia.org/wiki/POSIX_terminal_interface#Input_processing diff --git a/content/clopher/04-Native_interface.md b/content/clopher/04-Native_interface.md new file mode 100644 index 0000000..08a8e52 --- /dev/null +++ b/content/clopher/04-Native_interface.md @@ -0,0 +1,273 @@ +Title: TUI Slang: Speak like the natives +Date: 2019-06-15 +Categories: Series +Tags: Clopher - TUI Gopher client +Slug: clopher04 +Lang: en +Summary: Interfacing between Clojure, Java and Native code. + + +> Read all the posts in [Clopher series here]({tag}clopher-tui-gopher-client). + +The previous post introduced `termios` as a native interface to configure the +terminal input processing. With termios we managed to make our C programs get +input character by character processing them as they came with no buffering but +we didn't integrate that with our Clojure code. Now it's time to make it. + + +### Run before it's too late + +Before we dig in the unknown, I have to tell you there are other alternatives +for the terminal configuration. The simplest one I can imagine is using +`stty`[^1] as an external command. I learned this from [Liquid][liquid], a +really interesting project I had as a reference. If you want to see this work +check the `adapters/tty.clj` file in the `src` directory of the project.[^2] + +Of course, it has some drawbacks. `stty` is part of the GNU-Coreutils project +and you have to be sure your target has it installed if you want to rely on +that. I'm not sure about if it's supported in non-GNU operating systems[^3]. + +In my case, I decided to stay with `termios` interface to deal with all this +because I didn't really want to rely on external commands and it's supposed to +be implemented in any POSIX OS. The good (bad?) thing is it made me deal with +native libraries from Clojure and had to learn how to do it. + + +### The floor is Java + +When dealing with low-level stuff we have to remember Clojure is just Java, and +most of the utilities we need to use come from it. This means the question we +have to answer is not really *"how to call native code from Clojure?"* because +if we are able to call native code from Java, we will be able to do it from +Clojure too (if we spread some magic on top). + +#### So, how to call native code from Java? + +First I checked the [Java Native Interface (aka JNI)][jni], but I thought it +was too much for me and I decided to check further. Remember there are only a +couple of calls to make to `termios` from our code, so we don't really want to +mess with a lot of boilerplate code, compilations and so on. + +My research made me find [Java Native Access (aka JNA)][jna] library. If you +check the link there you'll find that the Wikipedia[^4] describes it as: + +> JNA's design aims to provide native access in a natural way with a minimum of +> effort. No boilerplate or generated glue code is required. + +Sounds like right for me. Doesn't it? + +I encourage you to check the full Wikipedia entry and, if you have some free +time at the office or something, to check the implementation because it's +really interesting. But I'll leave that for you. + +##### A lantern in the dark + +JNA is quite easy to use for the case of Clopher, even easier if you realize +there is [lanterna][lanterna], the TUI library, out there, using it internally +so you can *steal*[^5] the implementation from it. Lanterna is a great piece +of software I took as a reference for many parts of the project. Digging in the +internals of large libraries is a great exercise and you can learn a lot from +it. + +First of all, like many Java projects, the amount of abstractions it has is +crazy. It takes some time to find the actual implementation of what we want. +This isn't like this for no reason, the reality is they need to create this +amount of abstractions because the part of the library that handles the widgets +can work on top of many different terminal implementations, including a +[Swing][swing] based one that comes with Lanterna itself. + +Clopher only targets POSIX compatible operating systems so we can go directly +to what we want and read the termios part directly discarding all the other +compatibility code. This code is quite easy to find if you see the directory +tree of Lanterna: there's a `native-integration` folder in the root directory. +If you follow that you'll arrive to [`PosixLibC.java`][posix-lanterna] that +uses JNA to interact with termios. + +The implementation provided by Lanterna is quite complete, they declare a +library with the functions they need and the data structure introduced in the +previous chapter. Once the library interface and the necessary data structures +are defined from Java they can be called with JNA, like they do in the file: +[`NativeGNULinuxTerminal.java`][nativegnu-lanterna]. + +#### How to call JNA from Clojure, then? + +Calling Java code from Clojure is quite simple because Clojure have been +designed with that in mind, but this is not only that. Thanks to the Internet, +there's a great [blogpost by Nurullah Akkaya][nakkaya] describing a simple way +to use JNA from Clojure. From that, we can move to our specific case. + +`termios` has its own data structure so we need to define it so the JNA knows +how to interact with it. The problem is that Clojure doesn't have enough OOP +tools to do it directly so we need to make it in plain Java. The good thing is +that we don't really need to create anything else. + +If we remove some unneeded code from Lanterna's termios structure +implementation it will look like the implementation I made at +`src/java/clopher/Termios.java`: + + ::clike + package clopher.termios; + import com.sun.jna.Structure; + + import java.util.Arrays; + import java.util.List; + + + /** + * Interface to Posix libc + */ + public class Termios extends Structure { + private int NCCS = 32; + + public int c_iflag; // input mode flags + public int c_oflag; // output mode flags + public int c_cflag; // control mode flags + public int c_lflag; // local mode flags + public byte c_line; // line discipline + public byte c_cc[]; // control characters + public int c_ispeed; // input speed + public int c_ospeed; // output speed + + public Termios() { + c_cc = new byte[NCCS]; + } + + // This function is important for JNA, because it needs to know the + // order of the fields of the struct in order to make a correct Java + // class to C struct translation + protected List<String> getFieldOrder() { + return Arrays.asList( + "c_iflag", + "c_oflag", + "c_cflag", + "c_lflag", + "c_line", + "c_cc", + "c_ispeed", + "c_ospeed" + ); + } + } + +Once the struct is defined, it's time to use it from Clojure. `clopher.term` +namespace has the code to solve this. Summarized here: + + ::clojure + (ns clopher.term + (:import [clopher.termios Termios] + [com.sun.jna Function])) + + (def ^:private ICANON 02) + (def ^:private ECHO 010) + (def ^:private ISIG 01) + (def ^:private ECHONL 0100) + (def ^:private IEXTEN 0100000) + + (def ^:private VTIME 5) + (def ^:private VMIN 6) + + ; The macro we saw at the blogpost by Nurulla Akkaya + (defmacro jna-call [lib func ret & args] + `(let [library# (name ~lib) + function# (Function/getFunction library# ~func)] + (.invoke function# ~ret (to-array [~@args])))) + + ; Wrapper for the tcgetattr function + (defn get-config! + [] + (let [term-conf (Termios.)] + (if (= 0 (jna-call :c "tcgetattr" Integer 0 term-conf)) + term-conf + (throw (UnsupportedOperationException. + "Impossible to get terminal configuration"))))) + + ; Wrapper for the tcsetattr function + (defn set-config! + [term-conf] + (when (not= 0 (jna-call :c "tcsetattr" Integer 0 0 term-conf)) + (throw (UnsupportedOperationException. + "Impossible to set terminal configuration")))) + + ; Example to set the non-canonical mode using the flags at the top of the + ; file + ; Yeah, binary operations. + (defn set-non-canonical! + ([] + (set-non-canonical! true)) + ([blocking] + (let [term-conf (get-config!)] + (set! (.-c_lflag term-conf) + (bit-and (.-c_lflag term-conf) + (bit-not (bit-or ICANON ECHO ISIG ECHONL IEXTEN)))) + (aset-byte (.-c_cc term-conf) VMIN (if blocking 1 0)) + (aset-byte (.-c_cc term-conf) VTIME 0) + (set-config! term-conf)))) + + +Pay attention to all the mutable code here! +`aset-byte` function helps a lot when dealing with all that. + +Be also sure to check termios' documentation because the calls act in a very +C-like way, returning a non-zero answer when they fail. + +We need an extra point in our code to solve the Java-Clojure interoperability: +we have to tell our project manager that we included some Java code in there. +If our project manager is Leiningen, we can just tell it where do we store our +Java code. Be careful because Leiningen [doesn't like if you mix Java and +Clojure in the same folder][leiningen]. + + ::clojure + (defproject + ; There's more blablabla in here but these are the keys I want you to + ; take in account + :source-paths ["src/clojure"] + :java-source-paths ["src/java"] + :javac-options ["-Xlint:unchecked"]) + + +### Look back! + +Now you can configure your terminal to act non-canonically and serve you the +characters one by one as they come. It's cool but you'll see there are some +problems to come for the next chapters. Don't worry! They'll come. + +> This is like a heroic novel where the character (in this case you) fights +> monsters one by one leaving their dead corpses in the dungeon floor. Looking +> back will let you remember how many monsters did you slaughter in your way to +> the deep where the treasure awaits. Remember to take rest and sharpen your +> sword. This is a long travel. + +> Prepare yourself for the next monster. Let the voice of the narrator guide you +> to the unknown. + +Why don't you mix what you learned on the previous chapter with what you +learned from this one and try to make an interactive terminal program yourself? + +I'll solve that in the next chapter, but there's some code of that part already +implemented in the repository. You can check it while I keep writing and +coding. Here's the link to the project: + +[https://gitlab.com/ekaitz-zarraga/clopher][clopher] + + +See you in the next episode! + + +[^1]: Use the man pages, seriously: [`man stty`](https://linux.die.net/man/1/stty) +[^2]: I've also been in contact with Mogens, the author of the project, who is + a really good guy and gave me a lot of good information. +[^3]: But who cares about them anyway? +[^4]: *the free encyclopedia* +[^5]: If it's free software it's not stealing and it's exactly what you are + supposed to do with it. + +[liquid]: http://salza.dk/ +[jni]: https://en.wikipedia.org/wiki/Java_Native_Interface +[jna]: https://en.wikipedia.org/wiki/Java_Native_Access +[lanterna]: https://github.com/mabe02/lanterna +[swing]: https://en.wikipedia.org/wiki/Swing_(Java) +[posix-lanterna]: https://github.com/mabe02/lanterna/blob/master/native-integration/src/main/java/com/googlecode/lanterna/terminal/PosixLibC.java +[nativegnu-lanterna]: https://github.com/mabe02/lanterna/blob/263f013a2ee1d522eb86b8f1d315423fb1f79711/native-integration/src/main/java/com/googlecode/lanterna/terminal/NativeGNULinuxTerminal.java#L123 +[nakkaya]: https://nakkaya.com/2009/11/16/java-native-access-from-clojure/ +[leiningen]: https://github.com/technomancy/leiningen/blob/master/sample.project.clj#L302 +[clopher]: https://gitlab.com/ekaitz-zarraga/clopher |