Update content

2 files changed, 480 insertions, 0 deletions

diff --git a/content/clopher/03-TUI.md b/content/clopher/03-TUI.md
new file mode 100644
index 0000000..fb5ad02
--- /dev/null
+++ b/content/clopher/03-TUI.md
@@ -0,0 +1,207 @@
+Title: TUI: A look to the deep
+Date: 2019-05-30
+Categories: Series
+Tags: Clopher - TUI Gopher client
+Slug: clopher03
+Lang: en
+Summary: Research and work on the Terminal based User Interface of Clopher, the
+    gopher client.
+
+> Read all the posts in [Clopher series here]({tag}clopher-tui-gopher-client).
+
+This software have been introduced as a Gopher client but, as you can probably
+deduce from the previous post, the Gopher part is probably the simplest one.
+The complexity comes with user interaction. *People are hard*. That's why we
+are going to delay that as much as possible, trying to cover all the points in
+the middle before we *jump to the unknown*.
+
+Just joking. In fact, we have to shave many yaks before thinking about user
+interaction anyway. This text talks about them.
+
+### Are you talking to me?
+
+Let's remember we can classify programs by two different categories like this:
+
+- **Non-interactive programs**, often called *scripts*, are programs that take
+  an input and return an output. There's no interaction with the user in
+  between.  An example of this could be the command `ls`.
+
+- **Interactive programs** receive user input while they run and respond to the
+  user while they are running. An example of this could be the machine that
+  sells you the tickets for the subway, it asks you where are you going, then
+  tells you the price, take your money and so on. All of this with the program
+  constantly running.
+
+Remembering what we talked about Gopher: it's a *stateless* protocol. There's
+no *state* stored in the server so all the queries *must* contain all the info
+related to them. *Queries are independent*.
+
+This, somehow leaves the door open to two possible implementations of Clopher.
+The *non-interactive* one would work like `curl`. Getting the IP, port,
+selector string and an optional search string as input it would open the
+connection retrieve the result and return it.[^1]
+
+But Clopher is designed as an **interactive program**. More like `lynx`, where
+you interactively ask for the pages and have a *local state* that records your
+history and other things. This is a decision, it's not imposed by the protocol.
+
+### Shell*f boycott*
+
+There are some different ways to handle user interaction in TUI based programs.
+The simplest one is to read by line, waiting until the user hits `ENTER` to
+read the result. That's the behaviour of the classic `scanf` function of C and
+many others like `input` in Python, etc.
+
+In programs like Clopher, where the design is similar to `lynx` or `vi`, this
+kind of input makes no sense at all. The program needs to be able to capture
+every key pressed by the user and perform action in response to them. For
+instance, in `vi` when the user hits `i` in normal mode it needs to change to
+insert mode and when the user presses `i` in insert mode it needs to change the
+contents of the buffer.
+
+The design of these kind of programs is simple to understand, it's an infinite
+loop[^2] where key presses are captured and they change the *state* of the
+program.  When the user hits the key combination that halts the program the
+loop is broken.
+
+In simple C code the program would look like this:
+
+    ::clike
+    #include<stdio.h>
+
+    int
+    main(int argc, char * argv[]){
+        char c;
+        // Create some state
+
+        while(1){
+            c = getchar();
+            if( c == 'q'){ // Exit if user pressed `q`
+                return 0;
+            }
+            // Update state here
+            putchar(c); // Show the character for debugging
+        }
+    }
+
+Or the simplified Clojure equivalent:
+
+    ::clojure
+    (loop [c     (char (.read *in*))
+           state (->state)] ; Create some state
+
+      (when (not= c \q)     ; Exit if user pressed `q`
+        (print c)           ; Show the character for debugging
+        (recur (char (.read *in*))
+               (update-state c state))) ; Update state
+
+
+Looks simple, right?
+
+Wait a second, there's a lot of stuff going on under the hood here. If you run
+the code in any POSIX compatible operating system (I didn't test on others,
+and I won't) you'll find the code might not be doing what we expected it to:
+The `getchar` (or `.read`) calls will wait until `ENTER` is pressed in the
+input buffer and then they'll get the characters one by one. But we want to get
+them as they come!
+
+#### Saints and demons  — canonical mode
+
+In POSIX operating systems, the input is buffered by default. But that behavior
+can be configured following the POSIX terminal interface under the name
+**canonical** mode or **non-canonical** mode. The mode we are looking for is
+the non canonical mode. You can read more about it in [the Wikipedia][canon].
+
+Choosing the non-canonical mode has some extra options: one controls the number
+of minimum characters to have in the buffer to perform a `read` operation and
+the other defines the amount of tenths of second to wait for that input[^3].
+Choosing the right value for those fields (`c_cc[MIN]` and `c_cc[TIME]`)
+depends on the kind of interaction we are looking for.
+
+#### Make Dikembe smile — blocking
+
+Setting `c_cc[TIME]` field to `0` means the `read` operation will wait
+indefinitely until the minimum amount of characters defined with `c_cc[MIN]`
+are waiting in the buffer. Together with that, the `c_cc[MIN]` can be `0` that
+means the read operations will wait until there are `0` characters in the
+buffer, or, in other words, they won't wait.
+
+Be aware that both fields can provoke the read operations in the input buffer
+be non-blocking operations and that will cause the read operation to return
+with no value.
+
+In the case of Clopher, I decided to set the `c_cc[MIN]` to `1` so the read
+operations block until there's at least one character in the buffer (that means
+they will always return something) and the `c_cc[TIME]` to `0` so the read
+operations have no timeout and will block until a character arrives.
+
+Depending on the application you are developing, you might choose other kind of
+blocking configuration. For instance, setting a timeout can let you process
+other parts of the system and wait for the input in the same thread.
+
+#### We're talking about practice? — termios
+
+So now we know where to find this theoretical configuration it's time to put it
+in practice. In POSIX the standard way to access this is via `termios`[^4].
+It has some details that are not specified and depend on the implementation, so
+it might have some differences from Linux to BSD or whatever.
+
+`tcsetattr` and `tcgetattr` calls can be used to set and read the terminal
+configuration via termios. Check this example, compile it and compare it with
+the C code of the previous example:
+
+    ::clike
+    #include<stdio.h>
+    #include<termios.h>
+
+    int
+    main(int argc, char* argv[]){
+        // Get interface configuration to reset it later
+        struct termios term_old;
+        tcgetattr(0, &term_old);
+
+        // Get interface configuration to edit
+        struct termios term;
+        tcgetattr(0, &term);
+
+        // Set the new configuration
+        term.c_lflag &= ~(ECHO | ECHONL | ICANON | IEXTEN | ISIG);
+        term.c_cc[VMIN]  = 1;   // Wait until 1 character is in buffer
+        term.c_cc[VTIME] = 0;   // Wait indifinitely
+        //TCSANOW makes the change occur immediately
+        tcsetattr(0, TCSANOW, &term);
+
+        char ch;
+        while(1){
+            if(ch == 'q'){
+                // Set old configuration again and exit.
+                // If it's not set back the normal configuration of the
+                // terminal will be broken later!
+                tcsetattr(0, TCSANOW, &term_old);
+                return 0;
+            }
+            ch = getchar();
+            putchar(ch);
+        }
+    }
+
+All the code has enough comments to be understood but there are some weird
+flags it's better to check in termios documentation.[^4]
+
+### But this is C code and Clopher is written in Clojure!
+
+I know but this is becoming long and boring. Why not wait until I get some
+spare time and write the next chapter? You have tons of information to check
+until I write it so you won't be bored if you don't want to.
+
+See you next.
+
+
+
+[^1]: In fact, you can navigate the Gopherverse like this with `curl`.
+[^2]: Unsurprisingly called *main loop*. Programmers are very creative.
+[^3]: That `read` operation is what `getchar` is doing under the hood.
+[^4]: `man termios` or visit [online `man`
+  pages](https://linux.die.net/man/3/termios)
+
+[canon]: https://en.wikipedia.org/wiki/POSIX_terminal_interface#Input_processing
diff --git a/content/clopher/04-Native_interface.md b/content/clopher/04-Native_interface.md
new file mode 100644
index 0000000..08a8e52
--- /dev/null
+++ b/content/clopher/04-Native_interface.md
@@ -0,0 +1,273 @@
+Title: TUI Slang: Speak like the natives
+Date: 2019-06-15
+Categories: Series
+Tags: Clopher - TUI Gopher client
+Slug: clopher04
+Lang: en
+Summary: Interfacing between Clojure, Java and Native code.
+
+
+> Read all the posts in [Clopher series here]({tag}clopher-tui-gopher-client).
+
+The previous post introduced `termios` as a native interface to configure the
+terminal input processing. With termios we managed to make our C programs get
+input character by character processing them as they came with no buffering but
+we didn't integrate that with our Clojure code. Now it's time to make it.
+
+
+### Run before it's too late
+
+Before we dig in the unknown, I have to tell you there are other alternatives
+for the terminal configuration. The simplest one I can imagine is using
+`stty`[^1] as an external command. I learned this from [Liquid][liquid], a
+really interesting project I had as a reference. If you want to see this work
+check the `adapters/tty.clj` file in the `src` directory of the project.[^2]
+
+Of course, it has some drawbacks. `stty` is part of the GNU-Coreutils project
+and you have to be sure your target has it installed if you want to rely on
+that. I'm not sure about if it's supported in non-GNU operating systems[^3].
+
+In my case, I decided to stay with `termios` interface to deal with all this
+because I didn't really want to rely on external commands and it's supposed to
+be implemented in any POSIX OS. The good (bad?) thing is it made me deal with
+native libraries from Clojure and had to learn how to do it.
+
+
+### The floor is Java
+
+When dealing with low-level stuff we have to remember Clojure is just Java, and
+most of the utilities we need to use come from it. This means the question we
+have to answer is not really *"how to call native code from Clojure?"* because
+if we are able to call native code from Java, we will be able to do it from
+Clojure too (if we spread some magic on top).
+
+#### So, how to call native code from Java?
+
+First I checked the [Java Native Interface (aka JNI)][jni], but I thought it
+was too much for me and I decided to check further. Remember there are only a
+couple of calls to make to `termios` from our code, so we don't really want to
+mess with a lot of boilerplate code, compilations and so on.
+
+My research made me find [Java Native Access (aka JNA)][jna] library. If you
+check the link there you'll find that the Wikipedia[^4] describes it as:
+
+> JNA's design aims to provide native access in a natural way with a minimum of
+> effort. No boilerplate or generated glue code is required.
+
+Sounds like right for me. Doesn't it?
+
+I encourage you to check the full Wikipedia entry and, if you have some free
+time at the office or something, to check the implementation because it's
+really interesting. But I'll leave that for you.
+
+##### A lantern in the dark
+
+JNA is quite easy to use for the case of Clopher, even easier if you realize
+there is [lanterna][lanterna], the TUI library, out there, using it internally
+so you can *steal*[^5] the implementation from it. Lanterna is a great piece
+of software I took as a reference for many parts of the project. Digging in the
+internals of large libraries is a great exercise and you can learn a lot from
+it.
+
+First of all, like many Java projects, the amount of abstractions it has is
+crazy. It takes some time to find the actual implementation of what we want.
+This isn't like this for no reason, the reality is they need to create this
+amount of abstractions because the part of the library that handles the widgets
+can work on top of many different terminal implementations, including a
+[Swing][swing] based one that comes with Lanterna itself.
+
+Clopher only targets POSIX compatible operating systems so we can go directly
+to what we want and read the termios part directly discarding all the other
+compatibility code. This code is quite easy to find if you see the directory
+tree of Lanterna: there's a `native-integration` folder in the root directory.
+If you follow that you'll arrive to [`PosixLibC.java`][posix-lanterna] that
+uses JNA to interact with termios.
+
+The implementation provided by Lanterna is quite complete, they declare a
+library with the functions they need and the data structure introduced in the
+previous chapter. Once the library interface and the necessary data structures
+are defined from Java they can be called with JNA, like they do in the file:
+[`NativeGNULinuxTerminal.java`][nativegnu-lanterna].
+
+#### How to call JNA from Clojure, then?
+
+Calling Java code from Clojure is quite simple because Clojure have been
+designed with that in mind, but this is not only that. Thanks to the Internet,
+there's a great [blogpost by Nurullah Akkaya][nakkaya] describing a simple way
+to use JNA from Clojure. From that, we can move to our specific case.
+
+`termios` has its own data structure so we need to define it so the JNA knows
+how to interact with it. The problem is that Clojure doesn't have enough OOP
+tools to do it directly so we need to make it in plain Java. The good thing is
+that we don't really need to create anything else.
+
+If we remove some unneeded code from Lanterna's termios structure
+implementation it will look like the implementation I made at
+`src/java/clopher/Termios.java`:
+
+    ::clike
+    package clopher.termios;
+    import com.sun.jna.Structure;
+
+    import java.util.Arrays;
+    import java.util.List;
+
+
+    /**
+     * Interface to Posix libc
+     */
+    public class Termios extends Structure {
+        private int NCCS = 32;
+
+        public int c_iflag;           // input mode flags
+        public int c_oflag;           // output mode flags
+        public int c_cflag;           // control mode flags
+        public int c_lflag;           // local mode flags
+        public byte c_line;           // line discipline
+        public byte c_cc[];           // control characters
+        public int c_ispeed;          // input speed
+        public int c_ospeed;          // output speed
+
+        public Termios() {
+            c_cc = new byte[NCCS];
+        }
+
+        // This function is important for JNA, because it needs to know the
+        // order of the fields of the struct in order to make a correct Java
+        // class to C struct translation
+        protected List<String> getFieldOrder() {
+            return Arrays.asList(
+                    "c_iflag",
+                    "c_oflag",
+                    "c_cflag",
+                    "c_lflag",
+                    "c_line",
+                    "c_cc",
+                    "c_ispeed",
+                    "c_ospeed"
+                    );
+        }
+    }
+
+Once the struct is defined, it's time to use it from Clojure. `clopher.term`
+namespace has the code to solve this. Summarized here:
+
+    ::clojure
+    (ns clopher.term
+      (:import [clopher.termios Termios]
+               [com.sun.jna Function]))
+
+    (def ^:private ICANON 02)
+    (def ^:private ECHO   010)
+    (def ^:private ISIG   01)
+    (def ^:private ECHONL 0100)
+    (def ^:private IEXTEN 0100000)
+
+    (def ^:private VTIME 5)
+    (def ^:private VMIN  6)
+
+    ; The macro we saw at the blogpost by Nurulla Akkaya
+    (defmacro jna-call [lib func ret & args]
+      `(let [library#  (name ~lib)
+             function# (Function/getFunction library# ~func)]
+         (.invoke function# ~ret (to-array [~@args]))))
+
+    ; Wrapper for the tcgetattr function
+    (defn get-config!
+      []
+      (let [term-conf (Termios.)]
+        (if (= 0 (jna-call :c "tcgetattr" Integer 0 term-conf))
+          term-conf
+          (throw (UnsupportedOperationException.
+                   "Impossible to get terminal configuration")))))
+
+    ; Wrapper for the tcsetattr function
+    (defn set-config!
+      [term-conf]
+      (when (not= 0 (jna-call :c "tcsetattr" Integer 0 0 term-conf))
+        (throw (UnsupportedOperationException.
+                 "Impossible to set terminal configuration"))))
+
+    ; Example to set the non-canonical mode using the flags at the top of the
+    ; file
+    ; Yeah, binary operations.
+    (defn set-non-canonical!
+      ([]
+       (set-non-canonical! true))
+      ([blocking]
+      (let [term-conf (get-config!)]
+        (set! (.-c_lflag term-conf)
+              (bit-and (.-c_lflag term-conf)
+                       (bit-not (bit-or ICANON ECHO ISIG ECHONL IEXTEN))))
+        (aset-byte (.-c_cc term-conf) VMIN  (if blocking 1 0))
+        (aset-byte (.-c_cc term-conf) VTIME 0)
+        (set-config! term-conf))))
+
+
+Pay attention to all the mutable code here!
+`aset-byte` function helps a lot when dealing with all that.
+
+Be also sure to check termios' documentation because the calls act in a very
+C-like way, returning a non-zero answer when they fail.
+
+We need an extra point in our code to solve the Java-Clojure interoperability:
+we have to tell our project manager that we included some Java code in there.
+If our project manager is Leiningen, we can just tell it where do we store our
+Java code. Be careful because Leiningen [doesn't like if you mix Java and
+Clojure in the same folder][leiningen].
+
+    ::clojure
+    (defproject
+      ; There's more blablabla in here but these are the keys I want you to
+      ; take in account
+      :source-paths      ["src/clojure"]
+      :java-source-paths ["src/java"]
+      :javac-options     ["-Xlint:unchecked"])
+
+
+### Look back!
+
+Now you can configure your terminal to act non-canonically and serve you the
+characters one by one as they come. It's cool but you'll see there are some
+problems to come for the next chapters. Don't worry! They'll come.
+ 
+> This is like a heroic novel where the character (in this case you) fights
+> monsters one by one leaving their dead corpses in the dungeon floor. Looking
+> back will let you remember how many monsters did you slaughter in your way to
+> the deep where the treasure awaits. Remember to take rest and sharpen your
+> sword. This is a long travel.
+ 
+> Prepare yourself for the next monster. Let the voice of the narrator guide you
+> to the unknown.
+
+Why don't you mix what you learned on the previous chapter with what you
+learned from this one and try to make an interactive terminal program yourself?
+
+I'll solve that in the next chapter, but there's some code of that part already
+implemented in the repository. You can check it while I keep writing and
+coding. Here's the link to the project:
+
+[https://gitlab.com/ekaitz-zarraga/clopher][clopher]
+
+
+See you in the next episode!
+
+
+[^1]: Use the man pages, seriously: [`man stty`](https://linux.die.net/man/1/stty)
+[^2]: I've also been in contact with Mogens, the author of the project, who is
+  a really good guy and gave me a lot of good information.
+[^3]: But who cares about them anyway?
+[^4]: *the free encyclopedia*
+[^5]: If it's free software it's not stealing and it's exactly what you are
+  supposed to do with it.
+
+[liquid]: http://salza.dk/
+[jni]: https://en.wikipedia.org/wiki/Java_Native_Interface
+[jna]: https://en.wikipedia.org/wiki/Java_Native_Access
+[lanterna]: https://github.com/mabe02/lanterna
+[swing]: https://en.wikipedia.org/wiki/Swing_(Java)
+[posix-lanterna]: https://github.com/mabe02/lanterna/blob/master/native-integration/src/main/java/com/googlecode/lanterna/terminal/PosixLibC.java
+[nativegnu-lanterna]: https://github.com/mabe02/lanterna/blob/263f013a2ee1d522eb86b8f1d315423fb1f79711/native-integration/src/main/java/com/googlecode/lanterna/terminal/NativeGNULinuxTerminal.java#L123
+[nakkaya]: https://nakkaya.com/2009/11/16/java-native-access-from-clojure/
+[leiningen]: https://github.com/technomancy/leiningen/blob/master/sample.project.clj#L302
+[clopher]: https://gitlab.com/ekaitz-zarraga/clopher