Cross-compiling Zig on an old Kindle

“What is the hardest bug you’ve solved?”

I usually like to ask this question in the interviews. It gives the candidate an opportunity to share something unique they’ve uncovered. The following log is a story of one such debugging journey I took recently.

Prelude

When I hacked my kindle and was planning to turn it into a dashboard, I had an interesting idea: Could I use Zig on this legacy device? I like the ergonomics of Zig and I thought it would be a fun detour to write the network component in Zig. Although I wasn’t planning to use it, running Zig old kindle turned out to be an interesting challenge.

The Zig HTTP Client

This part was easy. I picked up a working example from zig.news1 verbatim. After testing it on my local machine I cross-compiled it for kindle ABI.

Click here to expand the hello-web.zig program 
const std = @import("std");
const writer = std.io.getStdOut().writer();

pub fn main() !void {
    // Where we are going we need dynamic allocation
    const alloc = std.heap.page_allocator;
    var arena = std.heap.ArenaAllocator.init(alloc);
    const allocator = arena.allocator();

    //I like simple lifetimes so I am just clearing all allocations together
    defer arena.deinit();

    //The client is what handles making and receiving the request for us
    var client = std.http.Client{
        .allocator = allocator,
    };

    //We can set up any headers we want
    const headers = &[_]std.http.Header{
        .{ .name = "X-Custom-Header", .value = "application" },
        // if we wanted to do a post request with JSON payload we would add
        // .{ .name = "Content-Type", .value = "application/json" },
    };

    // I moved this part into a seperate function just to keep it clean
    const response = try get("https://jsonplaceholder.typicode.com/posts/1", headers, &client, alloc);

    // .ignore_unknown_fields will just omit any fields the server returns that are not in our type
    // otherwise an unknown field causes an error
    const result = try std.json.parseFromSlice(Result, allocator, response.items, .{ .ignore_unknown_fields = true });

    try writer.print("title: {s}\n", .{result.value.title});
}

//This is what we are going to parse the response into
const Result = struct {
    userId: i32,
    id: i32,
    title: []u8,
    body: []u8,
};

fn get(
    url: []const u8,
    headers: []const std.http.Header,
    client: *std.http.Client,
    allocator: std.mem.Allocator,
) !std.ArrayList(u8) {
    try writer.print("\nURL: {s} GET\n", .{url});

    var response_body = std.ArrayList(u8).init(allocator);

    try writer.print("Sending request...\n", .{});
    const response = try client.fetch(.{
        .method = .GET,
        .location = .{ .url = url },
        .extra_headers = headers, //put these here instead of .headers
        .response_storage = .{ .dynamic = &response_body }, // this allows us to get a response of unknown size
        // if we were doing a post request we would include the payload here
        //.payload = "<some string>"
    });

    try writer.print("Response Status: {d}\n Response Body:{s}\n", .{ response.status, response_body.items });

    // Return the response body to the caller
    return response_body;
}
 zig build-exe hello-web.zig \
 -lc -O ReleaseSmall \
 -fstrip -fsingle-threaded \
 -target arm-linux-gnueabi

I copied the file to the kindle using scp and ran it on the kindle, only to see it fail with an error.

[root@kindle apps]# ./hello-web

URL: https://jsonplaceholder.typicode.com/posts/1 GET
Sending request...
error: CertificateBundleLoadFailure

It was strange because the curl command was working fine on the kindle.

[root@kindle us]# curl -s https://jsonplaceholder.typicode.com/posts/1
{
  "userId": 1,
  "id": 1,
  "title": "sunt aut facere repellat provident occaecati excepturi optio reprehenderit",
  "body": "quia et suscipit\nsuscipit recusandae consequuntur expedita et cum\nreprehenderit molestiae ut ut quas totam\nnostrum rerum est autem sunt rem eveniet architecto"
}
Asking for help

The great thing about Zig is its active community. I went to the Zig Discord and asked if anyone has done something similar before. The fine folks on that forum were intrigued and provided invaluable help in debugging the program. The first step was to check if the SSL Certs are present and valid. They were present.2

[root@kindle us]# curl -v 'https://jsonplaceholder.typicode.com/posts/1' 2>&1 | grep -e 'CAfile' -e 'CApath'
*  CAfile: /etc/ssl/certs/ca-certificates.crt
*  CApath: none

[root@kindle us]# ls -la /etc/ssl/certs/
total 238
drwxr-xr-x    2 root     root          1024 Nov 28 14:17 .
drwxr-xr-x    3 root     root          1024 Nov 28 14:17 ..
-rw-r--r--    1 root     root          2796 Nov 28 13:39 ca-certificates-maru-prod.crt
lrwxrwxrwx    1 root     root            44 Nov 28 14:17 ca-certificates-maru.crt -> /etc/ssl/certs/ca-certificates-maru-prod.crt
-rw-r--r--    2 root     root        236607 Nov 28 13:39 ca-certificates-prod.crt
lrwxrwxrwx    2 root     root            39 Nov 28 14:17 ca-certificates.crt -> /etc/ssl/certs/ca-certificates-prod.crt

I verified if the certificates were present at the place where the binary was looking for them. They were present which made the mystery even deeper.

[root@kindle apps]# strings hello-web | grep ssl
/etc/ssl/ca-bundle.pem
/etc/ssl/certs
/etc/ssl/cert.pem
/etc/ssl/certs/ca-certificates.crt
ssl_key_log_path
ssl_key_log_file
http_enable_ssl_key_log_file
ssl_key_log

I recompiled the binary with debug symbols but it didn’t help much.

[root@kindle apps]# ./hello-web

URL: https://jsonplaceholder.typicode.com/posts/1 GET
Sending request...
unexpected errno: 38
unexpected errno: 38
....
Enter GDB

With options dwindling, it was time to debug the program and understand what was happening. Kindle doesn’t have strace, but luckily it came with an ancient gdb (v8.1 from 2018).

GDB Crash course

I relied on the following gdb commands:

  • b *ADDR break at address
  • b funcman break on function
  • i b see breakpoints
  • n next line
  • ni next instruction
  • s next line, follow into functions
  • si next instruction
  • bt backtrace

I ran the program using gdb hello-web and then used catch syscall 5 to stop at the first invocation of the open syscall.

Click here to expand the GDB trace 
Catchpoint 2 (call to syscall 322), 0x00085c34 in os.linux.arm.syscall4 (number=openat, arg1=4294967196, arg2=2130700608, arg3=655616, arg4=0)
    at /opt/homebrew/Cellar/zig/0.14.0/lib/zig/std/os/linux/arm.zig:55
55      /opt/homebrew/Cellar/zig/0.14.0/lib/zig/std/os/linux/arm.zig: No such file or directory.
(gdb) n

Catchpoint 2 (returned from syscall 322), 0x00085c34 in os.linux.arm.syscall4 (number=openat, arg1=4294967196, arg2=2130700608, arg3=655616, arg4=0)
    at /opt/homebrew/Cellar/zig/0.14.0/lib/zig/std/os/linux/arm.zig:55
55      in /opt/homebrew/Cellar/zig/0.14.0/lib/zig/std/os/linux/arm.zig
(gdb) n
posix.openatZ (dir_fd=-100, file_path=0x7effe940 "/etc/ssl/certs/ca-certificates.crt", flags=655616, mode=0)
    at /opt/homebrew/Cellar/zig/0.14.0/lib/zig/std/posix.zig:1809
1809    /opt/homebrew/Cellar/zig/0.14.0/lib/zig/std/posix.zig: No such file or directory.
(gdb) n
1810    in /opt/homebrew/Cellar/zig/0.14.0/lib/zig/std/posix.zig
(gdb) n
fs.Dir.openFileZ (self=..., sub_path=0x7effe940 "/etc/ssl/certs/ca-certificates.crt", flags=...)
    at /opt/homebrew/Cellar/zig/0.14.0/lib/zig/std/fs/Dir.zig:889
889     /opt/homebrew/Cellar/zig/0.14.0/lib/zig/std/fs/Dir.zig: No such file or directory.
(gdb) n
892     in /opt/homebrew/Cellar/zig/0.14.0/lib/zig/std/fs/Dir.zig
(gdb) n
919     in /opt/homebrew/Cellar/zig/0.14.0/lib/zig/std/fs/Dir.zig
(gdb) n
crypto.Certificate.Bundle.addCertsFromFilePathAbsolute (cb=0x7efffad4, gpa=..., abs_file_path=...)
    at /opt/homebrew/Cellar/zig/0.14.0/lib/zig/std/crypto/Certificate/Bundle.zig:203
203     /opt/homebrew/Cellar/zig/0.14.0/lib/zig/std/crypto/Certificate/Bundle.zig: No such file or directory.
(gdb) n
unexpected errno: 38

This told me that the failing line was https://github.com/ziglang/zig/blob/0.14.0/lib/std/crypto/Certificate/Bundle.zig#L203 and it was failing somewhere in https://github.com/ziglang/zig/blob/0.14.0/lib/std/crypto/Certificate/Bundle.zig#L224

To make debugging simpler, I fleshed out the cert loading part into a new file debug-certs.zig.

❯ cat debug-certs.zig
const std = @import("std");
const writer = std.io.getStdOut().writer();

pub fn main() !void {
    var gpa = std.heap.GeneralPurposeAllocator(.{}){};
    const allocator = gpa.allocator();
    var bundle = std.crypto.Certificate.Bundle{};
    try bundle.rescan(allocator);
    try writer.print("Got: {any}\n", .{bundle});
}

(gdb) n
crypto.Certificate.Bundle.addCertsFromFilePathAbsolute (cb=0x7efffad4, gpa=..., abs_file_path=...)
    at /opt/homebrew/Cellar/zig/0.14.0/lib/zig/std/crypto/Certificate/Bundle.zig:203
203     /opt/homebrew/Cellar/zig/0.14.0/lib/zig/std/crypto/Certificate/Bundle.zig: No such file or directory.
(gdb) ni
0x000f4acc      203     in /opt/homebrew/Cellar/zig/0.14.0/lib/zig/std/crypto/Certificate/Bundle.zig
(gdb) s
crypto.Certificate.Bundle.addCertsFromFile (cb=0xaaaaaaaa, gpa=..., file=...)
    at /opt/homebrew/Cellar/zig/0.14.0/lib/zig/std/crypto/Certificate/Bundle.zig:224
224     in /opt/homebrew/Cellar/zig/0.14.0/lib/zig/std/crypto/Certificate/Bundle.zig

I went deeper into the loading mechanism and stopped at the OS syscall.

(gdb) n

Catchpoint 3 (call to syscall 4), 0x0009c58c in os.linux.arm.syscall3 (number=write, arg1=2, arg2=190156, arg3=18)
    at /opt/homebrew/Cellar/zig/0.14.0/lib/zig/std/os/linux/arm.zig:44
44      /opt/homebrew/Cellar/zig/0.14.0/lib/zig/std/os/linux/arm.zig: No such file or directory.
(gdb)

At this stage, I performed a backtrace that gave me the following verbose log:

Click here to expand the GDB trace for backtracking 
(gdb) bt
#0  0x0009c58c in os.linux.arm.syscall3 (number=write, arg1=2, arg2=190156, arg3=18) at /opt/homebrew/Cellar/zig/0.14.0/lib/zig/std/os/linux/arm.zig:44
#1  0x00085a10 in os.linux.write (fd=2, buf=0x2e6cc <__anon_21027> "unexpected errno: ", count=18)
    at /opt/homebrew/Cellar/zig/0.14.0/lib/zig/std/os/linux.zig:1149
#2  0x0006f26c in posix.write (fd=2, bytes=...) at /opt/homebrew/Cellar/zig/0.14.0/lib/zig/std/posix.zig:1262
#3  0x000585c4 in fs.File.write (self=..., bytes=...) at /opt/homebrew/Cellar/zig/0.14.0/lib/zig/std/fs/File.zig:1327
#4  0x00053228 in io.GenericWriter(fs.File,error{NoSpaceLeft,DiskQuota,FileTooBig,InputOutput,DeviceBusy,InvalidArgument,AccessDenied,BrokenPipe,SystemResources,OperationAborted,NotOpenForWriting,LockViolation,WouldBlock,ConnectionResetByPeer,ProcessNotFound,NoDevice,Unexpected},(function 'write')).typeErasedWriteFn (context=0x7efff3d4, bytes=...) at /opt/homebrew/Cellar/zig/0.14.0/lib/zig/std/io.zig:348
#5  0x00057c94 in io.Writer.write (self=..., bytes=...) at /opt/homebrew/Cellar/zig/0.14.0/lib/zig/std/io/Writer.zig:13
#6  0x00052de0 in io.Writer.writeAll (self=..., bytes=...) at /opt/homebrew/Cellar/zig/0.14.0/lib/zig/std/io/Writer.zig:19
#7  0x000b2414 in fmt.format__anon_18802 (writer=..., args=...) at /opt/homebrew/Cellar/zig/0.14.0/lib/zig/std/fmt.zig:133
#8  0x0009cbb4 in io.Writer.print__anon_15075 (self=..., args=<error reading variable: Cannot access memory at address 0x12>)
    at /opt/homebrew/Cellar/zig/0.14.0/lib/zig/std/io/Writer.zig:24
#9  0x00085b50 in io.GenericWriter(fs.File,error{NoSpaceLeft,DiskQuota,FileTooBig,InputOutput,DeviceBusy,InvalidArgument,AccessDenied,BrokenPipe,SystemResources,OperationAborted,NotOpenForWriting,LockViolation,WouldBlock,ConnectionResetByPeer,ProcessNotFound,NoDevice,Unexpected},(function 'write')).print (
    self=..., format=..., args=...) at /opt/homebrew/Cellar/zig/0.14.0/lib/zig/std/io.zig:312
#10 debug.print__anon_12321 (args=...) at /opt/homebrew/Cellar/zig/0.14.0/lib/zig/std/debug.zig:215
#11 0x0006f948 in posix.unexpectedErrno (err=NOSYS) at /opt/homebrew/Cellar/zig/0.14.0/lib/zig/std/posix.zig:7502
#12 0x00072448 in fs.File.stat (self=...) at /opt/homebrew/Cellar/zig/0.14.0/lib/zig/std/fs/File.zig:553
#13 0x0005987c in fs.File.getEndPos (self=...) at /opt/homebrew/Cellar/zig/0.14.0/lib/zig/std/fs/File.zig:358
#14 0x000f62f8 in crypto.Certificate.Bundle.addCertsFromFile (cb=0x7efffad4, gpa=..., file=...)
    at /opt/homebrew/Cellar/zig/0.14.0/lib/zig/std/crypto/Certificate/Bundle.zig:225
#15 0x000f4ad0 in crypto.Certificate.Bundle.addCertsFromFilePathAbsolute (cb=0x7efffad4, gpa=..., abs_file_path=...)
    at /opt/homebrew/Cellar/zig/0.14.0/lib/zig/std/crypto/Certificate/Bundle.zig:203
#16 0x000f2cb0 in crypto.Certificate.Bundle.rescanLinux (cb=0x7efffad4, gpa=...)
    at /opt/homebrew/Cellar/zig/0.14.0/lib/zig/std/crypto/Certificate/Bundle.zig:100
#17 0x000f072c in crypto.Certificate.Bundle.rescan (cb=0x7efffad4, gpa=<error reading variable: Cannot access memory at address 0x12>)
    at /opt/homebrew/Cellar/zig/0.14.0/lib/zig/std/crypto/Certificate/Bundle.zig:61
#18 0x000f024c in debug-certs.main () at debug-certs.zig:11
#19 0x000f012c in start.callMain () at /opt/homebrew/Cellar/zig/0.14.0/lib/zig/std/start.zig:656
#20 start.callMainWithArgs (argc=<optimized out>, argv=<optimized out>, envp=...) at /opt/homebrew/Cellar/zig/0.14.0/lib/zig/std/start.zig:616
#21 start.posixCallMainAndExit (argc_argv_ptr=0x7efffca0) at /opt/homebrew/Cellar/zig/0.14.0/lib/zig/std/start.zig:571
#22 0x00000000 in ?? ()
Backtrace stopped: previous frame identical to this frame (corrupt stack?)

Scanning through the trace, I saw that at Bundle.zig:225 the program was having trouble reading the file. The const size = try file.getEndPos(); was calling fs.File.stat and getting an unexpected error. This was strange, why would it not be able to read a file… unless there was something wrong with the os calls made by Zig?

I checked if zig supports 3.16+ Kernels3 and then dived in the source code. Looking at the Zig changelog, I found that the current version v0.14 was not using os.fstat! It was using a later API linux.statx which was not supported on the ancient kindle kernel. Luckily this change was recent and v0.13 was using os.fstat.

I downgraded the zig version to v0.13 and it worked like a charm!

[root@kindle apps]# ./hello-web

URL: https://jsonplaceholder.typicode.com/posts/1 GET
Sending request...
Response Status: http.Status.ok
 Response Body:{
  "userId": 1,
  "id": 1,
  "title": "sunt aut facere repellat provident occaecati excepturi optio reprehenderit",
  "body": "quia et suscipit\nsuscipit recusandae consequuntur expedita et cum\nreprehenderit molestiae ut ut quas totam\nnostrum rerum est autem sunt rem eveniet architecto"
}
title: sunt aut facere repellat provident occaecati excepturi optio reprehenderit
Conclusion

In the end I managed to cross-compile a Zig program and run on the kindle device. I eventually used inbuilt curl command in the end, but it was certainly an interesting experience to cross compile zig. What impressed me was how easy it was to get started and how helpful the community was. The linter also worked right away in my NeoVim editor without any extra changes and provided valuable feedback. For a language that has not reached v1.0 yet, this is an incredible feat.

  1. https://zig.news/andrewgossage/easy-web-requests-in-zig-with-clientfetch-k43 ↩︎

  2. https://github.com/ziglang/zig/blob/7e751e8040ad6afb9ae0a040ae4080c2f4b2f443/lib/std/crypto/Certificate/Bundle.zig#L80 ↩︎

  3. https://ziglang.org/download/0.13.0/release-notes.html#Support-Table ↩︎