Open Source, Sustainability, Privilege, and Mastodon

Ever since I was introduced to the idea of Open Source Software(OSS), I have questioned the sustainability of it. After all, how can something be free, in a system where everything has a cost? Unfortunately, it’s difficult to have in depth discussions about my concerns, as it’s such a polarized subject.

However, with the #TwitterMigration of users fleeing a platform maintained by a corporation, to an open source platform run by volunteers, it seems that now is very important time to talk about this.

For anyone unfamiliar with the term Open Source Software, Wikipedia defines it as software “that is released under a license in which the copyright holder grants users the rights to use, study, change, and distribute the software and its source code to anyone and for any purpose.” It’s not necessarily free software, but it often is.

I should also note at this point, that this article isn’t going to be a dichotomous look at OSS, claiming that it’s either good or bad. It’s about the issues and risks inherent to it.

Sustainability of Open Source Software?

The main concern I’ve always had with OSS is around the sustainability of it. Whether we like it or not, we live in a society where everything has a cost. Hosting software/documentation on a server costs money. Writing/updating software, costs time. These costs need to be paid somehow.

When it comes to corporate software, the costs are covered by the customers. Whether the user is directly paying for the software, or the company selling user’s data to advertisers and the like, the company in question is managing a revenue flow. They have salespeople, marketers, and accountants to ensure that the revenue continues, and thus, that software developers can be paid.

This is why, when I first learned about OSS, I asked, how can it be sustainable without any reliable source of revenue? The short answer is, it isn’t sustainable.

I’m sure that at this point, all the OSS advocates will be screaming about all the different open source projects that have been going strong for years. Surely that proves that it’s sustainable? Not really, because Open Source Software is made possible by inequality, which isn’t sustainable.

While there are cases where Open Source Software is being provided or supported by a corporation, it’s generally written by people who aren’t being paid for it. This means that the people writing it have both the time, and the energy to sit down and write software that they’re not being paid for.

This is where privilege comes in, because having the time and/or energy to write software for free isn’t a luxury that everyone has. Software developers do, because we’re paid so well that it’s easy to order in some delivery and sit at our desks to write software after the work day is done. However, we live in a world where far too many people are working 2 jobs just to survive. Sitting down to write software after pulling a double is an unrealistic expectation.

The reason why volunteer driven OSS isn’t sustainable, is because this privilege of having the time/energy to write it, comes from the high levels of inequality that many software developers benefit from. I won’t go into the hows and whys that such high inequality is unsustainable, because so much has been written about it already.

One of the horrible effects of this requirement of privilege on OSS comes in the area of job hunting. When job hunting in the software development world, it’s largely expected for the application to have some demonstrated interest in software through Open Source contributions. As pointed out earlier, this is much harder for someone working 2 jobs to survive, so it skewers the job hunt in favour of someone with a privileged background.

The Impact on Mastodon

As I pointed out at the beginning, the #TwitterMigration makes this a very important time to talk about this intersection of privilege and the Open Source community.

The thing about Mastodon, as opposed to Twitter, is that all the software is Open Source, meaning that it’s written by volunteers. Similarly, the servers are all run by volunteers, who rely either on their own money, or on donations from the community.

This ensures that most, if not all, of the developers, mods, and admins on Mastodon are people with a rather high level of privilege. Anyone who’s studied intersectionality knows that privilege and inequality falls along the lines of race, gender identity, and sexuality.

This isn’t to say that for-profit tech companies don’t have issues with racism, sexism, homophobia, and transphobia. They do, but that’s also been written about a lot, by folks more knowledgeable than I.

In fact, we’ve already seen some terrible cases of mismanagement on Mastodon caused by this. Tracy Chou, noted software engineer was kicked off Mastodon for pointing out the very same privilege that I’m talking about here. Noted Journalist and Economist Erica Ifill had her Mastodon account suspended without any reason whatsoever.

While Twitter does have a horrible record itself for standing up to white supremacy (I’ve seen accounts with the N-word in the account name), actions like this on the part of major Mastodon servers is leaving a lot of people from marginalized communities thinking that Mastodon is even worse for them.

The Flipside

As I said, this isn’t a dichotomous look at OSS. Personally, I do believe that Open Source Software is very important, in certain areas. Things like programming languages, popular frameworks (like Vue.js and Ruby on Rails), and IDEs should always be free and Open Source to encourage adoption and help newcomers to the industry. I learned C/C++ with an IDE (and encyclopedia-sized manual) that was given to me by a friend of the family. Not an option for most folks.

If there’s a bottom line to all of this, if we want OSS to continue, and be sustainable, we have to address the problems of privilege that are inherent to the way we currently do it.

Cross-compiling Crystal for the Raspberry Pi (Updated for V1.4+)

Since the first time I wrote about cross compiling Crystal apps for the Raspberry Pi, many great things have happened with the Crystal language, including the official release of V1.0. Unfortunately, an official build for ARM processors isn’t among those changes. Thankfully though, there have been changes to the Crystal compiler that make cross compiling easier than in previous versions. So here I’ll outline the steps needed.

For this blog, I’m cross compiling from a Macbook Pro running Catalina, to a Raspberry Pi 2 running Ubuntu Server. There may be changes based on which OS you’re using on either your development or target device, but hopefully they’ll be minor. At the very least, the steps should be the same for any Debian based distro. If you have any issues with it, please let me know in the comments.

Preparing the Pi

As with previous versions, the first thing we should do is install any necessary packages on the Pi.

$ apt install llvm libgc-dev libpcre3-dev libevent-dev

For sake of completion, here’s what all of these packages do. Feel free to skip this part over if you’re not interested in these details. You’ll still be able to complete the process.

  • llvm – A set of compiler and toolchain technologies. The Crystal compiler uses the llvm to build efficient native code. In our case, it’s able to give information required for cross compiling code for another processor architecture.
  • libgc-dev – The Boehm–Demers–Weiser garbage collector, which the garbage collector used by the Crystal language. This library is one of the reasons why Crystal needs an operating system to run, and can’t be used on bare metal platforms.
  • libpcre3-dev – Perl Compatible Regular Expressions library. A common regular expression library used by the Crystal language.
  • libevent-dev – This library provides asynchronous event notifications and a mechanism for event based callbacks.

If you’ve read the previous article before coming here, this is the first big update. There’s no longer any need to have a libcrystal.a file with the newer versions of the compiler. So now we can move on to determining in the target flag we use when cross compiling.

$ llvm-config --host-target

This command provides a string describing the target flag that we’ll need when running the compiler on our main computer. In my case, it returned armv7l-unknown-linux-gnueabihf, but depending on what version of the Pi you’re using, yours may be slightly different.

Cross-Compiling in macOS

I’m using the same one line app as the previous article, saved as hello.cr:

puts "Hello, world!"

The command to compile is similar to last time:

$ crystal build hello.cr --cross-compile --target "armv7l-unknown-linux-gnueabihf"

The italicized part of this command should be replaced with whatever you saw on your Pi when you ran the llvm-config --host-target command. This command produces the hello.o file, and provides a shell script to link the file. For me, the instruction looked like this:

cc hello.o -o hello -rdynamic -L/usr/local/lib -lpcre -lm -lgc -lpthread -L/usr/local/Cellar/libevent/2.1.12/lib -levent -lrt -lpthread -ldl

If you followed the previous article to this one, you’ll notice that this output is much simpler, and doesn’t have any references to libcrystal.a. This is a key improvement over previous versions of the compiler, removing an entire step on preparing the Pi.

The other key improvement is that this shell script is exactly what you need to run on the Pi to link your hello.o file, so make sure you don’t erase it. Personally, I save the script to a .sh file, and use scp to copy it along with the .o file.

Linking on the Pi

We’ve reached the final step, but this is also the step where we find out if we made any mistakes.

Copy your hello.o file over to your Pi, and run the shell script that you were provided by the Crystal compiler. If there are any necessary libraries that you forgot to install on the Pi during the first section, this is where you’ll get an error, about an unknown flag. Unfortunately, there’s no simple library to tell you what library you need to install for each flag, so you’ll have to look it up for yourself.

An important thing to note here is that this shell script is unique to the Crystal program that we’re compiling. This means that larger, more complex Crystal programs may have additional flags in this script, and thus, you’ll have to install the proper libraries to your Pi before running this script.

If everything went according to plan, you should have your hello file in your directory. Run it, and you’ll get your expected output:

Hello, world!

Congratulations. You’ve now cross compiled a Crystal program and run it on your Raspberry Pi.

Conclusion

There’s one more caveat that I want to mention. Since there’s no official Crystal build for ARM processors, there are still a few bugs when cross compiling. Apps that run perfectly fine on my Mac, often produce unexpected outputs on my Pi.

The good news is that while ARM isn’t an officially supported platform, the Crystal core team does know that they’ll have to adopt it sooner or later (especially with the M1 MBPs on the market), and do welcome issues raised on Github regarding any problems.

Cross-compiling Crystal for the Raspberry Pi (prior to V1.0)

This article is for an outdated version of the Crystal compiler. The updated version can be found here.

Despite only learning about it a couple of years ago, the Crystal programming language has fast become one of my favourite programming languages. The syntax and core library are a joy to work with, while the nature of compiling the language allows for more lightweight installs and faster execution than an interpreted language like Ruby or JavaScript.

With these advantages, one of my first thoughts was to use this language for embedded applications. Yet, unfortunately, all this time later, there still isn’t an official build of the compiler for ARM processors, like those used in the Raspberry Pi. So the choices become, either go through the arduous task of building the compiler from source, or cross-compile code on a different type of machine.

While Crystal’s official documentation does describe how to cross compile Crystal apps, there are some specifics needed for the Raspberry Pi. So, if like me, you’re not familiar with command line compilers (I haven’t used one in about 20 years), then hopefully you’ll find these instructions useful.

Preparing the Pi

I tested these commands on a fresh install of Raspbian, so they should cover every package needed to link and run a basic Crystal app.

First thing we do is install necessary packages.

$ apt-get install llvm libgc-dev libpcre3-dev libevent-dev

For sake of completion, here’s what all of these packages do. Feel free to skip this part over if you’re not interested in these details. You’ll still be able to complete the process.

  • llvm – A set of compiler and toolchain technologies. The Crystal compiler uses the llvm to build efficient native code. In our case, it’s able to give information required for cross compiling code for another processor architecture.
  • libgc-dev – The Boehm–Demers–Weiser garbage collector, which the garbage collector used by the Crystal language. This library is one of the reasons why Crystal needs an operating system to run, and can’t be used on bare metal platforms.
  • libpcre3-dev – Perl Compatible Regular Expressions library. A common regular expression library used by the Crystal language.
  • libevent-dev – This library provides asynchronous event notifications and a mechanism for event based callbacks.

The other library that we need is a Crystal library, that we need to build from source. Fortunately, it’s just one file, so it won’t give you a headache. Just run the following commands from the directory where you plan to compile your Crystal program.

$ wget https://raw.githubusercontent.com/crystal-lang/crystal/master/src/ext/sigfault.c
$ cc -c -o sigfault.o sigfault.c
$ ar -rcs libcrystal.a sigfault.o

This will produce the file libcrystal.a. We’ll need this in our final step.

Before we create our program on a different machine, there’s one more command we need to run to get the necessary information.

$ llvm-config --host-target

This command provides a string describing the target flag that we’ll need when running the compiler on our main computer. In my case, it returned armv6k-unknown-linux-gnueabihf, but depending on what version of Pi you’re using, yours may be slightly different.

Cross-Compiling in macOS

I used a Mac for this portion of the process. I assume that it’ll also work in Linux, but if anyone does try on Linux without success, please let me know.

The next step is creating a Crystal app. For demonstration purposes, a simple one liner will do:

puts "Hello, world!"

Let’s save this as hello.cr.

Next, we run the command that gives us our (.o) object file, and a few instructions.

$ crystal build your_program.cr --cross-compile --target "armv6k-unknown-linux-gnueabihf"

The italicized part of this command should be replaced with whatever you saw on your Pi when you ran the llvm-config --host-target command. This command produces the hello.o file, and provides an instruction on how to link the file. On my Mac, this instruction looks like this:

cc hello.o -o hello -rdynamic -L/usr/local/Cellar/crystal/0.35.1_1/embedded/lib -L/usr/local/lib -lpcre -lm -lgc -lpthread /usr/local/Cellar/crystal/0.35.1_1/src/ext/libcrystal.a -L/usr/local/Cellar/libevent/2.1.12/lib -levent -lrt -ldl

Unfortunately, this instruction won’t work on the Pi, because it’s dependent on having Crystal installed through Homebrew, which isn’t going to be there on our Pi. However, the hello.o file created is useful to us, so let’s copy that over to the Pi.

Linking on the Pi

We’re finally at our final steps. First, ensure that the libcrystal.a file that you created in the first section is in the same folder as the hello.o file that you copied over from your main computer. Simply run this command:

$ cc hello.o -o hello -rdynamic -lpcre -lm -lgc -lpthread libcrystal.a -levent -lrt -ldl

This will produce a file named ‘hello’ without any extension. Running this file with:

$ ./hello

Will produce our desired output:

Hello, world!

Congratulations. You’ve now cross compiled a Crystal program and run it on your Raspberry Pi.

Conclusion

While it’s my express wish that this helps you get started running Crystal on your Raspberry Pi, there is more to running cross compiled apps. This program was very simple, just a single output. Some programs can be more complicated, and may need more libraries installed on your Pi, and these instructions won’t take care of that.

I also maintain hope that in the future, we’ll have an official build of the Crystal compiler for ARM, and a community of developers who want to develop Crystal libraries and programs for the Raspberry Pi. In the meantime though, at least we have these instructions to get us started.

Tech’s Role in the Erosion of Privacy

There’s an interesting debate happening right now among those in the cyber-security/privacy space. It revolves around an assignment given by a Kate Klonick, an Assistant Professor at a law school in the US. This assignment was for her students to go to a public place, and attempt to identify the strangers they see through observation.

 

 

 

Interestingly, her thread seems to have divided the privacy community. Some are praising this assignment as a “great assignment” or “brilliant.” Some related amusing anecdotes of people doing the same thing at parties. One person pointed out that all the students’ searches will be recorded by Google, and this will teach the students to “…become people who don’t respect the dignity of others…”

 

It’s an interesting debate to say the least. On one hand, it’s useful for these students to learn how easy it can be to gleam information about you from what you advertise to the world without realizing it. On the other hand, it’s a clear invasion of privacy of those people who the students will be spying on. One point brought up by one of the opponents of this study was that it reflects on the erosion of privacy.

This brought me to reflect on an annoying modern trend: people in public using their smartphones to record everything and everyone in sight. Sometimes I’ll be at a club, and someone’s got their smartphone on video mode, panning across the entire room, capturing everyone there. Sometimes there’ll be a couple of people in the middle of the dance floor, and 10 people around them, all recording them.

Back when I was in my 20s (yeah, I’m dating myself, so what?), digital cameras became commonplace. They were pretty convenient, and you could easily carry one in your backpack or purse. Even though they were so convenient, you didn’t see people taking pictures or videos of everything. In fact, seeing someone use their camera was the easiest way to spot a tourist. Most importantly, you didn’t see people taking pictures of complete strangers.

So what’s happened between then and now? Well, aside from all my grey hairs, we now have the surveillance economy. Now we have Google, Facebook and other social media companies making billions of dollars off the information people post online, and getting them all addicted to continuing to do so. Nowadays, young people are taught that it’s cool to violate a stranger’s privacy, and post a recording of them on Facebook in order to get more likes.

I sometimes refer to these folks as “Zuckerberg drones,” because that’s basically what they are. Taking these photos and videos, and then posting them to Facebook, Instagram, Snapchat or whatever, provides more data for these companies to use to run through their algorithms to determine how better to sell you something you don’t need, or simply get further addicted to their services. It’s of no benefit to the people doing the recording, but it’s of great benefit to the people running those companies.

The oddest thing of all of this, is how vehemently these people defend their privacy invading behaviour. They refuse to accept how incredibly rude it is to take a recording of someone else without their consent.

The question of consent brings us back to the situation that started this whole debate. Since this was an optional assignment (with no grades affects), every student who takes  part in it has given their consent. Those folks that they’ll be observing however, won’t even know what’s happening, so they won’t be giving any consent.

I don’t think it would be that hard to create a situation where students could learn the same lesson while still respecting consent. Universities never seem to have a shortage of willing participants in research studies, so this professor could have found a way to only involve consensual targets for her students to observe. This way, the lesson is still learned, while people’s rights are respected.

JWTs: More Than You Think

JSON Web Tokens, or JWTs, are a very important aspect of internet technology, be it Internet of Things, or Internet of People. Knowing this makes it very confusing as to why there’s so little discussion about it, and so much misinformation about it. So I’m writing this blog post to share some information, and hopefully clear away the mists that surround this subject.

There are already many libraries in most major languages for handling JWTs, but for the code examples in this article, I’ll be using the jose gem for Ruby.

The Misconception

Most of the times that you look up JWTs, you’ll see something similar to the item below.

{
  alg: RS256,
  typ: JWT
},
{
  iss: "Chris Larsen",
  sub: 'JWT',
  aud: 'Silicon Halton',
  iat: Time.now
},
2xauJJstlosXp40c2BAwJhueGUqTqq50pu5ICcJuQQA

While this is a JWT, it’s not the only kind of JWT. This is what’s called a JSON Web Signature(JWS). There is another type of JWT called a JSON Web Encryption(JWE). In common usage, most developers use JWT as a synonym for JWS. Even many of the JWT libraries ignore the JWE functionality, only providing signature and verification features.

JWS vs. JWE

The JWS and JWE specifications both provide useful, yet different functions. A JWS adds a digital signature to a set of claims, allowing the receiver to ensure that none of the data has been altered in transmission. A JWE on the other hand, encrypts the data sent, to ensure that it cannot be read by a third party. Both formats make use of Base64URL encoding in order to provide a URL-safe token that can be sent as the bearer token of an HTTP request, or as part of an HTTP response.

It’s also possible to sign AND encrypt data, by nesting one JWT within another. That is functionality that I won’t be covering in this article, but it’s possible to read about it in the JWT spec.

A number of algorithms are included in the specs for both signing and encrypting JWTs. The most popular algorithms are HMAC, which uses a symmetric key, and RSA, which uses private/public key pairs.

JWS

The typical use case for a JWS is for client authentication. The steps for this usage would be:

  1. Client signs in using credentials (most typically a combination of username/email and password)
  2. Server checks credentials against user database, and upon success, will generate and return the encoded JWS, with any required information included in the payload.
  3. Subsequent requests from the client to the server will include the encoded JWT for authentication.

The process of building a JWS is as follows:

  1. Encode JWS Protected Header as BASE64URL(UTF8(JWS Protected Header))
  2. Encode JWS Payload as BASE64URL(UTF8(JWS Payload))
  3. Encrypt ASCII(BASE64URL(UTF8(JWS Protected Header)) || ‘.’ || BASE64URL(JWS Payload)) using the algorithm specified in the header and the key
  4. Concatenate these values in the order Header.Payload.Signature
  5. This provides the URL safe JWS Compact Serialization

Note: Base64URL encoding is NOT a substitute for encryption. Only the signature of a JWS is encrypted. If a third party is able to get your JWS, then they will be able to easily decode it and read the contents of the header and payload. If you’re transmitting sensitive information in the JWS, you should also encrypt it as a JWE. Also, as always, you should be securing your connection using TLS, even when sending a JWE.

Here’s an example of creating and verifying a JWS using the HS256 symmetric algorithm.

require 'jose'

# HS256 Symmetric Signature
secret = 'some128bitsecret'
jwk = JOSE::JWK.from_oct(secret)
header = { "alg" => "HS256" }
payload = { "iss" => "Chris Larsen", 
                     "sub" => "JWTs", 
                     "aud" => "The blog reading public", 
                     "iat" => 1540121863 
}

signed_hs256 = JOSE::JWT.sign(jwk, header, payload).compact
puts "Signed and Encoded JWS: #{signed_hs256}"
#=> Signed and Encoded JWS: eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJTaWxpY29uIEhhbHRvbiBTb2Z0d2FyZSBQMlAiLCJpYXQiOjE1NDAxMjE4NjMsImlzcyI6IkNocmlzIExhcnNlbiIsInN1YiI6IkpXVHMifQ.2xauJJstlosXp40c2BAwJhueGUqTqq50pu5ICcJuQQA

verification = JOSE::JWT.verify_strict(jwk, ["HS256"], signed_hs256)
puts "Verification of JWS: #{verification.first}"
#=> Verification of JWS: true

 

Here’s an example of creating and verifying a JWS using RS256 encryption. You’ll note that the encrypted signature is a lot larger using this algorithm, resulting in a larger token.

# RS256 Assymetric Signature
private_key = JOSE::JWK.generate_key([:rsa, 4096])
public_key = JOSE::JWK.to_public(private_key)
header = { "alg" => "RS256" }

signed_rs256 = JOSE::JWT.sign(private_key, header, payload).compact
puts "Signed and Encoded JWS: #{signed_rs256}"
#=> Signed and Encoded JWS: eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJTaWxpY29uIEhhbHRvbiBTb2Z0d2FyZSBQMlAiLCJpYXQiOjE1NDAxMjE4NjMsImlzcyI6IkNocmlzIExhcnNlbiIsInN1YiI6IkpXVHMifQ.fzvQ4VPoA1DimHrP_b4kJY5k9vOgaR5rCZ85sKx1jYduuA0wLuu65M5hrvL3AeTs8xrkgqfK-joyU0PKBogJ76Dj5hb8yUmjxuvaG9kGxKM0VoJIS6V87N6wiQHu9SWCcIkYqovPnc01nDn7vAfsdHt6DkVRw2C3uWOR3kmwQT0iXWSjGMtm_uotqMexBNLghAf3LO131B8qltSguVeH-mtNFO18FNlMA6XlYB56dL-QYTWXk4VH5ocNiveUwmzlH8HmCq4Ms2tahXxHkHJRdQVmQU6EOodpldnx1CeZA7M-T7XBuB4byaYkpazrhotG8t4Z-SNuonWuE02D4N4vLpI2K2eVY4gjbqBmX5R8NHS8gY7gjmpbq3Xu2cgZIt0mgXMSUyrv-mYbVoTWBqx3Fs7lsXKCmBy6AGKxhMqFJPVFiW3WgjptEBhHmg0wRZxOW9RxtYqKO3h-2ytWlpZ06Wj_e4RuXeh6DkKwgBw1a9I1fbl-IyVfJt6cOLGEn7L9pmgZn1zcGtDcTvQdBqjwuX3DL3YDfRSb-Oxe5I-7tWRS_sMUuilULw4ITo0owo1z_rljtF5HvFyt8qU7_b6Be-kZcBAOSGVAoX7EyZznREpRpNM2taZB6XDrm7ZPkzPrHlHsaB7d1IfF-RwRk3X0tRGT1l-E-S3_1w6ZbwCh__k

verification = JOSE::JWT.verify_strict(public_key, ["RS256"], signed_rs256)
puts "Verification of JWS: #{verification.first}"
#=> Verification of JWS: true

While the first example used the same key for both signing and verifying, this example uses assymetric key pairs. In the case of signatures, the private key is used to sign an item, and the public key is used to verify it. This provides a more secure method than the symmetric key, as the private key, used to generate the signature doesn’t have to be shared. While sharing a symmetric key to the receiver of the token, it’s possible for it to be intercepted by a third party, and thus used to alter the data in the payload, and then provide a signature that matches the altered data. The public key doesn’t provide that ability, and thus is safe to share openl.

JWE

The process for building a JWE is a bit more complicated than that of the JWS. One of the key things to note about a JWE, is that there are two algorithms listed in the header. One is used to encrypt the plaintext of the message, and is included with the name “enc.” The other algorithm is listed in the header as “alg,” and is used to encrypt the key that encrypts the message.

  1. Encode JWE Protected header as BASE64URL(UTF8(JWS Protected Header))
  2. Generate a random Content Encryption Key(CEK)
  3. Encrypt the CEK with the recipient’s public key using the algorithm specified by “alg” in the header
  4. Base64url-encode the JWE Encrypted Key
  5. Generate a random Initialization Vector, and Base64URL encode it
  6. Generate the Additional Authenticated Data as a Base64URL encoding of the Protected Header
  7. Encrypt the plaintext with the algorithm specified by “enc” in the header, using the CEK as the encryption key, the Initialization Vector, and the Additional Authenticated Data value
  8. Concatenate these values in the order ProtectedHeader.EncryptedKey.InitializationVector.Ciphertext.AuthenticationTag
  9. This provides the URL safe JWE Compact Serialization

Here’s an example of encrypting a JWE using a symmetric algorithm:

require 'jose'

message = "Jodie Whittaker is the greatest Doctor ever!"

# A128GCM Symmetric Key Encryption/Decryption
secret = 'some128bitsecret'
jwk = JOSE::JWK.from_oct(secret)
encrypted_a128gcmkw = JOSE::JWE.block_encrypt(jwk, message, { "alg" => "A128GCMKW", "enc" => "A128GCM" }).compact
puts "encrypted JWE: #{encrypted_a128gcmkw}"
#=> encrypted JWE: eyJhbGciOiJBMTI4R0NNS1ciLCJlbmMiOiJBMTI4R0NNIiwiaXYiOiJnMW5Ld20xTlpjWExlMnpMIiwidGFnIjoiUkpSRll4cEVqY0NkMXN3a25MNnR2USJ9.rk77No8yWkIwH0TJpo60Nw.aJOp6rNyPnbivzcD.XVV3p1pzXH68wZvkRxU1Z3w8qHpvMuPudhqfsyaDP-556nII_MksS48rmx0.0jIODwJGT8emRqRexbkHWg

decrypted = JOSE::JWE.block_decrypt(jwk, encrypted_a128gcmkw).first
puts "Decrypted message: #{decrypted}"
#=>Decrypted message: Jodie Whittaker is the greatest Doctor ever!

Now, here is an example of encrypting the same message using an assymetric algorithm:

#RSA Assymetric Key Encryption/Decryption
puts "-----Assymmetric Key Encryption/Decryption-----"
private_key = JOSE::JWK.generate_key([:rsa, 4096])
public_key = JOSE::JWK.to_public(private_key)

encrypted_rsaoaep = JOSE::JWE.block_encrypt(public_key, message, { "alg" => "RSA-OAEP", "enc" => "A128GCM" }).compact
puts "encrypted JWE: #{encrypted_rsaoaep}"
#=> encrypted JWE: eyJhbGciOiJSU0EtT0FFUCIsImVuYyI6IkExMjhHQ00ifQ.G0XdCld3CSlX1ukuLX9wMiQKmBLDqWN4Af2Ow7a2zHiKKjFoEZ45DpNrvIki0wqDxojCX2iPcFZfWgOyS1vtutgSlUtFlrKwpUbD5pYt0R6iBGPIUWCa9fp-qaEiMOm0TVh00Z75R23Ziqr_SZ1ZlCzFPn5tbNfqrVIfxwMZN1hwXvBkF80DDQ-wqYQEUPKsshC65tckxM-xyqinYnmGGUuX3tjfPWlhG25i0AqEEhXWJ_uoy6BZrRFpdTQbtmRUerRQjFAIvloWRrl0a0cLMYcZAcSOSlpMVYRHWYmxhZzrB2GUKGlXp3LPtpBrMGf-o-5yL-ZmfHJXX4PsECp7wwg-3nPtdMSzBiPDZF2UPVFvbKGAqsliwfzZisGXJVZ3MsSrPjf0eMJibYyojRVCxEzfWhFx_ksGIEGWOc9rFhVGP6ruBO7VcYaEAitFx4vqvZ5tlEcJ7j_mjkiSu6LaU7xwD1lwZrGEGuOex9AknexNwSFTmPGTJW0_KxyZ73AJ7Ii8wLh8ckHUujOOvH3PyPmKqJNz_L-3-tMk5j9GoRaNS2LAUDhuCITC4WQoY64UPBSmkx8hqGHYdtzKmM_V8_F0qDih-WGxs4PvAV24y_ZESL-5NCBJgyJimWA8SWyP5T8uW9NkOPqMMxkXJDm0wBu7MRIxyEsHa8TDQl-iNiw.lYqyW-AXLSd-AEEV.cweZr5AUIY7vh1AnteK10lmdytJCWDXrBKLY0Trlj-kMp8Nyzb8_IAlIaZo.ZmTXFeamO5WlyifXxt2dJQ

decrypted = JOSE::JWE.block_decrypt(private_key, encrypted_rsaoaep).first
puts "Decrypted message: #{decrypted}"
#=> Decrypted message: Jodie Whittaker is the greatest Doctor ever!

Similar to the JWS examples, the first example using a symmetric key, while the second example here uses a public/private key pair. The risks inherent to the symmetric key are just as bad with a JWE. Any third party who intercepts a symmetric key would be able to decode the JWE and read the private data within. With a public/private key pair, the information is more secure. The receiver can send their public key to the sender, who then uses that key to encrypt the information, secure in the knowledge that the token can only be decrypted using the holder of the receiver’s private key.

Conclusion

Libraries for working with JWTs already exist in all major programming languages. Some of them only implement some functionality, and not all, so it’s best to do your research when choosing a library to use. JWT.io provides an extensive (although not complete) list of libraries for creating and using JWTs.

If you have any questions, please feel free to ask below, or reach out to me through Twitter.

Additional Resources

RFC7519 – JSON Web Token

RFC7165 – Use Cases and Requirements for JSON Object Signing and Encryption (JOSE)

JWT.io – An information website created and maintained by security company Auth0. While it does provide a great deal of information, it also ignores the JWE aspects of the JWT spec.

Private and Protected Methods: To Test, or Not To Test, That is the Question

For the last little while, I’ve had a dilemma upon me regarding the testing of private and protected methods. In keeping with the 3 rules of TDD, I wanted every bit of code that I wrote to be preceded by a test that would check that bit of code.

This lead me to the desire to test the private instance methods of a class I was building. I’d never had this come up before, so I started searching. Found some answers on StackOverflow, implemented them and everything was going fine.

Should I Test Private Methods?

Then started to ask myself if I SHOULD be doing this. As I said in the first paragraph, I did this because I wanted to test every little bit of code that I write. Afterwards though, I started to ask myself if that’s really the best way to do it.

Now yes, you do want full test coverage. We have tools like Ruby’s simplecov to ensure that every line of code you write is checked by your tests. However, think about what tests are actually supposed to accomplish. Your tests are supposed to mirror how other developers are going to use your library. They’re not going to call the private methods, they’re going to call the public methods that then call the private methods. So no, you really don’t need to test private methods.

What’s the harm if I want to?

Well, that’s the other question, isn’t it. Maybe you want to be absolutely sure that all your code works, private and public. That’s certainly a capital idea.

However, there are two downsides to directly testing your private methods.

First, comes refactoring. Proper TDD workflow is Red-Green-Refactor. You write your failing tests, you write code that makes it pass the test, then you refactor. Sometimes you refactor over and over again. The reason we do it in this order is so that we can refactor with the certainty that we’re not breaking the code. So, what happens when that refactoring involves changing your tests? Well then, we lose the certainty, and we run a risk of developing test apathy. That’s the last thing you want, in yourself, or your team.

The other reason to not test your private methods directly is for ensuring test coverage. Tools like the previously mentioned simplecov show you parts of your code are covered by tests and what parts aren’t. Test coverage tools are also good for helping you find code bloat. That is, parts of your code that never execute. If you’re testing your private methods directly, they’ll always be covered by tests, which means that they won’t be revealed as code bloat by test coverage tools. However, if you only test your public methods, and everything is covered other than your private method, well, you know you can get rid of it.

Conclusion

With all this in mind, I say DON’T test private methods. Your refactoring will be smoother, your test coverage will be more certain, and you’ll be fighting off test apathy.

TDD in Crystal with minitest.cr

Please note that this tutorial was written for Crystal 0.24.1 and minitest 0.3.6. I can’t be certain that these instructions will work for other versions of either.

Welcome back to my blog. Today we’re going to explore Test Driven Development(TDD) with the Crystal language. The Crystal compiler does include a fully featured spec library that was inspired by Rspec. However, I always preferred Minitest over Rspec in Ruby, so if you’re like me, you’ll be very pleased to know that there’s a minitest inspired Crystal library, appropriately called minitest.cr. So I’m going to show you how to use minitest.cr to create and run unit tests in your Crystal projects.

Creating The Project

First thing we’re going to do is create our project. Since we’ll be including a third party library in Crystal (also known as a Shard), we’ll be making use of the standard Crystal project structure. Luckily, the Crystal compiler has a built-in feature to create the project folder and necessary files. From the command line of the directory you want to create your project in, type the following command:

$ crystal init lib learning_minitest

Allow me to explain this for those of you who haven’t used this feature before. If you’re familiar with this command, feel free to skip to the next section.

The crystal command is the command for accessing the Crystal compiler, and its suite of built-in tools. In this case, we’re using the init tool to create our project. The lib command means that we’re creating a library. The other option is app, which is used for creating an application. After that, comes the directory(if not the current directory) and project name.

When run, this command will create a folder called learning_minitest and the project skeleton, complete with a git repository and .gitignore file.

Note: Make sure to use an underscore and not a dash in the project name. The Crystal compiler takes a dash to mean that you’re extending a library. In this case, that means you’ll get a module called Learning, and a module called Minitest within the Learning module. That’s not what we want. We just want one module called LearningMinitest.

Shard.yml

Third party libraries in Crystal are referred to as Shards. They’re very analogous to Ruby’s Gems. Following this analogy, Crystal uses a YAML file called shard.yml, which simultaneously fills the functions of Ruby’s gemfile and gemspec files. For those of you who haven’t worked with a shard file yet, I’ll briefly cover the features that are related to this tutorial. If you are familiar with the shard file, feel free to add the minitest shard (version 0.3.6), and move on to the next section.

The crystal init command we used earlier created a basic shard.yml file, which will look something like this:

name: learning_minitest
version: 0.1.0

authors:
 - Chris Larsen <clarsen@example.org>

crystal: 0.24.1

license: MIT

Now there’s a lot of information you can put into a shard file, but for our purposes, all we need to do is add a dependency. For other features, I recommend looking at the shard.yml specification page.

Right below where it says says crystal: 0.24.1, place the following lines:

development_dependencies:
   minitest:
     github: ysbaddaden/minitest.cr
     version: "0.3.6"

 

Shards accepts two different types of dependencies, regular and development, which are indicated by the line stating dependencies: or development_dependencies:, respectively. In this case, you’re not going to need your tests in production, so we’re declaring minitest as a development dependency.

Now that our dependency is declared, we need to actually download the files for it. The following command does that for us:

$ shards install

This will install the necessary files from the git repositories of all the dependencies listed in your shard.yml file. If you need to update the version of the shard you’re using, you can use the shards update command after updating the version number in your shard file.

Creating Tests

Since the creation of a Crystal project doesn’t give us a test folder, we’ll have to do that manually. Your test folder should reside in the top level directory of your project structure, just as it would in Ruby, and be named test. Within it, we’re going to need a file for our tests. In the interests of convention, I suggest ending the name of the file with _test.cr. In this case, I’m going with learning_test.cr. To start with, add the following lines to your file:

require "minitest/autorun"

require "/../src/learning_minitest.cr"

class LearningTest < Minitest::Test

end

The first require statement is part of the minitest library, which allows us to run the test methods that we’ll write. The second is the file in our own project that will house the class that we’re going to test. Lastly, the class, which is a subclass of Minitest::Test is where we put all our tests.

The minitest.cr library contains all the same life cycle hooks that the original minitest contained; before_setup, setup, after_setup, before_teardown, teardown, and after_teardown.

However, I must provide a quick word of warning. In Ruby, many developers use the setup methods to initialize an instance of the class. In Crystal, due to the compiler, it won’t work that way. Since the compiler doesn’t know that the setup methods will be called before every test, it treats any item initialized in those methods as a union type with nil.

This can cause all kinds of problems, so I don’t recommend doing it. The alternatives are to use memoization, as in:

def cat
  @cat ||= LearningMinitest::Cat.new
end

Or, you can define any instances to test right below the class declaration, and give them a default value, like so:

class LearningTest < Minitest::Test
  @cat = LearningMinitest::Cat.new

end

I prefer memoization, so let’s add those lines inside the class. In case you’re wondering, yes, we’re creating a Cat class.

Now, let’s create our first test. Simple, yet effective:

def test_cat_likes_petting
  cat.pet
  assert cat.happy
end

Since minitest.cr is a third party library, there aren’t any default tasks that run all your test files, so you run them manually, one at a time, like so:

$ crystal run test/learning_test.cr

All test methods must be prefaced with test_ in order to be recognized as test methods and run during this process. Running them now should give you an error:

Can't infer the type of instance variable '@cat' of LearningTest

@cat ||= LearningMinitest::Cat.new

This is because we haven’t yet created the class that we’re testing. So let’s do that now.

Creating Library

The crystal init command created a learning_minitest.cr file in our src directory. Right now, it looks like this:

require "./learning_minitest/*"

# TODO: Write documentation for `LearningMinitest`
module LearningMinitest
   # TODO: Put your code here
end

We’re going to create our Cat class within this module. We also need to add the pet method and happy read-only property. Remove the two # TODO statements and add the following lines after the module definition:

class Cat
  getter happy : Bool

  def initialize
    @happy = false
  end

  def pet
    @happy = true
  end
end

If you’re not familiar with the getter line there, that defines a boolean instance variable called happy, and also creates a getter method for it, thus making it read-only. You can use the same format to define write-only properties with setter, or read-write properties with property. The Bool keyword can also be replaced with any class name to define an instance variable of that type.

Now that our class is defined, along with the pet method and happy instance varible, we can run our test and expect a successful output.

.

Finished in 00:00:00.001435720, 696.5146407377483 runs/s

1 tests, 0 failures, 0 errors, 0 skips

This output is quite possibly the biggest difference between the Ruby minitest, and the Crystal version. Whereas Ruby gives you the number of runs and assertions, this one gives you the number of tests. So a test method with 5 assertions, will still output as 1 test.

Lifecycle Hooks

As stated earlier, the same lifecycle hooks exist as in the Ruby version, so let’s use one of those now. Add the following setup method to learning_test.cr:

def setup
  cat.name = "Pandora"
end

Now to test this:

def test_cat_has_adorable_name
  assert_equal "Pandora", cat.name
end

Which should result in an error. This we fix by adding a property to our Cat class, and assign a default value to it.

property name = ""

Yes, another feature of our instance variable declarations is the ability to assign a default value that the variable holds if it’s not assigned anything during the initialization process.

Now, both of our tests should pass.

..

Finished in 00:00:00.001404765, 711.8628382683224 runs/s

2 tests, 0 failures, 0 errors, 0 skips

There’s a lot more to minitest.cr, but most of it is exactly like the original Ruby minitest.

If your tests aren’t passing, or there’s anything you don’t understand about what we’ve done, you can either refer to the github repository for this project, or leave a comment below.

Many thanks to ysbaddaden for creating this shard, without which, this blog post would not have been necessary. The source code for the shard can be found here.

Crystal Grep

Today, we’re going to have some more command line fun with Crystal. Along the way, we’re also going to cover Regex, IO and other subjects.

This tutorial was influenced by my friend 8bitmiker and his recent YouTube video, where he shows you how to create a similar program in Ruby. The final version in Crystal can be found here.

To start with, we’re going to make sure that the program is called with an argument. Without any argument, there’s nothing for this program to do. You may remember from my previous blog post that Crystal uses the same ARGV array constant for command line arguments as Ruby. This, and other similarities allows us to use the same line as used in the Ruby version of this program.

abort "Need a regex pattern" unless ARGV.size > 0

The abort command does what you’d expect it to do, it exits the program immediately.

While Ruby has Array methods called length, size and count, Crystal has a non-alias philosophy, and in this case, has chosen size to return the number of elements in an array.

Combine these two with the in-line unless (another Ruby beauty), and the program will exit on the first line if the user didn’t give it any command line arguments.

Next, we’re going to take that Regex from the ARGV array, and store it. That way, we don’t have to keep accessing ARGV, and also make our code more readable. Using another Ruby inspired method, shift, will do the trick:

regex = ARGV.shift

Just like in Ruby, shift removes the first element in an array, and returns it. In this case, passing it into the aptly named regex variable. Now, it’s important to note, ARGV is an array of strings. The regex variable therefore, contains a string of the regex pattern that the user entered. Later, we’ll convert that string to a Regex object.

Now that we have our second line, we can show just how important that first line is. If the user doesn’t enter any command line arguments, ARGV will be an empty array, and trying to run shift on an empty array will result in an Index out of bounds error. Hence, the need to abort the program if there’s no arguments provided. Remember, Crystal may look like Ruby, but it’s not Ruby.

Let’s run a quick test of the program at this point. For a program like this, it’s best to compile and run the compiled version. I saved the file as cgrep.cr, so we can compile it with the command:

$ crystal build cgrep.cr

Now we run it with:

$ ./cgrep

This will of course return the message Need a regex pattern since we didn’t give it a regex pattern. To run it properly, you need to not only give it the regex pattern, but also give it some input to process.

$ ls -l | ./cgrep .*

If you’re not incredibly familiar with Command line interfaces, the pipe(|) operator takes the output from the left side, and inputs it to the standard input of the command on the right side. So this line takes the output from ls -l, and sends it to the standard input of our Crystal program.

Unlike the original Ruby program from 8bitmiker, Crystal doesn’t possess the $_ operator, so we’ll access our standard input using the built-in STDIN variable. STDIN is an IO::FileDescriptor object, so you can use any methods available to this class on your input. Since ls -l returns multiple lines, we’re going to use the popular each_line method, which iterates over each line in the input.

STDIN.each_line do |i|
end

And there’s our main loop. It will iterate over each line of the input, and pass that line into the i variable for access within the loop. Now, what we do we want to do with that line? Well, we want to compare it to the regex. To do that, we need to turn the string representation of our regex into a proper regex using interpolation:

%r(#{regex})

We also have the familiar match operator (=~) to see if the regex matches the line from standard input, and output if we have a match, like so:

puts i if i =~ %r(#{regex})

Placing that into the loop, and we have our finished program:

abort "Need a regex pattern" unless ARGV.size > 0
regex = ARGV.shift

STDIN.each_line do |i|
  puts i if i =~ %r(#{regex})
end

Running it with the same command as before:

$ ls -l | ./cgrep .*

Will produce an output similar to this:

total 2048
-rwxr-xr-x 1 chrislarsen staff 1037464 27 Dec 14:36 cgrep
-rw-r--r-- 1 chrislarsen staff 128 4 Jan 14:20 cgrep.cr

This is definitely not a production ready program. Ideally we’d have a check to make sure that the regex string is a regex pattern, and we’d interpolate that into a Regex object before the loop instead of wasting resources doing that conversion in the loop, but it does work as desired. It’s what 8bitmiker would call “sloppy and dirty.”

Constant Learning is Fun

Yes, you read that right. Learning is fun. No, I don’t work as a teacher, and I’m not doing this as a plug for academia. There’s no hidden agenda behind this blog post.

One of the many things that I’m doing in the tech world is learning the new Crystal programming language. My first project in Crystal is an interface for the Open Weather Map API. Recently, I learned that there was a far better(and simpler) way of coding the majority of this library. So here in late December, I spent half a day rewriting my only Crystal project from scratch.

As I headed out to a holiday party afterwards, I reflected on the fact that I was in the process of rewriting the entirety of my Crystal code. Naturally I did have a fleeting thought that this amounted to a lot of wasted time. However, this thought was very quickly replaced with a feeling of accomplishment.

You see, the time wasn’t wasted, because the first and second versions of my code were stages in my learning process. While the code of my first draft was very bad, I did learn a lot about this new project while writing it. While I am completely rewriting that first draft, the fact that I am, so soon after finishing it shows just speed at which I’m learning.

I’m not going to be sharing the technical aspects of what I learned in this blog post, but to keep your eyes open in the future for a post on the JSON.mapping macro, and how to use it to create elegant and efficient wrappers for your favourite web APIs.

In the meantime, allow me to pass on some helpful advice for beginner and expert programmers alike who find themselves in this same situation. Be proud that you wrote bad code, because you’re now good enough to recognize that it’s bad code. Replacing that bad code is proof of how good you are of a developer.

Misogyny in Our Industry

While most of my blog posts will focus on specific technologies and tutorials, every once in a while, I will talk about the culture and direction of the technology industry. The good, the bad, and the ugly.

For this first article, I’m going to address one of the ugliest parts of the industry, misogyny. This has been a problem for as long as the tech industry has existed, but recent illuminations of inappropriate behaviour by executives and VCs, as well as systemic sexism in a few of Silicon Valley’s well known startups.

Following the news of these poorly behaving executives, I came across news that was equally disturbing. An anonymous VC stated that he was cancelling meetings with female engineers because he was scared of ‘behaving wrongly.’ Two months after reading this article, I still can’t fathom how any grown man doesn’t have enough self-control to prevent himself from ‘behaving wrongly’ during a business meeting.

Uber’s Culture is Not Unique

Susan Fowler’s detailing of systemic sexism at Uber is widely known in this industry, and while she didn’t claim that these problems are unique to Uber, I’ve heard many people suggest that they are. However, a couple of months after this information was made public, Uber released their diversity report, which actually shows a greater percentage of female employees compared to other tech giants (although they do have smaller percentage of women in leadership and tech positions compared to most of the other firms). The numbers simply don’t support the idea that other companies aren’t the same. We need to look at the general culture of the tech world to fix it.

(Full disclosure: I do currently work as an Uber driver. This is not the reason why I don’t single out Uber for systemic sexism. Quite the opposite. I remain working as an Uber driver, because I understand that they’re not the only company with this problem.)

Remember the Gamergate controversy? Female game developers were being harassed, even to the point of receiving death threats and having their personal information shared online. Don’t forget the iLookLikeAnEngineer movement, motivated by people telling a woman that she couldn’t be an engineer because of the way she looks. It’d be nice if this problem were limited to one company, but it’s far worse than that.

Personal Observations

Living in Canada, and being a part of the Toronto tech community, I’m proud to say that sexism in our community isn’t anywhere near as bad as what it is in Silicon Valley. I’m lucky to know a number of women in the industry, who have well-respected positions at local companies. However, I won’t go so far as to say that our tech community doesn’t have sexism. I can recall one night where I heard a woman talk about sexism in the industry, to have a guy reply, “It’s a boy’s club, deal with it.” Sadly, I have seen far too much of guys not treating women very well, and making absolutely ridiculous excuses for it.

I don’t know how to talk to women!

By far, one of the most ridiculous excuses for inappropriate behaviour towards women. Women are people. There’s no reason why talking to them should be any different than talking to men. That female developer sitting to your left? You should talk to her the same way that you talk to the male developer sitting on your right. If you don’t talk to women the same way you talk to men, then your view of the world is sexist.

I’m socially awkward!

There are two reasons why being social awkward isn’t an excuse for behaving inappropriately towards women. First reason, if someone really were socially awkward, then they’d be equally inappropriate towards men as they are towards women, and I’ve never seen that to be the case. Second reason is, we’re adults. We know how to learn, grow and evolve as people. In the tech world, we’re constantly learning new tech skills, so there’s no reason you can’t learn social skills too.

The Rock Test

Possibly the most brilliant idea for my fellow heterosexual men on how to relate to women is The Rock Test. Blogger Anne Victoria Clark suggested this test as a way to avoid sexually harassing women. The test is simply this:

Treat all women like you would treat Dwayne “The Rock” Johnson.

This test even has the approval of Dwayne Johnson himself. If you’re ever in doubt of whether or not to say something to a woman, do this.

The Recap

For any men who may be reading these, let’s go over some key points so that you can be sure to be part of the solution, and not part of the problem.

  1. Women are people, just like men.
  2. Further to 1, when in any professional setting, talking to a woman should be the same as talking to a man.
  3. Building on 2, workplaces, business meetings, investment discussions and tech socials are professional settings, not pickup bars. Behave as such.
  4. If any thought pops into your head that you’re unsure of, put it through The Rock Test before opening your mouth, or moving any part of your body.

Edit: 2 days after publishing this post, I came across the news that Silicon Valley companies are looking to hire models for their holiday parties. Particularly, models with short skirts and cleavage. Seems that they’re not learning.