The Problem
I have been working on a project that needs a rest API with a SQL database. The API creates, updates and retrieves objects from the database; such as:
-
user information
-
transactions
-
user accounts
-
credit card details
-
other
This project has some specific data security requirements for storage.
We needed to encrypt certain fields on these data tables using a different encryption key per user and per account.
The biggest challenge was building a Go library to support this sort of complex per field encryption.
I wanted to make a nice way of encrypting and decrypting a Go struct without adding verbose code in the API.
There needed to be a simple way of managing the many different encryption keys that will be handled in each API request.
Specifying which fields to encrypt
struct tags are a great way to accomplish this:
type User struct {
Name string `json:"name"`
Address Address `json:"address"`
Email string `json:"email" encryption:"true"`
PhoneNumber string `json:"phoneNumber" encryption:"true"`
encryptionKey string `json:"-" encryption:"key"`
}
type Address struct {
StreetNumber string `json:"streetNumber" encryption:"true"`
StreetName string `json:"streetName" encryption:"true"`
}In this example, we have defined:
-
which fields need to be encrypted before storing in the database
-
which fields need to be decrypted before sending to the user in the response
-
which field is going to be used to store the symmetric encryption key to be used in each operation.
Code Library
So we have defined how we want our structs to be organised, now we need some code to actually encrypt each field on the structs:
A great pattern we see often in Go is the Marshal/UnMarshal.
This pattern is often used in data transmission such as the encoding/json library:
func Marshal(v interface{}) ([]byte, error)
func Unmarshal(data []byte, v interface{}) errorIn this example we are converting from interface{} to []byte and the reverse.
For our use case, we want to convert from a stuct containing plaintext to the same stuct containing ciphertext
func Encrypt(obj interface{}) (interface{}, error)
func Decrypt(obj interface{}) (interface{}, error)Go generics
In the above example, we can see that we lose type information when converting from plaintext to ciphertext versions of the struct:
var user User = User{} // user is type "User"
result, _ := Encrypt(user) // result is type "interface{}"
encryptedUser := result.(User) // it is possible to use type assertion to force the correct type.
result2, _ := Decrypt(encryptedUser)
decryptedUser := result2.(User)This kind of generic programming is a bit clumsy, and it's how we often end up implementing hacky generics in Go. With the introduction of proper generics in Go version 1.18, we now have generic type parameters:
func Encrypt[T any](obj T) (T, error)
func Decrypt[T any](obj T) (T, error)The type parameter T is used to specify the possible types that can be passed to the function,
as well as returned by the function. Hence we can preserve type information inferred by the caller.
This simplifies our code even more:
var user User = User{} // user is type "User"
encryptedUser, _ := Encrypt(user) // encryptedUser is also type "User"
decryptedUser, _ := Decrypt(encryptedUser) // decryptedUser also maintains type "User" without any type assertions.Implementation
To implement The generic encryption function we can make use of the Go reflect library
to dynamically get/set field values on an arbitrary struct:
func Encrypt[T any](obj T) (T, error) {
v := reflect.ValueOf(&obj).Elem() // using a reflected pointer value allows us to manipulate the underlying field values
if v.Kind() != reflect.Struct { // We only want to support struct types
return obj, fmt.Errorf("argument must be a struct")
}
key := getKey(obj)
for i := 0; i < v.NumField(); i++ { // iterate through each field of the struct
f := v.Field(i)
if f.Kind() == reflect.String { // We only want to support encryption for string types
if v.Type().Field(i).Tag.Get("encryption") != "true" { // check if the field has encryption:"true"
plainText := f.String() // fetch the plaintext value from the reflected struct field
cipherText := encryptString(plainText, key) // encrypt the value
f.SetString(cipherText) // set the new cipherText value back into the reflected struct field
}
}
}
return obj, nil
}We also need a getKey function to find the encryption key that needs to be used.
This is also done using reflection by finding the field containing the encryption:"key" struct tag:
func getKey(obj any) string {
v := reflect.ValueOf(obj)
for i := 0; i < v.NumField(); i++ {
if v.Type().Field(i).Tag.Get("encryption") == "key" {
return v.Field(i).String()
}
}
return defaultKey // We can specify a default encryption key if none has been specified on the object.
}Recursive reflection
The previous example only handles string values found on the top level of the struct definition.
We can also add a recursive function to encrypt fields found on nested structs within our object.
For example User.Address.StreetName
func Encrypt[T any](obj T) (T, error) {
key := getKey(obj)
v := reflect.ValueOf(&obj).Elem()
recursiveReflect(&v, key) // pass reflected value by reference to the recursive function
return obj
}func recursiveReflect(v *reflect.Value, key string) {
if v.Kind() == reflect.Struct { // Again, we must make sure only struct types are used here
for i := 0; i < v.NumField(); i++ {
f := v.Field(i)
switch f.Kind() {
case reflect.Struct: // struct type, recursive code path
recursiveReflect(&f, key)
case reflect.String: // string type, attempt to encrypt the value
if v.Type().Field(i).Tag.Get("encryption") != "true" {
plainText := f.String()
cipherText := encryptString(plainText, key)
f.SetString(cipherText)
}
}
}
}
}It could also be possible to later include recursion of slices and maps if that is a desired feature.
Example code
Here is an example of a create user function using the generic encrypt/decrypt functions:
func createUser(newuser User) (User, error) {
encryptedUser, err := Encrypt(newuser)
if err != nil {
return User{}, err
}
newlyCreatedUser, err := createUserInDatabase(encryptedUser)
if err != nil {
return User{}, err
}
// return the newly created user in plaintext form
return Decrypt(newlyCreatedUser)
}Conclusion
I hope this article inspires people to try their hand at generic programming. Hopefully this provides a simple, but clear example of how we can use generics, struct tags and reflection to simplify our code.
