Refined Types for Validated Configurations

# Refined Types for Validated Configurations

## Viktor Lövgren

<span class="vertical-align-middle">
  <i class="socicon-github"></i> [@vlovgr](https://github.com/vlovgr)
  <span> / </span>
  <i class="socicon-twitter"></i> [@vlovgr](https://twitter.com/vlovgr)
</span>

---

<span class="vertical-align-middle">
  <i class="fa fa-globe" aria-hidden="true"></i> [tech.ovoenergy.com](http://tech.ovoenergy.com)
</span>

<span class="vertical-align-middle">
  <i class="fa fa-twitter" aria-hidden="true"></i> [@OVOTechTeam](https://twitter.com/ovotechteam)
</span>

---

# This talk was compiled with [tut](https://github.com/tpolecat/tut)

```scala
import com.typesafe.config.ConfigValueFactory.fromAnyRef
import com.typesafe.config.{Config, ConfigFactory}
import eu.timepit.refined.api.{Refined, RefType, Validate}
import eu.timepit.refined.auto._
import eu.timepit.refined.boolean.And
import eu.timepit.refined.collection.NonEmpty
import eu.timepit.refined.numeric.{Greater, Interval, Positive}
import eu.timepit.refined.pureconfig._
import eu.timepit.refined.refineV
import eu.timepit.refined.types.numeric.PosInt
import eu.timepit.refined.types.string.NonEmptyString
import eu.timepit.refined.W
import java.net.ServerSocket
import scala.util.Try

def show[T](t: T): Unit =
  pprint.pprintln(t, width = 70)
```

---

# Why do we need configurations?

* Use the same binary in multiple environments

* Avoiding re-compile whenever settings change

* Keep secrets outside of version control

Bottom line: often necessary

---

# Configurations:
# What are some of the difficulties,<br/>and what can be done about them

---

# Why do we use configuration files?

* Change and override settings without re-compile

<br/>

Configuration library then helps with:

* loading/overriding values from properties and environment variables

* reusing and referencing across parts of your configuration

* merging and loading from files, URIs, and class path

* handling type conversions (`String => T`)

---

# Example: [Typesafe Config](https://github.com/typesafehub/config)

```scala
final case class Settings(config: Config) {
  object http {
    def apiKey = config.getString("http.api-key")
    def timeoutSeconds = config.getInt("http.timeout-seconds")
    def port = config.getInt("http.port")
  }
}

val config =
  ConfigFactory.parseString(
    """
      |http {
      |  api-key = ${?API_KEY}
      |  timeout-seconds = 10
      |  port = 989
      |}
    """.stripMargin
  ).resolve()

val settings = Settings(config)
```

---

# What's not so good here?

```scala
settings.http.apiKey
// com.typesafe.config.ConfigException$Missing: No configuration setting found for key 'http.api-key'
//   at com.typesafe.config.impl.SimpleConfig.findKeyOrNull(SimpleConfig.java:152)
//   at com.typesafe.config.impl.SimpleConfig.findOrNull(SimpleConfig.java:170)
//   at com.typesafe.config.impl.SimpleConfig.findOrNull(SimpleConfig.java:176)
//   at com.typesafe.config.impl.SimpleConfig.find(SimpleConfig.java:184)
//   at com.typesafe.config.impl.SimpleConfig.find(SimpleConfig.java:189)
//   at com.typesafe.config.impl.SimpleConfig.getString(SimpleConfig.java:246)
//   at Settings$http$.apiKey(<console>:32)
//   ... 429 elided
```

---

# What's not so good here?

```scala
val invalidConfig =
  config.
    withValue("http.api-key", fromAnyRef("")).
    withValue("http.timeout-seconds", fromAnyRef(-1))

val settings = Settings(invalidConfig)
```

```scala
show(settings.http.apiKey) // API key should not be empty
// ""

show(settings.http.timeoutSeconds) // Should be positive
// -1

show(settings.http.port) // Ports 0 to 1023 are system ports
// 989
```

---

# What's difficult with configurations?

* Often involves writing boilerplate code

* Tedious to test, rarely gets tested

* Almost never gets validated

* Mistakes can lead to disastrous results

---

# How should configurations be like?

* No boilerplate for declaring or loading

* Validated at compile-time or during startup

* Validation encoded as part of the types

* Secrets never touch persistent storage

---

# Eliminating boilerplate: [PureConfig](https://github.com/pureconfig/pureconfig)

```scala
final case class HttpSettings(
  apiKey: String,
  timeoutSeconds: Int,
  port: Int
)

final case class Settings(http: HttpSettings)

val settings = pureconfig.loadConfig[Settings](config)
```
```scala
show(settings)
// Left(ConfigReaderFailures(KeyNotFound("http.api-key", None), List()))
```

---

# What's still not so good here?

```scala
val settings = pureconfig.loadConfig[Settings](invalidConfig)
```

```scala
show(settings)
// Right(Settings(HttpSettings("", -1, 989)))

show(settings.map(_.http.apiKey)) // API key should not be empty
// Right("")

show(settings.map(_.http.timeoutSeconds)) // Should be positive
// Right(-1)

show(settings.map(_.http.port)) // Ports 0 to 1023 are system ports
// Right(989)
```

---

# Encoding validation: [refined](https://github.com/fthomas/refined)

```scala
// type NonEmptyString = String Refined NonEmpty
// type PosInt = Int Refined Positive

final case class HttpSettings(
  apiKey: NonEmptyString,
  timeoutSeconds: PosInt,
  port: Int Refined Greater[W.`1023`.T]
)

final case class Settings(http: HttpSettings)

val settings = pureconfig.loadConfig[Settings](config)
```

---

# Encoding validation: [refined](https://github.com/fthomas/refined)

```scala
show(settings)
// Left(
//   ConfigReaderFailures(
//     KeyNotFound("http.api-key", None),
//     List(
//       CannotConvert(
//         "989",
//         "eu.timepit.refined.api.Refined[Int,eu.timepit.refined.numeric.Greater[Int(1023)]]",
//         "Predicate failed: (989 > 1023).",
//         None,
//         Some("http.port")
//       )
//     )
//   )
// )
```

---

# Can we do better?

* Actually, port numbers are between 0 and 65535 (inclusive)

```scala
type NonSystemPortNumber =
  Int Refined Interval.Closed[W.`1024`.T, W.`65535`.T]
```

---

# Can we do even better?

* What about clashing ports?

* What if we allow impure functions?

```scala
final case class OpenPort()

implicit val openPortValidate: Validate.Plain[Int, OpenPort] =
  Validate.fromPartial(new ServerSocket(_).close(), "OpenPort", OpenPort())
```

```scala
show(refineV[OpenPort](989))
// Left("OpenPort predicate failed: Permission denied (Bind failed)")

show(refineV[OpenPort](10000))
// Right(10000)

show(refineV[OpenPort](65536))
// Left("OpenPort predicate failed: Port value out of range: 65536")
```

---

# Impurity means we need to be careful

* We get compile-time validation from refined using macros

* Means we might do the open port check at compile-time

```scala
final case class HttpSettings(
  apiKey: NonEmptyString,
  timeoutSeconds: PosInt,
  port: Int Refined OpenPort
)
```

```scala
HttpSettings(
  apiKey = "y2UX83yLeoGudKMZv9vb",
  timeoutSeconds = 10,
  port = 989
)
// <console>:37: error: OpenPort predicate failed: Permission denied (Bind failed)
//          port = 989
//                 ^
```

---

# Can we improve compile-time safety?

* Do you really need to use configuration files?

* What about writing your configurations in Scala?

* We should not store any secrets in source code

* Store configuration code elsewhere, later compile with your application

* Alternatively, load the minimum necessary from the environment
---

# A pure Scala configuration

```scala
final case class HttpSettings(
  apiKey: NonEmptyString,
  timeoutSeconds: PosInt,
  port: NonSystemPortNumber
)
// defined class HttpSettings

val settings =
  HttpSettings(
    apiKey = "y2UX83yLeoGudKMZv9vb",
    timeoutSeconds = 10,
    port = 10000
  )
// settings: HttpSettings = HttpSettings(y2UX83yLeoGudKMZv9vb,10,10000)
```

\* Assuming single environment, no secrets

---

# Loading values from the environment

Typically means we have to deal with:

* supporting different configuration sources

* converting from String to different types

* handle errors and accumulate them

* somehow manage multiple environments

* integrate with libraries like refined

---

<h1 style="margin-bottom: 0;">Ciris</h1>
<h2 style="margin-top: 0;"><a href="https://cir.is">https://cir.is</a></h2>

---

# What is Ciris about?

* Put as much as possible of your configurations in Scala

* Load only the necessary values (secrets, environment, …)

* Encode validation by using appropriate data types

* Deals with error handling and error accumulation

* Supports loading values from different sources

* Dependency-free core, modules for library integrations

---

# Loading the same configuration

```scala
import ciris._
import ciris.refined._

val settings =
  loadConfig(
    env[NonEmptyString]("API_KEY"), // Reads environment variable API_KEY
    prop[NonSystemPortNumber]("http.port") // Reads system property http.port
  ) { (apiKey, port) => HttpSettings(apiKey, 10, port) }
```

```scala
show(settings.left.map(_.messages))
// Left(
//   Vector(
//     "Missing environment variable [API_KEY]",
//     "Missing system property [http.port]"
//   )
// )
```

---

# Dealing with multiple environments

```scala
import _root_.enumeratum._

object configuration {
  sealed abstract class AppEnvironment extends EnumEntry
  object AppEnvironment extends Enum[AppEnvironment] {
    case object Local extends AppEnvironment
    case object Testing extends AppEnvironment
    case object Production extends AppEnvironment

val values = findValues
  }
}
```

---

# Dealing with multiple environments

```scala
import configuration._
import ciris.enumeratum._

val settings =
  withValue(env[Option[AppEnvironment]]("APP_ENV")) {
    case Some(AppEnvironment.Local) | None =>
      loadConfig { HttpSettings("changeme", 10, 4000) }
    case _ =>
      loadConfig(
        env[NonEmptyString]("API_KEY"),
        prop[NonSystemPortNumber]("http.port")
      ) { (apiKey, port) => HttpSettings(apiKey, 5, port) }
  }
```

```scala
show(settings)
// Right(HttpSettings(changeme, 10, 4000))
```

---

# Loading unary products and coproducts

```scala
import ciris.generic._
import shapeless.{:+:, CNil}

type SystemPortNumber = Int Refined Interval.Closed[W.`0`.T, W.`1023`.T]
final case class NonSystemPort(value: NonSystemPortNumber)
final case class SystemPort(value: SystemPortNumber)
final case class Port(value: NonSystemPort :+: SystemPort :+: CNil)

final case class Settings(port: Port)

val settings =
  loadConfig(prop[Port]("http.port"))(Settings)
```

\* Also works with value classes

---

# Loading quantities with unit of measure

```scala
import ciris.squants._
import _root_.squants.energy.Power
import _root_.squants.space.Area

implicit val source =
  ConfigSource.fromMap(ConfigKeyType("squants key")) {
    Map("power" -> "12.0 kW", "area" -> "2.4 km²")
  }
```

```scala
show { loadConfig(read[Power]("power"))(identity) }
// Right(12.0 kW)

show { loadConfig(read[Area]("area"))(identity) }
// Right(2.4 km²)
```

---

---

# Summary

* Configurations are typically associated with:  
  boilerplate, mistakes, no tests, no validation

* Use [PureConfig](https://github.com/pureconfig/pureconfig) to eliminate boilerplate code

* Use [refined](https://github.com/fthomas/refined) to encode validation in data types

* You can improve compile-time safety by writing as  
  much as possible of your configurations in Scala

* [Ciris](https://cir.is) is a library helping you to do so

---

# Questions?

## Thanks!

<h3 style="margin-bottom: 0;">Slides are available at:</h3>
<h3 style="margin-top: 0;"><a href="https://vlovgr.github.io/refined-types">vlovgr.github.io/refined-types</a></h3>