Scala and Kotlin Under One Roof

In one of our latest open-source projects, we’ve combined Kotlin and Scala code under a single Gradle project. Since both languages work on the Java Virtual Machine, we were confident we could pull this off. This post describes some of the bumps we found along the way and what we learned to solve them.

How we got there

Karat is a new library we are developing to bring the power of temporal logic to testing. In short, that means you can express sequences of actions like "at some point after this HTTP request is received, there should be another message of this form in this Kafka queue." This extends property-based testing by making the actions also available for randomization. Both Kotlin and Scala have mature and well-known libraries for property-based testing, namely Kotest and ScalaCheck. We decided that we wanted to integrate with those as tightly as possible, with the goal of enabling people to use temporal logic in a setup they’re already familiar with.

At the core of this style of testing, we find an algorithm that decides whether a particular temporal formula is true at a given point in the sequence, and how we should continue in the next step (the interested reader can check the details in this paper about Quickstrom). It was clear that duplicating this code and the whole framework to describe the temporal formulae was not desirable. Furthermore, we didn’t want to manage several different projects for each of the integrations with libraries. The decision was to host the common code and the integrations in a single repository.

Project set up

Scala projects often use SBT as a built tool, whereas Kotlin developers use Gradle almost exclusively. Fortunately, Gradle bundles a Scala plug-in. We found no problem in depending from the Kotlin common core into the Scala code.

Multiplatform

Both Kotlin and Scala support several targets for compilation; the former using the overarching Kotlin Multiplatform, the latter using sibling projects like Scala.js and Scala Native. Our initial feeling was that making all those platforms work together would be too difficult for a small gain, so we set JVM as our initial goal.

At some point, we decided to change the common library into a Kotlin Multiplatform project to target the same platforms as Kotest and Ktor do. To our (pleasant) surprise, Gradle was able to provide the JVM version correctly to the Scala platform, which needed no further adjustment.

Scala and binary compatibility

Alas, Scala 2 has a complicated binary compatibility (which Scala 3 solves). In practice, that means each library must be compiled several times, once for each supported compiler version. This is something you don’t notice using SBT,

libraryDependencies += "org.scalacheck" %% "scalacheck" % "1.15" % "test"

but is reflected in the artifacts published in Maven Central as a suffix. The single ScalaCheck library becomes scalacheck_2.13 and scalacheck_3.

Gradle doesn’t understand this convention. We started by depending on a single version of the library, which was good enough to make progress. However, when we looked at publishing the library, it was clear that we had to depend on the correct library for each compiler version. Fortunately, there’s a [Scala Multi-Version plug-in for Gradle (https://github.com/ADTRAN/gradle-scala-multiversion-plugin) that solves the issue. When you declare your Gradle dependencies, you use %% to stand for the compiler version; here’s how it looks using version catalogs.

scalacheck-core = { module = "org.scalacheck:scalacheck_%%", version.ref = "scalacheck" }

That plug-in also takes care of packaging and publishing different artifacts, according to the Scala versions you declare as supported.

JUnit Platform vs. MUnit vs. JUnit

Kotest uses the JUnit 5 runner, which means that you need to include useJUnitPlatform to make those tests available to Gradle.

tasks.withType<Test>().configureEach {
  useJUnitPlatform()
}

ScalaCheck doesn’t have direct integration with JUnit, but we found that we could run ScalaCheck through MUnit, which does support it. This requires some changes to the way you declare the properties (you cannot use Properties) but it was a small change overall. Alas, after setting everything up, tests were not discovered!

It turns out that MUnit uses the JUnit 4 runner instead. This requires a tiny change in the Scala project.

  tasks.withType<Test>().configureEach {
-   useJUnitPlatform()
+   useJUnit()
  }

Generics

Java supports generics, Scala supports generics, Kotlin supports generics, so what’s all the fuss about? The problem is that Java takes a different approach to variance than Scala and Kotlin do, and this permeates the entire ecosystem.

Before moving on, Scala supports a more powerful form of generics in higher-kinded types, which become very relevant when working on FP-oriented stacks like Typelevel‘s. Kotlin doesn’t support those, and this is one of the reasons we decided that Scala code would consume Kotlin code, but not the other way around.

Variance refers to the way we relate generic types and subtyping. For example, if Dog is a subtype of Mammal, does it mean that List is a subtype of List? The answer depends on how we use the Dogs in the List, and is given to the compiler as variance annotations. And here comes the difference: in Java, those annotations are given solely in methods,

public static void pet(List<? extends Mammal> list) { ... }

whereas both Scala and Kotlin support annotations as part of types,

interface List<out A>: Collection<A> { ... }

Both Kotlin and Scala support the same feature, but the underlying virtual machine doesn’t. That means that both compilers need to translate from their source code into another form, plus some additional information to recover the variance not encoded in the Java types. Alas, this additional information is not saved in the same form, so a completely reasonable Kotlin type signature becomes a terrible mess from the Scala point of view.

At the moment of writing, we found no other solution than explicitly casting to types with variance on the Scala side. For example, here, we give a generic type with variance,

formula.asInstanceOf[Formula[Info[_ <: Action, _ <: State, _ <: Response]]]

to an argument that was declared as having type Formula<Info> in Kotlin, where the arguments to Info has out variance. Obviously, the best solution here would be for Scala and Kotlin compiler authors to agree on some form of shared variance annotations.

Kotlin-specific features

The rest of the problems were related to language features specific to Kotlin. As in the case of generics, understanding the translation from Kotlin to Java bytecode proved to be incredibly useful in these cases.

Top-level and extension functions

In Kotlin, you can define a function (or a property) directly in the file, as opposed to requiring a surrounding type, as in Java or Scala 2 (Scala 3 does have top-level definitions).

fun pet(cuties: List<Mammal>) { ... }

You can take this a step further and "fake" that this function has been declared over an existing type using extension functions (this has also been added to Scala 3, which previously required implicit definitions).

fun List<Mammal>.pet() { ... }

When consuming that Kotlin code from Scala, the functions reveal once again their true form. In this case, they are declared as static members of a new class defined with the name of the type,

public class ExtensionsKt {
  public static void pet(List<? extends Mammal> cuties) { ... }
}

Fortunately, Scala allows importing all the static members of the class into scope. That means that we can fake having pet as a top-level definition,

import ExtensionsKt._

As with generics, it would be beneficial if Kotlin and Scala (or the entire JVM) supported a shared way to declare a member as top-level.

Coroutines

If there’s one feature that sets Kotlin apart from the rest of the JVM languages, that’s definitely coroutines (although my colleagues are working hard on bringing this style to Scala 3!). Declaring a function as suspend requires minimal changes in Kotlin code,

suspend fun List<Mammal>.pet() { ... }

but has profound effects on the way the code is compiled. Ilmirus has written an in-depth guide on the matter, but for our purposes, we only need to know that a suspended function doesn’t return normally. Instead, it ends by calling a function passed as an additional argument. In our example above, the code looks as follows from a JVM perspective.

public class ExtensionsKt {
  public static pet(List<? extends Mammal> cuties, Continuation<Unit> c) {
    ...
    // end by calling the continuation
    c.resume(Unit)
  }
}

In theory, interacting with this code shouldn’t be too difficult; we just need to build a Continuation to specify how to "return." Since Continuation is a functional interface, we can use well-known lambda expressions for it. Alas, practice threw us a rock and every possible way to interact with suspended functions ended on IllegalStateException or ClassCastException. After too many unsuccessful attempts, we admitted defeat and created almost duplicate definitions of some of the core types and functions.

Having to say goodbye to one of our favorite Kotlin features leaves us with a sour taste. We look forward to Project Loom as a shared substrate for coroutines in the entire JVM, which may make the story here have a much happier ending.

Conclusion

We’re pretty happy with the current state of interoperability between Kotlin and Scala, and we’re moving forward with the "two languages under one roof" model in our project. It’s no secret that we, the functional team at Xebia, love both Scala and Kotlin. If you’re interested in talking to us, you can use our contact form, or head to our training platform to learn more about these two ecosystems.