Better Living Through sbt

Luke Amdor

@rubbish

Screenshot 2014-03-01 14.31.50.png
  • 50ish developers (including intern army)
  • Half do scala full time

Large number of projects

Number of libs

Number of deployables

Separate repositories

Very few multi-module projects

sbt

As number of projects grow

PAIN

setup becomes painful

clashing dependencies becomes painful

maintenance bcomes painful

variance becomes painful

maven

Parent pom.xml

Apache parent pom

In sbt terms, organizational plugin

banno-sbt-plugin

sbt plugin

standardized settings

bag of tricks

Not really the conventional plugin layout

depends on other plugins

  • sbt-assembly
  • aether-deploy
  • sbt-revolver
  • sbt-release

import com.banno._

name := "aws-utils"

BannoSettings.settings

addBannoDependencies(
  "banno-utils",
  "banno-health"
)

libraryDependencies += 
  "com.amazonaws" % "aws-java-sdk" % "1.6.10"

Specs2.settings

Settings

object BannoSettings {
  val settings =
    Seq(organization := "com.banno",
        version in ThisBuild := "1-SNAPSHOT",
        scalaVersion := "2.10.3"
      ) ++
    Seq[Setting[_]](bannoDependencies := Seq.empty,
                    libraryDependencies <++= bannoDependencies) ++
    Seq(publishArtifact in (Compile, packageSrc) := false,
        publishArtifact in (Compile, packageDoc) := false) ++
    Seq(javaOptions ++= Seq("-Djava.awt.headless=true", "-Xmx1024M", "-XX:MaxPermSize=512m")) ++
    Revolver.settings ++
    BannoCi.settings ++
    BannoNexus.settings ++
    BannoCommonDeps.settings ++
    BannoCompile.settings ++
    BannoRelease.settings ++
    BannoPrompt.settings ++
    BannoIvy.settings
}

Seq(organization := "com.banno",
    version in ThisBuild := "1-SNAPSHOT",
    scalaVersion := "2.10.3"
  ) ++ ...

Seq(publishArtifact in (Compile, packageSrc) := 
      false,
    publishArtifact in (Compile, packageDoc) :=
      false) ++

Seq(javaOptions ++= Seq("-Djava.awt.headless=true",
                        "-Xmx1024M",
                        "-XX:MaxPermSize=512m")) ++

Revolver.settings ++

BannoCi.settings ++

val ci = taskKey[Unit]("ci")
ci <<= 
  publish.dependsOn(
    (test in Test).dependsOn(clean)
  )

BannoNexus.settings ++

publishTo <<= (version) { v =>
  if (v.endsWith("SNAPSHOT"))
    Some(bannoSnapshots)
  else
    Some(bannoReleases)
}

BannoCommonDeps.settings ++

val settings: Seq[Setting[_]] = Seq(
  libraryDependencies ++= Seq(
    "org.joda" % "joda-convert" % "1.1",
    "joda-time" % "joda-time" % "2.0",
    "org.slf4j" % "slf4j-api" % "1.7.5",
    "org.slf4j" % "log4j-over-slf4j" % "1.7.5",
    "org.slf4j" % "jcl-over-slf4j" % "1.7.5"
  )
) ++ LogbackDeps.settings

BannoCompile.settings ++

scalacOptions ++=
  Seq("-deprecation", "-feature",
      "-language:implicitConversions",
      "-language:higherKinds",
      "-language:existentials",
      "-language:postfixOps","-Ywarn-adapted-args",
      "-Ywarn-dead-code", "-Ywarn-inaccessible",
      "-unchecked")

scalacOptions in Test +=
  "-language:reflectiveCalls"

BannoRelease.settings ++

BannoIvy.settings

Dependencies

Dependency Sets

BannoCommonDeps.settings ++

joda-time and slf4j+overs/logback

Other dependency sets

  • Akka.settings
  • Spray.client
  • Spray.server
  • Spray.caching
  • Scalaz.settings
  • Specs2.settings
  • Unfiltered.settings
  • ….

object Akka {
  val version = SettingKey[String]("akka-version")
  val settings = Seq(
    libraryDependencies ++= 
      Seq("com.typesafe.akka" %% "akka-actor" % version.value)
  )
}

  • Set version and allow project override.
  • Sometimes version is dependent on scalaVersion

BannoIvy.settings

Dependency management is fine art

  • By default:
    • Removes all other logging frameworks
    • some duplicates
  • lots of XML transformations for ivyXML setting
  • BannoIvy.addExclude(..)
  • BannoIvy.overrideVersion(..)

clear-local-banno-artifacts

clearLocalBannoArtifacts := {
  import IO._
  import Path._
  delete(userHome/".ivy2"/"cache" /"com.banno")
  delete(userHome/".ivy2"/"local" /"com.banno")
  delete(userHome/".ivy2"/"cache" /"scala_2.10" /"sbt_0.13" /"com.banno")
}

Versioning and Releases

Wants:

  • fluidity during development
  • reproducable artifacts

addBannoDependency(artifactId)

Keeping a set of Banno "owned" dependencies

val bannoDependencies =
  SettingKey[Seq[ModuleID]]("banno-dependencies")

libraryDependencies <++= bannoDependencies

addBannoDependency(artifactId)

bannoDependencies <+=
  "com.banno" %% artifact % bannoDepVersion

Use snapshots when developing, stable releases when releasing

val bannoDepReleasedVersion =
  SettingKey[String]("%s-released-version"
                       .format(artifactId))

bannoDepVersion <<=
  (version, bannoDepReleasedVersion) {
  (v, released) =>

  if (v.endsWith("SNAPSHOT")) snapshotVersion
  else released
}

Released version settings are kept in versions-banno-deps.sbt

BannoRelease.settings

leverages sbt-release

Steps our release goes through

  1. Updates Banno deps to newest releases (queries Nexus)
  2. Updates version to newest release (queries Nexus)
  3. Cleans / Tests
  4. Commits / Tags
  5. Pushes release
  6. Publishes
  7. Updates version back to SNAPSHOT
  8. Commits / Pushes

Data Services Build Pipeline.png

Most of the time developing, binary SNAPSHOTs are OK

Fast cross-project development mode

symlink external banno dependency

Screenshot 2014-03-01 07.19.25.png http://code.technically.us/post/9545154150/local-external-projects-in-sbt

automatically pick up symlinks under root dir and turn them into external project references

/Users/luke/code/b/aws-utils:
total used in directory 48 available 19016313
drwxr-xr-x  12 luke  staff   408 Mar  1 11:56 .
drwxr-xr-x  64 luke  staff  2176 Feb 25 22:40 ..
drwxr-xr-x  16 luke  staff   544 Mar  1 11:56 .git
-rw-r--r--   1 luke  staff   151 Aug 28  2013 .gitignore
-rw-r--r--   1 luke  staff    20 Aug 28  2013 README.md
lrwxr-xr-x   1 luke  staff    30 Mar  1 11:56 banno-utils -> /Users/luke/code/b/banno-utils
-rw-r--r--   1 luke  staff   305 Mar  1 11:57 build.sbt
drwxr-xr-x   6 luke  staff   204 Jan  8 15:27 project
drwxr-xr-x   4 luke  staff   136 Aug 28  2013 src
drwxr-xr-x   8 luke  staff   272 Feb 12 17:46 target
-rw-r--r--   1 luke  staff    40 Aug 28  2013 version.sbt
-rw-r--r--   1 luke  staff   150 Feb 12 17:46 versions-banno-deps.sbt

class BannoBuild(id: String) extends Build {
  // this is weird since if we're the symlinked project we read from the root of the symlinkee project
  def findSymlinkedProjectFiles(cwd: File = file(".")): Seq[File] = {
    val currentSymlinkProjects = cwd.listFiles.filter(Symlink.isSymlinkDirectory)
    val allSymlinkProjects = for {
      dir <- currentSymlinkProjects
      symlinksUnderDir = findSymlinkedProjectFiles(dir)
    } yield (dir, symlinksUnderDir)

    val maybeUs = allSymlinkProjects.collectFirst {
      case (dir, symlinks) if dir.getName == id =>
        symlinks.map(f => file(f.getName))
    }
    maybeUs getOrElse currentSymlinkProjects
  }
  lazy val symlinkedProjects =
    findSymlinkedProjectFiles().map(sp => RootProject(sp): ClasspathDep[ProjectReference])

  lazy val proj = Project(id = id, base = file("."), dependencies = symlinkedProjects).settings(
    name := id
  ).settings(
    BannoSettings.settings: _*
  )
}

object Symlink {
  def isSymlinkDirectory(dir: File) = dir.exists && dir.isDirectory && isSymlink(dir)
  def isSymlink(file: File) = file.exists && {
    val canon = new File(file.getParentFile.getCanonicalFile, file.getName)
    canon.getCanonicalPath != canon.getAbsolutePath
  }
}

import com.banno._

lazy val root = BannoBuild("aws-utils")

BannoSettings.settings

addBannoDependencies(
  "banno-utils",
  "banno-health"
)

libraryDependencies +=
  "com.amazonaws" % "aws-java-sdk" % "1.6.10",

Specs2.settings

Things banno-sbt-plugin could be better at

Not so banno specific (maybe!)

Tests through sbt scripted

Cruft

  • New macro hotness
  • Old dep versions lingering

Thanks