DevOps(4)Build RPM for CentOS

DevOps(4)Build RPM for CentOS

1. On CentOS

Install the RPM build ENV
>sudo yum install rpm-build
>sudo yum install gcc
>sudo yum install rpmdevtools
>sudo yum install rpmlint


2. On Ubuntu
>sudo apt-get install lintian
>sudo apt-get install fakeroot
>sudo apt-get install rpm


Error Message:
rpm:packageBin) java.io.IOException: Cannot run program "rpmbuild"

Solution:
Install the RPM build ENV
>sudo yum install rpm-build
>sudo yum install gcc
>sudo yum install rpmdevtools
>sudo yum install rpmlint

Error Message:
(localpoint-bridge-api-rpm/rpm:publish) java.io.IOException: destination file exists and overwrite == false

Solution:
I change the file 
>vi ~/.sbt/repositories 
[ivy] sbt.override.build.repos=true

But it does not work. I will check the artifactory side.

Error Message:
debian:genChanges) Cannot generate .changes file without a changelog

Solution:
Place a file changelog.txt in the project place will help.


Publish the RPM Build
>;project projectname; clean; project projectname-rpm; clean; rpm:publish

Package the PRM Build
>;project projectname; clean; project projectname-rpm; clean; rpm:packageBin

Package the Debian Build
>;project projectname; clean; project projectname-rpm; clean; debian:packageBin

3. Some Configuration in the Project Directory
I just do not plan to spend a lot of time on this right now. So I will just log the conf here.
build.properties
// 0.13.2 defaults to publishing ivy artifacts with overwrite = false, which we don't want to deal with yet
sbt.version=0.13.6

Build.scala
import sbt._
import Keys._
import com.typesafe.sbt.packager.Keys._
import scalariform.formatter.preferences._
import com.typesafe.sbt.SbtScalariform._
import spray.revolver.RevolverPlugin._

import org.sbtidea.SbtIdeaPlugin._


object ApplicationBuild extends Build {
  lazy val ourVersion = Process("git describe --match=[^(jenkins)].* --always --long").lines.head

  val akkaVersion = "2.2.0-RC1"

  val sprayVersion = "1.2-M8"

  val publishEnv = SettingKey[String]("publishEnv","artifact environment to publish to")

  override lazy val settings: Seq[Project.Setting[_]] =
    super.settings ++ Seq(
      scalaVersion := "2.10.3",
      version := ourVersion,
      organization  := "com.sillycat",
      resolvers := Seq(),
      externalResolvers := Seq(
        Resolver.defaultLocal,
        “sillycat Artifactory" at "http://repository.sillycat.com/artifactory/remote-repos"
      ),
      credentials += Credentials("Artifactory Realm", "repository.sillycat.com", "builder", “sillycat2011!"),
      publishTo := Some(“sillycat Artifactory Dev" at "http://repository.sillycat.com/artifactory/sillycat-dev/")
    )

  lazy val all = Project(id = "all",
    base = file(".")).
    aggregate(
      main,
      gerd,
      auth,
      util,
      localpointBridgeApi,
      migration,
      messageQueue,
      profilesSpark,
      localpointAuthCenter
    )

  /**
   Several projects have 2 ids, e.g. foo and foo-rpm
   The 1st is for ease of development, so that changes to guts are recognized without publishing.
   It depends on main, which will put guts on the classpath before libraryDependencies.
   The 2nd is specifically for building the rpm, and uses the same sourcetree as the project above.
   It doesnt use dependsOn; the guts jar is included from libraryDependencies,
   so guts must be published before the rpm can be built.
  */

  lazy val main = Project(id = "guts",
    base = file("guts")).
    dependsOn(util).
    settings(scalariformSettings ++ Revolver.settings ++ Seq(
      name := "guts",
      scalacOptions := Seq("-unchecked", "-deprecation", "-feature", "-encoding", "utf8"),
      publishEnv := "develop",
      libraryDependencies ++= Seq(
        // do not upgrade mysql connector, 5.1.27 broke hibernate
        "mysql" % "mysql-connector-java" % "5.1.26",
        "org.apache.cassandra" % "cassandra-all" % "1.2.8",
        "com.datastax.cassandra" % "cassandra-driver-core" % "1.0.3",
        //https://github.com/jorgeortiz85/scala-time
        "org.scalaj" % "scalaj-time_2.10.0-M7" % "0.6",
        "com.typesafe.slick" %% "slick" % "1.0.1",
        "com.typesafe.slick" %% "slick-testkit" % "1.0.1" % "test",
        "com.typesafe" % "config" % "1.0.0",
        "org.scalatest" % "scalatest_2.10" % "1.9.1",
        "io.spray" % "spray-json_2.10" % "1.2.3",
        "c3p0" % "c3p0" % "0.9.1.2",
        "com.typesafe" %% "scalalogging-slf4j" % "1.0.1",
        "com.esotericsoftware.kryo" % "kryo" % "2.21",
        "org.bouncycastle" % "bcprov-jdk16" % "1.46",
        "org.jasypt" % "jasypt" % "1.9.0",
        "com.google.guava" % "guava-parent" % "14.0.1",
        // this is needed for guava
        "com.google.code.findbugs" % "jsr305" % "2.0.1",
        "redis.clients" % "jedis" % "2.0.0",
        "org.apache.kafka" % "kafka_2.10" % "0.8.0",
        "org.msgpack" %% "msgpack-scala" % "0.6.8",
        "com.googlecode.flyway" % "flyway-core" % "2.2.1"
      ),
      fork := true,
      publishArtifact in Test := true,
      publishTo <<= publishEnv { (p: String) =>
        if(p == "master")
          Some(“sillycat Artifactory master" at "http://repository.sillycat.com/artifactory/sillycat-master")
        else if(p == "feature")
          Some(“sillycat Artifactory feature" at "http://repository.sillycat.com/artifactory/sillycat-feature")
        else if(p == "release")
          Some(“sillycat Artifactory release" at "http://repository.sillycat.com/artifactory/sillycat-release")
        else if(p == "maven")
          Some(Resolver.file("file",  new File(Path.userHome.absolutePath+"/.m2/repository")))
        else
          Some(“sillycat Artifactory dev" at "http://repository.sillycat.com/artifactory/sillycat-dev")
      }
    ): _*)

  lazy val auth = Project(id = "localpoint-auth",
    base = file("localpoint-auth")).
    dependsOn(main % "test->test;compile->compile").
    dependsOn(util).
    settings(scalariformSettings ++ Revolver.settings ++ Seq(
      name := "localpoint-auth",
      scalacOptions := Seq("-unchecked", "-deprecation", "-encoding", "utf8"),
      libraryDependencies ++= Seq(
        "io.spray" % "spray-can" % sprayVersion,
        "io.spray" % "spray-routing" % sprayVersion,
        "io.spray" % "spray-testkit" % sprayVersion,
        "io.spray" % "spray-caching" % sprayVersion,
        "io.spray" % "spray-client" % sprayVersion,
        "com.typesafe.akka" %% "akka-actor" % akkaVersion,
        "com.typesafe.akka" %% "akka-testkit" % akkaVersion,
        "com.typesafe.akka" %% "akka-transactor" % akkaVersion
      ),
      fork := true
    ): _*)

  lazy val util = Project(id = "localpoint-auth-util",
    base = file("localpoint-auth-util")).
    settings(
      autoScalaLibrary := false,
      crossPaths := false
    )

  lazy val gerdSettings: Seq[Setting[_]] = scalariformSettings ++ Seq(
    name := "gerd",
    // gerd actually has more than one consumer, but this is the first one in the chain
    mainClass := Some("com.sillycat.gerd.EntryEventConsumer"),
    libraryDependencies ++= Seq(
      "log4j" % "log4j" % "1.2.17",
      "com.github.sgroschupf" % "zkclient" % "0.1",
      "com.yammer.metrics" % "metrics-core" % "2.2.0",
      "org.apache.kafka" % "kafka_2.10" % "0.8.0" intransitive()
    )
  )

  lazy val gerd = Project(id = "gerd",
    base = file("gerd")).
    dependsOn(main)

  lazy val gerdRpm = Project(id = "gerd-rpm",
    base = gerd.base).
    settings(gerdSettings ++ Packaging.settings ++ Packaging.server ++ Seq(
      target := file("gerd/target-rpm"),
      packageSummary := "Generalized Event Routing and Distribution (gerd)",
      packageDescription := "consumers that process device traffic events from kafka",
      publishTo := Packaging.rpmCalcPublishTo("gerd"),
      libraryDependencies += "com.sillycat" %% "guts" % ourVersion,
      ideaIgnoreModule := true,
      rpmPost := Some(Packaging.rpmCalcPostChown("gerd"))
    ): _*)

  lazy val localpointBridgeApiSettings: Seq[Setting[_]] =
    scalariformSettings ++ Revolver.settings ++
    Seq(
      name := "localpoint-bridge-api",
      mainClass := Some(“com.sillycat.localpoint.console.api.LocalpointBridgeAPIServer"),
      scalacOptions := Seq("-unchecked", "-deprecation", "-encoding", "utf8"),
      libraryDependencies ++= Seq(
        "org.specs2" %% "specs2" % "1.13" % "test",
  "io.spray" % "spray-io" % sprayVersion,
  "io.spray" % "spray-can" % sprayVersion,
  "io.spray" % "spray-routing" % sprayVersion,
  "io.spray" % "spray-caching" % sprayVersion,
  "io.spray" % "spray-http" % sprayVersion,
  "io.spray" % "spray-testkit" % sprayVersion,
  "io.spray" % "spray-util" % sprayVersion,
  "io.spray" % "spray-client" % sprayVersion,
        "com.typesafe.akka" %% "akka-actor" % akkaVersion,
        "com.typesafe.akka" %% "akka-testkit" % akkaVersion,
        "com.typesafe.akka" %% "akka-transactor" % akkaVersion
      ),
      fork := true
    )

  lazy val localpointBridgeApi = {
    val dir = "localpoint-bridge-api"

    BuildInfo.write(dir,
      Process("git rev-parse HEAD").lines.head,
      Process("git rev-parse --abbrev-ref HEAD").lines.head,
      ourVersion)

    Project(id = "localpoint-bridge-api",
      base = file(dir)).
      dependsOn(auth).
      dependsOn(main % "test->test").
      dependsOn(messageQueue).
      settings(localpointBridgeApiSettings: _*)
  }

  lazy val localpointBridgeApiRpm = Project(id = "localpoint-bridge-api-rpm",
    base = localpointBridgeApi.base).
    settings(localpointBridgeApiSettings ++ Packaging.settings ++ Packaging.server ++ Seq(
      target := file("localpoint-bridge-api/target-rpm"),
      packageSummary := "localpoint bridge api",
      packageDescription := "localpoint bridge api for configuration of brand-specific settings",
      publishTo := Packaging.rpmCalcPublishTo("localpoint-bridge-api"),
      libraryDependencies ++= Seq(
        "com.sillycat" %% "localpoint-auth" % ourVersion,
        "com.sillycat" %% "message-queue" % ourVersion
      ),
      ideaIgnoreModule := true,
      rpmChangelogFile := Some("changelog.txt"),
      maintainer := "Carl <[e-mail]>",
      rpmPost := Some(Packaging.rpmCalcPostChown("localpoint-bridge-api"))
    ): _*)

  lazy val migration = Project(id = "migration",
    base = file("migration")).
    dependsOn(main).
    settings(scalariformSettings ++ Seq(
      name := "migration",
      scalacOptions := Seq("-unchecked", "-deprecation", "-encoding", "utf8")
    ): _*)

  lazy val messageQueueSettings: Seq[Setting[_]] = scalariformSettings ++ Seq(
    name := "message-queue",
    mainClass := Some("com.sillycat.messagequeue.Workers"),
    libraryDependencies ++= Seq(
      "com.rabbitmq" % "amqp-client" % "3.2.1",
      "net.jodah" % "lyra" % "0.3.2",
      "org.slf4j" % "slf4j-api" % "1.6.1",
      "com.sillycat" %% "push_notifications_api" % "0.1-0-g20d290d",
      "com.lmax" % "disruptor" % "3.2.0",
      "org.javolution" % "javolution-core-java" % "6.0.0"
    ),
    initialCommands in console := """
import com.sillycat.messagequeue.disruptor._
import com.sillycat.messagequeue.rabbitmq._
import com.sillycat.messagequeue.rabbitmq.CampaignPoller._
import com.sillycat.messagequeue.pushy._
import com.sillycat.messagequeue._
import com.sillycat.localpoint.model.DAL
import org.joda.time.{DateTimeZone, DateTime}""",
    initialCommands in consoleQuick := ""
  )

  lazy val messageQueue = Project(id = "message-queue",
    base = file("message-queue")).
    dependsOn(main).
    dependsOn(main % "test->test").
    settings(messageQueueSettings: _*)

  lazy val messageQueueRpm = Project(id = "message-queue-rpm",
    base = messageQueue.base).
    settings(messageQueueSettings ++ Packaging.settings ++ Packaging.server ++ Seq(
      target := file("message-queue/target-rpm"),
      packageSummary := "message queue processor",
      packageDescription := "worker that reads from message queue and sends push notifications",
      publishTo := Packaging.rpmCalcPublishTo("message-queue"),
      libraryDependencies += "com.sillycat" %% "guts" % ourVersion,
      ideaIgnoreModule := true,
      rpmPost := Some(Packaging.rpmCalcPostChown("message-queue"))
    ): _*)

  lazy val profilesSparkSettings: Seq[Setting[_]] = scalariformSettings ++ Seq(
    name := "profiles-spark",
    mainClass := Some("com.sillycat.localpoint.profile.ProfileUpdater"),
    libraryDependencies ++= Seq(
      "org.apache.spark" %% "spark-core" % "0.9.0-incubating",
      "org.hectorclient" % "hector-core" % "1.1-3",
      "org.apache.cassandra" % "cassandra-all" % "1.2.14"
    )
  )

  lazy val profilesSpark = Project(id = "profiles-spark",
    base = file("profiles-spark")).
    dependsOn(main).
    settings(profilesSparkSettings: _*)

  lazy val profilesSparkRpm = Project(id = "profiles-spark-rpm",
    base = profilesSpark.base).
    settings(profilesSparkSettings ++ Packaging.settings ++ Packaging.app ++ Seq(
      target := file("profiles-spark/target-rpm"),
      packageSummary := "profiles spark job",
      packageDescription := "spark job to update profiles in cassandra",
      publishTo := Packaging.rpmCalcPublishTo("profiles-spark"),
      rpmPost := Some("""
hn=`/bin/hostname | /bin/cut -d'.' -f1`
if [ $hn = "cassops" ]; then
  echo "10 0 * * * spark /usr/bin/profiles-spark" > /etc/cron.d/profiles-spark
fi
"""),
      rpmPostun := Some("""
rm -f /etc/cron.d/profiles-spark
"""),
      libraryDependencies += "com.sillycat" %% "guts" % ourVersion,
      ideaIgnoreModule := true
    ): _*)

  lazy val localpointAuthCenterSettings: Seq[Setting[_]] =
    scalariformSettings ++ Revolver.settings ++
    Seq(
      name := "localpoint-auth-center",
      mainClass := Some("com.sillycat.localpoint.authcenter.Boot"),
      scalacOptions := Seq("-unchecked", "-deprecation", "-encoding", "utf8"),
      fork := true
    )

  lazy val localpointAuthCenter = Project(id = "localpoint-auth-center",
    base = file("localpoint-auth-center")).
    dependsOn(main % "test->test;compile->compile").
    dependsOn(util).
    dependsOn(auth).
    settings(localpointAuthCenterSettings: _*)

  lazy val localpointAuthCenterRpm = Project(id = "localpoint-auth-center-rpm",
    base = localpointAuthCenter.base).
    settings(localpointAuthCenterSettings ++ Packaging.settings ++ Packaging.server ++ Seq(
      target := file("localpoint-auth-center/target-rpm"),
      packageSummary := "localpoint auth center",
      packageDescription := "server process for localpoint authentication requests",
      publishTo := Packaging.rpmCalcPublishTo("localpoint-auth-center"),
      libraryDependencies += "com.sillycat" %% "localpoint-auth" % ourVersion,
      ideaIgnoreModule := true,
      rpmPost := Some(Packaging.rpmCalcPostChown("localpoint-auth-center"))
    ): _*)

}

BuildInfo.scala
import sbt._
import java.io.PrintWriter

object BuildInfo {
  /*
   * Generate buld information to be used by the /bldinfo.txt route
   *
   * logo taken from http://www.network-science.de/ascii/ using the "slant" font.
   */
  def write(dir: String, gitHash: String, branch: String, version: String): Unit = {
    val logo = """
          BUILD-DATE : %s
       GIT REV-PARSE : %s
          GIT BRANCH : %s
             VERSION : %s
 """
    try {
      val buildDate = new java.util.Date().toString()
      val writer = new PrintWriter(new File(dir + "/src/main/resources/bldinfo.txt" ))
      writer.write(logo.format(buildDate, gitHash, branch, version))
      writer.close
    } catch {
      case e: Exception => {
        println("Nonfatal build exception thrown creating bldinfo.txt : " + e.getMessage())
        e.printStackTrace()
      }
    }
  }
}

Packaging.scala
import sbt._
import Keys._

object Packaging {
  import com.typesafe.sbt.packager.Keys._
  import com.typesafe.sbt.SbtNativePackager._

  def app = packageArchetype.java_application

  def server = packageArchetype.java_server

  def settings: Seq[Setting[_]] =
    packagerSettings ++
    deploymentSettings ++
    Seq(
      version in Rpm ~= { _.replace("-", "_") },
      rpmVendor in Rpm := “sillycat",
      rpmLicense in Rpm := Some("proprietary"),
      publishArtifact in (Compile, packageBin) := false,
      publishArtifact in (Rpm, packageBin) := true
    )

  /** calculate rpm publish location; default may add stuff like scala version that we dont want */
  def rpmCalcPublishTo(name: String) = Some(
    Resolver.url(“sillycat Artifactory Rpm",
      new URL("http://repository.sillycat.com/artifactory/rpm-repo/6/os/x86_64/Packages/%s/".format(name))
    )(Patterns("%s-[revision].[ext]".format(name))))

  /** because we're lazy about logging and just use a single file, on a reinstall make sure the log file is writeable by the user. */
  def rpmCalcPostChown(name: String) = s"""
chown ${name}:${name} /var/log/${name};
chown -f ${name}:${name} /var/log/${name}/*.log;
"""
}

plugins.sbt
resolvers := Seq()

externalResolvers ++= Seq(
  Resolver.defaultLocal,
  “sillycat Artifactory" at "http://repository.sillycat.com/artifactory/remote-repos"
)

credentials += Credentials("Artifactory Realm", "repository.sillycat.com", "builder", “sillycat2011!")

addSbtPlugin("io.spray" % "sbt-revolver" % "0.7.2")

addSbtPlugin("com.github.mpeltonen" % "sbt-idea" % "1.6.0")

addSbtPlugin("com.typesafe.sbt" % "sbt-scalariform" % "1.3.0")

addSbtPlugin("com.eed3si9n" % "sbt-assembly" % "0.11.2")

addSbtPlugin("net.virtual-void" % "sbt-dependency-graph" % "0.7.4")

//addSbtPlugin("com.typesafe.sbt" % "sbt-native-packager" % "0.7.0-RC2")

addSbtPlugin("com.typesafe.sbt" % "sbt-native-packager" % "0.8.0-M2")

It can build rpm and debian. I will try these later.

References:
http://www.scala-sbt.org/sbt-native-packager/installation.html
http://www.scala-sbt.org/sbt-native-packager/GettingStartedApplications/MyFirstProject.html
https://github.com/sbt/sbt-native-packager

https://github.com/sbt/sbt-native-packager/issues/103

https://github.com/sbt/sbt/issues/1156
http://www.scala-sbt.org/0.12.2/docs/Detailed-Topics/Launcher.html
http://www.scala-sbt.org/0.13/docs/Proxy-Repositories.html