Adventures of hackerdad()

Blogging about Life, the Universe, and my baby girl.

500 Dialects

The world is full of amazing insights, and you never quite know from where or how the next one will appear. Case in point, I spend a lot of time at opodz, this really amazing co-working space in a quiet corner of downtown Los Angeles. When I come after hours, there’s one of two security guards to give me a friendly hello in the lobby. This particular morning, he’s listening to some kind church music that sounds Byzantine.

“Are you Greek?” I ask.

“No, I’m from Congo”.

He’s been in the US a while, big guy, the kind you want to have on your side in case of a bar fight, but with a big smile to boot. I’ve said hello to him lots of times over the past two months, but this is the first time strike up a conversation instead of rushing to my computer.

He goes on to relate, among many other things, that in his country, deocracy is really hard. There are 500 dialects. He explains it like this, if the two tribes happen to be nearby each other, they’ll speak different dialects but still be able to understand each other, more or less. If the US were like that, then people from Los Angeles and San Francisco would speak a different language, but could kind of understand each other, but people from Arizona and California would be lost.

In a way, we all speak a different language because each of us has arrived to the here and now with a different set of experiences.

I’m going up to San Francisco on Tuesday to attend SF-Scala. I’m looking forward to an awesome presentation about Scala collections and saying hello to great people I met at the Scala Symposium last August. The talk will be in English, of course, but still, I can’t take it for granted that we all “speak the same language.” No worries, good communication starts with understanding that good communication isn’t easy.

The next time you walk by a guy in a hurry to get to your computer, take a moment to say something more than just a friendly grunt, you just might learn something!

Three Steps

Last night my daughter Athena reached out her arms for me. I helped her to stand, and then I reached out my arms, inviting her to walk into my arms. She broke her previous record by taking two steps before she crashed landed into the foam mat, and it made me ponder the meaning of 1.0 as a software milestone.

Time plays tricks on the mind. It was just a few months ago that Athena began to crawl, but it wasn’t an immediate event, but rather a gradual process. Back in June, a hair short of seven months, she would go in circles on the mat because she hadn’t yet figured out how to coordinate her arms and legs. Before and after there were lots of milestones on the path to crawling. Somehow, I was expecting walking to be different. That she would just wake up one day and start walking. But it hasn’t been like that.

Last night, after she took three steps towards me, and we all applauded her, I stood her up again, inviting her to repeat her new record. Instead she stood there with those beautiful blue eyes and radiant smile, applauding her achievement and inviting us to do the same. But instead of clapping, I just held out my hands, inviting her to walk the two feet that separated us.

And this caused me to think about what exact my app needs to do to move from alpha to beta status. Ask twelve develoers what 1.0 means and you’ll probably get twelve different answers. Ask Facebook, TypeSafe, & NodeJS what /good enough/ means, and again you’ll probably get very different answers.

So, as I’m working on the MagicNotebook, I ask myself, “When will it be good enough?”

When is good just not good enough? When does that change?

There’s a sticky note pasted to my monitor. It says, “When will it be good enough?”. I have an old- fashioned windup clock that goes “tick-tick-tick-tick”, reminding me that I need to go faster. And the sticky note reminds me that sometimes /good/ is good enough and sometimes it isn’t. And now I have a memory burned into my brain of a little girl taking her first three steps, reminding me that the there will not be an event that clearly delineates the transition from version 0 to version 1, when my app transitions to a state of “almost but not quite good enough” to “good enough”.

User-Centered Software Testing With Scala

Today I want to talk a bit about how I reoriented my tests from a file-center to a user-centered perspective. I’ve been spending a lot of time thinking about testing lately.

My code base is still relatively small, just under 2k, but the past week or so I haven’t added any new features while I’ve been changing my focus from we’re-a-startup-and-there’s-no-time-to-test to a more sane approach. After all, building an application from scratch is both a sprint and a marathon.

I’ve settled on two testing styles that I like within the ScalaTest framework: FunSpec and FeatureSpec. FunSpec is for my unit testing and FeatureSpec is for my integration testing. Here’s what a simple Funspec looks like:

1
2
3
4
5
6
7
8
9
10
11
@DoNotDiscover
class DbUserTest extends FunSpec with Matchers with EitherValues
    with DbUserJson {

  describe("A DbUser") {
    describe("Should be instantiated from valid json") {
      validUserJson.validate[DbUser].left.get.email should be(
          "bart@simpsons.com")
    }
  }
}

The @DoNotDiscover annotation is because all of my tests are organized into suites, which helps keep the test output organized and clean, important since the tests are an important source of documentation and specification for my MagicNotebook app.

The integration testing … well, this morning I was looking at how I was organizing my tests, and they really were file-centric rather than user-centric. Fundamentally, the tests were asking whether isolated parts of the application (database, cache, & models, for example) were working together as expected. But the tests were organized around different files and mimicked the application file/folder structure. Here’s what it looked like when I woke up this morning:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
 % tree test
test
├── integration
   ├── IntegrationSuites.scala
   └── models
       ├── db
          └── DbUserSpec.scala
       └── google
           └── GFileSpec.scala
├── resources
├── unit
   ├── models
      ├── db
         └── DbUserTest.scala
      └── google
          └── GFolderTest.scala
   └── UnitSuites.scala
└── utils
    ├── db
       ├── DbSetup.scala
       └── DbUserJson.scala
    ├── google
    └── MagicNotebookData.scala

So I decided to reorganize it around actual specifications that a user would understand. Here’s what it looks like now:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
 % tree test
test
├── feature
   ├── admin
      └── AccountFeatures.scala
   ├── explorer
      └── ViewDriveFeatures.scala
   └── FeatureSuites.scala
├── resources
├── unit
   ├── models
      ├── db
         └── DbUserTest.scala
      └── google
          └── GFolderTest.scala
   └── UnitSuites.scala
└── utils
    ├── db
       ├── DbSetup.scala
       └── DbUserJson.scala
    ├── google
    └── MagicNotebookData.scala

As you can see, the Integration tests are now organized around features, such as account management and views Google Drive file features. So, I’m on the cusp of breaking the piddling 2k loc metric, but I’m not in a rush to grow the code base because I want it to grow into a beautiful greenfield that will stay green and healthy, because I’m in it for the long haul, and I hope you are too!

Using a Database View With Scala Slick

Scala Slick is pretty darn cool. I’ve been using it for my app MagicNotebook.io, which is going to change the way teachers guide students through the writing process.

As I explored in my last post, it is important that design decisions should drive technology choices and not the other way around. To do that, you need to be prepared to think outside the box regarding what exactly those technology choices are, and that’s the focus of today’s post.

To review, here’s the current representation of a Google Drive folder in my app:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
case class GFolder(id: String,
                   eTag: String,
                   url: String,
                   iconUrl: String,
                   title: String,
                   owner: String,
                   parents: Set[String],
                   children: Set[String],
                   scions: Set[String],
                   created: LocalDateTime,
                   modified: LocalDateTime) extends GFile {
  def folders(implicit gFolders: Map[String, GFolder]): List[GFolder] = {...}

  def docs(implicit gDocs: Map[String, GDoc]): List[GDoc] = {...}

  def export(implicit gFolderMap: Map[String, GFolder],
                      gDocMap: Map[String, GDoc]): List[TreeTableRow] = {...}

And here’s the object that is persisted to cache so the user can work with his/her Google Drive files:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
case class MagicNotebook(tree: GFolder,
                         implicit val folders: Map[String, GFolder],
                         implicit val docs: Map[String, GDoc])

object MagicNotebook {

  def create(userId: String)(implicit s: Session) = {
    DbHomes.findById(userId).flatMap { fileId =>
      DbGFolders.findById(fileId).map { tree =>
        val folders = DbGFolders.findAllFor(fileId)
        val docs = DbGDocs.findAllFor(fileId)
        MagicNotebook(tree, folders, docs)
      }
    }
  }
  val empty = MagicNotebook(GFolder.empty, Map(), Map())
}

To create a MagicNotebook object, we start with the user’s id to get the fileId of the Google Drive folder that is the root of all the MagicNotebook files in the user’s Google Drive. With that, we build a GFolder object of that root folder along with a map of all the subdirectory GFolders and GDocs. But in the database, the representation of a GFolder is split among four tables. In addition, there is scions, representing the children, grand-children, great-grand-children, etc of a folder.

My first attempt to do this transformation was purely with Scala and Scala Slick code, and seemed a bit clunky. So it was time to think outside the box. The documentation for Scala Slick explains clearly how to map a Scala object to a database table, but nowhere in the documentation does it mention database views. I tested it with Play 2.2.0 and Scala Slick 1.0.1, and it turns out you can bind a Scala Slick Table to a database view! First, we begin by defining a view that uses a recursive query to find all scions (children, grand-children, great-grand-children, etc.) of a folder:

1
2
3
4
5
6
7
8
9
CREATE VIEW scion_view AS
  WITH RECURSIVE scions(id, scion) AS (
    SELECT c.id, c.child
    FROM children AS c
    UNION ALL
    SELECT s.id, c.child
    FROM children AS c, scions AS s
    WHERE c.id = s.scion)
  SELECT * FROM scions ORDER BY id, scion;

Next, we need to string-agg function that will flatten multiple rows of a fileIds from a one-to-many relationship to a one-to-one relationship as a single comma-delimited string row:

1
2
3
SELECT DISTINCT
  id, string_agg(child, ',' ORDER BY child) AS child_str
FROM children GROUP BY id;

With these two building blocks in place, we can create the view that maps directly to the Scala GFolder object:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
CREATE VIEW gfolder_view AS
  SELECT
    f.id, f.e_tag, f.url, f.icon_url, f.title, m.name, f.file_owner,
    p.parent_str, c.child_str, s.scion_str, f.created, f.modified
  FROM
    gfiles AS f
      JOIN mimes AS m ON (f.mime_type = m.name)
      LEFT JOIN (SELECT DISTINCT id, string_agg(parent, ',' ORDER BY parent) AS parent_str
                 FROM parents GROUP BY id) AS p ON (f.id = p.id)
      LEFT JOIN (SELECT DISTINCT id, string_agg(child, ',' ORDER BY child) AS child_str
                 FROM children GROUP BY id) AS c ON (f.id = c.id)
      LEFT JOIN (SELECT DISTINCT id, string_agg(scion, ',' ORDER BY scion) AS scion_str
                 FROM scion_view GROUP BY id) AS s ON (f.id = s.id)
  WHERE
    m.category = 'folder';

The only transformation required to map this view into a GFolder object is to split the comma-delimited String of parent, child, & scion fileIds, which is rather trivial to do with Scala. So here is the Scala object:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
case class DbGFolder(id: String,
                     eTag: String,
                     url: String,
                     iconUrl: String,
                     title: String,
                     owner: String,
                     parents: Option[String],
                     children: Option[String],
                     scions: Option[String],
                     created: LocalDateTime,
                     modified: LocalDateTime)

object DbGFolders extends Table[DbGFolder]("gfolder_view") {
  def id = column[String]("id")
  def eTag = column[String]("e_tag")
  def url = column[String]("url")
  def iconUrl = column[String]("icon_url")
  def title = column[String]("title")
  def owner = column[String]("file_owner")
  def parents = column[String]("parent_str")
  def children = column[String]("child_str")
  def scions = column[String]("scion_str")
  def created = column[LocalDateTime]("created")
  def modified = column[LocalDateTime]("modified")
  def * = id ~ eTag ~ url ~ iconUrl ~ title ~ owner ~ parents.? ~
          children.? ~ scions.? ~ created ~ modified
          <> (DbGFolder, DbGFolder.unapply _)

  def findAllFor(root: String)(implicit s: Session): Map[String, GFolder] = {
    val query = for {
      (_, gFolder) <- DbScions.innerJoin(DbGFolders)
                              .on(_.scion === _.id)
                              .where(_._1.id === root)
    } yield gFolder

    query.list().map {v =>
      GFolder(v.id,
              v.eTag,
              v.url,
              v.iconUrl,
              v.title,
              v.owner,
              v.parents.map { parentStr =>
                parentStr.split(",").toSet }.getOrElse(Set()),
              v.children.map{ childStr =>
                childStr.split(",").toSet }.getOrElse(Set()),
              v.scions.map { scionStr =>
                scionStr.split(",").toSet }.getOrElse(Set()),
              v.created,
              v.modified)
    }.groupBy(_.id).mapValues(_.head)
  }

  def findById(fileId: String)(implicit s: Session): Option[GFolder] = {
    Query(DbGFolders).filter(_.id === fileId).list().headOption.map {v =>
      GFolder(v.id,
              v.eTag,
              v.url,
              v.iconUrl,
              v.title,
              v.owner,
              v.parents.map { parentStr =>
                parentStr.split(",").toSet }.getOrElse(Set()),
              v.children.map{ childStr =>
                childStr.split(",").toSet }.getOrElse(Set()),
              v.scions.map { scionStr =>
                scionStr.split(",").toSet }.getOrElse(Set()),
              v.created,
              v.modified)
    }
  }
}

Thus, moving the application logic to denormalize a GFolder back to postgres allowed me to significantly improve the Scala code readability.

Formula Flavored Coffee

I just poured the remains of eight ounces of baby formula into my coffee cup. I made eight ounces. Athena, my beautiful daughter who is 0.8661202186 years old, just finished off four ounces and was sated, leaving the rest for me.

There was a time when I was a coffee snob. Back in those days, I would never dream of putting anything into my coffee except the pure unadulterated black juice itself. Back in those days, I would be writing down my thoughts in a chic cafe with my Pelikan fountain pen and leather notepad filled with Rhodia paper. These days, I pen my thoughts using gvim on a laptop that runs Arch Linux, and instead of a chic cafe I’m sitting on a big green foam playmat with my baby girl. Times change.

I still like fine coffee, but I find that a bit of milk reduces the acid bite. I could go to the refrigerator and pour some milk into my coffee cup, but I feel the weight of that half-empty bottle. There’s a bit of that milk in Athena right now, growing her bigger, so in some since there’s a bit of Athena in that bottle, and I just can’t stand the thought of pouring that down the drain.

Formula-flavored coffee tastes great, just have to get used to the little lumps of formula that didn’t quite completely dissolve. Time to get back to coding.