The idea is simple but fundamental.  When we started Saffron and began working with one of the big intelligent agencies, one true believer in what we were doing would provoke others by saying, “It’s all just counts.  What else is there?”  What did he mean?  When dealing with the analysis of massive data, so much of what is computed needs to be computed over counts.   More deeply, information and knowledge is based on the frequencies of what we see in the world.  Counts are fundamental to knowing what we know.

Let’s use even the most basic definition of a statistical mean or “average”, computed as the sum of counts divided by the number of counts.  An average count is useful as a measure of expected value (although other measures are also possible from counts).  Knowing nothing else, you can have an expectation about how tall a person is or often a business person travels.  Meeting someone who is 7 feet tall is rare.   A million mile traveler is unusual.   In general, these deviations from expectation are surprising, unusual, and potentially interesting.  This simple measure, based on count frequencies, is fundamental to many forms of analysis.

Now let’s look at a more advanced but still fundamental measure, such as entropy, a measure of disorder — or information — contained in a frequency distribution.    If we define information formally, as in Information Theory, then counts are critical to this definition and its application.  Think of the “analogies” query in the SaffronMemoryBase REST API.  How do we find entities that are similar to a given entity?  The entire method starts with getting a signature of the most informative attributes for the given entity.  What is most informative?  If the analysis were over a “bad guy” database and in fact all the persons are described as “gender: male”, then being male doesn’t tell us much.  In fact, it holds zero information as defined by an entropy measure.  But suppose the given entity was born in Timbuktu and only one other person was born in Timbuktu?  This is maximally informative when reasoning by similarity.

This example describes a method for entity resolution, determining who might actually be whom such as to resolve name variants and aliases.  Or you might want to find similar bad guys with a similar signature.  If you are a manufacturer, we might need to find similar parts, such as for part replacement strategies.  Also for predictive analytics, the “classifications” method in the REST API is a form of nearest neighbor classification, also reasoning by similarity.    As with “analogies”, the memory base supports a larger process of easily and quickly computing over connections and counts, but computing entropy is part of the process.   If the classifications are for buying and selling, sniffing whether transactions are suspicious or not, or deciding whether a product is desirable or not, the differentiating power of each attribute can be weighted by the distribution of frequencies.  If the counts of the attribute are distributed over the classifications according to chance, then it doesn’t seem to matter.  To the degree an attribute is more associated with one class, then it is informative to helping us make the classification.

The information in counts comes from data but is stored in a memory “above” the data.  I recently heard someone in the intelligence community say, “Everything we want to compute begins with collecting the counts.  We spend too much query time just to get these counts” (using map-reduce to compute of massive data at query time).   So why not start with a count-based representation? Why waste time searching and scanning for counts when they can be always at the ready for more advanced analysis?   The requirements for massive data analysis, whether for entity analysis or predictive analysis, are growing.   SaffronMemoryBase is a massive connection engine as well as a massive correlation engine, both of which are fundamental to analysis.  Connections and counts.  What else is there?

This week as I was working on some administrative functions for Sierra I found myself thinking there had to be a better way, and it turns out there is.  As I’ve discussed before when Grails launches it knows what mode/environment it’s running in.  The default environments are “development”, “test”, and “production”.  The Grails configuration scripts Config.groovy, DataSource.groovy, etc… give you handy “environments” blocks, but what about classes such as Bootstrap.groovy.  What should we do there?  The answer is the grails.util.Environment class.  This class knows what the current running environment is.  If we use this in Bootstrap.groovy we now have a way to only load our sample data if we are in “development” mode.  This is how it looks:

if (Environment.getCurrent() == Environment.DEVELOPMENT) {

// Insert sample data here

}

While working on the administration console for Sierra this week I ran into a case where I needed to paginate a set of results.  Having built web applications for years now I have some pretty tried and true methods for doing this.  However, in the past while using other frameworks these methods have involved me writing a decent chunk of code.  With Grails that chunk of code is no longer needed.  Or, a least I don’t have to write it.  Let me show you a simple example.  Let’s say we have a “show” method on a UsersController that either fetches a single user (if we pass in an id), or returns a list of all users.  It might look something like this:

def show = {

if (params.id && User.exists(params.id.toLong())) {

def user = User.get(params.id)

[user: user]

} else {

// default max num for the page to 10 if not specified

if (!params.max) {

params.max = 10

}

render view: "list", model: [userList: User.list(params)]

}

}

The last piece needed to get this working is a simple one line addition to your list.gsp file so that those URL parameters actually get passed in and the pagination controls get rendered.  That line looks like this:

<g:paginate controller="users" action="show" total="${User.count()}" />

Now for the “Bad”, or at least the not so pleasant.  I spent some time this week implementing a load balancing solution for the Sierra REST APIs.  While doing this I decided that I needed to externalize some of the Grails configuration.  When you create a Grails application your configuration normally lives in Config.groovy.  This is great to get started, but as your app matures you don’t want to have to recompile just to point at a new database, etc… Config.groovy gives you the ability to easily load external configuration files using a command like this:

grails.config.locations = ["classpath:${appName}-config.properties"]

If you do just this things work great, but I started thinking that I probably wanted things to work differently in my development environment than in my production environment in regards to loading external configuration.  Grails has thought about this desire and gives you the ability to define “environments” blocks within your Config.groovy.  I thought that using this along with loading external configuration would be the perfect combination.  Not so fast Jared.  It turns out that these two features don’t play nicely together.  I think this is a bug, or simple stupidity on my part.  I’m open to either option.  Using Grails 1.1.1 if you put a “grails.config.locations” definition inside of an “environments” block the external configuration will not be loaded.  I’ve logged the bug with the Grails folks here.

I wonder fun next week will hold?

I had been wanting to use and learn jQuery for quite some time.  We (Saffron Technology) had used Prototype in the past for a lot of our applications.  I noticed a while back that there seemed to be a lot of buzz surrounding jQuery and wanted to see what it was all about.  The hope was that jQuery would provide a quick an easy way to consume the JSON responses from the Sierra REST APIS.  It well exceeded my expectations in this department.

The development of the first few pages and API calls went very smoothly and things looked great while testing on my local development workstation.  However, as I tried to use the test harness to access remote Sierra instances I noticed I wasn’t getting the responses I would expect, in fact, I wasn’t getting responses at all.  It turns out this was due to web browsers enforcing tight security when it comes to calling other domains/sites from the domain on which you’re currently running.  This issue is well documented in this wikipedia article.

It turns out that the Javascript powers that be have a pretty handy hack that gets around this issue for well meaning developers like myself.  The solution has the somewhat unfortunate name JSONP (also explained here).  Using JSONP involves altering the way you make your REST calls on the client side and changing how JSON responses are formed on the server side.  On the client side you need to add a request parameter called “callback”.  In Javascript that’s as simple as the following line of code:

url += ‘&callback=?’;

On the server side you have to detect the existence of the “callback” parameter and make sure the JSON response is properly wrapped with the value of the “callback” parameter (in the above code using the value of “?” tells jQuery to generate a value).  The grails code in our controllers that handles this logic looks like this:

// if we have a JSONP callback we need to wrap the response with the callback

if (params.callback) {

render “${params.callback}(${json as JSON})”

} else {

render json as JSON

}

The rest of the “magic” is handled by jQuery.  It just works.  Calling remote Sierra instances now return responses just as my local development workstation.  If you’re interested in taking a look at the test harness code you can grab it using the instructions here.

About 6 months ago we began using Grails for our new and ever growing set of REST apis.  Overall our experience has been great.  Grails has really allowed us to accelerate development and focus on the things that are important to us.  Naturally, when we started talking about Sierra and knew that we’d need to build a user dashboard, admin console, etc… we turned to Grails.

Grails is implemented using Groovy, which is a scripting language that runs in the Java JVM.  Seeing as the rest of the Saffron stack uses Java this was another strong selling point for us, but it also means that in order to stand up Sierra we’d have to jump through all of the same hoops that come along with running a Java stack.

In the past all of our software and applications were installed and run in LAN environments.  This allowed us to ignore some of the issues that come along with deploying a Java application on the Internet.  Not so with Sierra, which is SaaS.  I hope to cover some of those challenges here which may, or may not be of some use to others.

1. Hosting – We needed some place to run the web front-end for Sierra.  While our new office does have sweet server closet we didn’t want to host it here.  We needed a place with more reliable power, bandwidth, etc…  We also needed an environment that would be relatively easy for us to manage.  We chose Slicehost.  With Slicehost we were able to get a server slice up and running in minutes.  It really is a great service.
2. Port 80 – Whenever you use a web browser to hit a web server your traffic typically travels over port 80 (unless your using https).  The port number is usually assumed and you don’t even see it in your URL.  Unfortunately Java web containers (Tomcat, Jetty, etc…) don’t easily run on port 80.  They typically run on port 8080.  The reason for this is one of security.  Most Unix OS’s don’t allow a non-root user to start a process that listens on a port below 1024.  You don’t want to run as root because that could be bad.  So, your left trying to figure out how to let people hit “http://saffronsierra.com” and not “http://saffronsierra.com:8080″.  There are number of potential solutions to this problem.  Using iptables seems to be the most commonly used/recommended approach and is the one we chose.
3. Too Many Redirects – Some Grails plugins give you to ability to state what URL endpoints needs to travel over http vs https.  This gives you the flexibility of not having the explicitly state http vs https in your markup.  They do this by issuing http redirects.  This works great in a development environment, but can cause problems in production.  As we moved over to our slice we kept getting the browser error  ”Too many redirects”.  Browsers have a threshold as to the number of times a particular request can be redirected.  I’m guessing, but I think the solution to the port 80 problem threw us over the redirect limit.  In order to get around this we had to explicitly define “http://” or “https://” in our markup.  While this doesn’t give us the flexibility of being able to separately define things on the fly in configuration, not markup, it will work for now.
4. Spam – Sierra needs to send out emails to user for certain events (registration, service start, etc…).  We setup Postfix to send mail on localhost.  However, I noticed that whenever I would send a test email to my Gmail account it would get flagged as spam.  After reading and tinkering a bit I figured out that by simply modifying some of the defaults in the “main.cf” configuration file for Postfix Gmail no longer thinks Sierra email is spam.  The steps I followed to edit main.cf are nicely listed in this Slicehost article.

I’m sure there were other challenges, but those are the “big” ones that come to mind.

Here are some example applications we are considering:

• Tracking exercise activity sites (http://www.mapmyrun.com/)

Online tracking of exercise activity is booming, but it’s not clear exactly what we could analyze here (it would depend a lot on what is available). But looking for patterns (grouping by age, diet, activity, etc..) and determining outcomes based on those observations is one possibility. Maybe an application that can suggest the best exercise routine based on real data of people like you (not just statistical averages).

• Build upon data set from http://www.data.gov

Data.gov has a ton of data available, but it arguably of little value unless we can see associations and patterns in the data. But finding a clean use case may be difficult.

• Swine Flu/H1N1 research using CDC data

This, or other, CDC data could be very powerful. Imagine being able to identify where an issue originated based on the pattern of previous disease profiles.

• Cell phone call log analysis

Although this data is private, it might be useful for either individual users, businesses or phone companies to be able to identify patterns in usage.

• Twitter/Social Media analysis

This is a hot topic right now and we have thought of several apps that might be a good fit. One example is an application that you feed all your social media contacts (twitter, facebook, myspace, delicious, etc…) and point out a few “important” contacts in the data set. These could be people that you know personally, or people that matter to you (or your org) because they have been influential in the past. Then Sierra could look for people like them based on any qualities (things they tweet, location, connections, etc…) and tell you who else might be influential.

• Website widget (what people are saying about…)

Similar to the previous example, but you would feed it blogs, twitter accounts, comment feeds and other conversational information and it could pick out the stuff important to your organization. Then it could be displayed as a widget on your site showing “what people are saying about us”. Would need some serious filters on this one though…

• Analyzing Gordon Bell’s MyLifeBits data (http://research.microsoft.com/en-us/projects/mylifebits/)

Gordon Bell has been collecting data for quite some time and has several search and indexing applications to find his stuff. But Sierra would be so much more effective and finding things because it works much like the human brain: looking for patterns of associations. Now we just need to figure out how to get all that data from Gordon….

What we are looking for is complex data sets (and unstructured data) where finding a “needle in a haystack” would be of value.