Preface

In my previous blog, Part 1, I went over what I did "wrong" during the development of Ping Collector. In Part 2, here's what I did *sorta* right.

What I did Right

  • I finished the project.

Ping Collector is my first major software creation in which everything planned from the beginning was finished. About 5% of the code was provided with the help of a friend during a "Codeathon" event we did, but the majority was setup by myself. Here is a list of things I had very little to no experience with programming from the ground up before I started Ping Collector:
      • Google App Engine
      • Memcache
      • Non-relational database use (Google Datastore)
      • Database design around a non-relational database.
      • Cross-environment Symmetrical Encryption (Implementation of algorithms, not the algorithm itself)
      • Hashing
      • Request Limiting
      • Client authentication (Is the data I am receiving on my server by a permitted client?)
      • User Authentication (What user is the data I am receiving coming from? Custom implementation.)
      • Windows Services
      • Various platform support for the Client and its Installer (.NET 2.0+, Win XP - Win 8)
      • InstallShield Express with various custom actions to accomedate Service installation and Windows 8.
      • Encrypted Storage of credentials on Windows environments.
      • Retrieving/Storing values in registry (Windows)
      • Log4Net
Even if the product doesn't generate many donations, or doesn't succeed in changing the quality of internet connections around the world, I still learned a huge amount.

The "Request Gateway" method which was once implemented as Before Advice for all methods before it was decided that Spring Aspect Oriented Programming was too complex for the Google App Engine environment.

public void requestGateWay(HttpServletRequest req,
 HttpServletResponse resp, int requestCooldown, String functionName)
 throws TooManyRequestsException, UnsupportedISPException {
 System.out.println("REQUEST GATEWAY");
 String isp = ispService.getOrg(req.getRemoteAddr());
 if (isp != null && testingBypass == false) {
   if (!isp.toLowerCase().contains("windstream") && testingBypass == false) {
     throw new UnsupportedISPException("Only Windstream internet users are supported currently.");
   }
 }
 // may god have mercy on your latency.
 Long ipExpiration = (Long) cache.get(req.getRemoteAddr() + functionName);
 if (ipExpiration == null) {
 System.out.println(req.getRemoteAddr() + "'s requests limited for "
 + requestCooldown + " minutes after method call");
 Expiration expir = Expiration.byDeltaSeconds(requestCooldown * 60);
 cache.put(req.getRemoteAddr() + functionName,
 expir.getMillisecondsValue(), expir);
} else {
   Date expirDate = new Date(ipExpiration);
   PrettyTime pt = new PrettyTime();
   System.out.println("IP Address " + req.getRemoteAddr()
   + " requested too frequently. Next request available in "
   + pt.format(expirDate));
   resp.setStatus(429);
   throw new TooManyRequestsException(new RequestLimitInfo(false,
   "You are requesting the feature, "" + functionName
   + "" too frequently. Please try again in "
   + pt.format(expirDate) + "."));
  }
 }
  • Google's (NoSQL-like, non-relational, BigTable-based) Datastore was a great choice.

Taking a non-relational database approach to this was somewhat daunting at first, mostly because all of my database experience had mostly been through the normal relational databases (Oracle, SQL Server, SQLite). I am pretty comfortable with normalization and relational database design so making one for a system of this size would of been easy. However, there were a few problems with that. First off, Google SQL Server costs money right off the bat, unlike the datastore. Second, I wanted to be able to scale the database easily if Ping Collector was to suddenly gain a lot of traction. I was able to make two Entities, a user and a ping report. Ping Report entities were added with the user entity as its ancestor. This allowed a sort of organization/pseudo-normalization that I was worried about losing from a relational database. I kept it simple, in retrospect. The datastore is FAST. Very fast. Whenever I generated pie charts for all pings collected over  a month (note, the query returned tens of thousands of entities), it was done in under 3-4 seconds. This is in addition to generating the graph.

A humorous, early test snippet generating a piechart based on the data collected.

 public void testPieChart(HttpServletResponse resp,
 @RequestParam(required = true) String user, @RequestParam(required = true) String sauce,
 @RequestParam(required = false) boolean peakTime) throws IOException,
 NoPingsFoundException {
if (!secretSauce.equals(sauce)) {
 resp.sendError(403, "You didn't provide the secret sauce.");
 } else {
 Filter filter = null;
 PieChart chart = PingPieGraph.generatePingDistributionChart(PingDAO
 .getAllPingsBetweenDates(user, filter, peakTime));
 String title = "Report for " + user;
 if(peakTime == true){
 title += " during Peak Time (4pm-12am EST)";
 }
 title += " Date: " + new DateTime().toString("dd/MM/YYYY"); ;
 chart.setTitle(title,Color.BLACK,10);
 System.out.println(chart.toURLString());
 resp.sendRedirect(chart.toURLString());
 }
 }
 
  • Taking it relatively slow.

I work a normal software development job, doing roughly 40 hours a week. I'm passionate about programming and development, but I can get burnt out on it. There are less than 15 .java source files for the server side and about 5 times less than that for the C#/.NET client (although I am guilty of not being entirely too object oriented on that end. There were times where development was done in bursts (a weekend, or a night when sleep didn't come easy.) Sometimes I would just push out a service call or a function in the evening. A benefit to moving slow was that I was able to incorporate the knowledge I had learned developing other projects at work (my knowledge of the Spring framework came mostly from this) Balancing the side project, work, and my social life is harder than the actual development at times! Which is why I am proud that I didn't stress too much, although there was always a tinge of defeat when telling a friend about what I was working on, even though I didn't have tons to show for it at that point in time.

Obfuscating model classes which are serialized WILL make you have a bad day

[System.Reflection.ObfuscationAttribute(Feature = "properties renaming")]
class PingCollectorResponse
 {
 [JsonProperty(PropertyName = "message")]
 public String message { get; set; }
 [JsonProperty(PropertyName = "code")]
 public int code { get; set; }
 [JsonProperty(PropertyName = "error")]
 public Boolean error { get; set; }
}

Conclusion

Some other non-technical thoughts:
  • Start small, work up. The first iteration and alpha version of Ping Collector simply sent emails with the latency data, and it was made during my short vacation after falling six feet onto my back in a unfortunate crowd surfing accident (I love my metal, a little too much). Gradually, features were added and goals were set, and Ping Collector is where it is today.
  • If you lack the skill or confidence to build something, put monetary goals aside.
  • StackOverflow is a godsend.
  • Program things for your mother. She helped pay for your education.
--Z