Java Concurrency, Volatile and Synchronized Blocks

Why Synchronized Blocks are not Enough

So I was reading “Effective Java” by Josh Bloch and got to the chapter on Concurrency (chapter 10).  I thought I had a basic understanding of synchronization blocks and volatile, but I was a little wrong.

  • Volatile – Ensures reading a variable is the most up-to-date at the time of the read
  • synchronized – Ensures a “lock” is put on whatever is being modified within your block.  Mutual exclusion of threads accessing anything being modified within the block.  Avoids other threads seeing things in an invalid state.

Now my definitions are not 100% clear, but they work for me and any comments would be appreciated.

It might be a good time to read this guy’s article.

Based on the above definitions…

Synchronizing modification of a variable does not guarantee that those changes are communicated to another thread, but why?

It is because of optimizations by the JVM.  In “Effective Java” this example is given:

// Cooperative thread termination with a volatile field
public class StopThread {
  private static boolean stopRequested;
  public static void main(String[] args) throws InterruptedException {
    Thread backgroundThread = new Thread(new Runnable(){
      public void run() {
      int i = 0;
      while (!stopRequested)
    stopRequested = true;

Without using the keyword volatile for the stopRequested variable you let the JVM optimize the while loop to potentially something like this:

int i = 0;

This would result in the thread never terminating.


The volatile not only ensures that the different threads update their memory between one another, but also protects you from unseen optimizations.

Now I would like to test this myself, because it seems puzzling to me that this optimization would occur on a non-final variable, but according to Josh Bloch
this is what the HotSpot JVM will do. It is called hoisting optimization.

Yii Rights Extension – Error 403. There must be at least one superuser!

The Problem

I am new to yii and wanted to use a php framework for a new personal site.  After reading a bunch and my own experience with CakePHP and CodeIgnitor I decided to try something new and Yii seemed like a good choice.

It seems pretty well organized and has tons of extension already.  Notably I was interested in their User Management plugins and social integration.  This led me to yii extensions: Rights, yii-user.

I installed yii-user without too much trouble, but still was not perfect and documentation for a newb was crappy to say the least.  But fumbled my way thought it with an hour or 2.  Still better than cakePHP ACL (but not as powerful).

So I needed an ACL (Access Control Lists), which led me to Rights extension.  Which seemed easy enough to install, but quickly got to:

Error 403.  There must be at least one superuser!

 Solving Error 403.  There must be at least one superuser!

What is not told by this error is this is actually a sql error due to constraint violations.  Unfortunately all these errors are nicely captured and completely hidden.  EVEN IN DEBUG MODE!!!!  Why the hello would anyone do that!

So after 1.5 hours of trying to find where the problem was I go to RInstaller->run() method.  Which i stepped thought the code at this point and found the error.

The easiest solution was to comment out the foreign key contrains in the data/schema.sql file.

NOTE: I also Commented this following out in RAuthorizer.php, but couldn’t tell you if this is necessary.

if( $superusers===array() )
throw new CHttpException(403, Rights::t(‘core’, ‘There must be at least one superuser!’));

Simple OAI Repository in PHP

Getting Started

If you don’t know what an OAI repository is, then go here.

If you just want the code, check it out here at github.

So a couple of months ago I needed to test against an OAI repository.  Basically we have an OAI harvester, but were having issues with a customer’s repository.  The main problem is the repository in question would fail deep into the harvest.  So I wanted to pinpoint where this error happens during the harvest, but the logs around the error were no help.  The errors were trapped and we use a 3rd party harvester jar, so I couldn’t easily get better error messaging.  So what was the error?

  • Is it bad XML?
  • Bad UTF-8?
  • Some other error?

Now as a java shop someone had written a clojure oai simulator, but if you have ever written some clojure you may feel my pain.  Im not against functional languages, but really the dependency management is a huge pain and java compounds these problems.  So it uses lein for this, but when this stopped working my clojure world came to an end.  Hence I decided to write a quick php example.

PHP Implementation

So we need to implement the basic functionality:

  • Identify
  • ListRecords
  • ListSets
  • Error Handling

The basic requests work on the URL var “verb”.  For example http://localhost/my-oai/oai.php?verb=Identify

Getting URL vars in PHP

Anytime you use variables submitted by users, you should be careful.  So lets make a utility class that will handle some of the basic functionality we will be using.

class Utility{
	 * Wrapper for getting a URL variable.  Gurantees a string is returned.  Either
	 * the variable requested or '' if the URL variable doesnt exist.
	public static function myGet($index){
		return (isset($_GET[$index])) ? $_GET[$index] : '';

Now we made things static so it is a little easier to use. And as we require more helper functions in the future we can easily place them here. Really all we did was protect ourselves from accessing an index that doesn’t exist.

Now that we have a safe GET command, how should we set things up?  Since we have several actions that will do certain things, this sounds like we should use switch() statement.  Here is the start of my oai.php file.

//we are returning xml, so we need to set the header
//for "Content-Type"
header("Content-type: text/xml; charset=utf-8");

require_once 'Utility.php';

//handle the verb
$verb = Utility::myGet('verb');
$set = Utility::myGet('set');

 * Handle the verbs
	case 'resumptiontoken':

	case 'ListRecords':

	case 'Identify':

	case 'ListMetadataFormats':

	case 'ListSets':


So based on these actions we need to display some xml. We could do this via building an XML document using PHP, but really this is overkill. The approach we will take is storing the XML as strings and doing some basic variable replacement. For this we can use PHP sprintf.

How to Structure our Verb / Actions

So as we discussed everything will use this sprintf command, but we want to extend the functionality a little.  So we will create a base class which has this shared functionality so we can extend it and re-use the functionality.  And to be a little over the top I included an interface.

interface Verb{
	public function doVerb();

class Base{
	public function getRequest(){
		$prot = ($_SERVER['SERVER_PORT'] === '443') ? 'https://' : 'http://';
		return $prot . $_SERVER["SERVER_NAME"] . htmlentities($_SERVER['REQUEST_URI']);

class VerbBase extends Base{

	public function sprintf($args){

		$args = func_get_args();
		$file = array_shift($args);

		$str = file_get_contents($file);

		if(empty($str) || $str < 0)
			throw new Exception('Failed to read file ('.$file.').');

		array_unshift($args, $str);

		return call_user_func_array('sprintf', $args);

Implementing Our Verbs

Lets create a ListRecords.php file which implements our Verb interface and extends our VerbBase class.


 * List Sets class
 require_once 'Verb.php';

 class ListSets 
		extends VerbBase
		implements Verb{

	public function ListSets(){

	public function doVerb(){
		$sets = "";
		foreach(scandir('sets') as $file)
			if($file[0] !== '.' && $file !== 'defaults')
				$sets .= "\n\t<set>\n\t\t<setSpec>$file</setSpec>\n\t\t<setName>Set $file</setName>\n\t</set>";

		return $this->sprintf('sets/defaults/listsets/listsets.xml', date(DATE_ATOM), $sets);


As you can see the doVerb() implementation scans a directory listing called ‘sets’ and creates a string.  We then use our sprintf implementation to create the xml.

Our Remaining Verbs

Since the remainder of the verbs do basically the same thing we can use a single class to perform the rest.  I named the file ListRecords.php but this is misleading.


 * List records class
 require_once 'Verb.php';

 class ListRecords 
		extends VerbBase
		implements Verb{

	private $rawSet;
	private $set;
	private $setDir = 'sets/';

	public function ListRecords($set = null){
		$this->rawSet = $set;
		$this->set = $this->setDir . ((empty($set)) ? 'badset' : $set);

	public function doVerb($resumption = null){
		//set doesnt exist
			return $this->doBadSet();

		$loadFiles = array();
		$files = scandir($this->set);

		foreach($files as $file){
			if($file[0] !== '.' && substr($file,-3) === 'xml'){
				array_push($loadFiles, $file);

		//no xml to load
		if(count($loadFiles) <= 0)
			return $this->doBadSet();

		$key = 0;
			$parts = explode(':',$resumption);
			$key = $parts[1];

		$content = file_get_contents($this->set . '/' . $loadFiles[$key]);

		//for convienence we remove any resumption tokens
		$content = preg_replace('/\<resumptionToken\>.*\<\/resumptionToken\>/i','',$content);
		$content = preg_replace('/\<resumptionToken[\s+]?\/\>/i','',$content);

		if(count($loadFiles) > 1){		

			//remove all end stuff
			$content = str_replace('</ListRecords>','',$content);
			$content = str_replace('</OAI-PMH>','',$content);

			$content .= ($key < count($loadFiles)) ? "<resumptionToken>{$this->rawSet}:$key</resumptionToken>" : '<resumptionToken />';

			$content .= "</ListRecords></OAI-PMH>";


		return $content;

	private function doBadSet(){
		$url = $this->getRequest();
		return $this->sprintf('sets/defaults/badset/badset.xml', date(DATE_ATOM),$this->set,$url);



The only part that is unique to ListRecords verb is the resumption code logic.  It may have made send to try and split this out a little better, but I didn’t see a clear way to do this that didn’t result in a lot of code duplication.

The End Result

So if you remember the oai.php we can easily fill in our code now to get the following.

header("Content-type: text/xml; charset=utf-8");

require_once 'Utility.php';

//handle the verb
$verb = Utility::myGet('verb');
	$verb = 'resumptiontoken';

$set = Utility::myGet('set');

 * Handle the verbs
	case 'resumptiontoken':
		$parts = explode(':', Utility::myGet('resumptionToken'));
		$set = $parts[0];

	case 'ListRecords':
		require_once 'ListRecords.php';

		$verb = new ListRecords($set);
		echo $verb->doVerb(Utility::myGet('resumptionToken'));

	case 'Identify':
		require_once 'ListRecords.php';

		$verb = new ListRecords('defaults/identify');
		echo $verb->doVerb();

	case 'ListMetadataFormats':
		require_once 'ListRecords.php';

		$verb = new ListRecords('defaults/listmetadataformats');
		echo $verb->doVerb();

	case 'ListSets':
		require_once 'ListSets.php';

		$verb = new ListSets();
		echo $verb->doVerb();

		require_once 'Verb.php';

		$verb = new VerbBase();
		$url = $verb->getRequest();
		echo $verb->sprintf('sets/defaults/badrequest/badrequest.xml', date(DATE_ATOM), $url);



So you can see we have a very simple clear oai.php file which handles our basic required actions.  Enjoy and ask any questions if you have any.

So Why did I Bother?

There are already other Open Source OAI implementations out there.  And some of them see good.  But what I found was they were overly complicated and were intended to be a “real” repository that people wanted to hook up to a database and have all kinds of admin functionality.  So I decided I needed a real bare bones repository that met my needs.

Hopefully this saves someone else time out there.  You can download all the code and some more utility functionality than what I discussed in the article at github.

Creating an Image Preloader

Image Preloader using JQuery

I recently was creating an image viewer for a customer.  It had the usual qualities of an image viewer where you need preload images to improve response times for the user.


  • Load X images behind/infront of current image
  • Handle image load errors
  • Callback for load complete
  • Tie attributes to an image

In addition, we will use JQuery to facilitate this.

Lets Get Started

So if we abstract this a little, we want to have an Image Object with the following attributes…


  • url – The url of the image to load
  • callback – A callback function for when the image load is complete
  • scope – With javascript, closure, we want to call the callback within a scope other than window.  This will hold this information.
  • loaded – Boolean so we can track the images state.
  • loadFailed – So with an image, we may have failed to load or timeout.  So we need to track that.
  • timeout - How long to wait on things to load.
		 * @classDescription Image object which holds information about the image which is preloading.
		 * This object can be prototyped with extendImage.
		 * @param {String} url The url of image
		 * @param {Function} callback The users callback function
		 * @param {Object} scope The scope to call callback function in
		 * @param {Boolean} loaded Whether the image has been loaded
		 * @param {Jquery Object} $img The reference to the jquery image object in preloading node.
		 * @param {Object} objRef This object's reference
		 * @see extendImage()
		_Image : function(url, callback, scope, loaded, $img, objRef){
			this.url = url;
			this.callback = callback;
			this.scope = scope;
			this.loaded = loaded;
			this.$img = $img;
			this.func = objRef._imageLoadComplete;
			this.loadFailed = true;
			this.timeout = 5000;//5 seconds
			this.start = new Date().getTime();
			this.finish = 0;

Now We Need to Store the Image Objects

So if we are going to have an image object to represent each image, then we need a way to store these and be able to get them.  Unfortunately javascript doesn’t by default have a good way for use to store and retrieve data.  The default array, is somewhat limited.  Fortunately we can create indexes of the array with strings.  Basically giving us a simple hash table.

However, I kinda wanted to spice this up a little.  I could have used the URL of the image as the index, but what fun is that.  So lets create a hash of the URL and use that as an index.

hashCode = function(str){
	str = str + "";
	var hash = 0;
	if (str.length == 0) return hash;
	var len = str.length;
	for (var i = 0; i < len; i=i+1) {
		char = str.charCodeAt(i);

		hash = ((hash<<5)-hash)+char;
		hash = hash & hash; // Convert to 32bit integer

	return hash >>> 1; //I want positive numbers

Now you will probably argue that looping over each character  is bad, why not just use the URL.  Well, your probably right, but this is more fun.

hash = ((hash<<5)-hash)+char;

This is better explained here.

Initiate the Pre-loading of an Image

We will need a simple function to load images. Everything that refers to “this” refers to the object that wraps all these functions.

preload = function(url, callback, scope, timeout){
	var hash = this._hashCode(url);
	if(this.isLoaded(hash)) return;

	if(typeof scope === "undefined") scope = window;

	//create image object
	var $img = $(document.createElement('img'))
	this._images[hash] = new this._Image(url, callback, scope, false, $img, this);

	//bind the onload...pass vars using closure ;) 
	this._images[hash].$img.load(function(obj, hash){
		//only executes on successful load
		return function(){obj._images[hash], obj._images[hash]);};
	}(this, hash));

	//we always want to trigger our functions, but jquery only
	//callsback on success, so have to keep polling at the image timeout
	if(typeof timeout !== "undefined") this._images[hash].timeout = timeout;
	this.waitOnImage(url, this._imageLoadComplete, this, this._images[hash].timeout, 2);

	this._$loadNode.append(this._images[hash].$img); //append loaded image to the loading DOM element

	return this._images[hash];

There are actually a couple interesting segments of this code.

scope - This will allow us to scope the context to the callback.  This is useful to avoid namespace collisions and ensure you are triggering the correct event to specified image loading.

this._images[hash].$img.load(function(obj, hash){
		//only executes on successful load
		return function(){obj._images[hash], obj._images[hash]);};
	}(this, hash));

This uses the JQuery default load callback.  The problem with the load event is it is unreliable.  It won’t fire if the image fails to load and won’t fire reliably cross-browser and in other circumstances.  These are describer in the jquery link.

Handle the load event not working

To do this we basically need some sort of polling to check if the image is loaded.  That is the waitOnImage() function.

WaitOnImage Function

		 * Bind specific callback to wait on image
		 * @param {String} url The url of image to wait on load for
		 * @param {Function} callback Callback function to call when loaded
		 * @param {Object} scope The scope to call callback in.  Default to window.
		 * @param {Integer} perdiod Time in milliseconds between recalling wait
		 * @param {Integer} maxTries max attempts to call this function
		 * @param {Integer} tryCount defaults to 0.  Used for cancelling this.
		waitOnImage : function(url, callback, scope, period, maxTries, tryCount){
			if(typeof period === "undefined") period = 50;
			if(typeof maxTries === "undefined") maxTries = 100;
			if(typeof scope === "undefined") scope = window;
			if(typeof tryCount == "undefined") tryCount = 0;

			var hash = this._hashCode(url);
			var imgObj = this._images[hash];

			if(typeof imgObj === "undefined"){
				this.preload(url, null, null);
			} else if(imgObj.loaded || tryCount > maxTries){, imgObj);

			//use closures to call self
				function(url, callback, scope, period, maxTries, tryCount, _this){
					return function(){
						_this.waitOnImage(url, callback, scope, period, maxTries, tryCount);
				}(url, callback, scope, period, maxTries, ++tryCount, this)


The interesting part of the code here is the setTimeout function call.  This accomplishes 2 things.

  • Keeps this from blocking.  Meaning, setTimeout will execute in parallel to this code blocks execution.  If you look at the preload function you see it call this function, which if it just continuously called itself, it would block and prevent the function from completing until the image finished loading.  This would be bad as images can sometimes take quite a while to load.
  • allows for closure to scope the variables to this execution.


So we have covered the basics of creating a preloader, but its so basic we really need a mechanism to extend its functionality.

		 * Access the prototype to _Image to add your own attributes.
		 * @param {String} name Name of the reference in prototype.
		 * @param {Object} obj The function that will be referenced by name.
		extendImage : function(name, func){
			//todo add prototype stuff
			this._Image.prototype[name] = func;

This alows us to add whatever we want to the _Image object.  Its really just for accessibility, but a nice feature none the less.  Check out the source code and documentation here.

Hash Function in Javascript

Writing a Hash Function in JavaScript

So when I was writing my image preloader I had the idea where I wanted to store an image object in an array to easily look up.  Now the key would be the url, but URLs are big and can be ugly, so I really wanted to use a hashmap.  But javascript does not have a native implementation.  In addition, why should it, its arrays are associated and I could very well just use the url as the key.

Anyway, although it probably made more sense to use the URL as the key I wanted to use a hash for lookup.  Thought it would be fun.

So I did a quick google and found this

The Algorithm

s[0]*31^(n-1) + s[1]*31^(n-2) + ... + s[n-1]

Straight Forward Implementation

function hashCode(str){
	len = str.length;

	hash = 0;
	for(i=1; i<=len; i++){
		char = str.charCodeAt((i-1));
		hash += char*Math.pow(31,(len-i));
		hash = hash & hash; //javascript limitation to force to 32 bits

	return hash;

Now this is very straight forward, but isnt the implementation they gave in the link above.  That implementation is actually rather clever and avoids using an expensive multiplication and power function.

The Clever Approach

function hashCode2(str){
	var hash = 0;
	if (str.length == 0) return hash;
	for (i = 0; i < str.length; i++) {
		char = str.charCodeAt(i);		hash = ((hash<<5)-hash)+char;		hash = hash & hash; // Convert to 32bit integer
	return hash;

The Break Down

So I was a little confused and wanted to understand how these can be the same, so I dusted off my binary math hat and stepped through it.

char = str.charCodeAt(i)

Just gets the ansii value or utf-8 character code value.  So for example if str = “hello”, then str.charCodeAt(0) would return 104.  Check out this character chart.

hash = ((hash<<5)-hash)+char;

This is the nitty gritty scary part right.  Its actually not that bad and make sense once we step through it.


This is a simple bit shift operator of 5 to the left, but lets review bit shifts of the value 0001.

0001 is 1 in base 10
0010 is a bit shift left 1 and is 2 in base 10

var x = 1; //0001 in binary

Binary Value Bit Shifts Left Code Example Base 10 Value Algebra Equivalent Exponential
0001 0 x<<0 1 x * 1 x * 2 0
0010 1 x<<1 2 x * 2 x * 2 1
0100 2 x<<2 4 x * 2 * 2 x * 2 2
1000 3 x<<3 8 x * 2 * 2 * 2 x * 2 3

As you can see a bit shift is the same as multiplying by 2 to some power.  So lets look at what we can rewrite things to.

hash<<5 or
hash * 2^5 or
hash * 32

So we can rewrite

hash = ((hash<<5)-hash)+char; as

hash = ((hash * 32) - hash) + char; or

hash = hash * (1 * 32 - 1) + char; or

hash = hash * 31 + char;

Whew we figured out where the 31 came from.  Originally I thought it had to do with a prime number or something, but I was way off.

But what about the exponential?

To figure this out we need to step through the for loop

for (i = 0; i < str.length; i++) {
		char = str.charCodeAt(i);
		hash = ((hash<<5)-hash)+char;
		hash = hash & hash; // Convert to 32bit integer

1: hash = hash*31 + char1
2: hash = (hash * 31 + char1) * 31 + char2
3: hash = ((hash * 31 + char1) * 31 + char2) * 31 + char3

What I did above was replace hash with the value from the previous iteration to show the expansion of the for loop.

So at iteration 3 of the for loop we have…

hash = ((hash * 31 + char1) * 31 + char2) * 31 + char3

Lets do some algebra on this…

hash = ((hash * 31 * 31 + char1 * 31) + char2) * 31 + char3

hash = ((hash * 31 * 31 * 31 + char1 * 31 * 31 + char2 * 31 + char3


hash = hash * 31^3 + char1 * 31^2 + char2 * 31^1 + char3 * 31^0

which gives us what we were looking for…where hash = 0  to start

hash = 0 * 31^3 + char1 * 31^2 + char2 * 31^1 + char3 * 31^0

which gives….

hash = char1 * 31^2 + char2 * 31^1 + char3 * 31^0

hash = s[0] * 31 ^ (n-1) + s[1] * 31^(n-2) + s[2] * 31^(n-3)

There we go.  We have proven it works!

Arcgis, How to Find the Point User Clicked On

About This Post

The main goal of this post is to explain how I figured out finding which point a user clicked on in an generated map.  You would think the API would cover this, but unfortunately it is a problem that others found.  Here is thread about the problem.  I spent considerable time trying to solve this problem.  Because I knew it was possible, but why wasn’t it in the javascript API?


  • Load a Feature Layer with your data
  • Listen for a map click and identify which point the user clicked on.

Simple goal right?  You wouldn’t believe that this was actually quite difficult.  If there is now a simple solution in the API, I swear as of 1/1/2013 there was not.

Side Note:
The infoTemplate is passed the data, but I couldnt figure out for the life of me how to get this content.  We know its there, but I didn’t want to spend forever unraveling the arcgis code base looking for a way to do this.  I never ended up being able to do it this way, so I had to force it with the arcgis javascript API.  We could wait till the infoTemplate loaded and then parse its HTML elements, but that isn’t a very good solution either.

What we need?

Solution / Example

1. Setup Our Feature Layer

Lets example this part of the code in initOperationalLayer function from the downloaded example above.

featureLayer = new esri.layers.FeatureLayer("",{

mode: esri.layers.FeatureLayer.MODE_ONDEMAND,
outFields: ["*"],
infoTemplate: infoTemplate

We want to replace our feature layer in bold and underlined above with the our recreational one I referenced above.



If you run this, you will now see our map shows the available recreation centers that were in our feature layer map.

2. Lets Hook into the Map onClick Event

So at the bottom of the initOperationalLayer function we want to add the following code:

//hook into the onClick event
dojo.connect(featureLayer, "onClick", function(evt){

	//we need to query the feature layer for this point
	var query = new esri.tasks.Query();

	//get the location of mouse click..The API doesnt tell you about this.  This is the magic part
	//I stumbled across this value by using debugger and console.log(evt) and searching the object.
	query.geometry = evt.graphic.geometry;
	query.spatialRelationship = esri.tasks.Query.SPATIAL_REL_CONTAINS;

	//select the actual point
	featureLayer.selectFeatures(query, esri.layers.FeatureLayer.SELECTION_NEW,
			//user clicked on something, then we should have features
			if(features.length > 0){
				var data = "";
				for(var x in features){
					data = "Feature "+x+":n";

					//building a string of the queried data
					for(var y in featureLayer.fields){
						var name = featureLayer.fields[y].name;
						data+= name+" : "+features[x].attributes[name]+"n";
				alert("Failed to select the feature");


The special part is this line:

query.geometry = evt.graphic.geometry;

This gives the query the “spatial reference” (I guess that is what you might call it) to perform query on Feature Layer for.

3.  That it, we are done.  You can get the solution here.

Using Google Scripts to Host a Website

The Objective…

1.  Have a framework for hosting static HTML pages at no cost
2.  Linking between pages should still work

Where can I host some content?

There are probably lots of places out there where you can host for free.  Your internet service provider might have free space.  There might even be places in yahoo to do this.  And of course there are (this blog), blogger, etc…  These will allow you to get content online for free, but limits how you create the content and how you maintain it.

For me I was looking for a way to host my documentation (a continuation of this post).  So I wanted to use my documentation that is available on, but be able to view it as a web page.  Really I was just playing around and having some fun.

Here is how I got it done

Setup your google scripts.  See my other post on this.

In order to host the github pages, we need to dynamically get the HTML we want to load.  In my example I will be using for hosting, but you could use things like or the like.  Google scripts gives an easy way of dynamically getting content.


However, following the steps from my previous blog post we can show the content, but linking will not work.  So we will have to parse the downloaded document.

Parsing the document and rewriting links

1.  Find “<a ”
2.  Check that there is an “href” attribute
3.  Parse the actual link
4.  Replace the link contents

The problems I came across

1. First pass, I didn’t account for links without “href” attribute
2.  Thought it was so easy I could just write it procedurely with no functions…that was a mistake.  Meaning, just because your playing around, doesn’t mean you should ignore breaking breaking a problem down into simpler parts.
3.   Its really hard to diagnose problems in google scripts…
3a.  Put all the code in your .gs files so you can step through the code.  This would have saved me a bunch of tme.

Finding the Next Valid “<a ” tag

* Return the this position
* <a class="someclass" href ="/some/url"
* ^
* pos
* @param String The raw HTML to parse
* @param int The start position to begin parsing of the rawHtml
* @return int The position or -1 on fail
function getNextAtagUrlStart(rawHtml, pos){
var temp = rawHtml.substring(pos,rawHtml.length).indexOf("<a ");
var start = pos + temp;

//no starting a tag
if(temp == -1)
return -1;

//the href.  We dont want to look too far ahead
var maxLen = Math.min(start+50,rawHtml.length);
var temp = rawHtml.substring(start,maxLen).indexOf("href");

//this atag has not href, so lets look again
if(temp == -1)
return getNextAtagUrlStart(rawHtml,start+2);

start += temp+4;//4 is for the "href" string

//ship whit spaces...href       =    "
while(rawHtml[start] == " ") start++;

//skip the  "="

//skip till we get the url (actually the " or ')
while(rawHtml[start] == " ") start++;

//if this is an absolute linke skip it.  This assumes that internal links
//are relative.
if(rawHtml.substring(start+1,start+5) == "http")
return getNextAtagUrlStart(rawHtml,start);

//return the position of the link including the leading " or '
return start;

Get the End of the URL string

//skip the url portion of url
function getEndOfUrl(rawHtml, start){

//should give a " or '
var wrapper = rawHtml[start];

//end tokens
var end1 = wrapper+">";
var end2 = wrapper+" ";

var token = function(pos){
return rawHtml.substring(pos,pos+2);

while(token(start) != end1 && token(start) != end2)

//this take use to the position after the closing " or '
return start+1;

Get only the <body> Content

function getInnerText(rawHtml){
var start = rawHtml.indexOf("<body");
var end = rawHtml.indexOf("</body>")+6;

return rawHtml.substring(start,end);

Create your HTML Template File


<!--  Use the w3 default template so things look ok -->
<link rel="stylesheet" href="" type="text/css">


<? output.append("<div"+body.substring(5,body.length-7)+"</div>"); ?>


Put it All Together

So google scripts uses a defaul “main” function.
1.  doGet – This handles GET requests.  Just normal webbrowser
2.  doPost – This handles POST requests.  These more than likely come from form submits.

First I put some global parameters…

var base, domain, tempalte, file;

This doGet function will set these global variables…

function doGet(e) {
var tpl;

//need to just handle when we are missing template
if(typeof e == undefined || e.parameter.file == null || e.parameter.domain == null){

tpl = HtmlService.createTemplateFromFile('404');//load the 404 html file
} else {

//choose template to load.  I have a base one described above and some others with specifc styleings
template = (typeof e.parameter.tpl == undefined || e.parameter.tpl == null) ? "base" : e.parameter.tpl;

base = ScriptApp.getService().getUrl();
domain = e.parameter.domain;
file = e.parameter.file;

tpl = HtmlService.createTemplateFromFile(template);

//dynamically load the html body
var response = UrlFetchApp.fetch(domain+file).getContentText();

var body = getInnerText(response); //get <body...</body>
var pos = getNextAtagUrlStart(body,0);
while( pos < body.length && pos != -1 ){
var endUrl = getEndOfUrl(body,pos);
var url = getUrl(body, pos+1,endUrl-1);

//splice in the url
body = body.substring(0,pos+1)+url+body.substring(endUrl-1,body.length);

pos = getNextAtagUrlStart(body,pos);


//make the body varaible available to the HTML template file we loaded
tpl.body = body;

//render the page
return tpl.evaluate();



In the end there are some short comings of my implementation.

1.  Doesn’t handle forms
2.  Doesn’t seem to handle anchor tags for bookmarking and internal references

Here is an example where I load my pages from

Jquery UI Select Box Plugin

 Jquery UI Select Box

This is available on JQuery as a plugin.

So a couple of weeks ago I was working on a project for a customer and decided I needed to use JQuery UI since I was going to have some popup boxes and wanted some pretty buttons.  So I applied JQuery UI no problem, but forgot that it does not have a select box.  Yeah I could work around it using Autocomplete or just try and style the select menu up a little with some css, but that would’t be very cross browser consistent.

And then I thought how hard could it be?  I mean all I need to do is change the css and put an overlay to hide the drop down arrow.  And then I could simply trigger the drop down to show.  Something like this…


Basically I would just inspect the jquery css styles and apply some of the classes to the html to make a jquery styles selct box overlay.  So if the HTML would look something like this…

<div class="my-ui-select">
	<span class="ui-spinner ui-widget ui-widget-content ui-corner-all">
		<select style="border:none;" >
			<option>Option 1</option>
			<option>Option 2</option>
		<a class="ui-spinner-button ui-spinner-down ui-state-default ui-corner-right">
			<span class="ui-button-text"><span class="ui-icon ui-icon-triangle-1-s"></span></span>

I figured it would be pretty simple to then bind a click event to my <a> tag.  Which is a custom drop down arrow using jquery styles.

//bind event to <a> tag
$('.my-ui-select a').click(
		//trigger the select menu click event

Simple enough right?  Well in actually there is a huge difference in programmatically triggering an event versus a user’s mouse triggered event.  This is probably due to security concerns.

Possible Work Around

So in my searching I came across this entry in stack overflow.  One of the recommendations is to change the size of the select menu.  This would work if you absolutely position the element and use the wrapper description I gave above.

The javascript would look something like this…

$('.my-ui-select a').click(function(){
	//trigger the select menu click event
	var select = $(this).parent().find('select').first();

Now, this is a very reasonable work around but it has its downsides.

  • The drop down list will not look like other JQuery UI elements when the drop down happens.  Some css styling might make this work alright though.
  • The size attribute can be a little tricky to get right.
    • If your select list has 2 elements, making it a size of 2 will give a lot of white space.
  • Positioning is tricky
    • The <a> tag has to be in-front of the default drop down arrow, but the wrapper which creates the border has to be below the select menu so the default actions work on the select menu.  Getting this position correct on all browsers will be tricky, but not impossible.
  • The scroll bar is default and cannot be removed (might be able to apply styleing, but this wont be corss 100% browsers).
  • In the end, it just doesn’t look quite right.  And this is just in Chrome.  Figuring out how to tweak styles to be good in all browsers will be a serious pain.

In the end you end up with something like this…

Download the above example here.  Please keep in mind, this was only for Chrome and looks bad in some other browsers.  You will need to spend some time playing with the css, but in the end you should be able to get this working fairly nice.  Also might be a great lightweight solution.

My Solution


So the above solution is pretty good and super simple.  You are using the native select menu with a little bit of a hack.  This is great, because all your events for the select menu make sense.  But the down side is it has some issues looking polished and the drop down doesn’t really follow the other Jquery UI.

So I needed to resolve these problems.  My solution was to combine the css styling used above coupled with Jquery  UI AutoComplete Widget.

How will this work…

  • Use above css to make something that used JQuery UI css to look nice.
  • Use jquery fn to create our own selector function
    •  Basically we will parse select elements, hide them and create our new css to make things look nice.
  • Apply AutoComplete Widget to our styled drop down menu to give the illusion of a select menu.

Why did I go this route?

  • Cross browser functionality
  • Cross browser looks
  • Maintain Jquery UI consistancy in my own widget
  • Use Jquery UI widget to hadle the heavy lifting of the select menu.  I just need to write an API wrapper for AutoComplete widget to make it do a little more.

In the end this was very simple to implament.  About 100 lines of code and very straight forward.  The real issues arose in event handling.  The AutoComplete Widget API is ok, but frankly a little lacking.  So working around its limitation and forcing it into my own needs took a little fiddling.

Here is the example page.

If you have any questions let me know.  The code should be pretty simple to step through on your own, but I would be happy to explain what I can.

Free Hosting with Google Scripts

The Problem…

I have some projects that I work periodically and wanted to be able to host examples of on the web, but don’t want to pay for hosting (

So the hosting needed the following

  • Insert my own javascript (So blogger, wordpress, etc… will not work)
  • Dynamically load content from external host (my source code on github)

The Solution…

I was playing around with google looking for a way to host content and came across Google Scripts.  The documentation can be a little bit of a mess, but after a couple hours of playing around I was able to figure out the basics.  A more detailed version of this will come later.

So for this I wanted to host my documentation for project.  I know I could have done this with github wiki, but this has limitation as I cannot actually insert example javascript.

Steps Taken…

1. Setup a project (

2. Create an HTML template file
Create HTML

3. Save this into the HTML template

<?  //dynamically load the html body
var response = UrlFetchApp.fetch(“”);

4. This uses the scripts API to grab the content. Then it uses the default global variable “output” to insert the loaded content.

5. Edit the Code.js and insert this content

// Script-as-app template.
function doGet() {
return HtmlService.createTemplateFromFile(‘selectbox2′).evaluate();

6.  Publish your project

Here is my example.


There are some downsides…
1.  The anchor tags dont work correctly.  I’ll look into fixing this.
2.  How I currently did this I would have to setup a template for each page.