iSpot & EML project blog

The latest developments that are taking place to iSpot are to do with the functionality around the species dictionary. This will allow multiple dictionaries to be used and put in order of priority for use. In time this will also allow a priority to be assigned based on specified location.

The new functionality allows users to visit a page for any valid term i.e. any term, either scientific or english, that appears in the active species dictionaries’. This page will at the least show any observations that have an ID that match the term and for dictionaries that specify a hierarchy for their terms this page will show all observations that are also in any of the sub groups of the taxonomy hierarchy. Along with this page there is an accompanying block that appears on the observation page that gives hierarchy links where the dictionary provides them.

This, I think, will be a very powerful new feature of iSpot which will dynamically create a valid URL for any term in any active species dictionary, including synonyms which will automatically display the page for the recommended scientific name. I’ll be sending a link round to the project team members for this new functionality on the approval site so we can have a discussion about any implications and any additional work around this that may be needed.

I’m presently working on the admin area for the species dictionary which allows administrators to activate or disable individual dictionaries along with setting the priority hierarchy of the active dictionaries. It will also allow editing of the terms within each dictionary so allowing administrators of dictionaries to correct or add any terms.

Just an update as to some of the things that have changed on the site over the pasts couple of weeks:

• The drop down lists for the predictive text fields on Scientific and English name fields have now been given scroll bars which has allowed the returned results to be upped from a maximum of 20 now to 100
• The ‘get recommended’ function has been altered slightly to allow for the possibility of multiple recommended scientific names that may be in different iSpot groups (which is the case with a small number of dictionary entries). The results for these names now contain the informal group as part of the displayed name to the user to allow the user to differentiate between them. On the whole most users won’t see any change.
• The reputation system has now been changed to cap the amount of points a user can pick up on each observation. This now means a user can only ever gain a maximum per observation equal to the amount an ‘expert’ would bestow by agreeing. This stops users from potentially having a large reputation from only a small number of observations.
• the ‘add this’ button which allows people to add a link to various sites such as facebook has now been updated to have a smaller list of sites in the list and also not pop up straight away when the mouse pointer moves over the button

After an annoying start to the week, with my MacBook Pro crashing after I was prompted to install updates, I decided to make some changes to how the adding of a location occurs whilst my Macbook was being resurrected.

When adding an observation, a user can select a location they’ve previously used by using the predictive text box for the location title. Up to now this has automatically overwritten any information already present i.e. title, latitude, longitude, notes and hide the precise location tick box. The problem with this was that it’s very easy to miss the fact that the latitude and longitude have been changed which could potentially lead to inaccurate recording of an observations location. This has now been changed so if a user already has added location information and then uses the predictive text box to select a title the user will be asked if they want to use all the details associated with the selected title or just the title itself.

My MacBook is now usable so I’ll be pressing on with the mobile app/API development side of things.

The first version of the iSpot API is now ready for testing on the dev site along with the completed documentation as a wiki page. This follows on from the development of the latest version of the mobile app spec and should now allow all the necessary calls to be made for a functioning mobile app to be produced. The android developed that we’ve employed has said he should be ready to commence with development as of next week so this ties in nicely. It also allows for myself to get back to the development of the iPhone version which I also hope to continue with next week.

The final piece of work that will need to be carried out on the site before launch of the mobile apps is to create a mobile version of some of the site pages that will need to be accessed by the apps, such as the OAuth authorization pages and the login pages.

This week after completing the first version of the API documentation I’ve been continuing with the implementation of it. Along with this I’ve also been involved with the recruitment of an Android developer to produce the Android version of the mobile app. So long as all the paper work gets completed I’m hopeful that the developer should be starting on the Android app this Thursday which means I need to make sure the API is somewhere close to completion.

In the course of working on the API I also uncovered a potential bug in the ‘changes’ page in the ‘my spot’ area of the site so hopefully this may resolve the odd unexplained result that users get on this page.

This week I’ve also set up a new section in the forums to complement the ‘in your area’ part of the site that is being created.

The user profile page on iSpot has had a bit of a re-design, this came about as a result from a combination of other things I’ve been working on. The new format gives the users’ reputation section now as a table instead of the previous, hard to read, sentence for each group.

In my opinion I think this is a real improvement and seems to give much more meaning to the figures shown relating to number of observations, identifications and agreements per group for a user.

Apart from the normal amount of tasks that come along to side track me, I’ve been working on a re-write of the spec for a mobile app which has then lead me to continue with the creation of documentation for the iSpot API.

On the mobile app spec, I’ve spent some time creating mock up images of the various screens that will comprise the app. I felt this was needed so an android app and iPhone app would have half a chance of looking similar if being developed simultaneously. I’ve also revised some of the functionality with more of an emphasis on tabbed layouts which I feel work better. The final piece of the spec which is yet to be completed are the references to the calls needed to the iSpot API, which leads me on to this.

So, continuing with the creation of a wiki page that documents the iSpot API. This is not just a case of writing up each of the calls but actually deciding what these calls should be, what parameters need to be passed in and what type of response should be produced. I think I’m 90% there now although it’ll probably be something that continues to change/grow over time.

As part of continuing the API documentation I’ve been looking at the parts of the API that I’ve already implemented. This has lead me to get RSS output working now, which means that any API call that would benefit from being formatted as RSS, now can be.

The latest round of performance improvements has now taken place. Although just prior to the implementation of these the site was mentioned on ‘Bang goes the theory’ and coped with the small spike in traffic this caused. I think this is evidence of improvements brought from the previous round of performance changes.

This latest round has focused on the move of the site onto higher spec hardware with considerably more RAM to allow the site to cope with many more concurrent requests. Along with the RAM there have also been changes to the web server configuration, with two instances of Apache running; a smaller footprint gateway web server which is responsible for serving all the static page requests such as images and stylesheets and then a second web server being passed all the requests for dynamic pages e.g. drupal pages. There are many more requests for static pages than dynamic so this should allow the smaller footprint server to deal with these requests whilst saving memory for the dynamic page server to use.

The use of a gateway server also means we can map calls to the bayesian keys website to a URL without the :8080 in it, also we can have the keys website hosted on an entirely different server whilst still using the ispot URL.

Along with the move of the site I have also now set up the ability to monitor this site in a much more detailed way so allowing us to pinpoint bottlenecks/issues with the site and rectify these. This has already lead to the identification of several bottlenecks that have been fixed.

On the database server side of things, this has also been moved to a new server with much more RAM and has been set up for this along with making sure that the database configuration is matched to the PHP configuration on the web server. There has also been a change to the types of tables used to store the website data, this is in line with changes that have taken place in Drupal 7.

There are still many things that can be improved to help the site’s performance, these include:

• adding more caching using things such as Apache Solr and Memcache
• combining many of the images used in the site theme to one file so minimizing requests to the server
• move the Drupal core code over to Pressflow(a re-worked version of Drupal with greater scalability)
• changing various aspects of the site code such as image handling and badges/roles.

Before moving the site to the new servers we carried out some benchmark testing on them and, although it’s hard to mirror real world usage, the results seem to suggest that the new servers are capable of handling somewhere in the region of 10 times the amount of traffic of the old servers… time will tell. The beauty of the new architecture is that we can up capacity if we believe that we may need to, so we are in a much better situation for potential increases in site traffic.

I’ve now started work on the implementing the API for iSpot to allow external applications access to services provided by the site. I’ve concentrated on actual calls that we’re going to be handling as this follows on from the work that Richard Lovelock did in setting up the security layer using OAuth.

The REST server that comes bundled with the Services module that we’re using in Drupal doesn’t seem to have support for the multipart/form-data encoding type that allows files to be uploaded alongside normal form fields. This means that the handling of uploaded files is problematic. It looks possible to extend the REST server which is what I’ve now done to allow the handling of the multipart/form-data, although I’ve not yet tested this.

Assuming the extended handling that I’ve written works then creating the rest of the functionality to handle the calls from the API should be reasonably straight forward.

An elusive bug that’s been an issue for sometime has at long last been dealt with. The bug was to do with the ‘get recommended’ button, it seemed to not work 100% of the time. Recently one of the ispot users realised that it was associated with when they were uploading photos. Having tracked down exactly what events were triggering it allowed me to track down the actual cause and fix it.

This has been an annoying bug that we’ve been aware of for sometime so it’s been great to finally get it fixed… so thanks to the iSpot user for helping to work out what was going on!!