iSpot & EML project blog

Just an update as to some of the things that have changed on the site over the pasts couple of weeks:

• The drop down lists for the predictive text fields on Scientific and English name fields have now been given scroll bars which has allowed the returned results to be upped from a maximum of 20 now to 100
• The ‘get recommended’ function has been altered slightly to allow for the possibility of multiple recommended scientific names that may be in different iSpot groups (which is the case with a small number of dictionary entries). The results for these names now contain the informal group as part of the displayed name to the user to allow the user to differentiate between them. On the whole most users won’t see any change.
• The reputation system has now been changed to cap the amount of points a user can pick up on each observation. This now means a user can only ever gain a maximum per observation equal to the amount an ‘expert’ would bestow by agreeing. This stops users from potentially having a large reputation from only a small number of observations.
• the ‘add this’ button which allows people to add a link to various sites such as facebook has now been updated to have a smaller list of sites in the list and also not pop up straight away when the mouse pointer moves over the button

After an annoying start to the week, with my MacBook Pro crashing after I was prompted to install updates, I decided to make some changes to how the adding of a location occurs whilst my Macbook was being resurrected.

When adding an observation, a user can select a location they’ve previously used by using the predictive text box for the location title. Up to now this has automatically overwritten any information already present i.e. title, latitude, longitude, notes and hide the precise location tick box. The problem with this was that it’s very easy to miss the fact that the latitude and longitude have been changed which could potentially lead to inaccurate recording of an observations location. This has now been changed so if a user already has added location information and then uses the predictive text box to select a title the user will be asked if they want to use all the details associated with the selected title or just the title itself.

My MacBook is now usable so I’ll be pressing on with the mobile app/API development side of things.

The user profile page on iSpot has had a bit of a re-design, this came about as a result from a combination of other things I’ve been working on. The new format gives the users’ reputation section now as a table instead of the previous, hard to read, sentence for each group.

In my opinion I think this is a real improvement and seems to give much more meaning to the figures shown relating to number of observations, identifications and agreements per group for a user.

Apart from the normal amount of tasks that come along to side track me, I’ve been working on a re-write of the spec for a mobile app which has then lead me to continue with the creation of documentation for the iSpot API.

On the mobile app spec, I’ve spent some time creating mock up images of the various screens that will comprise the app. I felt this was needed so an android app and iPhone app would have half a chance of looking similar if being developed simultaneously. I’ve also revised some of the functionality with more of an emphasis on tabbed layouts which I feel work better. The final piece of the spec which is yet to be completed are the references to the calls needed to the iSpot API, which leads me on to this.

So, continuing with the creation of a wiki page that documents the iSpot API. This is not just a case of writing up each of the calls but actually deciding what these calls should be, what parameters need to be passed in and what type of response should be produced. I think I’m 90% there now although it’ll probably be something that continues to change/grow over time.

As part of continuing the API documentation I’ve been looking at the parts of the API that I’ve already implemented. This has lead me to get RSS output working now, which means that any API call that would benefit from being formatted as RSS, now can be.

The latest round of performance improvements has now taken place. Although just prior to the implementation of these the site was mentioned on ‘Bang goes the theory’ and coped with the small spike in traffic this caused. I think this is evidence of improvements brought from the previous round of performance changes.

This latest round has focused on the move of the site onto higher spec hardware with considerably more RAM to allow the site to cope with many more concurrent requests. Along with the RAM there have also been changes to the web server configuration, with two instances of Apache running; a smaller footprint gateway web server which is responsible for serving all the static page requests such as images and stylesheets and then a second web server being passed all the requests for dynamic pages e.g. drupal pages. There are many more requests for static pages than dynamic so this should allow the smaller footprint server to deal with these requests whilst saving memory for the dynamic page server to use.

The use of a gateway server also means we can map calls to the bayesian keys website to a URL without the :8080 in it, also we can have the keys website hosted on an entirely different server whilst still using the ispot URL.

Along with the move of the site I have also now set up the ability to monitor this site in a much more detailed way so allowing us to pinpoint bottlenecks/issues with the site and rectify these. This has already lead to the identification of several bottlenecks that have been fixed.

On the database server side of things, this has also been moved to a new server with much more RAM and has been set up for this along with making sure that the database configuration is matched to the PHP configuration on the web server. There has also been a change to the types of tables used to store the website data, this is in line with changes that have taken place in Drupal 7.

There are still many things that can be improved to help the site’s performance, these include:

• adding more caching using things such as Apache Solr and Memcache
• combining many of the images used in the site theme to one file so minimizing requests to the server
• move the Drupal core code over to Pressflow(a re-worked version of Drupal with greater scalability)
• changing various aspects of the site code such as image handling and badges/roles.

Before moving the site to the new servers we carried out some benchmark testing on them and, although it’s hard to mirror real world usage, the results seem to suggest that the new servers are capable of handling somewhere in the region of 10 times the amount of traffic of the old servers… time will tell. The beauty of the new architecture is that we can up capacity if we believe that we may need to, so we are in a much better situation for potential increases in site traffic.

I’ve now started work on the implementing the API for iSpot to allow external applications access to services provided by the site. I’ve concentrated on actual calls that we’re going to be handling as this follows on from the work that Richard Lovelock did in setting up the security layer using OAuth.

The REST server that comes bundled with the Services module that we’re using in Drupal doesn’t seem to have support for the multipart/form-data encoding type that allows files to be uploaded alongside normal form fields. This means that the handling of uploaded files is problematic. It looks possible to extend the REST server which is what I’ve now done to allow the handling of the multipart/form-data, although I’ve not yet tested this.

Assuming the extended handling that I’ve written works then creating the rest of the functionality to handle the calls from the API should be reasonably straight forward.

Along with all the normal maintenance work of iSpot I’ve had to spend some time this week on fixing the forums on the site. This was to do with the updates of various modules and also the core drupal code to patch potential security vulnerabilities. This left some users unable to post to various of the forums and also changed the breadcrumbs that were displayed for the forums. It took longer than I’d have liked to track down exactly what was happening but once I had it didn’t take too much to fix it.

This week I’ve also managed to add a new section on the site for the mentors to keep the users up to date with news and events. It’s only on the approval site at present while we work out the exact format of the pages but the area is called ‘in your area’ and has it’s own link from the main navigation at the top of the site.

The replacement of the current location functionality is pretty much ready for release. I’ve now finished the redevelopment of what was the my spot locations tab, now called the map tab, this is now just one large map with pins for each of a users observations. The pin icons themselves correspond to the group that the observation is in expect for locations with multiple observations, for these there is a different colour pin with the iSpot logo on.

In preparation for the live release of this I’ve updated the beta site – ispot-beta.open.ac.uk so this is now open to anyone how would like to try out/test the new features before hand.

The other bugs that there had been with the observation input form have now been resolved although there may still be others waiting to be uncovered

At last!!! The long awaited changes to the locations on iSpot are ready for approval. These include:

• the observation location being shown in a popup similar to the popup used for showing people who’ve agreed with a determination.
• pins used in place of vague green circles.
• a 1km grid square used instead of a pin where the user wants to hide the exact location.
• a re-worked input including; search by place name or by OS grid coordinate.
• use of meta data from images that are uploaded, allowing users to fill in details of the observation form from these.

I’ve made the new version of the site available on the approval site and I’ll email the link to people. After people have had a chance to test out the new features I’ll collect feed back and make any appropriate changes before testing again and then making live.

These changes will now pave the way for adding many new features based around locations, such as searching for observations in a given area.