iSpot & EML project blog

The end of last week saw the design ready to be uploaded to the approval server. After the showing the team this at Mondays meeting, various suggestions were made for slight changes along with requests for a couple of new features to be added for the launch of the re-designed site.

This week has been taken up with implementing all the requests. The design now works on IE6 although it’s not quite to the standard of the newer browsers. The various small general issues have been fixed with the exception of one or two things yet to complete. The two big new features that have been added are:

• images to the search results, I’ve also changed the displaying of identification search results they’re now showing the observation they’re made on and added images to these too
• a related observation carousel on every observation page. This looks the same as the latest observations carousel on the front page but obviously is populated with observations that share the same likely ID

Another feature that we’ve yet to add is an extra section on the front page under the latest observations that will look similar but will be populated with old observations that are yet to have a likely ID, we were thinking of calling this block something like ‘can you help’.

The final thing that we’ve done this week is created a new domain for the beta testing of the re-design prior to launch. This is accessible externally and is a separate site to the internal approval site. The URL is……. ispot-beta.open.ac.uk

The site re-design is going very well, although we haven’t quite managed to get it to the point of uploading it to the approval site. I expect to now give people access to it on the approval site sometime in the coming week but I’d prefer it to be as near to completion as possible so any comments will be more valid. It present I’m putting the finishing touches to the ‘Add an observation’ form page and Rich is working on getting the latest observations carousel on the front page working.

This week we’ve been concentrating on creating a new theme for the site using the approved re-design. The first two and a half days were taken up with creating a design for the various elements of the site that are used elsewhere other than the front page. The rest of the week has been taken up with starting to implement this new design.

I’m hoping that by the end of next week we should have a site ready for viewing on the development server. I’ll send an email around when it’s in place for people to look at.

The recently configured back-up architecture and plan were given a chance to be tested over the past couple of weeks with a live swap over for iSpot and the Evolution Megalab. This was brought about by a disk failure on the live web server. The switch over included moving from the live web and database/file server to two completely different back-up web and database/file servers which then assumed the role of live servers whilst maintenance was performed on the original live web server.

The switch to the back up servers went fairly smoothly and with only a minimal amount of  downtime of the live websites. With the exception of the quiz on Megalab (which is something that needs to be looked at), both sites continued to function as normal on the back up architecture.

The disc failure problem on the live web server was rectified after about two weeks which then allowed the site to revert to the original live web server and database/file server set up. A procedure to do these switches had been documented and followed. We had originally believed that the the swap back to the live set-up had gone very smoothly, and generally speaking in terms of the technical tasks performed it had gone smoothly. However there were a couple of issues which resulted in some significant down time of the live sites:

1. When the live site files were copied back to the live web servers from the ‘backup live’ servers the websites were put in to maintenance mode so that the public could not access them and no data was written to the databases. A flag is set in the database to put the sites in to maintenance mode. After the files and databases had been copied back on to the live servers, the DNS records were changed so that requests to iSpot and the Megalab were then directed to the original live servers. The flag on the databases on the live servers were set to be online so that as the DNS changes propagated to the DNS servers of various ISPs, users trying to access the live sites would then be able to access them on the live servers (this can generally take between 24 and 72 hours or so and is out of our hands). The oversight on our part that caused a problem was that there is a scheduled task (as part of the original back up routine) to back up the live database and save/overwrite the copy on the back up server. This task performed its duty and overwrote the database on the back up server from live but now with the offline/online flag set to online. This happened within an hour of switching back to the live set up and before the DNS propagation had taken effect so that requests for iSpot were generally still pointing to the back up server which had now (inadvertently) been set to online, so users began posting observations and we believed that we were seeing the site appearing on the live servers. Then at hourly intervals the database on the back up server (which was being updated with user observations etc) was over-written with the database from the live site – which was effectively old data. The issue was picked up when a new news feature item dissappeared from the home page. The scheduled task was then stopped and the flag was set to make the sites offline again on the back up servers. Once the propagation of the DNS records started to take effect, users were successfully directed to using the sites on the live servers again.
2. The second problem was that we had requested internally that the external hosting company be notified to update the DNS settings of iSpot and the Megalab and our request was not fulfilled. When we chased up about the request we were told that it was the responsibility of a different person so we then had to chase up the request with a different person to get it actioned. We weren’t given any notice that it wasn’t the responsibility of the person that we logged the request with.

Lessons learnt

• Apart from the obvious frustrastion of a period of a couple of hours where data was lost, we were generally very pleased with how the back up architecture performed and how smoothly the transition from each system went.
• We have updated our documentation to reflect any changes to the procedure to make sure that data isn’t lost in the future.
• During the switch back to live it came to light that there may be an alternative and more efficient way of switching between servers that will result in less downtime of the site for users. Instead of asking the external hosting company to switch the destination of the request to iSpot and the Megalab on their DNS servers and waiting for these changes to propagate, there is a tool that we can use that can internally (internal to our network) control the routing to various servers/IP addresses for given requests. This would effectively mimic a change to an external DNS record but as there would be no external propagation required the switch between servers would be almost instantaneous which should therefore result in only a minimal amount of down time for the live servers.

There’s been the usual few maintenance points this week including, a new webservice for S159 to allow validity checking of the exported data, the clear my unread link getting fixed and some changes to the expert badges. As well as these I’ve extended the idea of the likely ID idea to now give an ‘initial ID’ on an observation that doesn’t yet have a likely ID, this means that most observations will now have some kind of ID for the summary view.

Aside from the above, the two big features that we’ve started work on this week are:

• A next & back feature that will allow users to navigate between observations without having to keep going to the parent page.
• An overhaul of the location side of the site.

Both of these features have some complexities to them that need considering in developing them but should provide powerful additions to the site.

This week has seen us able to again focus work on developing the site further, here’s what we’ve been doing this week:

• the addition of a knowledgeable user role for each of the top level groups. Allows users to have their reputation seeded without having to be classed as an expert.
• likely ID feature added. This rates each identification in an observation based on the sum total of the reputation of each user that has agreed with the reputation of the ID author, if the highest scoring ID is above a certain threshold the ID gets a ‘likely ID’ badge and is shown on the summary view of the observation.
• A bug with the oucu login has been fixed. This bug was preventing some users from seeing certain content.
• S159 students will now get a different badge upon completion of the course. This had previously been talked about but not implemented until now.
• A new login feature that allows users to login using an existing account from one of the following providers: google, yahoo, wordpress, facebook, twitter and windows live. At present this will only allow users to create a new profile on iSpot by logging in using this method but the module is being actively developed so linking to an existing profile may soon be possible.

We have now at last got the automated backup for the live site working on an hourly basis. There are still a couple of things that we’re waiting for to do with the networks to allow us to have a complete mirror site working for both the research and backup site, but the actual data is all now on an hourly backup cycle. The remaining network things should hopefully be sorted out today which will mean that we should have the research copy of the site up in the next few days.

On the maintenance front, I spent some time rewriting the role checking code for S159 students which should now mean no more problems with students not being assigned the correct privileges. It does mean that the code runs a bit more frequently than I’d wanted but this doesn’t impact on the user experience.

This week has seen the addition of links to both EOL and the NBN in iSpot. These links have been added to all identifications that have a valid taxon name in the NBN species dictionary, the scientific name is given precedence but if the scientific name is invalid and there is a valid English name then that is used instead. At present these links are only created once the identification has been published but I feel these links may be a good aid prior to publishing an identification and so it may be a good idea to look into how best to let users use these resources separately from publishing identifications.

As well as these links being added some of the other things that we’ve also done are as follows:

• Rich has added more custom reports for use by the iSpot team including User statistics which has some interesting information
• The active S159 students now have a badge so they can spot each other on the site
• it’s now possible to view who’s agreed with an identification, this still needs a bit of work to get the popup using a fixed size box and then the use of a scroll bar but it’s usable for the time being
• many small improvements to both the look & feel and functionality spread throughout the site!!

We’ve been doing a lot of work on addressing issues concerning various pages, both their look and usability. We’ve got the current theme working correctly on pages that previously had problems and have created a better way of viewing teasers of observations. The add some images page has been tidied up as has the second step page to it, which now also makes it clearer to users that they need to complete this step to use the images in the site.

The lightbox functionality has been tweaked for displaying images and this is now being used as the more feature on the latest observations block on the front page. The latest observations page that used to be linked to from this more link has been changed to the browse observations page a link to which can be found in the right hand ‘browse’ menu of the site. This page has lots of added functionality and allows users to apply filters to all the observations and browse the result.

The search functionality of the identification fields has also been improved and it now deals with hyphens as well as being able to search on multiple partial words. Hopefully this will be helpful to Mike and the problem with fungi

This is more of a reminder to myself in case this slips my mind at some point in the future.

The BeautyTips module that I’m using for creating the help icon popup uses an external library called ExplorerCanvas to allow it to work in IE.

I’ve just found that IE 7, and probably the earlier ones too, have a limit on the number of CSS documents that can be imported into a document, this limit is 31. This then means that IE 7 can throw a javascript error if the javescript is trying to use createStyleSheet() which in my case happens on line 112 of excanvas.js. This will only be a problem in the development environment where CSS optimisation is disabled. The live site has this enabled and so there should never be a large number of CSS documents being importedand so no problem.