Behind the Scenes - Glendale App

Glendale Skye Auroras

Back

Introduction

Ever wondered how the Glendale App works? How does it achieve such unrivalled accuracy? How can a teuchter on Skye claim to have created the world's only 100% accurate aurora alert app?

I am going to try to explain a little about what goes on behind the scenes and what it is actually doing.

History

The first version of the app was released in November of 2015. It included the live reports, IMF and Solar Wind panels. All alerts had to be done manually. I would take test photos regularly and then manually send out push alerts on the app and Twitter once I had picked up the aurora.

In December of 2015, I applied for permission to use live data feeds from the Tromsų Geophysical Observatory's magnetometer array and, in January 2016, they very kindly granted me access on the condition that their data is not used for commercial gain. This allowed me to fully automate the app and produce the most accurate forecasting and alert system that exists anywhere in the world.

The Iceland version of the app was released on January 17th, 2018, together with beta versions for all north European countries.

Final calibration of the Scandinavia version of the app was completed in April 2019. Some investigative work on a Greenland version of the app was started in the summer of 2019. Initial trials of the app in Canada were done in October 2019 with work on the full app started in November 2020.

Why Bother?

It was clear from the research that I had been conducting for many years here on Skye that accurate, automated aurora alerting and forecasting were achievable, given access to the relevant data feeds. I wanted the technical challenge of proving that it could be done, validating my own research and, ultimately, producing a tool that would give aurora-hunters the best possible chance of catching them.

There were some specific things that I absolutely wanted to avoid: sending alerts during the day and regurgitating data from the SWPC. Everything on the app must be processed to add some value. Only information that is actually needed to predict auroras should be included.

Where does the app get its data?

The app uses live data feeds from (at the time of writing) thirteen magnetometers in Norway, one in Sweden and four in Greenland, together with satellite data from DSCOVR and the SDO.

It also uses the real-time sun/moon position and twilight times which I calculate in the app rather than relying on external sources.

Live sighting reports are submitted by registered users of the app.

What web technologies does it use?

The server-side code is written in PHP 7 using a MySql 5.3 database on a Linux cloud server. The client scripting is entirely jQuery on the Jquery Mobile framework. It employs the very latest internet standards for Progressive Web Apps to provide a native-app-like experience, including off-line capabilities, using the service worker, push, cache, indexDB, share and background sync Javascript APIs.

All client/server communications use Ajax and JSON with minimal packet sizes to ensure optimal performance on 2G and 3G mobile networks.

All features are load-on-demand to keep bandwidth usage to an absolute minimum, whilst still allowing all critical data to refresh at one minute intervals.

Data Collection

Once a minute the app pulls the latest data from the magnetometer array, DSCOVR and the SDO. The satellite data is adjusted for impact time at the Earth and sorted into order of arrival. This gives the most accurate possible picture of what is happening within the Earth's magnetosphere at any given moment. Additional derivative data are also calculated, including clock angle, dynamic pressure and energy transfer into the magnetotail.

The app is automatically monitoring the data feeds for failure and will automatically switch in alternative sources in the event that a feed goes offline.

Data Analysis - Substorms

The first iteration of my substorm tracker used a 2D signature scanner on every magnetometer feed. The signature scanner was analysing deflections on the magnetometer over time, to identify which phase of the substorm we were in. The outputs from the multiple 2D scanners were then combined to determine when the alerts needed to be issued.

2D scanning had many flaws and inefficiencies. The most notable being that it wasn't easy to create a single figure that represented the overall strength of the substorm at any instant. This made it very difficult to compare different substorms to identify which was stronger.

I have since developed a 3-dimensional scanner, which analyses slices across the magnetometer array: deflection vs time vs latitude. Picture the TGO stackplot graph as a landscape with hills and valleys. This algorithm is extraordinarily complex and has taken me over a year to perfect. The outputs it produces, however, are very simple and useful: the strength and the latitude of the substorm... and, therefore, by proxy, the strength and latitude of the aurora.

My 3D substorm tracker is similar to the Ovation Model that you will see on many sites, except that mine is measuring where the actual substorm/aurora is, rather than trying to guess where it might be based on satellite data. Unlike the Ovation Model, mine is taking account of the substorm process.

The latest work I have in hand is an extension to my model which calculates the width of the aurora in degrees of latitude. This allows me to make a fairly accurate estimate of the angle above the northern horizon to which the aurora will extend from any viewing point.

On November 28th, 2020 I began work on a new 4-dimensional substorm tracker which also incorporates longitude. It analyses deflection vs. time vs. latitude vs. longitude. This will provide increased accuracy for Iceland post midnight UTC and also allow 100% accurate alerting for Canada and North America.

At the end of each substorm, the app automatically creates a spreadsheet containing all of the collected data in the 36 hours upto and including the subtorm, together with detailed parameters from my model. This allows me to analyse the performance of the app and models in detail afterwards. It allows adjustments to be made and then 'replayed' against the real data to further refine the model.

Data Analysis - IMF / Solar Wind

Having sorted the satellite data by arrival time and calculated all the derived values, the app scans the IMF and Solar Wind to identify patterns that will cause a substorm to develop. e.g. prolonged periods of negative Bz.

The app also scans for forward and reverse interplanetary shock signatures and will issue an alert within minutes of a shock arriving at the satellites.

Forecasting

My subtorm tracker allows the growth, onset, expansion and recovery phases to be clearly differentiated. This allows usually a few hours advanced warning to be given that a substorm is developing. The IMF and wind analyses allow this to be extended slightly earlier.

The 27-day long-range forecast is usually accurate within 6 hours if the coronal hole stream remains of similar scale. This is in spite of the day/night shift that occurs on each rotation.

All forecast and nowcast information on the app is personalised to the user's location. The complex, generic model data and analyses are calculated once per minute and cached. They are then personalised and adjusted for every user's precise location. Every location in the UK, Europe and Iceland will see their own specific nowcast.

All conditions of endless twilight and the sun never rising/setting are catered for, so that the nowcast can advise when auroras are impossible.

This table shows the average amount of advance warning it gives, in hours, for substorms that go on to reach yellow, orange, red or major levels:

Mean Times From...YellowOrangeRedMajor
Start to Onset2.00 hours2.62 hours3.30 hours4.20 hours
Onset to Peak2.25 hours3.18 hours4.20 hours4.80 hours
Start to Peak4.25 hours5.80 hours7.60 hours9.00 hours

Take, for example, a modest substorm that reaches 'yellow' alert levels only. The app will begin advising that a subtorm is developing 2 hours before it issues the onset alert. Once the onset alert has been issued there will typically be another 2.25 hours before the substorm reaches its peak. This is plenty of time for people to drive to viewpoints should they need to.

The stronger the substorm eventually becomes, the longer it takes to develop, so my app will give much more advance warning that a strong aurora will happen.

Live Reports

Registered users can make live reports by clicking buttons for 'aurora', 'nothing' or 'cloudy'. I stick rigidly to the 'on camera' rule for UK users, as this helps reduce the number of false reports. For latitudes above 63N the app switches to naked eye reporting and forecasting.

The app automatically validates every aurora report using a complex set of criteria. Some reports it will reject immediately and others will be flagged as 'maybe' and displayed with a question mark. The app automatically notifies me of the borderline reports, so that I can approve them manually.

The commonest causes of dubious reports are during twilight start/end and around the moonrise/set, so inexperienced users may have these reports automatically flagged by the app during those times.

The other cause of dubious reports are when there is no, or minimal, geomagnetic activity and again the app will automatically flag up these reports for moderation.

In all of these cases, the app is making a complete assessment of the situation at the time of the report including the sun & moon positions at the user's precise location and the current substorm progression.

Users are graded, using a star-rating system, based on the accuracy and frequency of their reporting. This gives other users of the app confidence that a report can be trusted. These experienced users are given more tolerance in the validation of their reports to ensure that vital edge-case sightings do not get flagged as dubious or rejected.

During the night, the app displays the latest live report from each user. Up until November 2017, it would show the same during the day but there was some bogus coverage on social media that the app was sending red alerts unnecessarily when the map only showed ticks in the north the day after. This was because the users in the south had stayed out until the aurora faded and then posted a cross (nothing). To mitigate against this mistaken perception of poor performance, the map now shows the latest ticks during the day to give a proper summary of the successes during the previous night.

In September 2019, I added an advanced button panel that allows trusted users in the UK to report a little more information about the type of aurora they are getting, e.g. diffuse, rays, arc or overhead. This was for a trial to allow me to collect more data that would allow me to improve the forecasts in future years. If this data proves useful to my research then I will extend it to other countries and possibly allow even more details about the aurora type to be submitted.

Live Photo Uploads

In March 2020, I added the capability for users to upload photos or back-of-camera shots to show other users what they are getting.

Allowing uploads creates a risk that users might abuse the system and upload nasties. To prevent this, I only allow trusted users to upload images. I also developed an image recognition algorithm using artificial intelligence to identify aurora and noctilucent cloud photos. If the AI is confident an image is aurora or NLC then it will be shown immediately to other users on the app. Otherwise, the image is flagged up to me for approval before it is shared out.

Images from my own archive were used to initially train the AI algorithm. User uploads are now also used to train further, so that over time it will continue to improve in accuracy.

To protect user's copyright, I automatically watermark all images with their name upon upload and images are automatically reduced in size, with only watermarked, smaller images shared out on the app. I only hold the user's last uploaded image on the server. All previous uploads are overwritten.

Iceland, Scandinavia, Greenland & Canada

The app has versions for Iceland and all countries in northern Europe. If it has the user's position, it will automatically switch to the correct forecasting engine for their location. For latitudes above 63N the app gives advice based on naked eye and below that it gives advice based on camera.

The work on the Iceland Version began in September 2017 in collaboration with Caroline Weir, an experienced and like-minded aurora-hunter, who is based in Reyjavik. Caroline provided extremely detailed logs of the aurora's elevation, strength and characteristics at different times during substorms. This allowed my existing UK model to be extended to provide incredibly accurate substorm tracking up to latitudes around 67N.

The data from Caroline was also essential in filling in some gaps in the UK model where on a few isolated occasions aurora-hunters were able to detect aurora in the sky ahead of the model.

In December 2018, work began on calibrating the app for higher latitudes in Norway, Sweden and Finland. This was primarily done using all-sky cameras and live webcams, backed up by user reports on the app. By mid January 2019, the app was already producing extremely accurate results for Scandinavia, with minor adjustments and testing continuing until the end of the 2018/2019 season. In the 2019/2020 season, Darren Beardmore provided invaluable assistance with final testing and calibration of the app for the high arctic, whilst working as an aurora guide in Lapland.

In August 2019, Rachel Bibby from the Isle of Lewis became the first person to test a preliminary Greenland version of the app and got the first successful tick from Greenland.

In October 2019, Rebecca Douglas from Kent tested the app against an aurora in Canmore, Canada and showed that the app perfectly tracked the aurora in the sky. She became the first person to report on the app from Canada. Work on the full Canada project started on November 30th, 2020 when I had obtained access to every magnetometer in Greenland and was, therefore, able to 100% accurately track substorms across Canada.

Alerts

Onset alerts are the most difficult alert to issue. I have spent countless hours refining and perfecting these. They are essential for the higher latitudes (>63N), as they are the heads-up that the aurora is coming visible by the naked eye. They are more an advisory for the lower latitudes. The problem is that, when they need issuing, the magnetometers are close to ambient, so it is really important to determine the transition in the substorm phases accurately. It is also important not to spam the users with onset alerts when the magnetometers wobble around background levels before committing themselves.

Up until December 2017, I used dynamic alert levels. Yellow was not always -200 nT, for example. When the moon is stronger, the substorm needs to be stronger. However, when the user is further north moon becomes less of an issue, so I reverted to static alert levels to ensure that higher latitude users received their alerts sooner.

I can take account of all these things based on a clear, dark sky and a camera. It is impossible to take account of light pollution and weather conditions. I set the levels based on what is ordinarily achievable on a camera, in a dark place, in the UK, with clear sky. If the user's view north is over the UK land-mass (e.g Cumbria, Lancashire, the Midlands), then a red alert might not get them an aurora, whereas for someone further south in Norfolk or Lincolnshire, with views over the sea, it could.

The amount of processing required to issue individual alerts for every user's precise location would be astronomical, so I issue the same alerts to all users. I send multiple levels of alert, so if red is not good where the user is, they can wait for the higher level alerts instead.

In the earliest versions of the app I used only three alert levels: yellow, orange and red. These provided basic warnings for all parts of the UK but once the red alert was sent it left nowhere else to go, which meant that when a substorm escalated to more extreme levels users would not get alerted. I later introduced the major, severe and extreme levels that were calibrated at levels when aurora would become intense at UK-equivalent latitudes and had a high chance of going overhead.

Software Development

The Progressive Web App (PWA) technologies were only just becoming available when I started the app. Google were early-adopters and started adding the capabilities to their browsers in summer of 2015. I wanted to create the app as a web app to give complete compatibility across all devices and platforms. Devices that did not yet support PWAs would revert to running the app like a web-site until their vendors had caught up with the new internet standards.

Being novel and state-of-the-art technologies they were buggy and documentation was poor, which meant an endless cycle of code rewrites and workarounds over the first 18 months of the app's development until they had stabilised.

Google were first to support push alerts on their Chrome browser for Android and PC but their initial version did not support payloads, which meant a very convoluted process to get the actual text message of the alert to the device. Next to support push was Firefox, and it supported payloads, but not push without payloads, which meant a complete rewrite of my push code. Then Chrome added push with payload.

Microsoft have committed to providing PWA support and push alerts by late 2018. Apple have a prototype version of Safari with very basic PWA support (service worker and cache APIs only) due for release in spring of 2018. They have not yet committed to implementing push alerts.

I was able to extend the capabilities of the app as new APIs became available, for example, I was able to add offline aurora reporting when Google implemented the background sync API into Chrome.

By 2020, web apps were handled as fully native apps by Android devices and totally indistinguishable from 'store' apps once installed.

Smart Speakers

In February 2020, I developed the first version of the app for the Google Home range of smart speakers. This allowed people to ask for the latest aurora status or the forecast for a specific date. In future releases I would like to allow users to receive alerts and regular updates via their smart speaker.

Users can get a forecast by saying "OK Google, talk to Aurora Alerts UK. Get me the latest forecast or "...Get the the forecast for Tuesday".

Testing

I conduct detailed testing and analysis of the app's performance every night based against the actual aurora that I capture on camera, reports from users and posts on social media. The model is carefully calibrated against the actual aurora in the sky to ensure that the substorm strength figure tracks the aurora strength in real-time. The model has now become so refined that bugs and deviations become very easy to identify.

There are over 13,000 people actively using the app and most evenings there can be over 2000 people permanently logged on every hour. Over 23,000 users have registered. The web server is taking over 170,000 requests an hour during big auroras and over 80,000 on a quiet night.

One of the important things about the app is that it is 'self-testing'. It displays the precise time when the substorm started, peaked and ended, so it is very easy for users to correlate that with their own observations.

There have been over 116,000 reports on the app and, again, it is very easy for people to see the correlation between the substorm progression and the live reports coming in on the app.

Feedback from testers in Iceland and Scandinavia has been that the strength of the aurora in the sky directly correlates with the trendline graph displayed in the substorm panel and the 24-hour substorm strength graph.

There is nowhere for me to hide. I am presenting all of the relevant information on the app that allows users to verify its accuracy themselves.

'Stealth' Auroras

The app works by tracking substorms and is, therefore, 100% accurate at alerting and forecasting auroras that are driven by substorms. However, auroras can occur spontaneously at high latitudes without a substorm occurring. I call these as 'stealth' auroras, as they slip in under the radar without causing any deflections on the magnetometers. These tend to be much weaker, quieter and more diffuse than substorm-driven auroras. The are caused by charged particles entering the atmosphere on the daylight-side of the Earth and at the poles, instead of entering on the dark-side through the substorm process.

They are common at high arctic latitudes, which leads aurora hunters there to believe that all apps are inaccurate. They don't see any obvious relationship between lights in the sky and the scientific data because they are seeing so much of both kinds. At lower latitudes, substorm-driven auroras are crucial to a good show and the accuracy of my substorm tracker becomes very obvious. Even in the high arctic, it is worth using my app, as the substorm-driven auroras are much more powerful and dynamic. You do not want to miss them.

Occasionally, stealth auroras are picked up on camera in the far north of the Scotland and are very low and weak. These are often used as 'evidence' that Glendale App does not work by detractors. However, detailed analysis with accurate timings often shows that the aurora came very close to the beginning or end of a substorm and there was a small lead or lag in the magnetometers detecting the activity.

True stealth auroras picked up from UK equivalent latitudes only occur a few times a year and represent only a tiny percentage of auroras recorded. They are certainly not a worthy reason to advise people against using Glendale App because they might miss one. It is also difficult to justify advising people to ignore my app and stand outside, all night, every night, on the off-chance of catching a very austere show that it might not detect. My app will not fail to detect any worthy show.

Research

Over six years of research and development has gone into the app. I spent the years from 2013 through to 2015 photographing the night sky at intervals of 5 to 20 minutes. The stages of the aurora were then correlated against magnetometer and satellite data. At the time, this involved using a ruler to measure the deflections on the Stackplot graphs and calculating the magnitude from the scales.

I analysed precisely which data sources and signatures were needed to produce visible auroras here on the Isle of Skye. I also looked in detail at the effects that twilight and moon had on the levels at which auroras became visible. I worked with photographers around the UK to determine how the various latitudes affected the visibility of the aurora compared to Skye.

The accuracy of the substorm tracker and my automated data recording & analysis systems allow me to test, behind the scenes, many of the common myths and theories about auroras. I program the app to automatically collect and analyse data that are needed to validate or disprove them.

One example of this is determining how the alignment and strength of the IMF during the growth phase of the subtorm affects the peak strength in the expansion phase.

When I can prove a theory, I will the incorporate the results of my research into the app.

I came to this with no knowledge whatsoever, so my mind was completely open. It quickly became clear that the official scientific bodies and universities had made a few wrong turns along the way and put themselves in a cul-de-sac:

The principal official scientific indexes used to measure activity, such as K or Kp, are 'longitudinal' measures. They take an average reading from magnetometers dotted around the globe. What you actually need to predict auroras is to work laterally, using magnetometers that are dotted latitudinally.

Many scientists believe that auroral activity is best measured using the magnetometer is closest to the observer. When in fact, you need to use the magnetometer that is directly underneath the aurora to the north of the observer, typically around 10 degrees north of the observer in the UK.

Another mistake is believing that being closer to the auroral oval, e.g. in Lapland, makes it easier to understand the aurora. Yet to gain a proper understanding you actually need to stand-back and view the aurora from a distance, like the UK.

Competitors

It took almost five years for another developer to clone features of my work into another app. In September 2020 another app began using Scandinavian magnetometer deflections to provide crude 'aurora strength' indications, nowcasts and long-range forecasts. This was done by a French developer, who was living in Iceland. It also included aurora reporting with sightings recorded on a map, as Glendale App has done since 2015. The similarity of the nowcast messages and supplementary information suggested that it was being calibrated against my app rather than the actual aurora in the sky.

Funding

The app is entirely self-funded. I pay for all server costs to make it free for everybody to use. The app carries no adverts and no SPAM is sent. No revenue is generated whatsoever. I freely give all my spare time to auroral substorm research, development of the app and technical support for users.

Andy Stables
Originally written on December 19th, 2017. Updated on October 7th, 2020.