Ever wondered how the Glendale App works? How does it achieve such unrivalled accuracy? How can a teuchter on Skye claim to have created the world's only 100% accurate aurora alert app?
I am going to try to explain a little about what goes on behind the scenes and what it is actually doing.
The first version of the app was released in November of 2015. It included the live reports, IMF and Solar Wind panels. All alerts had to be done manually. I would take test photos regularly and then manually send out push alerts on the app and Twitter once I had picked up the aurora.
In December of 2015, I applied for permission to use live data feeds from the Tromsø Geophysical Observatory's magnetometer array and, in January 2016, they very kindly granted me access on the condition that their data is not used for commercial gain. This allowed me to fully automate the app and produce the most accurate forecasting and alert system that exists anywhere in the world.
The Iceland version of the app was released on January 17th, 2018, together with beta versions for all north European countries.
Final calibration of the Scandinavia version of the app was completed in April 2019. Some investigative work on a Greenland version of the app was started in the summer of 2019.
Initial trials of the app in Canada were done in October 2019 with work on the full app started in November 2020 and completed on December 14th, 2020.
A preliminary beta version of the app for Australia & New Zealand was released secretly on January 1st, 2021.
It was clear from the research that I had been conducting for many years here on Skye that accurate, automated aurora alerting and forecasting were achievable, given access to the relevant data feeds. I wanted the technical challenge of proving that it could be done, validating my own research and, ultimately, producing a tool that would give aurora-hunters the best possible chance of catching them.
There were some specific things that I absolutely wanted to avoid: sending alerts during the day and regurgitating data from the SWPC. Everything on the app must be processed to add some value. Only information that is actually needed to predict auroras should be included.
The app uses live data feeds from (at the time of writing) 31 magnetometers: fourteen in Norway, one in Sweden, five in Finland, five in Greenland, six in Canada and two in Alaska, together with satellite data from DSCOVR, ACE, the SDO and SOHO.
It also uses the real-time sun/moon position and twilight times which I calculate in the app rather than relying on external sources.
Live sighting reports are submitted by registered users of the app.
The server-side code is written in PHP 7 using a MySql MariaDB 10.5 database on a Linux cloud server. The client scripting is entirely jQuery on the Jquery Mobile framework. It employs the very latest internet standards for Progressive Web Apps to provide a native-app-like experience, including off-line capabilities, using the service worker, push, cache, indexDB, share and background sync Javascript APIs.
All client/server communications use Ajax and JSON with minimal packet sizes to ensure optimal performance on 2G and 3G mobile networks.
All features are load-on-demand to keep bandwidth usage to an absolute minimum, whilst still allowing all critical data to refresh at up to one minute intervals.
Once a minute the app pulls the latest data from the magnetometer arrays, DSCOVR, ACE, the SDO and SOHO. The satellite data is adjusted for impact time at the Earth and sorted into order of arrival. This gives the most accurate possible picture of what is happening within the Earth's magnetosphere at any given moment. Additional derivative data are also calculated, including clock angle, dynamic pressure and energy transfer into the magnetotail.
The app is continually monitoring the data feeds for failure and will automatically switch in alternative sources in the event that a feed goes offline.
Until March 2021, all data panels on the app were updated at one minute intervals for all users. However, escalating load on the web server made this unsustainable. Users now receive an update rate that varies dependent on their trust rating and whether they contribute by making reports.
The first iteration of my substorm tracker used a 2D signature scanner on every magnetometer feed. The signature scanner was analysing deflections on the magnetometer over time, to identify which phase of the substorm we were in. The outputs from the multiple 2D scanners were then combined to determine when the alerts needed to be issued.
2D scanning had many flaws and inefficiencies. The most notable being that it wasn't easy to create a single figure that represented the overall strength of the substorm at any instant. This made it very difficult to compare different substorms to identify which was stronger.
I have since developed a 3-dimensional scanner, which analyses slices across the magnetometer array: deflection vs time vs latitude. Picture the TGO stackplot graph as a landscape with hills and valleys. This algorithm is extraordinarily complex and has taken me over a year to perfect. The outputs it produces, however, are very simple and useful: the strength and the latitude of the substorm... and, therefore, by proxy, the strength and latitude of the aurora.
My 3D substorm tracker is similar to the Ovation Model that you will see on many sites, except that mine is measuring where the actual substorm/aurora is, rather than trying to guess where it might be based on satellite data. Unlike the Ovation Model, mine is taking account of the substorm process.
An extension to my model which calculates the width of the aurora in degrees of latitude was beta-released on October 25th, 2021. This allows me to make a fairly accurate estimate of the angle above the northern horizon to which the aurora will extend from any viewing point. By October 2021, I had finally worked out how to determine the upper and lower latitudes of the actual aurora in real-time and was able to show pictorially on the maps precisely where the aurora was sitting.
On November 28th, 2020 I began work on a new 4-dimensional substorm tracker which also incorporates longitude. It analyses deflection vs. time vs. latitude vs. longitude. This provided increased accuracy for Iceland post midnight UTC and also allowed 100% accurate alerting for Canada, Alaska and Australasia. This was completed on December 23rd, 2020.
In September 2021, I converted the 4D tracker to work entirely on the geomagnetic coordinate system and introduced 5 new magnetometers at strategic locations across Canada.
My 4D substorm tracker has been carefully designed to allow any magnetometer, anywhere in the world, to be added or removed at any time without degrading the output. This is especially important when magnetometers go offline temporarily. If any magnetometers go offline, others will automatically plug the gap, so that the algorithm continues to track the substorm perfectly.
At the end of each substorm, the app automatically creates a spreadsheet containing all of the collected data in the 36 hours upto and including the substorm, together with detailed parameters from my model. This allows me to analyse the performance of the app and models in detail afterwards. It allows adjustments to be made and then 'replayed' against the real data to further refine the model.
Having sorted the satellite data by arrival time and calculated all the derived values, the app scans the IMF and Solar Wind to identify patterns that will cause a substorm to develop. e.g. prolonged periods of negative Bz.
The app also scans for forward and reverse interplanetary shock signatures and will issue an alert within minutes of a shock arriving at the satellites.
My substorm tracker allows the growth, onset, expansion and recovery phases to be clearly differentiated. This allows usually a few hours advanced warning to be given that a substorm is developing. The IMF and wind analyses allow this to be extended slightly earlier.
The 27-day long-range forecast is usually accurate within 6 hours if the coronal hole stream remains of similar scale. This is in spite of the day/night shift that occurs on each rotation.
All forecast and nowcast information on the app is personalised to the user's location. The complex, generic model data and analyses are calculated once per minute and cached. They are then personalised and adjusted for every user's precise location. Every location in the world will see their own specific nowcast.
All conditions of endless twilight and the sun never rising/setting are catered for, so that the nowcast can advise when auroras are impossible.
This table shows the average amount of advance warning it gives, in hours, for substorms that go on to reach yellow, orange, red or major levels:
| Mean Times From... | Yellow | Orange | Red | Major |
|---|---|---|---|---|
| Start to Onset | 2.00 hours | 2.62 hours | 3.30 hours | 4.20 hours |
| Onset to Peak | 2.25 hours | 3.18 hours | 4.20 hours | 4.80 hours |
| Start to Peak | 4.25 hours | 5.80 hours | 7.60 hours | 9.00 hours |
Take, for example, a modest substorm that reaches 'yellow' alert levels only. The app will begin advising that a substorm is developing 2 hours before it issues the onset alert. Once the onset alert has been issued there will typically be another 2.25 hours before the substorm reaches its peak. This is plenty of time for people to drive to viewpoints should they need to.
The stronger the substorm eventually becomes, the longer it takes to develop, so my app will give much more advance warning that a strong aurora will happen.
From 2019 until 2021, I studied in detail the conditions that cause the magnetotail to become unstable and trigger a drop (substorm expansion). I programmed the app to scan for these and in January 2022 was able to extend the nowcast to give very accurate times when the next burst of activity would occur.
I also studied in detail the different convections patterns by which energy from the solar wind is transferred into the magnetosphere to predict when enough energy had been transferred to trigger a drop on the magnetometers.
My work in these areas is so advanced that I am able to give specific times when the substorm expansions are likely to occur. This is a ground-breaking advance in aurora prediction. Nobody else had ever attempted to put a specific time on when the drops will happen.
My substorm tracker is so accurate that I am able to automatically test the predicted drop times against the tracker data to measure its performance and verify its accuracy. I can also test other suspected patterns against the data to check if they are correct.
My drop prediction times are dependent on the type of pattern. The amount of advance warning varies because sometimes it is necessary to wait for more data to be certain that a drop will happen.
The first official release of my drop predictor was on October 27th, 2022 but many pattern types had their predictions piloted in beta versions throughout 2022.
I have tested the drop-time prediction accuracy since the feature was released. In 95% of cases, the predicted time will be before the peak of the drop by an average of 16 minutes. A very satisfactory result and putting yourself in place at the predicted time will give you a very high chance of success. Even in the 5% of case when it gets it wrong, it will be predicting a time that is only 17 minutes after it actually drops, on average.
The predicted drop times are shown on the app between 30 and 100 minutes in advance.
Not all drops are predictable. In cases where the Bz remains negative for many hours, it becomes impossible to estimate within +/- 1 hour of when a drop might occur but my work in this field is continuing to try to improve that.
Registered users can make live reports by clicking buttons for 'aurora', 'nothing' or 'cloudy'. I stick rigidly to the 'on camera' rule for UK users, as this helps reduce the number of false reports. For latitudes above 63N the app switches to naked eye reporting and forecasting.
The app automatically validates every aurora report using a complex set of criteria. Some reports it will reject immediately and others will be flagged as 'maybe' and displayed with a question mark. The app automatically notifies me of the borderline reports, so that I can approve them manually.
The commonest causes of dubious reports are during twilight start/end and around the moonrise/set, so inexperienced users may have these reports automatically flagged by the app during those times.
The other cause of dubious reports are when there is no, or minimal, geomagnetic activity and again the app will automatically flag up these reports for moderation.
In all of these cases, the app is making a complete assessment of the situation at the time of the report including the sun & moon positions at the user's precise location and the current substorm progression.
Users are graded, using a star-rating system, based on the accuracy and frequency of their reporting. This gives other users of the app confidence that a report can be trusted. These experienced users are given more tolerance in the validation of their reports to ensure that vital edge-case sightings do not get flagged as dubious or rejected.
During the night, the app displays the latest live report from each user. Up until November 2017, it would show the same during the day but there was some bogus coverage on social media that the app was sending red alerts unnecessarily when the map only showed ticks in the north the day after. This was because the users in the south had stayed out until the aurora faded and then posted a cross (nothing). To mitigate against this mistaken perception of poor performance, the map now shows the latest ticks during the day to give a proper summary of the successes during the previous night.
In September 2019, I added an advanced button panel that allows trusted users in the UK to report a little more information about the type of aurora they are getting, e.g. diffuse, rays, arc or overhead. This was for a trial to allow me to collect more data that would allow me to improve the forecasts in future years. This proved useful to my research, so in November 2021 I extended it to other countries.
In July 2022, I added a feature for reporting the height in degrees of the aurora, noctilucent clouds, rays, arcs, etc. This was essentially for my own research records but I made it available to any trusted user that wanted to use it.
In March 2020, I added the capability for users to upload photos or back-of-camera shots to show other users what they are getting.
Allowing uploads creates a risk that users might abuse the system and upload nasties. To prevent this, I only allow trusted users to upload images. I also developed an image recognition algorithm using artificial intelligence to identify aurora and noctilucent cloud photos. If the AI is confident an image is aurora or NLC then it will be shown immediately to other users on the app. Otherwise, the image is flagged up to me for approval before it is shared out.
Images from my own archive were used to initially train the AI algorithm. User uploads are now also used to train further, so that over time it will continue to improve in accuracy.
To protect user's copyright, I automatically watermark all images with their name upon upload and images are automatically reduced in size, with only watermarked, smaller images shared out on the app. I only hold the user's last uploaded image on the server. All previous uploads are overwritten.
The app has versions for Iceland, all countries in northern Europe, Canada, US, Australia & New Zealand. If it has the user's position, it will automatically switch to the correct forecasting engine for their location. For latitudes above 63N the app gives advice based on naked eye and below that it gives advice based on camera.
The work on the Iceland Version began in September 2017 in collaboration with Caroline Weir, an experienced and like-minded aurora-hunter, who is based in Reyjavik. Caroline provided extremely detailed logs of the aurora's elevation, strength and characteristics at different times during substorms. This allowed my existing UK model to be extended to provide incredibly accurate substorm tracking up to latitudes around 67N.
The data from Caroline was also essential in filling in some gaps in the UK model where on a few isolated occasions aurora-hunters were able to detect aurora in the sky ahead of the model.
In December 2018, work began on calibrating the app for higher latitudes in Norway, Sweden and Finland. This was primarily done using all-sky cameras and live webcams, backed up by user reports on the app. By mid January 2019, the app was already producing extremely accurate results for Scandinavia, with minor adjustments and testing continuing until the end of the 2018/2019 season. In the 2019/2020 season, Darren Beardmore provided invaluable assistance with final testing and calibration of the app for the high arctic, whilst working as an aurora guide in Lapland.
In August 2019, Rachel Bibby from the Isle of Lewis became the first person to test a preliminary Greenland version of the app and got the first successful tick from Greenland.
In October 2019, Rebecca Douglas from Kent tested the app against an aurora in Canmore, Canada and showed that the app perfectly tracked the aurora in the sky. She became the first person to report on the app from Canada.
Work on the full Canada project started on November 30th, 2020 when I had obtained access to every magnetometer in Greenland and was, therefore, able to 100% accurately track substorms across Canada.
Work on the Canada version of the app was completed on December 14th, 2020. It required significant rework to the app to handle 8 timezones across Canada, build the 4D substorm tracker and create a completely independent alerting system to allow them to receive alerts during what would be daytime in Europe.
Work on the Australia & New Zealand version of the app started on January 1st, 2021 and Olivia Williams from NSW became the first person to report aurora on the app from Australia on February 7th, 2021. The first official release for Australasia was on February 22nd, 2021.
Onset alerts are the most difficult alert to issue. I have spent countless hours refining and perfecting these. They are essential for the higher latitudes (>63N), as they are the heads-up that the aurora is coming visible by the naked eye. They are more an advisory for the lower latitudes. The problem is that, when they need issuing, the magnetometers are close to ambient, so it is really important to determine the transition in the substorm phases accurately. It is also important not to spam the users with onset alerts when the magnetometers wobble around background levels before committing themselves.
Up until December 2017, I used dynamic alert levels. Yellow was not always -200 nT, for example. When the moon is stronger, the substorm needs to be stronger. However, when the user is further north moon becomes less of an issue, so I reverted to static alert levels to ensure that higher latitude users received their alerts sooner.
I can take account of all these things based on a clear, dark sky and a camera. It is impossible to take account of light pollution and weather conditions. I set the levels based on what is ordinarily achievable on a camera, in a dark place, in the UK, with clear sky. If the user's view north is over the UK land-mass (e.g Cumbria, Lancashire, the Midlands), then a red alert might not get them an aurora, whereas for someone further south in Norfolk or Lincolnshire, with views over the sea, it could.
The amount of processing required to issue individual alerts for every user's precise location would be astronomical, so I issue the same alerts to all users. I send multiple levels of alert, so if red is not good where the user is, they can wait for the higher level alerts instead.
In the earliest versions of the app I used only three alert levels: yellow, orange and red. These provided basic warnings for all parts of the UK but once the red alert was sent it left nowhere else to go, which meant that when a substorm escalated to more extreme levels users would not get alerted. I later introduced the major, severe and extreme levels that were calibrated at levels when aurora would become intense at UK-equivalent latitudes and had a high chance of going overhead.
The Coronal Holes panel was added to the app in the autumn of 2020. A few other sites had made simple forays into scanning solar imagery for coronal holes but I wanted to make something much more advanced, yet very simple to use. I find the holes, mark them with a box on the imagery and then automatically calculate when the wind from them will be geoeffective at Earth. I log all the holes found and can then automatically compare the hole with it's size on its prior rotation to notify users not only when it will be geoeffective but also whether it has grown or shrunk in size.
The CME detector was added to the app in May 2021 and work is still continuing on this. I wanted to automatically scan the LASCO imagery to find coronal mass ejections following a flare or filament eruption. Again, this had been tackled before, by the CACTUS team, but I wanted to go a bit further by automatically identifying the CME and highlighting its outline graphically. Once, the outline can be 'lassoed' then it is easy to accurately estimate the ejection velocity of the plasma. I also wanted to clean up the images and boost the CME to make it easier to spot halos on fainter ones.
I devoted six full weeks to version 2 of my CME detector around July of 2023. This made significant improvements in reduction/removal of streamers to prevent them from mistakenly being confused with CMEs. The identification of fainter CMEs was improved enormously. This made the 'lassoing' much better at outlining the true location of the CME. The algorithm was almost completely re-written, which will make it easier to tweak the detection logic in future.
In the spring of 2018, I added features to the app to help people to see Noctilucent Clouds and report their sightings. I send alerts to users when I have confidence that reliable reports are being made on the app. The app also shows sun elevation/azimuth angles and illumination angles for noc and cirrus clouds.
I have researched, through many years of my own observations, and those reported by other app users, the optimum times to view noctilucent clouds at any given location.
In spring of 2021, I became the first person in the world to discover that the optimum time for viewing noctilucent clouds is directly linked to a specific elevation angle of the sun at any given latitude. I call this 'Stables Law of Noctilucent Clouds'. The optimum sun elevation angle for viewing nocs is a linear function:
optimum elevation angle = a * latitude - b.
I also showed that 68% of reports occur within +/-1.5° of the optimum elevation angle. I was able to use this to calculate precise times that users need to be out looking for nocs. I was also able to flag up reports that were outside the expected sun elevation angle range for investigation.
By July 2022, I had proven beyond all reasonable doubt that my Stables Law was valid and accurate. During the great noc nights, when the app was receiving reports from 50+ users across the UK and Europe, all of the reports were made when the sun was precisely within the predicted elevation angles calculated by Stables Law. I have analysed over 3600 reports from latitudes 61°N down to 47°N.
All other NLC sites still describe nocs as being randomly visible when the sun is anywhere between -6 and -17 degrees below the horizon. It was a major advancement in the science for me to be able to prove that this was incorrect and instead identify a very narrow 3 degree band at each latitude when the nocs would be visible if they were present.
Glendale App was the first app to provide noctilucent cloud alerts based on user sightings and the first to allow users to report their sightings. Prior to 2018, there was only one other NLC app but it simply provided an alert when the sun reached -6 degrees below the horizon and provided no other advice.
Nocs remain totally unpredictable. You have to go look at the optimum times shown by my app and, if they are present, you will see them. A lot of credence is given by NLC social media sites to radar echoes but I have looked at these extensively and found no correlation whatsoever with actual sighting reports.
In June 2023, I finally cracked a formula that allows me to predict which part of the sky the nocs are most likely to be seen in, how bright they might be and how far above the horizon they could reach. This allowed me to start work on a more accurate model of noctilucent cloud illumination patterns. I was able to add a graphic to the app that illustrates which part of the sky nocs are most likely to appear in and how bright they will appear relative to the background sky.
The Progressive Web App (PWA) technologies were only just becoming available when I started the app. Google were early-adopters and started adding the capabilities to their browsers in summer of 2015. I wanted to create the app as a web app to give complete compatibility across all devices and platforms. Devices that did not yet support PWAs would revert to running the app like a website until their vendors had caught up with the new internet standards.
Being novel and state-of-the-art technologies they were buggy and documentation was poor, which meant an endless cycle of code rewrites and workarounds over the first 18 months of the app's development until they had stabilised.
Google were first to support push alerts on their Chrome browser for Android and PC but their initial version did not support payloads, which meant a very convoluted process to get the actual text message of the alert to the device. Next to support push was Firefox, and it supported payloads, but not push without payloads, which meant a complete rewrite of my push code. Then Chrome added push with payload.
Microsoft have committed to providing PWA support and push alerts by late 2018 but they weren't working in the app until February 2021. Apple have a prototype version of Safari with very basic PWA support (service worker and cache APIs only) due for release in spring of 2018. They have not yet committed to implementing push alerts.
I was able to extend the capabilities of the app as new APIs became available, for example, I was able to add offline aurora reporting when Google implemented the background sync API into Chrome.
By 2020, web apps were handled as fully native apps by Android devices and totally indistinguishable from 'store' apps once installed.
In February 2020, I developed the first version of the app for the Google Home range of smart speakers. This allowed people to ask for the latest aurora status or the forecast for a specific date. In future releases I would like to allow users to receive alerts and regular updates via their smart speaker.
Users can get a forecast by saying "OK Google, talk to Aurora Alerts UK. Get me the latest forecast or "...Get the the forecast for Tuesday".
I conduct detailed testing and analysis of the app's performance every night based against the actual aurora that I capture on camera, reports from users and posts on social media. The model is carefully calibrated against the actual aurora in the sky to ensure that the substorm strength figure tracks the aurora strength in real-time. The model has now become so refined that bugs and deviations become very easy to identify.
There are over 50,000 people actively using the app and most evenings there can be over 2000 people permanently logged on every hour. Over 70,000 users have registered. The web server is taking over 300,000 requests an hour during big auroras and over 180,000 on a quiet night.
One of the important things about the app is that it is 'self-testing'. It tests itself against both user reports and collected data. It displays the precise time when the substorm started, peaked and ended, so it is very easy for users to correlate that with their own observations.
There have been over 380,000 reports on the app and, again, it is very easy for people to see the correlation between the substorm progression and the live reports coming in on the app.
Feedback from testers in Iceland, Scandinavia, Canada and Australasia has been that the strength of the aurora in the sky directly correlates with the trendline graph displayed in the substorm panel and the 24-hour substorm strength graph.
There is nowhere for me to hide. I am presenting all of the relevant information on the app that allows users to verify its accuracy themselves.
In February of 2023 there was a major aurora that attracted a huge amount of press coverage. This resulted in a massive increase in sign-ups on the app. The server was pushed to its limits on a few nights in the weeks that followed. A lot of snide comments about the app going off when it is needed most, on social media, ensued.
The problem with an aurora app is that it induces a denial-of-service attack on itself whenever it issues an alert. The app can be running perfectly fine, with a thousand people logged on and then suddenly a red alert gets triggered. Thousands of people receive the alert and click into the app, at the same time, creating a massive surge in traffic that stops the app in its tracks.
When this happens, I don't have many options. If I leave the app running, it will die for everyone and be unable to send alerts.
What I do is restrict access to newer users, so that the regulars continue to receive the normal service as usual. The alerts are still sent to everyone, so those who were clever enough to follow my alerts on android, Telegram and Twitter will still get them even when the app appears 'down'.
With the server overloading coming close to the end of season in the northern hemisphere, it didn't make sense to upgrade my server in the spring, as it would be idling and costing me a lot of money for the whole summer. My plan was to upgrade the server in September, doubling my monthly running costs.
Even doubling the capacity of the server will not prevent overloading on the 'big' nights. On major nights, the traffic increases by 10+ times what is normal. A server powerful enough to handle every surge would cost thousands of pounds a month.
I regularly get computer "experts" telling me that I need to move my app into Azure or Amazon so that the server can automatically upscale to meet high demand. What these people don't realise is that the basic cost of these services for a 'quiet' night on the app is ten times what I currently pay. On a heavy night, were I to allow them to upscale to meet demand, I would rack up four-figure costs in a single night.
The other thing I get told a lot is that 'I would be willing to pay a modest subscription to support a bigger server'. Even if I were allowed to do this, the consequence would be that 90% of users would quit the app and the benefits of high-volume user reporting would be lost. It would die the death of many other online platforms. Even if they were prepared to pay, I would then have to deal with people demanding refunds every time the server had a hiccup or they missed an alert.
Another option regularly proposed is a 'go fund me' page. This still leaves a problem that the server will have hiccups. Anyone that donated will feel that they were cheated because they contributed to a bigger server and it still failed on the 'big' event.
I've had people complaining that it was my fault they missed the last big aurora because my app went down and they never received the alert. Yet the app didn't go down, it was just restricted access. All the alerts were sent as normal. Upon probing, I find that they aren't subscribed to my alerts on Telegram or Twitter ("I don't do Telegram or Twitter"). I am just the target of their anger at missing out. They have to help me, to help them, by signing up for the alerts on Telegram.
I spent every spare minute of my time in March and April finding innovative ways to make the code more efficient, to allow the app to take more traffic before it needed intervention. I cut the processor usage significantly so that when it did need to have restricted access it was only for half an hour at the peak on the nights when the app was being hammered.
It is ironic that the app often gets hammered the most on the night after a big aurora, when nothing is happening. This is because the people who missed it sign up on my app in the expectation of a repeat performance.
The server was upgraded on September 12th, 2023 to 4 vCores and 8 GB of RAM. I then spent the next week converting all of the PHP 7.4 code to be compatible with PHP 8.2 and the PHP processor was switched onto the latest version at mid-day on September 18th, 2023.
The app works by tracking substorms and is, therefore, 100% accurate at alerting and forecasting auroras that are driven by substorms. However, auroras can occur spontaneously at high latitudes without a substorm occurring. I call these as 'stealth' auroras, as they slip in under the radar without causing any deflections on the magnetometers. These tend to be much weaker, quieter and more diffuse than substorm-driven auroras. They are caused by charged particles entering the atmosphere on the daylight-side of the Earth and at the poles, instead of entering on the dark-side through the substorm process.
They are common at high arctic latitudes, which leads aurora hunters there to mistakenly believe that all apps are inaccurate. They don't see any obvious relationship between lights in the sky and the scientific data because they are seeing so much of both kinds. At lower latitudes, substorm-driven auroras are crucial to a good show and the accuracy of my substorm tracker becomes very obvious. Even in the high arctic, it is worth using my app, as the substorm-driven auroras are much more powerful and dynamic. You do not want to miss them.
Occasionally, stealth auroras are picked up on camera in the far north of the Scotland and are very low and weak. These are often used as 'evidence' that Glendale App does not work by detractors. However, detailed analysis with accurate timings often shows that the aurora came very close to the beginning or end of a substorm and there was a small lead or lag in the magnetometers detecting the activity.
True stealth auroras picked up from UK equivalent latitudes only occur a few times a year and represent only a tiny percentage of auroras recorded. They are certainly not a worthy reason to advise people against using Glendale App because they might miss one. It is also difficult to justify advising people to ignore my app and stand outside, all night, every night, on the off-chance of catching a very austere show that it might not detect. My app will not fail to detect any worthy show.
When I talk about 'research', I do not mean reading papers and theories written by other people. I mean back-to-basics, analysing raw data and studying the actual aurora/NLC in the sky myself to produce contemporary, ground-breaking understanding of how these things actually work in the real world.
Over 10 years of research and development has gone into the app. I spent the years from 2013 through to 2015 photographing the night sky at intervals of 5 to 20 minutes. The stages of the aurora were then correlated against magnetometer and satellite data. At the time, this involved using a ruler to measure the deflections on the Stackplot graphs and calculating the magnitude from the scales.
I analysed precisely which data sources and signatures were needed to produce visible auroras here on the Isle of Skye. I also looked in detail at the effects that twilight and moon had on the levels at which auroras became visible. I worked with photographers around the UK to determine how the various latitudes affected the visibility of the aurora compared to Skye.
The accuracy of the substorm tracker and my automated data recording & analysis systems allow me to test, behind the scenes, many of the common myths and theories about auroras. I program the app to automatically collect and analyse data that are needed to validate or disprove them.
I came to this with no knowledge whatsoever, so my mind was completely open. It quickly became clear that the official scientific bodies and universities had made a few wrong turns along the way and put themselves in a cul-de-sac:
The principal official scientific indexes used to measure activity, such as K or Kp, are 'longitudinal' measures. They take an average reading from magnetometers dotted around the globe. What you actually need to predict auroras is to work laterally, using magnetometers that are dotted latitudinally.
Many scientists believe that auroral activity is best measured using the magnetometer is closest to the observer. When in fact, you need to use the magnetometer that is directly underneath the aurora to the north of the observer, typically around 10 degrees north of the observer in the UK.
Another mistake is believing that being closer to the auroral oval, e.g. in Lapland, makes it easier to understand the aurora. Yet to gain a proper understanding you actually need to stand-back and view the aurora from a distance, like the UK.
I "discovered" substorm phases, the substorm current wedge, electrojets, the Russell-McPherron effect, steady magnetospheric convection, et al, from first principles, without ever having heard of them, simply by studying the night sky, magnetograms and satellite data feeds. It is only through searching online for information about the patterns I was observing that I found the scientific papers describing them.
My current focus of research is detecting patterns that cause magnetospheric instability and using them to predict when the drops will occur on the magnetograms.
Many people mistakenly assume that I am an expert physicist but nothing could be further from the truth. I am a computer scientist that has acquired a little knowledge of the physics along the way. I analyse the vast strings of data being churned out by scientific instruments measuring magnetic fields and compare them against aurora in the sky. These strings of data are encrypted messages telling us what the aurora is doing, like you chatting to your mum on WhatsApp about your day. The aurora in the sky is the decoded message. What I try to do is the find the cipher that decrypts the hidden messages in the scientific data that tell me what the aurora is doing. Looking for patterns, sorting, unjumbling, filtering, shifting, cleaning, combining, merging. Like an IQ test such as the quiz show 'Only Connect', hosted by the wonderful Victoria Coren Mitchell, where we are given clues and have to find the fourth item in the sequence. I am trying to find the pattern in the clues that tells us what the outcome will be.
I should explain that I do not see myself as being in competition with anybody else. What I do is unique and has never been done before. It costs me money to do this. The more people use the app, the more it costs me, so I have no incentive to 'sell' more copies or sign up more people. It is others that see me as a rival. However, more people reporting their sightings on the app helps me to build up a solid body of evidence of its accuracy and helps me to train the app to give more accurate forecasts at any specific location. What I seek are 'quality' contributors, who make accurate, reliable reports.
The 'market-leader' is AuroraWatch UK (AWUK), who have two apps built using their simple API. They use a single magnetometer that is located in Shetland, which isn't really far enough north to detect most auroral activity. Their alerts are very often retrospective, often being issued 10 minutes to 1 hour after the professional magnetometers at the same latitude have recovered. This became particularly prevalent after an incident whereby Glendale App had alerted of a major aurora, visible over large parts of the UK, but AWUK never alerted and this was picked up by the press the following day. Since then the AWUK data often appears to be manipulated after Glendale App issues higher level alerts to induce their apps to also send out alerts. This leaves a lot of people scratching their heads because their apps show 'no significant activity' throughout the substorm and then after the magnetometers have recovered their data shows a sudden large spike of activity that triggers alerts. Their API and 24 hour charts latch any momentary activity against a whole hour of the day to obfuscate the precise time at which supposed activity was detected.
It took almost five years for another developer to clone features of my work into another app. In September 2020 another app began using Scandinavian magnetometer deflections to provide crude 'aurora strength' indications, nowcasts and long-range forecasts. It also included aurora reporting with sightings recorded on a map, as Glendale App has done since 2015. The similarity of the nowcast messages, supplementary information and the reproduction of mountweazels show that it was being copied from my app rather than calibrated against the actual aurora in the sky.
A number of people began contacting me during 2021 asking about the internal workings of my substorm tracker algorithm. One wanted to sell the secrets through online tutorials and another wanted to replicate the algorithm in their own paid aurora app in order to sell more copies.
The problem with clones is that their aim is not advancing the science. They are just trying to copy from Glendale App in order to make gain for themselves. Once the other apps have reverse-engineered Glendale App and taken all the users, how will the science be moved forward? It is only through my continuing efforts to advance the science of aurora-forecasting that we have come so far. The rogues that make cheap Rolexes, aren't trying to advance the craft of watch-making, they just want to make a fast buck by imitating quality craftmanship.
In July 2022, another developer formally announced that a team were intending to clone noctilucent cloud features from Glendale App into a 'dedicated' noctilucent cloud app that would be launched prior to the 2023 noc season.
It has been a constant struggle to get people to try the app for themselves. I won't pay to advertise the app, as it is purely a pro bono project. I rely entirely on word of mouth.
Social media would be the obvious way to spread the word. However, many facebook group admins somehow feel threatened by the app.
The admins of a UK aurora facebook group started a campaign against the app as soon as it was launched. They initially claimed that it was 'unscientific' and unreliable. These claims were quickly refuted by their members. Their next tactic, which still continues at the time of writing, is to hide or delete any comment that mentions Glendale App. This is quite entertaining because a member will ask "what's the best app to use for aurora-hunting". Many kindly folk will recommend Glendale App and a few will recommend their own favourites. All comments that recommend Glendale App are swiftly hidden or deleted by the admins, leaving behind only those suggesting other apps or to 'follow this page'. Facebook shows say 14 comments on the post but only 2 are visible to members because the admins have hidden them. In late 2021, they issued specific rules to members that Glendale App was not to be mentioned by name. This was after a few very astute ladies had spotted that every time Glendale App was mentioned the comments disappeared and spoke out about it publicly on the group. This led to some very comical discussions on the group about 'the app whose name cannot be spoken'.
It is difficult to comprehend what would motivate a Scottish aurora group to actively seek to prevent their members from knowing about a Scottish aurora app, that is based on solid scientific research carried out in Scotland and generally acknowledged to be the most accurate in the world. Many people spend hundreds, even thousands, of pounds travelling to Scotland, Iceland or Scandinavia in the hope of catching an aurora and it is heart-breaking that people would wilfully direct them to apps they know don't work purely out of spite.
Another UK facebook aurora page has a policy of actively advising followers against Glendale App on the grounds that the Aurora Watch UK app is their recommended app for alerts. This is in spite of the fact that Glendale App always alerts... and always alerts ahead of the AWUK app, giving users more advance warning of substorm expansion.
I was banned from a Canadian facebook group after asking an admin to try my app because he did not want anyone to know that auroras were predictable. He preferred them to be a 'surprise' and didn't want anybody to know when substorms were happening.
Another UK group admin told me that he would never recommend Glendale App, and would always remove any comments from users that recommended it, because people trusted Glendale more than his own aurora forecasts. He would regularly post up aurora alerts, paraphrased directly from the alerts on Glendale App but several minutes after my app issued them. They would be accompanied by a screenshot of the TGO Stackplot Graph and a link to SWL to give people the impression that he had been sitting watching a graph all night rather than following alerts from Glendale App.
I was banned from a Swedish group because the admin wanted to be the sole source of all aurora information. Only he could declare when an aurora was happening. Substorms did not apply to his part of Sweden because it had unique conditions that only he was able to advise on.
I was totally blocked from a large Australian facebook group, before even asking to join, because the admin had heard that I had launched an Australis version of the app and he was trying to monetise a camera-based alert system. He did not want anyone to find out about Glendale App.
In reality, Glendale App is very helpful for facebook groups because every night the groups are swamped with posts from people asking 'will there be aurora tonight?', 'where will it be seen from?', 'is it there yet?', 'is there anything happening?' etc. If they use Glendale App they can see at a glance whether there is any activity and where it is being seen at any time of day or night, anywhere in the world. It doesn't sleep. People then use the groups for sharing their photos and more insightful posts.
You will often see Facebook admins advising people to 'follow this page for alerts'. The problem there is that Facebook's renowned 'algorithms' decide what people see and when. It can be three days later before the appropriate posts appear in users' feeds. Facebook groups are a terribly unreliable way of alerting people about activity in real-time. The groups are great for longer-term predictions, general chit-chat and sharing photos.
If you use Glendale App, you automatically become an aurora 'expert' in the eyes of those that don't. Why would you ever want to share your secret advantage with anybody else?
I know of a number of facebook admins that use the forecasts and predictions from Glendale App but pretend that they are making their predictions using data from other sources. They generally recommend aggregator sites and avoid mentions of Glendale App on their own sites in an attempt to stop people from realising that they are cribbing their information directly from Glendale. Some even declare publically that they never use Glendale App whilst my logs show that they are very heavy users and always check my app before making their posts online. Imagine the outrage these people would feel had someone taken one of their photos, tweaked it slightly in photoshop and then passed it off as their own work?
I jot these things down just so that I will remember them, not to embarrass anyone. They chart the trials and tribulations, the daily grind of trying to help people to see auroras and the characters I encounter along the way. At some point somebody will write a book or a screenplay telling the story of Glendale App. You never know, it might even be me! Will you be a hero or a villain? Which actor or actress will play you?
For balance, there have been many positives on social media. Some facebook group admins positively embraced my app and encouraged their members to try it. It's perhaps not surprising that these admins are so experienced, knowledgeable and confident in their own abilities not to feel threatened by it. You should join these groups in your locations of interest and you will be well served by their teams.
The fact that so many have signed up to the app is purely down to kind people recommending it to their friends on social media. It has spread entirely through word of mouth.
In the summer and autumn of 2022, I had a few enquiries from people asking about differences between my trendlines for Bt, Bz, Speed and Density when compared to those given on popular aggregator sites. I could see that the aggregators were wrong and easily explain why mine were correct. It wasn't until Christmas of 2022 that I found the time to look at exactly why all of the other sites were presenting incorrect data.
What I discovered was that some time around 2021, NOAA had introduced a flaw into the data feeds that provide plasma and IMF data. This was an attempt to workaround issues caused by a faulty Faraday cup on DSCOVR. However, when used, it caused some significant problems on the graphs:
NOAA SWPC used the flawed feed on their own "official" graphs. Aggregator sites copied what the SWPC were doing. Social media "experts" continue to lift and share screenshots of these erroneous graphs from the aggregator sites daily.
Glendale App has always used the pure, raw data from the satellites and was immune to this flaw. Consequently, users of the app were able to see further into the future what was coming, they could see live data when all other sites had stopped showing it and they saw fewer false 'shocks'.
I've had many trolls come and go along my journey. Let's call them the 'trollies'. The most determined bunch are three guys whose main theme is 'Don't use Glendale App. Just stand outside all night and look at the sky'.
Someone asks for advice about aurora apps on facebook and one of them will appear with their familiar refrain. I would have explained here the reason why they are wrong but this dialogue really happened on FB (on 30/11/2022) and the public dealt with it admirably:
Person #1: Can I just ask, what app do people recommend for spotting the Aurora?
Various: Glendale
Trolly: Just go out on clear nights and look north.
Person #2: Having the app definitely helps with info on solar storms and likelihood etc.
Trolly: it’s still not as accurate as being outside and looking.
Person #3: But not everyone has the right area right outside their door. They might need to drive to a good spot, away from light pollution for example.
Trolly: which means waiting for an app to ping will be too late.
Person #3: The app uses prediction, like a weather forecast. They are actually quite accurate. Glendale, the one mentioned in some of the comments is very good. So you could head out to a good spot in plenty time, if there was a good chance any particular evening.
Person #4: Not true it will be too late. I had left my app on when back in Norfolk and when the app pinged went outside and the 1/4 mile from my bungalow to the beach, seeing the Aurora over The Wash when I got there.
Trolly: How much of it did you miss?
Person #4: Dunno Trolly but I saw more of it than I would've if the alert hadn't gone off and I'd stayed at home
QED
After a couple of years, one of these trolls started a new tactic of constantly telling people on facebook that Glendale App was rubbish because if you waited till people posted sightings you would miss the show, so they should use an aggregator site instead. I was intrigued by this because he was a heavy user of the app and regularly reported his sightings, so I switched off his ability to make reports.
On the next aurora night, I received an angry message wanting to know why every time he reported a sighting it disappeared. I said that I had put him on a partial block list because he was telling people the app was rubbish and advising other sites, yet he was using it heavily. His reply was that he needed to post a tick because his friends look for his ticks to decide when to head out. Hmmm.
I've had a few users whose enthusiasm turns excessive. They have to report more auroras than anyone else, they have to shout first, they have to get aurora every night, they have to be #1. In order to achieve this, the quality of the reporting starts to deteriorate. They report twilight, moonglow, light pollution and other unidentifiable traces of colour in their photos as aurora. They have a tick on the app every night when everyone else is reporting nothing, there is no geomagnetic activity at the time, the app is saying unlikely. I get lots of emails/messages from other users further north saying how can this guy be reporting aurora when I'm in, say Shetland, with a crystal clear sky, and there is no aurora whatsoever?
When I query their reports the problems begin. They are experts and it's 100% aurora. You are wrong. Your app is wrong about there being no activity. Finally, they claim that I am removing their reports because they are proving my app wrong.
Once they reach this stage there is no reasoning with them. They stop reporting on the app. They block me on socials. They start telling everyone on social media that the app doesn't work. They setup a fake account and continue to use the app secretly. They decide that their new mission in life is to prove the app wrong, to expose a conspiracy.
They twist events to fit their new narrative. An example was a tweet along the lines of "proof that Glendale App doesn't work: I was just out photographing aurora and the alert arrived when I got in". Makes the app sound useless.
When I looked at the timeline of events, there was a yellow alert six minutes after the guy tweeted he'd got aurora. The app had alerted the onset 30 minutes earlier and had been saying for the previous 90 minutes that the next burst of activity was expected at that time.
By April 2023, the trolling was becoming so severe that I began to consider whether I should just close the app down for UK users. One site devoted to trolling my app, and latterly me personally, had attracted enough followers that the equivalent of one in every twenty of my users was supporting it. This was at a time when the server needed replacing and I was facing a doubling of my monthly costs simply to receive potentially twice the abuse. I don't need or deserve this kind of attack for what is simply a hobby, that I used to enjoy doing, that costs nobody else anything and hurts nobody.
My initial aim was to run the app for a complete solar cycle, which would give me the complete set of data for the research projects I have on-going. The final loose ends of research don't need UK user reports. I could switch off reporting for UK users to save myself from the trolling, whilst still running the app for my own research and for users outside the UK. This would remove the need for a new server. On the other hand, it's not fair that a small minority should ruin it for everyone else.
The app is entirely self-funded. I pay for all server costs to make it free for everybody to use. The app carries no adverts and no SPAM is sent. No revenue is generated whatsoever. I freely give all my spare time to auroral substorm and noctilucent cloud research, development of the app and technical support for users.
Andy Stables
Originally written on December 19th, 2017. Updated on April 10th, 2023.