Cloud incident
Incident Report for Easee
Resolved
This incident has been resolved.
Posted Jan 04, 2021 - 11:11 CET
Update
All systems are operational and most chargers and Equalizers are online.

However some chargers may still be offline and has not been able to reconnect, please read through the updated guide below if your charger is offline.

Some devices may also sporadically be unavailable and feel more β€œsluggish” from time to time.

We may still need to deliberately force offline mode on some chargers to maintain system stability.
This state may be necessary over the next few days before we have the first of the long term improvements ready.

We are working in parallel on our long term efforts to find permanent solutions to the challenges we have been faced with lately.

-------------------------
Offline chargers instructions
-------------------------

Is your charger offline? πŸ“΅
If your charger is still offline please see this guide
https://www.notion.so/easee/Is-your-charger-is-still-offline-90a2743e3327461ba223e6450d9a3e63

LED blinking? 🚦
If the LED is blinking once you plug in your vehicle then you can use any Easeekey/RFID tag to start charging. The key-tag does not have to be in your Profile or does not need to have been used before. Any RFID key should work, you can use your access badge, wireless visa, a Phone with NFC or similar. For those of you who are unable to use your chargers while our cloud is offline, we suggest using public charging stations or emergency chargers πŸ”Œ

Do you have a schedule? ⏱
The schedule set for your charger is not possible to disable other than from the Easee app. It is not possible to turn off this schedule from the WiFi interface. The car will start charging when the schedule is set to start.

Cable locked? πŸ”’
You can unlock the cable from the charger by accessing the local WiFi interface. See more here https://www.notion.so/easee/Easee-charger-local-interface-b9ff6002acef4c3794531f68a6664d79
Posted Dec 19, 2020 - 19:55 CET
Update
All systems are operational and most chargers and Equalizers are online.
However some chargers may still be offline and has not been able to reconnect, we suggest to turn off and on the power to the charger if that is the case.

Some devices may also sporadically be unavailable and feel more β€œsluggish” from time to time.

We may still need to deliberately force offline mode on some chargers to maintain system stability.

This state may be necessary over the next few days before we have the first of the long term improvements ready.

We are working in parallel on our long term efforts to find permanent solutions to the challenges we have been faced with lately.

-------------------------
Offline chargers instructions
-------------------------

LED blinking? 🚦
If the LED is blinking once you plug in your vehicle then you can use any Easeekey/RFID tag to start charging. The key-tag does not have to be in your Profile or does not need to have been used before. Any RFID key should work, you can use your access badge, wireless visa, a Phone with NFC or similar. For those of you who are unable to use your chargers while our cloud is offline, we suggest using public charging stations or emergency chargers πŸ”Œ

Do you have a schedule? ⏱
The schedule set for your charger is not possible to disable other than from the Easee app. It is not possible to turn off this schedule from the WiFi interface. The car will start charging when the schedule is set to start.

Cable locked? πŸ”’
You can unlock the cable from the charger by accessing the local WiFi interface. See more here https://www.notion.so/easee/Easee-charger-local-interface-b9ff6002acef4c3794531f68a6664d79
Posted Dec 18, 2020 - 07:53 CET
Update
All systems are operational but several chargers have been set to offline mode to maintain stability.

That means that some of you may experience that your charger is offline or intermittently have connection issues.

This state may be necessary over the next few days before we have the first of the long term improvements ready.

We are working in parallel on our long term efforts to find permanent solutions to the challenges we have been faced with lately.

-------------------------
Offline chargers instructions
-------------------------

LED blinking? 🚦
If the LED is blinking once you plug in your vehicle then you can use any Easeekey/RFID tag to start charging. The key-tag does not have to be in your Profile or does not need to have been used before. Any RFID key should work, you can use your access badge, wireless visa, a Phone with NFC or similar. For those of you who are unable to use your chargers while our cloud is offline, we suggest using public charging stations or emergency chargers πŸ”Œ

Do you have a schedule? ⏱
The schedule set for your charger is not possible to disable other than from the Easee app. It is not possible to turn off this schedule from the WiFi interface. The car will start charging when the schedule is set to start.

Cable locked? πŸ”’
You can unlock the cable from the charger by accessing the local WiFi interface. See more here https://www.notion.so/easee/Easee-charger-local-interface-b9ff6002acef4c3794531f68a6664d79
Posted Dec 17, 2020 - 17:59 CET
Update
All systems are operational but several chargers have been set to offline mode to maintain stability.

That means that some of you may experience that your charger is offline or intermittently have connection issues.

In parallel, we are working on our long term efforts to find permanent solutions to the challenges we have been faced with lately.
Posted Dec 17, 2020 - 15:50 CET
Monitoring
A fix has been implemented and we are monitoring the results.
Posted Dec 17, 2020 - 09:06 CET
Update
We now have all systems operational, Operator environments, Easee app, Easee portal and API is up and running.

Some customers may still be lagging behind for certain operators as the systems have intentional delays built in to not overload the systems.

We are closely monitoring everything to ensure that everything is stable πŸ‘€πŸ‘¨β€πŸ’»β˜‘οΈ

We are still monitoring all functions and working on getting the historic data up.

We will issue a new update later on progress ❀️
Posted Dec 17, 2020 - 09:06 CET
Update
We are slowly bringing the system back to life so many of you will probably notice that things are starting to work again. The system is still not completely restored but we are going to work in a gradual and controlled manner bringing everything back.

The system is now running on optimized systems which have been implemented over the last few days. We are more optimistic now and expect better stability on this optimized system. We have several other initiatives that have been started to ensure future proof and scalable solutions both online and offline.

We are working together with all operators in bringing back all systems. Some of you may still not be able to charge because of the current state where we are gradually bringing things back.

The app you are using to control your charger may still have instabilities and not reflect the real-life status of the charger or Equalizer. Connection status may vary between displaying as online or offline.

We will let you know more as soon as we have more information, we thank you all for your patience in this situation β€οΈπŸ™
Posted Dec 16, 2020 - 17:36 CET
Update
We have now deliberately brought the system down to avoid overloading it.

Most chargers should now allow offline charging based on the steps below.

We will monitor the system before enabling them in a controlled manner again.

We will update you as soon as we know something more.

-------------------------
LED blinking? 🚦
If the LED is blinking once you plug in your vehicle then you can use any Easeekey/RFID tag to start charging. The key-tag does not have to be in your Profile or does not need to have been used before. Any RFID key should work, you can use your access badge, wireless visa, a Phone with NFC or similar. For those of you who are unable to use your chargers while our cloud is offline, we suggest using public charging stations or emergency chargers πŸ”Œ

Do you have a schedule? ⏱
The schedule set for your charger is not possible to disable other than from the Easee app. It is not possible to turn off this schedule from the WiFi interface. The car will start charging when the schedule is set to start.

Cable locked? πŸ”’
You can unlock the cable from the charger by accessing the local WiFi interface. See more here https://www.notion.so/easee/Easee-charger-local-interface-b9ff6002acef4c3794531f68a6664d79
Posted Dec 15, 2020 - 21:11 CET
Update
Thank you for all your trust and support during the last week. This has been a tough week for all of us ❀️

We are currently in a situation where we almost have all systems operational, however we cannot guarantee that it will remain that way continuously over the next few days.
Statuses in your app may at times be wrong and the system might feel slower than normal. We are closely monitoring everything to maintain control.

We have clearly underestimated our massive growth and cloud capacity. We feel your frustration towards the situation and the consequences that follow. This is not good enough – we are working day and night to solve the ongoing challenges with our cloud – getting closer to a future proof and scalable solution.

We have a specific plan on how to get the world balanced again – and fix this!

We are now making changes to the offline behaviour for all chargers so that you should be able to charge even if our cloud is not responding as it should.

At the same time we have our whole team working on the future-proof and scalable solution.

-------------------------

LED blinking? 🚦
If Easee is the operator of your charger and if the LED is blinking once you plug in your vehicle then you can use any Easeekey/RFID tag to start charging. The key-tag does not have to be in your Profile or does not need to have been used before. Any RFID key should work, you can use your access badge, wireless visa, a Phone with NFC or similar. For those of you who are unable to use your chargers while our cloud is offline, we suggest using public charging stations or emergency chargers πŸ”Œ

Do you have a schedule? ⏱
The schedule set for your charger is not possible to disable other than from the Easee app. It is not possible to turn off this schedule from the WiFi interface. The car will start charging when the schedule is set to start. Cable locked? πŸ”’
You can unlock the cable from the charger by accessing the local WiFi interface. See more here https://www.notion.so/easee/Easee-charger-local-interface-b9ff6002acef4c3794531f68a6664d79
Posted Dec 15, 2020 - 18:37 CET
Update
We are working together with external industry experts on fine-tuning our cloud and we have good progress here.
At the same time, we are working on an Β«off the shelfΒ» cloud solution made for the rapid growth we are experiencing, in order to prevent that this ever happens again.

We are working day and night and we hope to be able to get things back up and running as soon as possible. We thank you all again for your patience, this is not something we take for granted. An update will be issued as soon as we know more πŸ™Œβ€οΈ

---------------------------

LED blinking? 🚦
If Easee is the operator of your charger and if the LED is blinking once you plug in your vehicle then you can use any Easeekey/RFID tag to start charging. The key-tag does not have to be in your Profile or does not need to have been used before. Any RFID key should work, you can use your access badge, wireless visa, a Phone with NFC or similar.

For those of you who are unable to use your chargers while our cloud is offline, we suggest using public charging stations or emergency chargers πŸ”Œ

Do you have a schedule? ⏱
The schedule set for your charger is not possible to disable other than from the Easee app. It is not possible to turn off this schedule from the WiFi interface. The car will start charging when the schedule is set to start.

Cable locked? πŸ”’
You can unlock the cable from the charger by accessing the local WiFi interface. See more here https://www.notion.so/easee/Easee-charger-local-interface-b9ff6002acef4c3794531f68a6664d79
Posted Dec 14, 2020 - 21:39 CET
Update
The system is currently offline again. We are working on bringing it back gradually. This means that all chargers are offline.

Most chargers should work as defined by their offline settings meaning that you should be able to charge. Some chargers may have settings that do not allow charging when offline.

LED blinking? 🚦
If Easee is the operator of your charger and if the LED is blinking once you plug in your vehicle then you can use any Easeekey/RFID tag to start charging. The keytag does not have to be in your Profile or does not need to have been used before. Any RFID key should work, you can use your access badge, wireless visa, a Phone with NFC or similar.

For those of you who are unable to use your chargers while our cloud is offline we suggest using public charging stations or emergency chargers πŸ”Œ

Do you have a schedule? ⏱
The schedule set for your charger is not possible to disable other than from the Easee app. It is not possible to turn off this schedule from the WiFi interface. The car will start charging when the schedule is set to start.

Cable locked? πŸ”’
You can unlock the cable from the charger by accessing the local WiFi interface. See more here https://www.notion.so/easee/Easee-charger-local-interface-b9ff6002acef4c3794531f68a6664d79
Posted Dec 14, 2020 - 16:10 CET
Identified
We are now deliberately forcing several chargers in offline mode in order to limit the number of connections to our cloud while we are working on stabilising it.

Most chargers should work as defined by their offline settings meaning that you should be able to charge. Some chargers may have settings that do not allow charging when offline.

LED blinking? 🚦
If Easee is the operator of your charger and if the LED is blinking once you plug in your vehicle then you can use any Easeekey/RFID tag to start charging. The keytag does not have to be in your Profile or does not need to have been used before. Any RFID key should work, you can use your access badge, wireless visa, a Phone with NFC or similar.

For those of you who are unable to use your chargers while our cloud is offline we suggest using public charging stations or emergency chargers πŸ”Œ

Do you have a schedule? ⏱
The schedule set for your charger is not possible to disable other than from the Easee app. It is not possible to turn off this schedule from the WiFi interface. The car will start charging when the schedule is set to start.

Cable locked? πŸ”’
You can unlock the cable from the charger by accessing the local WiFi interface. See more here https://www.notion.so/easee/Easee-charger-local-interface-b9ff6002acef4c3794531f68a6664d79
Posted Dec 14, 2020 - 15:18 CET
Update
We are experiencing sporadic degraded performance. We are working on finding out the root cause of this and working together with all our partners to get everything up and running again.

Some of you may experience sporadic dropouts and your app may indicate from time to time that your charger is offline. We will get back to you as soon as possible when we have more news.
Posted Dec 14, 2020 - 13:01 CET
Update
"Houston, Tranquility Base Here. The Eagle has Landed.!” πŸ¦…βš‘οΈ

We are finally back πŸ™ŒπŸΌπŸŽ‰

We now have all systems operational, Easee app, Easee portal and API is up and running.

Some customers may still be lagging behind for certain operators as the systems have intentional delays built in to not overload the systems.

We are closely monitoring everything to ensure that everything is stable πŸ‘€πŸ‘¨β€πŸ’»β˜‘οΈ

We thank you all for your patience in this unfortunate incident β€οΈπŸ™

We can guarantee you that we have learned so much and we have a much better understanding of what we need to do going forward.
We still have a lot of work ahead of us to be able to scale all our systems along with with our amazing growth πŸ“ˆ

Although ever so unfortunate, we have learned the hard way here and we will use this knowledge to become the best at what we do! πŸš—βš‘οΈπŸŒπŸ’š
Posted Dec 14, 2020 - 00:44 CET
Monitoring
We have now updated enough chargers to be able to bring back systems to their normal operation as well.

We have just started activating OCPP services, APIΒ΄s, Easee app and Easee Portal. We are starting to bring back services in a controlled manner.

The online and offline state in the Easee app and portal may be wrong at times but this will get better over time.

We are not 100% back yet and some things may not work as it should but we are getting very close πŸ‘βš‘οΈβ˜ΊοΈ
Posted Dec 13, 2020 - 23:35 CET
Update
Over the weekend we have prepared a new firmware version for the chargers which is more "friendly" on our cloud, both in its behaviour on how It connects to the cloud and also how intense it is on updating live data which is not needed.

We have been gradually letting in more and more chargers all day while at the same time updating their firmware. This has been done in a controlled manner while closely monitoring the cloud systems.

We are now getting close to having all devices connected again. Once all chargers have been updated and as long as everything is stable we will gradually start activating all services again.

The positive results we have seen today means that we are in a much better position now than we have been in a long time βš‘οΈπŸ™Œβ€οΈ

We will keep you all informed as soon as we know more! 😊
Posted Dec 13, 2020 - 22:42 CET
Update
We are currently updating around 50% of the chargers to a new version which is β€œkinder” on our cloud πŸ˜…

This is a forced update so cars currently charging will stop charging to install the update.

Your chargers may also reboot during this process.

We will issue a new statement on the situation we are in soon.

Even though some chargers are online no apps or systems will work until we enable those services again.

Here is the link to the changelog for the firmware we are installing. https://easee.upvoty.com/f/changelog/charger-v258/
Posted Dec 13, 2020 - 14:24 CET
Update
We were not able to restore the systems with all the parallel attempts we had today. Currently it seems like we need to utilize the weekend to work on deeper architectural changes to our cloud services on order to restore the system properly πŸ› πŸ§°πŸ‘·β€β™‚οΈ

Everyone in Easee and our dear partners are working together with us on solving this issue.

We understand that many of you are frustrated, we are and will be doing everything we can to never end up in a situation like this again.
At the same time we are working on the sideline to make plans for how our products can continue to work as well as they potentially can in situations like this in the future.

LED blinking? 🚦
If Easee is the operator of your charger and if the LED is blinking once you plug in your vehicle then you can use any Easeekey/RFID tag to start charging. The keytag does not have to be in your Profile or does not need to have been used before. Any RFID key should work.

For those of you who are unable to use your chargers while our cloud is offline we suggest using public charging stations or emergency chargers πŸ”Œ

Do you have a schedule? ⏱
The schedule set for your charger is not possible to disable other than from the Easee app. It is not possible to turn off this schedule from the WiFi interface. The car will start charging when the schedule is set to start.

Cable locked? πŸ”’
You can unlock the cable from the charger by accessing the local WiFi interface. See more here https://www.notion.so/easee/Easee-charger-local-interface-b9ff6002acef4c3794531f68a6664d79

We would like to thank everyone supporting us and cheering for us in this unfortunate situation.
Our developers are working day and night to get back on track, we thank you for the support it means so much πŸ™β€οΈ

We will continue to update you all as soon as we have some news to share.
Posted Dec 11, 2020 - 21:19 CET
Update
We are still working on trying to stabilize our system to be able to bring back all devices. We are experiencing a high load on our services as we are trying to bring back devices online. We are making progress but it is going slower than we hoped for. We hope to bring back all devices today but we cannot guarantee anything at current point of time.

We will give you another update soon when we know more.
Posted Dec 11, 2020 - 16:13 CET
Update
The system is deliberatly now set to be unavailable while we are working on the fix.

We started implementing a fix this afternoon and we were planning to gradually bring back chargers that are running on strict offline behaviour tonight. As of now we are not able to start bringing them back gradually until tomorrow.

LED blinking?
If Easee is the operator of your charger and if the LED is blinking once you plug in your vehicle then you can use any Easeekey/RFID tag to start charging. The keytag does not have to be in your Profile or does not need to have been used before. Any RFID key should work.

Do you have a schedule?
The schedule set for your charger is not possible to disable other than from the Easee app. It is not possible to turn off this schedule from the WiFi interface. The car will start charging when the schedule is set to start.

We are sorry for the inconvenience this is causing. We are doing everything we can to resolve this as soon as possible❀️⚑️
Posted Dec 10, 2020 - 23:36 CET
Update
We have started implementing a fix and are planning to gradually bring back chargers that are running on strict offline behaviour.

We hope to be able to start bringing them back later tonight.
All other chargers will be brought back gradually over the next day.

LED blinking?
If Easee is the operator of your charger and if the LED is blinking once you plug in your vehicle then you can use any Easeekey/RFID tag to start charging. The keytag does not have to be in your Profile or does not need to have been used before. Any RFID key should work.

Do you have a schedule?
The schedule set for your charger is not possible to disable other than from the Easee app. It is not possible to turn off this schedule from the WiFi interface. The car will start charging when the schedule is set to start.

We are sorry for the inconvenience this is causing. We are doing everything we can to resolve this as soon as possible❀️⚑️
Posted Dec 10, 2020 - 17:56 CET
Update
We are still experiencing partial connection errors. We are working hard to find out how we can stabilise our systems to bring everything back. You will experience partial system instabilities.
Posted Dec 10, 2020 - 14:04 CET
Update
We are almost back to normal, most of you should see that your chargers are online in the app you use. However, some of you might from time to time experience slowness or wrong statuses. We are working on getting all systems operational again.

Historical data could be missing but will over time come back, for example, historical consumption.
Posted Dec 10, 2020 - 09:52 CET
Identified
The issue has been identified and a fix is being implemented.
Posted Dec 10, 2020 - 07:45 CET
Update
We are having problems recovering our cloud services after the outage. We have our whole team working on restoring the system.
The system is now completely offline so most chargers should work as defined by their offline settings meaning that you should be able to charge. Some chargers may have settings that do not allow charging when offline.

We will be doing everything we can do resolve the issue as fast as possible.
Posted Dec 09, 2020 - 21:47 CET
Investigating
The problem we experienced earlier seems to have hit us again. We are investigating the incident.
Posted Dec 09, 2020 - 20:20 CET
This incident affected: Easee Cloud & API and OCPP.