Howdy, Stranger!

It looks like you're new here. If you want to get involved, click one of these buttons!

Sign In with OpenID
Advertise on LowEndTalk.com

In this Discussion

Hetzner cloud in falkenstein down

Hetzner cloud in falkenstein down

mkshmksh Member
edited July 3 in Outages

Not affected myself. Just noticed because a couple services i use are down. I normally like Hetzner a lot but jesus that looks clueless: https://www.hetzner-status.de/en.html

Edit: Lol. The second it's posted at lowend talk the issue gets fixed. Massive importance or just coincidence? You decide ;)

Comments

  • MikePTMikePT Member, Provider

    Wasn't expecting it to be offline tbh...

    MXroute.io - SMTP Relay Service, powered by MailChannels, fully automated, LET plans

    MXroute.com - Email Hosting, powered by MailChannels

  • TomTom Member, Host Rep

    Humph, mine were off too. This has been happening alot recently :|

    I recommend Clouvider, Ramnode, HostUS

  • mkshmksh Member

    @MikePT said: Wasn't expecting it to be offline tbh...

    Yeah, i was also quite surprised when i saw that all the down servers belonged to Hetzner. Checked their status and was like: Ouch...

  • mkshmksh Member

    @Tom said: Humph, mine were off too. This has been happening alot recently :|

    Really? I guess i'll be holding on to my old non cloud VPS a bit longer then.

  • williewillie Member, Moderator

    Mine is up and shows 22 days uptime. Do you mean there was a network interruption? Any idea how long it lasted? My guess is other Hetzner VPS were also affected. No idea about dedis.

  • MikePTMikePT Member, Provider

    @mksh said:

    @MikePT said: Wasn't expecting it to be offline tbh...

    Yeah, i was also quite surprised when i saw that all the down servers belonged to Hetzner. Checked their status and was like: Ouch...

    @Tom said: Humph, mine were off too. This has been happening alot recently :|

    Yep and yep. But shit happens, and happens with anyone so let's just call it a day. Though lately I've been noticing a few network hiccups.

    MXroute.io - SMTP Relay Service, powered by MailChannels, fully automated, LET plans

    MXroute.com - Email Hosting, powered by MailChannels

  • MikePTMikePT Member, Provider

    @willie said: Mine is up and shows 22 days uptime. Do you mean there was a network interruption? Any idea how long it lasted? My guess is other Hetzner VPS were also affected. No idea about dedis.

    Network interruption, https://www.hetzner-status.de/en.html

    MXroute.io - SMTP Relay Service, powered by MailChannels, fully automated, LET plans

    MXroute.com - Email Hosting, powered by MailChannels

  • mkshmksh Member

    @willie said: Mine is up and shows 22 days uptime. Do you mean there was a network interruption? Any idea how long it lasted? My guess is other Hetzner VPS were also affected. No idea about dedis.

    Yeah, seems there was some flaky router. According to the status page it started acting up around 12AM (GMT+1?) with numerous tries to repair it until about 1:00PM. I guess there were multiple shorter downtimes during the night/morning (must have been up sometimes judging by the side effects on the affected services). Anyways, at least between 12PM and 1:00PM 2 services i use were completely offline. My stuff in Nuremberg was unaffected anyways.

  • williewillie Member, Moderator

    Interesting. Any idea if Nuremberg-Falkenstein connectivity was affected? One thing I like about Hetzner Cloud is you can easily have servers in both locations (plus Finland for that matter), with DNS round robin (I guess IP failover won't work if FLK is split from NBG)

    Hey @Hetzner_OL how about offering anycast routing between the 2 (or 3) locations?

  • mkshmksh Member

    @willie said: Interesting. Any idea if Nuremberg-Falkenstein connectivity was affected?

    Yeah, it was. That's actually how i noticed it. I have a VPS in Nuremberg that tried to connect one of the affected services but failed. Traceroute timed out. IP belonged to Hetzner. Shrugged it off at first (non profit services go down every day...) but when i couldn't reach a couple websites and all of them also resolved to Hetzner i got suspicious.

  • ZerpyZerpy Member

    My mysql replica experienced a 41-minute delay in replication, so I'd expect downtime to be roughly 41 minutes :-D

    On the bright side, no impact on prod traffic

  • hexukhexuk Member

    It still looks like there are issues (warning as soon as you login to the panel about the router). Is Hetzner to be trusted with large client websites? Got a few high-end clients I was thinking of migrating to the cloud panel but are they stable enough?

  • mkshmksh Member
    edited July 3

    @hexuk said: It still looks like there are issues (warning as soon as you login to the panel about the router). Is Hetzner to be trusted with large client websites? Got a few high-end clients I was thinking of migrating to the cloud panel but are they stable enough?

    According to the status page the issue is fixed sine 1PM local time. If they are still warning people about potential issues i'd call that being pretty open about it. Their communication is always excellent anyways. Even if the issue handling looks a bit messy this time hats of for having the balls to post it as is. Other providers would simply stay quiet and sit it out.

    On stability: While i haven't used their cloud line i am with them for about 10 years now and haven't experienced a single noteworthy downtime yet. Actually i was connected to one of their VPS almost 24/7 for a rather long time. During which i maybe witnessed about 2-3 <5m network hickups in a year and i am not even sure if the reason might have been on my side. I also have a VPS with around a year of uptime now if that says anything. For dedis you can pretty much assume they only reboot if you request it or there is a hardware failure (rare).

  • NeoonNeoon Member

    The dedis from Hetzner are solid, these cloud fuckups are also common for example at OVH from time to time.

    General rule, the more stuff you add together, the more complex and fragile it will get.
    Every time I tell people to get a dedi, even if its just a small one, its far better then any of the cloud "shit" or vps, whatever you wanna call it.

    Thanked by 3mksh vimalware willie
  • williewillie Member, Moderator
    edited July 4

    hexuk said: Got a few high-end clients I was thinking of migrating to the cloud panel but are they stable enough?

    Hetzner Cloud is still new and maybe a little bit iffy based on this thread and another recent one. But, there is no server or service that will never have an outage. If you have low tolerance for downtime you have to set up a high availability solution with multiple servers.

    I'd be interested to know if this outage only affected the cloud product, or also the regular VPS and dedicated servers.

    Also regarding high-end clients: this is low end talk, which might not be such a good match for high-end anything. I've been using Hetzner stuff happily, but I'm a low end user and if my stuff is down I either don't notice or I'm at worst slightly inconvenienced. I don't have anything on Hetzner where a even a much longer temporary outage would cause me any real difficulty.

    Also regarding high-end clients: other people are seeing better network speeds than me, but from my own experience I'd say if you have a lot of transatlantic traffic, Hetzner isn't the best possible choice. Their network within EU is much better than their transatlantic routes. I'm in the US and use Hetzner anyway because the hardware value is good enough for the trade-off to be worth it, but I do have to manage my expectations about network performance. If my requirements were more stringent I'd have to pay more and look for better connectivity.

  • williewillie Member, Moderator

    Regarding the dedis: I've had one about 3 years and it's been very solid, zero crashes and just one unexpected downtime due to a hardware failure (bad cpu fan). This was not a crash because I shut down the server on purpose, but it was unexpected downtime because the server didn't come back up when I expected it to. I opened a ticket and Hetzner staff found and fixed the problem, all within about 30 minutes, which I thought was great.

  • TomTom Member, Host Rep

    Offline again :(

    It's only a couple minutes (15 or so?) at a time but gets very annoying e.g. IRC connections dropping

    I recommend Clouvider, Ramnode, HostUS

  • rick2610rick2610 Member

    I have one cloud node offline now, yesterday all my nodes and dedis were down.

    Thanked by 1willie
  • mkshmksh Member

    @Tom @rick2610 is this still ongoing? At least right now the services i use that were affected yesterday seem to be online.

  • TomTom Member, Host Rep

    Nah, down for about ~45 mins

    I recommend Clouvider, Ramnode, HostUS

  • mkshmksh Member

    @Tom said: Nah, down for about ~45 mins

    So you are up again?

  • TomTom Member, Host Rep

    For now, ya

    I recommend Clouvider, Ramnode, HostUS

  • Hetzner_OLHetzner_OL Member, Provider

    Hi LET readers, Please see: https://www.hetzner-status.com/#9223 and https://www.hetzner-status.com/#9237 We will continue to post updates on our Hetzner Status site. And I will post more information when I can. @willie "Hey @Hetzner_OL how about offering anycast routing between the 2 (or 3) locations?" -- Please give me a few days to ask my colleagues about this. --Katie, Marketing

    Thanked by 3Tom willie Wolveix
  • williewillie Member, Moderator
    edited July 4

    @Hetzner_OL thanks for the update. I see the status page says "We are currently experiencing a fault on our router fsn1-dc8-ex9k2." Does that mean only DC8 is affected? If yes, maybe that leaked info about where the cloud servers are ;). Actual reason for asking is that some of us have dedis in other DC's. I think mine is in DC11.

  • vimalwarevimalware Member

    I only saw a 15 min network blip on my HTTP reverseproxy VM on a Skylake dedi in FSN-DC1 (around 0712-0727 Hrs UTC.)

    I don't have any cloud instances; so I guess I missed that outage.

    Thanked by 1willie
  • ZerpyZerpy Member

    @willie said: @Hetzner_OL thanks for the update. I see the status page says "We are currently experiencing a fault on our router fsn1-dc8-ex9k2." Does that mean only DC8 is affected? If yes, maybe that leaked info about where the cloud servers are ;). Actual reason for asking is that some of us have dedis in other DC's. I think mine is in DC11.

    It has always been known where the cloud servers are - the location in their panel literally states "fsn1-dc8"

    Thanked by 2willie Hetzner_OL
  • Hetzner_OLHetzner_OL Member, Provider

    @willie - I am unaware of any issues in FSN1-DC11 . I would write a support request via Robot. Please do this any time that you are having connectivity issues that have not yet been posted on Hetzner Status.

    @vimalware - Ditto for FSN1-DC1. --Katie, Marketing

  • williewillie Member, Moderator

    I haven't had any trouble with my dedi that I know of. I just wondered if the network prob was isolated to DC8, but it sounds like it wasn't. My dedi is mostly a compute and storage box so if it has occasional network outages I generally won't notice.

  • rick2610rick2610 Member

    The cloud in Falkenstein was again down for about 20 min today. I am a customer of Hetzner for 6 years but i never experienced so many problems as the last two weeks.

  • GulfGulf Member

    @rick2610 said: The cloud in Falkenstein was again down for about 20 min today. I am a customer of Hetzner for 6 years but i never experienced so many problems as the last two weeks.

    Wait till the end of the World Cup :)

    Thanked by 1inthecloudblog
  • ClouviderClouvider Member, Provider

    @Gulf said:

    @rick2610 said: The cloud in Falkenstein was again down for about 20 min today. I am a customer of Hetzner for 6 years but i never experienced so many problems as the last two weeks.

    Wait till the end of the World Cup :)

    Some networks cope well with the extra traffic : >

    Clouvider Leading UK Cloud Hosting solution provider || UK Dedicated Servers Sale || Tasty KVM Slices || Latest LET Offer

    Web hosting in Cloud | SSD & SAS True Cloud VPS on OnApp | Private Cloud | Dedicated Servers | Colocation | Managed Services

  • MikePTMikePT Member, Provider

    Yeah I too noticed the network hiccup today. It's been happening quite a lot lately. 3rd time in the last week.

    MXroute.io - SMTP Relay Service, powered by MailChannels, fully automated, LET plans

    MXroute.com - Email Hosting, powered by MailChannels

  • GulfGulf Member

    @Clouvider said: Some networks cope well with the extra traffic : >

    Their employees are still drunk in Russia :) Like these Swedish guys.

  • Could Hetzner clouds Nuremburg and Falkenstein together be considered analogous to AWS EC2's 'Availability Zones'.

    Would someone using a Hetzner floating IP between these DCs with quorum and failover have been able to route around the DC8-FSN network issue?

  • sibapersibaper Member

    so one filled ticket asking whats going on?

    reply from hetzner : 'Unfortunately an urgent maintenance had to be performed on the cloud node. For this purpose, all servers located on it were shut down and restarted after the work was done.'

    We only support unsupported OS!

Sign In or Register to comment.