Your browser is no longer supported! Please upgrade your web browser now.

Harvest Availability Related to Hurricane Sandy

Since Monday night, the flooding and power outages caused by Hurricane Sandy have severely affected our availability. Harvest’s primary datacenter is located in the Chelsea neighborhood of New York City. The building is home to Google’s NY operations and a major datacenter in the north eastern U.S. While flooding has not affected our datacenter, the power outages in New York City, despite backup generators, have caused repeated issues.

Our team has been working around the clock to keep Harvest available. This post is to keep you informed of our current status and share the details of this event with you thus far.

Current Status

Saturday, Nov 3rd 7:00am ET – At approximately 5:52PM ET on Friday Nov 2nd, Harvest became unavailable due to problems related to the restoration of commercial power in the datacenter. The Harvest team worked through the night to safely restore service after the transition back to commercial power. By 7:00am ET, Harvest service is once again accessible to all customers. We thank you for the extra patience during this weekend evening which allowed us to restore our service properly. With commercial power in place at the data center, we take a big step towards stable conditions.

Thursday, Nov 1st 1:45pm ET – We continue to operate normally on backup generators with plenty of fuel at the datacenter. Based on ConEd’s latest estimate, they expect power to be restored in lower Manhattan by Saturday.

Wednesday, Oct 31st 7:30pm ET – Harvest experienced intermittent network issues from approximately 6:17pm ET till about 7:00pm ET. During this time, some customers were not able to access Harvest. The network issues were resolved at the datacenter and service should be 100% accessible at this time.

Wednesday, Oct 31st 12:30pm ET – Harvest is available and operating normally. Due to the power outage in lower Manhattan, all services are still powered by backup generators. Our datacenter has ample fuel and the ability to refuel as needed. To minimize downtime for our customers, we will continue to operate as-is and monitor the situation closely. Based on a statement from Con Edison (the power company in New York City), it will take an estimated 3 more days before commercial power is restored in lower Manhattan. We will provide updates if and when estimates change.

In the meantime, we have a backup plan in place to migrate our services should we need to (more details on that below). Note that all data continues to be safely backed up on-site and off-site in several locations during this time.

Please also note that you can always check our status at http://harveststatus.com

What’s Been Happening

Around 11:35pm ET on Monday, Oct 29th, Harvest went offline due to a power outage in our datacenter. During this time, datacenter staff worked to restore power and network connectivity. By 6:25am ET on Tuesday, Oct 30th, service had been restored. Harvest continued to operate on backup power generators during this time. Harvest operated normally for the rest of Oct 30th.

At 5:48am ET on Tuesday, Oct 31st, Harvest experienced a second outage window due to power failures in neighboring datacenters. This caused network connection problems which made Harvest unreachable by customers. By 7:00am ET, network paths had been re-routed and made Harvest accessible to most customers again. By 9:00am ET, all known network issues were resolved and Harvest became accessible for all customers.

Contingency Plan

While our core datacenter issues were being addressed, our team has been working tirelessly to ensure that we can reliably serve Harvest from a different datacenter if necessary. After around the clock efforts led by Harvester Warwick Poole to deploy and test the setup in a new datacenter, with support provided by our entire team, we are ready to take this step if necessary.

In the event of another extensive outage, we will immediately begin to switch service to a new datacenter located in Dallas, Texas. We expect this move to take approximately 2 hours, during which time Harvest will not be available. Should we need make the switch, we will alert all account owners via email and provide information through Twitter and harveststatus.com.

Thank You

Thanks to all our customers for the understanding and overwhelming support sent our way during this crisis. We couldn’t be more proud to have you good people as our customers. We will continue to do our best to provide service to you during this challenging time. If you have any concerns, please do not hesitate to contact us at support@harvestapp.com.

Thoughts or questions about this post? Need some help?
Get in touch →

This was posted in #status, Product News.
  • Thanks for keeping up in the loop! Be safe.

  • Thanks for letting us know and for staying up and running. Good contingency planning!

  • Excellent customer service. It’s very reassuring to know you have everything under control with backups and Contingency plans in place :-).
    Can’t really ask for better.

  • Super appreciative of your communication and hard work! Makes me that much more glad I chose Harvest :)

  • Sigurður Finnsson on October 31, 2012

    Thank you for keeping us so well informed. This is top service from you as usual. Good luck getting things into control again!

  • Thank you for your efforts to keep up and running to support us. Top notch customer service. Our thoughts are with your team and everyone within the region impacted.

  • I appreciate the updates. Keep up the great work!

  • Linda Kiss on October 31, 2012

    Status updates and contigency planning information is very reassuring, thank you. Wishing the entire Harvest Team respite and rest soon.

  • Harvest rocks — I deeply appreciate everyone’s tireless work to help keep the billing and invoicing backbone of our businesses going. I hope y’all stay safe, and that things stabilize enough for you to take care of yourselves and your homes soon. Take care!

  • Kathleen McGrorty on October 31, 2012

    As a Business Continuity Professional, I am impressed with your timely and informative communications. They are very detailed and to-the-point. As a customer, I am confident I chose the right service provider! Kudos to the Harvest recovery team! Best wishes and be safe.

  • Excellent job keeping things running! Also, thank you for keeping us in the loop – that’s more important than you may realize. I hope things return to normal as soon as possible. Much, much luck to all of you, and keep up your great work!

  • Great work getting things up so quickly… wishing you good luck on your personal recovery following the storm…

  • If you go down, I completely understand, its great to know the details of what is happening instead of being left in the dark. I agree with everyone – keeping us in the loop is more important than you may realize. Thanks for the updates

  • Honestly, this is top-notch. Spectacular service. Thanks for your tireless efforts.

  • Keep up the great work. The communications updates are fantastic. As a former Dreamhost web customer, that was the main area where they let me down. I loved the service, but when they went down, they also went incommunicado. You are guys are the MODEL for how this is done. Great job, Harvest Team.

  • Glad you all are safe, that’s the first priority. Your efforts to keep things running are very appreciated by the people at this small business. Your communications are just another indicator that your service/product offering are truly first class. Thank you all.

  • Great job in your crisis communications! Hats off to all of you, I’ve been in your shoes for smaller disasters and you are all shining through this crisis. Your hard work is so appreciated, and thoughts and prayers are with you and your families as you work through not only your professional crisis, but personal as well.

  • Thank you all for your support! The dev and support teams have been working around the clock and your encouragement really helps keep the spirits up. We’ll continue to keep you posted.

  • Henrique Mascarenhas on October 31, 2012

    Harvest serves people from all the world. I dont understand why there’s no redundancy. Thanks for keep us informed

  • Just a +1 here on the outstanding customer support and quick response, even in the face of a life-threatening disaster. You guys rock and roll!

  • You are going a fantastic job in spite of harrowing conditions. Makes me so glad I subscribe to your services! Hope you get a nice rest soon. Thanks for everything!

  • From the safe hills of Caistor UK, we can only stand in awe and hope if we ever have a major problem in our business, we can rise to the occasion and deliver the same high quality service to our clients and great communications. Your work is appreciated, as I suspect many of you have also personal challenges to face in the coming days, wishing you all the best, were pleased to be a client of yours. Pete

  • As a former NJ native (and with tons of family and friends in the metro area) I’m glad you guys are ok and secondly have been able to restore service without much disruption. As we were preparing for month end, we initially panicked, but were very happy to see the system up and running!

    Kudos to the Harvest team. I love SAAS

  • Thanks for keeping us posted. Good support and great service.

  • Glad to hear you’re okay. Thanks from another happy customer who thinks your service and communication on this have been great.

  • Dave Smith on October 31, 2012

    thanks for the great service and attention to detail….. keeping us informed in light of the crisis you are personally and professionally experiencing is appreciated and admired.

  • I appreciate the updates , you doing great job.

  • chris howden on October 31, 2012

    thanks for the updates guys, and working so hard to being it online so quickly.

  • On behalf of your subscribers in Australia thanks for the terrific customer service standard you are maintaining during this crisis. As many others have said already, being kept informed, keeping the communication open, is a key priority at times like this. Your product is terrific too, its made my business management so much easier, as a single person consultancy. Best wishes to all the team!

  • You guys rock. Thanks for caring about your service and product!

  • Thanks for keeping us up to date. Our thoughts are with your team and everyone affected by the storm.

  • Nicolas Pascual on October 31, 2012

    Thanks guys, you rock!

  • I wanted to echo what everyone else is saying. Fantastic work and great transparency. We are proud to be customers and as a former New Yorker my heart is heavy for what you all are going through. Hang in there. We have tons of patience out here.

  • Harvest has been such an amazing service that has helped us manage our small, growing business – I love you guys! My heart stopped for a minute when I got your email about the outage, but you have gained my confidence and continued admiration with your great communication and above and beyond dedication to ensuring our data is safe and the service is available. Thanks for all you are doing and we wish you a speedy and safe recovery!

  • Well done to all of you – downtime for us here in Australia was minimal and you have done an amazing job under such trying circumstances. We can’t begin to comprehend the magnitude of the devastation. Keep safe and thank you for your ongoing commitment to your customers.

  • Moz Nugent on October 31, 2012

    Echoing the other positive comments here.

  • Hi, as another Aussie user, thanks for the open and clear communication and work to support your business and us your clients.

    Appreciate your looking at other emergency data centre options.

    All the best with the overall recovery and ongoing operations strategy.

    Kind regards

  • What a great team with a positive attitude for a difficult situation. Thanks for putting the customer first and keeping us updated!

  • Hey Harvest Team,

    Thank you for your superhero-esque efforts to keep things up and running during this crisis time. Keep well and safe, and thank you for making such a clear and effective plan for situations such as this. Your vigilance is appreciated!

  • Thanks for all your updates and hard work in what is clearly a very difficult time for you and your fellow New Yorkers (and others in the region). All the best with the recovery in the days ahead.

  • Fantastic service even with all the uncontrollable things been thrown at you… appreciate the commnication to really makes us at ease…

  • Sam Gribble on November 1, 2012

    On behalf of the users from South Africa; thanks very much for the update, your transparency and making sure that you do all to keep your great service up-and-running. All the best in dealing with Hurricane Sandy.

  • If You’re Going Through Hell, Keep Going – Winston Churchill.

    Thanks from the UK.

  • Thanks very much for the hard work. It’s so reassuring to know that you are on the case. Your vigilance is very much appreciated!

  • We love you, Harvest! Keep the spirits up and good luck!
    More thanks from South Africa.

  • Great and hard work! We appreciate this update and wish you all the best.

  • Impeccable up to the storm! Harvest is really a quality service managed by involved and responsible people. Bravo

  • Good work! Thank you very much for updating us.

  • Thanks for the updates!

  • We hope that the Harvest team are all okay and not too exhausted from the overtime and mad weather! Thanks for the updates.

    The Laser Red Team

  • Thanks very much to all the Harvest team for working so hard to keep Harvest running during this difficult time for you, when I’m sure a lot of you have your own worries at home. Not only that but you have also gone to the trouble to keep us in the loop as to what is going on – excellent customer service – thank you very much.
    From all of our little team over here in England we hope you stay safe and wish you the best in getting your lives back to normal.

  • John Ambrose on November 1, 2012

    Thanks for doing your utmost to keep the service up. Wishing you and all your families all the best as you sort out everything.

  • Thanks for the update Danny. Appreciate all the work you guys are doing. I think a lot of us hadn’t realize how close you are to the hardest hit areas in Manhattan. As someone who has dealt with DR plans and testing of such setup in a past career (long time ago), I hope you will be able to find a robust alt site in a different state to mitigate locale risk.

    Hope everyone on the Harvest team and their families are safe.

  • inDemand - Nickko on November 1, 2012

    Thanks for the update,

  • YOU ARE STRONG… AND NOW EVEN STRONGER.

    Thank you for all you do and the way you do it!
    SIMPLY GREAT!

    Best from souther Switzerland ;)

    Florian

  • Jan Bertman on November 1, 2012

    Thank you for the update Danny and all the great work you guys do.

    Living in New Orleans and going through Katrina, and recently Isaac, we clearly understand what you are going through. Our thoughts are with you – hope you, your team and all their families, friends and animals are safe and warm!

    Best Regards,
    Jan

  • Great custom service and loving the forethought even in your time of crisis.

    Thank you for a great service Mr Wen & the rest of the Harvest Team.

    Good luck & keep safe.

  • Thanks for your hard work Harvest. Your service and communications are unparalleled. Speedy recovery out there in NY. Best of luck.

  • Thank you for keeping us up to date so efficiently, as long as we know what’s going on we can work around it. Great to know about your Dallas contingency plan too! :-)

    Keep up the good work, and well done for getting through what must be a hugely difficult time for you all.

    Massive thanks, Kate

  • Thank you for all your hard work. Good luck & stay safe.

  • Really appreciate your hard work in not only getting the systems back up and running so quickly, but for sparing the time to keep us updated too – I’m sure all of the team has their own individual worries about homes and families too, not just work!

    We’re glad we made the move to Harvest – your customer service rocks :)

    Best wishes from the UK.

  • I reiterate what Adrian Bell has said.

    I appreciate that your commitment to customer service has not wavered even in such difficult circumstances.

    Best wishes to you all and your families at this difficult time.

    Here in Boston, we are counting our blessings.

  • Sorry to hear you have been affected. I have some good friends and colleagues in this area. I used to work in Chelsea.

    Thanks for all your effort to get things back to normal. Be safe.

    I’m a Harvest brand champion. Like Adrian above, I’m also glad I made the switch and will continue to send business your way. Excellent product and service.

    Sending you some Harvest Love from Canada.

    Take care.

    Joanne

  • Thanks for the update.

  • Note from a Swedish user. I’m also very impressed with your timely communication. Thank you for your hard work, now have a well-deserved rest.
    It’s always a breeze to create invoices with Harvest :)

  • Thanks for being so upfront and honest. Reliability aside, transparency is important to keep us informed, so thank you.

    Have you considered setting making the whole setup redundant so that there would be no downtime to migrate to another datacenter in the event of a future failure?

    We virtualise our setup so that our applications are spread over several datacenters around the US. If one were to catastrophically go offline, the others would pick up the slack instantaneously, no downtime.

    Just a thought.

    Cheers
    Ben

  • You should put your DR in Arizona.

  • You guys rock. Thanks so much for both keeping your customers in the loop, and for having such incredible contingency plans. I feel completely comfortable leaving this section of my business with you guys, you definitely know what youre doing!

  • Lee Staniforth on November 2, 2012

    Thank you for the updates guys – very professional and reassuring – truly hope you are all okay on that side of the pond!

  • Thank you so much. Keep up the great work! It’s invoicing time – so thankful you are working so hard so we can all get paid. Thanks Harvest!

  • Thanks for the information updates and the quick response. Good work guys.
    Good plans for the future as well. You should put a datacenter in Sweden, nothing happens here :)

  • Thanks for the hard word.. Be safe :)

  • Thanks for the hard work. Be safe :)

  • Thanks for the hard work and the excellent communications. Hope you and your families are well.

  • I am a harvest customer and great job on your local plan with generators.. but wouldn’t it be easier to fail over to Dallas the same night things went down? From what I see it looks like there was no contingency plan set until this disaster happened. Failing over to a hot site would probably be easier than dealing with a datacenter that is dealing with the aftermath of a hurricane… I live in Houston and I know what thats like.

  • Good that you guys did work on that and brought Harvest back. This is our daily need. Thanks to your team !!

  • Thanks for the great communication.

    If you need help setting up a replicated cloud infrastructure across two regions, so that you no longer rely on any one provider and can failover without anyone noticing…

    I’ve done it for other companies, contact me. Initial consultation is free.

    ;-)

  • Ryan Platte on November 3, 2012

    I sincerely appreciate the proactive communication and heroics. But I pay you to stay up. After the effects of the hurricane wane, please get set up to switch over to a distant data center promptly when you’re facing 5 minutes or more of downtime. I am very surprised that a team of your caliber isn’t ready to fail over somewhere quickly and without fanfare.

    Happy to help if you need some extra hands to get this done.

  • Thanks guys for always being great with communication. Very few companies out there are willing to keep their customers in the loop such as you have done here. Its just one of the reasons you’ve found a life long customer in me. Keep up the amazing work!

  • Thank you for your great customer care!

  • Maureen McCarthy on November 4, 2012

    Thanks for your hard work and excellent updates. It has been tough for all of us in New York. Hang in there — things are improving, particularly now that your power (and my power) has been restored!

  • les pederson on November 5, 2012

    Great Job Gang, you are dealing with extreme conditions. You did a fantastic job — keep up the good work. It did not hurt me at all to make notes with a pen and paper for a few days, to track my time. Yep, a pen — still works.

    Thank You

  • Great job, guys. Something to consider though, a desktop app that can work both on and offline, for those just in case moments. I invested in Klok 2 and it works reasonably well with Harvest. I say reasonably well because I am forced to use stop and start times, so I’m finding myself having to combine line items so only 1 appears on the invoice . . .

Comments have been closed for this post.
Still have questions? Contact our support team →