WP-ORG System Status

Updated: Sep 6, 2019   As many of you are aware, West-Point.Org (WP-ORG), has been down since August 1st. We sincerely apologize for the inconvenience this has caused and the frustration that you may feel. We are currently bringing web sites back online and expect to have all available by this weekend.

  • Problem discovered during scheduled maintenance: During scheduled maintenance on July 28th, we discovered some network issues that we worked to resolve. What was to have been a 1-hour downtime turned into several days of having the systems go up/down in odd ways.
  • Ransomware attack: The networking issues that we discovered turned out to be the by-product of a ransomware attack on WP-ORG.
    • All of our main storage systems were compromised and the files were being encrypted to remove our access to them.
    • It is important to emphasize that data was not being pulled from the system, but instead being encrypted.
  • Current situation:
    • We are currently having each system assessed by experts and will bring all of our services back online.
    • 2019-08-08: Credit card sites are back online as of 4:30pm ET on August 8th.
      • All passwords have been changed for backend access. Megan Klein will contact all backend owners of live credit card sites for their new password. She has your contact information.
    • 2019-08-12: We have obtained all past and current files from our backup server and moved them to a secure hard drive.
    • 2019-08-12: We provisioned a new sterile file server with 20TB of storage that will allow us to bring up the remaining services.
    • 2019-08-13: Sterile database server provisioned and all databases restored to their last state. All systems that rely on the database (e.g., web, listserves, etc.) require that we restore this system as a priority.
    • 2019-08-14: Restoration of email files, web files, and listserve files started.
    • 2019-08-16: Restoration of all email files completed and validated.  
      • Email access settings will remain the same for your PC or mobile device, so once this is up then your systems should start pulling mail again. If your email is not working, then insure that your settings are to send and receive at users.adc.west-point.org. In the past, users.west-point.org worked, but now it must specifically be users.adc.west-point.org. 
    • 2019-08-19: Testing finally done for email. You may now send and receive email at your @west-point.org address. 
      • We ran into several issues getting the new system to work with new versions of POP and IMAP that are more secure than the older versions we were running.
      • Through a configuration error during the downtime, some email between 7/31 and 8/15 was returned to sender.
    • 2019-08-27: New firewall installed.
      • Credit card sites and email traffic were paused while the firewall was installed.
      • New firewall policies may have interrupted email for some users even after the services were turned back on. Those issues have been addressed.
    • 2019-08-29: Majordomo listserve functionality is restored and tested.
      • All list membership is restored to the status prior to the incident.
      • The ability to change list membership with our moderator tools will be restored at a future date.
    • 2019-08-31: All email should be working properly. If your email is not working, then insure that your settings are to send and receive at users.adc.west-point.org. In the past, users.west-point.org worked, but now it must specifically be users.adc.west-point.org. Contact feedback@west-point.org if you have issues.
    • In progress: Web site restoration.
      • 2019-09-03: Some virtual sites restored. 
      • 2019-09-06: Main web site restored with SSL capability. We are going through each hosted sub-site (e.g., /class/usma1989/) to determine if the site needs any updates to the underlying code.
      • 2019-08-31: Hosted virtual web sites are still in the process of restoration.
    • Future:
      • SFTP access
        • Access to your files via SFTP will come at a later date. We are still planning that infrastructure.

Some common questions that you may have

Q: Should I change my password?
A: We are assuming that any password used in WP-ORG is now compromised because the attackers had access to the system. We will be forcing every user to change their password. If your WP-ORG password was the same as one you have used outside of WP-ORG (though it shouldn’t be…that is a security risk), you should IMMEDIATELY change that password. 

Q: I use WP-ORG for email. Did I lose any email?
A: This is a multi-part answer. First, the email that was already delivered to your inbox at WP-ORG should be recoverable. Second, email that has been addressed to you during the outage will attempt to be delivered for 48-72 hours before it is sent back as undeliverable. Therefore, yes, some email addressed to you since August 1st will be lost (returned) but others will be waiting and will be delivered when our email is back online.

Author: Dian Welle