Ok folks,

So here's where we are at:

• Public Site functionality is being restored
• New members process should be working now
• Forums restored
• ATM panel restored
• Instructor Panel Restored
• Cert Center Restored

Training Center is being worked on now.  I am still chasing bugs here and there in the transfer and instructor systems, but we are getting them worked out.  any help requests should be sent through our new help desk system at support.vatusa.net.

Thanks

I would like to first thank everyone for the support, patience, and understanding as we have worked to resolve these problems and challenges.  The outpouring of wonderful comments is very flattering and humbling, and I appreciate it deeply.  

I have described the last couple of days to the other VATUSA Staff members as putting band-aids on a gunshot wound, but I think we have bandaged this thing back up enough to get back to business while we finalize the VATUSA Strategic Plan for Information Management Systems over the next few days.  

Some special thanks go to Rob and Michael for their time in helping me crack the code on the certcenter problem, their input was invaluable.


Thank you all again, and we will keep patching glitches as we find them

6:00pm Update:

Brandon, I just saw your post, thank you for the offer.  I may contact you shortly.


Myself, Michael (USA3) and Rob (USA9) have been collaborating for the last few hours on the cert center and I am pleased to report that we have overcome the worst of the obstacles which was fixing the authentication system.  At this time we are going to break for dinner and then come back to working on the cert center.  Essentially what is left to be done is to update the form pages to start working right.  This is a fix that we have already performed several times in other systems, and I feel like we will be able to overcome most of the problems with relative ease.

This is a very good development on two fronts, first that we are getting the system back online, and second, that this is coming together due to a very positive, goal oriented team effort on the part of the VATUSA staff.  I am so proud and thankful to be working alongside such a dedicated group of guys who have no fear about rolling up their sleeves and helping out.

more to follow.......

UPDATE: SUNDAY 12:00pm

Even the best laid plans are subject to Murphy's law!  But we have managed to bring another system back online, just not the one I wanted.

The bad news:
-Cert Center/Testing login is down, but it looks like testing and reassigns are working

The Good News:
-The instructor site / promotions panel is back online
-The ATM control panel is back online and functional as far as I can tell (if anyone accepts a transfer today let me know if it works.)
-All data and records are intact as far as I can tell
-The training center is still up and running
-Email is running
-New Member Application is working and the Basic Controller test is working
-Forums are working
-Everything that is broken can be fixed!


Cert Center Update:
Ok guys, for you code nerds (like me) out there, you'll understand what the problem is when I say that all forms and sessions in the cert center only work under a php4 type server setup.  The config file I need to change is sitting in the directory just above us and is owned by VATSIM.  I have submitted our case for having the config file modded so the cert center will start working.  This does not mean that they will do it, only that I have asked.

For the rest of you, essentially the cert center is a very heavily modified software package from several years back and the problem is when you fix one thing, something else breaks, so before we commit to any further major work in the Certcenter, I would like to try to just get the server to agree with the software.

IF we get the server mod, then everything should start spinning again.  If not, Staff will have to make a judgment on what  move we want to make, whether to attempt a major salvage operation, or start anew.  The good news is that we are down to the last major system that needs to be repaired.  I will update again once I have heard from the powers above as to what the server config outcome will be.

Answer to the who did question
We know that the attack began on the 28th and the last hop into the system came from a system in Holland.  I have not explored it beyond that, and probably will not until recovery is complete.

more to follow..........

UPDATE: SUNDAY 7:30am

Sometimes, even when your not working on a project, good things can still happen....
The new website seems to be holding up well, no major problems observed or reported yet.  Also, last night it appears that all backend data systems operated normally.  I received several successful job completion emails, and no failures.   Very good sign.  I believe that we are in decent shape so far.  So the update list is.....

The bad news:
-Promotions panel is down
-Cert Center/Testing login is down, but it looks like testing and reassigns are working

The Good News:
-The ATM control panel is back online and functional as far as I can tell (if anyone accepts a transfer today let me know if it works.)
-All data and records are intact as far as I can tell
-The training center is still up and running
-Email is running
-New Member Application is working and the Basic Controller test is working
-Forums are working
-Everything that is broken can be fixed!


Today's goal(s):

1-Start on the Cert Center
2-....let's see how #1 goes first

more to follow..........

9:30pm Update:

We have managed to put together a new website as the old one was just not going to be very fixable, and was due to be replaced anyway.  The department pages are not all filled in yet, but each department head will be working on that soon.  A few things are not in there yet:
-C3 applications
-NOTAMS/Mass Email System
-ATC Booking (but it does not regularly get used anyway)

The bad news is:
-The ATM control panel is basically good for read only purposes
-Promotions panel is down
-Cert Center/Testing login is down

Good News is
-All data and records are intact as far as I can tell
-The training center is still up and running
-Email is running
-New Member Application is working and the Basic Controller test is working
-Forums are working
-Everything that is broken can be fixed!


Now let's talk about what the plan is:

1)Get the ATM/DATM control panel back up and running at 100%.  Ultimately most systems will be merged into the new site framework, but for now I believe that this system can be repaired with out TOO much pain.  This system is crucial for day to day operations, and has to be attacked first.  It is about 50-60% operational right now.

2)Certcenter, this is where the tests are and it is a very old legacy system.  Currently, here's what I know; The testing system must work to some extent becuse people are taking the Basic Controller Exam to join since we restored services.  That is a good sign.  What appears to be messed up the worst is logins/authentication.   IF that is fixed and nothing else major is wrong, then we are luck and just bought some time.  If not, then we are in a tough spot and will have to come up with an effective replacement in short order and that will be a difficult task as it is a very complex system.

3)Promotions/Instructor Site:  for the time being we ae going have to do promotions the old fashioned way.  Send us an email and we'll get it handled.  Again, the plan will be to try and restore the current system first, and if that fails, we'll have to overhaul.

4)Backend Systems: we will have to evaluate our automated systems to see if they are still integrating and communicating with VATSIM properly

5)NOTAMs/Email:  This will be the last system we get to, but it will get restored in one form or fashion.


Once all this is done, we will take a breath and then go back to the original goals that have been being discussed over the last week or so.

More to follow.......

Ok guys,

We are hobbling along with some glitches here and there and we are working to get those solved one by one.  Please bear with us as these issues are resolved.

Updates will be forthcoming, but just probably not every hour like before.