VATUSA Forums
General => General Discussion => Topic started by: Andrew Podner on May 30, 2009, 04:33:27 PM
-
Ok guys,
We are hobbling along with some glitches here and there and we are working to get those solved one by one. Please bear with us as these issues are resolved.
Updates will be forthcoming, but just probably not every hour like before.
-
Great job getting it back Andrew! I know you've been busting your butt all day long on it.
-
Thanks Tom,
Here's the latest update:
Most of our systems look ok on the surface but there are a lot of underlying issues that are killing us. the Webhost required a lot of server configuration changes to close loopholes and that basically put our legacy systems in a bad way. First off, our systems are secure. Our data has not been compromised. That is the good news.
The bad news is:
- The public site is not working correctly
- The ATM control panel is basically good for read only purposes
- Promotions panel is down
- Cert Center/Testing is down
- notams & email....unknown at this time
- pilot/atc booking....down
Good News is
- A new public site will be online within 24 hours
- The training center is still up and running
- Email is running
- New Members appears to be working (not sure about basic controller exam yet)
- Forums are working
- Everything that is broken can be fixed!
So please be patient, bear with us, and we will emerge from this stronger, faster, and more secure. Thank you all for your support as your VATUSA staff continues to put this back together. Expect further updates on a regular basis, beginning with a recovery plan that will be posted later this evening.
-
I'd like to echo what Tom has already said. Andrew has been doing a fantastic job today getting us back online. Initiation was smooth
More info will be released as we begin to piece things back together and figure out what needs to be done.
-
Andrew has been working all day non-stop on this. Thank you very much Andrew.
-
I am very impressed by the response to this crisis. Great work Andrew!
-
Thanks Andrew! I look forward to seeing the improvements that emerge from this!
Justin
-
9:30pm Update:
We have managed to put together a new website as the old one was just not going to be very fixable, and was due to be replaced anyway. The department pages are not all filled in yet, but each department head will be working on that soon. A few things are not in there yet:
-C3 applications
-NOTAMS/Mass Email System
-ATC Booking (but it does not regularly get used anyway)
The bad news is:
-The ATM control panel is basically good for read only purposes
-Promotions panel is down
-Cert Center/Testing login is down
Good News is
-All data and records are intact as far as I can tell
-The training center is still up and running
-Email is running
-New Member Application is working and the Basic Controller test is working
-Forums are working
-Everything that is broken can be fixed!
Now let's talk about what the plan is:
1)Get the ATM/DATM control panel back up and running at 100%. Ultimately most systems will be merged into the new site framework, but for now I believe that this system can be repaired with out TOO much pain. This system is crucial for day to day operations, and has to be attacked first. It is about 50-60% operational right now.
2)Certcenter, this is where the tests are and it is a very old legacy system. Currently, here's what I know; The testing system must work to some extent becuse people are taking the Basic Controller Exam to join since we restored services. That is a good sign. What appears to be messed up the worst is logins/authentication. IF that is fixed and nothing else major is wrong, then we are luck and just bought some time. If not, then we are in a tough spot and will have to come up with an effective replacement in short order and that will be a difficult task as it is a very complex system.
3)Promotions/Instructor Site: for the time being we ae going have to do promotions the old fashioned way. Send us an email and we'll get it handled. Again, the plan will be to try and restore the current system first, and if that fails, we'll have to overhaul.
4)Backend Systems: we will have to evaluate our automated systems to see if they are still integrating and communicating with VATSIM properly
5)NOTAMs/Email: This will be the last system we get to, but it will get restored in one form or fashion.
Once all this is done, we will take a breath and then go back to the original goals that have been being discussed over the last week or so.
More to follow.......
-
Wow, you've outdone yourself already! Nice work. I'm really hoping we can get our ARTCC tests back from the CertCenter.
-
Keep up the great work, Andrew.
-
Harold, the tests themselves are in there and fine, as are all previous results. Even if we move it to another system, you will not have to start from scratch
Wow, you've outdone yourself already! Nice work. I'm really hoping we can get our ARTCC tests back from the CertCenter.
-
Wow!! Great job Andrew. The new site also looks fantastic. Nicely done today.
-
For the time being, promotions will need to be emailed to me so I can upgrade the member through CERT. Please include the name and CID of the student.
-
What a fantastic job today Andy! You always got things done fast and eloquently when I first met you, and you are still doing things that same way! Can't wait to see your next ideas and improvements!
-
UPDATE: SUNDAY 7:30am
Sometimes, even when your not working on a project, good things can still happen....
The new website seems to be holding up well, no major problems observed or reported yet. Also, last night it appears that all backend data systems operated normally. I received several successful job completion emails, and no failures. Very good sign. I believe that we are in decent shape so far. So the update list is.....
The bad news:
-Promotions panel is down
-Cert Center/Testing login is down, but it looks like testing and reassigns are working
The Good News:
-The ATM control panel is back online and functional as far as I can tell (if anyone accepts a transfer today let me know if it works.)
-All data and records are intact as far as I can tell
-The training center is still up and running
-Email is running
-New Member Application is working and the Basic Controller test is working
-Forums are working
-Everything that is broken can be fixed!
Today's goal(s):
1-Start on the Cert Center
2-....let's see how #1 goes first
more to follow..........
-
Andrew,
To echo the other comments, you've done a phenomenal job in very short order. It is very, very much appreciated by the ARTCC staffs and the entire membership. A huge, huge THANK YOU!!
-
yes Thank You for everything you are doing.
-
Andy,
G-d is testing how good you really are in these past 24 hours
You have done great things in so little time to get this back on the road. I appreciate all of your efforts, as do all of the VATUSA members I am sure, and I look forward to seeing your progress as USA6.
Your friend,
-
This is a mini update as of about 10:30am
Right Now I am working on some local custom server configuration setting to see if we can make it cooperate with our legacy systems. Trying to overhaul the certcenter to make it agree with the server is not working out real well. I am focusing on trying solutions that plug leaks quickly. Other than the new public site, none of these are permanent solutions, just patches so we can buy time to fix it right.
more to follow...........
-
Out of curiosity, do we have an idea of who might be responsible for this?
-
Out of curiosity, do we have an idea of who might be responsible for this?
No idea at the moment, but it does look to be somewhat random although the attack was centered on an old inactive system running on our site. One would think that the hacker would need to know it was there, but I really don't know and Andrew can provide more insight than I can on this issue.
I do know that it took down the entire VATSIM.info server, including a couple other divisions (China and India to my knowledge).
-
UPDATE: SUNDAY 12:00pm
Even the best laid plans are subject to Murphy's law! But we have managed to bring another system back online, just not the one I wanted.
The bad news:
-Cert Center/Testing login is down, but it looks like testing and reassigns are working
The Good News:
-The instructor site / promotions panel is back online
-The ATM control panel is back online and functional as far as I can tell (if anyone accepts a transfer today let me know if it works.)
-All data and records are intact as far as I can tell
-The training center is still up and running
-Email is running
-New Member Application is working and the Basic Controller test is working
-Forums are working
-Everything that is broken can be fixed!
Cert Center Update:
Ok guys, for you code nerds (like me) out there, you'll understand what the problem is when I say that all forms and sessions in the cert center only work under a php4 type server setup. The config file I need to change is sitting in the directory just above us and is owned by VATSIM. I have submitted our case for having the config file modded so the cert center will start working. This does not mean that they will do it, only that I have asked.
For the rest of you, essentially the cert center is a very heavily modified software package from several years back and the problem is when you fix one thing, something else breaks, so before we commit to any further major work in the Certcenter, I would like to try to just get the server to agree with the software.
IF we get the server mod, then everything should start spinning again. If not, Staff will have to make a judgment on what move we want to make, whether to attempt a major salvage operation, or start anew. The good news is that we are down to the last major system that needs to be repaired. I will update again once I have heard from the powers above as to what the server config outcome will be.
Answer to the who did question
We know that the attack began on the 28th and the last hop into the system came from a system in Holland. I have not explored it beyond that, and probably will not until recovery is complete.
more to follow..........
-
3:30pm Update:
Cert Center: Still waiting to find out if we will be able to get the server reconfigured. I know that there is frustration about this and everyone who needs to take an exam is very anxious to do so. I promise you that the cert center is my #1 priority and we are doing all things possible to rectify the issue. Thank you for your patience.
-
Andrew
I did a lot of work with the PHP Test. I made some modifications. When I was working on the Atlanta Site we used a version that was modified similar to the one VATUSA's uses. You may not have to modify that php config file.
If you want to talk more email me. brandon.bartell@g-m-x.com (remove the dashes). Maybe when can get on teamspeak I have some ideas.
Brandon
-
6:00pm Update:
Brandon, I just saw your post, thank you for the offer. I may contact you shortly.
Myself, Michael (USA3) and Rob (USA9) have been collaborating for the last few hours on the cert center and I am pleased to report that we have overcome the worst of the obstacles which was fixing the authentication system. At this time we are going to break for dinner and then come back to working on the cert center. Essentially what is left to be done is to update the form pages to start working right. This is a fix that we have already performed several times in other systems, and I feel like we will be able to overcome most of the problems with relative ease.
This is a very good development on two fronts, first that we are getting the system back online, and second, that this is coming together due to a very positive, goal oriented team effort on the part of the VATUSA staff. I am so proud and thankful to be working alongside such a dedicated group of guys who have no fear about rolling up their sleeves and helping out.
more to follow.......
-
6:00pm Update:
Brandon, I just saw your post, thank you for the offer. I may contact you shortly.
Myself, Michael (USA3) and Rob (USA9) have been collaborating for the last few hours on the cert center and I am pleased to report that we have overcome the worst of the obstacles which was fixing the authentication system. At this time we are going to break for dinner and then come back to working on the cert center. Essentially what is left to be done is to update the form pages to start working right. This is a fix that we have already performed several times in other systems, and I feel like we will be able to overcome most of the problems with relative ease.
This is a very good development on two fronts, first that we are getting the system back online, and second, that this is coming together due to a very positive, goal oriented team effort on the part of the VATUSA staff. I am so proud and thankful to be working alongside such a dedicated group of guys who have no fear about rolling up their sleeves and helping out.
more to follow.......
I am going to PM you my number. Give me a call. I have done a lot of work with that PHPTest and I know how much of a pain it is to work with. I am happy to help out with anything that needs to be done.
-
********EDITED AT 9:50pm******* 9:30 PM UPDATE
The bad news:
-I'm Tired
The Good News:
-Cert Center is working in safari and IE7, and Firefox
-The instructor site / promotions panel is back online
-The ATM control panel is back online and functional as far as I can tell (if anyone accepts a transfer today let me know if it works.)
-All data and records are intact as far as I can tell
-The training center is still up and running
-Email is running
-New Member Application is working and the Basic Controller test is working
-Forums are working
-Everything that is broken can be fixed!
Cert Center:
It is possible to assign and complete exams in the cert center; however, the whole system is scheduled for replacement and we will commence work on this immediately.
Tests will not be added or edited until a new system is up and running. Staff members needing a change to a local test can contact me and we will work it out on the backend.
Thanks again for your patience. If anyone has problems, let me know.
-
You're doing an amazing job, Andrew. Keep it up.
-
I would like to first thank everyone for the support, patience, and understanding as we have worked to resolve these problems and challenges. The outpouring of wonderful comments is very flattering and humbling, and I appreciate it deeply.
I have described the last couple of days to the other VATUSA Staff members as putting band-aids on a gunshot wound, but I think we have bandaged this thing back up enough to get back to business while we finalize the VATUSA Strategic Plan for Information Management Systems over the next few days.
Some special thanks go to Rob and Michael for their time in helping me crack the code on the certcenter problem, their input was invaluable.
Thank you all again, and we will keep patching glitches as we find them
-
If you retired today (please don't) you would go down as one of the greatest data techs that VATUSA has ever had. You've done a masterful job in getting all this sorted out and working again, and over a weekend at that! Everyone on the system owes you a debt of gratitude.
[size=](Or at least a beer!)[/size]
-
Andrew and others who worked on this:
Thanks, the efforts are much appreciated.
-
Great job guys. Fantastic work.
Cheers!
-
Awesome job guys. From making a new web site in just 2 days that is just awesome.
-
Great Job Andrew. The fix was so fast I didn't even know anything was wrong. (Seriously, I had no idea the CertCenter was down until I read this thread) Was wondering why I haven't received any passing/failing emails from my students lately...
Great Work Andrew!
-Matt
ZSE I1
-
Bad news for all...
Andrew's wife is expected to divorce him for spending all his time on VATUSA webpage....
Just kidding Website looks great. Take a break soon/slow down otherwise you'll burn out for sure!
-
Andy,
You stepped up for your rookie debut with the bases loaded, and you hit a rocket right out of the park. The only problem now is... you'll have to do something EVEN BETTER to keep us impressed
Just kidding of course! Thanks for all your work in these few hectic days. They are greatly appreciated, and as Tom said, they will not go unnoticed!
Justin
-
Don't worry Justin, innovations are in the works......
stay tuned!