VATUSA Forums
General => General Discussion => Topic started by: Daniel Hawton on June 17, 2017, 04:05:40 AM
-
Ever wonder what 24 hours of flights in VATUSA looks like? I did.
Using data collected by AirStats, I applied some math and created a time lapse of VATSIM traffic from 6/16/2017 0000Z until 6/16/2017 2359Z (6/15 8:00pm US Eastern - 6/16 7:59pm US Eastern). See 24 hours of traffic in 3 minutes. It involved 2038 flights and 129,700 position reports (and about double that of mathematically calculated estimated positions).
Only flights that had flight plans, were on the within 500 feet AGL and 3nm of the departure and arrival airports for those phases and remained connected to VATSIM the entire time were logged. Only aircraft with a speed of 30 knots or more are shown. Position reports are speed based, <100 knots, every 10 minutes, 100-250 knots every 5 minutes, 250+ knots every 2 minutes. Positions are calculated for every minute.
VATUSA Timelapse: https://youtu.be/H_VJMFyBNvA Data obtained from: https://www.airstats.net
Hope you enjoy it.
-
Very cool!
-
Very cool, thanks for doing it and for sharing it!
-
Nice!
-
Awesome!
-
Nice. Too bad you didn't let it run another 4 hours and catch the entire Fort Houston Live event!
-
Awesome!
-
Nice. Too bad you didn't let it run another 4 hours and catch the entire Fort Houston Live event!
I do take requests ;) airstats is logging data 24x7x365 and I wrote it to be useful for any time period.
-
Yes, very cool!
If you decide to do one again, it might be interesting to include a clock showing the Zulu (or ET) time throughout the video. To the extent practical, showing which controllers are online would also be kind of interesting.
Pairing this with some of the major events we run (e.g., Cross The Pond) would be really neat, too.
-
Yes, very cool!
If you decide to do one again, it might be interesting to include a clock showing the Zulu (or ET) time throughout the video. To the extent practical, showing which controllers are online would also be kind of interesting.
Pairing this with some of the major events we run (e.g., Cross The Pond) would be really neat, too.
I thought about that including ATC, but I don't track ATC due to the number of boundaries I would have to collect to make it usable and that they are constantly changing. It's a significant investment of time that I don't really have. The clock has been added, the VATUSA and VATSIM time lapse videos were proof of concepts mostly, the next will have a clock
-
Do you need the request beforehand?
-
Nope. It would be great to have it within 24 hours of the event, if after, or earlier than as after the flight has been logged for 24 hours the positions are archived to AWS... it makes data retrieval in bulk a little more difficult but not impossible.
-
Nice. Too bad you didn't let it run another 4 hours and catch the entire Fort Houston Live event!
I do take requests ;) airstats is logging data 24x7x365 and I wrote it to be useful for any time period.
We'd love it if you could do a timelapse of the ZHU/ZFW Live event +/- a few hours!
-
Nice. Too bad you didn't let it run another 4 hours and catch the entire Fort Houston Live event!
I do take requests ;) airstats is logging data 24x7x365 and I wrote it to be useful for any time period.
We'd love it if you could do a timelapse of the ZHU/ZFW Live event +/- a few hours!
It will be published tonight.
Sneak peak:
(https://image.prntscr.com/image/nLWU3y8aT7meZp1dJ9nJrg.png)
-
Nice. Too bad you didn't let it run another 4 hours and catch the entire Fort Houston Live event!
I do take requests ;) airstats is logging data 24x7x365 and I wrote it to be useful for any time period.
We'd love it if you could do a timelapse of the ZHU/ZFW Live event +/- a few hours!
It will be published tonight.
Sneak peak:
(https://image.prntscr.com/image/nLWU3y8aT7meZp1dJ9nJrg.png)
Much appreciated, thanks Daniel! I'm looking forward to the results.
-
Nice. Too bad you didn't let it run another 4 hours and catch the entire Fort Houston Live event!
I do take requests ;) airstats is logging data 24x7x365 and I wrote it to be useful for any time period.
We'd love it if you could do a timelapse of the ZHU/ZFW Live event +/- a few hours!
+/- 1.5 hours at https://www.youtube.com/watch?v=joiIc-Mig3U
-
Nice. Too bad you didn't let it run another 4 hours and catch the entire Fort Houston Live event!
I do take requests ;) airstats is logging data 24x7x365 and I wrote it to be useful for any time period.
We'd love it if you could do a timelapse of the ZHU/ZFW Live event +/- a few hours!
+/- 1.5 hours at https://www.youtube.com/watch?v=joiIc-Mig3U
Sweet! Much appreciated!
-
Nice. Too bad you didn't let it run another 4 hours and catch the entire Fort Houston Live event!
I do take requests ;) airstats is logging data 24x7x365 and I wrote it to be useful for any time period.
Atlanta would greatly appreciate a time lapse of the Charlotte Live event from the 9th, +/- 1.5 hours or so.
-
Nice. Too bad you didn't let it run another 4 hours and catch the entire Fort Houston Live event!
I do take requests ;) airstats is logging data 24x7x365 and I wrote it to be useful for any time period.
Atlanta would greatly appreciate a time lapse of the Charlotte Live event from the 9th, +/- 1.5 hours or so.
This will take a few days as all the positions have already been archived.
-
+/- 1.5 hours at https://www.youtube.com/watch?v=joiIc-Mig3U
Thanks Daniel!
-
Thank you Daniel. That was pretty cool to watch. Amazing to think of all we do during an event.
You rarely have time to stop and think about it during, makes it even better to go back and see it later.
Appreciate your time with this.
JB
-
A timelapse of CalScream would be much appreciated. It's this Friday, from 2345-0400Z.
-
Nice. Too bad you didn't let it run another 4 hours and catch the entire Fort Houston Live event!
I do take requests ;) airstats is logging data 24x7x365 and I wrote it to be useful for any time period.
Atlanta would greatly appreciate a time lapse of the Charlotte Live event from the 9th, +/- 1.5 hours or so.
Finished the new formulas, which calculates positions at quarter-minute intervals instead of minute intervals. Makes the timelapse a lot smoother. Aircraft will still appear to turn on a dime, that's just because I only log positions every 2 minutes from the feeds.. positions between data feed positions are calculated based upon the calculated bearing and speed between points.
ZTL's event: https://youtu.be/7RxlD1d4N3k
-
Add some weather in there ;)
-
Add some weather in there ;)
Considering the global audience of AirStats and that there are no reliable world wide weather radars let alone archived .. that's not a possibility for me.
-
Could we get a Timelapse of CalScream yesterday from 2345-0500Z? Thanks!
-
Add some weather in there ;)
Considering the global audience of AirStats and that there are no reliable world wide weather radars let alone archived .. that's not a possibility for me.
And that there are so many simulator weather options, such an overlay may not be representative of what the pilots are trying to work around.
-
Could we get a Timelapse of CalScream yesterday from 2345-0500Z? Thanks!
Yes, saw your post.. the data is processing right now as we speak.. when I get home from work in ~1.5 hours it'll be captured and rendered and be posted for view.
-
Yes, saw your post.. the data is processing right now as we speak.. when I get home from work in ~1.5 hours it'll be captured and rendered and be posted for view.
Awesome, thanks!
-
Yes, saw your post.. the data is processing right now as we speak.. when I get home from work in ~1.5 hours it'll be captured and rendered and be posted for view.
Awesome, thanks!
Posted: https://youtu.be/Pvm13uXHzRU
-
Posted: https://youtu.be/Pvm13uXHzRU
Your soundtrack is perfect :D.
-
Doing one for the Anchorage FNO in August or should I? :)
-
This may be a bit much to ask for, but would it be possible to get our hands on your scripts to generate these such that we, as the community, could generate them as we wanted?
-
This may be a bit much to ask for, but would it be possible to get our hands on your scripts to generate these such that we, as the community, could generate them as we wanted?
All the data is obtained from my AirStats database, so you wouldn't be able to generate the same thing.
-
This may be a bit much to ask for, but would it be possible to get our hands on your scripts to generate these such that we, as the community, could generate them as we wanted?
All the data is obtained from my AirStats database, so you wouldn't be able to generate the same thing.
Unfortunate.
So next question... expose the data via some kind of web services for our consumption? :-)
-
This may be a bit much to ask for, but would it be possible to get our hands on your scripts to generate these such that we, as the community, could generate them as we wanted?
All the data is obtained from my AirStats database, so you wouldn't be able to generate the same thing.
That's a lot of bandwidth I don't want to pay for ;)
Unfortunate.
So next question... expose the data via some kind of web services for our consumption? :-)
-
This may be a bit much to ask for, but would it be possible to get our hands on your scripts to generate these such that we, as the community, could generate them as we wanted?
All the data is obtained from my AirStats database, so you wouldn't be able to generate the same thing.
That's a lot of bandwidth I don't want to pay for ;)
Unfortunate.
So next question... expose the data via some kind of web services for our consumption? :-)
Given the absence of response, I'll let my imagination assume that it's an affirmative response :D
-
This may be a bit much to ask for, but would it be possible to get our hands on your scripts to generate these such that we, as the community, could generate them as we wanted?
All the data is obtained from my AirStats database, so you wouldn't be able to generate the same thing.
That's a lot of bandwidth I don't want to pay for ;)
Unfortunate.
So next question... expose the data via some kind of web services for our consumption? :-)
Given the absence of response, I'll let my imagination assume that it's an affirmative response :D
My response ended up inside the quote... that's a lot of bandwidth demands I don't wish to pay for. That's the reason the API isn't public and won't be public.
-
This may be a bit much to ask for, but would it be possible to get our hands on your scripts to generate these such that we, as the community, could generate them as we wanted?
All the data is obtained from my AirStats database, so you wouldn't be able to generate the same thing.
That's a lot of bandwidth I don't want to pay for ;)
Unfortunate.
So next question... expose the data via some kind of web services for our consumption? :-)
Given the absence of response, I'll let my imagination assume that it's an affirmative response :D
My response ended up inside the quote... that's a lot of bandwidth demands I don't wish to pay for. That's the reason the API isn't public and won't be public.
Are you accumulating the data simply by scraping the status output?
-
This may be a bit much to ask for, but would it be possible to get our hands on your scripts to generate these such that we, as the community, could generate them as we wanted?
All the data is obtained from my AirStats database, so you wouldn't be able to generate the same thing.
That's a lot of bandwidth I don't want to pay for ;)
Unfortunate.
So next question... expose the data via some kind of web services for our consumption? :-)
Given the absence of response, I'll let my imagination assume that it's an affirmative response :D
My response ended up inside the quote... that's a lot of bandwidth demands I don't wish to pay for. That's the reason the API isn't public and won't be public.
Are you accumulating the data simply by scraping the status output?
Lots of math and frequent updates. Since AirCharts serves a mirror of the data feeds, I get to process them more rapidly than the 2-4 minutes other sites are generally restricted to. There are a number of calculations done and analysis before the position report is logged.
-
This may be a bit much to ask for, but would it be possible to get our hands on your scripts to generate these such that we, as the community, could generate them as we wanted?
All the data is obtained from my AirStats database, so you wouldn't be able to generate the same thing.
That's a lot of bandwidth I don't want to pay for ;)
Unfortunate.
So next question... expose the data via some kind of web services for our consumption? :-)
Given the absence of response, I'll let my imagination assume that it's an affirmative response :D
My response ended up inside the quote... that's a lot of bandwidth demands I don't wish to pay for. That's the reason the API isn't public and won't be public.
Are you accumulating the data simply by scraping the status output?
Lots of math and frequent updates. Since AirCharts serves a mirror of the data feeds, I get to process them more rapidly than the 2-4 minutes other sites are generally restricted to. There are a number of calculations done and analysis before the position report is logged.
Other than offering services as another mirror, is there any way for the average Joe like myself to get higher fidelity data in the manner you do?
-
Other than offering services as another mirror, is there any way for the average Joe like myself to get higher fidelity data in the manner you do?
No. There are limits to how often you can pull from the mirrors and there isn't a perfect interval for when the data arrives at the mirror, either. In fact, most mirrors monitor for abusers pulling more often than every 2-4 minutes and will block access.. so you can't just constantly call it to see if it's updated. Truthfully, the data feeds weren't really designed to be used for a FlightAware-esque style capture.. as I was building my data capture scripts is when I realized the update interval didn't line up with the update interval specified in the data feed itself as each tweak and run of my script was generating different results within 2 minute spans. Just happened to be one of the benefits having direct access to it.
-
Other than offering services as another mirror, is there any way for the average Joe like myself to get higher fidelity data in the manner you do?
No. There are limits to how often you can pull from the mirrors and there isn't a perfect interval for when the data arrives at the mirror, either. In fact, most mirrors monitor for abusers pulling more often than every 2-4 minutes and will block access.. so you can't just constantly call it to see if it's updated. Truthfully, the data feeds weren't really designed to be used for a FlightAware-esque style capture.. as I was building my data capture scripts is when I realized the update interval didn't line up with the update interval specified in the data feed itself as each tweak and run of my script was generating different results within 2 minute spans. Just happened to be one of the benefits having direct access to it.
Interesting. I suspected that the 2 minute interval wasn't accurate, but didn't want to piss anybody off looking in to it.
Is there any mechanism for individual users to get direct data feed access? This is VATSIM, after all, it's not like we'd be looking at Terabytes of bandwidth a month on a status page. I'd almost be willing to host another mirror if it could get me the data I wanted.
-
Other than offering services as another mirror, is there any way for the average Joe like myself to get higher fidelity data in the manner you do?
No. There are limits to how often you can pull from the mirrors and there isn't a perfect interval for when the data arrives at the mirror, either. In fact, most mirrors monitor for abusers pulling more often than every 2-4 minutes and will block access.. so you can't just constantly call it to see if it's updated. Truthfully, the data feeds weren't really designed to be used for a FlightAware-esque style capture.. as I was building my data capture scripts is when I realized the update interval didn't line up with the update interval specified in the data feed itself as each tweak and run of my script was generating different results within 2 minute spans. Just happened to be one of the benefits having direct access to it.
Interesting. I suspected that the 2 minute interval wasn't accurate, but didn't want to piss anybody off looking in to it.
Is there any mechanism for individual users to get direct data feed access? This is VATSIM, after all, it's not like we'd be looking at Terabytes of bandwidth a month on a status page. I'd almost be willing to host another mirror if it could get me the data I wanted.
Negative. And there are some serious bandwidth draws.. in excess of 300GB per month from my mirror (and I pull from my feeds via the harddisk). Running a mirror isn't a small thing, really. In comparison, the VATUSA website pulls between 6 and 7GB per month and Aircharts pulls about 3 and 4GB.
-
Other than offering services as another mirror, is there any way for the average Joe like myself to get higher fidelity data in the manner you do?
No. There are limits to how often you can pull from the mirrors and there isn't a perfect interval for when the data arrives at the mirror, either. In fact, most mirrors monitor for abusers pulling more often than every 2-4 minutes and will block access.. so you can't just constantly call it to see if it's updated. Truthfully, the data feeds weren't really designed to be used for a FlightAware-esque style capture.. as I was building my data capture scripts is when I realized the update interval didn't line up with the update interval specified in the data feed itself as each tweak and run of my script was generating different results within 2 minute spans. Just happened to be one of the benefits having direct access to it.
Interesting. I suspected that the 2 minute interval wasn't accurate, but didn't want to piss anybody off looking in to it.
Is there any mechanism for individual users to get direct data feed access? This is VATSIM, after all, it's not like we'd be looking at Terabytes of bandwidth a month on a status page. I'd almost be willing to host another mirror if it could get me the data I wanted.
Negative. And there are some serious bandwidth draws.. in excess of 300GB per month from my mirror (and I pull from my feeds via the harddisk). Running a mirror isn't a small thing, really. In comparison, the VATUSA website pulls between 6 and 7GB per month and Aircharts pulls about 3 and 4GB.
Matter of perspective, I suppose. 300GB/mo is small potatoes these days.
-
Yeah, but Daniel lives in Alaska. They just got internet last week.... ;)
-
Yeah, but Daniel lives in Alaska. They just got internet last week.... ;)
No.. that's Barrow. Barrow, for instance, has bandwidth caps that are measured in thousands of megabytes per month (15,000). ;) Fairbanks is a mix between 21st and 20th century. Hell, there are 2 Blockbusters still.
-
Other than offering services as another mirror, is there any way for the average Joe like myself to get higher fidelity data in the manner you do?
No. There are limits to how often you can pull from the mirrors and there isn't a perfect interval for when the data arrives at the mirror, either. In fact, most mirrors monitor for abusers pulling more often than every 2-4 minutes and will block access.. so you can't just constantly call it to see if it's updated. Truthfully, the data feeds weren't really designed to be used for a FlightAware-esque style capture.. as I was building my data capture scripts is when I realized the update interval didn't line up with the update interval specified in the data feed itself as each tweak and run of my script was generating different results within 2 minute spans. Just happened to be one of the benefits having direct access to it.
Interesting. I suspected that the 2 minute interval wasn't accurate, but didn't want to piss anybody off looking in to it.
Is there any mechanism for individual users to get direct data feed access? This is VATSIM, after all, it's not like we'd be looking at Terabytes of bandwidth a month on a status page. I'd almost be willing to host another mirror if it could get me the data I wanted.
Negative. And there are some serious bandwidth draws.. in excess of 300GB per month from my mirror (and I pull from my feeds via the harddisk). Running a mirror isn't a small thing, really. In comparison, the VATUSA website pulls between 6 and 7GB per month and Aircharts pulls about 3 and 4GB.
Matter of perspective, I suppose. 300GB/mo is small potatoes these days.
Depending on the provider. On my Digital Ocean droplet it ends up using more than 1/10th of my monthly bandwidth allocation just for the mirror, not counting the other services I run on it. Hence why AirStats will not have a publicly accessible API.
-
If you're still taking requests, a time-lapse of Northeast Corridor from tonight (6/29) would be cool to see.
-
Dan,
Think ZDV could have a time lapse from two hours before and after our recent Telluride Summer Festival event?
-
Dan,
Think ZDV could have a time lapse from two hours before and after our recent Telluride Summer Festival event?
https://www.youtube.com/watch?v=FBbXDwIkmtk
-
Think ZDV could have a time lapse from two hours before and after our recent Telluride Summer Festival event?
https://www.youtube.com/watch?v=FBbXDwIkmtk
Daniel -- weird question. Any chance the Zulu time stamp on this is an hour off? In watching this playback, I'm almost certain that I'm the "blip" that zooms in from the upper right corner of the screen at 2243z, splits the difference between KDEN and KCOS, flies an ugly rendition of the hold at Cones, then limps his t-storm-battered Citation into KTEX at around 2343z according to your screen. Which would make that 743pm EDT (local time for me in Baltimore). Yet my recollection is that I finished up that flight an hour earlier, almost exactly 645pm local, which would've been 2245z.
Not a nitpick (because I absolutely love these timelapse videos!), but, I figure you want the times to be accurate!
Actually, seconds before hitting "Post" it occurred to me to check Vataware, which seems to confirm my timeline above. Figure you might want to look into it!
http://www.vataware.com/flight/596bd28d535233693f000003
-
Think ZDV could have a time lapse from two hours before and after our recent Telluride Summer Festival event?
https://www.youtube.com/watch?v=FBbXDwIkmtk
Daniel -- weird question. Any chance the Zulu time stamp on this is an hour off? In watching this playback, I'm almost certain that I'm the "blip" that zooms in from the upper right corner of the screen at 2243z, splits the difference between KDEN and KCOS, flies an ugly rendition of the hold at Cones, then limps his t-storm-battered Citation into KTEX at around 2343z according to your screen. Which would make that 743pm EDT (local time for me in Baltimore). Yet my recollection is that I finished up that flight an hour earlier, almost exactly 645pm local, which would've been 2245z.
Not a nitpick (because I absolutely love these timelapse videos!), but, I figure you want the times to be accurate!
Actually, seconds before hitting "Post" it occurred to me to check Vataware, which seems to confirm my timeline above. Figure you might want to look into it!
http://www.vataware.com/flight/596bd28d535233693f000003
I've verified the timestamps on my database nodes, they're all showing the same. They sync times several times daily with a well known time server. In fact, your flight didn't get logged at all: https://www.airstats.org/search/1155655 -- there are several checks it does, and if something doesn't fit its criteria it scratches the flight rather than logging as incomplete. Either the data feed I get didn't get your FP info before you went airborne, it didn't see you arrive at the other end, or you were disconnected from VATSIM for at least 5 minutes.
-
Looks Very Nice!
-
Could I trouble you for a timelapse of ZMP's Northern Migration last week? 07/21 2230-0500z period?
-
Could I trouble you for a timelapse of ZMP's Northern Migration last week? 07/21 2230-0500z period?
Coming in the next couple hours.
-
Could I trouble you for a timelapse of ZMP's Northern Migration last week? 07/21 2230-0500z period?
Coming in the next couple hours.
https://youtu.be/ILGZX0PoEG0
-
Could we get a time lapse from ZOA's FNO last night. 10/06 2300-0600Z. Thanks!