Arcooda Cab Views 2 to 4 play in Slow Motion. View 1 fine - Build 1.70.18 (5th June)

[seattlemark]

i have it set to exclude the C:\Program Files (X64)\Steam\SteamApps\Common folder .

Question. I don't even have "C:\Program Files (X64)\Steam" on my PC. I just have "C:\Program Files (x86)\Steam", and I am on a 64bit PC and operating system. Where does one get a 64 bit version of Steam, and does it matter?

 

[SilverBalls]

As a reminder I said earlier I disabled all Avast shields and got 60 FPS. Instead of that I also tried excluding the two TPA folders but I floated around 54 FPS with a jittery ball so it didn't work as well as disabling the shields.

I then tried disabling the Avast Behaviour shield only and get a full 60fps. So the behaviour shield is the culprit for me.

 

[Mr 230268]

Just a short hint from my side. I solved the problem with a new graphic card and a new cabel setup:

http://www.digitalpinballfans.com/showthread.php/13353-Arcooda-Cab-Views-2-to-4-play-in-Slow-Motion-View-1-fine-Build-1-70-18-(5th-June)?p=273673&viewfull=1#post273673

I do not use any virus scanner or any protection software. From my point of view they are all bull****. Windows defender is more than enough.

 

[gust334]

I believe slow cameras are due to page faults from TPA.

My Windows 10 system saturates at ~2600 page faults per second.

Each single page fault from TPA triggers about 15 antivirus page faults.

Page faults counts can be observed in the Details tab of Windows task manager.

There are approximately zero page faults on the fast camera in Cabinet mode.
There are approximately zero page faults with any camera in non-Cabinet mode.

 

[SilverBalls]

Question. I don't even have "C:\Program Files (X64)\Steam" on my PC. I just have "C:\Program Files (x86)\Steam", and I am on a 64bit PC and operating system. Where does one get a 64 bit version of Steam, and does it matter?

I am 64-bit too - there is only one folder for Steam - ie: the one you mentioned.

 

[GAH1068]

I am 64-bit too - there is only one folder for Steam - ie: the one you mentioned.

The steam folder is in x86 not x64. Sorry, I had a brain cramp.

 

[SilverBalls]

I believe slow cameras are due to page faults from TPA.

My Windows 10 system saturates at ~2600 page faults per second.

Each single page fault from TPA triggers about 15 antivirus page faults.

Page faults counts can be observed in the Details tab of Windows task manager.

There are approximately zero page faults on the fast camera in Cabinet mode.
There are approximately zero page faults with any camera in non-Cabinet mode.

Nice find! I think you found the needle in the haystack. I just watched the Page Fault Delta in the task manager and it generates about 32k per second on my PC. Switch to the fast view and it's mostly zero. I just read page faults result in more disk activity, also the overhead of them.occurring plus the impact to the antivirus must explain the perf hit. If this is right Farsight owe you free tables for life I would say .

 

[seattlemark]

I believe slow cameras are due to page faults from TPA.

My Windows 10 system saturates at ~2600 page faults per second.

Each single page fault from TPA triggers about 15 antivirus page faults.

Page faults counts can be observed in the Details tab of Windows task manager.

There are approximately zero page faults on the fast camera in Cabinet mode.
There are approximately zero page faults with any camera in non-Cabinet mode.

First of all I want to thank gues334 for mentioning Page Faults being able to be observed in the Windows Task Manager! I never knew that. (Turns out "Page Faults" and "Page Faults Deltas" are extra columns that can be added through the view menu item of Processes tab.) Once I added Page Faults and Page Faults Delta, I had (my Windows 7) Task manager open on one monitor, while I tested TPA Cabinet Beta on the other. I had it sorted by the Page Faults Delta (by clicking on that column's header). Then I started testing TPA and switching between Views 1 - 4. Sure enough, I had 0 page faults on view 4. Views 1, 2, and 3, threw tons of Page Faults, with both that column and Page Faults Delta increasing dramatically. As soon as I switched back to View 4, Page Faults Delta would drop back to 0, and Page Faults would stop increasing. Note, however, I did not notice any similar thing in Task Manager for Avast, and Malwarebytes.

 

[SilverBalls]

First of all I want to thank gues334 for mentioning Page Faults being able to be observed in the Windows Task Manager! I never knew that. (Turns out "Page Faults" and "Page Faults Deltas" are extra columns that can be added through the view menu item of Processes tab.) Once I added Page Faults and Page Faults Delta, I had (my Windows 7) Task manager open on one monitor, while I tested TPA Cabinet Beta on the other. I had it sorted by the Page Faults Delta (by clicking on that column's header). Then I started testing TPA and switching between Views 1 - 4. Sure enough, I had 0 page faults on view 4. Views 1, 2, and 3, threw tons of Page Faults, with both that column and Page Faults Delta increasing dramatically. As soon as I switched back to View 4, Page Faults Delta would drop back to 0, and Page Faults would stop increasing. Note, however, I did not notice any similar thing in Task Manager for Avast, and Malwarebytes.

Interestingly the PF Delta column is the number of faults that are generated since the last update in the Task Manager. I think the default 'Normal' update interval is 1 second so for me it produced 35,000 pers second which is insane. Avast also didn't show faults for me in the task manager however the side affects might vary from setup to setup. It would be good to hear from Farsight today, but I just realised there is a holiday in the US today.

 

[gust334]

Here in the USA we celebrate winning our independence from the domination of England on this day.
I have a friend outside London who says they hoist a pint to "Good Riddance" on this day.

Back on topic... page faults occur in any software and are not necessarily evil things. Every software has circumstances where a page might not be in the present working set. However, so many faults generated per second, continuously for hours, is a strong indicator of something wrong in memory management.

I haven't been able to find any concrete data on live antivirus innards, but I speculate TPA is shot-gunning memory accesses all over the virtual memory space, and every time one causes a new page to be mapped, the antivirus sees the new address range and decides it needs to scan the new page and some number above and below it... and since those pages are not yet mapped, they generate their own faults. This speculation could explain why those people using antivirus that has a live scanning process see significantly more degraded performance than those without live scanning.

 

[hawkeyez88]

I've been having a rough time with this too - doesn't matter what I change this game is painfully sluggish during play.

 

[seattlemark]

First of all I want to thank gues334 for mentioning Page Faults being able to be observed in the Windows Task Manager! I never knew that. (Turns out "Page Faults" and "Page Faults Deltas" are extra columns that can be added through the view menu item of Processes tab.) Once I added Page Faults and Page Faults Delta, I had (my Windows 7) Task manager open on one monitor, while I tested TPA Cabinet Beta on the other. I had it sorted by the Page Faults Delta (by clicking on that column's header). Then I started testing TPA and switching between Views 1 - 4. Sure enough, I had 0 page faults on view 4. Views 1, 2, and 3, threw tons of Page Faults, with both that column and Page Faults Delta increasing dramatically. As soon as I switched back to View 4, Page Faults Delta would drop back to 0, and Page Faults would stop increasing. Note, however, I did not notice any similar thing in Task Manager for Avast, and Malwarebytes.

Correction to what I had mistyped earlier. Actually, "Page Faults" and "Page Faults Deltas" are extra columns that can be added through the view menu item of the Details tab.

 

[seattlemark]

I now have a new gaming PC (that I am going to run wires to the video pinball cabinet I will be building). With the new PC I am able to get 60 fps in all 4 views. I am running on it with the non-beta. The new PC is currently using the preinstalled McAffee Live Safe for anti-virus. With this new PC, I just ran the same test I had described in my earlier post. Result- Even though I am getting 60 fps in all four views on this particular PC, I still see the huge numbers of Page Faults on the PinballArcadeCabinet process in view 1, 2, and 3. As with my old PC, I don't notice any Page Faults in PinballArcadeCabinet with View 4. So I am wondering if these same Page Fault results are reproducible on any PC that isn't seeing the slowdown, including the ones FarSight/Arcooda are testing on.

I will be using the new PC for normal playing of Pinball Arcade going forward, but I still wanted to run this test on it to help in the debugging for those that have the Slowdown. Plus I can go back to my old PC for testing as well.

Added Note: My new PC is running Windows 10 64-bit with 16 GB of RAM, and my old PC is running Windows 7 64-bit with 6 GB of RAM.

 

[hawkeyez88]

I now have a new gaming PC (that I am going to run wires to the video pinball cabinet I will be building). With the new PC I am able to get 60 fps in all 4 views. I am running on it with the non-beta. The new PC is currently using the preinstalled McAffee Live Safe for anti-virus. With this new PC, I just ran the same test I had described in my earlier post. Result- Even though I am getting 60 fps in all four views on this particular PC, I still see the huge numbers of Page Faults on the PinballArcadeCabinet process in view 1, 2, and 3. As with my old PC, I don't notice any Page Faults in PinballArcadeCabinet with View 4. So I am wondering if these same Page Fault results are reproducible on any PC that isn't seeing the slowdown, including the ones FarSight/Arcooda are testing on.

I will be using the new PC for normal playing of Pinball Arcade going forward, but I still wanted to run this test on it to help in the debugging for those that have the Slowdown. Plus I can go back to my old PC for testing as well.

SeattleMark - i'm going to use your idea and run this on one of my more powerful machines and see if it produces the same slowness for me in all views.

 

[SilverBalls]

I reckon we will hear from Farsight on this soon, will probably take them a bit of time to see what's going on and do a fix. Pretty ineffecient how it is atm, so there is hope that improving the mem management will do the trick.

I am away from I'm home so cannot try, but did anyone disable the virtual memory page file to see if the views improve in speed and the FPS hits 60?

 

[seattlemark]

SeattleMark - i'm going to use your idea and run this on one of my more powerful machines and see if it produces the same slowness for me in all views.

Hi hawkeyez88. Sounds like I may not have been completely clear. On my new gaming PC , when I reran the test, I did not have slowdowns (and hopefully that will be the same for you when you retest on your more powerful machine). But on my new gaming PC, I did again see the Page Faults on the PinballArcadeCabinet process when observed in Task Manager when I reran the following test, which I am extracting from my original post about my original test on my old PC (with some adjustments to that earlier post) :

"Page Faults" and "Page Faults Delta" are extra columns that can be added through the view menu item of the Details tab. Once I added Page Faults and Page Faults Delta, I had Task manager open on one monitor, while I tested TPA Cabinet on the other. I had it sorted by the Page Faults Delta (by clicking on that column's header). Then I started testing TPA and switching between Views 1 - 4. Sure enough, I observed in Task Manager 0 page faults on view 4. Views 1, 2, and 3, threw tons of Page Faults on the PinballArcadeCabinet process, with both that column and Page Faults Delta increasing dramatically. As soon as I switched back to View 4, Page Faults Delta would drop back to 0 for the PinballArcadeCabinet process, and Page Faults would stop increasing.​

 

[gust334]

Not a peep from Farsight since Monday. Maybe they took the week off for the holiday. Hope they're rested and ready to continue debug next week.

I would expect disabling the virtual page file will do nothing to affect soft page faults.

(When one has sufficient physical RAM, there is no reason to page anything out to a page file.)

 

[SilverBalls]

Not a peep from Farsight since Monday. Maybe they took the week off for the holiday. Hope they're rested and ready to continue debug next week.

I would expect disabling the virtual page file will do nothing to affect soft page faults.

(When one has sufficient physical RAM, there is no reason to page anything out to a page file.)

Maybe they are still recovering from the 4th July celebrations.... after hoisting a few pints (I guess the 'other side' also do it ).

Did you find a way to determine the number of soft vs hard page faults out of interest or can we not get that level of breakdown? I read that sometimes when a program needs a large chunk of memory it can obviously be fragmented all over the shop, and memory locations can get repeatedly obtained/released, etc., so to help reduce the overhead and risk of a page faults, a program can request/reserve a chunk of memory upfront (a bucket) which it then holds onto. If it turns out that the page faults that are responsible for the slowness I wonder if the slow views could do something like this. It's all speculation / guesswork but you never know it may give Farsight some ideas or take them down another line of thought

 

[seattlemark]

Hi hawkeyez88. Sounds like I may not have been completely clear. On my new gaming PC , when I reran the test, I did not have slowdowns (and hopefully that will be the same for you when you retest on your more powerful machine). But on my new gaming PC, I did again see the Page Faults on the PinballArcadeCabinet process when observed in Task Manager when I reran the following test, which I am extracting from my original post about my original test on my old PC (with some adjustments to that earlier post) :

"Page Faults" and "Page Faults Delta" are extra columns that can be added through the view menu item of the Details tab. Once I added Page Faults and Page Faults Delta, I had Task manager open on one monitor, while I tested TPA Cabinet on the other. I had it sorted by the Page Faults Delta (by clicking on that column's header). Then I started testing TPA and switching between Views 1 - 4. Sure enough, I observed in Task Manager 0 page faults on view 4. Views 1, 2, and 3, threw tons of Page Faults on the PinballArcadeCabinet process, with both that column and Page Faults Delta increasing dramatically. As soon as I switched back to View 4, Page Faults Delta would drop back to 0 for the PinballArcadeCabinet process, and Page Faults would stop increasing.​

Turns out that Page Faults and Page Faults Delta are columns that can be added to Task Manager's Processes tab on a Windows 7 PC, and instead it's the Details tab of a Windows 10 PC.

 

[seattlemark]

I reckon we will hear from Farsight on this soon, will probably take them a bit of time to see what's going on and do a fix. Pretty ineffecient how it is atm, so there is hope that improving the mem management will do the trick.

I am away from I'm home so cannot try, but did anyone disable the virtual memory page file to see if the views improve in speed and the FPS hits 60?

I ran the test Silverballs was wondering about. Using information in https://www.howtogeek.com/126430/ht...-windows-page-file-and-should-you-disable-it/ , I disabled the page file on my old Windows 7 PC (which has 6GB of installed RAM). After the required reboot, I started up TPA on one monitor and Task Manager on the other. Results were the same as before on this computer:

a) Slowdowns on views 1,2,3.
b) Task Manager showing large increasing number in Page Faults Delta for PinballArcadeCabinet process in views 1-3, with Page Faults also increasing the entire time in those three views.
c) Full 60fps on view 4.
d) For PinballArcadeCabinet process in Task Manager, Page Faults Delta drops down to zero in View 4 and Page Faults stops increasing.