Welcome to September’s Progress Report! Firstly, we’re aware that this report is severely late and would like to apologise for the delay. RPCS3’s progress reports are solely written by volunteers and most of our regular writers could not contribute to this report due to personal commitments. Going forward, we are actively looking at ways to streamline the report to allow us to publish them in a timely manner. If you hate seeing RPCS3’s reports get delayed and would like to contribute to them, please apply here.

In addition to the following report, further details of Nekotekina and kd-11’s work during September and upcoming contributions can be found in their weekly reports on Patreon. This month’s Patreon reports are:

Status update from kd-11 (2019-09-14)
Status update from Nekotekina (2019-09-25)
Status update from kd-11 (2019-09-30)

Table of Contents

Major Improvements
Games
Other Improvements
Conclusion

After the large improvements in August, September featured a lull in testing games as our regular testers were focused on testing various pull requests opened during the month. Due to this, most categories only showed modest changes. However, the Loadable category did decrease from 16 games down to just 14 games now.

On Git statistics, there have been 17640 lines of code added and 7957 removed through 89 pull requests by 14 authors.

Major RPCS3 Improvements

RSX improvements and fixes (#6483, #6488, #6572, #6667)

This month, kd-11 made a multitude of RSX improvements and addressed a number of bugs present in the graphics output. Let’s take a quick look at the major improvements from the list. First up, kd-11 restructured the command buffer to to allow faults in vertex upload and defers the renderpass open to allow recovery after a fault in the middle of vertex upload. This fixed scenarios where flushing during draw calls would lead to inconsistent object state and “recovery” draw calls would be guaranteed to fail or crash the driver. This reduced the flickering when Write Color Buffers was enabled in games such as NieR and also improved the rendering of few post-processing effects.

Next up, kd-11 looked into optimising RPCS3’s handling of textures. Firstly, he implemented faster texture upload by looking for opportunities to upload the full data block in one go as opposed to parts. Secondly, he implemented auto-matching of pitch dimensions. Analysing statistics from multiple games revealed that most most use pitch alignment of 1, indicating that there is no reason to force a 256-byte alignment when uploading. This allows fast-copy to work which should reduce the load on the GPU and CPU while also drastically lowering the memory footprint of smaller textures. Finally, compute and transfer operations are now batched when decoding texture data which greatly lowers the overall number of compute invocations.

During this time, some users reported graphical issues in Metal Gear Solid 2 and 3 where Snake would be semi-transparent. This in itself was nothing unexpected, but with further user interaction, it was discovered that issue was present only with AMD graphics cards. This piqued kd-11’s interests and he began investigating into the issue. Initial testing revealed that this issue did not occur when MSAA was disabled and helped narrow the investigation to the implementation of the rendering pipeline search when using MSAA.

Before we proceed, here’s a quick look at RPCS3’s rendering technique. RPCS3 determines all the factors considered for a draw call and hashes them into a key (known as a renderpass key). These structures are then cached and fetched dynamically as and when the game requires it. However, the samples used in anti-aliasing were were previously not part of this key as they were assumed to remain static. After running hardware tests, it was discovered that it was indeed possible (albeit rarely) for a game to change the sample mask when rendering and such a change went unnoticed by the backend. This meant that not only the rendertarget configuration is important (with its renderpass key), but also the active sample mask. And as a complete surprise to nobody, Metal Gear Solid 2 was found to do this. Once the bug was out in the open, kd-11 promptly fixed it and Snake was once again visible in all his glory!

Finally, kd-11 finished the month off with a couple stability fixes. Asynchronous flips are allowed to safely occur in the handle descriptor reset with thread::end during a begin-end cycle without crashing the drivers. Also, the width up from surface cache was aligned before bpp normalization to avoid a result of 0 width and fence/event timeout was increased to multiple seconds. This was necessary as it was possible for the driver thread to “starve” if the emulator is compiling LLVM modules with all threads, causing a small timeout to fail. This fixed a spurious crash in Gran Turismo 6 after the attract video.

sys_usbd Implementation (#5688)

Like any console, the PlayStation 3 had a host of accessories such as the Guitar Hero guitar controllers, Rock Band drum kits, SingStar microphones, Logitech Driving Force GT racing wheel and even toys such as the Skylanders Portal that could be connected for additional functionality. The PlayStation 3 had dedicated software libraries to handle communication with these accessories. The cellPad library was used to communicate with controllers such as the DualShock 3, Guitar Hero guitar, drum kit, dance mat and many more. Microphones such as the SingStar microphone utilised the cellMic library whereas devices such as the PSEye camera used the cellCamera library. The goal of these libraries was to streamline and standardise the interface that games used to communicate with various peripherals. RPCS3 has a HLE implementation of all these libraries with varying degrees of completeness that allow games to correctly recognise the peripherals connected to the PC.

However, this was not the only way games could communicate with accessories on the PlayStation 3. There existed an additional library known as cellUsbd which allowed games to directly access and communicate with USB devices without any management. This library was particularly beneficial for games that required functionality not available through the dedicated libraries. However, RPCS3 did not support this cellUsbd as it required an LLE implementation of sys_usbd (the lv2 syscalls used by cellUsbd) and was only used by a handful of titles. That is until GalCiv decided to take a look into it. If the name rings a bell, it’s the same developer who expanded cellPad to support various PlayStation 3 accessories in February and implemented cellMic in July. In his mission to expand RPCS3’s support for peripherals, it was only a matter of time till he set his eyes on sys_usbd.

Development had it’s bumps but after 6 months of tackling issues, the work on sys_usbd was finally completed and merged into the master build of RPCS3. The new USB system was designed to handle abstraction from the grounds up which means implementing USB hardware emulation is relatively simple. As a proof of concept, GalCiv also implemented an emulated version of the Skylanders Portal along with its figures and added a GUI Manager to easily select the figures during gameplay. Here’s a video showcasing Skylanders Giants that now progresses ingame thanks to this:

There is still work left to be done with sys_usbd such as adding support for streaming USB devices (used in few SingStar titles) which will be addressed at a later point. In the meantime, users who wish to play games that utilise cellUsbd can refer to the guide on our Wiki. Use of this library requires an additional setup process as both Windows and Linux do not allow applications raw access to USB devices for security reasons. Another important point to note is that the decision on which library to use is decided by the game. So a particular peripheral, say a GH guitar, may be accessed via cellPad by DJ Hero and via cellUsbd by DJ Hero 2.

Improvements by Nekotekina (#6471, #6602, #6631, #6637, #6645)

Continuing the trend of previous months, Nekotekina undertook a broad task of finding low-level bugs and errors in the codebase and replacing them with the correct implementation. As was the case with other developers, we don’t have time to cover all the changes made so here’s a summary of the major improvements made. Firstly, he completed the implementation of g_fxo as explained in the July progress report.

Next, Nekotekina focused on how RPCS3 handles unimplemented HLE functions. The previous designed required the emulator to throw an exception every time an unimplemented HLE function was encountered. Due to this, RPCS3’s codebase was plagued by exceptions. Work was previously done in the past by Nekotekina to reduce their negative influence of using exceptions, but proved to only be a stop-gap measure. Exceptions were originally used in RPCS3 as developers were unable to find a better alternative at the time, but it is now apparent that emulation has very little place for such heavyweight C++ features. It is especially detrimental with the JIT recompilers where exceptions can be thrown even when there was no good excuse to throw them. To minimise the use of exceptions, Nekotekina changed the behavior of unimplemented HLE functions to no longer throw exceptions but rather show up as a TODO in the log and return CELL_OK. Also, some native filesystem errors at RPCS3 startup are now tolerated, whereas previously they would throw and exception and crash the emulator. A trivial set of changes, but ones that will help push RPCS3 towards the goal of completely disabling the use of such features in the future. As a side effect, the TTY output can now be completely disabled by creating a directory called “TTY.log” in place of file!

Next up, Nekotekina tackled a few Windows specific issues. First up, the use of certain undocumented Windows syscalls for keyed events was dropped this month. Not much information is available for these syscalls, but analysis revealed that WinAPI begun abandoning the use of these syscalls around Windows 8. Nekotekina opted to replace these syscalls with native semaphores which have been available since Windows XP. After this, Nekotekina fixed an error in internal path creation function, where RPCS3 tried to create parent folders when they already existed. An attempt to do so would sometimes produce unexpected errors with the previous implementation.

Finally, Nekotekina finished his work of making the savedata saving function atomic. This greatly increases the reliability of creating user saves and provides safeguards against events such as emulator crashes, computer BSoDs or sudden power loss. With atomic savedata, the old savedata remains in place until the new data is successfully created. Once the new savedata has been created, a maintenance function will delete the old savedata. Within the same pull request, Nekotekina addressed one of the Windows-specific workarounds that was made to allow access to deleted files. On the original PlayStation 3’s filesystem, Linux and many other POSIX-compatible operating systems, deleting opened files is allowed and files will remain “invisible” on disk until the last user closes them. But on Windows, this feature is typically emulated by moving the file somewhere and marking it for deletion. Nekotekina changed the location of placing these files that are being deleted in order to not interfere with removing empty directories.

Games

Thor: God of Thunder

First up in this month’s roster is the console exclusive hack and slash game, Thor: God of Thunder. This title previously did not progress past Intros but was recently found to be fully playable!

Top Spin 3

The third installment in Sega’s Tennis series,Top Spin 3 is now playable on RPCS3. With this, both titles in the series released on the PlayStation 3 now run at full speed on the emulator.

NCAA Football 11 & 12

Finally, two titles from the NCAA Football series, NCAA Football 2011 and NCAA Football 2012, have been found to be Playable. Previously, these titles suffered from lackluster audio quality and were not considered to be playable. However, with the recent improvements made to the emulator’s performance, the audio no longer stutters and the games can now be comfortable played even on mid-range hardware.

Other Improvements

There have been numerous other pull requests merged during the month that couldn’t make it to the Major Improvements section. We have collected a list of all such improvements here, and attached a brief overview to each. Make sure to check out the links provided for them if you are interested, as their GitHub pages usually uncover further details as well as the code changes themselves. To see this whole list right on GitHub, click here.

Nekotekina

6471 – Don’t throw exceptions on unimplemented HLE functions. See coverage in major improvements here;

6484 – Fixed regressions caused by PR 6393;

6494 – Removed spin wait loop in cpu_thread::check_state and fixed an out-of-bounds error in the PPU Interpreter;

6503 – Fixed GCC compilation;

6504 – Improved ALSA backend;

6507 – Simplified more code using waitable atomics and implemented uniform wait abortion for named_thread;

6543 – Added few atomic fixes;

6560 – Updated Internal API;

6576 – Minor bug-fixes and maintenance;

6580 – Minor bug-fixes and maintenance;

6619 – Fixed a bug in cellSaveData;

6602 – Dropped using of undocumented Windows syscalls for keyed events. See coverage in major improvements here;

6631 – Fixed a bug in fs::create_path. See coverage in major improvements here;

6637 – Improved cellSaveData implementation to make savedata atomic. See coverage in major improvements here;

6645 – Complete the removal of fxm. See coverage in major improvements here;

6655 – Refactored endianness support;

6666 – Restored experimental optimized operators for se_t.

kd-11

6359 – Completed rewrite of DMA transfer handling;

6455 – Addressed static analysis warnings;

6467 – Changed implementation to not include read-only data when attempting to do section merge;

6483 – Restructured commandbuffer scoping to allow faults in vertex upload. See coverage in major improvements here;

6485 – Minor shader generation improvements;

6488 – RSX texture optimisations. See coverage in major improvements here;

6493 – Fixed OpenGL decode description for uint_24_8; Fixed broken source reference when doing Z scaling; Fixed missing target when using Vulkan surface transforms without scaling; Removed unused surface transform argb_to_bgra as Vulkan now has proper texture decoding;

6498 – Fixed implementation of fast texture copies when linear mipmapped images are uploaded;

6512 – Further fixed issues with the texture cache;

6534 – Addressed bugs in blit engine when moving inverted regions;

6535 – Improved use of CPU vector extensions;

6546 – Set SSE4.1 as target for accelerated index path;

6571 – Restructured flip code and frame scoping;

6572 – Fixed pipeline search when using MSAA. See coverage in major improvements here;

6594 – Avoided recursion in flip handler;

6644 – Fixed DEVICE_LOST error seen with Nvidia turing graphics cards when using the Vulkan renderer;

6654 – Added missing initialization;

6656 – Fixed scaling issues with the native overlay;

6663 – Restructured RSX deferred flip queue handling;

6667 – Allowed asynchronous flips to safely occur in the handle descriptor reset with thread::end during a begin-end cycle without crashing the drivers. See coverage in major improvements here;

6673 – Fixed missing point size export that addressed a graphical regression in Hyperdimension Neptunia;

6683 – Addressed few regressions with the OpenGL renderer.

Megamouse

6465 – Disabled a speculative check that broke controller vibration in several games;

6497 – Based on user feedback, increased the maximum setting for mouse acceleration to a factor of 30;

6416 – Added error_code and a lot of function parameters including proper log messages to most sceNp functions. This helps with debugging and further implementation. Also added some basic checks for these functions (some of which were wrong but got fixed already);

6545 – Fixed regressions caused by the above pull request;

6500 – Added error_code to cellGem, cellMouse and cellVdec. This improves log error messages;

6505 – Enabled the CLI arguments –stylesheet which loads RPCS3 with a custom stylesheet, –style which loads RPCS3 with the chosen system style and –styles which enumerates available styles. Also, added “None” to the stylesheet options in the settings dialog which disables any styles or stylesheets;

6591 – Fixed the enter button assignment in the native On-Screen Keyboard.
In RPCS3 users can select if they wish to use circle or cross as enter buttons in the native dialogs (the other one is used for cancel). This was broken for the On-Screen keyboard because it switched those buttons internally again after they were already switched by the overlay control utility itself, effectively rendering that option useless in that case;

6593 – Fixed a minor issue in Ninja Gaiden Sigma that caused the game to open the menu after saving a game if you held any button too long after dialogs. Hotline Miami had a different issue that was fixed by keeping the so called pad interception intact while there was still a button pressed after leaving dialogs. Since the pad interception flag is passed to the game we assumed that games just ignored input and did nothing else. But Ninja Gaiden Sigma will in fact open the menu whenever that flag is activated during regular gameplay, which is exactly what happened with the previous fix. As a solution, the emulator no longer keeps the pads intercepted after dialogs but simply doesn’t give the game any input while buttons are kept pressed when closing dialogs. This solution works correctly with both Hotline Miami as well as Ninja Gaiden Sigma. Also, removed some unused code in the pad utility;

6615 – Added the missing DualShock 3 button remapping and settings. Fixed an evdev regression caused by the above pull request that disabled a random button. Also, unified some of the pad handlers’ behaviors into their parent class and adds some virtual functions to simplify future implementations;

6638 – Addressed new compiler warnings caused by PR 6615;

6642 – Fixed the accidentally removed DualShock 3 pressure sensitive button presses in PR 6615;

6520 – Added some more error checks to sceNpTrophy. Ideally this should fix some random games, improve general accuracy and help with debugging by adding more accurate log errors;

6636 – Added a missing override in the keyboard pad handler that disabled the keyboard in the pad settings dialog;

6677 – Use the current config value for FPS in the game window title instead of initializing the game window with a fixed setting. This will be useful once the settings can be changed during gameplay in the future.

elad335

6482 – Fixed passive reservation locks checking at Emu.Stop() (non-TSX)
Fixed a race condition on the non-TSX path that could have resulted in SPU threads waiting for the PPU thread to unlock its memory lock when the PPU thread has already exited because the emulation has stopped. This was addressed by releasing the memory lock on emulation stop;

6583 – Allowed scNp2Init to initialize sceNp more than once internally. This fixed few regressions in sceNp2Init caused by PR 6545;

6604 – Don’t escape from check_state() before ensuring signal is resetted.
Fixed an old race condition which could have resulted in cpu_thread::check_state() to not reset all flags it should before it awakes. This problem was detected by regressions in games since recent sys_cond_wait improvements in because it was one of the places in code which relied on proper behaviour of cpu_thread::check_state().

6585 – Implement more mnemonics of PPU BC, BCCTR and SYNC instructions for the PPU disassembler. BC is the conditional branch instruction and has a lot of different variations based on encoding;

6587 – Since all cellSaveData functions operate through a single gigantic function called savedata_op() with multiple error checks, it is very hard to tell which error check failed based on parameters alone. This pull request added the official naming for failed conditions in error checks which provides more descriptive logs and eases debugging efforts;

6657 – Fixed error reporting in sys_rsx syscalls and disabled logging of “PSN offline” error codes in sceNp functions as it was impacting performance;

6595 – Optimizations of the two SPU instructions (CGX & BGX) for both SPU recompilers;

6661 – Fixed bugs in sceNpUtilCmpNpIdInOrder, sceNpUtilCmpNpId and sceNpUtilCmpOnlineId’s comparison operation and result;

6659 – Masked FIFO PUT on RSX execution;

6678 – Improved performance of FIFO PUT register masking which was implemented in the above pull request; Improved synchronization of rsx FIFO command semaphore_release.

RipleyTom

6476 – Reverted File.h encoding to circumvent an obscure GCC bug;

6393 – Simulate BSD File System for savedata file presentation order.
Megamouse discovered that Blazing Angels 2: Secret Missions of WWII gave a buffer for only one input file to the savedata function and thus made file presentation order critical. This makes file order the same as on the PlayStation 3 (first file written is presented first and deleted files leave an empty space for future files);

6557 – Updated minimum CMake version to ensure compatibility with Visual Studio 2019;

6567 – Changed implementation to report file size in bytes for better compatibility with the auto-updater;

5688 – Implemented sys_usbd (USB passthrough/emulation). See coverage in major improvements here;

6577 – Added back initialization checks back in sys_usbd which were incorrectly removed in the above pull request;

6592 – On Linux, Qt changes the conversion locale to that of the OS, which forces it to use OS locale default for numbers. Fixed games not showing that they had been updated in the UI (strtod() was failing because of different languages using different decimal markers ‘,’ or ‘.’);

6622 – Implement sys_usbd_event_port_send
Implemented a missing sys_usbd function (sys_usbd_event_port_send) which makes games calling cellUsbdEnd functional again;

6635 – Only free the list if libusb_get_device_list succeeded as gathering device list failed on some systems and crashed the emulator because of invalid ptr.

MSuih

6454 – Added disk cache removal to UI;

6511 – Added handler descriptions to gamepad settings;

6554 – Fixed Debug build compilation error;

6652 – Implemented game version logging.

CookiePLMonster

6538 – Make “Clear shader cache” clear all caches, not just the first one;

6524 – Added SCP extension support to the XInput pad handler.

jbeich

6522 – Dropped unused is_leaf_function;

6624 – Fixed compilation on BSD when using Clang 9.

JohnHolmesII

6440 – Added a message for the user when using a .rap file with its extension in uppercase (.RAP);

6519 – Fixed compilation warnings with clang relating to null pointer checks in sceNp module.

isJuhn

6438 – Added rounding_alignment argument to get_dir_size method to allow GetSizeKB functions to report size more accurately.

6480 – Removed SSE4.1 requirement.

windyboy1704

6559 – Corrected a typo in TSX instruction tooltip.

youwereeatenbyalid

6542 – Added a “stack mode” to the TTY output window.

arabek

6510 – Fixed compilation warnings found in sceNp module.

kevinxucs

6470 – Converted EROFS error to avoid crashes on a read-only filesystem.

Joaozin-tech

6437 – Updated the latest firmware version to 4.85.

Closing Words

We hope you liked this report and look forward to the next one! If you would like to contribute to the project, you can do so either by contributing code, helping the community or becoming a patron. RPCS3 has two full-time developers working on it who greatly benefit from the continued support of the many generous patrons. In exchange, patrons also get special support over on our Discord server and get access to early updates directly from our lead developers. If you are interested in supporting us, consider visiting our Patreon page at the link below and becoming a patron, or join our Discord server to learn about other ways of contribution.

We’re always looking for dedicated writers to help us write these reports. If you have the skill, time and are willing to help, please apply here. Also, come check out our YouTube channel and Discord to stay up-to-date with any big news.

This report was written by HerrHulaHoop, MarioSonic2987, GalCiv, Megamouse, Digitaldude555 and elad335.