A new version of DLSS is here, together with an alternative upscaling that works on all GPUs
It’s been a little over a year since Nvidia announced the 2.0 generation of their FPS-increasing AI upscaling DLSS, with significantly better quality thanks to temporal reconstruction and stabilization. Now it has been refined into DLSS 2.3, which brings a new AI for better quality, but at the same time Nvidia also introduces a purely spatial upscaler, which is actually a competitor or alternative for the similar AMD FSR technology.
The latest version of AI upscaling and reconstruction is called DLSS 2.3 and was officially unveiled today. Ironically, the public has known about it for a while, as the files for this version were already publicly available. And thanks to the ability to replace libraries in games, players have already been able to test DLSS 2.3.
DLSS 2.3: Improved against ghosting
According to Nvidia, this version should have extended the use of motion vectors, but it’s not specified what exactly has changed. In theory, it could be an increase in the number of consecutive frames the game uses for temporal filtering and reconstruction. But the number was apparently quite high already, so it’s more likely that the change is in how they are evaluated and “blended” into the result. At the same time, Nvidia is also continuing to train the AI model used in DLSS, so that the interpolation and temporal part could see various gradual improvements in various ways.
DLSS 2.3 should improve ghosting in some situations (frame blending artifacts, traces of previous frames) that are sometimes manifested in DLSS 2.x due to the temporal nature of this filter. DLSS 2.0 has motion compensation, where it uses motion vectors to put together pixelsfrom different parts of the frame according to how objects move in them, so in most cases ghosting is unlikely to occur. However, due to different rotations, changes in size, perspective and so on, these past instances of objects may not correspond directly to the current instance, and pixels that don’t quite fit together are combined, so sometimes the combination goes wrong and ghosting, blurring and artifacts occur, where you can see something in the image coming from the previous frame, that shouldn’t be there in the current one. DLSS 2.3 is supposed to be more successful and smarter in this regard, so (at least in some cases) ghosting is now avoided.
You can see an example of this in this Cyberpunk 2077 screenshot around the rear view mirror. This game is supposedly getting DLSS 2.3 now (the was planned for arrival on November 16th). Another game where DLSS 2.3 should be available is Doom Eternal, where it should have a similar effect of diminished ghosting.
Nvidia has shown an example of the handling of flying embers in Doom Eternal. DLSS 2.1 has been mishandling them and produced long blurry trails instead of singular glowing particle. It seems the AI model or the temporal reconstructing element failed to properly motion compensate the embers in the consecutive frames on the dark background. Small moving particles seem to have been a troublesome case for the temporal reconstruction in general, and DLSS 2.3 could seemingly be helpful in this case.
Nvidia Image Scaling or Spatial Upscaling
The second part of the news is not directly about DLSS anymore. Nvidia has now updated the “ordinary” upscaling or resampling algorithm that can be used through its drivers. This does not use a neural network (AI) based interpolator like DLSS, it is a conventional “2D” spatial algorithm instead. Nvidia calls it “Nvidia Image Scaling” or, for purposes of comparison with DLSS, “Spatial Upscaling“. The company hasn’t presented this kind of upscaling much lately, with their emphasis on DLSS, but that’s changing now, with a new version shown and introduced on 16th.
The point of this simpler scaling algorithm is to be usable almost everywhere – it doesn’t need any tensor cores (it runs on general-purpose GPU shaders) and it doesn’t depend on motion vectors (so it won’t suffer from artifacts like ghosting). The dependence on motion vectors and past frames is something DLSS 1.0 also doesn’t entail, but it still has a dependency on tensor cores (DLSS 1.0 doesn’t run on anything other than GeForce RTX 2000/3000 cards). The AI filter also tends to change the image content more, plus it has perhaps more complexity. But in any case, Nvidia probably sees an opportunity for regular upscaling now too. In fact, the technology presented in this way is probably a close counterpart of AMD’s FSR (FidelityFX Super Resolution) upscaling.
Open source and universal solutions
And in fact, it’s also a competitor. Nvidia states that this scaling will be similarly freely available to developers (the code should be open sourced on Github under the MIT license) and should work on competing GPUs and platforms, developers will be able to use it even on consoles (which have AMD GPUs). Apparently the integration into games is also supposed to be very easy, with plugins for the engines and developer tools available.
So Nvidia Image Scaling is probably very much an alternative that Nvidia offers to game developers in place of FSR, although it’s hard to say whether it’s explicitly about beating or replacing competitors’ solutions. Both could probably be offered for use in games in theory, but since both algorithms perform the same role, then it would probably only have the slight benefit of allowing the player to choose between two versions of the filter, where each has a slightly different image (and presumably performance impact). But anyway, this method of upscaling should have the same advantages as FSR, i.e. simplicity, universal operation on all GPUs, portability and multiplatform support.
This new version of Image Scaling is said to have evolved from the Nvidia Image Sharpening feature that was available from the Nvidia Control Panel and could be used to combine upscaling and sharpening already. But this week, a new version was released with GeForce GameReady 496.70 drivers that has an improved algorithm that Nvidia says should have higher performance and better quality.
Like FSR, Nvidia Image Scaling consists of a scaling algorithm (“Spatial Scaling”) and a sharpening filter (“Sharpening”) applied to improve the appearance of edges. The resolution at which the game is rendered can be adjusted, as can the strength of the sharpening filter, using a slider. While the approach AMD takes with FSR is emphasizing a few presets (although the algorithm likely allows free parameter settings as well), Nvidia does not present any presets, just free adjustments.
So there is more than one set of image quality and effects under the Nvidia Image Scaling umbrella and character of the output depends on both of these parameters (which is good to keep in mind when comparing images between these different technologies).
Better quality than with FSR?
According to Nvidia, this Spatial Upscaling is the highest quality spatial resampling/upscaling algorithm of its kind for games. It doesn’t logically have the capabilities of DLSS 2.0 (Nvidia doesn’t seem to consider comparing it to DLSS 1.0), but according to Nvidia the image quality should be higher than AMD FSR. The filter used is a 6-tap filter using directional scaling with four selectable kernels, and the sharpening algorithm is adaptive. However, both operations (scaling and sharpening) are done in one pass for performance reasons. All operations are of course performed by shader code on general purpose compute units. We don’t know if they use FP16 calculations for speedup.
Based on the cited number of taps, the algorithm might be more computationally intensive than FSR, which would fit with the fact that Nvidia says the output image quality should be higher. FSR should use a 2-tap filter (EASU), which is probably targetting higher speed more aggressively. However, in addition to producing sharper output, more taps also increase ringing when used in resampling algorithms. AMD’s FSR/EASU has added additional processing that act as a simple anti-ringing filter. Whether there is also some anti-ringing in Nvidia’s algorithm, we don’t know. If not, then it’s possible that Image Upscaling might have a more sharp, but at the same time slightly less clean image.
It is quite common for different resampling algorithm methods to aim at different trade-offs. Ringing manifests itself as ripples around edges. Sometimes they are masked because they pass as sharpness or graininess of the texture, but it’s quite often that these artifacts become a distracting fault.
However, the quality of both solutions will only be possible to assess after more independent testing. For example, Nvidia showed a comparison in Necromunda: Hired Gun, where Nvidia’s Image Scaling algorithm scaled from 2954×1662 pixels to 4K slightly faster than AMF FSR Ultra Quality, but it probably wasn’t exactly the same job because Nvidia used the upscaling option forced from the drivers, so post-processing ran at a lower resolution than with AMD FSR. This might not have a big performance impact, but we don’t know yet if Image Scaling, for example, has better results on GeForce cards (and AMD FSR on Radeons), which could be due to differences in how the two companies’ GPUs have FP16 precision operations implemented. FSR is apparently optimized for AMD’s Rapid Packed Math technology and runs slower on GeForce, Nvidia on the other hand might have shaders written for optimal performance on Turing/Ampere architectures.
Global version for all games…
The Image Upscaling filter can be used from the Nvidia Control Panel, but it can also be easily enabled from GeForce Experience. Using these options, it can be used globally even for games that don’t have direct support for it. It’s used for non-selective postprocessing of the entire final image in this case.
The disadvantage is that this method is apparently not compatible with many image capture and recording software tools. Because scaling and sharpening occurs as the last step after rendering and it urns within the GPU driver, your footage may end up being captured at a lower rendering resolution instead of at the output resolution (and it will probably be taken without the full filter effect). Thus, you may need to use a hardware card that stores video from the HDMI video output to record game video.
… or integration into games like FSR
The other way this upscaling can be used is by developers integrating it directly into the game, like it typically is with AMD FSR. This will be possible using the Nvidia Image Scaling SDK libraries. If Nvidia Image Scaling is integrated right into the game, while it will use the same algorithm that can be globally “forced” in the drivers, it will have some extra benefits. Namely, instead of applying it to the entire finished image, the scaling filter will be placed in the rendering pipeline at a more appropriate place – after anti-aliasing and color mapping, but before further post-processing and also before the game’s GUI/HUD is rendered. This way it won’t be affected by effects of the upscaling and sharpening, which can have quite a detrimental effect on the HUD graphics, or on text.
It’s also important that in this cases, Nvidia Image Scaling scales and sharpens before post-processing filters like adding noise happen. Sharpening noisy image is again something that can lead to problematic artifacts, it often doesn’t look good. Conversely, adding noise to an upscaled image can actually be quite beneficial because it masks various artifacts and shortcomings produced by the upscaling. AMD FSR does exactly the same, so if Nvidia Image Scaling is integrated directly into the game, both these solutions combining scaling and sharpening algorithms will be on equal footing (before they get to show off their effect).
Solutions for higher resolutions
Nvidia recommends using Image Scaling only for 1440p and 4K output resolutions and with relatively lower scaling factors, since in these conditions the difference in quality versus native rendering isn’t as large. For gaming with 1080p output resolution, it won’t be very useful anymore, as the source rendered image will already be quite lacking in detail. DLSS 2.x is viable at low resolutions since it is saved by that temporal reconstruction from previous frames; that is not present in NIS and the results will therefore be bad. But the same is 100% applicable to the competing alternative (FSR), it is an inherent feature of the approach used.
ICAT: easy comparison of images or recordings
For the purpose of comparing quality of game fame screenshots (or captured videos), Nvidia has published the ICAT tool that allows you to compare game images or videos. Up to four sources can be compared, side by side, or you can use a before/after slider to help you spot changes.
The program also allows you to align images of different resolutions or videos temporally (if the length does not match). You can also zoom to a specific location and analyze the differences.
Nvidia uses this solution internally for their own development, but as of this week, ICAT should be released publicly. So you will be able to use it as well.
By releasing this tool, Nvidia wants to motivate reviewers to not only look at performance when testing different upscaling techniques, but also to compare how the image looks, i.e. not to compare methods like AMD FSR or the now new Nvidia Image Upscaling with a particular DLSS mode if the DLSS has higher quality.
Help for equalizing image settings in performance tests?
However, this will not be all that easy to implement, as image quality assessment can be a very fuzzy area. Aside from personal preferences, people can display blindness or on the opposite side high sensitivity to a certain kind of artifact in an image and not another kind. And it can be very difficult to consider which artifacts are worse and which are (relatively speaking) better.
When comparing different upscaling techniques, including those using artificial intelligence, there are often cases where one setting/solution is visibly better in some image area or scene, but other scenes or parts of the image may instead show degradations. So what exactly is equivalent quality is going to be pretty hard to decide.
Source: Nvidia
English translation and edit by Jozef Dudáš, original text by Jan Olšan, editor for Cnews.cz
- Contents
- A new version of DLSS is here, together with an alternative upscaling that works on all GPUs
