03 April 2020

Win10 Bug: .evtx Files Rapidly Fills C: Free Space


This very nasty Windows 10 bug has been around for over two years at least; crippling, often associated with "Feature Updates", i.e. new versions of Windows 10, and still there from at least as far back as 1803, to current 1909.

Note that each of those hyperlinked words in the previous paragraph, is a link to a forum thread on this issue, so while it not be common enough "all at once" to attract attention and get fixed, it's always around, and always eating systems - no magic bullets, typical advice is a shrug and "just re-install Windows" or equally-hi-impact brute-force "fixes".

I suspect it's a generic category of bug within the Microsoft Store and UWP Apps subsystem, regardless of which of these Apps is the "cause" of the problem on any particular system. Never use Apps, Microsoft Store, or UWP stuff?  Too bad, that "updater" or "installer" will still flood your drive with pointless error messages and make it impossible to use your PC.

What you will see

The bug presents as an inexplicable runaway filling of free space on the C: drive, no matter how much free space you had there before.  Disk Cleanup doesn't show the bulk that needs to be cleared; Settings, Storage Sense may pop up and show the material as in "Temporary Files", yet not in any of the checkbox sub-categories offered to be cleared.

Users will then turn to Windows Directory Statistics (WDS) and/or TreeView or similar, and may get side-tracked into arguing which is better, etc.  I use WDS, and it will show a massive "Unknown" accounting for the lost storage space.  If I right-click WDS and "Run As Admin", I will then see this bulk as thousands of small files (between 68k and 20M) within %WinDir%\Temp

Most of these will be .evtx files, as "opened" by Event Logger; the rest will be .txt files, and these will be date-stamped as being spawned several times a minute, if not every few seconds, until the free space is exhausted.  Deleting these doesn't help; they will immediately start flooding again.

What appears to be cause


AppXSvc is a Windows service that "deploys UWP Apps"; I found little documentation of the service, but finds this Fortnet zero-day alert, FWIW.  Looking at...

Computer\HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\AppXSvc

...via Regedit, we see the following settings:

Start = 3, i.e. Manual
Type = 32 (0x20), i.e. can share address space with other services of the same type
ErrorControl = 1, i.e. warn but do not abort starting Windows

So, something else starts it all the time, as it's always running yet not set to start automatically.

Recovery actions, as seen via the Services UI (where everything is "greyed out"), appear reasonable...

1st failure: Restart the service
2nd failure: Restart the service
3rd and subsequent failures: Take no action

...so if this service repeatedly falls on its ass, it should stop running, limiting the impact to at most 2 sets of .evtx and related error logging files in %WinDir%\Temp.  So what's going wrong, here?

I suspect whatever is starting this wretched (and for most of us who only use "real" Windows preograms, totally useless) service isn't paying any attention to those Recovery actions, and is just endlessly banging away, restarting the service "Manually".  If each time the service considers itself to have been launched for the first time, it will always "Restart the service".

Also, in one of the two cases I've seen first-hand (manual Media Creation Tool upgrade to 1909 from inside Windows), I noticed odd behavior in the Services UI.  Specifically, the service Properties (as seen via Services UI) offered to Start the service, even though Ctl+Alt+Del Task Manager showed it to be still running.  If the service and/or managing code gets confused about whether it's running or not, that too may screw up the "FFS stop trying to start the &^$& thing, it's already failed 3 times" logic.

The other system I saw and managed via TeamViewer, was after an auto-upgrade to 1903.  On that system, setting the AppXSvc Start value to 4 (Disable) will hopefully kill the service, plus I wrote a brute-force batch file set to "Run As Admin" as follows:
@Echo Off
Set Secs=15
Set Mask=*.evtx
Echo.
Echo Deleting %WinDir%\Temp\%Mask% files every %Secs% seconds...
:LoopForever
    Echo.
    Del %WinDir%\Temp\%Mask%
    Echo.
    Timeout 15
GoTo LoopForever
None of this a proper fix, especially on an SSD where you don't want tends of thousands of pointless file writes every hour or few.  At the basic level, Microsoft needs to muzzle the UWP App subsystem so it doesn't stomp all over the system willy-nilly, and ensure that every logging process has a basic LIFO clue so as not to consume all available storage space.  A specific fix would be nice, too, but we also need a more respectful vendor-to-user relationship.

PS: What is it with HTML text editing (e.g. in Blogger) that messes up blank line spacing around subheadings, etc.?




1 comment:

cquirke said...

Following up; I have an afflicted Win10 laptop set aside and ready for testing, as this is an uncommon bug and thus may be hard to repro(duce).

ne solution mentioned in the various threads I read on this bug, is to check Settings, Privacy, Background Apps and turn all of these off. I checked that on this afflicted laptop, and they are already all turned off.

I last worked on this laptop months ago, hit the problem, and must have worked purely "from orbit" (WinPE boot, off Internet) since then.

Today, I tried Win10 PE DVD boot, which ignored CMOS Settings to boot DVD before hard drive, and booted to the Win10 "choose OS" boot menu (which is populated by Windows 10, EaseUS To To Backup PreOS, and the same Win10 installation set to boot as Safe GUI and Safe Cmd). Laptop was again kept unplugged from Internet, no auto-WiFi, etc.

On that menu, I booted into Safe Cmd, and from there, did the following:
- check %WinDir%\Temp via Dir *.evtx /a /s
- this showed a load of files, 60G free space
- repeating this showed no shrinking of free space, nor increasing file count
- implies "the bug" is not in effect
- Regedit, HKLM\System..CCS..AppXSvc
- Start was = 3 (Manual), OK
> set Start to 4 (Disable)
- on sole user's Desktop, create but not run a batch file, to...
- Del %WinDir%\Temp\*.evtx, then Timeout 15, loop repeatedly
- did NOT run this batch file, to preserve "forensics"
- for emergency use after normal boot

Then I well-exited Safe Cmd via Ctl+Alt+Del, Shutdown, Restart; the next boot I accepted the default to boot Windows 10 normally, still offline. This boot shows the expected 57G or so free space, which did not decrease as would have happened if the bug were still in effect. So I did not need to run the emergency batch file, and did not do so.

This is the condition in which I left the laptop, in case Microsoft get back to me and wish to study the stricken system further, as an example of this nasty bug.