Tuesday, June 23, 2015

First of all, I had this included in my post since the get-go, but it was overlooked as it wasn't at the beginning of the post. With that said, I'm moving it here, and clarifying a bit more. I was not the sole person involved, it was a multiple-person discovery. Here were the people involved:

wavly - The user that had the problem, and the reason we had anything to even discover in the first place.BrianDrab - Assisted wavly in their Windows Update problem, and investigated with us why it was resetting and disabling the user from keeping it the setting they wanted to.niemiro - Was largely involved in the discovery by investigating/reverse engineering SW Update.zcomputerwiz - Was largely involved in the discovery by suggesting registry auditing.tom982 - Was largely involved in the discovery by investigating/reverse engineering SW Update.Tekno Venus - Was largely involved in the discovery by investigating/reverse engineering SW Update.Me (Patrick Barker) - Was involved in the discovery by further reverse engineering and investigating SW Update and its behavior after the above people, and creating the blog post.

I've also seen a few (very few) articles even say I was the individual who was helping with the Windows Update issue(s) wavly was having. For the record, I personally don't know a damn thing about the technicalities of Windows Update, how to fix broken updates, etc. The user that was assisting wavly with the Windows Update issue(s) was BrianDrab, as I had mentioned in this post, just apparently not mentioned enough (or clearly enough). I merely further investigated and reverse engineered SW Update, and brought Disable_Windowsupdate.exe and its silent behavior to light.

Onto the post...

On my home forum Sysnative, a user (wavly) was being assisted with a WU issue, which was going well, aside from the fact that wavly's WU kept getting randomly reset to "Check for updates but let me choose whether to download or install them" after every single reboot of Windows. It was figured out eventually after using auditpol.exe and registry security auditing (shown below later) that the program that was responsible for resetting WU was Disable_Windowsupdate.exe, which is part of Samsung's SW Update software.

SW Update is your typical OEM updating software that will update your Samsung drivers, the bloatware that came on your Samsung machine, etc. The only difference between other OEM updating software is, Samsung's disables WU from working as the user intends it to.

Do note that it does check for a Samsung environment, and if one is not detected, the program will in general run really buggy. A lot of its features won't drop or work as intended either, which is why a lot of manual work needs to be done to investigate this program.

What devices does SW Update run on?

Samsung notes:

SW Update allows you to download and install the newest drivers, updates, and software for your Windows PC.

So most likely only desktop and laptop type devices that run the Windows OS.

Uninstalling SW Update

UPDATE: I've received confirmation from a Samsung NP350V5C-A06UK user (Windows 8.1) that uninstalling SW Update via the Programs and Features list does in fact remove all of its installed parts, including the service. With that said, it does indeed stop resetting Windows Update's settings after reboots. So the solution to having SW Update constantly reset your Windows Update settings and disabling it from working as you intended, is to simply uninstall SW Update.

-- Initially today I had this saying it did not stop it from resetting, but wavly got back to me and said they were mistaken.

Here we can see some more information, such as its agent's sleep is set to 300 seconds, its first execution timestamp, and the creation of the "SW Update" service. I'll break down the service stuff:

Type (0x00000110): As far as I know, this implies it's a Win32 program that can be started by Windows' Service Controller, and that it obeys the service control protocol. This type of Win32 service runs in a process by itself.

Start: (0x00000002): This implies it's set to load or startup automatically for all startups, regardless of the service type. Its loader is the Service Control Manager, where as the 0x0 (boot) would be the kernel, and 0x1 (system) would be the I/O Subsystem.

ErrorControl: (0x00000001): This implies if the driver fails to load or initialize, proceed regardless with startup, however display a warning.

We note that its ImagePath is:

C:\ProgramData\Samsung

If you show hidden files & folder and navigate here, you have two folders - "SW Update Service", and "SWUpdate". If you actually have a Samsung machine, you instead have two "SWUpdate" folders, and they both contain XML files. If we take a look at one (BASW-A0394A05_1B33BCEB.xml):

We can see its using the xcopy command to inevitably "drop" Disable_Windowsupdate.exe in \ProgramData\Samsung. %ALLUSERPROFILE% is an environment variable for \ProgramData on >Vista, and \Documents and Settings\All Users on XP.

We can confirm this by checking ourselves:

Note that the exe is actually signed by Samsung themselves:

So a big thing is the question as to how this persistently
resets Windows Update from working after you change it and reboot, and it's
actually not SW Update. SW Update is basically just there to genuinely
do its job, which is to update Samsung's drivers, software, etc.

What's actually causing Windows Update to persistently become reset and not allow the user to set it the way they want it to, is the fact that Disable_Windowsupdate.exe creates a scheduled task that runs at every logon to ensure that Windows Update is indeed consistently reset to "Check for updates but let me choose whether to download or install them".

So first off, as I noted earlier in the post, if you're trying to run the Samsung update software + disabler, etc, on a non-Samsung environment, it's really buggy. My VM was going through convulsions trying to just take screenshot examples after frequent restarts, etc, so there's a few minutes in between each screenshot.

Here's what WU looks like directly after installing SW Update:

Note that it's set to 'Check for updates but let me choose whether to download and install them'.

Let's change it to 'Install updates automatically (recommended)':

Cool, let's restart and check again.

Oh, this doesn't look right. Let's check the settings:

Uh...

There's a bit more to it that I'd like to get to eventually, but I suppose this is enough to get the point across. Anyway, with this known, I decided to try Samsung's chat to see if they knew of it:

You are now chatting with 'Rep'. There will
be a brief survey at the end of our chat to share feedback on my performance
today.

Your Issue ID for this chat is *purged*.

Rep: Hi, thank you for reaching out to
Samsung technical support. How may I assist you?

Rep: SW Update tool helps in automatically
detecting the hardware on the laptop and installs the supporting drivers for
them. I am afraid; this tool has directly no effect on the registry of your
laptop or Windows Updates.

Rep: When you enable Windows updates, it
will install the Default Drivers for all the hardware no laptop which may or
may not work. For example if there is USB 3.0 on laptop, the ports may not work
with the installation of updates. So to prevent this, SW Update tool will
prevent the Windows updates.

So thanks to Rep over at Samsung, we now know Samsung's motive to disabling WU.

OEMs, come on... has Superfish taught us nothing?

Upload/report this as malware to Microsoft/MSRC, etc, because that's exactly what it is. Why would you ever tamper with WU in such a fashion (or in general), in a way a generic user cannot control, leaving them vulnerable?

x86 MD5

3727acd09814c0d5ce8fd3d6be705254

x64 MD5

d0a3a1c266845ef1e2cdf65c226facae

x86 SHA-256

61da7461e8a60a20e9d2b595edff89a0898c8f2d47d2be847c8a7ceff0fc4bd4

x64 SHA-256

7b9547acf8b3792b48fe5a02f7d5f3e0dfba8e57055d60f479bb8adfed99871c

Small edit: I edited out the Samsung rep's real name to just 'Rep'. It was clearly a tier 1/2 support just doing their job, and I of course don't want them getting in any trouble since this appears to be blowing up. After all, as I said, this isn't their fault at all.

Update

According to a few news articles, here's Samsung's latest statement:

"It is not true that we are blocking a Windows 8.1 operating system
update on our computers. As part of our commitment to consumer
satisfaction, we are providing our users with the option to choose if
and when they want to update the Windows software on their products,"
said Samsung.

"We take product security very seriously and we
encourage any Samsung customer with product questions or concerns to
contact us directly at 1-800-SAMSUNG."

I
don't understand what this statement is implying, and it may have been a
loss in translation between whichever article reporter/editor got the
statement from Samsung, because I never implied it specifically blocked a
"Windows 8.1 OS system update", just that their SW Update software is
preventing Windows Update from automatically installing updates, and
forcing the user tohave it set to "let me choose whether to
download and install". If you attempt to change it, it'll switch right
back on a reboot. Microsoft has openly stated that they do not like the
fact that it's persistently changing, or even existing in the first place without the user's consent. It's disabling Windows Update from working as the user intends it to.

However you look at this, Samsung's solution to what we can guess is a device driver workaround was not done in the best way, or a safe way. I mean, come on, the exe is named Disable_Windowsupdate.exe.
In any case, if it appears I am acting as an enemy to Samsung, I'm not. I'm just a
22 year old cashier with a love for Windows internals that found a
security risk for Windows' Samsung users with a few others. That's it.

Update #2

According to a few news articles, here's Samsung's latest statement:

“Samsung has a commitment to security and we continue to value our partnership with Microsoft. We will be issuing a patch through the Samsung Software Update notification process to revert back to the recommended automatic Windows Update settings within a few days."

I'm very glad Samsung is committed to implementing a resolution to this issue so soon. Ultimately, in a perfect world, I hope OEMs will learn from Superfish/SW Update, as it would be disheartening to see a similar issue occur in the future. I feel OEMs need to disclose whatever they intend with their users with their software, and if possible, giving them a choice.

If this is done, it's not "under the table" anymore so to speak. If Samsung's users were notified in the first place that their Windows Update settings were being actively modified, then even though it still potentially may have been a question of poor implementation/methods, it probably wouldn't have been seen as malicious or questionable behavior in the first place as it would have at least been known.

Wednesday, May 20, 2015

Today I'll be investigating an issue involving Bitdefender, which is turned out to be a Windows bug/issue more than Bitdefender, although there are developmental changes that could be made aside from a hotfix to stop this issue. Bitdefender's 0x4A bug check issue has been prevalent for quite awhile now, but there's little to no documentation on solving it or what's causing it, just a few things to try like updating Bitdefender, uninstalling it, etc

Throughout all of the 0x4A Bitdefender related crashes, the NT kernel was labeled as the fault:

Probably caused by : ntkrnlmp.exe

Given we're seeing ntdll, we can likely imagine the reason for the NT
kernel being blamed as being the fault of the crash is because most of
the API from ntdll is implemented in the NT kernel variants, with this being ntkrnlmp.exe because this system has a multi-processor without physical address extension configuration.

All we can see if we're exiting user-mode code using the KiSystemServiceExit function, and we go off the rails right there - KiSystemServiceExit+0x245. This function is in charge of handling the various call-styles used to enter kernel-mode, and then returning to user-mode.

With that said, let's switch to the other processor within the system that was involved and see what's going on at the time of the crash. To find out the active processors on the specific system, we'll use !running:

I used knL as opposed to the other stack dump commands as I wanted to get the frame # feature for reference reasons.

Starting at frame # 2a, we can see the NtDeviceIoControlFile function calls IopXxxControlFile. The latter function appears to be undocumented, so I'm unsure as to what it does. What I do know is, the NtDeviceIoControlFile function is ultimately used to build descriptors for a driver. I imagine it's using the IopXxxControlFile function to aid in passing such to the driver.

Also, for what it's worth, although NtDeviceIoControlFile has since been superseded by DeviceIoControl, the former native function provides more information that may be beneficial to the caller (especially for debugging purposes). This is likely why Bitdefender chose to use the former function instead.

So after neatly putting together this disassembly of sorts, we can see that this is indeed how the NtDeviceIoControlFile function is passing on the buffer and such to the driver.

The IoAllocateMdl function in this specific case is used to ultimately associate the MDL with an IRP, which is why we call into the IoAllocateIrp function, to of course assign the IRP. IoGetAttachedDevice is called likely to return a pointer to the devobj, with help from the IoGetRelatedDeviceObject function to probably obtain the devobj from the file system driver stack.

ObReferenceObjectByHandleWithTag is called to increment the reference count of the object, and to write a four-byte value known as a "tag" so it can support object reference tracing for debugging purposes. Finally, the ProbeForWrite function is called to ensure that a user-mode buffer meets the following:

Resides in the user-mode portion of the address space.

Is writeable.

Is correctly aligned.

As all appears to have went well, we can see the driver we were ultimately building and passing descriptors to/for was bdfwfpf.sys, which is Bitdefender's firewall filter driver. As it's a driver in charge of a firewall, it of course uses the WFP API (Windows Filtering Platform) to achieve its goals (not just filtering and monitoring).

We can confirm this easily by looking at the very first driver/function call after Bitdefender's firewall, which is fwpkclnt.sys. Specifically, Bitdefender's firewall driver called it to inject new/cloned data to the data stream. Directly afterwords we have calls from the Network I/O Subsystem to continue the injecting, which is because fwpkclnt.sys exports kernel-mode functions, as opposed to fwpuclnt.dll which exports and handles the user-mode side.

To handle and/or continue the injection into the data stream, it looks like DPC(s) are used to handle it by calling KeInsertQueueDpc to create a queued DPC for execution.

-- After discussion with a fellow kernel-debugger friend of mine, we also
thought that the IRQL was possibly DISPATCH_LEVEL due to the multiple
injections, etc, therefore Windows deferred it to a DPC. Given this
possibly being the case, when the DPC was to be worked on, the system
service finished but the IRQL is still DISPATCH_LEVEL. Since that was the case, we get a bug check.

We continue through netio.sys' functions regarding the data stream injection, ultimately injecting the request to the stack and going through a few tcpip.sys functions.

To continue sending the data along, NDIS' NdisSendNetBufferLists function is called, and NDIS' filter driver (which I believe is pacer.sys), called NdisFSendNetBufferLists to send the list of network data buffers back to Bitdefender's firewall driver.

In order to do so, NDIS needs to call the HAL, which we can see through the function HalBuildScatterGatherList. What is supposed to happen next is, the HAL builds the scatter/gather list, and we go on through various registered miniport functions. However, this did not happen, and we go off the rails on frame #00 with a call to the miniport driver.

We get a lot of good information, and can see that Bitdefender's firewall filter driver is/was involved with this miniport. We know this, because we saw it all happening in the stack, but this just confirms it.

What appears to be happening here is multiple NBLs in a chain are being passed, the FwpsStreamInjectAsync0 function is called to pass Bitdefender's data, and then the chain is broken as the call goes on (see the NBL next member is zeroed out/null).

Possibly a fix (in Bitdefender's case) is to avoid multiple injections inside the stream callout routine, possibly taking NBLs in a chain and calling the FwpsStreamInjectAsync0 function just ONCE for each callout routine execution. Unsure, kernel development isn't my strong point : )

A fix for user's is to install this hotfix and hope it works, as it should. Overall, maybe Bitdefender instead of making any developmental changes could just raise awareness for this issue, like creating a well explained documentation page with a link to the hotfix. I think developmental changes would be a better workaround.