OAuth using PowerShell and certificate Service Principal to Azure Function App



The NDIS Blog: We're back!


Now with witty subtitles

Lately things have been quiet on the NDIS blog.  But that's about to change, because we've got some new blog posts lined up for you.  We'll start things off this week with a discussion on OID requests.

If you would like to suggest a topic, please leave a note in the comments.


TMF download page


Are you targeting Windows 8 or Windows Server 2012?  You don't need anything from here!  These operating systems already include all the TMFs you'll need in the PDB from the Microsoft Symbol Server.

For Windows 7 and Windows Server 2008 R2, here is a copy of the TMF decoders for NDIS.SYS:

→ Download here.

This file was last updated on July 24 2015, and includes traces for the following versions of NDIS:

  • RTM
  • KB977927
  • KB981765
  • KB2471472
  • KB2482122
  • SP1
  • KB2624668
  • KB2697537
  • KB2688892
  • KB2719857
  • KB2729608
  • KB2894906
  • KB2974617
  • KB3014793

NDIS traces messages at the following verbosity levels

Level Name Description
1 Severe errors NDIS doesn't use this level much. It's mostly interchangeable with the Errors level.
2 Errors An error that will definitely cause something to break. Example: miniport installation will fail because a registry key is missing.
3 Warnings An unusual condition that might need fixing. Example: miniport indicated a malformed status indication, and NDIS dropped it.
4 Info Informational messages. Example: miniport is going into low power mode now.
5 Verbose NDIS doesn't use this level much. It's reserved for a few messages that aren't usually useful

 

NDIS recognizes the following flags:

Flag value Name Description
0x00000001 Initialization  
0x00000002 Configuration  
0x00000004 Send The Send and Receive traces can be very noisy. Only enable them if you have a specific problem with the datapath. If possible, limit the number of packets sent while tracing these.
0x00000008 Receive The Send and Receive traces can be very noisy. Only enable them if you have a specific problem with the datapath. If possible, limit the number of packets sent while tracing these.
0x00000010 Protocol  
0x00000020 Bind  
0x00000040 Bus Traces miniport's access to underlying hardware, if the miniport uses NDIS APIs to do so. (WDM miniports will not go through NDIS to access their hardware).
0x00000080 Registry Traces how external drivers use the Ndis Configuration APIs
0x00000100 PnP Event Memory Allocation (PnP Event on Windows 7 / Windows Server 2008 R2 and later; Memory Allocation on Windows Vista / Windows Server 2008)
0x00000200 Light-Weight Filter  
0x00000400 OID Request  
0x00000800 Work Item Traces the occasional maintenance that NDIS needs to do on work items, including miniport reset
0x00001000 PNP  
0x00002000 Power Management  
0x00004000 Protocol Reference Counting (Windows Vista / Windows Server 2008 only)
0x00004000 Selective Suspend (Windows 8 / Windows Server 2012 and later)
0x00008000 Lock (Windows Vista / Windows Server 2008 only)
0x00010000 Reset  
0x00020000 WMI  
0x00040000 CoNDIS  
0x00080000 Reference Counting  
0x00100000 Memory Allocation  
0x00200000 NDISIF  
0x00400000 NDIS Port  
0x00800000 Status Indication  
0x01000000 Receive Queues (VMQ) (Windows 7 / Windows Server 2008 R2 and later)
0x02000000 SR-IOV (Windows 8 / Windows Server 2012 and later)
0x04000000 Miscellaneous Features that are not generally of interest to third party developers, like CEIP bookkeeping. (Windows 8 / Windows Server 2012 and later)
0x08000000 Bind Engine State changes for protocol and filter bind, unbind, pause, and restart (Windows 8.1 / Windows Server 2012 R2 and later)

ndistmf.zip


Troubleshoot a Windows bluescreen, a.k.a bugcheck, a.k.a blue screen of death


I have read a lot of posts in multiple forums on the internet where people ask "My machine keeps bluescreen-ining, what do I do?"... a common response is "Reinstall Windows and the problem will most likely go away". This is a wrong answer because if you simply reinstall Windows you don't know what caused the blue screen and if you don't know what caused it you cannot prevent it from happening again. So the correct answer is: find out what driver is causing the blue screen and then either a) stop using the driver or b) call your PC/device manufacturer and ask them for a fixed driver.

This post summarizes what a technically savvy user can do to troubleshoot and mitigate a bluescreen on his Windows PC.

Here are the quick steps:

1. Install the Windows Debugger

2. Load C:\Windows\MEMORY.DMP in the debugger

3. Load the debugging symbols for the crashing OS

4. Issue the '!analyze -v' command and read the output. It will tell you what driver caused the crash among many other useful things it will reveal.

Here is the long version:

What is a bug check?
When Windows encounters a condition that compromises safe system operation, the system halts. The system halt is a direct result from a kernel mode component (driver) calling either the KeBugCheck(...) or KeBugCheckEx(...).  This condition is called a bug check. It is also commonly referred to as a system crash, a kernel error, a Stop error, a Blue Screen, or a Blue Screen Of Death (BSOD) .

Windows has default settings which control whether the system will automatically restart after a bug check, whether the system will write a mini dump, kernel dump or a full memory dump and wheter it should overwrite the previous dump file. The dumps differ in size and thus they differ in the ammount of information saved at the time of the crash. You can view and change these settings by going to Control Panel -> System Properties -> Advanced tab -> 'Startup and Recovery' Settings. Here is the UI for the System Properties on Windows Vista:

 System PropertiesStartup and Recovery UI - Windows Vista

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

As you can see in the screen shot on the right, on Windows Vista the default is for the system to automatically reboot after a bug check and save a kernel memory dump. On Windows XP the default is for the system to automatically reboot after a bug check and save a mini dump.

The key takeaway is that : a) if your machine restarts when you didn't expect it to, the system bug checked; b) in order to figure out what driver caused the bug check it is best that you have a kernel dump (this is the default on Vista).

How do you diagnose what driver caused the bug check?

The steps are as follows:

  • Save the C:\Windows\memory.dmp file somewhere handy (note that the drive letter might be different, depending on which partition you installed Windows on) 
    • Note that depending on your system settings you might have a 'mini-dump'. To find all the dump files on the system do 'C:\>dir /s *.dmp'
  • Download the Windows Debugger. I highly recommend that you spend some time reading through the help file. Specifically, read the 'Debugging Techniques -> Elementary Debugging Techniques' section
  • Start WinDBG (debugger with UI front end) or alternatively you can use the command prompt debugger called 'kd'. 'kd' is located in the directory where you installed the debugger. Here is what you should see...

WinDBG Startup Screen 

  • Load the memory dump from #1 in the debugger by doing: File -> Open Crash Dump and pointing to the MEMORY.DMP file.

 

  • Load the symbols for the crashing OS by doing: File -> Symbols File Path and point to the directory where you extracted the symbols package. Here is what you should see...
  • You can download the symbols package for your OS version from here. Note: Pay attention to the CPU architecture when downloading symbols. The symbols are different for x86, x64 and IA64.
  • in the lower right corner is the command line, issue the command !analyze -v and you should see plenty of information about the crash. Here is an example:

0: kd> !analyze -v
*******************************************************************************
*                                                                             *
*                        Bugcheck Analysis                                    *
*                                                                             *
*******************************************************************************

DRIVER_POWER_STATE_FAILURE (9f)
A driver is causing an inconsistent power state.
Arguments:
Arg1: 00000003, A device object has been blocking an Irp for too long a time
Arg2: 8445c700, Physical Device Object of the stack
Arg3: 84fe9428, Functional Device Object of the stack
Arg4: 870ddae0, The blocked IRP

Debugging Details:
------------------

DRVPOWERSTATE_SUBCODE:  3

DEVICE_OBJECT: 84fe9428

DRIVER_OBJECT: 85269e68

IMAGE_NAME: NETw3v32.sys

MODULE_NAME: NETw3v32

FAULTING_MODULE: 8c210000 NETw3v32

DEFAULT_BUCKET_ID:  VISTA_BETA2

BUGCHECK_STR:  0x9F

PROCESS_NAME:  System

CURRENT_IRQL:  2

LAST_CONTROL_TRANSFER:  from 82c1ef3e to 82cd53a9

STACK_TEXT: 
80803e1c 82c1ef3e 0000009f 00000003 8445c700 nt!KeBugCheckEx+0x1e
80803e78 82c24841 80803f5c 00000000 82cf8e20 nt!PopCheckIrpWatchdog+0x165
80803eb0 82c7365e 82d0c1c0 00000000 85523e08 nt!PopCheckForIdleness+0x2d5
80803f88 82c724e0 00000000 00000000 00401229 nt!KiTimerExpiration+0x60c
80803ff4 82c61aa9 82a63a48 00905a4d 00000003 nt!KiRetireDpcList+0xca
80803ff8 82a63a48 00905a4d 00000003 00000004 nt!KiDispatchInterrupt+0x3d
WARNING: Frame IP not in any known module. Following frames may be wrong.
80803ffc 00905a4d 00000003 00000004 0000ffff 0x82a63a48
80804000 00000000 00000004 0000ffff 000000b8 0x905a4d

---------

 In the example above the driver at fault was NETw2v32.sys. Note that the !analyze command does not always give the correct driver that crashed the system. For example if driver A corrupts driver B's data structure and driver B access that data structure and crashes the !analyze command might blame driver B where in reality driver A was at fault. So whenever memory corruption is involved !analyze cannot be trusted completely.

- Rade Trimceski

sys_prop.jpg


Using C++ in an NDIS driver


Are NDIS drivers allowed to use C++?

The first question is easy: can NDIS drivers be written in C++?  The answer: yes.  In this case, NDIS doesn't have any official stance on C++, so we just fall back on the WDK's general rules.  As of Windows Driver Kit 8, Microsoft officially supports using a subset of the C++ language in drivers.  ("Subset?  What subset?"  There's more precise information here.)

The inevitable follow-up question is more nuanced: should NDIS drivers be written in C++?  The answer is: it depends.  Here are some facts that will help you derive a more specific answer:

  • The NDIS API is a C API.  There is no NDIS API that magically gets better or worse when you're coming from C++ versus C.
  • The NDIS team has no future plans to make a feature that requires C++.  We are well-aware that many of our developers are dedicated fans of C, and have strong opinions on C++.  Don't worry — C isn't going anywhere.
  • The NDIS team may, in the future, add minor conveniences that only light up in C++.  For example, the WDK macro ARRAYSIZE is defined differently for a C++ driver, which gives it better abilities to detect misuse with pointers.  NDIS.H may start adding macros that offer minor improvements for C++ code, just like WDM.H already has today.
  • Several major IHVs build their production NDIS miniport drivers using C++.
  • Several major IHVs build their production NDIS miniport drivers using C.
  • Microsoft builds some drivers in C and some drivers in C++.
  • Our NDIS sample drivers are all in C.  (This is largely for historical reasons, as these drivers were created before C++ was officially supported.  If we were creating a new sample today, we'd consider writing it in C++.)

In summary, then, either language works fine, and it all comes down to a matter of your preference.


Using the checked version of NDIS.SYS


I assert that this is a good way to find bugs

Installing the checked version of the operating system is an effective technique to quickly find bugs in your network driver.  If you're not familiar with checked builds (and even if you are), you should read the excellent documentation here.  Seriously, read it; I won't repeat it here.

What do you get with the checked build of NDIS?

The main difference is that NDIS's implementation has (as of Windows 8.1) approximately 2200 extra asserts.  While some of these asserts verify NDIS's internal bookkeeping is consistent, many of them verify that your driver uses NDIS's APIs correctly.  For example, NDIS asserts the current IRQL is correct when each MiniportXxx callback returns, to help catch the class of bug where your miniport driver leaks an IRQLs or spinlock.

Prior to Windows 7, using the checked build of NDIS is also the only way to see NDIS's debug traces.  But as of Windows 7, these traces are now available from WPP, so there's no longer a need to use the checked build solely for tracing.

What's the downside of using a checked build?

There are two downsides to using checked builds: performance and false-positives.

Checked builds are noticeably slower.  But they aren't as bad as you might think.  We still compile checked builds with most compiler optimizations enabled, so the only slowdowns are a few extra verifications here and there.  Still, the operating system has zillions of assertions, so those do add up.  You definitely don't want to use a checked build for any performance-related work.  But they're just fine for initial work on a new feature or functional testing.

False positives are also a problem.  Sometimes you'll see assertions that fail for reasons that don't seem to be related to your driver.  When you see an unfamiliar assertion, you'll first want to spent a moment to convince yourself that the assertion failure is really caused by your driver.  For example, if there's an assertion in win32k.sys about an invalid HRGN, that's probably not caused by any network driver.  Prior to Windows 8, the operating system was kind of "noisy"; a nontrivial percentage of its assertions would fire for benign reasons.  We worked hard to clean that up in Windows 8, so the asserts have a better signal-to-noise ratio.  (Like many Windows engineers, I used a checked build of the OS as my primary workstation for some time during Windows 8 development.  That was fun.)

If you discover an assertion in NDIS.SYS that you believe is a false positive, please let me know here and I'll try to clean that up.  (Unfortunately I'm not knowledgeable about non-networking drivers, so I can't promise I can help you with any random assertion that you come across.)

How can you get the checked build of NDIS?

MSDN has the story on how to download a copy of the checked build.  From there, you have two options:

Since MSDN already explains the first option, I won't repeat those instructions here.  Let's talk about the second option: how to selectively replace a few drivers.

First, identify the drivers that you want to replace.  Here's a table of drivers you can consider replacing:

Driver When to replace
The kernel & HAL Always
NDIS.SYS Always
TCPIP.SYS Miniport, LWF, and WFP callout drivers
NETIO.SYS (Windows Vista and later) Whenever TCPIP.SYS is replaced
FWPKCLNT.SYS WFP callout drivers
(your bus driver, e.g., PCI.SYS) Miniport drivers
NWIFI.SYS Native 802.11 drivers
VWIFIBUS.SYS VWIFIFLT.SYS VWIFIMP.SYS Native 802.11 drivers that implement MAC virtualization (WFD or SoftAP)
NDISUIO.SYS WWAN drivers
WMBCLASS.SYS WWAN drivers that implement the class driver model
VSWITCH.SYS Hyper-V extensible switch extension

Keep in mind — these are just guidelines.  You are not required to test with any particular set of drivers, and you might want to fine-tune the list depending on what subsystem you're targeting.  If you are unsure about which binaries to replace, remember you can always just install the entire checked OS, which gives you the maximum checked build coverage.

Now that you know which drivers to replace, you can extract them from the checked build media.  If you obtained installable media, you can mount the included INSTALL.WIM with DISM.EXE to get at the individual drivers, or you can just install the OS into a throw-away VM to get convenient access to its drivers.

Finally, you'll need to actually replace these drivers on your target OS.  Don't do this on a production OS machine; we can't officially support this.  The easiest way to replace binaries is to hook up a kernel debugger and use the .kdfiles feature.  For example, here's the mapfile that I use to replace NDIS.SYS on a test machine:

 map  \Windows\system32\DRIVERS\NDIS.SYS  c:\path\to\ndis.sys

Note that the name of the driver will depend on how the driver is loaded.  Use CTRL+D or CTRL+ALT+D in the debugger and reboot the target machine to see the official name of each driver.

Note that the process for replacing the kernel & HAL is special.

Oh, and sorry for the awful pun in the subtitle.


Using WDF in an NDIS driver


Can, Should, and How?

WDF is a framework that makes it easier to write Windows drivers.  NDIS is a framework for writing low-level Windows network drivers.  The purposes of these frameworks overlap a bit, and some people (okay, probably many people) are confused about the relationship between NDIS and WDF.  Today we'll set down a few guidelines.  But first – let's dispel one tenacious myth.

Myth: Some people think that NDIS drivers cannot use WDF.

In reality, you can use WDF in your NDIS driver.  I know this works rather well, because I have personally written several WDF-based NDIS drivers.

So where do people get the idea that WDF is incompatible with NDIS?  There are a few sources of this idea:

  • When writing an NDIS miniport driver, certain parts of WDF are not compatible with NDIS.  You must put WDF into a mode sometimes referred to as "miniport mode".  Not all WDF APIs are available in miniport mode.  See the step-by-step checklist here.  Note that this restriction only applies to NDIS miniport (and IM) drivers; protocols and LWFs can use the full breadth of WDF functionality.
  • Miniport drivers must also put NDIS into a special mode, called NDIS-WDM mode.  This is a poor name, because it seems to indicate that you must use WDM.  The reality is that NDIS-WDM mode just means your driver can use any non-NDIS framework.  (At the time that NDIS-WDM mode was invented, there were no other frameworks besides WDM, so the name didn't seem to be too constraining.  If it helps, you can think of it as NDIS-WD* mode.)
  • Most of the NDIS drivers that are included with Windows (like TCPIP) don't use WDF.  But this isn't because Windows developers are avoiding WDF; it's because most inbox drivers simply predate WDF.  If we were writing the network stack from scratch, we'd use more WDF.  New drivers like MSLLDP, an NDIS protocol driver included with Windows 8, are indeed based on WDF.

Now that we know you can combine WDF with NDIS, let's talk about whether you should combine WDF with NDIS.  In nearly all cases, an NDIS driver will work with or without WDF.  So you rarely have the decision forced upon you by the technology.  Ultimately, it will come down to what you decide, based (hopefully) on a good engineering judgment call.  Let's collect some evidence to help you make that decision.

Reasons you should use WDF in your NDIS driver

  • Your engineering team is already familiar with WDF.
  • You will be developing several drivers, including non-networking drivers.  (Might as well learn WDF now, and maybe you can share some library code between your drivers.)
  • Your driver already uses WDF.
  • You are writing an NDIS miniport that uses IRPs on its lower edge (USB, SDIO, etc.)
  • You are writing a protocol or LWF that interacts with non-NDIS parts of the OS (usermode IOCTLs, WSK requests, etc.)
  • Your code would benefit from WDF's clever object management system to avoid memory leaks.
  • You are new to Windows driver development, and have no idea where to start 😰

Generally speaking, it's a good idea to consider WDF.  But there are a few reasons why WDF might not be very useful to your NDIS driver:

Reasons that WDF won't help in your NDIS driver

  • Your engineering team is already very familiar with NDIS, but has no experience with WDF.
  • You are maintaining a mature driver that does not use WDF.
  • You are writing a simple NDIS miniport on a directly-connected bus (like PCI).
  • You are writing a protocol or LWF that has minimal interaction with the rest of the OS.  This driver mostly only calls NDIS APIs.
  • Your codebase must be compatible with platforms where WDF is not available (like Windows CE).

Mind you, it's still quite possible to link against WDF in these situations.  But you'll probably find that there aren't a lot of opportunities to actually use WDF APIs.  Integrating with WDF doesn't give a lot of value if you don't call its APIs.  In those cases, the pragmatic engineering decision may be to just not use WDF.

Okay, so let's suppose you've decided to give WDF a spin.  You'll eventually notice that WDF overlaps somewhat with NDIS.  For example, both frameworks have APIs for workitems (NdisQueueIoWorkItem versus WdfWorkItemEnqueue).  Which API should you use?  Again, in many cases, either framework's APIs will work.  Again, it's an engineering decision that ought to consider several factors, including maintaining consistency with your other code, etc.  But if you are new to NDIS and WDF, you can use this quick-reference table as a starting place for your decision-making process.

API family Use NDIS APIs? Use WDF APIs? Use WDM APIs?
Work items Avoid Preferred Do not use
Timers Avoid Preferred Do not use
Memory allocation Avoid Preferred Okay
Locks & interlocks Avoid (but RW locks are okay) Preferred Preferred
Events Avoid Preferred Preferred
String handling Avoid Preferred Preferred
DMA Preferred Preferred Avoid
Interrupts Preferred Not permitted Not permitted
DPCs (for miniports) Preferred for interrupts Okay for non-interrupts Avoid
DPCs (for non-miniports) Avoid Preferred Avoid
Processor information Avoid (except RSS APIs) (no equivalent) Preferred
IRPs and IOCTLs (for miniports) Required Not permitted Not permitted
IRPs and IOCTLs (for non-miniports) Avoid Preferred Avoid
Direct bus/port access Okay Preferred Preferred
Reading configuration Preferred for standard keywords Preferred for other registry values Okay for other registry values
File I/O Avoid (no equivalent) Preferred

Remember, the above table only contains guidelines.  It is still acceptable to ship a driver that uses an API marked "Avoid".  You should use the table to help nudge your decision-making when you have no other compelling reasons to use a particular API family.


Welcome to the NDIS and NDISTest blog


Welcome to the NDIS and NDISTest blog... we are just getting started so be patient.

We want this blog to be a place where you can come and learn about the latest NDIS APIs, NDISTest test tool, get answers to your questions and more... stay tuned...

- The Network Devices Platform team, a.k.a the NDIS team