On User Rights, Safe Mode etc.

Technorati tags: Safety, Safe Mode, XP, malware, maintenance OS

Edited for spelling; from...

http://www.spywarepoint.com/forums/t26963-p8-microsoft-zero-day-security-holes-being-exploited.html

On Fri, 29 Sep 2006 23:17:02 -0400, "Karl Levinson, mvp"
>"cquirke (MVP Windows shell/user)" wrote in

>>>All operating systems do that. They are designed to launch code at boot
>>>time by reading registry values, text files, etc. Because those registry
>>>values are protected from unauthorized access by permissions, someone
>>>would have to already own your system to modify those values, wouldn't they?

The weakness here is that anything that runs during the user's session is deemed to have been run with the user's intent, and gets the same rights as the user. This is an inappropriate assumption when there are so many by-design opportunities for code to run automatically, whether the user intended to do so or not.

>> Sure, but the wrong entities come to own systems all the time.

>My point is that this one example here doesn't seem to be a vulnerability if
>it requires another vulnerability in order to use it.

Many vulnerabilities fall into that category, often because the extra requirement was originally seen as sufficient mitigation.  Vulnerabilities don't have to facilitate primary entry to be significant; they may escalate access after entry, or allow the active malware state to persist across Windows sessions, etc.

>This isn't a case of combining two vulnerabilities to compromise a
>system; it's a case of one unnamed vulnerability being used to
>compromise a system, and then the attacker performs some other
>action, specifically changing registry values.

>If this is a vulnerability, then the ability of Administrators to create new
>user accounts, change passwords etc. would also be a vulnerability.

OK, now I'm with you, and I agree with you up to a point. I dunno where the earlier poster got the notion that Winlogin was there to act as his "ace in the hole" for controlling malware, as was implied.

>> Defense in depth means planning for how you get your system back; you
>> don't just faint in shock and horror that you're owned, and destroy
>> the whole system as the only way to kill the invader.

>That's a different issue than the one we were discussing. The statement
>was, winlogon using registry values to execute code at boot time is a
>vulnerability. I'm arguing that it is not.

I agree with you that it is not - the problem is the difficulty that the user faces when trying to regain control over malware that is using Winlogin and similar integration points.

The safety defect is that:
- these integration points are also effective in Safe Mode
- there is no maintenance OS from which they can be managed

We're told we don't need a HD-independent mOS because we have Safe Mode, ignoring the possibility that Safe Mode's core code may itself be infected. Playing along with that assertion, we'd expect Safe Mode to disable any 3rd-party integration, and would provide a UI through which these integration points can be managed.

But this is not the case - the safety defect is that once software is permitted to run on the system, the user lacks the tools to regain control from that software. Couple that with the Windows propensity to auto-run material either be design or via defects, and you have what is one of the most common PC management crises around.

>Besides, it's a relatively accepted truism that once an attacker has root,
>system or administrator privileges on any OS, it is fairly futile to try to
>restrict what actions s/he can perform. Anything a good administrator can
>do, a bad administrator can undo.

That's a safety flaw right there.

You're prolly thinking from the pro-IT perspective, where users are literally wage-slaves - the PC is owned by someone else, the time the user spends on the PC is owned by someone else, and that someone else expects to override user control over the system.

So we have the notion of "administrators" vs. "users". Then you'd need a single administrator to be able to manage multiple PCs without having to actually waddle over to all those keyboards - so you design in backdoors to facilitate administration via the network.

Which is fine - in the un-free world of mass business computing.

But the home user owns their PCs, and there is no-one else who should have the right to usurp that control. (Even) creditors and police do not have the right to break in, search, or seize within the user's home.

So what happens when an OS designed for wage-slavery is dropped into free homes as-is? Who is the notional "administrator"? Why is the Internet treated as if it were a closed and professionally-secured network? There's no "good administrators" and "bad administrators" here; just the person at the keyboard who should have full control over the system, and other nebulous entities on the Internet who should have zero control over the system.

Whatever some automated process or network visitation has done to a system, the home user at the keyboard should be able to undo.

Windows XP Home is simply not designed for free users to assert their rights of ownership, and that's a problem deeper than bits and bytes.

Public Conversations

On Win9x, SR, mOS II, etc.

Technorati tags: Safety, malware, maintenance OS, Bart

Lifted from ...

http://www.spywarepoint.com/forums/t26963-p9-microsoft-zero-day-security-holes-being-exploited.html

On Sun, 01 Oct 2006 20:45:23 -0600, "Dan W." <spamyou@user.nec> wrote:
>karl levinson, mvp wrote:
>> "Dan W." <spamyou@user.nec> wrote in message

>> Fewer vulnerabilities are being reported for Windows 98 because Windows 98
>> is old and less commonly used, and vulns found for it get you less fame

More to the point is that vulnerable surfaces are less-often exposed to clickless attack - that's really what makes Win9x safer.

You can use an email app that displays only message text, without any inline content such as graphics etc. so that JPG and WMF exploit surfaces are less exposed. Couple that with an OS that doesn't wave RPC, LSASS etc. at the 'net and doesn't grope material underfoot (indexing) or when folders are viewed ("View As Web Page" and other metadata handlers) and you're getting somewhere.

For those who cannot subscribe to the "keep getting those patches, folks!" model, the above makes a lot of sense.

>> Didn't XP expand on and improve the system restore feature to a level not
>> currently in 98 or ME?

There's no SR in Win98, tho that was prolly when the first 3rd-party SR-like utilities started to appear. I remember two of these that seemed to inform WinME-era SR design.

No-one seemed that interested in adding these utilities, yet when the same functionality was built into WinME, it was touted as reason to switch to 'ME, and when this functionality fell over, users were often advised to "just" re-install to regain it. I doubt if we'd have advised users to "just" re-install the OS so that some 3rd-party add-on could work again.

XP's SR certainly is massively improved over WinME - and there's so little in common between them that it's rare one can offer SR management or tshooting advice that applies to both OSs equally.

I use SR in XP, and kill it at birth in WinME - that's the size of the difference, though a one-lunger (one big doomed C: installation may find the downsides of WinME's SR to less of an issue.

>>> about Microsoft and its early days to present time. The early Microsoft
>>> software engineers nicknamed it the Not There code since it did not have
>>> the type of maintenance operating system that Chris Quirke, MVP fondly
>>> talks about in regards to 98 Second Edition.

>> If the MOS being discussed for Win 98 is the system boot disk floppy, that
>> was a very basic MOS and it still works on Windows XP just as well as it
>> ever did on Windows 98. [Sure, you either have to format your disk as FAT,
>> or use a third party DOS NTFS driver.]

That was true, until we crossed the 137G limit (where DOS mode is no longer safe). It's a major reason why I still avoid NTFS... Bart works so well as a mOS for malware management that I seldom use DOS mode for that in XP systems, but data recovery and manual file system maintenance remain seriously limited for NTFS.

>> I think Chris really wants not that kind of MOS but a much bigger and
>> better one that has never existed.

Well, ever onward and all that ;-)

Bart is a bigger and better mOS, though it depends on how you build it (and yes, the effort of building it is larger than for DOS mode solutions). You can build a mOS from Bart that breaks various mOS safety rules (e.g. falls through to boot HD on unattended reset, automatically writes to HD, uses Explorer as shell and thus opens the risk of malware exploiting its surfaces, etc.).

I'm hoping MS WinPE 2.0, or the subset of this that is built into the Vista installation DVD, will match what Bart offers. Initial testing suggests it has the potential, though some mOS safety rules have been broken (e.g. fall-through to HD boot, requires visible Vista installation to work, etc.).

The RAM testing component is nice but breaks so many mOS safety rules so badly that I consider it unfit for use:
- spontaneous reset will reboot the HD
- HD is examined for Vista installation before you reach the test
- a large amount of UI code required to reach the test
- test drops the RAM tester on HD for next boot (!!)
- test logs results to the HD (!!)
- you have to boot full Vista off HD to see the results (!!!)

What this screams to me, is that MS still doesn't "get" what a mOS is, or how it should be designed. I can understand this, as MS WinPE was originally intended purely for setting up brand-new, presumed-good hardware with a fresh (destructive) OS installation.

By default, the RAM test does only one or a few passes; it takes under an hour or so - and thus is only going to detect pretty grossly-bad RAM. Grossly bad RAM is unlikely to run an entire GUI reliably, and can bit-lip any address to the wrong one, or any "read HD" call to a "write HD" call. The more code you run, the higher the risk of data corruption, and NO writes to HD should ever be done while the RAM is suspected to be bad (which is after all why we are testing it.

A mOS boot should never automatically chain to HD boot after a time out, because the reason you'd be using a mOS in the first place is because you daren't boot the HD. So when the mOS disk boots, the only safe thing to do is quickly reach a menu via a minimum of code, and stop there, with no-time-out fall-through.

It's tempting to fall-through to the RAM test as the only safe option, but that can undermine unattended RAM testing - if the system spontaneously resets during such testing, you need to know that, and it's not obvious if the reboot restarts the RAM test again.

Until RAM, physical HD and logical file system are known to be safe, and it's known that deleted material is not needed to be recovered, it is not safe to write to any HD. That means no page file, no swap, and no "drop and reboot" methods of restarting particular tests.

Until the HD's contents are known to be malware-free, it is unsafe to run any code off the HD. This goes beyond not booting the HD, or looking for drivers on the HD; it also means not automatically groping material there (e.g. when listing files in a folder) as doing so opens up internal surfaces of the mOS to exploitation risks.


Karl's right, tho... I'm already thinking beyond regaining what we lost when hardware (> 137G, USB, etc.) and NTFS broke the ability to use DOS mode as a mOS, to what a purpose-built mOS could offer.

For example, it could contain a generic file and redirected-registry scanning engine into which av vendor's scanning modules could be plugged. It could offer a single UI to manage these (i.e. "scan all files", "don't automatically clean" etc.) and could collate the results into a single log. It could improve efficiency by applying each engine in turn to material that is read once, rather than the norm of having each av scanner pull up the material to scan.

MS could be accused of foreclosing opportunities to av vendors (blocking kernel access, competing One Care and Defender products), but this sort of mOS design could open up new opportunities.

Normally, the av market is "dead man's shoes"; a system can have only one resident scanner, so the race is on to be that scanner (e.g. OEM bundling deals that reduce per-license revenue). Once users have an av, it becomes very difficult to get them to switch - they can't try out an alternate av without uninstalling what they have, and no-one wants to do that. It's only when feeware av "dies" at the end of a subscription period, that the user will consider a switch.

But a multi-av mOS allows av vendors to have their engines compared, at a fairly low development cost. They don't have to create any UI at all, because the mOS does that; all they have to do is provide a pure detection and cleaning engine, which is their core compitency anyway.

Chances are, some av vendors would prefer to avoid that challenge :-)

>> XP also comes with a number of restore features such as Recovery
>> Console and the Install CD Repair features.

They are good few-trick ponies, but they do not constitute a mOS. They can't run arbitrary apps, so they aren't an OS, and if they aren't an OS, then by definition that aren't a mOS either.

As it is, RC is crippled as a "recovery" environment, because it can't access anything other than C: and can't write to anywhere else. Even before you realise you'd have to copy files off one at a time (no wildcards, no subtree copy), this kills any data recovery prospects.

At best, RC and OS installation options can be considered "vendor support obligation" tools, i.e. they assist MS in getting MS's products working again. Your data is completely irrelevant.

It gets worse; MS accepts crippled OEM OS licensing as being "Genuine" (i.e. MS got paid) even if they provide NONE of that functionality.

The driver's not even in the car, let alone asleep at the wheel :-(

>> I never use those or find them very useful for security, but they're
>> way more functional and closer to an MOS than the Win98 recovery
>> floppy or anything Win98 ever had. 98 never had a registry
>> editor or a way to modify services like the XP Recovery Console.

They do different things.

RC and installation options can regain bootability and OS functionality, and if you have enabled Set commands before the crisis you are trying to manage, you can copy off files one at a time. They are limited to that, as no additional programs can be run.

In contrast, a Win98EBD is an OS, and can run other programs from diskette, RAM disk or CDR. Such programs include Regedit (non-interactive, i.e. import/export .REG only), Scandisk (interactive file system repair, which NTFS still lacks), Odi's LFN tools (copy off files in bulk, preserving LFNs), Disk Edit (manually repair or re-create file system structure) and run a number of av.

So while XP's tools are bound to getting XP running again, Win98EBD functionality encompasses data recovery, malware cleanup, and hardware diagnostics. It's a no-brainer as to which I'd want (both!)

>>> that at the bare bones level the source code of 9x is more secure

>> It depends on what you consider security.

That's the point I keep trying to make - what Dan refers to is what I'd call "safety", whereas what Karl's referring to is what I'd call "security". Security rests on safety, because the benefit of restricting access to the right users is undermined if what happens is not limited to what these users intended to happen.

>> Win98 was always crashing and unstable,

Er... no, not really. That hasn't been my mileage with any Win9x, compared to Win3.yuk - and as usual, YMMV based on what your hardware standards are, and how you set up the system. I do find XP more stable, as I'd expect, given NT's greater protection for hardware.

>> because there was no protection of memory space from bad apps or
>> bad attackers.

Mmmh... AFAIK, that sort of protection has been there since Win3.1 at least (specifically, the "386 Enhanced" mode of Win3.x). Even DOS used different memory segments for code and data, though it didn't use 386 design to police this separation.

IOW, the promise that "an app can crash, and all that happens is that app is terminated, the rest of the OS keeps running!" has been made for every version of Windows since Win3.x - it's just that the reality always falls short of the promise. It still does, though it gets a little closer every time.

If anything, there seems to be a back-track on the concept of data vs. code separation, and this may be a consequence of the Object-Orientated model. Before, you'd load some monolithic program into its code segment, which would then load data into a separate data segment. Now you have multiple objects, each of which can contain thier own variables (properties) and code (methods).

We're running after the horse by band-aiding CPU-based No-Execute trapping, so that when (not if) our current software design allows "data" to spew over into code space, we can catch it.

>> Microsoft's security problems have largely been because of backwards
>> compatibility with Windows 9x, DOS and Windows NT 4.0. They feel, and I
>> agree, that Microsoft security would be a lot better if they could abandon
>> that backwards compatibility with very old niche software, as they have been
>> doing gradually.

The real millstone was Win3.yuk (think heaps, co-operative multitasking). Ironically, DOS apps multitask better than Win16 ones, as each DOS app lives in its own VM and is pre-emptively multi-tasked.

64-bit is the opportunity to make new rules, as Vista is doing (e.g. no intrusions into kernel allowed). I'm hoping that this will be as beneficial as hardware virtualization was for NT.

Win9x apps don't cast as much of a shadow, as after all, Win9x's native application code was to be the same as NT's. What is a challenge is getting vendors to conform to reduced user rights, as up until XP, they could simply ignore this.

There's also the burden of legacy integration points, from Autoexec.bat through Win.ini through the various fads and fashions of Win9x and NT and beyond. There's something seriously wrong if MS is unable to enumerate every single integration point, and provide a super-MSConfig to manage them all from a single UI.

>Classic Edition could be completely compatible with the older software
>such as Windows 3.1 programs and DOS programs. Heck, Microsoft
>could do this in a heartbeat without too much trouble.

Think about that. Who sits in exactly the same job for 12 years?

All the coders who actually made Win95, aren't front-line coders at MS anymore. They've either left, or they've climbed the ladder into other types of job, such as division managers, software architects etc. To the folks who are currently front-line coders, making Vista etc., Win9x is as alien as (say) Linux or OS/2.

To build a new Win9x, MS would have to re-train a number of new coders, which would take ages, and then they'd have to keep this skills pool alive as long as the new Win9x were in use. I don't see them wanting to do that, especially as they had such a battle to sunset Win9x and move everyone over to NT (XP) in the first place.

Also, think about what you want from Win9x - you may find that what you really want is a set of attributes that are not inherently unique to Win9x at all, and which may be present in (say) embedded XP.

If you really do need the ability to run DOS and Win3.yuk apps, then you'd be better served by an emulator for these OSs.

This not only protects the rest of the system to the oddball activities of these platforms, but can also virtualize incompatible hardware and mimic the expected slower clock speeds more smoothly than direct execution could offer. This is important, as unexpected speed and disparity between instruction times is as much a reason for old software to fail on new systems as changes within Windows itself.

>I will do what it takes to see this come to reality.

Stick around on this, even if there's no further Win9x as such. As we can see from MS's first mOS since Win98 and WinME EBDs, there's more to doing this than the ability to write working code - there has to be an understanding of what the code should do in the "real world".

 

Public Conversations

CDRW/DVDRW Primer

It can be a bit confusing figuring out R vs. RW and formal authoring vs. packet writing, but I'll try.  This skips a lot of detail, and attempts to zoom on what you'd need to know if starting on writing CDs or DVDs in 2007...

Here's the executive summary:

	R	RW
Authored	Fine	Fine
Packet-written	Can't	Sucks

R vs. RW disks

R(ecordable) disks are like writing in ink - once you've written, you cannot erase, edit or overwrite.

R(e)W(ritable) disks are like writing in pencil - you can rub out what you want to change, but what you write in there, has to fit between whatever else you have not rubbed out.

Authoring vs. packet writing

The "authoring" process is like setting up a printing press; you first lay out the CD or DVD exactly as you want it, then you splat that onto the disk.  You can fill the whole disk at once, like printing a book (single session), or you can fill the first part and leave the rest blank to add more stuff later, like a printed book that has blank pages where new stuff can be added (start a multisession).

The "packet writing" process is what lets you pretend an RW disk is like a "big diskette".  Material is written to disk in packets, and individual packets can be rubbed out and replaced with new packets, which pretty much mirrors the way magnetic disks are used.  This method is obviously not applicable to R disks.

RW disks can also be authored, but the rules stay the same; you either add extra sessions to a multi-session disk, or you erase the whole disk and author it all over again.

Overwriting

When you overwrite a file in a packet-writing system, you do so by freeing up the packets containing the old file and write the new file into the same and/or other packets.  The free space left over is increased by the size of the old file and reduced by the size of the new, rounded up to a whole number of packets.

When you "overwrite" a file in a multisession (authored) disk, it is like crossing out the old material and writing new material underneath, as one is obliged to do when writing in ink.  The free space drops faster, because the space of the old file cannot be reclaimed and re-used, and because each session has some file system overhead, no matter how small the content.

Standards and tools

There are a number of different standard disk formats, all of which must be formally authored; audio CDs, movie DVDs, CD-ROMs and DVD-ROMs of various flavors.  In contrast, packet-written disk formats may be proprietary, and supported only by the software that created them.

Nero and Easy CD Creator are examples of formal authoring tools, and several media players can also author various media and data formats.

InCD and DirectCD are examples of packet-writing tools, which generally maintain a low profile in the SysTray, popping up only to format newly-discovered blank RW disks.  The rest of the time, they work thier magic behind the scenes, so that Windows Explorer can appear to be able to use RW disks as "big diskettes".

Windows has built-in writer support, but the way it works can embody the worst of both authoring and packet-writing models.  I generally disable this support and use Nero instead.

Flakiness

RW disks and flash drives share a bad characteristic; limited write life.  In order to reduce write traffic to RW disks, packet writing software will hold back and accumulate writes, so these can be written back in one go just before the disk is ejected.

What this means is that packet written disks often get barfed by bad exits, lockups, crashes, and forced disk ejects.  Typically the disk will have no files on it, and no free space.  When this happens, you can either erase the disk and author it, or format the disk for another go at packet writing.  Erasing is faster, while formatting applies only to packet writing (it defines the packets).

I have found that packet writing software has been a common cause of system instability (that often ironically corrupts packet-written disks).  The unreliability, slow formatting, and poor portability across arbitrary systems have all led me to abandon packet writing in favor of formally authoring RW disks. 

Back to Basics

Evolution vs. Intelligent Design

Technorati tags: Information theory, memetics, evolution

Evolution vs. Intelligent Design = non-issue.

Evolution does not define why things happen. 

It is a mechanism whereby some things that happen, may come to persist (and others, not). 

Graded belief

Human thinking appears to have at least two weaknesses; an automatic assumption of dualities (e.g. "Microsoft and Google are both large; Microsoft is bad, therefore Google must be good"), and an unwillingness to accept unknowns. 

You can re-state the second as a tighter version of the first, i.e. the singleton assumption, rather than duality.

We don't even have words (in English, at least) to differentiate between degrees of belief, i.e. weak ("all things equal, I think it is more likely that A of A, B, C is the truth") and strong ("iron is a metal") belief. 

And I fairly strongly believe we strongly believe too often, when a weaker degree of certainty would be not only more appropriate, but is a needed component in our quest for Word Peace TM.

For example, religious folks have a fairly high certainty of what will happen after they die - but can we all accept that as they have not yet died, that something slightly less than "you're wrong, so I'll kill you" certainty should apply?

What is evolution?

My understanding of evolution, or Darwinian systems, revolves around the following components:

• limited life span
• selection pressure
• imperfect reproduction

That's what I consider to be a classic evolutionary environment, but you may get variations; e.g. if entities change during the course of their lifetime, do not reproduce, but can die, then you could consider this as a Darwinian system that is ultimately set to run down like a wind-up clock as the number of survivors declines towards zero.

In fact, implicit in that classic model is the notion of reproduction based on a self-definition that does not change during the course of an entity's lifetime (in fact, it defines that entity) but can change when spawning next generation entities.

Evolution is blind

Evolution per se, is devoid of intent.  I don't know whether Darwin stressed this in his original writings, but I weakly believe that he did; yet I often see descriptions of creatures "evolving to survive". 

As I understand it, game theory is a reformulation of evolution that centers on the notion of survival intent.

Evolution is something that happens to things, and doesn't "care" whether those things survive or not.  The "selfish gene" concept is an attempt to frame this inevitable sense of "intent" within Darwinian mechanics; there is no more need to ascribe a survival intent to genes, as there is to the phenotypes they define.

However, evolution doesn't have to be the only player on the stage, and this is what I meant about "evolution vs. intelligent design is a non-issue".

I don't think there's much uncertainty that evolution is at work in the world.  That doesn't weigh for or against other (intelligent design) players in the world, and that's why I consider the question a non-issue.

Intelligent players can apply intent from outside the system (e.g. where an external entity defines entities within a Darwinian environment, or the environment itself, or its selection pressures) or from within the system (e.g. where entities apply intent to designing their own progeny).

Example systems

I consider the following to be Darwinian systems:

• the biosphere
• the infosphere
• human culture, i.e. memetics

One could theorize that evolution is an inevitable consequence of complexity, when subjected to entropy.  Just as a moving car is not fast enough to exhibit significant relativistic effects (so that Newton's laws appear to explain everything), so trivial systems may be insufficiently complex to demonstrate Darwinian behavior.

This is why I'm interested in computers and the infosphere; because they are becoming complex enough to defy determinism. 

Normally, we seek to understand the "real world" by peering down from the top, with insufficient clarity to see the bottom. 

With the infosphere, we have an environment that we understand (and created) from the bottom up; what we cannot "see" is the top level that will arise as complexity evolves.

This creates an opportunity to model the one system within the other.  What was an inscrutable "mind / brain" question, becomes "the mind is the software, the brain is the hardware", perhaps over-extended to "the self is the runtime, the mind is the software, the brain is the firmware, the body is the hardware". 

We can also look at computer viruses as a model for biosphere viruses.  A major "aha!" moment for me was when I searched the Internet for information on the CAP virus, and found a lot of articles that almost made sense, but not quite - until I realized these described biological viruses, and were found because of the common bio-virus term "CAPsule".

Code

Common to my understanding of what constitutes a classic Darwinian system, is the notion of information that defines the entity. 

In the biosphere, this is usually DNA or RNA, a language of 4 unique items grouped into threes to map to the active proteins they define.

In the infosphere, this is binary code of various languages, based on bits that are typically grouped into eights as bytes.

In the meme space, languages are carried via symbol sets that are in turn split into unique characters, which are then clumped into words or sentences.  Some languages contain less information within the character set (e.g. Western alphabets), others more (e.g. the Chinese alphabet, ancient Egyptian hieroglyphics, modern icons and branding marks).

When we create computer code, we are laboriously translating the memetic language of ideas into code that will spawn infosphere entities.  This is not unlike the way a set of chromosomes becomes a chicken, other than that we view the infosphere and meme space as separate Darwinian systems.

The alluring challenge is to translate infosphere code into biosphere code, i.e. to "print DNA", as it were.  One hopes the quality of intent will be sound, by the time this milestone is reached, as in effect, we would be positioned to become our own intelligent creators.

Intelligent design

We know that entities in the infosphere are created by intent from outside the system; as at 2007, we do not believe that new entities arise spontaneously within the system.

We don't know (but may have beliefs about) whether there is intent applied to the biosphere, or whether the biosphere was originally created or shaped by acts of intent.

Conspiracy theorists may point to hidden uber-intenders within the meme space, the creation of which is inherently guided by self-intent.

That which was

Just as folks mistakenly ascribe intent to the mechanics of evolution, so there is a fallacy that all that exists, is all that existed.

But evolution can tell you nothing about what entities one existed, as spawned by mutation or entropic shaping of code.

There is a very dangerous assumption that because you cannot see a surviving entity in the current set, that such entities cannot arise. 

Think of a bio-virus with a fast-death payload a la rabies, plus rapid spread a la the common cold.  The assumption of survival intent leads folks to say stupid things like "but that would kill the host, so the virus wouldn't want that".  Sure, there'd be no survivors in today's entity set, but on the other hand, we know we have some historical bulk extinctions to explain.

We're beginning to see the same complacency on the risks of nuclear war.  We think of humanity as a single chain of upwards development, and therefore are optimistic that "common sense will prevail".  Even nay-sayers that point to our unprecedented ability to destroy ourselves, miss the point in that word "unprecedented".  As Graham Hancock postulated in his Fingerprints of the Gods, we may have been this way before.

This blindness applies to malware within the infosphere as well, and the saying "there's none so blind as those who can see" applies.  If we want authoritative voices on malware, we generally turn to professionals who have been staring at all malware for years, such as the antivirus industry.  These folks may be blinded by what they've seen of all that has been, that they fail to consider all that could be.

The Waking Hour

Low Heap Space in XP and Vista

Technorati tags: Bug, XP, Vista

Have you ever been motoring along in XP or Vista, opening up new tabs in IE7, running apps in the background, etc. and noticed new tabs don't show the pages, or that when you right-click a link, you don't get a context menu?  Or have you ever opened a couple of dozen photos in Irfan View and found the last few come up with no menu, and don't respond to hotkeys?

Did this bring back memories of "low resource heaps" in Win9x, or the need to restart DOS/Win3.yuk PCs several times a day to keep MS Office apps running properly?

You aren't going crazy, and yes, it's the same problem.  It's like deja vu all over again.

Background

Windows 3.x used two or three "heaps", i.e. areas of RAM set aside for certain items that are spawned by running programs.  There was a GDI heap for graphic elements, a user heap for UI and other elements, and a "system" heap that may have been a logical view of the first two (it's been a while, and I hoped I never have to remember the details again).

When folks started multitasking in earnest, these heaps would fill up and cause crashes, half-drawn UI elements, or spurious "out of memory" errors.  Adding RAM was as useful as adding a trailer to a removal van with a loading crew so dumb they insist on putting all metal objects in the glove compartment and then tell you "we're full" after moving your hi-fi.

Windows 95 design intended to move these heaps to 32-bit replacements, but it was found that doing so would break several applications (including, it was rumored, Excel 4.0) which wrote directly to heap objects in memory rather than using the proper API calls.

So Windows 95 went into a long public beta where these things were thrashed out.  Win9x left some items in "legacy" 16-bit heaps, and proactively cleared heap allocations when closing down VMs.  32-bit and DOS programs were run in their own VM, so this curbed "heap leakage" for these, but as 16-bit Windows all ran in a single shared VM, any resident 16-bit Windows program (e.g. Bitware 3.x) could hold this VM open forever.

Pundits would smugly point out that NT was a "true" 32-bit OS which did not make such compromises.  The "huge" 32-bit address range should mean 32-bit heaps would never run out again.

The broken rule is...

"Do not use finite global storage for the per-instance data of an unbounded number of instances". 

There are several instances of this rule being broken, such as the way Windows Explorer "remembers" settings for different folders. 

The converse rule...

"Do not scale a per-instance resource based on evolving global store capacity"

...is broken by IE''s grabbing of X% of total HD volume space for its web cache, as well as System Restore capacity allocation.  The first example was only fixed in IE7; the second, continues.

NT broke the first of these two scalability rules.  The NT developers did not fix the heap problem via dynamically-sized heaps that could expand up to the limits of 32-bit addressability.  Instead, they set various arbitrary limits at various times, in fact reducing these between versions 3.1 and 3.5 of the original NT. 

The fix

Apparently this has been a known issue at Microsoft, though most of us (myself included) didn't see the old "resource heap blues" for the first few years on XP.  I found some good background coverage in blogs and elsewhere, but the fix (adjusting certain registry settings) comes with some caveats:

Please do not modify these values on a whim. Changing the second or third value too high can put you in a no-boot situation due to the kernel not being able to allocate memory properly to even get Session 0 set up. 

I haven't applied the fix yet, but probably will, on my XP SP2 system with IE7 etc.  If it holds up here, and I have complaints from Vista users, then I may apply the fix to my new Vista PC builds as well.

Coverage

As you can guess, some blogs have been quite hostile to Microsoft, and this one is no exception (to me, it seems as if these folks missed the "Scalability 101" lecture).

However, I can see some reasons why they may have chosen not to create a dynamically-resizing heap system. 

One reason is to prevent DoS memory usage through deliberate leaks; another might be concern over exploitable race conditions that might arise if one process is creating heap objects while another deliberately releases heap space to provoke a downwards resize. 

There may be unacceptable overhead in managing more scalable data structures, such as linked lists, if heap objects are accessed as often as I suspect they might be.

Whatever such reasons might be, I'd suggest Microsoft get their story up really soon, before the trickle turns to a flood and users twig onto why they can't do "too many things at the same time" on their PCs - especially given the usual Vista mantra of "it's slow and needs more resources, but that's so it can scale up to future needs" that I've been known to wave around  :-)

Seek, And Ye Shall Find... What?

Technorati tags: safety

Re: Can a saved search be indistinguishable from real folders?

On Sat, 28 Jul 2007 05:34:00 -0700, Baffin

>To my mind, it would be elegant and correct for the operating system to
>present a 'saved search' folder to all appolications exactly as a real folder
>is presented.  Thus no changes should be required to any applications -- they
>should all just work as usual -- instant access to saved-search collections
>-- great!

Think through the safety and security implications of that.

>But as mentioned originally, I'm having problems getting many applications
>to work with saved-search folders -- am I doing something wrong?  Or, could
>Microsoft have implemented 'virtual folders' (saved searches)
>non-transparently?  If so, why?  What's the advantage worth all the
>disruption that would cause to applications?

I think there was a change in design intention on this. 

Originally, when Vista was to embed SQL within the WinFS file system, these "virtual folders" were to function more transparently as folders, as you expect.

When WinFS was dropped, functionalities of "virtual folders" were scaled  back - I'd thought they had been dropped alltogether.

Just as web mania drove MS to embed IE4 in Win98, with "View As Web Page" on your local file system, so search mania has driven MS to embed search into Vista. 

Just as there were safety downsides to blurring the edge between Internet and local PC (as well as HTML-everywhere also dropping scriptability everywhere), so may there be safety downsides to searching for rather than specifying the files etc. you "open".

>Why can't it be 'invisible' to all existing applications whether or not a
>folder is real or 'virtual' (ie., a saved search)?

The original intention of file names was to ensure that every file was uniquely named.  When this hit scalability issues, the concept of directories and paths was added.

The need to uniquely identify files is as strong as ever, in an age of pervasive malware, phishing, etc. but is also necessary to avoid "version soup" problems, reversion to pre-patch code, and confusion between old and new versions of data files that may be scattered across "live", backup, and off-PC storage locations.

When you throw away that specificity and just "search" for things, you need to be very sure about what you are looking at.  Not easy, through a shell that hides file name extensions, allows dangerous file types to define their own icons, etc.

So yes; I *definately* want it to be very obvious as to whether I am looking at a directory, or some virtual collection of found items.

Public Conversations

When ChkDsk Doesn't

Technorati tags: ChdDsk, file system, data recovery

Subject: Re: Unable to run CHKDSK with "Fix" option

On Wed, 25 Jul 2007 17:46:03 -0700, Paulie

I think it is beyond time that we had a proper interactive file system maintenance tool for NTFS.  ChkDsk is a relic from the MS-DOS 5 days; I wish NT would at least catch up to, say, MS-DOS 6 Scandisk.

Now folks will flame me for saying that.  "File systems are too complex for users to understand, just trust us to fix everything for you".  Fine; let's read on and see how well that works...

>My new Notebook is unable to run a CHKDSK with the Fix option selected.
>I can run a normal CHKDSK within VISTA and it works without a problem.
>If I choose the Fix option it schedules a scan on the next boot. Upon
>rebooting,
>CHKDSK will begin but it will freeze after 8% of the scan. The
>Notebook does not respond and I have to power it down and restart.

Great, so now we combine a possibly corrupted file system in need of repair, with recurrent bad exits.  What's wrong with this picture?

>Any idea what this may be?

Given that ChkDsk and AutoChk are closed boxes with little or no documentation of what they are doing (and little or no feedback to you while they are doing it), one can only guess.

My guesses would be one of:

1)  Physically failing HD

When a sector can't be read, the HD will retry the operation a number of times before giving up.  Whatever driver code that calls the operation will probably also retry a few times, before giving up, and so may the higher-level code that called that, etc. 

The result can be an apparent "hang" lasting seconds to minutes while the system beats the dying disk to death.

That's before you factor in futile attepts to paper over the problem and pretend it isn't there, both by the HD itself, and by the NTFS code.  Each will attempt to read the sick allocation unit's data and write it to a "good" replacement, then switch usage so that the dead sector is avoided in future.  And so on, for next dead sector, etc.

2)  Lengthy repair process

Scandisk and ChkDsk have no "big picture" awareness.  If they were you, walking from A to B, they would take a step, calculate if they were at B, then take another, and repeat.  If they were walking in the wrong direction, away from B, they'd just keep on walking forever.

So when something happens that invalidates huge chunks of the file system, these tools don't see the "big picture" and STOP and say "hey, something is invalidating the way this file system is viewed".  No; they look at one atom of the file system, change it to fit the current view, and repeat for the next.  If that means changing evey atom in the file system, that is what they will do.  Result; garbage.

3)  Bugginess

Whereas (2) is a bad design working as designed, sometimes the code doesn't work as designed and falls off the edge.

Needless to say, AutoChk and ChkDsk don't maintain any undoability. They also "know better than you", so they don't stop and ask you before "fixing" things, they just wade in and start slashing away.

>I have run full virus scans and updates on the drive and
>there are no issues. Other than this the Notebook runs fine. Its just a
>strange issue that i am unable to fix at this stage.

I would at least exclude (1) by checking the HD's surface using the appropriate tests in HD Tune (www.hdtune.com), after backing up my data.  You should be able to get a "second opinion" on the file system, but you can't; ChkDsk and AutoChk are all you have.

Public Conversations

Malware Poll Results

Technorati tags: Poll results

I played with Blogger's "poll" feature, asking whether malware was or was not reader's biggest PC headache.  The results:

• 16% - Malware is my biggest PC headache
• 66% - Malware is not my biggest PC headache
• 0% - I'm OK, I use Linux / BSD / other
• 16% - I'm OK, I use a Mac
• 0% - I'm not on Windows, but malware is still a worry

That's on a tiny sample size of 12 respondents  :-)

Public Conversations

Technorati tags: Blogging

A lot of what I write is posts in transient newsgroups and forums, and although I resolved to write things "properly" in web pages and then blogs, I tend not to do this.  I find it far easier to respond to another entity than write "cold" for a generalized audience.

As mentioned, I'm blending blogging with formal web site navigation using whatever tools come to hand.  I'll use Blogger's labels as category selectors so that this blog can function as a set of unrelated blogs.  These "virtual" blogs may include:

• Public Conversations
• The Waking Hour
• Maintenance OS
• Safety and Security

The first of these is Public Conversations, where I will simply paste what I see as interesting posts that are already public, edited only to allow natural line lengths and to break visible email addresses.

Over at my other blog, I've added a few new photo galleries, using "comments" to use these as pictorial articles rather than just a  bunch of loose photos.  So far, these include:

• Using Bart as a mOS
• Bugs (biological)
• Bugs (hardware)
• Bugs (software)
• Hall of Shame
• Bad Capacitors

Both Blogger and Live Spaces have their strengths and weaknesses, so I'll probably continue adding new content to both.

Ship Now, Patch Later

Technorati tags: Safety, security

Subject: Re: It would be nice if MS could settingle on a single subnet for updates

On Fri, 27 Jul 2007 15:13:52 +0100, "Mike Brannigan"
>"Leythos" <void@nowhere.lan> wrote in message
>> Mike.Brannigan@localhost says...

This thread is about the collision between...

    No automatic code base changes allowed

...and...

    Vendors need to push "code of the day"

Given the only reason we allow vendors to push "code of the day" is because their existing code fails too often for us to manage manually, one wonders if our trust in these vendors is well-placed.

A big part of this is knowing that only the vendor is pushing the code, and that's hard to be sure of.  If malware were to hijack a vendor's update pipe, it could blow black code into the core of systems, right pas all those system's defenses.

With that in mind, I've switched from wishing MS would use open standards for patch transmission to being grateful for whatever they can do to harden the process.  I'd still rather not have to leave myself open to injections of "code of the day", though.

>NO never ever ever in a production corporate environment do you allow ANY of
>your workstations and servers to directly access anyone for patches
>I have never allowed this or even seen it in real large or enterprise
>customers. (the only place it may crop up is in mom and pop 
>10 PCs and a Server shops).

And there's the problem.  MS concentrates on scaling up to enterprise needs, where the enterprise should consolodate patches in one location and then drive these into systems under their own in-house control.

So scaling up is well catered for.

But what about scaling down? 

Do "mom and pop" folks not deserve safety?  How about single-PC users which have everything they own tied up in that one vulnerable box?  What's best-practice for them - "trust me, I'm a software vendor"?

How about scaling outwards? 

When every single vendor wants to be able to push "updates" into your PC, even for things as trivial as prinyers and mouse drivers, how do you manage these?  How do you manage 50 different ad-hoc update delivery systems, some from vendors who are not much beyond "Mom and Pop" status themselves?  Do we let Zango etc. "update" themselves?

The bottom line: "Ship now, patch later" is an unworkable model.

>As you said your only problem is with Microsoft then the solution I have
>outlined above is the fix - only one server needs access through your
>draconian firewall policies.  And you get a real secure enterprise patch
>management solution that significantly lowers the risk to your environment.

That's prolly the best solution, for those with the resources to manage it.  It does create a lock-in advantage for MS, but at least it is one that is value-based (i.e. the positive value of a well-developed enterprise-ready management system).

However, I have to wonder how effective in-house patch evaluation really is, especially if it is to keep up with tight time-to-exploit cycles.  It may be the closed-source equivalent of the open source boast that "our code is validated by a thousand reviewers"; looks good on paper, but is it really effective in practice?

Public Conversations

Good Things Here...

Technorati tags: Malware, maintenance OS

Some folks get it...

http://blogs.technet.com/secguide/archive/2007/07/12/malware-removal-starter-kit.aspx

Using the Windows Preinstallation Environment (Windows PE) in combination with free anti-malware programs, the kit provides you with a low-cost, effective strategy and tool recommendations that you can use to vanquish malware attacks

...while some folks don't:

http://blogs.msdn.com/rflaming/archive/2006/09/20/763960.aspx

From a security perspective, when you get owned running under a Machine-wide account, game is over and you have to flatten the machine to get back to a secure state. 

By "it", I mean the defense-in-depth concept that the battle doesn't end when malware gets into your PC.  Machines get "owned" all the time; the majority of spam is carried by botnets running on such systems, and surveys have indicated a high percentage of PCs are running malware. 

If the only option for such systems is to "just" flatten and rebuild, many consumers will simply shrug and prefer to stay infected.  After all, they tolerate rootkits dropped from audio CDs, DoS (activation) payloads built into their OS, adverts from all over the place, etc. so why should they mind if a smidgen of bandwidth is used to DDoS unpopular entities such as the RIAA, or send out the same spam they get every day, either way?

The problem with "just wipe and rebuild" is not the pessimism that a cleaned PC will really be clean, but the optimism that a rebuilt PC will stay clean.  In reality, both approaches are complex battles that may be lost.

Security Guides Blog

The first link is from SecGuide, who may be the first Microsoft team to offer end users the tools they need to formally manage malware on infected PCs.  They may not be as far down that road as some Bart-based solutions, of which an example is shown in this slide show, but in-house Bart projects are usually too complex to be offered as an off-the-peg solution for end users to download and use.

The SecGuide approach is based on WinPE 2.0, which is now available for end users via the WAIK.  The process of integrating tools into WinPE, and building a WinPE boot disk, is pretty daunting, so I was wondering if combining David Lipman's Multi-AV tool with Bart PE would be easier?

In the big picture, we need to market the clean state against the accepted state of living with resident malware.  A non-destructive cleaning approach is a key element, and it's good to see parts of Microsoft getting this.

Windows Installer

The second link is from Setup Sense and Sensibility, which is a fascinating insight into the Windows Installer and how this has developed in Vista in particular.  The perspective appears to be 100% rooted in the concerns of corporate networking, and centered on per-user permissions and control.

The trouble is, this approach just doesn't fit the outside world of free users and the one or few PCs they use.  There's no "admin" to "do things for" the user; no tight white-list of permitted applications, and the user should have full and unfettered control over the PC.  A single PC may represent the user's entire infrastructure, so there's no "easy way out" of wiping and rebuilding desktop systems while data is safe on the server. 

Moreover, the same user will do multiple different things in the same logon session that should have differentiated rights.  Simply giving all processes the same rights just because they occur in the same logon session is next to useless, as even the most limited user account rights will allow the user's data to be edited, overwritten or trashed.

I've covered aspects of this issue many times, such as the adverse effect of flattening natural obstacles and the janitor account concept.  UAC is a step in the right direction, as for the first time it leverages the user's control over automated processes - the reason it is so "ugly" is because it is so at odds with the assumptions underlying NT's development, i.e. that automation would always be done by "proper" entities and that the user should be swept aside to facilitate such automation.

Timeless Blogging

Technorati tags: Blogging

Can you have your blog and web it too?  I'd say so, and am doing that - blogging so that content is navigable in ways more like a web site, as described here.  My tactics are:

• Lists as the "top level"
• Permalinks, labels and tags
• Hover-tips to explain link destinations
• Closure via post-terminating "home" links

Blogging is still dominated by the timeline concept, and this is particularly ironic when a post solicits feedback, only to tell you that "new comments are disabled", as in this case.  Bah!  So I'll feedback here...

Yes, I'd love to see the SecGuide blog post discussion topics for feedback, as well as their usual announcement posts.  They can use tags to separate such traffic for those who only want to read announcements, vs. those who want to contribute to discussions, and still have a mix of both for the rest of us interested in both types of posts.

I plan to do something similar on my blogs, so that in effect each blog can function as multiple sites.  Blog-based discussions may be more survivable and discoverable than web or news forums, as the blog (or web site) provides a persistent navigation tree to reach this material.

Meta-Bug: UI Refresh That Doesn't

Technorati tags: Bug, Meta-bug

If you can read this post, you're lucky, because I think I've just edited this blog's framework into the ground!

The case was a meta-bug (i.e. a conceptual bug that underlies other bugs).  I was trying to add a poll to the blog, which was rejected due to some date format thing.  So; is "07/07/19" yy/mm/dd or mm/yy/dd?  Do my preferences for dd/mm/yy(yy) have any bearing on what the error report shows?  None of that is the bug, by the way; just bad error message design - in fact, it may be that the new Poll thing is just plain broken.

Here's the bug: If I try a different date that is also "wrong", will I be able to visually tell the difference between nothing happening when I click Save, and getting the exact same error message invisibly "refreshed" over the old one?  What happens if I ASSume that "nothing happened" and I machine-gun the dumb-ass Save button 100 times out of frustration?

Er... I wish the last question was rhetorical, heh.  I've just closed the browser window that has been "Saving..." my blog for the last unfeasibly large number of minutes.

So if you're looking at a blog layout that is currently less than fully-assed, with a dead poll that was "fished" before it was born, then you know why.

Bah!  User failure strikes again...

Malware - Is That All You Ever Think About?

Technorati tags: Malware

Folks could be forgiven for asking:

Why do you care
About malware?

Malware is the bulk of a larger problem which is vendor-pushed code.  Nothing can overwhelm support resources as widespread automatic insertion of bad code can do.

For in-house system administrators, it's a major headache, but for a tech servicing multiple single-PC sites, it can be a disaster.  If you offer an SLA (Service Level Agreement) that is insufficiently escaped by weasel-wording and disclaimers, then one big outbreak can put you out of business... how do you "resolve within 48 hours" when you have 100 sites per tech needing urgent attention within the same hour?

So yes; just as someone interested in completing university studies may switch to soldiery as driven by self-preservation demands, so I have an interest in malware.  And just as a soldier has an interested in keeping his weapons in working order, I have an interest in maintenance OSs such as Bart and WinPE 2.0, as well as the politics that keep these tools out of the hands of those who need them most.

Sharp readers will have noticed my definition of the "larger problem" encompasses automatic OS and antivirus updates, various ad-hoc "update" facilities built into arbitrary programs, Google's "update everything" tool, and codecs "needed" to play arbitrary content. 

All of these break best-practice rules on code changes:

• Do not allow others to change your code
• Log all code changes
• Ensure all changes are reversible
• Ensure changes do not "kick away the ladder"

In essence, the logic behind "code of the day" is broken:

• When our code breaks, it can't be trusted
• This happens too often to manage manually
• So trust us to push more code whenever we see fit

Does not compute.  Yes, I see the need to patch OS and exposed surfaces as soon as possible, but I also see the need to reduce exposed surfaces made of code that is not trivial enough to be relied on as defect-free.

And no, I don't recommend Google's "update everything" tool.

New Content From Here

I've just completed a wish list of around 30 or so I'd like to see fixed, changed or created in Vista (and in some cases, retro-fitted to XP).

I also did a commented picture show of a typical Bart mOS session, stepping though verifying each safety level before reaching for the next, as outlined in this PC Crisis article.

I've always wanted to re-use blog posts as structured web pages, so as to combine the simple creation and consistent style of a blog site with the ongoing re-usability of a formal web site.  I find I can do that on the other blog by combining the "list" feature with jump-pad blog posts full of links.  I'm used to this structure from a closed mOS wiki I did a while back.

What this may mean, is less (or perhaps, different) content here, with links from here to there.  I like the lack of adverts on this site, but I can't see as easy a way to get it to do what I'd like done.

Forthcoming Attractions

In the next week, I'll be low-profile in newsgroups (again) and doing Vista feedback and mOS issues that I will echo in my blogs. The mOS stuff will be here, while the Vista stuff I'll do over in the Vista Curve blog.

I expect there will be a lot of traffic, so suggest you peruse subject lines in the sidebar rather than desperately hitting the PageDown key :-)

mOS for Vista: WinPE, WinRE and Bart

With Vista, one loses some friends and gains others, with Vista-64 being particularly challenging. You can use XP-based Bart to do many things in Vista (especially hardware diagnostics) but nothing has the ability to redirect registry access on the fly as RunScanner does for XP on Bart, which is a significant limitation for Vista maintenance.

The players are:

• MS WinPE 2.0 for XP, 2003 and Vista
• MS WinRE for XP, 2003 and Vista
• Bart PE, current XP/2003 version
• Bart 4 for Vista, XP, 2003 etc.?

Bart 4 is but a rumor at this time, and may stay that way awhile. As MS WinPE 2.0 is now available to general users, and as Bart has experience with earlier MS WinPE, he may develop a GUI front-end for that instead (I'm just guessing here, but that's what I'd do).

My Vista Curve blog will cover WAIK, WinPE 2.0 etc. as I grow my Vista maintenance skills. There are also walk-thoughs for building MS WinPE 2.0 solutions, and blogs from the Microsoft WinRE and WinPE teams.

I was going to develop MS WinPE 2.0 as a separate mOS project, but a better way might be to cross-develop Bart PE to be used as an extension of the MS WinPE environment. This is suitable for techs with an \i386 file set, but is tricky otherwise. On the other hand one could easily build a gutted Bart with no bootability, no OS, and just the nu2menu interface to collected tools.

MS WinPE 2.0 seems harder to build and use than Bart PE. The Bart equivalent of those 4-page walk-throughs is "install Bart PE builder, run it, and follow the UI that pops up". Once you have the skills, similar things should be possible in MS WinPE 2.0, but those who find the Bart skills investment too high will bounce off WinPE.

MS WinPE runs in RAM, allows the boot CDR to be removed, and allows arbitrary CDs and USB sticks to be hot-swapped - so the potential is there to launch your Bart CDR from a MS WinPE boot via an accessible "pull tab". On the Bart side, you'd either have a Vista-specific nu2menu, or logic within the menu and/or plugins that filters and modifies things to work (or not be offered) in Vista.

My Bart project is already down this road, built for compatibility with...

• Running from output subtree on the build PC
• As accessed from DOS diskette boot
• Autorun from Windows 2000, XP
• Autorun from Windows 9x
• Booted into Bart mOS

...so adding "run from MS WinPE boot" is just an extra step.

You can already manually run a Bart PE CDR's menu from an MS WinPE boot session. Some tools don't work as needed code files or registry settings aren't present in the WinPE runtime, and that may be a ball-breaker to fix. Some tools don't run due to permissions issues, and may run if it were possible to do the equivalent of right-click, Run as Administrator within the WinPE environment. Others work just fine.

The nu2menu system has some breakage when run from an MS WinPE 2.0 boot. So far, one issue I've picked up is that the very useful GUI file/folder selector feature fails to populate the namespace tree, so you can't use that to set Temp etc.

As an MS WinPE builder, you'd likely have or will soon build skills in XML. The nu2menu system is XML-based, and could be used in two ways (as I do in Bart):

• UI front-end for the build process
• UI front-end or shell for the mOS runtime

The MS WinPE build process would benefit greatly from some sort of GUI front end, and nu2menu would be useful for that. It runs as an unobtrusive button in one corner of the screen, and from there it fans out like the traditional Win9x Start menu, but with a lot more power and logic.

HD Replacement and Bad Cluster Markers

This is a common scenario...

• Hard drive develops bad sectors
• File systems have some clusters marked as Bad
• You image partition(s) from old hard drive to new replacement
• New hard inherits bad cluster markers that are now inappropriate

...that can present in interesting ways, if file system is NTFS:

• BING images the sick hard drive OK
• BING images the contents to new hard drive OK
• You use BING to shrink the partition
• BING checks the file system OK
• BING shrinks the file system OK
• BING re-checks the file system, and fails!

The FATxx equivalent is more like this:

• BING images the sick hard drive OK
• BING images the contents to new hard drive OK
• You use BING to shrink the partition
• BING checks the file system, fails!
• You manually clear bad cluster markers
• You use BING to shrink the partition
• BING checks the file system OK
• BING shrinks the file system OK
• BING re-checks the file system OK

What is going on here? Well, it depends on the file system.

With FATxx, BING sees the bad cluster markers when it checks the file system before it attempts the resize, "sees" the bad cluster markers inherited when the file system was imaged off the old hard drive, and backs out of the resize. Only when you clear these bad cluster markers (as you should) will BING attempt the resize, and will then generally succeed.

With NTFS, BING fails to see the bad cluster markers and is happy to resize the volume, which is dangerously inappropriate if the volume is still on the sick hard drive. NTFS points to bad clusters by cluster address; if these point beyond the end of the file system after the partition is shrunk, they are seen as invalid by BING's post-resize file system check.

Clearing Bad Cluster markers in FATxx

If there's a better way, I haven't found it! What I do is edit the file system on the new hard drive using ye olde Norton Disk Edit, selecting the two FATs in different panes, searching these for the raw byte sequence that marks a bad cluster, and editing these to 0 (free cluster available for use). When done, a file system check can be expected to detect FAT32's boot record free space value to be incorrect, and I allow that to be fixed.

Obviously this is serious caveat territory, and I look up FAT details to see what the Bad Cluster value is, and then run into byte-order toe-stubbers while searching for these. Norton Disk Edit runs from DOS or a Win9x DOS mode, and DOS can't safely "see" beyond 137G, so more caveats there.

Clearing Bad Cluster markers in NTFS

I haven't found an easy way to do this, but the process is documented at Katy's Homepage - so if you know of any free tools that can automate this, please comment with details!

If the Bad Cluster pointer is invalidated by shrinking the NTFS volume or partition, as described above, then the invalidated pointer will be "fixed" (cleared?) by ChkDsk /F

The problem is those Bad Cluster records that remain valid, and thus are not cleaned up by ChkDsk /F, or (if marked clusters are not re-tested) by ChkDsk /R.

Blogger! Can't Update Site Links!

When I first set up this blog, I entered links to my main web site and older Win9x web site, while leaving in the Google News link that was there.

Hmm... according to Google News, "Palestinian Leaders Plead for Clam"... not clear if they're asking for clam to be served them as food, or perhaps clamency for an aquatic comrade in arms?

Anyway, what I wanted to do was add my new Vista blog, but I can't see anywhere to edit that list of links - and I did look everywhere. Strange!

Internet Explorer 7 Exits When Started?

I see Microsoft has an article describing this behavior, though in my experience the problem may be more general than the article suggests, though the mechanism may be the same.

What happens is that whenever you run Internet Explorer 7 (IE 7), or things that invoke it such as Windows or Microsoft Update, it vanishes off the screen as soon as it appears, with no error messages as to why it has done so.

I've seen this in previously-working IE 7 installations, but more commonly after something has interrupted an IE7 installation. What would do that? Automatic Update, that's what... as seen in the "Windows Bugs" photo set at my other blog.

I've seen this regularly, as others have not, so I wondered why this might be. Perhaps it's because the PCs in question have a lot of old patches to catch up on, so Automatic Update gets active as soon as they reach the Internet, plus I often install IE 7 from a saved copy off CDR at around the same time.

What is supposed to happen is that the Internet Explorer 7 install does its thing, then prompts you to restart Windows within its own series of successive blue dialog boxes. Instead, these dialogs are still indicating files being installed etc. while Automatic Updates pops up its usual grey dialog telling you to restart Windows, and if you cancel that, it will pop up again and again.

I've always wondered whether Automatic Update co-ordinates itself with what Windows or Microsoft Update are doing, or whether the same material gets downloaded by each at the same time, doubling the bandwidth consumed. This case suggests problems of that nature; the IE 7 installer should trap and disallow (or gracefully clean up) software-initiated shutdown requests, and/or prevent other items installing themselves while the IE 7 install is in progress. Similarly, Automatic Updates should detect Microsoft's own installation activity, be this locally or as managed from update web sites.

Sun Java JRE Bloat

Well, it took Microsoft long enough to finally scale down Internet Explorer's ridiculously bloated cache allocation; Internet Explorer 7 follows other browsers in sizing this to 50M, irrespective of hard drive volume size, and it may complain that the present cache size is too large (e.g. if it was set so via user).

However, what Microsoft has finally learned, Sun is still getting wrong. After installing the new Sun Java JRE 1.6, I saw a Java SysTray icon, and poked around; there's a slider for temporary file cache to be allocated to Java (separate from browser caches) and the duhfault is 1G! Needless to say, that got scaled back to 20M pretty quickly.

So, there's a new bloat factor to remember when folks run out of space on C:, over and above the bloat of multiple Sun Java JREs, as discussed earlier in this blog. At least new JREs no longer pass control to older (exploitable) ones when requested to do so by Java malware (sorry, "valued Java applets"); still, at 100M+ a pop, old JREs aren't too cheap.

Bad RAM, Bad RAM Tester Design

This long post covers Vista's mOS, MemTest86 and Microsoft's stand-alone RAM testing utility.

How bad RAM presents

If RAM was originally OK, then goes bad, you'd start to see random errors, crashes, lockups, reports of corrupted registry or other files and operations, and perhaps some spontaneous resets. This random pattern may develop some reproducible errors, where the contents of the hard drive have been corrupted, either from bad RAM per se or from recurrent bad exits.

RAM crashes at full speed, so you won't notice any slowdown of the system. This contrasts with bad sectors on the hard drive, which slows the system due to attempts to retry the operation, and/or copy contents of failing sectors to spare sectors. On most consumer PCs, there's no attempt to detect RAM errors after the BIOS boot phase; where such detection is present, the system will usually halt as soon as a RAM error is detected.

Why bad RAM matters

RAM errors can not only corrupt what is written to disk, but also where it is written to disk, at a level beneath that of the file system. A sector intended to be written to the contents of a file may instead be written over some core file system structure, e.g. if a hi-order bit in the raw sector address is flipped from 1 to 0.

RAM errors can corrupt code, causing crashes, but a greater risk may arise where the code does not crash. Many disk operation calls may use a status byte in a register to differentiate between read and write operations, so a bit-flip there could cause a write instead of a read. This is why no disk access can be considered safe; any disk access starts with reading crucial areas of the file system, and if those reads become writes, the disk contents could be trashed.

Why bad RAM may be tough to exclude

When I started out with PCs in the era of DOS 3.3, 286 processors etc. I wondered why there were so many RAM testing utilities around. Surely you would just copy data to a spare register, write it into RAM, read it back from RAM, and compare with the spare register?

I had found that in practice, several testers would pass RAM as "OK" even though swap testing would clearly demonstrate that problems would clear up on the suspect PC with good RAM and appear on a known-good PC with the suspect RAM added.

So I though a bit more about how RAM can fail; not just by returning different data compared what was written to it, but altering data in other addresses when certain addresses are accessed, or behaving differently according to whether the RAM is read for instructions vs. data, or whether it is being accessed by the processor, AGP, or some other device via DMA.

Also, some failures can crash, lock up or reset the system, instead of being presented as a nice list of bad addresses. If the RAM testing boot disk is left in the system during the test, a spontaneous reset may be missed, unless you happen to notice the test has been running for fewer hours than have elapsed since you started the test.

For a long time, I gave up on RAM testing utilities, and just did swap testing as above. My faith in RAM testers started to return with SIMM Tester, and grew stronger with MemTest86 and MemTest86+. But I find that even with these tools, either one of the two MemTest86 projects may detect errors where SIMM Tester does not, and 8 hours of MemTest86 may pass, only to throw errors somewhere in the next 12 hours of testing.

How to design a RAM testing utility

This isn't about test sequence and data intended to provoke errors due to local power starvation or whatever. Instead, it's about how this core of test routines should be wrapped into a safe and usable utility - as illuminated by issues raised earlier in this post.

Microsoft have a free stand-alone RAM tester that is called the "Windows Memory Diagnostic". But why is "Windows" in the tool's name, given this is a tool that should run at the system level, before any OS has booted up or is left running in the background?

I used this stand-alone form of the tool, and noticed something rather nasty about it - when set to repeat the test sequence, it clears the results of all previous test passes! It also does not indicate elapsed clock time, so if the tester disk is left in the boot drive, the test will restart and look exactly the same as if it had been running without any interruptions.

Any RAM failure is significant, even if it shows up only once in 24 hours of testing. If you use MemTest86 and one such error occurs, you will see it listed when you return after an overnight unattended test - whereas even if Microsoft's tester flagged it at the time, you will only see the "OK" result of the last test pass when you return in the morning.

There's no point in doing 24 hours of testing, if only the last pass (possibly the last 20 minutes of testing) is reported! Who is going to sit and watch an "unattended" RAM test loop for 24 hours, just in case one pass fleetingly shows an error on screen?

How to integrate RAM testing with a mOS

I'd love to include RAM testing within my maintenance OS, but I can't see a way to fully automate this. The mOS boot disk should not boot past the RAM testing component into loading the full mOS, because that involves a lot of disk operations that may be unsafe when RAM is bad. There's no safe and standard way that the RAM tester can set a flag that it is in session, that will persist after a spontaneous reset. The best I can think of would be to boot the mOS to a menu that defaults to testing RAM, but that does not timeout but will wait forever for a keypress.

So I can't see a safe way to incorporate RAM testing into a wizard-driven mOS intended for unskilled use. It would be lovely to have a boot disk that would do x hours of RAM testing, then test the hard drive for physical errors, then test and possibly fix file system logical errors, before commencing with formal scanning for malware. But without a safe way for the mOS boot to detect whether RAM had been recently (define "recently") tested without errors, the best I could design would be a mOS that booted to a 3-item menu (test RAM, continue with the wizard, or display help) and stayed there until a selection was made.

How to get all this sooo wrong

The good news is that the Vista DVD has RAM testing incorporated into the mOS. The bad news is that Microsoft made just about every mOS design mistake possible:

1. mOS boot will fall through to hard drive boot unless key is pressed
2. mOS runs a lot of code before the UI from which RAM can be tested
3. mOS looks for a Vista installation on hard drive before anything else
4. mOS drops RAM tester on hard drive, then reboots to run it
5. RAM tester does one pass only, unless this is overridden by user
6. RAM tester displays no results on screen
7. RAM tester writes results to Vista installation's logs on hard drive

OK, let's walk through what happens when you test a system that may have bad RAM. Microsoft seems to expect this RAM to be so bad that a test single pass will catch it, even though we know from experience that you may only see one error in 24 hours of testing (mistake 5).

If RAM is so bad that one test pass will always catch it, then it is surely too dangerous to run large complex GUI code (mistake 2), or to read into the logic of a Vista installation on the hard drive (mistake 3). If BIOS standard practice is to halt a system whenever bad RAM is detected, irrespective of what the OS was doing at the time, then surely it is foolhardy to boot up a complex OS from the hard drive (mistake 1), write material to hard drive (mistake 4), especially if the RAM has been proven to be bad (mistake 7)?

What happens if the nature of the defective hardware causes the system to reset, rather than lock up or carry on running so the tester can flag the error? Well, the Vista DVD will chain into the Vista installation on the hard drive and boot that (mistake 1), which is about the worst possible thing one can do - and this will happen even if you had explicitly excluded the hard drive from BIOS bootability, because the DVD boot chains directly into it irrespective of any BIOS-level settings you may have applied.

If the RAM did test bad, how would you know? It seems as if the only way would be by booting Vista from the hard drive and scratching around in Event Viewer. If the process of writing those results into Vista's logs didn't corrupt the contents of the hard drive, then booting Vista (with all the attendant paging, temp files and registry updates this may imply) to reach Event Viewer may well do so.

This is a bit like being a driving license tester faced with a pupil who immediately tries to mash down pedestrians a la Carmageddon at the start of the test. It's nice to see Microsoft (at last!) taking an interest in maintaining sick systems, but the lack of insight displayed is scary.

Learning Vista

See http://cquirke.spaces.live.com, which is where I'll blog my initial bewilderment and hopefully progress as I actually work with (as opposed to, look at) Vista.