Maintaining control of the repair/recovery process

One of the hallmarks of a self-healing process is the control applied to the width and depth of what parts of the operating system are automatically corrected back to an approved ideal state upon reboot. Based on requirements such as corporate policies, regulatory compliance, multi-user access and best practices, this setting changes from company to company, but also can be easily modified for various sub-groups within each organization.

To maintain proper control, an automated system must allow for or provide multiple levels of repair point control (for example-High, Medium and Low). A device will repair on every boot if it is assigned a Low/Medium/High level repair policy. However, an individual device (or group) can be assigned a “No Repair” setting to support an on-demand repair policy. This way, IT administration can control if and when a device needs to be returned to the last repair point. In fact, best practices suggest that repair should be implemented on demand rather that upon every reboot in order to maintain the continuous integrity of the permitted updates and allowable changes to the image assigned to that specific device.

When engaged to self-heal, Persystent always repairs the registry files (except for the keys excluded in filters). During boot up (after the BIOS loads), the process reapplies the approved image and the Repair Exempt filter. This way, any specific file and setting, such as Virus Definition Files, can be appropriately preserved. One of the chief benefits of the Persystent self-healing process is that the device is not being reset to a Zero-day state, but rather the last approved repair point. Additionally, the repair process only affects operating system and application files. The user’s data and files are not touched. A user profile is only impacted at the highest level of repair.

Further control of the repair process is evidenced by the flexibility of changing settings. The centralized WebUI allows IT administrators to change the repair levels against any individual or group at any time. This is done by simply adjusting the policy setting. This includes adding or modifying excluded files or other policies that can be applied to a named group of devices (i.e. identified by a characteristics like location, department, function or permission etc…) or defined by an event (i.e., updates, public daily usage by multiple users). The policies can be extended to when returns to ideal state can be scheduled and enforced.

The three levels of Repair:

Low Level Repair

  • Repairs any operating system or application files that are either modified or deleted back to the repair point state.
  • Deletes any new files/folders added in operating system and application folders.
  • User profiles are left intact. All change in the user’s profile are preserved and not repaired.
  • Any new files/folders created at the root of C:\ will be left intact.

Medium Level Repair

  • Repairs any operating system and application files that are either modified or deleted back to the repair point state.
  • Deletes any new files/folders added in operating system or application folders.
  • User profiles are left intact. All change in the user’s profile are preserved and not repaired.
  • Any new files/folders created at the root of C:\ are deleted.

High Level Repair

  • Repairs any operating system or application files that are either modified or deleted back to the repair point state.
  • Deletes any new files/folders added in operating system or application folders.
  • User profiles are deleted so that new user profiles will be created when a user logs on.
  • Any new files/folders created at the root of C:\ will be deleted.

Which repair setting is best?

Each company has unique compliance, security, device performance and administrative needs. This is why settings can be adjusted to meet specific requirements. IT administrators can add various policies that control the ability to add or manipulate certain registry, files, and services. Many companies enforce a variety of direct and unique policies that apply to a selection of their diverse user profiles. Most organization use low and medium settings for individual PCs based on the above noted considerations. High level repair settings are typically reserved for publicly accessed devices, classroom, kiosk and other multi-user machines.

Who chooses the repair point?

You do. Persystent’s imaging capabilities facilitates the creation and management of an image. A snapshot of this image is reapplied during the reboot process. When the repair is initiated, the self-healing (automatic corrective action) follows the repair level rules, exemptions and filters associated with the individual or group of devices and applies the last approved ideal state (image).

How often should a new repair point be created?

Best practices dictate that a new repair point should be taken immediately after authorized changes are made to the system. This can be automatically scheduled and executed with Persystent Suite. This includes changes such as Windows Updates, key application updates, installation of new applications, installation of new devices, etc… This will preserve the authorized changes and ensure the integrity of the repair process. Many companies schedule updated images weekly on a weekly basis; typically after the application of “Patch Tuesday” or a similar coordinated event. With Persystent, the process is considerably faster in that an entire image is not recreated. The process only identifies and incorporates the changes since the last approved repair point.

What exactly is repaired?

Depending on the level of control the self-healing applies corrective action against operating system and application files. However, this can be optionally expanded to include other files and folders that are not automatically part of the repair point by using our “Repair Point Include Filter” feature. The following is Persystent’s default repair point listing:

Default on Windows Vista, Windows 7/8/8.1
C:\Bootmgr
C:\Bootsect.bak

Captured on Windows Vista/Windows 7/8/8.1
C:\Windows (Excluding C:\Windows\CSC)
C:\Program Files
C:\Program Files (x86)\
C:\ProgramData
C:\Users\Public
C:\Users\Default
C:\Boot
C:\inetpub

The driving force behind Persystent’s multiple levels of repair is to allow for the maximum amount of control by IT while maintaining corporate standards of performance integrity. The flexibility of Persystent provides the right amount of protection, lifecycle expediency and compliance support for every machine under the enterprise umbrella.

With so many potential issues affecting critical systems, from user errors to malware infections to catastrophic failures (“blue screen of death’) IT departments constantly need to reimage machines from scratch or spend countless hours troubleshooting and repairing. The benefits of self-healing are obvious. It reduces helpdesk calls, promotes faster resolution of issues, eliminates the need for lengthy manual intervention, but most importantly it maintains a standard of performance through Persystent’s levels of repair.

 Download this article as PDF

Becoming a trusted adviser: helping clients prevent unforeseen expenses

As a service provider of any kind, the ultimate compliment is to be considered a “trusted adviser” by your client. But this status is more than simply getting a good reference or getting a customer to renew their annual contract. By its very title, a trusted adviser is an outside insider for a company. A consultant depended upon by an organization to provide valuable insight on how a company can best achieve its stated and latent goals.

For managed service providers, whose very purpose is to ensure various IT infrastructure and applications provide the expected results and value for the client, to become a trusted adviser means you have the responsibility to continually identify and implement ways to improve performance, anticipate challenges and constantly adapt to the transformative nature of technology.

Sounds easy enough. That is what you do, right? Whether you provide network support, security, help desk or a variety of other key services doesn’t immediately raise you to the level of trusted adviser. It simply means you provide an important service…and we assume you provide it very well.

Part of the trusted adviser’s job description is not only to improve performance, but to do so at the maximum level for minimal costs. The transition from service provider to trusted adviser means you are looking out for your client’s best interest, and not just service they can buy. To accomplish this, MSPs must address one of the biggest cost burdens that can affect the relationship: break/fix issues.

The labor required to manage this portion of the relationship is the biggest drain on margin. Regardless of whether a client purchased full coverage for a monthly fee, use a capped block of hours or pay out of pocket for each issue, somebody’s margin is affected when things go sideways.  It’s either money (margin) out of the MSPs pocket or out of the clients.

It’s not that issues arise, it’s just that the labor required to address problems is unpredictable. It could be a five minute fix or something that takes an application or network offline for an extended period of time while troubleshooting, fix planning and solution are applied.

Nothing erodes trusted adviser status faster than money. This is not to say an MSP should operate as a non-profit, but there are ways to proactively and automatically confront the break/fix issue without either side having to dig deep into profit margin. And, more importantly, provide a reliable means to attack unforeseen issues that eat time, upset productivity, and force reprioritization of potential revenue generating services. This is the road to trusted adviser status.

The ability to break out of “firefighter mode,” is the first step to creating lasting value for clients. The less time spent with your hair on fire, the more you can concentrate on tasks that support client business (and add to an MSP partner’s credibility and differentiation). For many MSPs services surround 6 general areas of coverage:

  1. Network Support
  2. Backup and Recovery
  3. Security
  4. End User Support/Help Desk
  5. Compliance
  6. Extra consulting services

The one constant through each of these services are the likelihood that break/fix will occur sooner or later. The ability to mitigate the risk associated with these problems and the labor required to properly diagnose and repair them can by automated configuration.

This doesn’t suggest a simple recovery tool. Instead of applying hours diagnosing and repairing, systems can self-heal upon reboot. It takes the client’s ideal image and removes the service issue. It’s simple. It’s automatic. And it removes problems that would otherwise require manual intervention and desk side visits.

Of course this doesn’t solve every problem, but if it can remove 60-70% of user-inflicted issues like changing critical settings, downloading malicious viruses, making unauthorized application changes, deleting necessary dll files, disabling BITS, and thousand other actions that compromise infrastructure integrity, not only are significant dollars saved, uptime and asset availability increased, but expensive personnel time is saved for higher value tasks.

There are several other benefits an MSP achieves by including automated self-healing as part of an overall package.

Scheduled versus variable labor: Labor costs take a huge bite out of the scope of service–especially when it comes to break/fix issues. An MSP and their client can create more fiscally stable relationship through precision budgeting. The client knows how much is going to pay each and every month and the MSP gains the stable recurring revenue. By using configuration automation and optimization, MSPs can reduce the specter of additional pass-along costs to the client or avoid absorbing the additional expensive labor costs. Now the conversation can move from “how much” to “how to improve” (from reactive to proactive).

Expand geographic reach: Many MSPs operate as regional entities because they do not have the personnel or the budget to adequately cover a larger (or even national) territory. From a cost perspective, self-healing eliminates a great many client visits. Typical on-site services like device restoration, no longer require a warm body in the room. This, in turn, reduces the need to travel and out-of-pocket time and costs. Without having to hop in a car or plane, you can provide effective service to a wider circle of clientele. Now when you visit a client, it is to provide proactive intellectual value and consulting expertise…or simply take them to dinner to thank them for the business.

Help Desk reduction: Resources show that by self-healing and rebooting to an ideal state eliminates more than 34% of all inbound help desk issues without manual intervention. If you consider that very time the help desk phone rings, it’s $20 (based on nat’l average). For more serious issues such as catastrophic device failure, infected operating systems/applications, unauthorized downloads, the cost is obviously greater–and not just in terms of tech/admin intervention, but lost productivity and potential loss of client trust. This doesn’t include scheduled maintenance tasks such as patching, updating and migration—which in itself requires a significant time and resource commitment. By adding a self-healing component to your existing slate of offerings, it reduces the number of help desk calls and, more importantly, allows an MSPs help desk pros to uncover root causes rather than continually fix the symptoms.

Removal of malicious changes: Through maliciousness or carelessness, your client’s network is under constant attack from botnets, malware, viruses and a variety of other negative impact influences. Although automatic configuration and reimaging can’t prevent Stan from sales downloading a suspect app or prevent organized element in Eastern Europe from worming into  a system, the continuous maintenance and reapplication of an ideal state can prevent lingering damage. Any time an unauthorized outside influence tries to change a registry, attach itself to a file, or embed itself in a supported application, the system rejects these modifications in favor of the ideal state…in real time. From an MSP perspective, this avoids the downtime needed to cleanse a network and helps preserve the continuity of critical information.

Of the six general service areas mentioned, it is obvious how configuration/recovery/repair/ reimage automation can helps issues related to the network, backup and end users, however some question the value to those who provide security and compliance services. The answer is simple. Although not a traditional security solution, it not only demonstrates control over network assets (as required in SANS, HIPAA, PCI and others), but enables the operating environment running smooth over the course of the lifecycle.

Because a trusted adviser is more interested in a long term relationship than any short term gains, it is imperative that MSPs find and propose new and innovative solutions to include within their base services. If clients consider a MSPs service as a commodity, then it is very simple to find another provider.

The difference between an expert and a trusted adviser really comes down to a single attribute: an expert provides good answers. A trusted adviser asks good questions. Can you reduce costs while increasing your quality of service?

Learn how to answer that question at www.utopicsoftware.com