The Latest Arrival

Under the hood


Well I finally gave in.  I have not enjoyed a book like this in a while, my brain was just soaking up the knowledge like a sponge last night when trying the Kindle sample. Yup I have gone and done it and got the Windows Internals book.

I just love knowing what goes on under the hood as it were. I have a good understanding of Windows internals from studying the Win32 API and COM fundamentals in the late 1990s early 2000s but this will give a good up to date look into how it all hangs together.

I went for the fifth edition mainly because the sixth is split into two parts and is not that much of an incremental change (6.0 to 6.1 of Windows). Plus the second half of the sixth edition is not out yet and this covers bits I am particularly interested in, that and Server 2012 is on the horizon too so guessing I will go and become one with the couch shortly.

User Interface Reading

"X marks the spot"


Learning how to read a user interface, or at least trying to actively see the interface you are using rather than accepting blindly "I click on this button and X happens..." is a good skill to develop.

What do I mean by actively seeing the interface?  Actively seeing is the act of trying to understand why the particular user interface, you are using to do your given administrative task, is designed the way it is and also inferring relationships and meanings from groupings or hierarchies as displayed.

For example on a given tree node snapin such as Computer Management (compmgmt.msc) we see a logical grouping of nodes :

The trick here is thinking to yourself why these nodes are arranged as such, the benefit is if another node is added, say Microsoft Message Queuing (MSMQ) you should have a good idea where to find the added node (Services and Applications for the unfamiliar).  It also allows you to collapse and ignore the items not relevant to your task.

But the basic thrust of what I am trying to get at is a user interface designer, hopefully, sat down and thought out this design and relationships and did so for a reason.  Once you learn to read the interface like a pro, coming into a new possibly unfamiliar layout these skills will allow you to orientate yourself quickly and look like the uber admin you always wanted to be.

Lets take a more complex example, the Exchange 2007/2010 Exchange Management Console (EMC) interface :

Now the initial reaction of a lot of administrators coming from the old Exchange System Manager is that they have repeated the nodes a few times, "Huh, there are two Hub Transport options, what gives?" etc...

The trick again here is to think why (always why).  In this case it is to allow tasks at different levels to be performed or even delegated.  One set of nodes allow you to perform actions that affect the entire Exchange organisation (Transport Rules for example) and one allows you to manage the Connectors for a given Hub Transport role.  Same for mailbox, one allows you to (in Exchange 2010) manage the Database Access Group (DAG) on that abstract level, the other allows you to manage the individual databases making up the smoke and mirrors that is the DAG to users.

This design also allows for the separation of roles and their installation to separate servers, pretty flexible, huh?  Even with the brief explanation above it is a good idea to sit down and have a think, especially if you interact with the EMC on a daily basis, it will give you a hopefully deeper understanding of how Exchange works and how the architecture of this system has changed in the newer versions.

Not all designs are great though, take the delete and refresh buttons on Disk Management :

I don't know about you but for me those buttons are way too close together for my liking.  The thing to take from this is again active seeing, make sure you are not clicking wildly on buttons and just OK-ing dialog boxes or you may have an interesting night trying to rescue a server...

Arr Tee Eff Em

"Physician, heal thyself"

In my mind there is nothing better that a technician can do to enhance their career than being able to find information out for themselves. I have mentioned it in previous posts, and will probably continue to do so as it is a theme of mine.

At a minimum reading the manuals or "RTFM" as the well known acronym goes (Read The F**king Manual for those unfamiliar with it) for the products you support is a must. Not all vendor documentation is created equal but the bigger players tend to have professional writers employed, so that goes a long way.

Most if not all documentation will be in soft copy or online, gone are the days of the product coming with printed manuals. I remember back in the day a couple of friends buying the Borland C compiler (ah you don't hear that name too often these days). It took two of them to carry the box home given the volumes of manuals supplied. Now that is value for money.  But these days the documentation will be in a sub directory on the installation media, available for download from the vendors support site, or purely online (ala Microsoft server product documentation).

Knowing how the product you support works, from at least the vendors point of view is a good start. Coupled with day to day use of the product will allow you to maintain and support it.  Following on from the vendor documentation are the many a varied books on a given subject.  It used to be that you would trundle home with the latest 1000 page tome on a given product and add this to your bookshelf, and in some cases actually even read it.  These days e pub is king, a Kindle is pretty cheap option, yes it ties you into the Amazon ecosystem but with products such as Calibre you are not necessarily tied down that much.

Having a core library of books on your electronic book reader of choice is a great idea, as a Windows System Administrator I have books on Active Directory, Windows Server, SQL Server and Exchange on my Kindle application among others, but having a good all round go to library is a must have these days.  Couple that with vendor documentation in PDF form and Adobe Reader or similar for your device and you are all set.

Being able to figure out what is what on a given server, taking a lie of the land as it were if you are new to the job or the job involves supporting various client set up is good, I have lost count of the times I have fired up MSInfo32 to check if the server is physical or virtual (tip look at the manufacturer), or in the case of VMWare you can normally tell from the system tray if the tools are installed (but not always).

I will be expanding on this theme of self reliance and finding things out for yourself and various tools, utilities and techniques involved.  I will leave you with one thing. If you are a Windows server or Active Directory administrator I encourage you to look at NLTest, one of my favorite utilities, you can find out a tonne of information with it.

Fun with Windows Server 8 Beta

"I'm sorry, Dave. I'm afraid I can't do that"


I support, along with the usual Microsoft server products, at work Symantec Enterprise Vault.  Recently Windows Server 8 beta was released.  I had mentioned this in a previous post, but now it was time to try it for real.

I created a VMWare Workstation virtual machine and spun up an image.  Then just because I can (you know that guy thing where we just do stuff to see if we can) I decided to see if the latest and greatest version of Enterprise Vault, Version 10, would even look at installing on this new car smell operating system.

At first signs it was promising :

Even this new OS did not phase the installer, and we are good to go, right?

Well not quite, it failed the .Net framework test, yes Windows Server 8 Beta comes with the .Net Framework version 4.5 but this is no go, good old Enterprise Vault needs version 3.5.

No problem I thought to myself we will just add this "feature" via the Server Manager :

Great this is looking promising :

Excellent we can add the required framework, everything looks good so far.

Then in the words of Homer Jay Simpson. "D'oh".

And we were going so well there, at least Server 8 gives you a nice Web 2.0 "flag" to show you where it all went wrong.

Yeah pretty, but not too helpful.  Aha I thought, I will just do what we do on other Microsoft operating systems.  I then proceeded to download the offline installation of the required .Net Framework.

As we have always done, a simple double click and a follow of the white rabbit, I mean wizard would have this installed lickety-split.  But no it has all changed, hence the title "Fun with Windows Server 8 Beta". Upon that innocent double click we get :

Odd, I did not choose "Windows Features" all I did was double click an installer that well, just installs on all other Microsoft operating systems.  Just to rub it in we then get :

So it's my way or the highway, you must use Server Manager.  Well good to know.  A little digging around on the web revealed that the actual cause of the install failure is due to lack of connectivity to the Internet.  Now the virtual machine is NATed to the host, so that should be OK there, but a quick check of the TCP/IP v4 settings showed :

Yes I could have set the IP settings to static, but this is just a play around server, and I am just seeing if Enterprise Vault would even install.  All that is needed is to add a secondary DNS, and Google DNS will do as well as any, then a quick trip back to Server Manager and vuala as they say :

Now all I need to get sorted is the ASP.NET 2.0 requirement and i should have it all installed. Fun and games indeed.

Communication

"Hello, Is there anybody in there? Just nod if you can hear me. Is there anyone home? "


Communication in a busy support environment is vital.  Nothing happens in isolation and the free flow of information from you to your co-workers and back can make for an efficient environment and also help present a unified front to customers.


There is nothing that makes you and your company look unprofessional to external customers and suppliers more than someone calling or emailing them about something a) you have already been notified about, such as planned downtime, b) having to repeat the exact same information to yet another tech about the problem they logged earlier in the day, or c) two techs calling about the same problem within minutes of each other.


They may not say it to you on the phone, and if they do count yourself lucky that they have as you are now aware of it and can catch this bad habit and hopefully with managements cooperation nip it in the bud. But you have no idea what they are thinking and saying to the right people in their company that could affect the next contract renewal when it comes around. Sure you are just a "low level support grunt" how can this affect you, yeah it is going to suck for the sales guy but hey that comes with the territory, right?  Well if enough companies decide to do business elsewhere than your current workplace, there goes your livelihood (as with most of the people in this world you probably rely on one income source) and replacing that is not going to be as fast or as easy as you imagine from within "safe" employment.


So how can you improve communication, well in a support environment, be it the traditional internal IT department or an outsourced help/service desk, you should have four (well five) lines of communication.


So what are these lines you speak of?


1. Talking


Yep that old fashioned get up and go talk to someone and tell them something technique (or if you want to tech it up and save burning a few calories, telephone them).  But nothing beats a quick face to face information dump and you also make sure that they have got the information.  Though just to be sure you should follow it up with the next line of communication.


2. Email


Email is great for two things, the communication is asynchronous, you can fire and forget and carry on with what you are doing. Say for example you have a router down to site a quick email fired off to your team even with a one liner subject like "Main Campus Building A router down, working on it" will alert your team that the servers behind this router are inaccessible so they don't need to waste time problem solving that. It also allows your first liners to smoothly handle calls or even call designated department contacts to stem the inevitable flow of "is the network down?" calls.  Another advantage of email is it gives you an audit trail (always good if you need to talk to management post outage) and also covers your butt as you have something with a time stamp showing when you were aware of the problem and working on it (though of course this would have been logged in your ticketing system, as any good sys admin would).  Which brings us to our third line of communication.


3. Your ticketing system


Not a lot of people think of the ticket/case/incident logging system as a form of communication but it is, say you are working on a complex case and encounter a bus incident.  No I don't mean the bus in your PC dies, I mean you get hit by a bus and are not able to communicate with your co-workers, having good call notes that are able to communicate the problem and the steps and logic you have used up until the bus incident will go a long way to help your colleagues out while you are mending and not thinking about work (you aren't thinking about work, right?) It also ensures continuity of service to the customer even if you experience a less catastrophic event such as a long boring, er... I mean productive meeting that you absolutely can't get out of.  Someone else can pick up where you left off and carry on providing that gold level of service you and your team are renowned for.


4. Instant Messenger (or IM to the internet generation)


Instant messengers are really useful in a support environment as they are for gossiping or organising that night out on the town with your buddies or gal pals. They can allow you to fill someone in live while they are on the phone to the customer with helpful information so y'all can look smooth to external clients.  Much like a news reader reading autocue while listening to their director over the in ear thingy.  IM is also great for asking quick questions or pointing people at useful technical articles and other on-line resources.  IM is pretty easy to set up, there are even some free Jabber based servers and clients if you are on a tight budget.  I would not recommend using a public based IM service such as Google Chat or MSN Messenger simply because you may need to send things like server IPs and names, and other bits of sensitive information via IM and you want to control where this flows through. I should not have to mention that you should never send a user name or password via IM (or even email for that matter, but secure thinking is whole other topic).


But wait you said there were five lines of communication, and I sure did.  The final line of communication is your documentation.  You do document the systems you support?  Of course you do. Having good, or even some documentation can go a long way to fill in the gaps without the newly hired staff and administrators needing to trek up guru mountain to ask those on high how it all hangs together.  Duct tape and bits of string normally, if anyone asks.  So having a good, well populated documentation system can go a long way and save money and time in the long run.  Something as basic as a network share with sub directories of documents is better than nothing.  Or you could use SharePoint from Microsoft for an on-line collaborative project. Another option going down the open source route would be Mediawiki, the same technology that powers that great source of movie and comic book lore, Wikipedia. As this was designed for the common man on the street to contribute to editing and updating for System Administrators should be a breeze. And for the text config file editing averse Windows admins out there it can even be set up on Ubuntu server (sudo apt-get really is not scary at all) with the very friendly Cherokee web server (no need to edit Apache or Nginx config files in Vim or even *shock* Nano).  Pop that into a VM on your ESXi/Vsphere set-up and you are good to go.


Communication does not need to be hard and there are multiple ways in the technical support sector to achieve this, if you take away one thought from this article I would want it to be communicate well and communicate early. If you think something is going to bite someone in the ass, give them a heads up, they'll appreciate it and you.

A Workman and His Tools

"The right tool, for the job..."


I have been thinking for a long time what makes a good support person and sets them apart from the rest.  Much like a master craftsman or artisan it is the tools that he uses and the techniques he applies to day to day tasks that make a difference.  Much of this comes with experience and from other master craftsmen in the field and has little to do with methodologies and frameworks such as ISO 9001 and ITIL.

Don't get me wrong these methodologies have their place, but having pride in what you do and knowing what tools are available to you make a whole lot of difference.  In IT there are varying tools for sifting and mining for our life blood, that is information.  We are information workers so a lot of the time it is not so much what you already know so much as how fast you can find the information and assimilate it.  As expected the Web holds a wealth of this, we are also sometimes referred to as Google wranglers. It may seem like cheating to go ask the mighty Google but at the end of the day users generally want their email up faster than being interested in your pride.

Speaking of pride, don't be afraid to ask co-workers for help, even if only to bounce ideas off or to perform a sanity check, a second pair of eyes on a problem is a always a good approach. There is a balance though relying on others to solve the issues in your support queue does not win friends and influence people. As we IT support staff know there is never enough hours in the day and the stress never stops.

So being able to find things out for yourself is a great skill to develop, if when trekking up the proverbial mountain to the resident guru to ask a question and the response is "what problem solving have you done to this point?" it is good to be armed with answers.  It will certainly go a long way to assist your guru in helping to point you in the right direction (after all a good guru would never just give you the answer, how would you learn anything if they did?) and earn some IT brownie points in the process.

Logs are a great place to start, I don't mean those things lumber companies produce, I am talking about the various log files, binary or text, produced by computer systems and the applications they run. As you would expect on a Windows system from desktop to enterprise level server cluster the go to place is the event logs (Start/Run/Compmgmt.msc is the fastest way to get there, yes I know you could run eventvwr.msc but with computer management you have other useful tools all in one place).  The two mainstay logs to look at here are System and Application.

System funnily enough is where all system activity happens, on most systems this will be things to do with services and disk subsystems.  Application is where, you guessed it, applications are supposed to log information, warnings and errors.  Generally well behaved applications do. There can also be specialised application logs created just for information from that application (Enterprise Vault springs to mind).

Other logs are also created such as the W3C logs on windows systems, sure you can open these up in good old dumb notepad and go log diving but there are better solutions out there such as Log Parser, this is a versatile tool for querying logs of all types and descriptions.

Another great skill to develop is finding what is on a given network or domain, it is good to be able to connect to a server and figure out the domain and networking information from there. A couple of good tools (again Windows, but then again I am mainly a Windows system administrator) are nltest (built into most Windows domain controllers and IPScan (caution some anti virus solutions may flag this as malware). IPScan will tell you what devices are on a given subnet range and also the ports open on these if you configure it so.  Nltest will allow you to find out the domain controllers or global catalog servers for a given domain. Being able to find things out when there is either no documentation or someone to ask will go a long way to making you professional and relied on.

Being on the look out for new and cool tools is something a good system administrator should constantly be doing.  When you find one you should put it in your toolbox ready to use in the right situation. Like a trusty screwdriver or Leatherman multi-tool, it may even save the day, or at least the bosses email.

I could not finish without mentioning one set of tools every Windows sys admin should know and use, even those starting out.  These being the tools and utilities from Sysinternals.  These should form the core of your toolbox along with the tools built in to Windows.  Download a copy today plus you can always stay current with the Sysinternals Live option.

A New Flavour

"Mmmmm, minty..."


I have been using UNIX since tech college, this was before it became a university.  My first introduction to UNIX was AIX.  Back in those days it was all telnet and text no GUI.  We had fun learning the intricacies of ls and battling with Vi.  Though most of us enterprising students simply FTPed the files over to Windows and used Notepad.  Not to mention writing SQL against Oracle on that same system, it truly was big iron stuff or so we thought.

Fast forward a few years, called down to an issue at site while I was working in Wellington, New Zealand.  There was a problem with the document management system talking to a Sybase SQL server running on Netware.  The kicker, command line access only.  Thank goodness for the previous experiences at tech.  I certainly was not put off and we (the resident sys admin and I) got everything hooked back up.

The next experience was when I went to work for the major telecoms supplier.  This time it was Solaris, back when it was the dot in dot com (late 90's to the slow ones in the back).  Again telnet was the tool of choice.  The entire eCommerce system was running on that, using at its core an installation of Isocor X.400 and EDI.  It was also running MQSeries and Post.Office SMTP servers.  This is where I got my love of mail servers from.   The whole thing was stitched together from bash scripts.  But it ran.  I had a lot of fun in that job grepping SMTP logs, battling a SPAM relay "issue" for a weekend due to oops firewall rules (thankfully not managed by our group).  I guess being in that environment and coming back to Windows (we did have a Windows NT server or two running Isocor was well) really helped with the flexibility in my approach these days.

So what is this new flavour all about?

I have decided to live with Linux at home, it is a good way to keep my skills current and also investigate new ones, I have been meaning to install and play with Nginx for a while as I have read good things about this web server.  To this end I have installed Mint Linux, sure Ubuntu is great but the whole Unity/Gnome 3 thing, I don't know.  I prefer a more traditional interface and if by moving towards a more tablet/mobile interface it broadens the market for these companies that is great, but Mint hits all the right buttons for me.

I like eye candy as well so one of the deciding factors was the new Cinnamon interface offered by Mint.

I am certainly going to enjoy this journey, but for others it may also be a journey you want to take, a recent article is showing that system administrators with Linux skills can earn up to 5% above the average. It also states that there will be a demand for this skill set in the years to come and major internet sites (Facebook, Twitter to name a couple) utilise Linux and open source technologies.

I also see it every day at work.  The deeper you get into VMWare the more you bump into its Linux heritage. Plus a lot if not all of the SAN and storage technologies grew out of the *NIX space.  These days it can't be just a Windows world, so I would urge you to explore other systems and technologies, even if only via a desktop virtual machine.