Problems with CG3000DCR Cable Gateway

There isn't a whole lot of info out there for this issue outside of a well done YouTube video by Christian McDonald and a few Comcast Community forum posts so I wanted to make sure people hear about this problem. We'd been having some problems with our Comcast service for a few days and the techs decided it would be a good idea to try a different modem. We had an older SMC gateway (SMCD3G) that has been a workhorse model for Comcast for a number of years now. We got the latest and greatest business gateway they had locally - a Netgear CG3000DCR.

Speeds were pretty good with it right away but that very same day we had a number of HTTP/HTTPS timeouts. That stuff happens so I wasn't too concerned but over the next couple of days it became an issue that was not going away. After some investigation we discovered 2 things - The most brief outages were simply HTTP timeouts that were inexplicable. The longer outages were happening because the modem would reboot itself from time to time.

Latency was also an issue. HTTP traffic would be fast for established connections but establishing a new connection was very slow. This was not the case for PCs connected to a public network we have for some public terminals. Internet there was lightning fast and new connections were very fast. Our setup was very similar to that YouTube video I linked to above - one firewall static IPs through the Netgear, the other firewall DHCP through the Netgear.

Firmware was the latest 1.32.03. Signal was good from Comcast and configuration was a direct copy from the old SMC gateway. Everything really pointed to the static IP problems others are complaining about with the Netgear. This morning a tech stopped by to put an SMCD3G back in. As soon as it was hooked up all our problems went away. Fast internet, low latency, no timeouts.

So beware the Netgear CG3000DCR for now. I think it's just fine if your firewall is configured with DHCP on the WAN. For those of you with static public IPs stick with the SMC gateways for now.

Dangers of IT Exceptionalism

After reading a tweet by Joe The IT Guy the other day I got to thinking about IT Exceptionalism and the dangers of that state of mind. Here's how that conversation went:

I think I understand what Joe was getting at and I don't completely disagree with him. What I believe is that allowing a disconnect between "colleagues" and "IT" by dehumanizing them to "customer" is a hazardous move. Instead of addressing how we provide service we've instead constructed barriers and disconnected ourselves from the people we're supposed to be helping. Touchy-feely, kumbya, blah, blah. I know.

IT has spent too much time separate from the business. Business leaders still struggle with how to deal with IT and technology because they still don't understand that IT is part of their business infrastructure not some money pit a bunch of nerds constructed. Problem is that IT has liked the autonomy and power that go with maintaining the technology black box. The nerds enforced a power stalemate with the popular jocks that were in the executive suite. 

The other problem, as I see it, is that IT has a uniquely global view of the enterprise. Often times the only other group of employees that have that view is executives (and facilities maintenance...don't underestimate how much they know about your business!). This perspective and the fact that business leaders don't understand IT has spawned a feeling of IT Exceptionalism that at times feels like its infected IT industry-wide.

Considering our co-workers, or even users in general, as "Customers" really minimizes the roll that they play in IT's success and widens the chasm that already exists between IT and the business interests it serves. There's obviously an appropriate time to consider users "Customers - when the core function of the business is providing IT services. Obviously, as an IT admin you're dealing with customers in the truest sense in that scenario.

When we need an analogy for IT-types to understand good service we seem to always go with the "customer" one for some reason. As customers, do we experience good service because we're customers? Often we do not. Customer service is not the purpose of the business unless for some reason the business has decided it is how it will differentiate itself (I'm totally ripping off +Rob England  here). Profitability is the real purpose. So we're asking IT to treat users as business does - Like sheep to be fleeced and dealt with begrudgingly.

Business can ignore, mistreat, under-deliver, and otherwise abuse customers as long as customers continue to pay. IT has acted that way with users and colleagues all too often in the past and it worked because there was no other way to do things without IT. Now in the days of BYOD, shadow IT, and cloud services IT can be bypassed and users and colleagues are doing the end-run in droves.

Perhaps instead of considering users as "Customers" we should treat them as co-workers, friends, fellow humans, lovers, family. Whatever it takes to have empathy and patience with them. To treat the people we are helping with respect and kindness. We need to understand service and learn what it means to serve. We have a great chance to have a positive impact on our fellow humans in IT. It's always "Service as an Opportunity" - SaaO.

SaaO or Service as an Opportunity

Let's bring it down for just a little bit. Many people who read my stuff are big into IT service and specifically IT Service Management. Let's talk a little about the service thing a little here. We all know it's important, but why service? Why did you choose to be in service?

I just went through a staff meeting where the topic was change. The employee assistance program folks were talking about how to deal with change and how the grieving process applies to our coping with change. Almost every change scenario they discussed involved an information system of some sort. As the IT guy in the room it felt uncomfortably targeted. If Information Systems causes this much grief why did any of us get into it?!

Service is really important. There's a service component in many belief systems or religions. We have service to country, serving as board member, public service. A high value is often placed on service.

A friend who's active with youth in a faith setting posted a Halloween message on Facebook with "A chance to serve" by trick or treating for donations to the local food shelf. I loved the excitement in the post - It was a chance to have fun and contribute to a service that is very needed in the community. Double the reward.

That's just it: Every chance to serve is an opportunity - an opportunity to have an impact on another person. Many people go into IT because they're "good with computers" or they're the super technical types. There's certainly a place for that. What's really needed, however, is people that come into IT because they're good with people. IT is really about the people first, technology second and when you understand the impact you can have on people through your service...well, that's when the magic starts to happen.

When that user calls for the umpteenth time about an issue I'm completely sick of I think of the opportunity I have. We're doing IT to serve.

vSphere Snapshots Choke in Backup Exec, IDE vs SCSI

We've been fighting with Backup Exec 2012 and our vSphere environment for the last month or two. Most VMs would backup just fine but a few would randomly choke and the job would end with cryptic errors like:

Final error: 0xe0009578 - Unable to copy the virtual machine disk using the VMware VixDiskLib.
Final error category: Resource Errors

Followed by:

V-79-57344-38264 - Unable to copy the virtual machine disk using the VMware VixDiskLib. VixDiskLib_Read() reported the error: Unknown error

Ugh! Unknown error, how I hate thee! After much screwing around with different settings, migrations, updates, etc, etc I finally, and hesitantly called Symantec. In the end I was connected with Jason who is a first rate tech and obviously knows his stuff. Very grateful because I've not had good luck with Symantec support in the past.

He determined the errors were definitely not a Symantec problem. Backup Exec relies on the vCenter to snapshot the VM and then present the snapshot to BE to slurp up. In the end vCenter removes the snapshot and BE will eventually verify its data with vCenter at the end of the job. Debug logs from Backup Exec showed no indication of any problem. Everything seemed normal. But here's where we figured out something was amiss:

WARNING: "VMVCB::\\<machine path>.vmdk" is a corrupt file. This file cannot verify.

Ah. Everything is now pointing at vSphere. The data being presented by vCenter to BE looked corrupt to BE and it was reporting this in the Exceptions in the job log.

There was one machine that was corrupted every backup job and a few others that would randomly show up corrupted as well. The common factors were

1. All were converted machines from an old Hyper-V environment
2. All had IDE hard drives rather than SCSI
3. All were important enough for me to be losing sleep over!

It didn't seem like IDE hard drives should cause this problem. It's all virtual so what does vSphere care? But to test it out I followed VMware's knowledge base article about editing the VMDK files to convert from IDE to SCSI. It's a simple if not hair-raising task. 

Then I ran a full backup of the virtual environment. This morning I came in to find one of the machines that I did not convert to SCSI choked. The VM that failed every time for the last month backed up flawlessly.

Now tonight I'll convert the rest over to SCSI and hopefully put this issue to bed finally!

My Current Setup

After reading my friend Bob's blog post about his Unsettled Computing I thought I'd post a little about my current technology setup. When I really thought about it I had a much more interesting set of kit than I realized. Interesting not in the fact that it's amazing but interesting in how much I've embraced "good-enough" and mobile. Here's my current setup:

• Samsung Galaxy S3
• Gen3 iPad
• 2009 MacBook Pro 15"

That's really it and the list is in order of importance. True, I do have an ancient HP workstation with WinXP on it that I use for occasional scanning. We also have an HP EX487 MediaSmart home server but that's becoming more of an archive than an active server. For work I've been using a Dell Latitude E6430s that would be a good replacement for the MacBook if I needed one.

Back to the point: My primary devices are A) an Android device, and B) an iOS device. When I need to type a lot, the MacBook comes out. If I had a decent bluetooth keyboard I'd bet I'd use the MacBook even less. The performance and capabilities of the mobile devices is so great now that I've transitioned to a nearly portable experience.

When I got my first Handspring Visor in late 1998 I knew mobile was going to have a big impact on my life. I guess I never imagined it would nearly supplant my desktop needs. The reality is, with the tools available today on mobile, not only can I do nearly everything I want to with mobile I can do things I never thought would be possible on mobile.

Now, if we could just get batteries to last longer...

Target Market Fail?

Just opened a package for a device I ordered from <well-known security product vendor>. It was well packed and obviously done by hand. At the very top with the instructions folded up in it was a t-shirt. Nice touch.

You might notice the label says "L". As a co-worker and I laughed about that he remarked, "Who do they think their target market is?" I agreed. There does seem to be a tendency for sys admins to we'll endowed in breadth or mass. In my case and my co-worker's, we're more of the vertically gifted variety. Someone else will get to walk around promoting <well-known security product vendor>.

Communication is Good, or Why Opaqueness is Not A Feature

Another day, another fiber cut. This time the telephone company's fiber met another backhoe and phone service was jacked up for 4 communities. Data stayed up (thankfully!). This outage was very exciting because phone service was completely down. Outbound and inbound calling did not work at all for our PRI and calling into the telco's exchanges from cell phones either gave a generic wireless provider message or told you that all circuits are busy.

Fun times when you work in the financial sector and members need to call you about their accounts!

I tried calling the telephone company but, naturally, their phones were also down. Fortunately I live a digitally enhanced life and stash all data I run across in one system or another. Found our sales rep's cell phone number and gave him a call for a status update. A short while later I got a fantastic email from him not only describing the issue but it also had the internal ticket that gave all the details! He continued to give me updates throughout the day, always including the internal tickets with the details. This was pure gold! It had service area affected, details, what services were down, and ETA for fixes.

Just think: what if they had just emailed all their business customers' technical contacts the internal ticket? No one would have called, no one would have been angry (hey, fiber cuts happen and I accept that), everyone would have had what they needed.

Being opaque when dealing with customers and partners is all too often seen as beneficial to the organization. Making people come to you for information feels like it keeps you in control of the relationship. The problem is, even for vendors of commodity services, only your competition will benefit from you not engaging your customers. Today communication is cheap and data is plentiful. We expect every vendor to be able to keep us informed and that's not necessarily an unreasonable expectation. With minimal effort, even a limited amount of data can be given out and that might be all it takes to engage or satisfy the customer.

You have information systems and communication platforms, use them to add value to your service. Your customers will be happy.

Invisible IT

Job satisfaction is often a problem in IT. Service sector is hard no matter where you work. You might even quip in song: Seldom is heard, an encouraging word. Service, especially when focused internally in a business, is an expectation and really doesn't get the recognition other business units receive when good things happen.

People involved in service do receive, on the other hand, a great deal of frustration, complaint, and sometimes anger. We're the clean up crew when things go wrong. When someone calls IT it's rarely because something good happened. This is something people in careers that are dedicated to service resign themselves to knowing full well that stress will become a routine part of life. But IT isn't just involved in support of existing services; There's a constant effort to improve and upgrade existing services. Updated services get pushed out constantly trying to keep up with demand and business case needs.

Last week my job involved going live with a new fiber connection to all our locations. We'd been getting by with some bonded T1 connections and one location had a single T1. Communications were via MPLS over these small pipes. After months of wrangling we managed to get fiber installed and went from 3Mbps over MPLS to a fat 50/50Mbps connection at each location. This is no small incremental increase!

The new connection went live Wednesday night and I got myself all prepped for problems Thursday morning. Only one problem cropped up and things went smoothly. "Surely," I thought to myself, "Someone will be sending IT brownies, or cookies, or something Friday morning!" The network response was amazing, applications were faster than ever, and call quality was dramatically improved. Friday morning rolls around and....nothing. No cookies, no doughnuts, not even a job-well-done pat on the back.

Now, don't get me wrong, management understands the improvements and we're not talking about total radio silence here. It's just that, from a job satisfaction point of view, you always hope that improvements would get the same response as an organization-wide outage does. IT labors in anonymity until things go wrong. The good things we do are preventative and steer the good ship around the hazards ahead. The fact is, service will always be harder than being served. That's not a bad thing; It is something to aspire to. There's fulfillment to be found in the service and the job well done.

IT vs I&T

I think it's time to change our name from "Information Technology" to "Information AND Technology".

It's a simple matter of reflecting what we're becoming and what we'll be expected to do in the future. I was chatting with a colleague who is working on becoming a newly minted systems/network admin when we both recognized that sys admins and network admins are a dying breed. That's not for lack of admins: shake any organizational tree and a dozen or more spare sys/net admins will fall out. The fact of the matter is that outsourcing continues tp dramatically reduce the need for admins. Cloud-based services are quickly moving infrastructure and systems far away from the business and the few systems that remain in-house can easily be assigned to a managed services provider (Started by a sys admin who was fired before you. Wink, wink!).

The same goes for generalist technicians across the board. Their positions are expensive and outside vendors are pounding on the gates offering to do the job from dramatically less money. Technicians have long-ago become a commodity and their employment has been buoyed by business' inability to change course quickly.

Going back to the conversation with my colleague: What was the solution for those of us in the endangered systems/network admin game? Information!

IT has a special place in business. We're often one of the few business units that spans the entire organization. Everything we do affects every team in the business. Many times the only other department with this kind of reach is facilities/janitorial. IT is often inward-facing and serves only the organization and, as because of this, we have a unique perspective that allows us to see the bigger picture. IT knows what everyone is doing and how they're doing it. IT knows the goals, the directives, the strategies, and the methods being used to achieve all these things. IT also has all the data.

It's time for IT to break up the melding of Information and Technology. Technology is the stuff, the gadgets. The business doesn't need gadgets (except when they do), what it really needs is Information. An I&T department would put its focus on Information and less on Technology. Someone has to shepherd the gadgets and gewgaws. Information, the life-blood of business, is where the real effort belongs - Process, information delivery, integration, standardization, automation, etc.

The old joke among admins, "Go away or I will replace you with a very small shell script" is coming true but at our own expense. It's time to evolve our careers, leave IT, and create the I&T Department.

The Failed Promises of Tech

The relationship between user and technology is a rocky one. It's a constant give and take between a group of typically non-technical people who are obligated to use technology to perform their jobs. It's not always easy. After nearly 23 years we have finally reached a point where you no longer have to advertise for "Microsoft Office proficient" candidates, generally speaking. We've transitioned the workforce into the digital age but there have been compromises along the way. (Caution, broad generalizations to follow)

Change? I ain't got time for Change!

Backwards compatibility is evil. When you think about it, backwards compatibility delays the inevitable. The problem is that it builds up a false promise: We'll keep accommodating you until the end of time. As a sector, IT has been teaching an entire workforce the wrong approach to technology. Tech involves a lot of change; It's nearly constant. Even more so now that we're in the cloud where our destiny is controlled by someone else with no connection to our own organizations.

I'm pretty sure we did the backwards compatibility thing because technology used to be expensive. It took a lot of money to invest in the hardware. Often times you also ended up sinking a lot of dough in bespoke software because what you wanted to do didn't have a retail product yet. Any subsequent new OS version had to be backwards compatible to run all that expensive software. The next version of the custom software needed to perform the job of the old one plus the new features AND run on old hardware. It was the economical thing to do at the time and it's not a bad strategy. The unintended consequence is that it created a culture.

We've been doing backward compatibility for decades and now an entire workforce expects it. "This is the way I used to do this process. Now it doesn't work. Make it work the way it used to!" This is a statement that I imagine raises the ire of any first level tech after an update. We techs think to ourselves, "Of course it doesn't work, you nitwit! Read the email we sent explaining the update and changes to your process!!" But users don't do that, do they? They don't because we've taught them that we will be able to build systems around them that are minimally impacting. They won't have to change the holy and sacred process because we'll make the system backwards compatible. Thought will not be required, carry on as usual.

IT is 20% Technical, 80% User Therapy

The resistance to change is really centered around the level of effort it takes. We all have work to do. Adding in a change that requires learning and understanding doesn't help the work get done. It IS difficult but workers encounter change in their jobs in many aspects other than technology. They complain about those as much as any tech change we throw at them. New position created? Complain. New rule for checking out company vehicle? Complain. New TPS report? Complain. Got rid of the TPS report? Complain.

Its time for Tech to deliver on a new promise. The path of least resistance does no one favors. Change must be based on need and requirements. Yes, sometimes that means backwards compatibility too (I'm not totally against it). The success of knowledge workers will depend on embracing change. Change is going to come fast and it's going to be impossible to avoid. IT can be part of the solution here. If we're thoughtful about the changes we introduce and work to help the people we support be OK with change we can start to transition the workforce from grudging digital users to digital adoptees or naturalized digital citizens. I believe they can learn to fully embrace the digital culture and come to terms with their new homeland.

Some day support calls will not be prefaced by a user saying, "I'm not very computer literate or techy..."

Silence is Golden?

I was doing some dishes and had some music playing. I had to leave the room for a minute and took my phone with me so the music stopped while the bluetooth went out of range. Arriving back in the kitchen I went back to the dishes and thought to myself, "I should really turn that back on." But I didn't. There was lots of silence. Birds in the distance. Cars went by. Silence. And I was ok with that.

As a young person (I guess I am still young by many peoples' standards) I couldn't be without music playing somewhere. Perpetual tunes. I had a beat up Sony Walkman that was a constant companion. That first CD player got used. A lot.

Now as I get a little older I'm finding myself listening to music less. Is music something less interesting to me now? I don't think that's the case. Are the music choices out there less to my liking? There seems to be plenty of items starred in my Spotify account.

Maybe with age silence is more precious. Probably more so as a parent. The part that frightened me a bit was that I don't miss the constant music. Am I becoming less musical? I don't think so. More focused, perhaps. I think 4'33" makes more sense to me these days.

My Kids: Digital Natives

My wife turned on the TV and promptly our 3 year old began to pester her to put on a cartoon he wanted. This was broadcast so she was trying to describe how that works.

Her: "You can only watch what they show you. You can't choose whatever you want."

3yr old: "But I want cartoons! Can't We turn it on? "

Her: "It just doesn't work that way."

3yr old: "But it works on Netflix!"

Yup. And advertisements are just as confusing. They have a tough time telling them apart from the real content. Pretty fascinating to watch the them watching. I doubt they'll ever learn to play during the ads like we did.

Content producers and "broadcasters" need to learn that the paradigm has changed. Business as usual won't work.

Restricting Technology: Not a Substitute for Good Management

It ain't easy being a small or medium business. You're not big enough to woo vendors with big contracts but you're too big to do things by the seat of your pants. Once, you were small enough to be lax and agile like that but now you've put on some extra weight and those additional employees and obligations (and laws) sure slow you down. We sometimes joke about the old dinosaurs, the massively big corporations, that change slowly and are burdened with an endless bureaucracy; Layers of middle-managers keeping the noble hourly employee in check.

To me, it's been interesting to watch SMB and Enterprise diverge from their traditional MO on technology. Where SMB used to be so nimble and innovative there seems to be a trend toward restriction and outsourcing. Enterprise seems to be leaning more toward relaxing and "insourcing". Obviously, legal requirements put limits on what any business can allow and require a certain amount of accountability but outside of those requirements this is the trend I see. To a certain

This trend to implement big brother on the SMB level reflects two facts (as I see it) about business:

1. Once products become cheap enough vendors have to create an "industry trend" to sell to lots of SMB to keep profits up.
2. Managing employees is hard. Really hard. The coward's way out is to use technology tools to insulate you from that job so that you can "focus on your business again."

Number 2 is being exacerbated by number 1. There's a lot of great tools out there in the enterprise sector. They serve a purpose and can be useful. The thing is, enterprise payed for an army of consultants to come in and implement management tools properly. SMB rarely does this, particularly on the "S" end of SMB.

Enterprise seems to have figured out that the IT police-state isn't sustainable. It becomes a nightmare and IT loses a lot of its effectiveness. There's a happier middle ground and larger organizations seem to be actively working on finding it. IT is about providing solutions and working with information not operating a Chinese great firewall to restrict users.

Sadly, I don't see an end to this in the near term. Industry trends are like a runaway bus that take everyone along with them for the ride. SMB needs to stop trying to buy all the enterprise stuff that finally came down to their price point (If I read another forum post by an SMB ITer asking which SAN to get I'll lose my mind). Prepare for the future of your business today. Be strategic and remember that technology should be part of, or at least play a part in, your business plan. It's a tool. Use it.

Who Are We To Decide?

It's rather unnerving to read fellow IT workers' comments about how their co-workers in other departments use technology tools. It makes me wonder if IT degree programs and certifications could benefit from some philosophy or anthropology courses. Apparently big picture is not a common IT way to think.

I was reading a forum posting about a business where users refused to give up their gigantic pst personal archives (a common lament that I've made as well). Many of the comments revolved around taking control and user smack-down. It just made me shake my head and move on to something else. Too bad. I was hoping to find some good information in that conversation.

Experience is a helpful thing to have. I've fought the email battle before: old server, no expansion possible, users constantly over their limits and using their inboxes as file storage. After I got done beating my head against that wall I learned something: Users will use the tools you give them to accomplish their tasks and it may not be in the way you envisioned. It might even be the "wrong" way. It's an evolutionary process and, just like evolution, there's no wrong way if it accomplishes the desired result (and complies with law, regulation, policy, etc, etc).

We IT hotshots come along in the middle of the evolutionary process and scream, "You're all doing it wrong!" But who are we to decide what's wrong? Who among us hasn't lamented the benighted user's decision to use a tool in the most asinine way? And yet those same users get their jobs done and the tool seems to help them.

Efficiency isn't the holy grail we think it is. Nor is conformity. Providing appropriate tools and enabling workers to do their job is far more important. The real purpose of IT is to make things work so the business value increases. Unless IT is the purpose of the business, we're here to do as we're told. We are *not* the enforcers.

Service Heritage

What's your "service heritage"?

A few things I've read lately made me think a little bit about my own perspective of customer service and IT service in general. Obviously, the way I approach service is flavored heavily by the experiences I've had both as a consumer but more strongly by the fact that I've been providing technical services to customers for over 16 years now. So why do I approach service in the way that I do?

All the different jobs, lessons my parents taught me, even my beliefs (religious and otherwise) all influence my approach to service. I've written about the good service template but you can follow that template to the letter and still screw up service. If you don't understand why you perform service and have a desire to  help then you're going to have a miserable time. There's certain elements of compassion, understanding, and drive that need to be present.

My Dad loved comic books and I remember him breaking out his version of the quote from Spider-Man: "With great power comes great responsibility."

When I first saw that what I could do with technology was magic to most people I understood that I was also responsible for helping those same people use the tech. Until everyone is a digital native (and probably even long after that) there are those to whom technology is otherworldly magic. We who fix or bend technology to our will are superheros. It's our responsibility to use that power for good.

I believe strongly that as we do our work we have an opportunity to make a difference in people's lives, to be a light in the darkness, as the saying goes. Good service can do that. We talk about value. We talk about metrics and business goals. That's part of working in a company and it gets really heartless if you focus on the business and money. In the end we're all human, and for those of us who are the service providers, the chance to make a difference can be really rewarding.

Think about your service heritage. Why do you work in service? Remember the why.

You Don't Like the Password Policy, Eh?

I was taking a CPR refresher course at work this evening and the instructor started going over using an AED. He had grabbed the one that we have in our building to show us the different parts. As he pulled out the actual defibrillator and explained how to turn it on one of our employees quipped:

"So we don't need to put in a password? Great, because if we did we'd never get in!"

This was met with quite a lot of agreement around the room.

"Passwords, yeah!"
"Can't stand having to change them all the time!"

Yup. Guess I'll move that project a few notches up the priority list.

Beside the auditors, who likes passwords? No one, that's who! I'd rather you didn't have one because then you won't call me asking me to reset it for you after you forgot it.

It's important to always be working to find ways to listen to your co-workers. There's many issues simmering below the surface and unless you build rapport between departments and people you'll never hear about them. Not all the issues are technical either! I've found that people generally put up with things unless it so adversely affects them they can't perform their job.

All these things that you don't hear about have a negative effect and impact the way IT is perceived in the organization. It's public relations!

If we want to aspire to the Information part of IT as much as the Technology part we need to pay attention to what our co-workers in other departments are saying.

Proxima DP6850 Projector Lamp Replacement

I searched high and low for instructions on how to replace the lamp on a Proxima DP6850+ projector. Couldn't find any anywhere and I didn't want to take ours down from the ceiling if I didn't need to. With no instructions available I decided the only way was to take it down. Here's the replacement procedure.

The lamp door on this model is on the bottom (the top of it's ceiling mounted with the controls facing the floor). So I would have needed to take it down one way or another.

There's a single door and a single screw holding the door. Go ahead and loosen the screw and tilt the door back to reveal the lamp inside.

You will find two screws at the front of the lamp holding it in place. Loosen these two screws. There's a handle on the top of the lamp. Use a fingernail or screwdriver to gently pop the handle up and pull the lamp straight up.

Before you place the new lamp in, inspect the face of the lamp to make sure there's no packing material or contamination on it. Use a clean, lint free cloth to remove any particles. 

Take the new lamp and slide it straight down into the housing. Make sure you seat it fully. There are two plastic tabs next to the screws at the front of the lamp. Verify that they are in place and that you have pushed the lamp down far enough. Similarly, made sure the back left of the lamp is pushed down far enough to seat the power connector. Tighten the screws at the front of the lamp.

Put the door back in place and tighten its screw.

Now that you have the projector down it's a good idea to make sure the filter is clean. This is located at the front of the projector and is held in by a plastic tab.

There are three parts to the filter assembly: the frame, a filter, and a square fitting to hold the filter in place.

Gently snap the square fitting out to remove the filter. I prefer to wash the filters myself. Shake or blow out the majority of the dust out with compressed air then run it under running water from back to front to clean it. Make sure it is completely dry before reassembly. I usually roll the filter in a towel and squeeze it dry.

Reassemble the filter aligning the shape of the filter with the shape of the outer frame. Gently snap the square fitting under the tabs of the frame. If the fitting's edges are bowed in you probably don't have it seated far enough.

When you're done, make sure you reset the hour timer for the lamp. Turn on the projector and make sure there's an active video source or you won't be able to access the on-screen menu.

Press and hold the reset button on the projector's keypad for 3 seconds. You'll see the lamp hours at the bottom of the screen. Use the arrow buttons to select "0" then press the reset button again for 3 seconds. This should set the timer back to 0.

Satisfaction In A Job Well Done

I ran across a tweet from one of the local newspapers:

Clicked and and saw that they're still using the SSRS report I made a while back to export the data to a nice format and email out in an automated fashion. There's also some other reports out there that they use for similar reports and at least one other agency using the same records software uses the same reports I created.

It was a simple job for the most part (once I taught myself the schema to a completely undocumented set of databases, no primary keys, and a spaghetti mess of relationships) but I tried to make it as nice as possible and easy to reuse. I'm sure a real developer could've done a better job but I doubt anyone would've been willing to pay the money.

There's a lot of satisfaction in a job well done. Especially because, since it was work in public sector, it benefits a lot of people and helps improve transparency.

Remember kids: If you're looking for a job straight out of college and are looking to maximize your impact, seriously consider a job in local government. You won't make a lot of money, but if you work hard you can make a big impact in a positive way.

Loss

Why is it when you lose someone suddenly there's reminders of them everywhere?

I lost my Dad on April 27th. After his service a week later and another full week after that of the "New Normal™" I've never wanted to give him a call more that I do right now. It's been a strange few weeks of well intentioned people asking enough questions that I'm finally forced to explain that Dad lost his battle with mental illness. In a way it's good they press on so much; It's important to talk about mental illness. Too often it's swept under the rug. Discussing how we lost Dad isn't cathartic to me any more. Now I just miss him terribly.

We were planning to go to a Twins game with Dad and some other family members this weekend. My kids were really looking forward to it. We're still going but one of the big reasons I was excited to go isn't there any more. An emptiness has taken the place of someone I love.

The other day I imagined myself in the place of Inigo Montoya in The Princess Bride, lost in his alcoholic haze while off trying to find the 6-fingered man to avenge his father's death. That's a little how I feel, wishing I could find the 6-fingered disease that killed my father to avenge him. Even if I could I'd be just like Inigo in the end, growling, "I want my father back you son of a bitch."

No Phone - You're So Lucky!

I was eating lunch at home today when my phone gave me an email alert. I was reading a newspaper (GASP! I know! They still exist and I subscribe.) so I was ignoring the alert. My three-year-old son immediately informed me, "Dad, your phone just made a noise!". "Yup I know," I replied, "It was telling me I have an email. What is your phone telling you?"
"Dad, I don't have a phone!"
"Lucky you!"

Mister 'I'm-going-to-be-four-and-all-grown-up-soon' smiled at me with an understanding look well beyond his years and proceeded to tell me all about the quesadilla he was eating.

Now, I love my smartphone. My Galaxy S3 has been a wonderful digital appendage that has nearly replaced my laptop. I use it. Lots. So why did I tell my son he was so lucky? Is that really how I feel about these amazing devices?

I don't feel trapped or tied to the electronics around me. They are my arms, my legs, the tools I use to do the work I love. Using them is as natural to me as breathing and they are extensions of me. I may not be a true digital native but I'm pretty darn close.

I guess I feel that the absence of the omni-present smartphone represents an innocence to me. He's not exposed to the broader world that I see through the portal that is my smartphone; The special tool that reveals the entire world to the beholder. A digital palantir to those who are powerful enough.

One thing I hope to give my kids is innocence. I'm not protecting them from the world; It will come up and beat them over the head eventually no matter what I do. What I can give them are beautiful, wonder-filled childhood days where anything is possible, imagination is the only limitation, and weapons wielded by good-guys turn bad-guys into good-guys. We lose that all too soon. They'll soon have their Galaxy S IX or whatever.

Evaluating Software Suitability

The method I've always used for determining how well a software tool fits as a solution essentially boils down to this:

1. Document process
2. Build Use Cases
3. Evaluate software fit with use cases. Double check fit using process documentation.
4. Evaluate software's fringe benefits (nice features that don't solve primary use cases/problems)
5. Evaluate cost, ROI, vendor viability, etc
6. Consider how software will impact future systems and designs.

I know there is probably better terminology for these steps. This is basically my home-brewed method and it's based on the purchasing processes used in the various organizations I've worked in over time. 

I'm sure there's a framework out there for this sort of thing but I haven't really been looking for one. I just want a good, simple process.

How do you evaluate how a software tool will fit your organization? See any flaws in my method (I'm sure there are some)?

Dad

Saturday morning we learned that Dad had lost his battle with mental illness. This is not the way this is supposed to end. The arc this story took was not even close to the plot we wanted; It was supposed to be different. This is not the final word, though. Mental health is something we should not, and cannot be silent about. But that is just a footnote to the entire story.

Dad was a man who deeply loved his family. He had compassion for everyone around him. His honest caring for people made him a lovable fellow. Dad could ask people deep, philosophical questions that would take them off guard but he asked because he genuinely wanted to know. The Tom Beran we were so lucky to love was creative, artistic, witty, thoughtful, hard working, and so many other adjectives that a thesaurus would probably be in order.

His children learned many life lessons from him. The chores we didn't want to do were good for us because "they build character". You love your family because you've got them for life. You work hard and do it the right way because you'll probably be the guy stuck fixing it next time it breaks. It takes a strong man to be gentle. Pray about things. Sometimes you light the fireworks all at once with the torch because it'll be fun to watch and, well, why not?

I miss you Dad. I've been missing you in many different ways for a long time now but time does not make it any easier. I remember being little and wanting to be like you, just like I imagine my boys think about me today. I'm going to keep trying to be the man you wanted me to be.

I love you Dad.

Providing Services Isn't Enough

I was doing a demo with some co-workers for a product we are considering subscribing to. A nifty product that facilitates polls to determine the best time for meetings among many participants. While we were looking through the interface a few questions were asked by my co-workers about polling internal participants and syncing calendars with the product in question. They basically wanted to sync many employee calendars to the product to view the best time to set poll questions.

Of course, you can already do that in Outlook. The Scheduling Assistant has been there all along and does exactly what they were looking for in that use case. I flipped my screen over to Outlook and showed them how it worked.

"Why hasn't anyone shown us this before?!"

The response I got, almost verbatim.

Look, IT can provide services and tools and tech all we want but in the end we have to communicate to the end-user what they can do with the STUFF we provide and put it in a context that makes sense to their jobs.

Front line staff are a great example of this scenario. Whether you're dealing with bank tellers, cashiers, customer service reps, etc they are essentially the modern assembly line worker.Their jobs are 100% volume based. Instead of assembling widgets they're assembling customer service. Front line workers don't get paid to be innovative (the low wages they get paid reflect that), instead they are paid to be as efficient and consistent as possible. As a result, staff in those positions are often most resistant to change. They stick to the old processes like glue because change impacts their primary purpose, often negatively.

However, they'll accept change in a heartbeat if it the usefulness and context to their jobs are made clear! I've seen cases where a new tool I offered was immediately adopted because the benefit and usefulness was obvious. The acceptance was not solely because of me providing the tool, it was in me matching the tool with the need and the legwork it took to explain its purpose and use.

This is more than a training issue, although training is certainly an aspect when trying to get people to use a tool. When IT is not working side-by-side with their co-workers in other business units the worse this problem gets. Context is the most important ingredient.

The Longest 10 Minutes

The recent events in Boston have brought people to comment on the professionalism and courage of law enforcement. Its sad that it takes a tragedy to show people the incredible things the thin line does for us every day. The big events are the only ones on the national news. The rest of the time the heroics are smaller but just as real.

During my years doing IT in public sector I spent quite a lot of time supporting law enforcement. There were times when I'd be working on equipment in a 911 dispatch center quickly backing away from the workstation as calls came in so that the dispatcher could take them. The calls were only audible to the dispatcher, of course, but the radio traffic was there for everyone to hear.

The longest 10 minutes you'll ever experience is when a call for assistance is received and emergency personnel are in route.

While you sit there knowing what's happening to the person calling for help and you're waiting for help to arrive you're stuck sitting there. Helpless.

Emergency personnel do the job because of a desire to help people. And they do that, day after day, despite the horrible things they see and deal with. It was an eye-opening experience to learn about the depth and breadth of what law enforcement and emergency personnel deal with on a daily basis that we, as citizens, are shielded from. Seeing the videos and pictures, and hearing the stories that don't get told outside of the office is frightening.

All of it is handled with professionalism and courage, and the vast majority of the time with compassion and understanding. Every single day.

Taking the time

The other day I had a conversation with a co-worker ("user" in IT speak). We started talking during my lunch break but I hadn't looked at the clock. After a while I started to get a little anxious about the time...was my break almost over? After a while, I decided that it didn't matter and committed myself to the conversation. Lunch went a half-hour over but I relaxed and was a much better participant in the conversation.

When I was in college I took a cultural anthropology class because, hey, a guy's gotta fill requirements. Right?

The professor pointed out the different ways that cultures think about time. We western folks are very linear and regimented. Work starts at 0800, lunch is at 1200, day ends at 1700, and so on. Some other cultures take a more circuitous view where time is more relative. 

The instructor's example was like this: Someone from a western culture might meet a friend while walking to work, stop to talk but cut the conversation off after looking at their watch and finding they're not on schedule to arrive on time. Other cultures would find that offensive and would take as much time as necessary to finish the conversation before continuing on their way. Time isn't as important as the person.

We often don't value people as much as we ought to. Relationships are important. Being a part of the organization is important. We use metrics and frameworks to make workers and processes as efficient as possible but people aren't machines, yet. Yes, timeliness is still important and we can't simply change ourselves and our workplaces into an anything-goes sort of schedule. As with most things in life, there's certainly a middle ground we can achieve.

More on IT Partnerships

I had a wonderful conversation this morning that finally made me understand my own misgivings about managed service providers (MSPs) and IT vendors. It was actually a conversation about our own business and relationship with our customers (actually members, we're a cooperative). I think I finally understand why I don't trust vendors and why thinking of a vendor as a "partner" is nearly impossible to me.

The problem sums up like this: An MSP/vendor in the IT space assumes a lot of our money but does not assume any of our business risk.

In the case of a financial institution doing business lending you have a lot of risk. When you agree to loan money you become inextricably entangled and self-interested in ensuring the success of the borrower. There is an inherent vested interest in ensuring the growth and continued success of the recipient of the service. You make sure they make the right decisions. You check up on them on a regular basis. You question them when decisions are made that jeopardize their success and your investment.

In the case of an MSP, or any other vendor looking to establish a "partnership" with IT there typically isn't that vested interest. Yes, you can write fancy contracts (if you have the legal team and staff to do so) or you can make the argument that the prospect of continued and expanded business is interest enough. I don't have the resources to demand such a contract and I don't believe that vendors are as driven by continued/expanded business as many people think. This is especially true in the SMB market where contracts are rather small and there's an awful lot of potential customers. Growth is ensured to any vendor as  long as they just keep signing on new clients.

So what does a partnership mean to me now? It is a relationship where an MSP/vendor provides me services I need AND assumes enough of my risk to create a vested interest in my success and growth.

I've had close relationships with some vendors and have grown to trust some of their employees. In the case where the employees were also personal friends I've been very comfortable trusting the organization. This is still not a vested interest. Am I going to disown a friend over their employer's bad behavior? Probably not. Trust is not enough. There needs to be a pain point to make sure that everyone is focused on the same goal: success.

Remember - no one is as interested in the success of your business as you are (well...besides your lending institution that is).

The Good Service Template

Funny thing about service - good service generally looks the same no matter where you approach it from. IT is not the first sector to approach the challenge of providing good service! I've had the disadvantage (or some might claim advantage) of learning the art of IT service through trial and error and the school of hard knocks.

I've lived through lost data, angry customers, irate phone calls, ten 1hr onsite service calls in one 8hr day, unreasonable expectations, small business, medium business, enterprise. Service ain't easy, and it's even harder under pressure. Everywhere I've been, and in every job I've done its been remarkable how good service never changes.

Here's the template:

• Make yourself available to provide service
• Interact plesantly with the service consumer
• Listen to the consumer
• Demonstrate that you've been listening
• Paraphrase or restate the problem to verify understanding
• Create plan for providing service or resolving problem
• Communicate plan to consumer
• Follow through with plan
• Communicate with consumer upon plan completion
• Follow up with consumer later

Managing service is an even larger thing, of course. You need frameworks and procedures to be able to pull the disparate disciplines of IT together into a manageable package. When you have MBAs with no IT experience and IT types with no management experience trying to heard the cats a framework helps keep everyone sane and on the same page.

I love to talk about service in IT. It's a fun topic because in our young industry is continually waking up to the fact that there are other people around us and that we are a part of a larger organization called a "business" or a "corporation".

Talking about service also helps IT/IS remember that we are dealing with PEOPLE not gadgets. People are the most important part and if you're going to be working in a SERVICE sector job you have to live that mentality.

Springtime In The Computer Room

All those expensive servers and switches in that computer room, remember those? We all know that we only go in there to push power buttons from time to time. Maybe run a new patch cable. Have you gone in to check the temperature lately?

In the northern parts of the US, at least, warm weather is still on its way so its not too late to check your A/C. If you're at a new job and you haven't learned where the filters are and who typically does maintenance now's the time to figure that out.

I was putting a filter in a pass-through grate the other day and while I was on a ladder I noticed that the head unit on one of our A/C systems had a door. I opened it up and found this:

Those filters don't look so good. Let's take a vacuum to that.

Ugh. That's just terrible.

Much better! That would explain why the airflow was so rotten. I'll be adding that to my quarterly checklist.

So remember kids, buildings are filthy. The world doesn't like your technology and is actively working to destroy it. Be proactive and practice good computer room hygiene.

How to Cable an EMC VNXe for Failover

I really struggled with cabling a new VNXe for failover with vSphere. It's not difficult. There's just not a lot of documentation or explanation out there for it. The best practices docs and howtos treat iSCSI like a 3rd class citizen and assume you'd really rather be using Fiber or NFS. So here's how you really do it and a little explanation about why. We'll go into the details of the vSphere configuration after covering the hardware end of things.

*Disclaimer: I am not an EMC expert, nor am I guaranteeing nothing horrible will happen to your equipment or vSphere environment. Use this guide at your own risk.

At its very core, this is how your network will look with a VNXe for fail over. This assumes you have 2 physical NICs in the host to provide multiple physical hardware paths between the host(s) and the network switches. All cabling is single lines with no trunking.

Note that the VNXe has a single network line from each SP to each switch. This is the most simple way to cable. The SPs are able to handle NIC teaming using LACP trunks but we're going to use vSphere's round robin scheme to get us the same functionality. The requirement of LACP trunks means that you need to rely on a higher end switch, such as a Cisco Catalyst 3560 or an HP Procurve 2910al. Using vSphere's built in capabilities you can use a less switch but be careful about your throughput and switching capacities! You can get burned skimping on switches.

I chose to have each NIC on the SPs set to a different IP address. In this case:

• SP A eth2 = 192.168.50.10
• SP A eth3 = 192.268.51.10 
• SP B eth 2 = 192.168.50.20
• SP B eth 3 = 192.268.51.20

Be sure that when you cable the SP NICs that you cable the correct subnets together! We'll end up with a NIC in the 192.168.50.xxx subnet and another NIC in the 192.168.51.xxx subnet on each host. Make sure the 50.xxx NICs are on the same switch and the 51.xxxx NICs are on the other switch.

Moving on to the vSphere end of things. There is an iSCSI requirement that you ensure VMkernels are tied to a single specific port. You can assign multiple ports to a vSwitch but you need to ensure that each VMkernel has the switch failover order overridden to assign a specific physical port to the kernel. This can get confusing and since I don't have a whole lot of vSwitches in my environment I choose to create VMkernels one per vSwitch. VMware advises that the number of vSwitches can impact performance so be aware of how that may impact your environment.

Now that we've created 2 vSwitches, each with their own iSCSI VMkernel we can assign them to the iSCSI software initiator adapter. Configure the Storage Adapters on your ESXi host and view the properties for the iSCSI software adapter. Under the Network Configuration you will have the option to add ports to the initiator. Add both of the iSCSI VMkernels created earlier.

Once you've configured either Dynamic or Static discovery for the your LUNs and rescanned the adapter for devices we can go to configure storage.

Add your storage as you normally would and go to the new datastore's properties. Click on the Manage Paths button in the lower right.

If everything is cabled correctly and operating like it should you should see two paths listed and a Path Selection policy set to Fixed. To set your paths to Active/Active you'll need to choose Round Robin. If you want Active/Standby, choose Fixed. Make sure you hit the Change button before closing the window to make sure your Path Selection policy is saved.

Now that multipath is setup you can move on to testing. You're doing this in a non-production environment, right? Remember that if you pull cables while testing and something isn't right you'll lose Datastore connectivity!.

Go ahead and start pulling cables off the switch to see how the paths are affected. You should see that when you pull one cable from the primary SP for a LUN that the datastore stays online. If you pull the corresponding cable from the other SP you should see a dead link show up in the manage paths interface for storage configuration. Pull the second cable from the primary SP and you should still have connectivity.

Its worth mentioning that some folks online have said it can take up to 60-90 seconds for an SP to fail over to the backup SP. Make sure you take that into account while testing. I seem to have nearly immediate fail over. Your mileage may vary.

Explanation:

The VNXe has a nifty design. There's internal communication links between the SPs. When they detect over the SAN network that an SP NIC is down the network traffic for that particular port is redirected to the corresponding port on the other SP. For instance, If Eth 2 on SP A loses connectivity, the traffic is redirected to SP B Eth 2. Same goes for the other ports. This is why EMC tells you to cable each SP identically. 

The only time you have a true failover scenario is when an SP actually goes down, say for a reboot after an update for instance. Any NIC failures are handled by redirecting traffic.

Because of the massive amounts of traffic generated, you really must NOT do SAN over a network with other general traffic. Not only is that terrible design in a fairly epic fashion, you run the risk of cross SP traffic flooding your network. I had that happen once when I cabled to a port that I forgot to put in the proper VLAN. Not fun.

The Slow IT Movement

If you haven't read about +Rob England 's SlowIT philosophy, I highly recommend that you go over to his blog and read up. He's really on to something here.

When I first read his ideas about slowing down, I completely agreed but didn't fully understand exactly why. His latest post on "The mad competitive scramble" cemented it for me. Its a cultural problem and it is poisonous. I look at it from a purely practitioner's perspective and often the mad dash to the next project is intoxicating or is forced on the IT staff. Since I don't have the 10000' view he has (nor the experience) I didn't fully appreciate how toxic the culture that brings about the scramble he's talking about.

The IT department is just like any other department. It's a tool for the business just like the accounting, HR, or any other departments. The purpose is to get the work done and maintain the viability of the organization.

I've been on the insider and consultant side of things and I cannot count the number of times I've had to reign in users and decision makers. Slow down. Let's look at the value/use cases/purpose/master plan/etc. Decision makers are often very good at seeing the big picture of an organization but seem to be inherently terrible at seeing the big picture of technology. To me it seems that the older folks in those positions are still mystified by tech and the younger decision makers learned all the wrong theories in school about tech. Understanding how tech fits into business isn't rocket science. It just takes a level headed approach and some basic knowledge about how it works.

Be competitive on your own terms (and speed). Learn to be good at customer service. Stop trying to change to compete.

P2V Server 2008r2 VMware Unable to Update boot.ini

There's not much documentation out there on this so I figured I better do my part and blog the solution so you don't have to pull your hair out like I did.

If you are trying to P2V or V2V Server 2008r2 with VMware vCenter Converter Standalone and you get this dreaded error don't worry. Turns out its a simple fix.

Symptoms:

• P2V/V2V conversion fails at 95-98%
• New VM in vCenter is not bootable
• Booting VM to install media to do a repair yields no visibly installed OS
• Errors include: Warning: Unable to update boot.ini, Error: An error occurred during reconfiguration
• Convert logs show: Unexpected Exception: converter.fault.ReconfigurationNoSystemVolumeFault

The problem, it turns out, is that the system reserved partition is not online. (VM KB2008012)) 

The fix is to online the System Reserved partition:

At the command prompt for the machine you are trying to convert

c:\>Diskpart

c:\Diskpart> List volume

c:\Diskpart> Select volume 1

Make sure the volume you select is the 100MB system partition. Dead give away is that it is offline.

c:\Diskpart> Online volume

c:\Diskpart> exit

This should bring the System Reserved volume online and now the conversion should work properly.

As a side note, after the conversion you may want to double check that the System Reserved partition is marked to be online at boot (See this Symantec article). You can do this by running this command before you exit diskpart in the above instructions: c:\Diskpart> automount enable

Trust But Verify: It Goes Deeper Than That

Managed Services Part 2

Read Part 1 Here

I continue to struggle with how an MSP fits into the broader strategy of a competent IT department. I've come to the conclusion that many of my issues are trust related. As I think through the list of service providers I've dealt with over the years I can only name three or four that I really trusted. All the others seemed to go to great lengths to destroy any vestige of trust and some even seemed to revel in the fact they were fleecing the customer.

Why does this happen? What is going on that MSPs and service providers in general feel no shame in providing terrible service? I mean, service is in the name of their industry category, right?

There are a few things that jump out at me.

1. All the businesses I've spent my IT career in are located in an area identified as "Rural". We're a solid 2 hours from the Twin Cities metro area. 
2. Contracts seem to universally favor the service provider
3. Many service providers are bigger than the companies they "serve"
4. Service is hard and service providers are often bare-bones staffed to maximize profits

It's hard to be a service provider. I've been in that boat and I know how much work it is to keep it above water. It doesn't need to be this way though. Good service is hard but it's not difficult to maintain a high level of service if you make it part of your culture.

As I lamented the "just trust us" mentality of MSPs this morning, a friend reminded me of the "Trust but Verify" edict. I think it goes much deeper than that (and so does he). That's really where I start to struggle.

You can monitor and manage an MSP or service provider just like you would an employee or even more than that. You can subject them to strict monitoring and compliance auditing. The fact remains that the worst that happens to a service provider is that you sever your relationship with them. They lose some money. If you have enough clout you might even be able to make sure they lose some reputation. Not much of a consequence for most providers, eh?

They have dedicated sales staff that are professional job seekers. Service providers interview very well. When you are courting a provider you'll never find them providing you references from companies that were unhappy with them. Due diligence is not an easy thing when it comes to service providers or MSPs. Sometimes there isn't much choice available (see #1 in the above list).

The relationship is skewed and very profitable for one party. I don't want to take lightly the qualities an MSP brings to the table. Expertise and knowledge are expensive and just as hard to find as good service is. For many, it's far cheaper and feasible to bring in an MSP than hire staff. It's unimaginable to expect a small business to hire an IT staffer when they only have 10 people on the payroll and technology is not their core business. MSPs fill an important role to be sure!

So I continue to have trust issues with MSPs and service providers. Accountability for the relationship would help a lot with my issues. I look forward to the conversations I'll be having over the next few weeks. Maybe I'll learn to trust again?

What is a Partnership?

A vendor came a-courtin' today. I'm probably too critical on vendors when they approach me. It is so frustrating, as someone who's been in the business for a while and qualified to do my job, to see the attitude I'm treated with as a person involved in decision making. All too often potential customers are assumed to be stupid, or at best, ill informed. As a result my defenses go up and I become a top-notch skeptic.

These guys stuck with me through it, though, and in the end a good conversation happened. Hats off to them for being persistent.

This conversation made me think a lot about what a partnership with a vendor looks like. When it comes to managed services I tend to be openly hostile. I've seen very few good experiences with MSPs. No one cares as much about your organization as you do and MSPs, in the end, are making money off your lack of investment in IT (or inability to make an investment). However, there times where its nice to have a vendor to dump some things off on. There is also a certain level of expertise and exposure a vendor has just by being in many environments.

So in the case of an organization where there are already competent IT personnel involved and technology is helping users meet or exceed corporate goals is there room for partnership with an MSP? It was interesting to think about that as the vendor I was talking with was trying to figure out what this sort of relationship would look like. I'm working on bringing nearly everything I can in-house and strategies to cut down the "IT stuff" (defined as tech busywork like password resets, account unlocks, stability-related outages, etc). That doesn't leave much room for the traditional MSP sweet spots of helpdesk and network administration.

So now I have some things to think about. I'd never considered how a vendor fits into my IT strategies. I've never found a vendor that had a broad enough expertise for me to "partner" with. That sounds like a blog post for another day.

Knowing is Half the Battle

A few months ago my family and I went to a friend's house for supper. It was a great chance to visit and see where some of my eggs and produce come from. Our friends run a sustainable ag produce farm and I've been very pleased to have purchased CSA shares from them. Some of the heirloom produce varieties are just so tasty its hard to pass up the opportunity!

Our friends also keep chickens and other fowl. I've really enjoyed the eggs we get from them from time to time. Anyone that's been around a farm where you have free-range birds knows that you are a prime target for predators. Southern Minnesota has seen a pretty healthy increase in the number of coyotes and other wildlife that loves chicken and turkey as much as we do. The reality is that you have to protect your animals.

Now I'm not going to argue rights to this or that, or controlling who has what, constitution, or whatever. That issue has nearly become a dead horse and everyone is holding a stick with which to beat it. What did come out of this visit with friends as a reminder that knowing how to use tools, regardless of their lethal capabilities, is sometimes not only helpful but also life saving.

We hadn't been in the house for more than 30 minutes when my eagle-eyed spouse noticed a rifle propped up in the corner of one room. Our friends don't have children so this wasn't a huge surprise to me; What surprised me was I was the only one in the group there who knew how to handle the weapon. First step when finding a firearm is finding if it is loaded, and this one was. 

I unloaded the gun, left the chamber open and handed it over to my friend who put it in a safer location. 

This seemed like a pretty scary incident to the folks that were there. I've been around and used firearms since I was young so it didn't phase me but everyone's reactions to the situation made an impression on me. It occurred to me how important training is.

Firearms are a part of our culture (for better or worse) and we all have a chance of coming in contact with a weapon at some point in our lives, especially when you live in a more rural area. There are so many deaths and injuries that could be prevented through simple education. My kids will certainly spend some time in a firearm safety class when they're old enough. We'll probably even head out to the farm some day and shoot at pop cans together. I would be terribly remiss as a parent if my children were in my situation at a friend's house and didn't know how to make a weapon safe.

'Round the World

I think I've had more international communication in the last 2-3 weeks than I've had in my entire lifetime. Between Twitter, G+, email, and even phone it's been a lot of fun figuring out time zones and marveling how technology allows us to do this. There are so many incredible people all over who make me think. I love it!

Thanks to everyone who's put up with me. I really enjoy your thoughtfulness and broad minds! I hope I'm returning the favor with good content and ideas.

Lost in Translation

The longer I continue in my career many things keep getting easier. Experience fills in the knowledge gaps. There's one thing that remains frustratingly difficult though: dealing with non-technical people. Often, when I've sat down with other tech workers and healthy doses of beer, the conversation seems to inevitably turn to non-technical users.

"This job wouldn't be so bad if it weren't for the users."

Experience naturally teaches us how to interact with non-technical users. You step on a lot of toes and hurt some feelings along the way but often the frustration remains even when you can gracefully talk a user back from the ledge and solve their problem. (True story: I've walked into an office once to find a user with gun drawn on his computer. Not even joking.) As technicians we can often look past the technophobia, the lack of understanding, even the anger that users are experiencing. The ignorance and the willful disinterest are much harder to accommodate.

There seems to be an inherent disconnect between non-technical users and IT staff. I'm convinced that much of it has to do with the way our minds work but also in the way we approach technology, our methods for dealing with new information and tools.

I think the best thing we can do to help bridge the gap between user and technical staff is eliminate the silo-ing of employees to some extent. We keep the IT folks in the basement - out of sight, separated off from the good-looking people in sales, no windows. Why do we do this? For one thing, IT staff embrace the separation. We're not usually cuddly or necessarily good with people. The separation is also a symptom of the way business tends to look at technology and IT staff; The viewpoint is that IT is not part of the business but rather a facility we just toss money at.

As an industry, IT needs to get people skills. The helpdesk needs to be social not just in the Social Media sense but in the squishy human contact sense. Bad people skills should not be tolerated any more. Its driving people to use Shadow IT solutions and doing bad things to business.

IT is people.

Change From Within

What does change from within look like? It seems to me that many organizations never experience that type of change. That seems to indicate that IT maybe isn't that influential or transformative to most organizations. Maybe our influence has waned after the initial shock of introducing technology into the business world. Perhaps we've assimilated too well for our own careers.

Whatever the reason for the lack of IT driven change I'm excited for it myself. The opportunities available for IT to transform the way we do business are so huge I can't wait to start!

So what does change from within driven by IT look like? Positive change, that is. We've seen and produced plenty of negative change.

Once you've solved the big problems in the server room the next step is to look at what's going on outside of IT. That's where technologists seem to fall on their face. Once you need to look outside of the IT Department we seem to get all awkward and nervous. IT is a unique department in that it is involved in every aspect of a business from building maintenance, to sales, to boardroom, etc. We have a broad viewpoint of the whole organization and it is time to utilize that knowledge and experience to help the business.

New Tricks For Today's Musician

I was practicing mandolin tonight and after going through a few pages in the lesson book I kept running into duets. I've been skipping them so far. I'm all by myself and don't know any other players right now (though there's some really good ones around New Ulm). What good is it to play a duet by yourself?

Then came the facepalm moment: I have technology! The music app I use on the iPad for all my sheet music has a simple recorder on it. So I figured I'd give it a try. Sure thing, works well enough!

I can't believe the tools we have at our disposal today. Recording used to be expensive and difficult. Now every device has a high quality microphone and digital recording capabilities that only used to be available to professionals.

So now I'm not going to skip the duets. I'll be accompanying myself.