HYPERBOLE, EMBELLISHMENT, AND SYSTEMS ADMINISTRATION TRUE TALES, SNAFUS, NEWS AND URBAN LEGENDS

[11/6/2006]

Training Users and Backup
POST A COMMENT

Why is it that people only remember to backup data after they’ve lost it?

 

We’ve all had that conversation.

 

User:                “I’ve lost all my data because my hard disk blew up”

Admin:            “Did you do any backups?”

User:                “Uh …. No”

Admin:            “Then, unless you want to pay some serious money to get someone to retrieve the data from the hard disk that failed, you’ve lost it”

User:                “Why didn’t you tell me to back up my data!”

 

Given that people have been doing important work on their home and work computers for years, you’d think that performing backups would be as obvious to them as locking up the car when you leave it in the car park at the mall.  Many people’s computers now store not only their documents, but other important things like family movies and thousands of irreplaceable pictures. Do they perform backups?

 

Unless they’ve gone through the experience of losing all their data, the answer is generally no.

 

In general I suspect that this is because users view the integrity of their data as being the sole responsibility of the systems administrator. This is true when everyone stores their backups and emails on the server, but in many organizations, backups and emails are stored locally on computers and are only copied to servers if they are going to be shared with someone else.

 

All you can probably do is send out a monthly notification reminding people to copy up any important data on their computers to the server so that it can be backed up with the organization’s other data. That way, when they tell you that you didn’t tell them that they needed to back up their data, you can gently remind them that you tell them each month to back up their data and if they’ve lost something important, it is no fault but their own.


- posted by Orin Thomas

[8/20/2006]

Real Physical Security
POST A COMMENT

A consultant was called out to look at a company's problematic database server hardware.

In the middle of the night prior to the consultant's arrival, there was a power blackout that lasted for fifteen minutes. Switches, routers and servers were protected by UPS, but this power blackout had somehow caused the fingerprint reader attached to the server room door to freak out and stop working.

The consultant had made his appointment to meet with the company's systems administrator early in the morning. When the consultant met with the sysadmin, the contact explained that the fingerprint reader could be fixed by resetting using a special key that was kept in one of the company's safes. Unfortunately one of the two people who had the combination to the safe was away sick and the other was away until mid afternoon attending an important meeting with a client.

With no obvious way into the server room to examine the problematic database server hardware, the consultant asked if he should come back tomorrow when the server room door issue was resolved.

"Oh no, it won't be too much of a problem" the sysadmin told him, "as a safety feature, the door can be opened from the inside even when the lock isn't working"
The consultant thought about pointing out that they were on the outside, not the inside, but held his tounge.
The sysadmin took the consultant to the server room door and told him to wait. The sysadmin then walked down the hallway, opened a door, and pulled a stepladder out of the cupboard. He came back up the hallway, put the stepladder next to the server room door and climbed up to the roof. The sysadmin carefully shifted aside some the roof tiles. He then climbed further up into the space between the roof tiles and the actual roof and moved some more tiles around. The consultant watched him climb fully into the roof space and then a few moments later there was a loud thump. A moment after that the server room door opened, banging against the stepladder. The consultant removed it.

"Whenever we can't get in, we just get someone to jump through the roof. That's why we keep the stepladder in that closet!" 
- posted by Orin Thomas

[7/19/2006]

Bad Subnetting
POST A COMMENT

This week’s snafu comes from a colleague of mine in Australia. The organization that the systems administrator worked for was one of Australia’s largest companies and had been allocated exactly fifty blocks of contiguous class C IP address ranges in the early 1990’s. It was one of those situations common in back then where addresses in the public range were handed out for each computer installed on a company’s network. In reality the organization had needed about 3 public IP addresses, but that’s another story.

 

The networking for this organization had been outsourced to a global company whose name you would recognize. It was this company that handled the organization’s router infrastructure. My colleague was only responsible for managing the organization’s servers and workstations.

 

The problem with laziness only became apparent when it came time to renew the company’s domain registrations. The IP address of the registrar’s website just happened to be located in a class C range adjacent to the ranges allocated to the organization. For some reason it was simply impossible to bring up the registrar’s website, even though it was accessible when my colleague connected through his private ISP’s dialup network

 

It took my colleague some time to figure out why a tracert would only get to the perimeter router. As the organization was large and geographically dispersed, my colleague wasn’t sure himself what public IP address ranges had been assigned to his organization. He eventually found the original documentation and wrote up a list.

 

My colleague then called the organization that was responsible for managing the routers. He asked that the route table on the perimeter router be read back to him. This is where the problem lay. It turned out that more than a year before, a guy who had since left the outsourcing company, had set up the routing tables on the organization’s perimeter router. Rather than perform a complex subnetting calculation to precisely map out the 50 class C addresses, he configured the router in a blunt way, telling the perimeter router that the organization’s internal network had a range equivalent to a class B block. Rather than enter many entries in the router table, he decided he could get by with one. Although it was possible to use an elegant solution involving classless interdomain routing, the outsourcing company decided upon, the less elegant solution of entering each of the 50 internal networks manually into the routing table.


- posted by Orin Thomas

[6/14/2006]

Why you should lock off USB
POST A COMMENT

I found this story on the Dark Reading website. It deals with a brilliant social engineering attack by Steve Stasiukonis of Secure Network Technologies Inc. It provides a really good argument for why blocking the use of USB thumb drives.



In the linked story, a credit union hired Secure Networking Technologies (SNT) to assess the security of their network. SNT took a novel approach. They scattered a collection of vendor giveaway USB thumb drives around the credit union’s parking lot several hours before work began. Each USB drive was filled with random image files and a custom trojan. The trojan would collect passwords, logins and other information from a user’s computer and then email it back to SNT.



The experiment ran 3 days. Of 20 scattered USB drives, 15 were found by employees. Each found USB drive was plugged into at least one of the credit union’s computers. In each case where someone plugged the drive into their computer, they executed the Trojan. SNT knew this because they received the password and login information from the computers where USB drives were attached.



This method of gaining access to a network is certainly a lot simpler than sitting out in the parking lot with a wireless card attempting to crack WEP!

 


- posted by Orin Thomas

[6/8/2006]

The Thirteen Year Old Enterprise Administrator.
(4 Comments)
POST A COMMENT

Many sysadmins report directly to people who do not have a technical background. Reporting to someone without a technical background can be problematic. For example: As some of you may have found, some managers see the membership of security groups as a status symbol. They figure that if the sysadmin who reports to them is a member of an important sounding group (such as the Enterprise Admins group), that they as the sysadmin’s manager should also be a member of that group. Administrators with patience and excellent communications skills might be able to explain to the manager why they should not be a member of such a group. Unfortunately, some of us, when broaching the topic of security risks come up against the philosophy:

 

“I could never be a security risk, I know what I’m doing, so I don’t see why I shouldn’t be a member of this important group.”

 

As a favor to his wife, a manager who held to such a philosophy used to bring his 13-year-old son and 9-year-old daughter in to the office with him on Saturday mornings. The manager’s 9-year-old daughter was not a problem, she would curl up on the couch and read whilst her father worked. The 13-year-old son was a different kettle of fish.

 

It is common knowledge that during non-working hours many IT support teams like to play LAN games such as Valve’s Counter-Strike. This IT support crew was no different and manager allowed his team to play games during off hours as his way of encouraging his team to bond. The IT support team’s computers were a generation ahead of the dated computer the manager’s family had at home and ran Counter-Strike a whole lot better. Whilst the daughter read, the manager would log his son on using his own user account at one of the IT team’s workstations.

 

This was the same user account that the manager had demanded be added to the Enterprise Admins security group.

 

Although on most Saturday mornings this was enough to ensure that the manager could get on with whatever work he was spending his Saturday morning doing, on one particular Saturday morning the excrement encountered the rotary cooling device. Rather than play Counter-Strike, the 13-year-old son decided to explore the contents of the workstation’s Administrative Tools menu.

 

The problems became apparent the following Monday morning. To understand the nature of problems, you need to think like a bored 13-year-old boy. If, when you put yourself in that mindset, you consider excessively using expletives to be the epitome of wit, you can probably predict the sorts of things that happened. Problems included, but were not limited to:

 

• Creatively renaming user accounts so that they included expletives
• Resetting user account passwords to expletives
• Adding security groups with names that contained expletives
• Giving Organizational Units names that were more expletive oriented
• Adding descriptions to random user accounts that included expletives
• Renaming existing security group names so that they now included expletives.

[5/28/2006]

The website is down because someone removed the X-Box
(3 Comments)
POST A COMMENT

  Public Universities are always running on a limited budget which means they sometimes come up with “innovative” solutions. A couple of years back, an X-Box with some variant of Linux installed had been put in the server room to support a subject designed to teach computer-illiterate Philosophy students how to build their own web pages. This unorthodox platform was chosen because one of the techs was a Linux enthusiast and had convinced “the powers that be” that a cheap web server solution for this subject could be implemented using an X-Box rather than a standard PC or server. Grateful to save money where they could, the project was approved.

 

Several years later and most of the staff in the department had moved on to bigger and better things. There was a new manager and a brand new set of helpdesk techs. The department’s Unix administrator was one of the few people left who knew what the X-Box was used for. Each year before the second semester class began he powered up the X-Box and updated its software. Few of the other staff ventured into the server room so there was little reason to wonder why an X-Box was sitting on the rack.

 

Five weeks into the “Web pages for Philosophy students” class the excrement encountered the rotary cooling device. The IT department’s help desk started receiving calls from Philosophy students who were unable to access their web projects.

 

Flummoxed, the help desk staff escalated the job ticket to the Unix administrator. Unable to remotely access the X-Box, he trotted off to the server room. He was surprised to find that the X-Box was no longer present. He did a quick search of the room but failed to find it. As it did not look as though someone had broken in to the room, there had to be another explanation.

 

The administrator went down to the new manager’s office to report the missing X-Box. The new manager was quiet for a moment and then sheepishly informed the administrator that it was he who had removed the X-Box. The manager had thought the X-Box was just a games console that the IT departments staff used for recreation when it got quiet. Noticing that the X-Box hadn’t been moved from the server room for some time and that his son was going to be at home on school holidays for the next two weeks, the manager decided to take the X-Box home so that his son would have something to entertain himself with.  The manager then drove home and retrieved the X-Box. The administrator got a labeling machine and plastered the words “This is actually a server” all across the console.


- posted by Orin Thomas

[4/24/2006]

The Vacation Problem
POST A COMMENT

In the past I’ve worked with two different guys at two separate jobs who, being limited to dialup at home, used the office internet connection to download a significant amount of data from the Internet. In most cases the data that they were downloading was harmless (accessing their MSDN subscriptions and so on), but the amount of data that was the issue.

 

Both these guys were fairly careful to hide their data downloading habits. Although some of the data was work related, a lot of it was stuff like ISOs of betas that would never be used in either work environment. Both guys configured downloads to occur at off peak times when no-one was around. Both had also found clever ways around having their downloads being specifically tied to them, though at the time both organizations didn’t have sophisticated proxy logging software.

 

What got each of them found out was the significant drop in the internet bill to each organization that occurred when each went on vacation for a month. Although each had gone to great lengths to hide their activity during their time at work, it was their inactivity that gave them away. It was only after the internet bill decreased significantly and then returned to normal levels when the staff members returned did respective managers start to get curious about what was going on.

Neither guy got fired, but their managers were a lot more watchful in future!


- posted by Orin Thomas

[2/27/2006]

They posted *what* on the website?
POST A COMMENT

Back when spirits were brave, the stakes were high, men were real men, women were real women and small furry creatures from Alpha Centuari were real small furry creatures from Alpha Centauri [sorry Douglas] most people in the local University’s Faculty or Arts ran their web pages off a DEC Alpha server.

 

All was good until one day an evil MA student from Criminology wrote a personal web page. The evil MA student had decided to do his thesis on black hat hackers. What better way to collect data on the aforementioned black hats than to publish a page challenging them to deface it? Believing his plan mighty and cunning, he even visited the dark IRC channels of the deep internet to inform the denizens there of his research and the challenge he proposed.

 

The first that the systems administrator heard of it was a tide of hack attempts against the server. This went on for several hours before an email arrived in the root account mailbox explaining that some idiot on an IRC channel was challenging people to hack the server.

 


- posted by Orin Thomas

[1/15/2006]

The case of the unlocking door
POST A COMMENT

Tim Ward posted this story on Slashdot [http://www.slashdot.org] by Tim Ward and is reproduced here with his permission. If you have an amusing IT story, submit it to me at orin.thomas@gmail.com and I’ll post it here with attribution.

 

Tim worked at a company where the front door was controlled by a lock that opened when it detected the proximity of an authorized RFID tag. This allowed members of staff to enter and exit the premises without having to physically swipe themselves in.

 

A member of his staff resigned and left the company. Several days after the employee left the receptionist noted that the front door was continually unlocking itself. Investigating further it was determined that the resigned staff member’s RFID tag was responsible for the door continuously unlocking itself.

 

When he resigned, the member of staff had not been asked to return his RFID tag. Noting this oversight he had mailed it back to the company. Of course as no one remembered to ask the guy for the tag when he left, no one remembered to disable the tag in the system.

 

The mysterious unlocking was explained when the letter containing the RFID tag was found in the HR department’s pigeon-hole, located less than a meter from the organization’s front door.

 


- posted by Orin Thomas

[1]  2  3   next page