Sometimes the Command Line is Faster
Not long ago I had a 4GB file that contained about 8 million lines and 1.6 million of those lines were of interest to me. Fortunately the file was on a Linux system so a quick grep command ("grep ^VI filename > out.txt") gave me what I needed. The grep command looked for any line that began with "VI" and copied that line to a file named "out.txt" and I instantly reduced a 4GB file to an 100MB file with 1.6 million lines. But what if I had to do this on a Windows machine? Easy! In fact, several possible solutions exist.
The Windows command line has both find and findstr, both of which perform a function similar to that of grep. These commands produce exactly the same output. The output is a list of lines in the file "test.txt" that begin with the characters "VI".
- Windows CMD: C:\Users\Bill>findstr ^VI test.txt
- BASH on a Linux machine: $ grep ^VI test.txt
But the Unix or Linux shell (BASH) has numerous commands that are missing in Windows. In many cases the Windows Power Shell has commands similar to those offered by BASH. Which solution you choose may depend on what you're most familiar with.
A Little Background
The first personal computers had only command lines. Graphical user interfaces were being investigated in the 1970s at the Palo Alto Research Center and it was PARC (not Apple) that introduced the first GUI-based operating system. The first commercial GUI came from Apple. The third, also copied from work at PARC, was Windows. Another graphical user interface (Graphical Environment Manager, or GEM) was developed by a former PARC employee and used by Ventura Publisher.
With the advent of graphical interfaces, many thought that the command line was dead but some tasks are far easier to perform from the command line. The Mac has one. Most Windows computers have two. With Unix and Linux computers, you have a choice of several. BASH is probably the most common.
Windows users know about COMMAND.COM from the old days through Windows 98. From Windows NT until now the command-line interface (or shell) has been CMD.EXE.
The second shell available to Windows user is POWERSHELL.EXE
COMMAND and CMD have never been fully scriptable. Batch files allowed some programming but Microsoft tried to improve things with the Windows 98 introduction of CSCRIPT.EXE, which made it possible for programmers to use JScript and VBScript that could interact with some applications. Documentation was bad and security was weak.
Work began on what would become Windows PowerShell in 2002 but it wasn't until 2006 that Microsoft had a release candidate. Initially an add-on product, Windows PowerShell is now incorporated into all versions of Windows.
Windows 7 users can press the Windows key, type "powershell", and press Enter to open it. And it's on the Start Menu, of course.
PowerShell version 2 was released in 2009 and, on 64-bit systems, is available in both 32- and 64-bit versions.
Multiple Solutions
That's PowerShell at the top, Cygwin in the middle, and CMD on the bottom.
Those who are most familiar with Windows may prefer to use the PowerShell or, for simple tasks, CMD. If you need a command that isn't available in one of the Windows shells, you might prefer to use the BASH commands. So you could set up your computer to dual boot Windows and Linux and switch between systems when you need to use BASH. Or you could install CygWin, stay in Windows, and have access to BASH commands.
If you set up your computer to boot Windows and Linux, you can run native Linux applications when you boot to Linux. You might think that installing CygWin would allow you to run Linux applications under Windows. This is not the case. It is also not, as the CygWin site says, "a way to magically make native Windows apps aware of Unix functionality."
But if you're familiar with Unix/Linux commands under BASH, you'll find that CygWin creates a Unix-like environment and command-line interface that runs under Microsoft Windows.
Cygwin was developed by Cygnus Solutions. When Red Hat acquired Cygnus, CygWin became free, open-source software under the GNU General Public License version 2. It is maintained by Red Hat, NetApp, and volunteers.
Users may start Windows applications from Cygwin in addition to using the commands that are familiar to Unix users.
If you download the entire CygWin environment, you'll find that it is huge: 1.6 GB. This is because it contains just about every package known to the Linux community. For example, you can install any (or all) of the following text editors: TeXmatics, bvi, ed, emacs, emacs-x11, emacs-el, gvim, hexedit, joe, lyx, mc, mined, nano, nedit, ted, vim, xedit, and xemacs.
The same is true for all of the other package types (audio, database, games, math, and Perl -- just to name a few.)
So the prirmary advantage CygWin brings is the ability for someone who is familiar with Unix/Linux commands to use those command on a Windows machine.
If you'd like to give CygWin a try, just follow the link.
But if you'd like to stick with a Microsoft solution, consider this as an example of what you can do:
I've mentioned using Sam Spade to investigate what's on a website without loading it into a browser. The trouble with Sam Spade, though, is that it hasn't been updated since 1999 and the developer has no plans to update it.
PowerShell makes it possible for you to examine a site without exposing your computer to any danger. Here are the commands to issue:
- $webClient = New-Object System.Net.WebClient
This command instantiates a new WebClient object. - $content = $webClient.DownloadString("http://www.blinn.com/index.html")
This command loads the contents of index.html from www.blinn.com. It should be clear how to load any given Web address and file. Just replace "www.blinn.com" with any server and domain name and replace "index.html" with the name of the file you want to review. - $content.length
This will return the length of the file. In this case, it's 12182 bytes. - $content.Substring(0,12182)
This command diplays the first 100,000 characters from the website. Nothing is interpreted. No scripts are run. You get to see exactly what the website would send to your browser. You can, of course specify any number of characters but if you choose a number that's larger than the number of bytes in the file, you'll get an error message.
The key here is that PowerShell is a powerfull shell. Besides being capable of executing all the command.com and cmd.exe commands you're familiar with, it exposes a vast array of Net Framework objects.
A Remote Access Tool By Any Other Name ...
We seem to be surprised, and possibly even enraged, when we learn that other nations are carrying out espionage. It's as if we reason that espionage is OK if we do it but not OK if somebody else does it. That's not to say that the "Shady RAT" exposed recently by McAfee Software doesn't concern me. It does. What it doesn't do is surprise me or enrage me.
The rest of today's program, with the exception of Short Circuits, will dwell on this topic.
Let's start with a couple of definitions. "RAT" is an acronym for "remote access tool" so that's where "RAT" comes from in the caper that McAfee refers to as "Shady RAT". And the "bad guys" in this operation are presumably in China although McAfee's Dmitri Alperovitch, the company's vice president of threat research, was very carful not to say either "China" or "Chinese". Instead, Alperovitch referred to a "state actor".
But that brought an immediate reaction from the Chinese government—the people not accused of conducting the spying—by way of the People's Daily, the newspaper operated by the Chinese Communist Party. Without quoting any government or party officials, the newspaper simply said "Linking China to Internet hacking attacks is irresponsible." Apparently this was in response to media comments that suggested, as I have, that China is one of the more likely and more capable suspects.
One of the more interesting aspects of this case is how long the spying has been going on: nearly two and a half years at an Asian Olympic committee and a South Korean government agency; more than two years at a US government agency, a California county government, and a US satellite communications company; and more than a year at organizations ranging from a US news organization's Hong Kong bureau, several defense contractors, a US real estate firm, and a construction company.
(See chart at the bottom of this article for details.)
Additionally, the types of organizations being spied on is fascinating. Quirky even. Defense contractors, US federal agencies, and the like make sense. So do think tanks and manufacturers, particularly if the intent is commercial espionage. But why a county government in California? Why a real estate firm? Why a German accounting firm? Why Denmark's satellite communications agency?
The Threat Is Real But Not New
McAfee's Dmiti Alperovitch says "For the last few years, especially since the public revelation of Operation Aurora, the targeted successful intrusion into Google and two dozen other companies, I have often been asked by our worldwide customers if they should worry about such sophisticated penetrations themselves or if that is a concern only for government agencies, defense contractors, and perhaps Google. My answer in almost all cases has been unequivocal: absolutely."
Some of the recent high-profile intrusions have made the problem a top-of-mind issue for some people but Alperovich says "these types of exploitations have occurred relentlessly for at least a half decade, and the majority of the recent disclosures in the last six months have, in fact, been a result of relatively unsophisticated and opportunistic exploitations for the sake of notoriety by loosely organized political hacktivist groups such as Anonymous and Lulzsec."
Spying on the Spies
It's ironc and amusing that McAfee learned so much about where the spies have been by gaining access to the intruder's command and control server. Says Alperovitch, "We have collected logs that reveal the full extent of the victim population since mid-2006 when the log collection began. Note that the actual intrusion activity may have begun well before that time but that is the earliest evidence we have for the start of the compromises. The compromises themselves were standard procedure for these types of targeted intrusions: a spear-phishing email containing an exploit is sent to an individual with the right level of access at the company, and the exploit when opened on an unpatched system will trigger a download of the implant malware. That malware will execute and initiate a backdoor communication channel to the Command & Control web server and interpret the instructions encoded in the hidden comments embedded in the webpage code. This will be quickly followed by live intruders jumping on to the infected machine and proceeding to quickly escalate privileges and move laterally within the organization to establish new persistent footholds via additional compromised machines running implant malware, as well as targeting for quick exfiltration the key data they came for."
McAfee identified 72 compromised parties. Alperovitch says that many more were present in the logs but there was insufficient information to accurately identify them. The organizations that were infiltrated are in 14 "geographic locations" according to McAfee.
The Victims:
The Geography:
One point that Dmitri Alperovich make crystal clear is the size of the threat: "I divide the entire set of Fortune Global 2000 firms into two categories: those that know they’ve been compromised and those that don’t yet know."
But nobody should be surprised, shocked, or even outraged that these kinds of activities are occurring. Nor should anyone be at all surprised to learn that the United States enagages is some of the very same activities.
The "Honor" Roll
| Victim | Country | Start | Duration |
|---|---|---|---|
| Olympic Committee of Asian Country #1 | Unidentified | Jul-07 | 28 |
| South Korean Government Agency | South Korea | Aug-06 | 27 |
| U.S. State Government #3 | USA | Aug-07 | 25 |
| U.S. Satellite Communications Company | USA | Feb-09 | 25 |
| U.S. Southern California County Government | USA | Jun-07 | 24 |
| U.S. Defense Contractor #1 | USA | May-07 | 21 |
| U.S. News Organization, Hong Kong Bureau | Hong Kong | Aug-09 | 21 |
| U.S. Defense Contractor #2 | USA | May-07 | 20 |
| U.S. National Security Think Tank | USA | Feb-08 | 20 |
| United Nations | Switzerland | Sep-08 | 20 |
| U.S. Construction Company #1 | USA | May-08 | 19 |
| South Korean Construction Company | South Korea | Jul-06 | 17 |
| U.S. Real-Estate Firm #2 | USA | Apr-07 | 17 |
| U.S. Defense Contractor #7 | USA | Apr-08 | 16 |
| World Anti-Doping Agency | Canada | Aug-09 | 14 |
| U.S. Electronics Company | USA | Feb-08 | 13 |
| U.S. Think-Tank | USA | Apr-10 | 13 |
| U.S. International Trade Organization | USA | Sep-06 | 12 |
| U.K. Defense Contractor | UK | Jan-09 | 12 |
| South Korean Steel Company | South Korea | Jul-06 | 11 |
| U.S. Defense Contractor #10 | USA | Jun-09 | 11 |
| Association of Southeast Asian Nations | Indonesia | Oct-06 | 10 |
| German Accounting Firm | Germany | Sep-09 | 10 |
| U.S. Defense Contractor #5 | USA | Feb-08 | 9 |
| U.S. Real-Estate Firm #1 | USA | Nov-06 | 8 |
| U.S. Federal Government Agency #1 | USA | Jul-07 | 8 |
| Taiwanese Electronics Company | Taiwan | Sep-07 | 8 |
| U.S. Federal Government Agency #4 | USA | Sep-07 | 8 |
| Taiwan Government Agency | Taiwan | Apr-08 | 8 |
| U.S. National Security Non-Profit | USA | Jul-08 | 8 |
| U.S. News Organization, Headquarters | USA | Aug-09 | 8 |
| U.S. Northern California County Government | USA | Jun-07 | 7 |
| U.S. Federal Government Agency #2 | USA | Aug-07 | 7 |
| Olympic Committee of Western Country | Unidentified | Aug-07 | 7 |
| Olympic Committee of Asian Country #2 | Unidentified | Sep-07 | 7 |
| U.S. Defense Contractor #3 | USA | Nov-07 | 7 |
| U.S. Defense Contractor #4 | USA | Dec-07 | 7 |
| U.S. Information Technology Company | USA | Apr-08 | 7 |
| U.S. Natural Gas Wholesale Company | USA | Mar-09 | 7 |
| U.S. Communications Technology Company | USA | May-09 | 7 |
| U.S. State Government #1 | USA | Jul-07 | 6 |
| UK Computer Security Company | UK | Feb-08 | 6 |
| U.S. Information Services Company | USA | May-08 | 6 |
| Denmark Satellite Communications Company | Denmark | Aug-08 | 6 |
| Canadian Government Agency #1 | Canada | Oct-09 | 6 |
| U.S. Federal Government Agency #3 | USA | Sep-07 | 4 |
| Non-profit Democracy-promoting Organization | Unidentified | Sep-07 | 4 |
| Canadian Information Technology Company | Canada | Jul-08 | 4 |
| Singapore Electronics Company | Singapore | Nov-08 | 4 |
| U.S. Construction Company #2 | USA | May-09 | 4 |
| U.S. Defense Contractor #8 | USA | May-09 | 4 |
| U.S. Solar Power Energy Company | USA | Sep-09 | 4 |
| Department of Energy Research Laboratory | USA | Jul-06 | 3 |
| U.S. Network Security Company | USA | Dec-07 | 3 |
| U.S. State Government #5 | USA | Apr-09 | 3 |
| U.S. Agricultural Trade Organization | USA | May-09 | 3 |
| U.S. Defense Contractor #9 | USA | May-09 | 3 |
| U.S. Insurance Association | USA | Aug-09 | 3 |
| U.S. Defense Contractor #6 | USA | Feb-08 | 2 |
| U.S. State Government #4 | USA | Apr-08 | 2 |
| U.S. Government Organization #5 | USA | Nov-09 | 2 |
| U.S. Defense Contractor #11 | USA | Dec-09 | 2 |
| Indian Government Agency | India | Sep-10 | 2 |
| Trade Organization | Unidentified | Jul-06 | 1 |
| Vietnam’s Government-owned Technology Company | Vietnam | Mar-07 | 1 |
| U.S. State Government #2 | USA | Aug-07 | 1 |
| International Olympic Committee | Switzerland | Nov-07 | 1 |
| U.S. Accounting Firm | USA | Jan-08 | 1 |
| U.S. Government Contractor #1 | USA | Apr-08 | 1 |
| U.S. Nevada County Government | USA | Apr-09 | 1 |
| U.S. Defense Contractor #12 | USA | Dec-09 | 1 |
| Canadian Government Agency #2 | Canada | Jan-10 | 1 |
Short Circuits
Pay-Per-View Porn Sales Drop
Is the US becoming more puritanical? Are we afraid big brother is watching? The Wall Street Journal reports that DirecTV subscribers are buying a lot less pornography these days. The same is true for Time Warner's porn channels. Despite what we might say, think, or like to believe, pornography has been, was, is, and probably always will be big business. So what's up with the drop?
The Wall Street Journal tends to obfuscate a bit: "It is hard to get a fix on how much porn contributes to cable and satellite companies' bottom lines because the companies aren't transparent about it. But adult content has been a consistent source of profit, because cable operators have leverage to command margins that can exceed 90% on rentals of generally interchangeable porn movies, analysts say."
So where have all those dollars gone?
Apparently they've gone to the Internet. Or maybe they're staying in consumers' wallets. There is, after all, no shortage of opportunities to view free porn on the Internet.
Again quoting the Wall Street Journal: Adult TV's woes echo broader challenges that the television business is facing from Internet video. Distributors and TV networks, for instance, are talking about restricting next-day Web streaming of TV shows, which some executives believe eats into ratings and makes it easier for television viewers to become "cord-cutters."
If you're wondering how big a deal this is, adult video-on-demand (aka "porn") brought in nearly $1 billion in revenue for cable companies in 2010 and that was down a bit. Now it's down a lot more but not because we're watching it less.
The newspaper says "Playboy Enterprises has seen its domestic TV revenue—which includes video-on-demand versions of Playboy TV and Spice, as well as traditional adult channels—fall significantly in recent years, from $75.8 million in 2007 to $44.4 million in 2010, the company's last report before going private."
The Puzzling Case of the Transparent PDFs
If you're one of those people who live on the edge and have the latest beta version of Firefox 6, you may have found that PDF documents are unreadable because they're transparent. I suspect that part of the problem is the transparency function of Windows Vista and Windows 7 but most of the problem comes from an unexpeted source.
Because there wasn't a great hue and cry about unreadable PDFs under Firefox 6, I assumed that the problem was something specific to my computer. It wasn't a problem that I experienced at the office where I'm still stuck on Windows XP. And it wasn't a problem I've experienced on my notebook computer where I use the default theme.
I've just provided enough clues that you should now know what the problem is.
As it turns out, some themes have an adverse reaction with Firefox 6 beta under Windows 7. Other themes don't exhibit the problem.
What I see with the GlowyWine theme enabled.
Not very readable is it?
What I see with the Gato Cinza theme enabled.
So the bottom line is this: If you encounter a problem with a new version of your browser, first consider which add-ons might be causing the problem.
The Absurdity of 99.9% Uptime
A message from Microsoft promotes the new online Office 365 service and, at the bottom boasts of 99.9% uptime. Sounds impressive, doesn't it? But if your telephone service provider offered 99.9% uptime, you would be on the phone day and night with the public utilities commission.
A service with 99.9% uptime will be unavailable approximately 31,536 seconds per year.
Here's the math:
60 seconds per minute
times 60 minutes per hours (3600)
times 24 hours per day (86,400)
times 356 days per non-leap year (31,536,000)
times 1.00
- 0.999 [0.001] = 31,536.
Now divide by 60 to get the number of minutes (525.6)
and by 60 again to get the number of hours (8.76).
Missing about 9 hours of service might not sound so bad and it wouldn't be if the entire 9 hours happened overnight when you didn't need the service.
But the fallacy of 99.9% uptime is that the serivce could be unavailable for about one and a half minutes every day and still meet the service level promise. The service may well be reliable and possibly reliable enough for general use. But those all-too-common claims of 99.9% uptime are just silly.
The author's image: It's that photo over at the right. This explains why TechByter Worldwide was never on television, doesn't it?
