According to a recent survey of Independent Oracle Users Group (IOUG) members, adoption of RAC (Real Application Clusters) and other clustering products is widespread, but grid lags behind. As I reported in my recent story “Grid computing adoption slow amid fears of complexity,” the main reason for this is that users are concerned about the cost and difficulty involved in deploying grid. Are these fears founded?

According to our resident RAC and availability expert, Bill Cullen, these fears aren’t surprising, founded or not:

That doesn’t surprise me at all. Tech managers are always leery of wasting money on the five-dollar solution for the five-cent problem. I’m not saying that’s what Grid is, but that’s what decision makers are fearful of. There is definitely a sense of “If it isn’t broke, don’t fix it,” and in many cases I agree. Because there is such a low availability of grid database expertise, I think many managers are scared of the complexity, training costs, and “mucking up” of existing production applications that are working well.

RAC alleviated a lot of existing issues, especially in the arena of higher availability,
and it also was the second generation of a problematic product (Parallel Server) that many people were anxious to get off of. So in this sense it cured the headaches of a lot of managers.

By contrast Grid allows additional scalability at more attractive costs but unless you are faced with specific challenges the natural inclination is to leave well enough alone.

Isn’t [complexity] always the concern? I remember when managers worried about the complexity of the 7.3 database. I may not go as far as to say “unfounded” but it is certainly missing the forest for the trees because the reality is Grid reduces complexity, makes larger environments more manageable, and lowers cost in the long run.

Fess up — how many of you are putting off grid indefinitely because you fear the effort of implementation? Or are you just afraid of change in general?

If your company already has RAC in place, does it serve your purposes just fine? Are you seeing benefits? Do you have any plans to increase the number of nodes? Are plans to take the plunge into grid in the works? Let us know your position.

-Elisa

Database 11g will boast a host of new, innovative features – such as Database Replay and its new Real Application Testing-related capabilities — but do these new fangled gadgets really matter to DBAs in the trenches?

I recently talked to one of my longtime sources, Forrester Research Inc. analyst Noel Yuhanna, at length about Database 11g. Yuhanna, a true DBMS expert who is extremely knowledgeable about the database marketplace, said there’s no question that Oracle is the leader when it comes to innovation.

“Database 11g shows Oracle’s leadership with innovations and advancements in the database technologies,” Yuhanna said. “Basically, it is continuing to extend the database features and functionality around key areas. They are availability, performance, security and unstructured data.”

Some of the innovations or enhancements that Oracle rolls out are a direct response to customer demand. According to Yuhanna, improvements in manageability features since Oracle 9i definitely fall into that category.

“I know that there are customers who are concerned about Oracle’s manageability in version 9, and 10 has improved quite a bit on manageability features,” the analyst explained. “In 11g, the trend is continuing where Oracle is going to extend upon automation and manageability, making it much easier to manage databases.”

I can see DBAs taking advantage of new manageability features, but what about some of the other bells and whistles. Sometimes when I talk to DBAs I get the feeling that they’re happy leaving well enough alone, and aren’t necessarily eager to try out new features and functionality — essentially for fear that they might break something.

What do you folks think? What is the process like for rolling out new features in your organization, and how long does that process take? Also, is Oracle helpful on that front? 

Let me know what you think and hopefully we can get a news story going on this topic.

– Mark

I’m happy to announce the debut of Oracle Talk.

Oracle Talk is our new biweekly podcast series all about Oracle database and applications management designed for DBAs, developers and managers of Oracle shops. Each half-hour episode focuses on a specific topic and features recent SearchOracle.com content, interviews with experts and authors and a round-up of Oracle-related news, hosted by yours truly, Tim DiChiara, editor of SearchOracle.com.

In the inaugural episode of Oracle Talk, we explore why database administrators are in such a tizzy about their career choice and what can they do to improve their lot. We also answer some career and certification questions submitted to our experts, as well as rounding up the latest news from the Oracle community and beyond. In future episodes, we’ll be focusing on topics such as the impending release of Database 11g, Oracle security, Oracle vs. SAP and surviving the patching and upgrading treadmill.

Besides Oracle Talk, we have a growing collection of over a dozen podcasts about other topics, which are listed in our podcast library. If you have any comments or suggestions about our podcasting efforts, let us know!

Thanks and have a great week,
Tim

There’s some interesting discussion going on this week over at Lewis Cunningham’s blog, as well as various other blogs and forums, asking whether open source companies are dabbling in hypocrisy — are they OK with taking business away from the big boys like Oracle, but not OK with other open source vendors treading on their own territory? — and if they’re really just monopolies waiting to happen.

Lewis C points to a discussion over the competition, as it were, between PostgreSQL and EnterpriseDB — specifically, a press release in which EnterpriseDB claimed ability to deliver better performance than PostgreSQL. A post on the PGSQL Advocacy forum denounced this claim as “cow dung.” Lewis finds this a little funny/baffling, or in his words, “hypocritical crap”:

The funny part to me is that this is not a new message. It only becomes a problem when the purity of PostgreSQL is called into question. Say what you want about the evil proprietary vendors (or even that evil OTHER open source database that must not be named! HINT: MySQL. Oh my gosh did I say that out loud?) but don’t diss THE POSTGRESQL!

CNET’s Matt Asay and Roy Russo at LoopFuse are musing about whether the open source business model is inherently monopolistic: “OSS companies focusing on the proprietary competition win out in the end, but if history is a guide, they also manage to squash their own OSS competitors by doing so,” writes Russo. Does “any market ultimately [have] room for only one purveyor of free software”? “So much for peace, love and open source,” says Asay. Asay goes on to say, however, that he thinks this is an oversimplification of the open source model. “There may not be room for Yet Another Open-Source Business Intelligence Vendor (YAOSBIY for short) ;-), but surely, there’s room for plenty more in this space who drive greater performance, superior ease of use, etc.? Open source becomes a facet of how such companies compete — an important one but not the outcome-determinative one.”

Do you think there’s truth to either of these claims? Are open sourcers ultimately as greedy and territorial as their proprietary counterparts? Do they have the right to take the moral high ground? Are there room for multiple open source vendors in the market?

Have a good weekend,
Elisa

I field the questions that come to our Ask the Expert section—deleting spam and nonsense and otherwise undesirable questions before forwarding the potentially answerable ones on to our panel of experts. I don’t have to answer them, and even I get annoyed when I see the same, often vague questions over and over (“How to back up my database?”) or something that obviously came from a homework assignment. (When you get 10 questions in a row from the same email address, all addressing different “problems,” it looks a little suspicious.)

That’s why I’m working on a collection of FAQ resources for our readers, so our experts won’t have to keep answering the same queries over and over again. One of our first new FAQ offerings is a three-part SQL FAQ assembled by our witty resident SQL guru Rudy Limeback. Rudy revisits the most common questions he’s taken over the past six years, from the most basic duh-type questions, to the homework questions, to genuinely complex and interesting ones. Check out all three parts of the FAQ:

• Part one: Novice questions: These are generally very simple and are asked by beginners who are quite unfamiliar with SQL. Common examples include “What is the difference between…” questions, such as “What is the difference between a candidate key and a composite key?” Other common novice questions depend on circumstances and context, such as “Which is faster, a subquery or a join?”
• Part two: Homework questions: Our experts are usually reluctant to answer these, but Rudy will take a homework question if it can teach something interesting like a subtle nuance of SQL to all readers. For example, “All employees under a given manager” is a common but tough problem that Rudy discusses comprehensively.
• Part three covers serious and complex SQL questions that come up frequently. This is the meatiest part of the FAQ, so dig in and enjoy! You’ll discover how to delete duplicate rows, how to find the first (or last) N rows in a table, how to handle pagination and much more.

If you’ve got a burning SQL question that’s not addressed here, send it to Rudy. He thrives on them.

-Elisa

For weeks now everyone has been reporting that July 11 was to be the launch date for Oracle Database 11g. That’s the impression I was under as I went forward with the reporting for our new Oracle Database 11g Special Report. But alas, the big Oracle event in New York was more of a Database 11g preview and introduction, rather than a product rollout. Now we’re told that the new release won’t be available for at least another month. And then only for the Linux platform.

Looks like I wasn’t the only who was highly confused by Oracle’s messaging around the so called “Database 11g Launch.” Tim at Oracle-Base blog apparently was too. He wrote:

“So all the pomp and ceremony is over and Oracle 11g is launched, but as yet I’ve not heard anything about a release date. Does anyone know when it is likely to hit the shelves. I kind-of [thought] that was the big news, but as yet I’ve not seen any press releases or blog entries that specify a date. It’s not available for download on OTN yet, so I’m assuming the “Launch” and the “Release” are not the same thing.”

 Well, Tim, I hope my little blog entry here clears up some of the confusion out there.

 – Mark 

Don adds these tales of woe to our growing collection of Oracle security bloopers:

**********************************

We received a call from a client who was complaining of performance problems on their Oracle database which was running on a standalone Linux server.  The company was in the business of providing credit information to third-party companies to access an individual’s probability of financial default.

Upon accessing the server, Oracle was apparent that something was terribly wrong. Even when idle, the Oracle database was performing I/O operations and the processors were active, even though Linux did not show any active processes.  The Linux “ps” command failed to reveal any active processes.

After a Linux expert was consulted the real issue was discovered.  A disgruntled Systems Administrator had left a time-bomb on the server, to be activated when their user account was removed from the /etc/passwd file, indicating that they had been fired.

This time-bomb was activated when the System Administrator left the company to “pursue other opportunities”, and the attack was both clever and devastating.  The attacker placed a Linux daemon process called “vacuum” on the Linux server and this process was constantly polling the Oracle database, seeking new information, and e-mailing Oracle to an overseas mailbox.

This attack has disclosed the entire Oracle database of confidential information to an unknown party, and the company was held fully responsible because they failed to institute a third-party employee to manage their server security.

The attack was very sophisticated and unobtrusive.  The malicious employee had replaced the standard Linux commands with a “root kit”, an attack method readily available on the Internet.  In a root Kit attack, the Linux commands are replaced with an alias to disguise the presence of the Oracle data stealing mechanism.  In this case, the process command “ps” was replaced with the command “ps|grep –i vacuum,” such that the process would not appear within Linux. 

******************************************* 

In this case, a hacker exploited a server vulnerability, siphoned confidential information from a company’s Oracle database and shipped it to a foreign nation that did not honor U.S.. copyright law.  A foreign crook then extorted the company, proving that they had the Oracle data, and threatened to disclose proprietary secrets to a competitor unless they were paid a significant sum of money.

Faced with the loss of their competitive advantage, the company contacted the FBI and was told that there was no reciprocity with the nation and that Interpol would not be able to investigate or arrest the extortionists.  Even worse, Oracle management had not detected the leak, and had no idea how the thieves had accessed their Oracle database.

********************************************

An Oracle database sdministrator for a major university was caught “enhancing” college transcripts to allow people to gain acceptance to top professional schools.  The DBA had complete control over the Oracle database and the auditing mechanism and was charging friends and acquaintances thousands of dollars to add courses and improve existing grades.  Because the DBA controlled the audit mechanism, she was able to completely erase all traces of the fraudulent changes.

This fraud went undetected for more than five years until a professor discovered the fraud.  The professor was asked questions about a former student as part of a pre-employment background check and discovered that the student had never taken his class even though the official university transcript indicated an “A” for the course.

Tim Hall at the ORACLE-BASE blog and Andy C at nbrightside have been writing about search engines (what percentage of their blog traffic comes from search engines, what percentage of that search traffic comes from Yahoo vs. Google, etc.). Search engine minutiae is endlessly fascinating to me, and these blogs prompted me to poke around our own stats to see what’s been bringing people to the Eye on Oracle blog lately.

I compiled this list of some of the amusing and/or unlikely search strings that have recently led folks to our humble blog:

• what do you mean theoretical database
• arrogant oracle database
• sap on sql sucks
• oracle pl/sql sucks
• oracle webcenter sucks (Note to self: Use the word “sucks” more for search engine optimization)
• learn oracle dba in one week
• what would a database administrator do (I can see the shirts now: WWDBAD?)
• database work is for suckers
• recent dumb in oracle
• WHINING AND GRINING (Grining? I get 439 hits in Google for this . . . does that make “grining” a word? But then “oralce” gets over 183,000 . . . there’s even an oralce.com! Talk about capitalizing on misspellings.)
• sheryl, rich does like you (????!)
• how does the eye function in easy languages
• how does anyone ever use oracle when sql server is so much easier to use (Apparently some people are trying to engage Google in an actual conversation)
• does anybody really do enterprise architecture (Nah. I think it’s a myth)
• hate oracle dst patch
• kramer dba seinfeld
• elisa gabbert blog (Aww. I have a fan!)

What kooky phrases are turning up in your referring URLs?

-Elisa

Last week, I asked you DBAs and consultants to send in the worst Oracle security nightmares you’ve come across. A few of you have responded so far. Read them and weep:

************************** 

Terry M. wrote:  

I was working as a software consultant going on site at a defense contractor.  Security was so tight that I had to be escorted to the bathroom and searched before going into and out of the site.

I was there to install a database monitoring software package for several of their Oracle 8i database instances.  The install requires the user to enter the sys ID and password to grant select on some data dictionary tables and the on site DBA that I was working with requested that I step outside of his cube while he entered the sys password.

After several failed attempts, and my hearing him curse a few times,  I noticed a repeating pattern of keystrokes which I immediately recognized — see if you can guess:

Tap Tap Tap Tap Tap Tap … Pause … Tap … Pause … Tap Tap … Pause … Tap … Pause … Tap Tap Tap Tap Tap Tap Tap

After about 5 minutes of listening to him fail to get the password correct and cursing, I finally had to speak up . . . as I turned around and walked back into his cube, I said “excuse me, but your sys password wouldn’t happen to be ‘change_on_install’ would it?” He immediately became suspicious and accused me of somehow watching him enter the password when I was clearly behind the outside of his cube wall.  I quickly told him that I bet I knew his system user password also: ’manager’.  He was astonished and extremely embarrassed when I explained to him that those two passwords were the default passwords for the sys and system accounts on every Oracle database installed.  And that it was common practice for every DBA to immediately change those passwords to secure their database instances.

Unfortunately, we lost the sale — he explained that he had over 100+ database instances that he had to go change the passwords on; ushered me out the door; and never called back to reschedule another visit.

**************************

Rick K. wrote:

Like most Oracle professionals, I subscribe to several Usenet groups so I can keep my skills current.  Well, a few years ago a DBA needed some assistance and posted a question in which he shared his tnsnames.ora file and wondered why he could not connect to SQL*Plus with the following syntax:

sqlplus system/SecurePswd@prod

Almost immediately several people connected to this person’s production system and was able to fish around the system.  Numerous people emailed the DBA back and pointed out that he just broadcasted to the world his production connection string and password. How crazy is that?

************************** 

Anonymous wrote:

I know a firm that has a partnership arrangement with several credit card companies.  These partnerships involve the credit card companies initiating an electronic process to create an account with the firm for their card holders to receive services from that, which are then billed to their credit cards. 

Unfortunately, the credit card companies seem to have a remarkable difficulty keeping track of which accounts are billed to which credit card numbers.  As a result, the credit card companies sometimes need to ask the firm for a list of accounts associated with certain credit card numbers.  On more than one occasion, a representative of a credit card company has sent an unencrypted email listing tens of thousands of credit cards numbers, thus breaching the PCI DSS which the credit card companies are trying to enforce.

************************** 

Sean S. wrote:

Unfortunately, this comes from the government, in fact, the military. I was brought in as a consultant to manage a set of Oracle8 databases for a branch of the US military. One in particular contained sensitive data which could be used to track the whereabouts of strategic military assets around the world. It was open to the internet, on port 1521, so that remote locations could connect through the application. When I came on board, the first thing I checked for were default passwords. Of course, scott/tiger was there. What’s worse was system/manager and sys/change_on_install were, too. So I approached the manager to tell her that the password needed to be changed.

“Oh no, you can’t do that!”

“Why not, I asked?”

It turns out that there was a committee of about 60-70 individuals, contractors, vendors, and representatives that met via conference call on a weekly basis to discuss the database and application. When the database was first installed, the subject of changing the password came up, but the committee couldn’t decide on a suitable password to change it to. Debate raged for several minutes over who had the best password and policy, and with no solution in sight, the idea of changing passwords was tabled until the group could reach agreement. No action was taken, and the subject never came up again, apparently.

Your government in inaction. Needless to say, I changed the password and told them they could change it back when I left.

But wait. It gets better…

Just prior to Y2K, we learned that foreign hackers were going to attempt to compromise military computer networks. A couple of security drills ensued, but a few days prior to Christmas, 1999, we had a new task at hand. We were instructed to label every cable leading in and out of every machine in the server room, be it a server, disk array, network switch, or monitor. On December 29, we powered down every machine, and unplugged them from everything. Literally. Both ends of every cable were disconnected, be it power, network, SCSI, or a keyboard. Machines were moved out into the middle of the room so you could walk around them and see they were physically disconnected from everything. Tiles on the raised floor were left up so that you could verify that nothing was plugged in. Everything was to be left in that state until at least January 3rd.

We were told in our briefing that this was to prevent terrorists from disrupting our activities. Of course, I raised my hand to ask what I felt was the obvious question: “Aren’t we doing for the terrorists exactly what we only think they might attempt–that is, disabling our computer systems–and assuring them of widespread success where they might not accomplish anything at all?”

The answer: “No. We’re doing this on our terms.”

That’s like saying that if we had intelligence about an attack on Pearl Harbor, that we should have sunk the Arizona on December 6th, in order to be doing it on “our terms.” Sigh.

************************** 

Sigh indeed. It’s like driving by a car crash — we’re drawn to it and repulsed at the same time. If you have any (anonymous) additions to this sad and funny parade of ignorance, let’s hear it!

Have a good week, Tim 

As we reported last week, a new survey shows that IT security pros have a “disturbing lack of confidence” in the ability of organizations to use sensitive information securely.

The survey looked at the data privacy and data protection concerns of 1,000 IT security workers and compliance professionals. It found that many see the potential for disastrous data loss and feel that their organizations aren’t equipped to deal with the risk. Well-known Oracle blogger and consultant Peter Finnigan agreed, saying “my experience [with] users of Oracle databases and database users in general is that databases tend to not be securely deployed. They are better than they have been in recent years but still not where they should be in terms of protecting data.”

Frankly, it’s hard for me to believe that DBAs aren’t already doing all they can to protect their data assets. If not, why not? The years of warnings haven’t been enough? The multiple and expensive break-ins didn’t jar you into action? Don’t think it can happen to you? You think your data isn’t all that valuable? Just plain lazy?

If you are an experienced DBA or a consultant, send me the worst (and/or funniest) security nightmares you’ve seen and we’ll post the most horrifying here in the blog (anonymously, of course). Come across a company using SCOTT/TIGER as their admin login? We want to hear about it!

Have a good holiday week,
Tim