I’ve had a few folks ask me a similar question about the AWR Warehouse, occuring numerous times this week.

“How can I limit what the user of the AWR Warehouse can view in the AWR Warehouse?”

“How can I add source databases to the AWR Warehouse without DBA privileges?”

This topic bridges into the area of confusion of use of the EM12c environment, (which I consistently promote for use by DBAs, Developers and all of IT) and then the AWR Warehouse, which shares it’s user interface as part of cloud control, that currently is a DBA’s deep analysis and research tool.

The request to limit privileges to add source databases, limit view access to targets, but also bind values, full SQL statements, and advanced performance data impacts the purpose of the AWR Warehouse.  I fully understand the security requirements for AWR Warehouse access as stated in Oracle’s support documentation:

“You can add and remove source databases provided you have access to the database target and to database credentials with execute privileges on the sys.dbms_swrf_internal package and the DBA role.”

https://docs.oracle.com/database/121/TDPPT/tdppt_awr_warehouse.htm#CACEEAHD

Why is this important?

The first reasoning would be for space considerations.  AWR data with long term retention could add up to a considerable disk space if just any database was added without careful planning to ensure the right database information is retained for the right databases.  The power of the Automatic Workload Repository, (AWR) is that its always on, but always on means its always growing and this is best left to a DBA resource to ensure that allocation of space is used wisely.

Second, when discussing limiting view of data in the AWR Warehouse-  When investigating an issue with any SQL execution, there are many factors to take into consideration.  One of the most significant and commonly important information to answering why a performance issue has occurred, requires me to look into differences in the amount of data resulting in the where clause and objects vs. the data provided to the optimizer.

If we take the following, simple where clause into consideration:

where a.col1=b.col2
and a.col2=:b1
and b.col7=:b2;

When the optimizer uses data provided to it from statistics, histograms and any dynamic sampling, there is going to be a number of choices that can be made from the following information provided.

1. Hash join on table a and b to address the join or perform nested loop if…
2. Column a.col2 is unique, making the join quite small or…
3. Adding b.col7 to a.col2 to the join will make it so unique that a nested loop is sufficient.

Now, what if the optimizer decided to perform a nested loop when 16 million+ rows were returned?

To the untrained eye, some may assume that the optimizer had made a poor choice or that there was a bug and would walk away.  More often, if you have the data provided by the values passed to the bind variables, along with the data provided to the optimizer, assumptions would fall away and a very different story would present itself.

This is why Oracle requires DBA privileges to add a source database to the AWR Warehouse and to work with the data provided as part of the AWR Warehouse. This feature provides an abundance of data that is most useful to the professional that knows how to work with the Automatic Workload Repository.  This professional, to have the access required to perform this type of analysis and research should be the database administrator, so the requirement for the AWR Warehouse now makes perfect sense.

Now to return to the query, adding in the values for the bind variables, a new picture develops to research:

where a.col1=b.col2
and a.col2=6002
and b.col7='YES';

We now can verify the statistics data behind the values for both a.col2 and b.col7 and accurately diagnose where the optimizer may have been mislead due to incorrect data provided to the Cost Based Optimizer.

This may be a simple explanation behind why I believe in the DBA privilege policy was chosen for source database additions and view options to the AWR Warehouse, but hopefully it sheds a bit of light onto the topic.

The term “Service” is used to represent numerous objects within Enterprise Manager 12c.  Although the most common reference to the term is related to the Oracle Management Service, (OMS), in this blog post, we are going to drill down and refer to all services identified in the drop down menu from Targets, Management Services and Repository.

The two main services we are going to cover here are set up as part of the Oracle Enterprise Manager 12.1.0.4 installation and shouldn’t be edited or reconfigured, but I think it’s good to know where everything is if you should ever need to investigate it or are working with Oracle Support.

By default the following services will display in the EM12c environment-

• EM Console Service
• EM Job Service

These two services are essential to successful Enterprise Manager task completion.  The EM Console Service is the service that keeps the console active and accessible and the EM Job Service controls not just the EM Jobs viewable in the EM Job Activity console, but also a ton of background processing for the Enterprise Manager.

The Main Console View

Once you’ve accessed the services from the target menu, as these are both monitored targets in the Enterprise Manager, you’ll see the following information:

Along with the two, default services we expect to see for a standard Enterprise Manager installation, we also can see the status, if there are any open incidents and what system the services belong to.  You can create and remove services from here, but unless you know what you’re doing, this is mostly the view to quickly filter service information.

Creating services from this console offers you three, two generic service types and one advanced service option called an Aggregate service.

Using our existing services as examples, it may give you a bit of an idea of how a System Based service is different than a Test Based one.

A System based service monitors one single system vs. a Test based one, which monitors a one or more test based systems.  Notice that the EM Jobs Service does ONE thing-  it monitors the Enterprise Manager Job Service.  The EM Console Service has to monitor EVERYTHING that is involved in the console, which is much more complex and requires a constant test to ensure all aspects are available.

And Aggregate Service is a combination of sub-services.  If one of the many sub-services is experiencing an issue, the entire Aggregate service is dependent upon that one failure.

There are a number of other services that Enterprise Manager 12c supports:

• Business Application
• Service Entry Point
• Oracle Database Service
• Oracle Applications Service
• Siebel Services
• Beehive Services
• EM Service
• And others….

You can create, manage and remove services via EM CLI commands, too.  The following commands will support the management of each:

• create_service
• assign_test_to_target
• change_service_system_assoc
• remove_service_test_assoc
• set_availability
• set_properties
• And many, many others….

To the right of the console list of services, are Key Components incident information.  This can assist the administrator when they are experiencing a problem, but the culprit isn’t displayed in the service, a related target may be the issue:

The Key Tests also show current status and any monitoring beacons that are attached to the service.

EM Console Service

If we click on the EM Console Service, we are taken to the home page for the service.

Centering in on the upper, left section of the home page, you notice that there are a number of tabs:

• Home
• Charts
• Test Performance
• System
• Monitoring Configuration
• Topology

For any generic service, you are going to see these tabs, but we’ll go through each one of the tabs, (outside of the home tab) for the EM Console Service so that you get a good feel for what is available in each and what tab has control over the configuration of the service.

Charts

Upon clicking the charts menu, you’ll see the basic information about page response per millisecond and page hits per minute:

Unless there is a serious performance issue in the console, I commonly don’t find a lot of useful data from the default view of “last 24 hrs” and recommend changing the view to [at least] the last 7 days.  You’ll be surprised how different the data will look when you take into account the nightly maintenance jobs that keep EM12c clean and effective:

Test Performance

The next tab is for test performance.  All services have tests involved that check and see if a service is actively running and passes either one or more tests that verify it is properly functioning and available.

The value for the last EM Console Service Test is shown above in the graph, but the real information lies lower in the performance timestamp.  The test isn’t run very often, (it is a performance impact to the system, so full testing of the service is only run once every 24 hours.)  You can see the Collection Timestamp in the right hand window of 6:55am, which will also match the extended Web Transaction used to verify that the console is accessible if you click on the EM Console Service Test at the bottom.

System

The System tab displays the latest information about the status and incidents of the components connected to the service.  

As you can see from the screen shot above, a recent test of a non-released patch has caused some grief, (this is why we test these things… :)) and there are a few applications components that were impacted and need to be reviewed and addressed.  Each of the components are linked on this page, so they can be easily accessed and the incident investigated.  As this is the EM Console Service, there are a number of related components, (16 total, as shown in the right hand totals) and you can change to the next page to display the rest of the components involved.

Monitoring Configuration

The Monitoring Configuration requires a section to itself, as this has some pretty impressive links in this tab.  We are going to go through each one of these, so you can get a solid understanding of what is available:

System Association

First thing you need to remember, DON’T CLICK ON THE “REMOVE SYSTEM” BUTTON.  Although I’ve never seen anyone attempt this from the console, I can’t imagine the damage that would be done if you chose to remove a key component from the EM Console Service unless instructed by Oracle Support, so just don’t attempt it to see what happens…

As discussed above, each component associated to the EM Console Service is displayed, along with the designation as a key component with a check mark.

Root Cause Analysis Configuration

By setting the analysis mode to manual, less stress is put on the system resources to collect root cause data at any issue, letting you control when the analysis is performed.  The recommendation is to leave this set to manual mode for analysis collections and only to change it [again] with the blessing of Oracle Support.

If any root cause analysis has been performed, you would view it to the right of the list of components and types, as the bottom section is reporting info only, listing each component and any test counts.

Service Tests and Beacons

This is how cloud control knows that the console service and key components are up and available.  A set of tests are run on a regular interval in the form of a web transaction.  This means that the test logs in, verifies a result from the URL for cloud control and verifies each key component is responsive.  If all tests result in a positive result, then the status is ‘up’.

A beacon is a target used to monitor service tests, primarily to measure performance of the service. One beacon belongs to at least each service test and is used to check connectivity between the client and the OMS host or the OMS host and the OMR host, etc.  Adding multiple beacons for one service adds no value, so if for some reason, you see multiple beacons assigned to a service, there’s a problem right there.

Availability Definition

This console looks very similar to the one above, but it covers test information about the EM Console Service Tests and EM Management Beacon:

By default, both of these require ALL KEY SERVICE TESTS must be SUCCESSFUL to be marked successful and return a status of ‘up’.  As you can see, the EM Management Beacon has not done so, which results in an unavailable status.  The next step in trouble-shooting this would be to click on the Beacon name link and investigate what  didn’t return successfully.

Performance Metrics

Performance metric data is where you set your thresholds for EM Console Service response, (and the readers say “Aha!  That’s another location I can get to the valuable metric threshold setting!”)

The metric value settings for the threshold are in milliseconds and if you are receiving notifications that this is beyond your response time, look at network connection, network connection between data centers, etc. and if it’s something you can’t correct, bump up the warning and critical threshold values to provide more value.

Usage Metrics

This setting by default isn’t set, but it’s one area, you can configure if you would like.  If you are concerned about page performance and want to report on this metric data, set up the with a logical value to start.  For a small web server, 12000 page hits per minute would be a pretty good warning value with 20000 page hits per minute critical.  Use this as a beginning base, test, rinse and repeat.

Now we’ve completed a high level review of the Monitoring Configuration, we can return to the last tab, Configuration Topology.

Topology

This last tab is for those of you that love topology charts.  This is the ultimate in configuration topology, allowing you to interact with all data, component dependency and connections to the EM Console Service.

This is also where you come to realize how dependent everything is on your EM Console Service-  no website, no Enterprise Manager access.  By hovering my cursor over the service, you can quickly see the dependent HA servers, database instance, management agent and other essential hardware and software components that are involved with the EM12c hardware and framework.  To the right is filtering options to drill down into a more manageable view:

High level information about each of the targets you want to drill down on are displayed and you can use this section to create a much easier to read topology:

The topology makes even more sense if you can view all the information on each of the targets vs. me hiding host/database names…

Hopefully this review of services helps explain a little bit more about this very, very important key component to Enterprise Manager 12c.  Next post on Services, I’ll go over the all important, EM Job Service.

Database Replay is a feature in EM12c that a lot of folks are unfamiliar with, but once they learn about it, well, now then they figure out how valuable it really is.  Where the ADDM Comparison Report comes in really handy when you want to see what changed, think about how much more value there is if you could test out what a change in parameters, patches, or in code would do to a database and THEN use the ADDM Comparision Report to view the “What if’s”?  This is where Database Replay really enhances the comparison report feature.

along with the ADDM Comparison, you also have the ability to replay with ASH Analytics, SQL Analyzer and Replay’s own Comparison Report.

Now you can generate the workload capture reports via the DBMS_WORKLOAD_CAPTURE.GET_CAPTURE_INFO package, but as this is EM12c Goth Girl’s blog you’re visiting, so we’re going to discuss using Enterprise Manager 12c to produce and execute the Database Replay.

Accessing Database Replay in EM12c

Database Replay is under the Performance drop down once you log into a target database.

Once you enter the wizard, you see the steps that need to be completed to perform a successful database replay:

1. Capture Production Workload
2. Prepare Test Database
3. Prepare for Replay
4. Replay Workload on Test Database

Capture Production Workload

This is broken down into three parts:

You’ll need to click on the icon at the very right to configure the first “configurable” step, Capture Workload.  The first step is to ensure you understand the requirements to export and store the data that will be required to run the replay.

After verifying that you have the disk space and the location to restore to, acknowledge both and click on Next.

Now you can choose what to exclude from your replay.  Keep in mind, the more you exclude, the less space and replay demands will be required.

The default looks like this:

But for our post, I’m going to trim it down…a LOT.

I’m going to keep the “Do not restart the database…” and then change the filter mode to “Inclusion” and just use the DBSNMP user schema.

This is all I want to capture and click on Next.

I now need to decide what database directory to use for my export and have the option to rename the database replay or use the default naming convention.  For this post, I’m just going to leave it with the default name.

Now it’s time to schedule the workload export.  This is done via the EM Job Service, so the job will show up in Cloud Control’s Job Activity page.

The default for the job is to run immediately, but as you can see in the example for this post, I’ve scheduled it for later, (meaning you can also schedule a replay to be performed after hours, (when users are less likely to be accessing the database.)   You also can set a Capture Duration limit, which means the process will rollback if it exceeds this time.  The default for the credentials used are the Preferred Host Credentials, so try to have these set up vs. creating new ones, an option I’m not crazy about having here in the wizard, but understand why it does exist.

If you don’t have the credentials set up beforehand, you will receive the following error upon proceeding:

Once you do have the credentials set, you’re able to proceed and go to the next step in the wizard.

For Part II, we’ll proceed into the next step of preparing the test database to refresh and receive the workload to replay!

Have a great Thursday and for those folks that I don’t see at RMOUG Training Days 2015 conference next week, the horror you must feel right now!

Collaborate 2015 at the Mandalay in Las Vegas is just around the corner and the sheer amount of Enterprise Manager focused content is phenomenal!  Oracle partners and power users around the world come together each year to provide the lucky attendees the best in use cases, tips and technical know -how regarding the best infrastructure management tool in the industry.

If you are one of the lucky masses who attend this incredible conference from IOUG, Quest and OAUG, you’ll be searching the massive schedule of sessions for the presenters that will get you to the level of expertise you desire in Enterprise Manager features and products.  So let’s save you a lot of time and just tell you about all the incredible sessions and offer you some links!

Sunday, April 12th

Our adventure starts out on Sunday with the IOUG Pre-Conference Workshops.  Join Werner De Gruyter, Courtney Llamas and me for a great hands on lab titled, “Everything I Ever Needed to Know About OEM, I Learned at Collaborate!”  from 9am-noon.   After you’ve refreshed with some lunch and networked with some of the other speakers and attendees, you can then head over to Brent Sloterbeek, from Gentex Corporation to sit in on his user experience session where he speaks on his great success expanding the use of EM12c from just the DBA staff to the entire IT department.

Monday, April 13th

Monday begins the impressive list of presenters on Enterprise Manager topics, beginning with my peer, the ever impressive Courtney Llamas explaining how you can take EM12c from “Zero to Manageability”.  The schedule has Werner and I as her co-presenters, but she knows this topic like the back of her hand, so if she does have us speak, it will mostly be both of us saying, “What she said!”

The first session choosing challenge of the conference starts right here, folks!  The wonderful Rene Antunez, from Pythian is also presenting at the same time on Database as a Service in the Cloud. I pity those who have to choose between these two sessions and please don’t ask me to choose between coworkers and Rene, who I mentored.  It’s just too difficult for me to make one!

After Courtney and Rene, you can head over and see Rich Niemiec speak on the Best in Database 12c Tuning Features, followed by another person I’ve mentored for years and am proud to be the mentor of, Golden Gate expert, Bobby Curtis, who’ll take some time off from GG to tell you about how to best implement Exachk for Exadata with EM12c.

At 3:15pm, we have another competition of sessions on EM12c that you need to choose between and I’m in the middle of it, too!  Ravi Madabhushanam, from App Associates will be discussing how Easy E-Business Suite patching can be with Enterprise Manager while I discuss one of the newest symbiotic product launches from the EM12c performance team, the Power of the AWR Warehouse.  I’m not finished yet-  3:15 is a popular session time!  Erik Benner, Steve Lemme,  Seth Miller, Rene Antunez, Charles Kim and Michael Timpanaro-Perrotta, (OK, so someone has a longer name than me! ) will be on a panel discussing how IT can benefit and should consider Database as a Service! But wait-  that’s still not all at 3:15pm, (yeah, it’s going to be a tough decision…)  Michael Nelson from Northrop Grumman  will be talking about the importance and how to implement the Automatic Diagnostic Repository.

Monday finishes up with three great sessions at 4:15, (sorry folks, you have to make another difficult decision again…:) )  One of my fantastic co-authors from the Expert Enterprise Manager 12c book is up to bat, Leighton Nelson,  Introducing Enterprise Manager .  It’s an excellent complement to Courtney’s session earlier in the day, so if you are new to the product, please consider this session.  For those of you who are looking for an advanced topics, we then have Kant Mrityunjay from AST Corporation discussing  Top Automation of Weblogic Tasks and for the database performance folks, we have Alfredo Krieg Villa from Sherwin Williams, discussing how to Stabilize Performance with SQL Plan Management in DB12c.

Monday evening offers you the chance to relax and network with fellow Enterprise Manager experts  at 5:30pm in the Exhibit Hall at the Oracle Demogrounds where you can view demos on Total Cloud Control and Applications Management, both powered by Oracle Enterprise Manager! It’s a great chance to speak with product managers, some of the Strategic Customer Program team members and Oracle community power users.

Tuesday, April 14th

Tuesday kicks off EM12c sessions at 9:45am with Ken Ramey from Centroid Systems focusing on security with Protect Your Identities, but will offer some bonus tuning tips, too!  At this same time, if you are looking for the best answers on how to upgrade to EM12c High Availability, head on over and see Bill Petro from American Express who will go over a real user experience on how it’s done right!

Bobby Curtis is back up to bat, (he’ll appreciate the baseball reference… )  speaking on his topic of specialty, GoldenGate monitoring with EM12c.   Erick Mader and Jon Gilmore from Zirous, Inc. will be up at 11am to tell you all about Oracle WebLogic Performance Tuning.   You also have the chance to learn about the power of Oracle Real User Experience, (aka RUEI) from Frank Jordan of ERP Suites during this time on Tuesday.  Rich Niemiec will be speaking at 11am, too, mostly on Database 12c features, but he does promise, since it is a session on the BEST Database 12c features, he will be covering a few Enterprise Manager features, too!

Alfredo Krieg Villa is back on Tuesday at 2pm, to discuss how to save your day with Enterprise Manager 12c Administration.  You lucky devils get to learn more about the glories of Automated Patching with EM12c from Fernando de Souza from General Dynamics IT at 4:30 to finish out your Tuesday OEM sessions.

Tuesday concludes with the Exhibitor Showcase Happy Hour in the Exhibit Hall at the Oracle Demogrounds where you get another chance at Cloud Control and Applications Management demos, along with great Las Vegas Mandalay venue drinks and food!

Wednesday, Aprile 15th

We start out Wednesday at 8am with Gleb Otochin from Pythian, (ask me about my first interaction with Gleb when we worked across from each other when I trained at the Ottawa office-  great story! ) He’s a brilliant guy and he’s going to tell you all about how to build Your Own Private Cloud.  Also at 8am, Collaborate has Shawn Ruff from Mythics discussing how to Manage and Monitor Fusion Middleware while one of my co-authors on the Enterprise Manager Command Line Interface book, Ray Smith, is going to show you how to Flaunt it if You’ve Got it with EM12c Extensibility.  You also have Frank Pound from the Bank of Canada who’s going to show you how easy it is Comply with Audits using EM12c.

You get a small break from EM12c content, so take a breath, network or if you want, you can attend other sessions not on our EM12c list.  I’ll forgive you, really…  At 10:45, Krishna Kapa from UBS is going to explain Oracle Database 12c Multitenant with Enterprise Manager 12c.

At 12:30, we then have the great IOUG Oracle Enterprise Manager 12c SIG.  If you are passionate about EM12c and want to get involved in the community, this is the place to be!  We are planning out this event, even as I write this and I can tell you, it’s going to be great!

At 2:45pm, the always wonderful, Kai Yu, from Dell, will show you how to Design and Implement Your Own Private Cloud.  During this same time, Claudia Naciff and Vladimir Lugo, of Loyola Marymount University, will be making my day by promoting  Migrating Your Cron Jobs to Oracle EM12c.  I love seeing folks simplifying management of their environments with Enterprise Manager and it’s great to see Claudia and Vladimir presenting on this topic.  Last, but not least in this time slot is  Raj Garrepally from Emory University, presenting how to use Oracle Enterprise Manager, along with other tools to ease the management of Peoplesoft Environments.

Erik Benner is up at 4pm to talk about  Servers and Systems and Storage, Oh My!  Don’t miss out learning about infrastructure management with EM12c.  It’s going to be another tough decision on sessions again at this 4pm time slot-  Leighton Nelson is also speaking with Sean Brown from World Wide Technology on Database as a Server with RAC using DB12c, along with a panel from Keith Baxter on Oracle Best Practices for Managing Oracle Applications.

Thursday, April 16th

The last day of Collaborate 2015 is made sunnier by starting the morning with Krishna Kapa from UBS as he gives another great session, this time on how to successfully combine Database 12c Multitenant Architecture with EM12c.   If Las Vegas isn’t sunny enough with the addition of that first session on the list, you also have Angelo Rosado from Oracle giving another session of great content on Managing Oracle E-Business Suite 12.2 with EM12c.

At 9:45, James Lui and Erik Benner are going to host the OAUG Oracle Enterprise Manager Applications SIG.  This is a great way to get involved with Fusion Middleware passionate folks in our community, provide feedback and be part of the best in the middleware world, (not to be confused with Tolkien’s middle earth, but we’ll know who’s who if anyone shows up dressed as an elf or hobbit, right? )  Another great session on DBaaS is up during this time for all you EM12c fans from someone I’ve enjoyed presenting with at Oracle, GP Gongloor.  He’ll show you how the future is now with Advanced Database Management with Database as a Service and EM12c.

Mark Saltsman and Kumar Anthireyan from Bias are presenting at 11am, explaining the 12 Things to Consider for Migrating EBS onto Exadata, which will include some great tips with EM12c.  Mark Scardina from Oracle is going to present on Database Clouds with Oracle RAC 12c and as we know, cloud is the almighty world we live in now!  My previous boss and always mentor, Alex Gorbachev is on at this time, too, presenting on Anomaly Detection for Database  Monitoring.  His summary description describes it as a novel approach, but with Alex, should we expect anything less? All of these great presenters are up against one of my favorite peers at Oracle, Pete Sharman, who is the master of masters when it comes to Database as a Service.  He will discuss Data Cloning and Refreshes Made Easy with EM12c Snap Clone.  If you want to learn from the master on DBaaS, this is the guy!  To make things even more difficult, my team members from SCP, Courtney Llamas and Werner De Gruyter are presenting in this same time slot to take you Under the Hood of the Enterprise Manager.  This session covers a lot of what we do as part of the SCP team and I know how much I learned from both of these wonderful teammates when I joined Oracle.  If you are considering implementing EM12c, this is an invaluable session.

To close out the conference on EM12c sessions, Courtney Llamas will easily convince everyone the power of Smarter Monitoring with Adaptive Thresholds and Time Based Metrics.  For anyone who is looking to “silence the white noise” in their environment, this is a not-to-be-missed session.  Anthony Noriega will also be presenting at during this time slot on a great case study revolving around Moving ASM Files that includes EM12c, so let’s give these great presenters on the last day our support  and don’t miss out on all the great content that is going on till the very end!

AND…

For those of you who are interested in Social Media, I’ll also be on the Social Media panel on Monday at 4pm, (as soon as I finish my AWR Warehouse session, so if you see me running down the hall, you know where I’m headed!) and if you are a WIT, (Women in Technology), you’ll soon see the announcement that I’m your speaker for the Monday luncheon.  I won’t ruin all the surprise and I’ll leave it to IOUG to let you in on the special topic I’ll be presenting on!

If you want a “just the facts” list of all the EM12c sessions for the conference, go to the following link, which will provide everything you need to make it easy to add your choices to My Show Planner.  See you at IOUG Collaborate 2015 in April at the Mandalay!

Just over a year ago, it became public that I was leaving Enkitec for Oracle.  Most folks had similar questions surrounding my decision, including my logic in giving up my ACE Director, an assumption that I would just disappear into the Oracle “machine”, considering the sheer size of the company and with my prominent company work history, what had prompted the decision.

In this last year, my career has definitely surprised more than a few folks.  I, in no way, disappeared into Oracle.  In fact, I’ve been made more prominent and in doing so, assisted in shining a light on the Enterprise Manager group.  I’m continually offered great opportunities to be more and do more with the OEM group, which I appreciate greatly.  I interviewed my potential managers as much as my employer interviewed me.  I believe your job is only as good as your boss and my managers are incredibly important to all that I’m able to do in the technical arena, along with the Oracle community.

  **Thanks to Philippe Fierens for the video

I presented at 11 conferences, a master class, three webinars and a number of local events.  During this time, I worked with customers and implemented a large number of Strategic Customer Program, (SCP) “Fast Track” engagements for Enterprise Manager.  These are projects that EM12c isn’t an after-thought to a project-  it’s the FOCUS of it.  I knew if I wanted to gain more knowledge of Enterprise Manager, that I was going to need to go back to the source and that meant Oracle.  There’s been no disappointment here-  I get to be part of the first line of huge OEM projects that implement HA Enterprise Manager environments with advanced features, plug-ins and other enhanced architecture designs.

Inside Oracle, I’m working with some of the best of the best in the Enterprise Manager world.  The Architects, Product Managers and Consulting Members of the Technical Staff that make Enterprise Manager the great product it is are who I get to interact with every day.  I have insight into the framework, plug-ins and other features that I simply would not get outside Oracle.  I have access to systems and development vision that I wouldn’t have as a customer.  The Strategic Customer Program, (SCP) is the first line to directing the product with the vision it requires, as so often many that are working on developing the features may be very distant from the customers-  where we are the ones interacting with them and are expected to come up with the answers.  This has shown in my blog posts, as I dive deeper into new EM12c features, plug-ins, etc.

My removal of my ACE Director, as it is a customer program, was in my mind, not a loss, but just something I had already achieved and I needed to go onto new challenges. I think my only concern was due to the small number of women involved in the program, but I found that I can still support the ACE program and am still working to support ACE submissions and mentoring.  That’s what’s really important to me. The only people I have seen viewing this as a mistake are commonly those that look down at others for something, no matter what and are dealing with their own insecurities, so I’ve found myself pretty unconcerned with these few.

After joining Oracle, I was able to locate the bylaws for Oracle user group involvement as an employee and retain my Conference Director role at Rocky Mtn. Oracle User Group, (RMOUG), by becoming a non-voting Board Member Emeritus.  I’ve found new ways to involve myself with the user group community, also taking on the role as the user group liaison for the Enterprise Manager group for all interaction.  I’ll be working with IOUG in the future to find ways to streamline and assist other user groups with consolidating our resources and helping each other be more successful- doing more with less, as many of the regional user groups are non-profit, have small budgets and limited resources.

Oracle has been very supportive of my Women in Technology initiatives.  They know this is something very important to the world of technology and my own hopes for a more balance IT workforce.  To have my manager email me about something I’m working on and end it with, “btw, read your <insert WIT article title here> the other day, great read!” and then tell me about some of the things he’s dong with his kids is really great to hear. That he recognizes the importance of what I’m doing, the essential work of mentoring those around us is impressive, too.

Some felt that my social media presence would be impacted-  assuming that Oracle’s Social Media Policy would suppress my personal brand after coming on board.  The opposite is quite true.  I’d been aware for quite some time, that if you had an issue with social media, it was due to companies NOT having a social media policy.  This leaves those that don’t understand social media or are uncomfortable with social media to make assumptions of what is allowed or what they think SHOULD be allowed.  I have never had an issue with a company surrounding social media when a company had a social media policy.  My social media value has done nothing but risen since joining Oracle and I thank everyone for their continued support.

My first year has been one of growth, opportunities and greater depth of knowledge in the Oracle Enterprise Manager product.  I am thrilled with the new opportunities that are on the horizon and look forward to the upcoming year!

Here’s a quick guide to my schedule of sessions at IOUG Collaborate 2015 for this year.  I’m looking forward to seeing everyone next week, (I’ll be arriving on Saturday, so I really should say in a couple days) and have no doubt we’re up for another phenomenal conference at Las Vegas, this year at the Mandalay!

Additionally, there are some great opportunities in professional development at the IOUG Strategic Leadership Program.  I’ll be speaking with James Lui on how to improve your value with personal brands and social media.

I’m the WIT luncheon speaker on Monday, so for those of you that signed up for this great, yearly event for women in tech before it sold out, I’m looking forward to meeting as many of the powerful women in our industry, (and hopefully a few men, too!) before and after the event.

There are also a few SIG’s that I’ll be attending that aren’t on the schedule:

Tue-12:30 p.m. – 1:00 p.m.Cloud Computing and Virtualization SIG Meeting (ID: 943)

Wed-12:30 p.m. – 1:00 p.m.Oracle Enterprise Manager SIG Meeting (ID: 949)

Thanks to everyone at IOUG for the opportunity to be so involved with this great conference and see everyone soon!

A common issue I’ve noted are dump files generated from the AWR Warehouse, but upon failure to transfer, the dumpfiles simply exist, never upload and the data is stuck in a “limbo” state between the source database, (target) and the AWR Warehouse.  This can be a very difficult issue to troubleshoot, as no errors are seen in the actual AWR Warehouse “View Errors” and no data from the source is present in the AWR Warehouse.

Empowered by EM Jobs

If you go to Enterprise –>  Jobs –> Activity and inspect the Log Report after a search for %CAW% jobs that perform the extraction, transfer and load that experienced a problem, you will then be able to view the error and can inspect the details of the issue.

If you double click on the job in question and note in the example above, you’ll notice that the jobs have the naming convention of CAW_RUN_ETL_NOW.  This is due to a force run via the AWR Warehouse console, (Actions –>  Upload Snapshots Now.)  If the job was a standard ETL run, the naming convention will be CAW_TRANSFER.

The job is a multi-step process, so you will see where the extraction step failed.  This is a bit misleading, as the previous failure that set into motion was a preferred credential issue that stopped the transfer step to the AWR Warehouse.  If you look far enough back in the jobs, you’ll find the original error, but now we are stuck in a loop-  the source can’t go forward once the credentials are fixed and yet it’s difficult for someone unfamiliar with the process to know where it all went wrong.  The first recommendation is often to remove the database and re-add it, but in this scenario, we are going to kickstart the process now that the credentials have been fixed.

Digging into a Job Error

As we stated earlier, you’ll see in the steps, that the extract failed, but the transfer would have been successful if a file had been created.  Double click on “Failed” to see the error that occurred:

In the SQL output, the error states, ORA-20137:  NO NEW SNAPSHOTS TO EXTRACT

Now we know that no files have been uploaded to the AWR Warehouse, yet, the logic written to the AWR Warehouse package that is in the DBNSMP schema in the source database thinks it’s already pulled all of these snapshots to an extract and created a dumpfile.  This error is very clear in telling the user what is going on.  There is a data telling the ETL process the data ALREADY has been extracted.

Disclaimer:  This solution we are about to undertake is for a BRAND NEW ADDITION TO THE AWR WAREHOUSE ONLY.  You wouldn’t want to perform this on a source database that had been loading properly and then stopped after successful uploads to the warehouse, (if you have one of those, I would want to proceed differently, so please keep this in mind before you attempt this in your own environment….) This fix is also dependent upon all preferred credential issues to be resolved BEFOREHAND.

The Kickstart

To “kickstart” the process after a failure, first, verify that there are no errors that aren’t displaying in the console:

select * from dbsnmp.caw_extract_metadata;

Next, gather the location for the dumpfiles:

select * from dbsnmp.caw_extract_properties;

There will be one line in this table-  It will include the oldest snapID, the newestID and the location of the dumpfile, (often the agent home unless otherwise configured.) This is the table the logic in the package is using to verify what has been already extracted.  We now need to remove this tracking information and the pre-existing dumpfiles created in the previous failed processes:

1. Make a copy of this table, (create table dbsnmp.caw_extract_prop2 as select * from dbsnmp.caw_extract_properties;)
2. Truncate dbsnmp.caw_extract_properties table.
3. Delete the extract, (dumpfiles) from the directory shown in the caw_extract_properties table.  Don’t remove anything else from that directory!
4. Log into the AWR Warehouse console.
5. Click on the source database you just scoured of the ETL extract files.
6. Highlight the database, click on Actions, Click on “Upload Snapshots Now.”
7. View the job via the link displayed at the top of the console and monitor to completion.
8. Once the job has succeeded completely, remove the dbsnmp.caw_extract_prop2 table and the backup files you moved that were created earlier from failed extracts.

You should now see successful upload jobs from this point on in the job, along with data in your AWR Wareshouse:

When installing Enterprise Manager 12c, the host value can come from a number of places for different applications/tiers.  For most, it comes from the environment variable $ORACLE_HOSTNAME, (for Windows Servers, %ORACLE_HOSTNAME%).

The OHS1 target, aka Oracle Apache in the middle tier of the EM12c environment pulls it’s value from the etc/hosts file, (for Unix as well as Windows) and so it is vulnerable with a virtual host name or host name change occurs.  It can, however, be updated post installation when the OHS1 target fails to return an active status in the EM12c console.

Update the Configuration File

The file that control the configuration of the OHS1 target is the topology.xml file that is located in the $OMS_HOME\user_projects\domains\GCDomain\opmn\topology.xml

Edit the topology.xml file and replace/add the following entries in bolded text, replacing Virtual_Cluster_name with the name of the Cluster:

 Save the file with the new changes.

Remove the OHS1 Target

Log into your EM12c console as the SYSMAN user, (or another user with appropriate privileges) and click on All Targets.  Either do a search for the OHS1 target or just scan down and double-click on it.  The target will show as down and display the incorrect associated targets with the HTTP Server:

You will need to remove and re-add the target to have the EM12c utilize the topology.xml file configuration update to the new host name.

To do this, click on Oracle HTTP Server–> Target Setup –> Remove Target. The target for the Oracle Apache server/HTTP Server, along with its dependents have now been removed.

Refresh the Weblogic Domain

To re-add the OHS1 target, we are going to use a job already built into EM12c.  Go back to All Targets the Targets drop down.  At the very top you will commonly see the EMGC_GCDomain, (Grid Control Domain, yes, it’s still referred to it as that… :))  Log into this target.  There are two “levels” to this target, the parent and then the farm.  Either one will offer you a job in the drop down to Refresh Weblogic Domain.

Once you click on this job, it will ask you to remove or add targets.  You can simply choose to Add Targets and the job will first search for any missing targets that need to be re-added.  Commonly it will locate 12 and display a list of the targets it wishes to add.  You will note that the OHS1 target now displays the CORRECT host name.

Close the window and choose to complete through the wizard steps to add these targets to the Weblogic domain.

Return to All Targets and access the OHS1 Target to verify that it now displays an active status-  it may take up to one collection to update the target status.

Metric thresholds have come a long way since I started working with OEM 10g.  I remember how frustrating it could be if an ETL load impacted the metric values that had to be set for a given IO or CPU load for a database when during business hours, a much lower value would be preferable.  Having to explain to the business why a notification wasn’t sent during the day due to the threshold set for resource usage for night time batch processing often went unaccepted.

With EM12c, release 4, we now have Time-based Static thresholds and Adaptive thresholds.  Both are incredibly valuable to ensuring the administrator is aware of issues before they become a problem and not let environments with askew workloads leave them unaware.

Both of these new features are available once you are logged into a target, then from the left side menu, <Target Type Drop Down Below Target Name>, Monitoring, Metric and Collection Settings.  Under the Metrics tab you will find a drop down that can be changed from the default of Metrics with Thresholds to Time-based Static and Adaptive Thresholds which will allow you to view any current setup for either of these advanced threshold management.

To access the configuration, look below on the page for the Advanced Threshold Management link-

Time-Based Static Thresholds

The concept behind Time-based Static thresholds is that you have very specific workloads in a 24hr period and you wish to set thresholds based on the resource cycle.  This will require the administrator to be very familiar with the workload to set this correctly.  I understand this model very well, as most places I’ve been the DBA for, I was known for memorizing EXACTLY the standard flow of resource usage for any given database.

In the Time-based Static Threshold tab from the Metrics tab, we can configure, per target, (host, database, cluster) the thresholds by value and time that makes sense for the target by clicking on Register Metrics.

This will take you to a Metric Selector page that will help you set up the time-based static thresholds for the target and remember, this is target specific.  You can choose to set up as many metrics for a specific target or just one or two.  The search option allows for easy access to the metrics.

Choose which metrics you wish to set the time-based static thresholds for and click OK.

You can then set the values for each metric that was chosen for weekday or weekend, etc.

You will be warned that your metric thresholds will not be set until you hit the Save button.  Note: You won’t be able to click on it until you close this warning, as the Save button is BEHIND the pop-up warning.

If the default threshold changes for weekday day/night and weekend day/night are not adequate to satisfy the demands of the system workload, you can edit and change these to be more definitive-

Once you’ve chosen the frequency change, you can then set up the threshold values for the more comprehensive plan and save the changes.  That’s all there is to it, but I do recommend tweaking as necessary if any “white noise” pages result from the static settings.

Removing Time-based Static Thresholds

To remove a time-based threshold for any metric(s), click on the select for each metric with thresholds that you wish to remove and click the Remove button.  You will be asked to confirm and the metric(s) time-based static threshold settings will be reverted to the default values or to values set in a default monitoring template for the target type.

Adaptive Thresholds

Unlike the Time-based Static Thresholds, which are based off of settings configured manually, Adaptive Thresholds source their threshold settings off of a “collected” baseline.  This is more advanced than static set thresholds as it takes the history of the workload collected in a baseline into consideration when calculating the thresholds.  The most important thing to remember is to ensure to use a baseline that includes a clear example of a standard workload of the system in the snapshot.

There are two types of baselines, static and moving.  A static baseline is for a given snapshot of time and does not change.  A moving baseline is recollected on a regular interval and can be for anywhere from 7-31 days.

The reason to use a moving baseline over a static one is that a moving baseline will incorporate changes to the workload over time, resulting in a system that has metric growth to go with system growth.  The drawback?  If there is a problem that happens on a regular interval, you may not catch it, where the static baseline could be verified and not be impacted by this type of change.

After a baseline of performance metric data has been collected from a target, you can then access the Adaptive Thresholds configuration tab via the Advanced Threshold Management page.

You have the option from the Advanced Threshold Management page to set up the default settings for the baseline type, threshold change frequency and how long the accumulation of baseline data should be used to base the adaptive threshold value on.

Once you choose the adaptive settings you would like to make active, click on the Save button to keep the configuration.

Now let’s add the metrics we want to configure adaptive thresholds for by clicking on Register Metrics-

You will be taken to a similar window that you saw for the Time-based Static Thresholds.  Drill down in the list and choose the metrics that could benefit from an adaptive threshold setting and once you are done choosing all the metrics that you want from the list, click on OK.

Note:  Once you hit OK, there is no other settings that have to be configured.  Cloud Control will then complete the configuration, so ensure you have the correct you wish to have registered for the target.

Advanced Reporting on Adaptive Thresholds

For any adaptive threshold that you have register, you can click on the Select, (on the right side of the Metric list) and view analysis of the threshold data to see how the adaptive thresholds are supporting the metric.

You can also test out different values and preview how they will support the metric and decide if you want to move away from an adaptive threshold and to a static one.

You can also choose click on the Test All which will look at previous data and see how the adaptive thresholds will support in theory in the future by how data in the baseline has been analyzed for the frequency window.

For my metric, I didn’t have time behind my baseline to give much in the way of a response, but the screenshot gives you an idea of what you will be looking at-

Removing Adaptive Thresholds

If there is a metric that you wish to no longer have a metric threshold on, simply put a check mark in the metric’s Select box and then click on Deregister-

You will be asked if you want to continue, click Yes and the adaptive threshold will be removed from the target for the metric(s) checked.

Advanced threshold management offers the administrator a few more ways to gain definitive control over monitoring of targets via EM12c.  I haven’t found an environment yet that didn’t have at least one database or host that could benefit from these valuable features.

As part of the projects I work on at Oracle, it’s often ensuring that those customers that wish to deploy Enterprise Manager, (EM12c) to large environments, have the correct settings and are tuned to offer the best performance from the Oracle Management repository database, weblogic and up to the console URL’s accessed by users.  This means that these large environments often receive recommendations from our experts  that differ from what EM12c comes “out of the box” with.

For those that aren’t receiving internal Oracle training on what to look for and how to tune EM12c tiers, there are some new features in release 4 that should be checked out by anyone using EM12c.

EM12c Health Overview

The Health Overview is accessible via the Setup menu, (right side of Console), Manage Cloud Control and Health Overview.

We’ll go over each of these new monitoring tools, but the Health Overview includes valuable information about the health of both the Oracle Management Repository, (OMR) and the Oracle Management Service, (OMS).

The overview page breaks down into easy to understand sections.  The first is basic information and workload on the OMS:

From here you can see all pertinent, high level information about the OMS/OMR, including OMS information, the number of agents, including status counts on availability and if a load balancer is used in the EM12c configuration.  Important target status availability is posted, along with how many administrators have been given access to the EM console.

Below this we can view the backlog graph on the right hand side of the page:

That right hand graph is important since along with the upload rate, you can see if there is a consistent backlog of XML files to be uploaded and that can signal performance trouble.  A backlog can cause problems, as this beings there is a backlog for the loader can delay receipt of critical alerts and information about a target.  If the backlog becomes too extensive, an agent can reach a threshold on how many files it can handle backlogged and stop collecting, which is a significant issue.  It’s important that if serious backlog issues are noted, tuning options to deter from them, (like add a load balancer to assist in managing the workload or a second OMS.)

Repository Details

The next section includes connection information, which also has the service name, the database name and database type, the port and job system status.  On the right is a graph showing if any backoff requests have been sent.  These occur when the OMS is busy processing an XML upload and requests the agent to hold off on sending anymore files until it has finished.

Notification Information

Scanning down from the backoff graph in the Health Overview displays the Notification backlog graph.  Knowing how backlogged your time-sensitive notifications are performing is crucial when someone asks if there is anyway to know why one of the notifications weren’t received in a timely manner and you can quickly assess if it is an issue with EM12c or if the problem resides elsewhere.

Alerts

The last section in the health overview covers incident management.  You can easily see if there are any Metric Collection Errors, (corresponding this to the backlog data above if necessary), access related Metric Errors and Related Alerts.

You also can choose to launch the Incident Manager from the Health Overview console if you wish to get more details about all incidents currently in the queue. This view is really to give you a very high level account of what incidents are currently open and related alerts and metric errors.  Use that button to launch the Incident Manager if you wish to see what the alerts are all about.

We’ll dig into the deeper diagnostic data offered in EM12c, release 4 for the OMR, OMS and Agents in subsequent posts, so until next time!

The new Oracle Enterprise Manager 12c Command Line Interface book is available via a number of locations, including Amazon and directly from Apress.  If you are an EM12c fanatic or just learning and want to learn more, consider the new book that will show you why the command line returns the DBA to the golden age, empowering Enterprise Manager to script and enable tasks at a global level!

There are a large number of “moving parts” when performance tuning or trouble shooting an Enterprise Manager environment.  The new EM performance features, (available in release 12.1.0.4) are there to assist you in understanding the source of the issue and can really make the difference for those that are unfamiliar with the challenges of Weblogic, java, network or other complexities that make up EM12c and aren’t commonly thought of as part of the DBA’s job role.

Now that we’ve finished with the Health Overview, we can look deeper into the health and performance of the two most well known components of the EM12c architecture, the Oracle Management Repository, (OMR) and the Oracle Management Services, (OMS).

Due to the impressive features offered in the new EM performance consoles, I’m going to break these up into multiple posts and start with OMR and focus on the Repository Tab.

The Repository

The Repository Tab is accessed via the same Setup menu in EM12c console:

Once accessed, there are a number of tabs at the top of the page, Repository, Metrics and Schema.  Starting with the Repository tab, (Left to Right) we’ll inspect what specific performance data is important when reviewing an OMR.

Repository Tab

The Repository page is going to display a number of graphs that tells you everything from specific information about the OMR database, incidents involving the OMR database and even how collections performance at the database repository level.  It is important to remember that this tab is all about the Repository, (OMR) and should not be confused with the Service, (OMS).

Basic Information

We begin by viewing information about database name, space allocated, space used number of current sessions actively connected to the OMR.

All of these are links, so you can click on the link and a you’ll be taken to a detailed view of the data and more information to investigate if you have questions.

For the Target type for the OMR, you can click on the target name, (db name) and the console will take you to the home page for the OMR database.

Click on Last Backup date and the console will take you to the backup activity report for the OMR database.

Click on Space Used and the console will then bring you to the Management Services and Repository page drill down for Repository Tablespace Used.

There is a ton of information in this location and we’ll dig deeper into it as a separate post, but just to understand how user friendly the interface is, note the links you have at your fingertips right here.

If you click on Management Service Repository Sessions, the following table with session type and counts will display:

Incidents

On the right hand side of the top of the page, we access to the incidents linked to the OMR.  No other incidents will be listed except the ones connected to the OMR, so this is a great place to check first when you are experiencing issues.

Notice that it includes incidents for page processing time outs to the OMS and collection timeouts.  This can be very helpful when you are experiencing slow response and need to know where the issue is sourced from.

Initialization Parameters for the OMR

Not only does the next graph identify what size category you fall into for your Enterprise Manager environment, (small, medium or large) but it also lets you know if any of your parameters are outside of the recommended sizing for that category.

In our example, you can see that we don’t have a set MEMORY_TARGET value and that is outside of our compliance as we recommend to have this set to one.  We can also view each of the values we do have set and how they compare to what Oracle thinks the minimum value for that category of OMR size should be.

Job Scheduler Status

To the right of the Init parameters is all the graph with information pertaining to all the jobs running in the OMR to support the Enterprise Manager environment.  Unlike Job Activity in the console, this is reporting all those jobs that are taking care of the Enterprise Manager.

If a job fails and you have the option to edit the schedule to run again, (the glasses icon) then you can click on the glasses and the following popup will show and you can then enter a new time for the job to retry:

Once you enter in the new time to run the job, click on Save and verify that the job has been successful in the console view, (green check mark vs. a red X.)

Collections Performance

At the bottom left, Collections is the next category that’s covered.  If collections aren’t uploading to the OMR, then the console isn’t able to provide the most up to date data and notifications of incidents and alerts aren’t sent out to notify administrators of issues.  Timely collections and the performance of collections is of a great concern to an EM Cloud Control administrator.

The graph is well laid out and shows clearly the number of collections in backlog and throughput performance.   The top of the graph, when hovered over, will show you the warning and critical threshold line for number of backlogs allowed.

Backlog is an issue, as if it gets too high and hits the threshold, your agent can stop uploading.  You can also see the duration, on average of the collections and view over time if the duration is increasing.  If you use a lot of metric extensions or plug-ins, this is something you’ll want to monitor, so this graph is extremely helpful when inspecting collection performance.

By hovering your cursor over the Collections Backlog line in the graph, I then am offered a number of options to look into the performance:

You have the option to click on Problem Analysis to look into the backlog, Metrics Detail or go to the Target Home.

Problem Analysis

As my EM environment is running quite smoothing at the OMR level, there isn’t a lot to show you in the Problem Analysis, but I wanted to at least give everyone a peak into this cool, new tool.

First of all, if I did have an issue, there would be collections showing in backlog.  This is very important for an administrator to check and ensure that backlog is not occurring.

As there is no backlog, you can see, my resource usage by my collections is pretty consistent and quite below the thresholds expected for most of the resource types shown:

You can also export the data from the table view, (small link at the bottom right of the screen, not shown) if you need the raw data.

You will note that my memory utilization is creeping, little by little to the critical threshold.  This is commonly due to java garbage collection causing a small memory leak and should be reviewed from time to time.  If it is considerable, the java heap should be examined and a more efficient value set.

Adding Metrics to the Performance Analysis

On the right hand side of the Performance Analysis, you will notice the Metric Palette.  This offers you the opportunity to go from the standard configuration to display more data on the existing metrics or add analysis on other metrics, such as Agents and Page Performance.

It’s important to know, even though you can be brought to this page from many different links within the OMR/OMS Performance pages, while you are in the Performance Analysis, you can inspect other performance metric factors than just the original ones you are reviewing.

For our example, we’ll add an additional metric graph,(Time estimates for clearing backlog) for review to the left hand analysis page-

We now have an additional graph on the left hand side analysis to compare to our existing data to see if the load times correlate to resource usage:

This can be done for dozens of metrics and offers some incredible analysis power when researching performance issues with EM12c.  The Performance Analysis link is one of the most powerful tools for locating where a bottleneck in performance is coming from and very quickly.  The fluid ability to add metrics to the graphs section and see how they correspond to the other resource usage is incredibly beneficial as well.

Metric Details

Now back to our Collections graph, if you remember we had three options when we click on the blue line:

By clicking on the Metrics Details link, we’ll then go to performance page for All Metrics.

This page displays information about the number of short and long running collections in backlog and will display the status if the threshold value has been hit for backlog quantity.  The page functions similar to Incidents, in that you can click on the right middle button to display the Task Class information highlighted to full page.

You are also offered the option to modify thresholds if the current values don’t meet the demands of the system is under currently, but know that the recommended values are their for a reason and the option to change them should be seriously researched beforehand.

Target Home

This link takes you to the Overview and Health page for the OMR.  I get to save a lot of typing by just sending you to my blog post on this great feature!

A final clarification, too-  the three options available, Performance Analysis, Metric Details and Target Home are options available for each metric by double-clicking in the Repository Collections Performance or the Metric Data Rollup graph, which we’ll discuss next.

Metric Data Rollup Performance

The last graph, in the right hand bottom corner, is the for metric data.  This graph displays the number of metric records rolled up and the throughput per minute for this data to be uploaded into the OMR.

We again have the ability to inspect performance analysis by double-clicking on the metric in the graph.

Each of the three options work almost exactly the same way as I demonstrated for the Collections Performance, but the data is based on the metrics rollup.

The main functionality of each of these sections is to realize how many different ways you can do performance analysis on different performance data:

Yes, even the legend can be clicked on and a detail option chosen.

That completes the review of the Repository Tab, remember, I have two more tabs to cover in posts before we dig into the Management Services and Agents performance consoles.

I’ve been told many times that the OMS for EM12c can take quite some time to start on Windows.  Some told me it took anywhere from three to up to fifteen minutes and wanted to know why.  I’ve done some research on the challenge and it is a complex one.

Let’s start this post by stating that even though I’m focusing on the OMS service that is part of the Windows installation of EM12c from Oracle, that in no way is it to blame, nor is it the only application to have this problem, (so this post may help many others) and it has more to do with over-engineering on MANY different non-Oracle levels and in no way is it a bug.  At the same time, it can really impact the quality of user experience with EM12c on Windows and it helps if you know WHAT is causing the challenge vs. what will easily have fingers pointed to as the blame. We all know that Oracle is blamed until proven innocent, so it’s important that we understand what is happening to correct the problem vs. just pointing fingers.

As most DBAs aren’t as familiar with the Windows OS Platform, lets quickly review what a Windows service is and why its important-

A Microsoft Windows services, formerly known as NT services, creates long-running executable applications that run in their own Windows sessions. These services can be automatically started when the computer boots, can be paused and restarted, and do not [require a] user interface.

When installing Enterprise Manager 12c on Windows or installing even the Oracle database on the Microsoft Windows OS platform, a service is created to support the application.  This service can be created a number of ways, but for Oracle, they support the following:

oradim - new -[sid] -intpwd [password] -maxusers [number] -startmode [auto|manual] -spfile [directory location of spfile]

emctl create service [-oms_svc_name <oms_service_name> -user <username> -passwd <password>]

and then we have Windows method of a service command:

sc create [service name] -binPath= "[path to executable to start app and argument]" start= [auto|manual] displayName= [name to display]

Each of these options are supported to create many of the different services that are needed to support different features/targets in Enterprise Manager and are used as part of the installation process via the Database Configuration Assistant, the Network Configuration Assistant and the Oracle Installer.

One of the enhancements that they are working on for EM12c is moving the java thread startup and stop from serial to multi-threaded processing.  This is going to speed up the start and stop of the OMS extensively, (anyone tracing the startup of an OMS to see where time is being spent will undoubtedly see that over 80% is the weblogic tier….)

Until this enhancement is made, the extended time trips a few safety measures that are built at a number of levels into services to ensure they stay up.  If a service isn’t up, well, you aren’t going to be using the application, so unfortunately for us, this is where the OCD of the development world has come back to haunt us….

Tracing and Advanced Logging

First, we need to get more info from our node manager to see what is starting the service and when it’s timing out and what is restarting it.  We can do this by going to the following:

$GCINST_HOME\NodeManager\emnodemanager

Make a backup copy and then choose to edit the original nodemanager.properties file

By default, the loglevel=info

There are numerous log level settings:

• SEVERE (highest value)
• WARNING
• INFO
• CONFIG
• FINE
• FINER
• FINEST (lowest value)

My recommendation is to set it to FINEST if you really want to log whats going on, but don’t leave it there, as it will produce a lot of logging and unless you are trouble-shooting something, there just isn’t any need for this amount of fine detail, so remember, a restart of the OMS service is required to update any change to the logging.

Update the loglevel info, save the file and restart the service.  The data will be saved to the following file:

$GCINST_HOME\NodeManager\emnodemanager\nodemanager.log

To understand more about tracing and logging, see the Oracle Documentation that can take you through it, (as well as save me a lot of typing… :))

Trace and Log Files

em.start                    Tells you if there were any time outs and at what step the timeout occurred.

OracleManagementServer_EMGC_OMS1_1srvc.log  This is the logged startup and shutdown of the actual service.

nodemanager.log     This is the log of the nodemanager’s interaction with the OMS service.

EMGC_OMS1.out    Steps of weblogic startup, java threads and times.

emctl.log                  Also shows timeouts set by emctl start process.

emoms_startup.trc  Shows timeout by connections, (including sqlnet timeouts)

emoms_pbs.trc      Shows actual timeouts at java level

There’s more data out there than this, especially if you use the EM Diagnostics kit, but just to start, it’s a good beginning.

Services

The OMS Service in Windows uses a standard naming convention, so it should look very similar to the one below:

Even though we are seeing one service, it can be controlled by many different daemons to ensure it is always running, as well as managing how long it has before timing out when it starts and restart options.

1. Service Timeouts:

There are two in the registry, depending on the version of Windows server that you have.  These are here to assist you, but due to redundancy, they could impact you as well. These two values control how long to wait for a service to start before timing out and how long to before killing a service or if unresponsive to kill.

To see these, you will be working in the registry.  The registry is the nervous system of the OS, so take great care when working with it and always make a backup of the folder you are working in before making any changes.

Click on Start –> Run Enter “Regedit” and click OK Go to Computer\HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control

Right click on the Control folder, choose export and save off the registry file as Services_TO.reg
In right hand “details” view and remove the following values, (either or both may be present) or even better, set them to a time that will allow enough time for the OMS to start before these come in and try to restart:
ServicesPipeTimeout
WaittoKillServicesTimeout

Remember, any changes you make here do not take effect until after you restart the computer.  You can revert any changes by importing the saved registry file backup you made beforehand and performing another restart.

2. Auto restart of the OMS by the Node Manager

The node manager’s job is to ensure that the Windows Service is up and running for OMS. It is there check the OMS service and if it sees its down, restart it.  If this is attempting to restart the OMS service while the registry setting are attempting to restart the OMS service, well, you are going to start seeing the issue here.

To stop Nodemanager from attempting to auto-restart service upon timeout:

Go to $GCINST_HOME/user_projects/domains/GCDomain/servers/EMGC_OMS1/data/nodemanager/startup.properties

Create backup of the startup.properties file and then open the file in an editor such as notepad or wordpad:
go to the following line: AutoRestart=true
Change value to “false“
Save the changes and the node manager will no longer attempt to autorestart the service if it sees it down once restarted.

3.  Clustered services added to a failover, Oracle Failsafe or other clustering process, (not RAC).

Clustering, at an OS level is primarily for high availability, so redundant checks and restart options are built in everywhere for Windows services added.  In the example of a failover cluster, the OMS service is added to the failover node.

This allows for it to automatically fail over with the shared Virtual server and shared storage to the passive node and start up if there is a failure.  The clu2 Virtual server has policy settings telling the OS what to do in case of failure and how to restart.  This, by default is applied to all dependent resources and shared storage allocated to it.

As you can see in the clu2 properties, the policies have been set if:

• A failure occurs, restart services and storage on the original active node.
• If the restart fails, then failover to the passive node.
• If the service or resource doesn’t start within 15 minutes, timeout.

You’ll also notice there is an option to not restart, as well as how soon a restart should be attempted.

You can update this at the server level properties, which will automatically propagate to the dependent resources, (it is the master of all policy settings, so you should set them here.)

We have now asked in case of failure, do not restart and don’t timeout for 30 minutes.

Summary

I’ve shown you all the redundant settings that have been built in to ensure that the service is restarted and how long it can attempt to start before timing out and if it should restart and how long between restarts.  The key to all this is knowing that only ONE should be managing this.  If you decide to let Oracle manage it, then use the Node Manager settings and disable option 1 and 3.  If you decide to let Microsoft handle it at the Service level, then disable 2 and 3 and so on.

Understand that if they are all left to manage on top of each other, you will have one timing out the start up while the another is still attempting to start and another notes it’s down and issues a restart.  If you wonder why it’s taking 15 minutes or more to start your OMS on Windows, I’ll bet money you trace out the session and you’ll find more than one process attempting to start or restart the poor thing in your logs.

Honesty dictates that we shouldn’t just blame a complex issue on any one contributor and realize that with added complexity comes the need for added skills to ensure that you have the best configuration to support the technology.  Taking the time to trace out and understand the issue will help make that happen.

I know others may want to know how to do this and I had challenges until Gagan Chawla, a teammate from Oracle, was kind enough to point out how to still get to this hidden, little gold nugget, so I’m posting it here for others!

Up till database plug-in 12.1.0.5, while in SQL Monitor SQL ID details page, you could click on a button called View Report and quickly view a large amount of valuable data about a SQL statement that had executed.  One of the sections in this report was binds, which listed what values were being passed for the bind variables.

If you are investigating a performance issue for the execution of a SQL statement, having bind values can give you a significant advantage.  It can tell you:

1. Is the value outside the min/max value on an existing histogram.
2. Do statistics lean towards another value being more prevalent.
3. Is the value passed in not in the correct format.
4. Does the value searched impact due it’s different from the values known and/or counts expected are off.

There are a number of other reasons, but to have this data and to have it easily accessible at your fingertips is very beneficial to the person trouble shooting.

Post the database plug-in, the feature is no longer where it once was.  From the SQL Monitoring, Monitored SQL Executions, if you were to a SQL ID of interest, you would then go to the SQL Details page.

There is a new report called “SQL Details Active Report“, but it doesn’t contain the bind values data.  This report is still very, very valuable:

It shows all the above data, along with a wait event vs. all resource usage graph at the bottom of the report.  You can save or mail the report and all it’s relevant data.  It would still be nice to have the previous report with the bind values that was once available from the details page and you can get to it, but you just need to make a few more clicks.

Go back to the main Monitored SQL Executions page and locate the SQL that you are interested in:

Bring your cursor to the status column for that SQL ID and double click.  This will take you the the Monitored SQL Executions, SQL Detail Page and on the right hand side, you will see the View Report button.

This button will bring you to the previous SQL ID Details report that includes the bind data.  Another thing to remember is that you must also be viewing a database that supports the feature, which means Oracle 11.2 or higher.

While enjoying the lovely Liverpool, UK weather at Tech14 with UKOUG, (just kidding about that weather part and apologies to the poor guy who asked me the origin of “Kevlar” which in my pained, sleep-deprived state I answered with a strange, long-winded response…. :)) a customer contacted me in regards to a challenge he was experiencing starting an agent on a host that was home to 100’s of targets.

        oracle_database.DB301.com - LOAD_TARGET_DYNAMIC running for 596 seconds
        oracle_database.rcvcat11 - LOAD_TARGET_DYNAMIC running for 596 seconds
        oracle_database.DB302.com - LOAD_TARGET_DYNAMIC running for 596 seconds
        oracle_database.B303.com - LOAD_TARGET_DYNAMIC running for 596 seconds
        oracle_database.DB304.com - LOAD_TARGET_DYNAMIC running for 596 seconds
        oracle_database.DB305.com - LOAD_TARGET_DYNAMIC running for 596 seconds
        oracle_database.DB307.com - LOAD_TARGET_DYNAMIC running for 596 seconds
        oracle_database.DB309.com - LOAD_TARGET_DYNAMIC running for 596 seconds
        oracle_database.B311.com - LOAD_TARGET_DYNAMIC running for 596 seconds

Dynamic property executor tasks running
------------------------------


---------------------------------------------------------------
Agent is Running but Not Ready

The output from the “emctl start agent” wasn’t showing him anything he didn’t already know, but I asked him to send me the output and the following showed the actual issue that was causing the Agent not to finish out the run:

MaxThreads=96
agentJavaDefines=-Xmx345M -XX:MaxPermSize=96M
SchedulerRandomSpreadMins=5
UploadMaxNumberXML=5000
UploadMaxMegaBytesXML=50.0
Auto tuning was successful
----- Tue Dec  9 12:50:04 2014::5216::Finished auto tuning the agent at time Tue Dec  9 12:50:04 2014 -----
----- Tue Dec  9 12:50:04 2014::5216::Launching the JVM with following options: -Xmx345M -XX:MaxPermSize=96M -server -Djava.security.egd=file:///dev/./urandom -Dsun.lang.ClassLoader.allowArraySyntax=true -XX:+UseLinuxPosixThreadCPUClocks -XX:+UseConcMarkSweepGC -XX:+CMSClassUnloadingEnabled -XX:+UseCompressedOops -----
Agent is going down due to an OutOfMemoryError

This host target was a unique environment in that it contained so many targets, especially database targets.  One of the reasons that the management agent was created and OEM processing removed from an internal database back-end process was to lighten the footprint.  As EM12c introduced numerous features that has assisted its direction towards the center of the Oracle universe, the footprint became heavier, but I’ve been very impressed with development’s continued investment into lightening that footprint, even when considerable additions with plug-ins and metric extensions are added.

With all of this, the server administrator may have a different value set to limits on resource usage than what may be required for your unique environment.  To verify this, I asked the customer to run the following for me:

ulimit -Su
ulimit -Hu

Which returned the following expected values:

$ ulimit -Su
8192
$ ulimit -Hu
3100271

The user limit values with these added arguments are to locate the following information:

-H display hard resource limits.
-S display soft resource limits.

I asked him to please have the server administrator set both these values to unlimited with the chuser command and restart the agent.

The customer came back to confirm that the agent had now started, (promptly!) and added the remaining 86 database targets without issue.

The customer and his administrator were also insightful and correctly assumed that I’d made the unlimited values not indefinitely, but as a trouble-shooting step.  The next step was to monitor the actual resource usage of the agent and then set the limits to values that would not only support the existing requirements, but allocate enough of a ceiling to support additional database consolidation, metric extensions, plug-in growth.

Last week’s release of 12.1.0.5 was a pleasant surprise for everyone out in the Oracle world.  This release hit the bulls-eye for another cloud target of Oracle’s, announcing the introduction of Enterprise Manager 12c’s offering a single pane of glass management of the hybrid cloud.  The EM12c team has been been trained and testing out the new features of this release with great enthusiasm and I have to admit, pretty cool stuff, folks!

Why is the Hybrid Cloud Important?

Many companies are still a bit hesitant to embrace the cloud or due to sensitive data and security requirements, aren’t able to take advantage of cloud offerings for their production systems.  Possessing a powerful tool like Enterprise Manager to help guide them to the cloud could make all the difference-

You’re going to start hearing the EM folks use the term, “Single Pane of Glass” a lot in the upcoming months, as it’s part of the overall move, taking Enterprise Manager from the perception that EM is still a DBA tool and getting everyone to embrace the truth that EM12c has grown into an infrastructure tool.

What is the hybrid cloud?

As we’ve discussed the baby-steps that many companies are taking, (vs. others that are jumping in, feet first! :)) with the hybrid cloud, the company can now uphold those requirements and maintain their production systems within on-premise sites, but enforce data masking and sub-setting, the sensitive data is never presented outside the production database, (including to the test master database that is used to track the changes in the snapclone copies…)  This then allows them with Database as a Service to clone development, test, Q&A environments to a less expensive cloud storage platform without exposing any sensitive data.

Once the datamasking or any other pre-clone data cleansing/subsetting is performed, then the Test Master database is created and can be used to create as many snap clones as needed.  These snaps can be used for development, QA or testing.  The space savings continues to increase as the snapclone copies are added, as the block changes are most of the space consumption in the test master database.  This can add up to a 90% storage savings over traditional database full copies.

Hybrid Cloning

The power of hybrid cloning is the Hybrid Cloud Gateway, a secure SSH tunneling, that allows seamless communication between on-premise systems and the cloud.

Types of Clones

There are four types of clones currently offered with Hybrid cloning-

• On-premise source cloned to the cloud.
• Cloud source, cloned to on-premise.
• Cloud source cloned in the cloud.
• Migrate from a schema in a database to a PDB in the cloud.

Simplicity is Key

The user interface is simple to engage, use to create a clone or clones, save off templates, build out a catalog to be used for a self-service portal and when cloning, the status dashboard is a great quick view of success on cloning steps:

If deeper investigation of any single step needs to be perforrmed, the logging is no different than inspecting an EM job log, (because an EM job is exactly what it is… :)):

I’ll be returning from Europe soon and hope to do more with the product, digging into this great new feature, but until then, here’s a great overview of 12.1.0.5’s brand new star!

The sales, support and technical teams were brought into the Denver Tech Center office to do some advanced training in Hybrid Cloud.  There were many take-aways from the days we spent in the office, (which is saying a lot-  most of you likely know how much I hate working anywhere but from home… :)) and I thought I would share a bit of this information with those that are aching for more details on this new and impressive offering from release 5.

If you’re new to Hybrid Cloud and want to know the high level info, please see my blog post on the topic.

Cloud Control Cloning Options

Cloud Control now includes a number of new options for database targets in EM12c.  These new drop down options include cloning to ease access to the new hybrid cloning.  Once you’ve logged into Cloud Control, go to Targets à Databases and then choose a database you wish to implement cloning features for.  Right click on the target and the drop downs will take you to the cloning options under Oracle Database –> Cloning.

There will be the following choices from this drop down:

• Clone to Oracle Cloud-  ability to directly clone to the cloud from an on-premise database.
• Create Full Clone-  Full clone from an image copy or an RMAN backup.
• Create Test Master- Create a read-only test master database from the source target.
• Enable as a Test Master- Use the database target as a test master, which will render it read-only and it would rarely be an option for a production database.
• Clone Management-  Manage existing cloning options.

Using a test master is essential for snap clones, which are a great way to offer great space savings and eliminates the time that is required for standard cloning processes.  The test master is in a read only mode, so it will need to be refreshed or recreated with an up to date copy, (which will then be another option in the drop down, “Disable Test Master”) for new cloning procedures.

Snapshot Clone

For the example today, we’ll use the following production database:

We’ll use an existing test master database to perform our clone from:

We can right click on the database and choose to create a clone.  This is going to be an artifact via a snapclone, so keep this in mind as we inspect the times and results of this process.

Upon choosing to create a snapshot clone of a pluggable database.  This will then create snapshot clone, each clone is just a copy of the file header with block changes involved on the read only or read-write clone.

Once you fill out the pertinent data for the clone, using the correct preferred credentials with SYSDBA privileges, name the new pluggable database, the name you’d like it displayed in  Cloud Control as and enter the PDB  administration credentials, password and confirm the password.  Once that’s done, choose if you’d like to clone it to a different container, (CDB) than what the source resides on and then add the database host and ASM credentials.

Once you click next, you have more advanced options to view and/or setup:

The advanced options allow you to change the sparse disk group to create the clone on, storage limits and database options.

You can then choose to have data masking to protect any sensitive data, but keep in mind, once you do so, you will no longer be using a snapclone due to the masking, but the option to implement it at this step is an option.  You can also set up any pre, post or SQL scripts that need to be run as part of the clone.  This could include resetting sequences, user passwords, etc.

The next step allows you to schedule the clone in the future or run immediately.

You can also choose the type of notifications, as this is simply an EM Job that is submitted to perform the cloning process.  Once you’ve reviewed the cloning steps chosen via the wizard, you can then submit.

Once the jobs been submitted, the submitter can monitor the job steps:

Success

Once the clone has completed, you can view each of the steps, including the time each took.

The source database was over 500 GB and was cloned in less than one minute!  You also will see the new cloned database in the targets list:

If curious, note that this is a fully cloned database that is on ASM, which you can view, just as you would for any other database.

Again, note the size and that this can be managed like any other database that you would have created via a DBCA template or through a standard creation process.

More to come soon and thanks to Oracle for letting us get our hands on the new 12.1.0.5 hybrid cloning!

I had a chance to take a closer look at the Zero Data Loss Recovery Appliance, (ZDLRA) via Cloud Control today and really liked the dashboard that’s included in the plugin for the product.  This post isn’t going to cover the installation of the plugin or configuration-  I was just intrigued by the dashboard and wanted to go over the great features that are part of it.

Accessing the ZDLRA

As with any Targets, the ZDLRA is going to show up in the All Targets menu and also has a target type filtered drop down in the Targets menu.

All ZDLRA targets discovered and configured will be listed under this view, (I’ve renamed mine to protect the innocent.

The reason for more than one ZDLRA is to ensure their is redundancy for disaster recovery, which you’ll see in the upcoming overview of the current environment we have set up for our High Availability Enterprise Manager environment.  If we double click on the first ZDLRA listed, we’ll come to the main dashboard, which we’ll dig into section by section, left to right.

ZDLRA Targets

As with all targets, there is the system target which can be expanded to show all the associated database type targets that make up the ZDLRA.

Below the database targets associations are the associated hardware targets, which also can be expanded to show that the ZDLRA is based off engineered system architecture.

Each of the associated targets are links that can take you to the home pages for each target shown in the list.

Backup Information

To the right of the target and associated target information, we start to view the high level information about what the ZDLRA handles-  backups.

At the bottom of the above graphic, notice that the ZDLRA is replicating copies of all backups to the ZDLRA2 to ensure that more than one copy of backups exist for DR purposes.

Summary

If you switch from the Current Activity to the 24hr historical view, then you can see a lot more information about the backups the ZDLRA is responsible for:

As you’ll see above, there are 7 databases that are currently being backed up to the ZDLRA.  You can click on “Backup”, which will then display more details about the backups:

You’ll now see the database targets that are backed up, the backup pieces, the amount of data received in GB’s and if available, the transfer rate for the backup.  As the backups are being replicated to the ZDLRA2, there aren’t any copies being pushed to tape, which we can also verify in the Replication section at the bottom.

We can click on the link for any of the Backup Pieces and it will then display the job task ID, the target, the database, backup information, elapsed time and if there are any errors, along with completion state.

Back on the main dashboard page, we can then inspect some of the backup load to the ZDLRA.  As you move your cursor across the graph, more information will be shown:

The blue displayed in the graph is the Backup Data Received on the first ZDLRA.  Highlighting a section on the graph will show the amount of data and date received.

Highlighting the green section in the graph will bring up a quick highlight of the amount of data replicated from the ZDLRA to the ZDLRA2, including date.  Orange is also shown for backups copied to tape, (if any are performed.)

Performance and Space Information

On the right side of the pane, we start with compliance information, can view the rate at which data is received to the ZDLRA and the send rate for replication to the ZDLRA2.

At the lower part of this section, we can view the storage information, including the recovery and reserved space.

Incidents and Events

At the bottom right, we have any incidents and events connected to just the ZDLRA targets.  This filter assists the recovery administrator to zero in on the incidents that are important to their responsibility role.  There’s no reason to go to the Incident Manager view and filter there, the data is already presenting what is important to them from this dashboard.

If we were to take the second incident down, “The total space requie to meet the recovery….” We can select this incident and dig into the details about the incident.

The general information, including thresholds and ability to investigate metric threshold history, etc. is displayed, just as for any incident.

The Events tab shows the history and event sequence as the thresholds were reached during the incident.

In the All Updates tab, you can see what rule set and that it’s a system generated rule, nothing created by an admin, this came as part of the plugin for space consumption and requirements.

You can then take a look at Related Events and see if any associated targets may be related to the incident, may have caused the incident, etc.

The ZDLRA dashboard within EM12c is clean, provides the important information that you need to know about the backups, any DR configurations/processing, target health and filtered incident management that makes it easy to manage for any DBA.  We all know, a DBA is only as good as their last backup and this makes it easy to know you’re good…:)

Happy Birthday to me!  So for my birthday, I give a present to you…  As I want all DBAs to sleep better at night, here are the top ten features you can use in Enterprise Manager Cloud Control to offer a good night’s rest instead of during the day at your desk…

1.  Disable the Default Rule Sets Shipped with Cloud Control.

Yes, you heard me.  I believe you should use them as a starting point or an example, but don’t put them into production.  These were examples set by development to see all that you could be notified on, but what you need to be woke up for should be anything mission critical that will SUFFER an outage if you DON’T respond.  Anything that can wait till the morning SHOULD wait till the morning.

Make copies of the default rules and disable the originals.  Plan on making as many copies and edits as necessary to ensure that you are only being notified on the appropriate targets, life cycle status and line of business that YOU are responsible for ensuring is up and available to the business.

2.  Implement Monitoring Templates and Default for Important Target Types.

Monitoring templates ensure that you are monitoring each target in the same way and for the same metric thresholds.  This ensures you start with metric thresholds that make sense for the target and should be applied to all targets of that target type.  Creating monitoring templates are easy when you create one target as an example and use it for the source of your template.

3.  Use Metric Thresholds for Individual Targets and Set Them to Not Be Overridden by Monitoring Templates

Now this might sound like a complete 180 from #2 on this list, but it’s not.  This is just like #1, break down and specialize for unique targets that have unique challenges.  This means, if you have a target backup drive that fills up to 97% each night, you shouldn’t be woke up for it.  This is expected behavior and you can either set a static threshold specific to this target or an adaptive threshold that won’t be overridden by the monitoring template for this target ONLY.

4.  Utilize Administration Groups

Administration Groups offer you advanced features and scalability to your Cloud Control environment that standard groups, and to a lesser extent, Dynamic groups, do not.  Line of business and life cycle management features that ensure you can break down notification groups, rule sets and other features, along with more advanced features with Database as a Service and other features to allow you to do more with less.  The natural life of a database environment is one of growth, so thinking ahead one, five and ten years is a great way to add value to the business as a database administrator.

5.  Create Metric Extensions for Unique Monitoring Scenarios

Enterprise Manager 12c is a self-service product.  So often there are unique situations that the business needs monitored for or the DBA notes creates a situation or outage, but isn’t, by default, a metric that comes with EM12c.  It’s easy enough to create a metric extension and take the concern and worry out of the situation, creating more value to the business.

6.  Add Corrective Actions, (Jobs)

Often when, a problem occurs, a DBA has a simple shell script or SQL they run and it corrects the problem.  If this is the case, why not have Cloud Control monitor for the issue, create an incident in the Incident Manager, send an email, then run the SQL or script as a Corrective Action?  The DBA will still know the problem occurred the next morning, but no one needs to be woke up to do what can be automated in the system.

7.  Use Patch Plans and Automate Patching

I understand, really.  Something could somehow, somewhere, some rare time go wrong, but the patch plans you can create in Enterprise Manager are surprisingly robust and full featured. If you’re still doing patching the old fashioned way and not patching environments in the more automated and global patch plan way, you’re wasting time and let’s face it-  DBAs rarely have time to waste.  You are a resource that could be utilized for more important tasks and quarterly PSU patching is just not one of those.

8.  Centralize Database Jobs to EM Jobs

The common environment is structured with multiple DBAs, often with one DBA as primary to a database environment and the others playing catch up to figure out how the primary has the database set up.  My favorite DBA to work with once told me, “Kellyn, love your shell scripts.  They make the world go ‘round.  I just don’t want to try to figure out how you write shell at 3am in the morning or what kind of scheduler is used on all the OS’s you support!”  I realized that I owed him to centralize all my environments with an interface that made it easy for ANYONE to manage it.  No one had to look at cron, the task scheduler or a third party scheduling tool anymore.  Everything was in Enterprise Manager and no matter what operating system, it all looked very similar with the logs in the same place, found in the same tab of the UI.  Think about it- this is one you do for the team, move those jobs to inside Enterprise Manager, too…

9.  Put Compliance Framework into Place

Compliance is one of those things that seem a mystery to most.  I’m often asked why environments really need it and does it make sense.  It can seem overwhelming at first, but the idea that you know what database environments, hosts and such are out of compliance helps to distinguish how to get your database environment all set up to ensure that business best practices are in place-  You have a baseline of compliance standards for configuration settings, installation and real-time monitoring to view globally via EM12c.

10.  Plug-ins to Offer a Single Pane of Glass View

A database is a database or that’s how the business sees it.  I have almost as many years in SQL Server as I do in Oracle.  I’ve worked in Sybase, Informix, Postgres and MySQL.  After being hired for my Oracle DBA skills in every job I’ve held, it never failed-  within 6 weeks, a mission critical database environment on a secondary database platform was discovered that a group, often outside of IT had implemented and now needed critical support of.  Enterprise Manager offers plug-ins to support all of the above database platforms and more.  It offers plug-ins for engineered systems, storage arrays and other hardware that the DBA is now expected to manage, too.  Why manage all of this from multiple systems when you can easily create a single pane to ensure you’re covered?

So there you have it, my top ten list.  There are, of course, 100’s of other great features in EM12c, but make sure you are taking advantage of these in the list!

More and more companies are consolidating environments.  Server sprawl has a high cost to any business and finding ways to consolidate to more powerful servers or to the cloud is a productive undertaking for any company.

The Consolidation planner in Enterprise Manager has been around for quite some time, but many still don’t utilize this great feature.

Creating a Consolidation Project

You can access the Consolidation Planner from the main Enterprise menu from Cloud Control in EM12c. There are two consolidation types currently available  physical to physical, (P2P) or physical to virtual, (P2V) consolidation.  For our example, we’ll perform a P2P, (physical) consolidation project.

Once you choose the option for your consolidation project, choose the servers you wish to add to your project.

The planner let’s you know the current resource usage of the hosts that you wish to include in your consolidation project.  This assists the administrator, giving a high level view if the servers are an option for consolidation.

If you scan to the right, you can view more information, including network and disk IO.

Note that the 07 host has high network I/O and capacity demands that will need to be taken into consideration if consolidating.  Once you are certain that you have all hosts that you wish to consolidate to a new host or engineered system, click on Next.

Choose the system you wish to consolidate to and the hosts will be displayed, along with current resource usage if already the home of databases.

As displayed to the right, we can see that the destination doesn’t contain the network waits that the 07 host is facing and so we can see that the potential candidates for consolidation are possible.

We can then click Next and choose the amount of workload data we would like to use for our comparison and if we would like to run the job now or schedule it:

Next you’ll decide if you wish to go with the default to include no pre-configured scenarios or if you would like to add these to enhance the results.

So for my example, I’ve added pre-configured scenarios, using the existing servers that I wish to consolidate:

Once satisfied, I can then click Next to review my consolidation project:

Review the project and if satisfied, click on Submit.

The Consolidation Project

The project is now in place and collecting data to ensure the most valid information is known for consolidation.

You can see that the project is in the process of collecting the data in the third column for each of the advanced scenarios I requested.  The job that collects all this data can be displayed from the Job Activity screen:

Back on the Consolidation Project screen, there are four tabs at the bottom that offer us information about our project.  This data can be viewed at anytime from the Consolidation console-

Notice that the amount of days that our review data is for hasn’t come close to being achieved, so this is shown in the General information tab.  We can also view the sources involved and see if any changes in resource usage has occurred:

We’ll need to return to this next tab after the data collections have occurred, but just to review, this is where we start to see how our consolidation data plays out:

We’ll stop here for now and revisit this in three weeks when the consolidation collection is finished and all the data is populated.  You now know how to create a project and next time we’ll go over how to use this data.