Saturday, April 26, 2014

This post will be based on the information concerning Exadata Administrators.. You will find information gathered on the way for being certified on Exadata..
The information provided will be item by item, and I think
this post will be useful for Exadata Admins even though they already have field experience..
This post will be unstructured,.I mean the information provided in this document is not grouped into subtitles, but it will present the whole picture about Exadata Administration, when you will read the all of it. The information provided in this document will be like snips from the Exadata knowledge..
In order to understand this documents, you should already know the basic concepts of Exadata.( like Storage indexes, smart scan and etc.. )

Let start;

You can make indexes invisible to increase the chance for using storage indexes.
You can load your data stored on the filtered columnd to maximize the benefit of storage indexes.
Bind variables can be used with storage indexes.

For media based backups, it s recommended to allocate equivalent number of channels and instances per tape drive.. For this type of backups, the network cables between Exadata and media servers should be connected through Exadata Database nodes.

Bonding on Exadata can be used both for reliability and load balancing, but when I look to a Production Exadata Machine , which is X3, I see that the bonding is configured for reliability.

active-backup or 1 -> Sets an active-backup policy for fault tolerance. Transmissions are received and sent out via the first available bonded slave interface. Another bonded slave interface is only used if the active bonded slave interface fails.

To guarentee proper cooling for an Exadata Machine, perforated floor tiles should be placed at the front, because the air flow is from front to back.

Creating multiple grid disks on a single disk in Exadata provides us multiple storage pools with different performance characteristics and multiple pools that can be assigned to different databases.

Here is a general information about Disk layout in Exadata
Physical disk -> Lun -> Cell Disk( its like a filsystem on Lun..) -> GridDisk ( it s like a partition)
Lastly Grid disks are served to the ASM diskgroups..
Note that , we can create flash based ASM diskgroups,too. They occupy may share space with the Flash Cache on the flash disks.

Things to consider for Exadata migrations,

For Transportable Database method-> Source database should be 10.2.0.5 and little endian
For Data Pump method ->, 10.2.0.5 is good, but it is time consuming..
For Data guard physical method ->, Source platform must be Linux, Solaris x86 or Windows and source must be on 11.2.

Standby Physical Standby:
-------------------------------------------------
Source platform must be Linux, Solaris x86 or Windows (see Document 413484.1) and source on 11.2
Source database in ARCHIVELOG mode
NOLOGGING operations to the database are not permitted, i.e. FORCE LOGGING is turned on at the database level

Source system must be 10gR2 (10.2.0.5) and little endian
Stricter service level requirements that obviate the required downtime with Oracle Data Pump

Transportable Tablespaces
-------------------------------------------------
Any source platform
EBS Release 12 with source database RDBMS release 10gR2 (10.2.0.5) or higher
EBS 11i with source database RDBMS release 11.2
Service levels cannot tolerate a significant outage. Using transportable tablespaces instead of Oracle Data Pump export/import can significantly reduce the outage time for large (> 300 GB) EBS databases. Thorough testing will determine the precise outage time.
For a point of reference, tests on an Oracle Exadata Database Machine quarter rack with the Rapid Install Vision database (about 300 GB) took about 12 hours. This time should remain about the same regardless of the amount of data in the database. This is because the metadata creation takes the longest time in the migration process and accounts for the bulk of time.

Oracle Data Pump
-------------------------------------------------
Any source platform
Source database is RDBMS release 10.2 or higher
To implement Oracle Exadata Database Machine best practices on the target
Service levels can tolerate a significant outage.
For a point of reference, tests on an Oracle Exadata Database Machine quarter rack with the Rapid Install Vision database (about 300 GB) took about 24 hours (export - 7:42; import - 16:42) using Network Storage and no dump file copy (i.e. the export dump storage was mounted on the source and the target).
Timings will vary depending on your system configuration and increase as the amount of data increases.

In Exadata, Oracle Enterprise Agents must be deployed to the compute nodes. .Oracle Exadata Plug-in deployed with the Agent. Plugins allow you to monitor the following key components of Exadata machine. There are several plugins for Grid Control and Cloud Control (such as Avocent MergePoint Unity Switch, Cisco switch, Oracle ILOM, Infiniband switch, PDU) .. Note that : A trap forwarder is required to catch cisco switch and kvm traps due to a port mismatch..

Exadata -> Action plan to replace a Flash disk when Griddisks are created them,

Ref: Replacing FlashCards or FDOM's when Griddisks are created on FlashDisk's (Doc ID 1545103.1)

If the flash card needs to be replaced, drop the disks used in +ASM for the FlashDisk
Drop FlashCache / Flashlog and delete celldisks of type Flashdisk
Shutdown
Replace
create flashdisks
Now create your griddisk's back on DiskType: FlashDisk
add back the disks to the diskgroup. (it is optional ->If you used 'force' when dropping the disks from ASM then Exadata auto management should automatically add these disks back into ASM.)

Note that, we have different failure types in Exadata. For example a disk which is in predictive failure state must be replaced immediately. ASM will drop these kind of disks automatically from the associated disk group , and start a rebalance operation.
For flash a flash disk also; if the disk is in predictive failure then ->
If the flash disk is used for flash cache, then flash cache will be disabled on this disk thus reducing the effective flash cache size. If the flash disk is used for flash log, then flash log will be disabled on this disk thus reducing the effective flash log size. If the flash disk is used for grid disks, then Oracle ASM rebalance will automatically restore the data redundancy..

If you want to change a memory dimm or want to make a similar activity, you just need to shutdown the affected cell.. The database, and the other cell servers will not be affected from this operation.
What you have to do is;
Ensure asmdeactivityoutcome of the Griddisks first..
Then inactive all the griddisks on that cell and ,shutdown the cell using shutdown -h now ..

quartery battery learn cycle is a normal maintanence activity.. It is used for charging the batteries. When this kind of activity is happening, you can end up with a performance degregation in terms of I/O, as this event may require the related device to be put in the write through mode.. Keep that in mind and, If you can or need, set the device back to write back mode...

Configure and use smart flash logs if you need a better LGWR performance, as LGWR writes redo data both flash and disk in parallel.. It considers whichever of these writes completes first, as done..
Smart Flash Logs are a new feature comes with 11.2.2.2.4 cell software. They are not for reading, they are used like a circular buffer for Redo writes.. Smart Flash logs can enhance the performance of an OLTP database. By default they occupy 512 MB per cell. (32 MB size per flash disk *16 flash disk) space in the Flash Cards.. So Smart Flash logs reduce the size of Flash Cache in a manner.
Note that , Flash Smart logs can be enabled or disabled using IORM according to the databases if needed..
Also LGWR writes and Controlfile IO will be automatically on high priority in IORM.. On the other hand, DBWR IO is managed automatically at normal priority.
Flash logs if needed somehow, can be dropped using drop flashlog command through the cellcli utility residing on the storage servers..

Consider using CTAS for bulk data loading from external tables to Exadata.. CTAS automatically uses direct path load.. Insert /*+append*/ can also be used for this kind of data loading, as by using the append hint, oracle will use direct path loading in insert operations, too..

cellip.ora is the configuration file , if you want to separete cells which are connected by the asm instances. This file is located on where ASM resides(compute node), and it basically tells ASM which cells are available. cellip.ora is located in every compute node, and its contents are like following;

In addition, you can connect any media servers which have infiniband cards, to Exadata via infiniband. Then you can connect those media servers to tape libraries to maximize the tape backup thorughput..
In terms of tape backups, Oracle docs suggest to have Disk-to-disk-to-tape, in other words D2D2T strategy, which allows keeping old backups on tape while retaining new/fresh backups on disks for achieving fast recovery times. You can consider the following Oracle Slide as a good Exadata-tape backup scenario;

Uncommited transactions and migrated rows can cause cell single block physical reads even if you are doing a Full table scan. Note that: Single block reads and Multi block reads may benefit from the Flash Cache,
Also note that, smart scan can not be done against Index Organized Tables and clustered tables.

When you need to apply a bundle patch to Oracle Homes in Exadata, you will need to use oplan utility.

oplan utility generates instructions for applying patches, as well as instructions for rollback. It generates instructions for all the nodes in the cluster. Note that, oplan does not support DataGuard .. Oplan is supported sincerelease 11.2.0.2 . It basically eases the patching process, because without it you need to read Readme files and extract your instructions yourself..

Again, choose the rollback strategy according to your needs and follow the patching instructions to rollback the patch from target.

It is mandatory to know which components/tools are running on which servers on Exadata;

Here is the list;

DCLI -> storage cell and compute nodes, execute cellcli commands on multiple storageservers ASM -> compute nodes -- it is ASM instance basically RDBMS -> compute nodes -- it is Databas software MS-> storage cells, provides a Java interface to the CellCLI command line interface, as well as providing an interface for Enterprise Manager plugins. RS -> storage cells,RS, is a set of processes responsible for managing and restarting other processes. Cellcli -> storage cell, to run storage commands Cellsrv ->storage cell. It receives and unpacks iDB messages transmitted over the InfiniBand interconnect and examines metadata contained in the messages Diskmon -> compute node, In Exadata, the diskmon is responsible for:,Handling of storage cell failures and I/O fencing ,Monitoring of Exadata Server state on all storage cells in the cluster (heartbeat),Broadcasting intra database IORM (I/O Resource Manager) plans from databases to storage cells,Monitoring or the control messages from database and ASM instances to storage cells ,Communicating with other diskmons in the cluster.

The following strategy should be used for applying patches in Exadata:

Review the patch README file (know what you are doing)Run Exachk utility before the patch application (check the system , know current the situation of the system)Automate the patch application process (automate it for being fast and to minimize problems)Apply the patchRun Exachk utility again -- after the patch applicationVerify the patch ( Does it fix the problem or does it supply the related stuff)Check the performance of the system(are there any abnormal performance decrease)Test the failback procedure (as you may need to fail back in Production, who knows)

Multiple Grid disks can be created on a single Cell Disks, as you know. While creating multiple Grid disks on a Single Disk, you can end up having multiple disks which have different performance characteristics . In order to have more balanced disk layout, you can use interleaving options or Intelligent Data Placement technology which is based on ASM..

The internal Infiband network used in Exadata transmits IDB messages between compute nodes and storage servers as well as Rac interconnect traffic packets between the compute nodes in the cluster.

Dbfs or a Nfs filesystem can be used as a stage for loading data from the external tables. If choosing Nfs to load data, the Nfs share should be mounted to the preferred compute node..
Dbfs can be created on DBFS_DG , as well as on a standart filesystem..
It can enhance performance if you need to bulk load data in to your Production System , which resides on Exadata.. By using DBFS created on ASM, you will have a parallelization in storage, which will enhance perfomance of IO.

Diagget.sh script can be used to gather the diagnostic information with software logs, trace files and OS information..

Exadata storage server has alerts defined on it by default.. This alerts are based on some defined metrics.. We can define new metrics also, this metrics will persist accross cell node reboots.

If you have a 11.1.0.2 database(little endian) and want to migrate it to Exadata with minimum downtime, you can upgrade it to 11.2.0 and use Data pump physical to minimize downtime or alternatively, you can use Golden Gate for this. Ofcourse you can use datapump, as well, but your with datapump the downtime will be significant..

According to the Non-interleaving disk configuration in Exadata(which is default), we can say that the first Grid diskdisks created using Create Griddisk... command will have the best performance. So the Diskgroup created based on the first created griddisk will have better performance than the other Diskgroup on the Same Exadata Machine. ...

If you want to do some administrating stuff on Exadata Storage servers, like dropping the cell disk and etc;
you can use celladmin OS account in storage servers to do this kind of an operation. You can use cellcli(on every cell) or dcli(from one cell to administer all of the cells) utility to execute your commands..
DCLI is a pyhton script. It is used to execute command on the cells remotely without login in to them.. (In first execution it create ssh keys with the following -> dcli -k -g mycells.)

In OLTP systems,
Exadata write-back flash cache can enhance performance , as it also caches database block writes.
Also Flash log can be useful for enhanching perfomance of OLTP systems, as fast response time for Log Writer is crucial in OLTP..
For Big OLTP systems, Flash cache and Flash log can enhance performance and High Capacity Disks in Exadata can meet the storage size needs..

Note that , IDP, IDB and IORM manager can only seen on an Exadata Environment.

IORM is used for managing the Storage IO resources.

Here is a diagram for the description of the architecture of IORM. (reference: Centroid)

So we can manage our IO resources based on the Categories, Databases and consumer groups. There is a hierarchy as you see in the picture.. The hierarchy used to distribute I/O.
IORM must be used on Exadata if you have a lot of databases running on Exadata Machine.. IORM is a friend of consolidation projects, in my opinion..

If you configured ASR manager in your environment, note that : database uses SNMP to transfer the notifications from database to ASR Manager and these notifications are forwarded using SNMP from ASR to Enterprise Managern In addition Faults are tranferred to the Oracle securely using https.

The fault telemetry sent from ASR manager is in the below format;

Telemetry Data:

System-id: System serial number.
¦ Host-id: hostname of the system sending the message.
¦ Message-time: time the message was generated.
¦ Product-name: Model name of the server

As known, Exachk is the utility to validate an Exadata Environments. We check the system using this utility time to time(before patches, after patches and etc..) Also, it can be scheduled to run regularly.. To schedule exachk, we can create a cron job or we can create a job in Enterprise Manager..

Compression on Exadata can only be done on the Compute nodes.. Decompression, on the other hand, can be done on Compute Nodes or Storage Cells.. Decompression can be done on Storage Cells if the associated operation is based on the Smart Scan.. Same rule applies for Encryption and Decryption, too..

If your index creation takes long time in Exadata, consider the following information:
Cell single block physical read can impact the peformance of an index creation activity. Cell single block physical reads are like db file sequential reads in a tradional system.
Migrated and chained rows can cause cell single block read events on an Exadata Machine. Also uncommited rows during a query is handled based on the consistency.. For supplying the consistency, Database nodes may require additional blocks. These blocks are sent by the Cell servers to the database nodes.. This activity can cause cell single block physical reads , as well. If you have a lot of blocks in one of these conditions, then your index creation time can take long time..

When migrating to Exadata, you should consider Database Type(oltp or warehouse), size of the source database, the version of the source database and the Endian format of the Source Operating Systems.. By analyzing these inputs you can choose an optimal migration method and strategy.

In Exadata Enterprise Manager monitoring, the communication flow from the ILOM of the Storage Servers to Enterprise Manager is through the Storage Server's MSprocesses. ILOM sends data using snmp to MSprocess, and MSprocess send the data to the Enterprise Manager using snmp. Data is triggered and transfered through the snmp traps.
Based on the preset thresholds defined in ILOM, we can monitor motherboard,memory,power, and network cards of Database Nodes using Enterprise Manager. We can see the faults and alerts produced for these hardware components from Enterprise Manager.

Oracle Auto Service Request (ASR) is a secure, scalable, customer-installable software feature of warranty and Oracle Support Services that provides auto-case generation when common hardware component faults occur. ASR manager can be installed on an external Oracle Linux or Oracle Solaris server. Also you can use one of Exadata db nodes for installing ASR manager (not preferred).

ASR manager communicates with Oracle using https..

In Exadata, some database work can be offloaded to Storage Servers as you know.. Besides operations like full table scan, single row functions and simple comparison operators, some joins can be offloaded to storage. Column filtering, Predicate filtering and Virtual Column filtering are the things that can be offloaded to the Exadata Storage Servers, as well.
If you want to see all the functions that can benefit from smart scan; you can use the following sql:
select name from v$sqlfn_metadata
The output will be like;

Operations like full table scan and fast full index scan executed in parallel always generate Smart scans.. For full table scan, you can see cell smart table scan, and for fast full index scan, you can see cell smart index scan events.. Fast full index scan operations can be executed through the smart scan because in fast full index scan, Oracle just reads the index block as they exist on the storage.. I mean a bulk read will be performed, and that makes Oracle to use smart scan even if it is an index operation. Smart scan is performed during Direct path read operations.. Parallel queries use direct path read, that 's why they make use of smart scans..

So if we need to gather all together; in order to have smart scans, we need to execute queries in parallel, we need to use direct path reads towards the process memory and we need to have cell.smart_scan_capable parameter set to TRUE for our ASM diskgroups.

We have Sun servers in Exadata.. Compute nodes and storage servers are acutally Sun Servers. They have ILOM cards on them. These ILOM cardscan be used to administer these servers remotely.. For Example ILOM can be used to power-on database servers or open a remote console for a Storage Server.

Note that, if you want to check the status of all ports located in the Infiniband Switch, you can use Enterprise Manager or ibqueryerros.pl script located in the Infiniband switch.

To properly shutdown the Exadata, we need to first stop the database and grid services. We may use crsctl stop cluster -all command for this. Then we need to shutdown Exadata storage servers first. Then we need to shutdown Database servers. Lastly we need to power off the network switches and cut the power using power switches on PDUs. Why firstly shutdown storage servers? ( this is still a mystery for me)

After Exadata migrations, analysis are made to drop the indexes . This activity is done to increase the chance of using smart scans on our query, but care must be taken while dropping those indexes. For example, in an OLTP system we need fast response times for single block reads, so dropping an index may result a negative performance impact for some of our OLTP queries.. So it s better to drop an index after analyzing the queries of the corresponding application. You can use invisible indexes to see the difference .. You can check Execution plans to see if Oracle wants to use smart scan instead of an index access for a query.. According to your analysis, make the decision to drop the unnecessary indexes..

As you know Exadata contains Compute Nodes and Storage Nodes.. For Storage Nodes, you dont have the choice to use a different OS than Linux. Exadata Storage Servers are Linux, and will continue to be Linux. Oracle Linux servers are coming with Uek kernels.
On the other hand, you can choose to have Solaris 11 rather than Linux for compute nodes. Operating systems are selectable at install time.

Note that, In Exadata we have different networks, like Management network and public network, infiniband network.

Following is a picture representing those networks; (it is for X4 actually, but it is useful)

Ssh, for example , works from the Management network, as it is an utility to manage the corresponding servers. If you want to change the network that ssh listens from, you can use ssh config file to do that.

Following inputs can be used for configuring the Exadata Machine at install time..

In order to actually have the Exadata Storage Servers send notifications via email (or alternately SNMP) each of the servers has to be configured with the appropriate settings. This is done using the ALTER CELL command in CELLCLI.

ALTER CELL smtpServer='mailserver', -

smtpFromAddr='exacel@blabla.com, -

smtpPwd='email_password', -

smtpToAddr='erm@blabla.com', -

notificationPolicy='critical,warning,clear', -

notificationMethod='mail'

The alerts may be stateful and stateless, as well..

If the alert is based on a threshold, it gets cleared automatically when it no longer violates the threshold;

For example; a filesystem alerts , as follows;

CellCLI> list alerthistory detail
name: 4_1
alertMessage: "File system "/" is 82% full, which is above the 80% threshold.
Accelerated space reclamation has started.
This alert will be cleared when file system "/" becomes less than 75% full."

Also alerts can be fired for Critical, Warning and Information purposes..

We have Storage indexes located in the physical memory of the Storage Servers.. Storage indexes are maintained automatically the cellsrv and they are not persistent across reboots..(as they are in memory).. Cellsrv build these indexes based on the filter columns of the offloaded queries.
In Storage indexes, Oracle builds range regions based on the column values. By the use of storage indexes Oracle can easily find where to look in the storage for a given column value..
Maximum 8 columns for a table are indexed per storage region, but different storage regions can have different columns indexed for the same table.

But for example, if we want to upgrade the storage server software, we need to use patchmgr utility..the patchmgr utility is a tool Exadata Database Administrators use to apply (or rollback) an update to the Oracle Exadata Storage Cell.

Here is another restrictions is ;

Oracle Exadata Storage Server Software and the operating systems cannot be modified, and customers cannot install any additional software or agents on the Exadata Storage Servers.

Lastly, I will mention about placing multiple Exadata Machines in a System Room/Data Center.

If you have multiple Exadata Database machines, you need to place them side by side while ensuring the exhaust air of one rack does not enter the airinlet of another. If you have multiple clusters running on several Exadata Machines, you can place the racks that are part of a common cluster together/side by side..

33 comments
:

I'm referring the above post to clear my Exadata (1Z0-027) Certification. I'm worried now because X4 Administration also got added in the same Certification. Now total there are 90 questions in 1Z0-027 Certification. Tomorrow is my exam date. Can you please tell me what was your score for this exam ?

I know, the exam has changed recently. X4 specific questions have been added. So you need to study for Exadata X4 , too. You need to refer DOCUMENTATION FOR EXA 11.2 & 12.1, It is on Oracle Support, it comes with Patch 10386736.

Hi Erman,official recommendation for tape-based baclup configurations goes like this:Configure one RMAN channel per tape drive and add tape drives to scale backup ratesthough this post is not accurate and may be misleading:'For media based backups, it s recommended to allocate equivalent number of channels and instances per tape drive..' Following this recommendation I can allocate more than 1 channel per tape which is contrary to BP:http://www.oracle.com/technetwork/database/features/availability/maa-tech-wp-sundbm-backup-11202-183503.pdfDo you agree?Any references supporting your statement?

I totally agree with all that you've just said.It's just a matter of how you understand of what you read.I saw the simialr answer for the question:'Which two are Oracle recommendations for media based backups performed for a databaserunning on a Database Machine?'while preparing for the exam.There is no word 'one rman channel' in this context, so equivalent number of channels and instances for me meansthat number of channels should match number of instances (per tape drive). It is not said here that one rman channel should be equivalent one tape drive.Besides if your statement (answer) is correct there will be no option to choose another 100% correct answer for this question.I think the problem is that answer is poorly formulated.

I understand what you mean. Okay. But this blogpost is not an exam review. The things written in here, is just some bunch of information which I have gathered while preparing the exam. The purpose of this blog post (also this whole blog) is to enhance te technical knowledge of mine and the ones who follow this blog. Thanks for you interest , really. I m glad for you to spend time reading and making comments about this post. The true and more understandable statement is then "allocate one RMAN channel to one tape drive." , "allocate two RMAN channel to two tape drives, one for each" and so on."

Yes, bonding can be be used both for improving performance and reliability.In this post, I wrote "it improves reliability in Exadata", because in version X3, bonding was active-backup by default.

When channel bonding is configured for the client access network during initial configuration, the Linux bonding module is configured for active-backup mode (mode=1). If a different bonding policy is preferred, then you may reconfigure the bonding module after initial configuration.,

For example: In case of Infiniband; In the default deployment of the Exadata X4-2 Database Machine, the InfiniBand bonding is set up in the active/active mode, on all database servers and all storage cells. Earlier models of the Exadata Database Machine (V2, X2-2, X3-2) have the InfiniBand set up in the active/passive mode.

(Oracle Exadata Database Machine - Backup & Recovery Sizing: Tape Backups : FIGURE 1 RMAN BACKUP USING INFINIBAND CONNECTIVITY AND ORACLE SECURE BACKUP)Looking at the best practices, MAA makes Exadata compute nodes to connect to Media servers using Single Infiniband QDRs.Each compute nodes uses 1 QDR to connect to IB switch. From IB switch to the Media servers, there are 2 QDR used per media servers. (active-active mode) I didnt understand this one. Why using 2 QDRs between IB switch and media servers? If the answer is reliability then why dont they use multiple QDRs between computenodes and the IB switch.. Strange.Anayways;continuing.. In MAA the Media Servers connect to the Tape Devices using 4 Fibers.each Media servers connects to the tape drives using 4 fibers (in active active mode..) I think this is for increasing fibers throughput...

The bonding and backup configruration is actually It depends on your needs..In the same paper there is another scenario.Example Scenario 2: Gigabit Ethernet

As you may see; there are some calculation there.. Throughput calculation.. According to these calculations:For example: Customer wants 4 hrs to backup 6TB So it makes @ 2TB/hr = 560MB/secAccording to this info, the configurations is done.

Hi Sir, Please, I am a bit confused about Oplan. I know Oplan is used to generate instructions specific to your environment. what about if you use RAC. Oplan will generate instruction for patching RAC oracle home and grid GRID infrastructure home or it will generate instructions for patching all the components on the database machine.

OPlan is a utility that facilitates the patch installation process by providing you with step-by-step patching instructions specific to your environment for both in-place and out-of-place patch installation. Currently, OPlan is supported for Exadata Database Bundle Patches. For the latest information, see "Oracle Software Patching with OPLAN" in My Oracle Support Note 1306814.1

crsctl stop cluster -all on database server Then we need to shutdown Database servers FirstThen shutdown Exadata storage servers Lastly we need to power off the network switches and cut the power using power switches on PDUs.

"crsctl stop cluster -all on database serverThen we need to shutdown Database servers FirstThen shutdown Exadata storage serversLastly we need to power off the network switches and cut the power using power switches on PDUs."

<<>>>>

Doc ID 1093890.1 is also correct and a practical way of doing it, as you prefer to SSH to cells from compute nodes and then carryout your tasks.

I believe it doesn't matter as long as cluster services are shutdown properly.

How to Replace a 10G Ethernet Card on an Exadata Database Node (X3-2/X4-2) (Doc ID 1672412.1)It is for replacing the ethernet card, but look what it says:Pull out any SFP modules installed in the original 10G Ethernet card and install them into the replacment 10G Ethernet card making sure to orient them properly so that they seat fully into the card.