Oracle Blogs | Oracle All Things SQL BlogOracle Blogshttps://blogs.oracle.com/sql/compendium/rss
Thu, 21 Mar 2019 17:59:06 +0000FeedCreator 1.7.3Announcing the Winners of the 2018 Oracle Dev Gym Championship for Database Designhttps://blogs.oracle.com/sql/announcing-the-winners-of-the-2018-oracle-dev-gym-championship-for-database-design
<p>On Thursday 14 February, 37 plucky contestants took part in the annual Database Design Championships over on <a href="https://devgym.oracle.com">Oracle Dev Gym</a>.</p>
<p>This competition tested the players knowledge of Oracle Database 18c in a series of five quizzes. These covered <a href="https://blogs.oracle.com/sql/how-to-create-alter-and-drop-tables-in-sql#create-temporary">temporary tables</a>, <a href="https://blogs.oracle.com/sql/how-to-create-alter-and-drop-tables-in-sql#create-external">external tables</a>, <a href="https://docs.oracle.com/en/database/oracle/oracle-database/19/vldbg/index.html">partitioning</a>, <a href="https://oracle-base.com/articles/12c/identity-columns-in-oracle-12cr1">identity columns</a>, and <a href="https://blogs.oracle.com/sql/how-to-store-query-and-create-json-documents-in-oracle-database">JSON in the database</a>.</p>
<img alt="" src="https://cdn.app.compendium.com/uploads/user/e7c690e8-6ff9-102a-ac6d-e4aebca50425/6a8a9853-c115-4f9b-b14e-7eee344d2c98/Image/279b6f7e9f015192ce940cfeba5f79ab/gratisography_275h_small.jpg" style="width: 2160px; height: 1440px;" />
<p style="text-align: center;">Ryan McGuire <a href="https://gratisography.com/">Gratisography</a></p>
<p>The results are now in, so it&#39;s time to announce the winners!</p>
<ol>
<li><a href="https://devgym.oracle.com/pls/apex/f?p=10001:26:::NO:RP,26:P26_USER_ID:4294">Iudith Mentzel</a> of Israel (86.5% correct)</li>
<li><a href="https://devgym.oracle.com/pls/apex/f?p=10001:26:::NO:RP,26:P26_USER_ID:29400">Sartograph</a> of Germany (82.7% correct)</li>
<li><a href="https://devgym.oracle.com/pls/apex/f?p=10001:26:::NO:RP,26:P26_USER_ID:3765">Chad Lee</a> of United States (80.8% correct)</li>
</ol>
<p>These three were head-and-shoulders above the rest, being the only players to clear eighty percent right. And huge kudos to to Iudith for taking the top spot for the second year in a row!</p>
<p>And, once again, a massive thanks to Elic, our quiz-reviewer supreme. His keen eye ensured that all the choices were accurate.</p>
<p>Think you know more about Oracle Database?</p>
<p>Find out in the <a href="https://devgym.oracle.com/pls/apex/f?p=10001:201:::NO:201::">weekly tournaments on Oracle Dev Gym</a>. These cover SQL, PL/SQL, and Database Design. Or if you&#39;re looking for something a little different, we also offer weekly ranked quizzes on Java and Logic.</p>
<p>We hope you can join us!</p>
<p>The complete rankings are:</p>
Rank
Name
Total Time
% Correct
Total Score
1
mentzel.iudith (5)
33 m
87%
6066
2
Sartograph (3)
31 m
83%
5825
3
Chad Lee (5)
29 m
81%
5730
4
NickL (4)
31 m
79%
5625
5
li_bao (5)
24 m
79%
5601
6
Andrey Zaytsev (5)
34 m
79%
5513
7
JeroenR (4)
30 m
75%
5276
8
Karel_Prech (4)
34 m
75%
5212
9
siimkask (5)
15 m
71%
5139
10
Maxim Borunov (5)
33 m
73%
5116
11
PZOL (4)
34 m
71%
5012
12
JustinCave (5)
26 m
71%
4995
13
Jan &Scaron;er&aacute;k (5)
31 m
71%
4974
14
seanm95 (5)
20 m
69%
4919
15
Sandra99 (5)
33 m
69%
4864
16
Chase Mei (5)
22 m
67%
4860
17
Aleksei Davletiarov (3)
23 m
69%
4858
18
NielsHecker (5)
32 m
67%
4821
19
Michal P. (5)
33 m
67%
4665
20
mcelaya (3)
30 m
65%
4579
21
Joaquin_Gonzalez (5)
24 m
65%
4550
22
Hertha Rettinger (2)
20 m
63%
4469
23
Rytis Budreika (5)
08 m
62%
4365
24
Talebian (2)
21 m
62%
4365
25
Ivan Blanarik (5)
20 m
62%
4317
26
Mike Tessier (2)
24 m
60%
4200
27
whab@tele2.at (3)
18 m
60%
4125
28
msonkoly (3)
34 m
60%
4112
29
RalfK (2)
24 m
60%
4051
30
richdellheim (2)
33 m
58%
3967
31
umir (4)
34 m
58%
3910
32
MarkusId (1)
27 m
56%
3890
33
Sachi (3)
17 m
54%
3881
34
Stelios Vlasopoulos (5)
34 m
58%
3860
35
Cor van Berkel (2)
34 m
54%
3710
36
K&ouml;teles Zsolt (1)
30 m
50%
3230
37
swesley_perth (3)
03 m
17%
1087
Mon, 25 Feb 2019 14:40:12 +0000https://blogs.oracle.com/sql/announcing-the-winners-of-the-2018-oracle-dev-gym-championship-for-database-designChris SaxonHow to Store, Query, and Create JSON Documents in Oracle Databasehttps://blogs.oracle.com/sql/how-to-store-query-and-create-json-documents-in-oracle-database
<p>JavaScript Object Notation (JSON) is a lightweight data transfer format. It&#39;s the de facto standard for document exchange.</p>
<p>So it&#39;s likely you&#39;ll want to send and receive JSON documents from and to your database. And store them in your tables. Oracle Database has a huge amount of functionality that makes this easy.</p>
<p>In this post you&#39;ll learn how to:</p>
<ul>
<li><a href="#store-json">Store JSON in a database table</a></li>
<li><a href="#query-json">Query a table for rows storing particular values within a JSON document</a></li>
<li><a href="#index-json">Create indexes to find these rows efficiently</a></li>
<li><a href="#generate-json">Generate JSON from relational rows and columns</a></li>
</ul>
<p>Let&#39;s get started!</p>
<img alt="" src="https://cdn.app.compendium.com/uploads/user/e7c690e8-6ff9-102a-ac6d-e4aebca50425/f4a5b21d-66fa-4885-92bf-c4e81c06d916/Image/3d7f43ad2e53b5466d2e28cff5a646a2/json.png" style="width: 512px; height: 333px; display: block; margin-left: auto; margin-right: auto;" />
<a name="store-json">How to Store JSON in Your Database</a>
<p>JSON documents are just text. So to store them all you need is a column that holds this.</p>
<p>Before we get into the details, a bit of a rant:</p>
<p>The <a href="https://json.org/">JSON homepage</a> describes it as:</p>
<p><em><b>JSON</b> (JavaScript Object Notation) is a lightweight <strong>data-interchange format</strong>. It is easy for humans to read and write. It is easy for machines to parse and generate. It is based on a subset of the <a href="http://javascript.crockford.com/">JavaScript Programming Language</a>, <a href="http://www.ecma-international.org/publications/files/ecma-st/ECMA-262.pdf">Standard ECMA-262 3rd Edition - December 1999</a>. JSON is a text format that is completely language independent but uses conventions that are familiar to programmers of the C-family of languages, including C, C++, C#, Java, JavaScript, Perl, Python, and many others. These properties make JSON an ideal <strong>data-interchange language</strong></em></p>
<p>Notice how data-interchange appears twice? And storage appears, well... never!</p>
<p>That&#39;s because JSON is primarily for <em>transferring</em> data between systems.</p>
<p>Personally I think traditional relational tables should be the default choice for data storage. Taking JSON input and it storing as-is in your database should be the last resort.</p>
<p>But I know that you may just want to dump JSON in your database and be done with it. So let&#39;s review how.</p>
<p>As JSON is text, the obvious choice for storing it is a VARCHAR2 for small documents. I.e. those under 4,000 bytes. Or 32,767 bytes if you have enabled <a href="https://oracle-base.com/articles/12c/extended-data-types-12cR1">extended data types</a>.</p>
<p>But if you&#39;re storing bigger documents, we recommend going for BLOB.</p>
<p>Why not the text-based CLOB you ask?</p>
<p>This is because BLOB stores the document as-is, avoiding character set conversions. It also means the documents may use less storage.</p>
<p>The downside of this is you need to do data conversions when you save or retrieve JSON in BLOBs. Luckily, as we&#39;ll see later, this is easy to do.</p>
<p>So to <a href="https://blogs.oracle.com/sql/how-to-create-alter-and-drop-tables-in-sql">create a table</a> that will store &quot;large&quot; JSON documents, run:</p>
create table departments_json (
department_id integer not null primary key,
department_data blob not null
);
<p>Regardless of which data type you use, you&#39;ll want to verify that you&#39;re storing real JSON. And not any old junk. To do this, add the IS JSON check constraint, like so:</p>
alter table departments_json
add constraint dept_data_json
check ( department_data is json );
<p>Now you&#39;ve got the table, it&#39;s time to start adding documents!</p>
<a name="insert-json">How to Insert JSON in the Table</a>
<p>You can now save a JSON document with a standard insert. If you&#39;ve used a character data type, just whack it in. If you&#39;ve gone with BLOB, add text-to-binary conversion:</p>
insert into departments_jsonjson
values ( 110, utl_raw.cast_to_raw ( &#39;{
&quot;department&quot;: &quot;Accounting&quot;,
&quot;employees&quot;: [
{
&quot;name&quot;: &quot;Higgins, Shelley&quot;,
&quot;job&quot;: &quot;Accounting Manager&quot;,
&quot;hireDate&quot;: &quot;2002-06-07T00:00:00&quot;
},
{
&quot;name&quot;: &quot;Gietz, William&quot;,
&quot;job&quot;: &quot;Public Accountant&quot;,
&quot;hireDate&quot;: &quot;2002-06-07T00:00:00&quot;
}
]
}&#39; ));
<p>If you&#39;ve added the IS JSON constraint, the database will also reject anything that&#39;s not real JSON:</p>
insert into departments_json
values ( 100, utl_raw.cast_to_raw ( &#39;Random junk&#39; ) );
ORA-02290: check constraint (CHRIS.DEPT_DATA_JSON) violated
<p>Great. But what if you want to edit part of a document saved in your table?</p>
<a name="json-mergepatch">How to Update JSON with SQL</a>
<p>Let&#39;s say there&#39;s a big corporate re-org, merging the Finance and Accounting departments. Now imaginatively titled &quot;Finance and Accounting&quot;.</p>
<p>So you need to change the name of the Finance department.</p>
<p>Pre-19c, to change part of a JSON document, you had to replace the whole thing. Leading to a big, clunky update like this:</p>
update departments_json
set department_data = utl_raw.cast_to_raw (
&#39;{
&quot;department&quot;: &quot;Finance and Accounting&quot;,
&quot;employees&quot;: [
{
&quot;name&quot;: &quot;Higgins, Shelley&quot;,
&quot;job&quot;: &quot;Accounting Manager&quot;,
&quot;hireDate&quot;: &quot;2002-06-07T00:00:00&quot;
},
{
&quot;name&quot;: &quot;Gietz, William&quot;,
&quot;job&quot;: &quot;Public Accountant&quot;,
&quot;hireDate&quot;: &quot;2002-06-07T00:00:00&quot;
}
]
}&#39;
) where department_id = 110;
<p>Oracle Database 19c introduces a new option: JSON patching with JSON_mergepatch. This only replaces the relevant section of the document. Just specify the attribute you want to update and its new value.</p>
<p>So instead of having to write the whole document, you can change just the department&#39;s value like this:</p>
update departments_json
set department_data = json_mergepatch (
department_data,
&#39;{
&quot;department&quot; : &quot;Finance and Accounting&quot;
}&#39;
)
where department_id = 110 ;
<p>Of course, as part of this re-org you&#39;ll probably need to merge in the Finance employees too. The employees attribute is an array. To patch an array, you have to replace the whole thing.</p>
<p>So you&#39;ll have an update like:</p>
update departments_json
set department_data = json_mergepatch (
department_data,
&#39;{ &quot;employees&quot; :
[ {
&quot;name&quot; : &quot;Gietz, William&quot;,
&quot;job&quot; : &quot;Public Accountant&quot;,
&quot;hireDate&quot; : &quot;2002-06-07T00:00:00&quot;
},
{
&quot;name&quot; : &quot;Higgins, Shelley&quot;,
&quot;job&quot; : &quot;Accounting Manager&quot;,
&quot;hireDate&quot; : &quot;2002-06-07T00:00:00&quot;
},
{
&quot;name&quot; : &quot;Chen, John&quot;,
&quot;job&quot; : &quot;Accountant&quot;,
&quot;hireDate&quot; : &quot;2005-09-28T00:00:00&quot;
},
{
&quot;name&quot; : &quot;Greenberg, Nancy&quot;,
&quot;job&quot; : &quot;Finance Manager&quot;,
&quot;hireDate&quot; : &quot;2002-08-17T00:00:00&quot;
},
{
&quot;name&quot; : &quot;Urman, Jose Manuel&quot;,
&quot;job&quot; : &quot;Accountant&quot;,
&quot;hireDate&quot; : &quot;2006-03-07T00:00:00&quot;
}
]
}&#39;
)
where department_id = 110 ;
<p>So this is still clunky if you&#39;re working with large arrays. But better than having to replace the whole document!</p>
<a name="query-json">How to Query JSON with SQL</a>
<p><img alt="" src="https://cdn.app.compendium.com/uploads/user/e7c690e8-6ff9-102a-ac6d-e4aebca50425/6a8a9853-c115-4f9b-b14e-7eee344d2c98/Image/d0bd6703b763344f037db6265ed92098/analytics.png" style="width: 512px; height: 333px; display: block; margin-left: auto; margin-right: auto;" /></p>
<p>So now you have JSON in your database. If you want to manipulate it, you can select the whole document into your app. And process it there.</p>
<p>But that&#39;s a lot of data to transfer if you&#39;re only interested in a couple of attributes. It&#39;s better to fetch the parts you want from the table.</p>
<p>Luckily Oracle Database offers many ways to get only the values you&#39;re interested in.</p>
<a name="json-dot-notation">Simple Dot-Notation Access</a>
<p>Provided you added the IS JSON constraint, you can access attributes by:</p>
<ul>
<li>Selecting the column</li>
<li>Adding the JSON path to the value you want</li>
</ul>
<p>So to get the department names, list out the path to it:</p>
select d.department_data.department
from departments_json d;
<p>Note you need to prefix the columns with the table name or alias. Or you&#39;ll get an &quot;ORA-00904: &quot;DEPARTMENT_DATA&quot;.&quot;DEPARTMENT&quot;: invalid identifier&quot; error.</p>
<p>You can also use this to get array elements. To do this, state the index of the element you want (remembering JSON arrays are zero-indexed), then its attribute.</p>
<p>For example, to find the name of the first employee in department 110, use:</p>
select d.department_data.employees[0].name
from departments_json d
where department_id = 110;
EMPLOYEES
Gietz, William
<p>Or perhaps you want the names of all the employees in a department. To do this, get the attributes at position * (asterisk). And you&#39;ll get an array back:</p>
select d.department_data.employees[*].name
from departments_json d
where department_id = 110;
EMPLOYEES
[&quot;Gietz, William&quot;,&quot;Higgins, Shelley&quot;]
<p>Remember you need the IS JSON constraint to do this.</p>
<p>So what if you forgot to add it?</p>
<p>In 18c you can get around this using the TREAT function with the AS JSON clause. Wrap this around a column in a subquery or view.</p>
<p>And you can use dot-notation again!</p>
with j_data as (
select treat (
d.department_data as json
) as department_data
from departments_json d
where department_id = 110
)
select j.department_data.department
from j_data j
where department_data is json;
DEPARTMENT
Accounting
<p>Note TREAT tells the database to <em>consider</em> the contents to be a JSON document. Key point being consider. If the contents aren&#39;t valid JSON, you won&#39;t get an error! And dot-notation access will fail silently, returning null instead.</p>
<p>This highlights a key limitation of dot-notation access. You have no control over the formatting and error handling. Everything comes back as a string (VARCHAR2(4000)). And it suppresses errors. Returning null if there&#39;s an issue accessing the element.</p>
<p>Which is a problem if you&#39;ve got messy data. If you want more control, you can use the JSON query functions JSON_value and JSON_query.</p>
<a name="json-value">Return a Single Value with JSON_value</a>
<p>Say you want to return Shelly&#39;s hire date. And return the value as a real, proper date.</p>
<p>Using dot-notation, the value is a VARCHAR2. So you need to to_date it afterwards. But with JSON_value, all you need to do is specify the returning value as a date (or timestamp)!</p>
select json_value (
department_data,
&#39;$.employees[1].hireDate&#39; returning date
) hire_date
from departments_json d
<p>Note this only works if you format datetime values in your document to <a href="https://en.wikipedia.org/wiki/ISO_8601">ISO 8601 standards</a>. If you&#39;ve used your own, custom format mask this won&#39;t work!</p>
<p>If you have chosen your own date format, by default the above will return NULL for the hire date. Because the database suppresses errors. Which can make tracking down problems in your JSON hard.</p>
<p>For example, if you search for a non-existent attribute you get nothing back:</p>
select json_value (
department_data,
&#39;$.nonExistentAttribute&#39;
) not_here
from departments_json d
where department_id = 110;
NOT_HERE
&lt;null&gt;
<p>This is because the error clause defaults to NULL ON ERROR.</p>
<p>So how do you get an exception instead?</p>
<p>Set the error clause to ERROR ON ERROR:</p>
select json_value (
department_data,
&#39;$.nonExistentAttribute&#39;
error on error
) not_here
from departments_json d;
ORA-40462: JSON_VALUE evaluated to no value
<p>JSON_value is great if you&#39;re getting a single value. I.e. not an array or object.</p>
<p>But what if you want to return all the details for an employee? Or all the employees in a department?</p>
<p>To do that, you&#39;re going to need JSON_query.</p>
<a name="json-query">Return a Document or Array with JSON_query</a>
<p><img alt="" src="https://cdn.app.compendium.com/uploads/user/e7c690e8-6ff9-102a-ac6d-e4aebca50425/6a8a9853-c115-4f9b-b14e-7eee344d2c98/Image/47230516b7bd97cebfd657e6e5e6f16f/developer_kit_512_trimmed.png" style="width: 512px; height: 408px; display: block; margin-left: auto; margin-right: auto;" /></p>
<p>The process for using JSON_query is like JSON_value. State the column holding JSON and the path you want to extract.</p>
<p>For example, if you want to return an array of the employees for a department, you can write:</p>
select json_query (
department_data,
&#39;$.employees[*]&#39;
returning varchar2 pretty
with wrapper
) employees
from departments_json d
where department_id = 110;
EMPLOYEES
[
{
&quot;name&quot; : &quot;Gietz, William&quot;,
&quot;job&quot; : &quot;Public Accountant&quot;,
&quot;hireDate&quot; : &quot;2002-06-07T00:00:00&quot;
},
{
&quot;name&quot; : &quot;Higgins, Shelley&quot;,
&quot;job&quot; : &quot;Accounting Manager&quot;,
&quot;hireDate&quot; : &quot;2002-06-07T00:00:00&quot;
}
]
<p>Note the with wrapper clause. You need to supply this if the path matches more than one value. Otherwise the call will fail silently, returning null due to the default NULL ON ERROR clause.</p>
<p>You can also use this to return one of the attributes of a nested object array as an array. For example, you can return just the names of each employee:</p>
select json_query (
department_data format json,
&#39;$.employees[*].name&#39;
returning varchar2 pretty
with wrapper
) employee_names
from departments_json d
where department_id = 110;
EMPLOYEE_NAMES
[
&quot;Gietz, William&quot;,
&quot;Higgins, Shelley&quot;
]
<p>This function is great if you want the result as JSON. But what if you want to display the document as traditional rows-and-columns?</p>
<p>Say you want a report, showing the details of each employee as separate rows?</p>
<p>Enter JSON_table.</p>
<a name="json-table">Convert JSON to Relational with JSON_table</a>
<p>The JSON_table function enables you to transform a JSON array into rows using SQL.</p>
<p>This makes it easier to view the data in reports. Or join it with relational tables in your database. Or take a document and save it to traditional tables!</p>
<p>To use JSON_table, pass the document as the first argument. Then list out the columns you want to project in the columns clause.</p>
<p>To convert an array to rows, you need to make it a nested path. And state the attributes you want to expose in another columns clause.</p>
<p>So to turn the department documents to a row per employee, you need to:</p>
<ul>
<li>Have an initial columns clause. This includes any department-level attributes you want to display.</li>
<li>Within this define a nested path, returning the employee array (employees[*]).</li>
<li>In this nesting, have another columns clause. This lists the employee attributes you want to return.</li>
</ul>
<p>Which gives a query like:</p>
select j.*
from departments_json d, json_table (
d.department_data, &#39;$&#39; columns (
department path &#39;$.department&#39;,
nested path &#39;$.employees[*]&#39;
columns (
name path &#39;$.name&#39;,
job path &#39;$.job&#39;
) ) ) j
where d.department_id = 110;
DEPARTMENT NAME JOB
Accounting Gietz, William Public Accountant
Accounting Higgins, Shelley Accounting Manager
<p>Neat. But kinda clunky.</p>
<p>You&#39;re repeating yourself a lot. The column names match the attribute names. Making mistakes likely.</p>
<p>So in 18c we&#39;ve simplified the syntax. If the names of the columns you&#39;re projecting match the attribute names in the document, all you need to do is list the attributes!</p>
<p>So you can simplify the above to:</p>
select j.*
from departments_json d, json_table (
d.department_data, &#39;$&#39; columns (
department,
nested employees[*]
columns (
name,
job
) ) ) j
where d.department_id = 110;
DEPARTMENT NAME JOB
Accounting Gietz, William Public Accountant
Accounting Higgins, Shelley Accounting Manager
<p>You can mix-and-match simplified and extended notation. So if you want to rename attributes, or include extra formatting you can.</p>
<p>For example, this adds employee&#39;s hire date to the results. In the process it renames the column. And specifies it as a DATE:</p>
select j.*
from departments_json d, json_table (
d.department_data, &#39;$&#39; columns (
department,
nested employees[*]
columns (
name,
job,
hire_date date path &#39;$.hireDate&#39;
) ) ) j
where d.department_id = 110;
DEPARTMENT NAME JOB HIRE_DATE
Accounting Gietz, William Public Accountant 07-JUN-2002
Accounting Higgins, Shelley Accounting Manager 07-JUN-2002
<p>So far we&#39;ve talked about using these functions to return parts of your document in the query. But you can also use JSON_value and JSON_object in your where clause. So you can find rows which store given values.</p>
<p>Which brings an important question: How do you do this efficiently?</p>
<a name="index-json">How to Search JSON Documents in Your Database</a>
<p>JSON documents can be huge. Reading all these in a table storing billions of rows is going to take a while.</p>
<p>To do this quickly, you&#39;re going to need an index. Luckily Oracle Database has a few options to speed up your JSON search SQL.</p>
<a name="json-function-based-index">JSON Function-based Indexes</a>
<p><img alt="" src="https://cdn.app.compendium.com/uploads/user/e7c690e8-6ff9-102a-ac6d-e4aebca50425/6a8a9853-c115-4f9b-b14e-7eee344d2c98/Image/377b2b6a4eb51086826ce4fb38f0b0ca/mathematics_757566_1280.jpg" style="width: 1280px; height: 896px; display: block; margin-left: auto; margin-right: auto;" /></p>
<p>If you know which attributes you&#39;re searching on in advance, you can <a href="https://blogs.oracle.com/sql/how-to-create-and-use-indexes-in-oracle-database">create function-based indexes</a> for your queries.</p>
<p>For example, say you want to allow staff to search by department name and get the corresponding document back.</p>
<p>So if you have a query like this:</p>
select * from departments_json
where json_value ( department_data, &#39;$.department&#39; ) = :dept_name;
<p>To make it fast, create an index with the exact-same function you use in your where clause, like so:</p>
create index dept_department_name_i on
departments_json (
json_value ( department_data, &#39;$.department&#39; )
);
<p>Be aware that this can only index one value per document. So you can&#39;t index a whole array.</p>
<p>And an attribute could flip between a single value and an array between documents. So this could lead to unexpected results.</p>
<p>To prevent this problem, and ensure index attributes are always scalars, add the ERROR ON ERROR clause. And NULL ON EMPTY to avoid errors for missing attributes.</p>
<p>So the full form of the above index is really:</p>
create index dept_department_name_i on
departments_json (
json_value (
department_data, &#39;$.department&#39;
error on error
null on empty
)
);
<p>Whatever you do, ensure the clauses you use in your where clause match those you used in the index!</p>
<p>So function-based indexes are fine if you know what you&#39;re looking for.</p>
<p>But what if you want to support ad-hoc queries?</p>
<a name="json-search-index">JSON Search Index for Fast Ad-Hoc SQL</a>
<p>Before Oracle Database 12.2 you could create an <a href="https://oracle-base.com/articles/12c/indexing-json-data-in-oracle-database-12cr1#full-text-search">Oracle Text index over JSON</a>. With this in place, the database could use the index for any JSON function query.</p>
<p>But this was kinda messy. So in 12.2 we simplified the syntax. With a JSON search index.</p>
<p>To use this, first add the IS JSON constraint to the column. Then create a SEARCH index with the FOR JSON clause:</p>
create search index dept_json_i on
departments_json ( department_data )
for json;
<p>This creates an Oracle Text index behind the scenes. So now, whatever attribute you&#39;re inspecting, you can find the associated rows.</p>
<p>Using JSON_textcontains, you can find any document which has a given value anywhere in the path you provide.</p>
<p>For example, to find documents which have the value &quot;Public&quot; anywhere in them, use:</p>
select *
from departments_json d
where json_textcontains ( department_data, &#39;$&#39;, &#39;Public&#39; );
<p>Note this has to be the exact word. The query above won&#39;t return documents that include &quot;Publication&quot; unless the value &quot;Public&quot; is also in the same document.</p>
<p>The database can also use the search index if you use JSON_value in your where clause.</p>
<p>But it indexes the whole document for every row. Which means the index will be bigger<span data-offset-key="3iru3-0-0"><span data-text="true"> than its function-based counterpart</span></span>. And less efficient.</p>
<p>For example, let&#39;s load up the table with JSON documents for all the departments with employees in the HR schema.</p>
<p>Then compare how the indexes perform when searching for the accounting department:</p>
select *
from departments_json d
where json_value ( department_data, &#39;$.department&#39; ) = &#39;Accounting&#39;;
<p>When using the targeted, function-based index, you&#39;ll get a plan like:</p>
------------------------------------------------------------------------------------------------------------------------
| Id | Operation | Name | Starts | E-Rows | A-Rows | A-Time | Buffers |
------------------------------------------------------------------------------------------------------------------------
| 0 | SELECT STATEMENT | | 1 | | 1 |00:00:00.01 | 2 |
| 1 | TABLE ACCESS BY INDEX ROWID BATCHED| DEPARTMENTS_JSON | 1 | 1 | 1 |00:00:00.01 | 2 |
|* 2 | INDEX RANGE SCAN | DEPT_DEPARTMENT_NAME_I | 1 | 1 | 1 |00:00:00.01 | 1 |
------------------------------------------------------------------------------------------------------------------------
Predicate Information (identified by operation id):
---------------------------------------------------
2 - access(&quot;D&quot;.&quot;SYS_NC00003$&quot;=&#39;Accounting&#39;)
<p>But the search index uses the following plan:</p>
-------------------------------------------------------------------------------------------------------------------
| Id | Operation | Name | Starts | E-Rows | A-Rows | A-Time | Buffers | Reads |
-------------------------------------------------------------------------------------------------------------------
| 0 | SELECT STATEMENT | | 1 | | 1 |00:00:00.01 | 28 | 4 |
|* 1 | TABLE ACCESS BY INDEX ROWID| DEPARTMENTS_JSON | 1 | 1 | 1 |00:00:00.01 | 28 | 4 |
|* 2 | DOMAIN INDEX | DEPT_JSON_I | 1 | | 1 |00:00:00.01 | 27 | 4 |
-------------------------------------------------------------------------------------------------------------------
Predicate Information (identified by operation id):
---------------------------------------------------
1 - filter(JSON_VALUE(&quot;DEPARTMENT_DATA&quot; FORMAT JSON , &#39;$.department&#39; RETURNING VARCHAR2(4000) NULL ON
ERROR)=&#39;Accounting&#39;)
2 - access(&quot;CTXSYS&quot;.&quot;CONTAINS&quot;(&quot;D&quot;.&quot;DEPARTMENT_DATA&quot;,&#39;{Accounting} INPATH (/department)&#39;)&gt;0)
<p>The key thing to note here is the buffers column. This is how many consistent gets each step does. The lower the number, the better.</p>
<p>The search index uses 28 consistent gets. Compared to 2 for the function-based index. That&#39;s an order of magnitude worse!</p>
<p>So if you know which queries you&#39;ll be running, e.g. if you&#39;re coding them in the app, use a function-based index.</p>
<p>Reserve search indexes for cases where you need to support ad-hoc queries.</p>
<p>Of course, you can create a generic search index and (many!) function-based indexes on the same column. So you can have targeted indexes for critical queries. And a fall back for rarely-executed general searches.</p>
<p>So we can use the JSON functions to extract values from a document. And efficiently find documents with given values.</p>
<p>But they rely on you knowing the structure of your JSON. You need good documentation and standards to know what the attributes are.</p>
<p>If your processes are&hellip; not the best, your documentation may be out-of-date. Or non-existent!</p>
<p>So you need to inspect every single row to find the attribute names. (Something you get for free if you used proper relational tables&hellip;)</p>
<p>Luckily you can find which rows (if any) have a given attribute. With JSON_exists.</p>
<a name="json-exists">Search for Attributes with JSON_exists</a>
<p><img alt="" src="https://cdn.app.compendium.com/uploads/user/e7c690e8-6ff9-102a-ac6d-e4aebca50425/6a8a9853-c115-4f9b-b14e-7eee344d2c98/Image/c4c0982c59a487bcab72fca11517dd99/search_5x_trimmed.png" style="width: 512px; height: 405px; display: block; margin-left: auto; margin-right: auto;" /></p>
<p>Say you want to know which documents have an employee with a hire date. Do that with JSON_exists by listing the path you&#39;re looking for.</p>
<p>Like so:</p>
select department_id
from departments_json d
where json_exists (
department_data,
&#39;$.employees.hireDate&#39;
);
DEPARTMENT_ID
110
<p>Or check if you&#39;ve saved anyone&#39;s salary:</p>
select department_id
from departments_json d
where json_exists (
department_data,
&#39;$.employees.salary&#39;
);
no rows selected
<p>Using this you can easily see which documents have which attributes.</p>
<p>Of course, this could still lead to a lot of trial-and-error to figure out what <em>is</em> in your documents.</p>
<p>Fortunately Oracle Database has a neat way to expose a document&#39;s attributes.</p>
<p>The JSON Data Guide.</p>
<a name="json-data-guide">Convert Documents to Relational with JSON Data Guide</a>
<p>The JSON Data Guide allows you to add columns to a table, returning values from a JSON document. Under the hood, this creates <a href="http://www.oracle-developer.net/display.php?id=510">virtual columns</a> calling JSON_value to get the values out.</p>
<p>To add these columns, you need:</p>
<ul>
<li>To create a JSON search index with the DATAGUIDE ON option</li>
<li>Be on Oracle Database 12.2 or higher</li>
<li>Create the columns by calling DBMS_JSON.add_virtual_columns</li>
</ul>
<p>This will then trawl through your objects, finding the attributes and adding columns to your table. Like so:</p>
alter index dept_json_ii
rebuild parameters ( &#39;dataguide on&#39; )
exec dbms_json.add_virtual_columns ( &#39;departments_json&#39;, &#39;department_data&#39; );
desc departments_json;
Name Null? Type
DEPARTMENT_ID NOT NULL NUMBER(38)
DEPARTMENT_DATA NOT NULL BLOB
DEPARTMENT_DATA$department VARCHAR2(16)
<p>As you can see, this added a column for the department. Which has a cryptic JSON$colname name. Which is case-sensitive. Meaning you have to use double quotes to access it:</p>
select &quot;DEPARTMENT_DATA$department&quot; department_name
from departments_json
where department_id = 110;
<p>It&#39;d be better if you could have properly named columns.</p>
<p>The good news is, you can!</p>
<p>DBMS_JSON includes a rename_column procedure. This maps a given attribute path and data type to a new name.</p>
<p>After setting this, you need to regenerate the columns. To this by calling DBMS_JSON.add_virtual_columns again. This drops and recreates the columns:</p>
begin
dbms_json.rename_column(
&#39;departments_json&#39;, &#39;department_data&#39;,
&#39;$.department&#39;, dbms_json.type_string,
&#39;DEPARTMENT_NAME&#39;
);
dbms_json.add_virtual_columns (
&#39;departments_json&#39;, &#39;department_data&#39;
);
end;
/
desc departments_json
Name Null? Type
DEPARTMENT_ID NOT NULL NUMBER(38)
DEPARTMENT_DATA NOT NULL BLOB
DEPARTMENT_NAME VARCHAR2(16)
<p>This is great.</p>
<p>But what about that employees array? How come there are no columns for that?</p>
<p>Well it&#39;s a one-to-many problem. There&#39;s no clear way to return all the elements of an array in a single row.</p>
<p>Fortunately, you can also use the JSON Data Guide to create a view. This uses JSON_table under the covers to turn an array to rows-and-columns:</p>
begin
dbms_json.create_view (
&#39;department_employees&#39;, &#39;departments_json&#39;,
&#39;department_data&#39;,
dbms_json.get_index_dataguide (
&#39;departments_json&#39;,
&#39;department_data&#39;,
dbms_json.format_hierarchical
)
);
end;
/
select * from department_employees
where department_id = 110;
DEPARTMENT_ID DEPARTMENT_DATA$job DEPARTMENT_DATA$name DEPARTMENT_DATA$hireDate DEPARTMENT_NAME
110 Public Accountant Gietz, William 2002-06-07T00:00:00 Accounting
110 Accounting Manager Higgins, Shelley 2002-06-07T00:00:00 Accounting
<p>Simple!</p>
<p>So you&#39;ve added your virtual columns or created views.</p>
<p>But what happens if someone adds a document with a new attribute? How do you keep the virtual columns in sync?</p>
<p>You could call DBMS_JSON.add_virtual_columns periodically. Or come up with some DDL-trigger based solution.</p>
<p>Or you could let the database do it for you automagically!</p>
<p>If you set the search index&#39;s parameters to DATAGUIDE ON CHANGE add_vc, the database will keep the columns in sync for you:</p>
alter index dept_json_i
rebuild parameters ( &#39;DATAGUIDE ON CHANGE add_vc&#39; );
<p>Bear in mind this will happen whenever the index updates.</p>
<p>This could be handy when you&#39;re prototyping in development. But I would advise against this in production; you could end up running <em>lots</em> of DDL against your tables!</p>
<p>But what if you want to know when a new document has new attributes? And let your developers know something&#39;s changed?</p>
<p>Well, you can write your own, <a href="https://www.oracle.com/pls/topic/lookup?ctx=dblatest&amp;id=GUID-EC076D41-D096-42A9-BCCA-AD894637FB5A">custom change procedure</a>. This could inspect the current virtual columns. Then see if they match those in the Data Guide. And send a notification to the relevant people if they don&#39;t.</p>
<a name="generate-json">How to Generate JSON with SQL</a>
<p><img alt="" src="https://cdn.app.compendium.com/uploads/user/e7c690e8-6ff9-102a-ac6d-e4aebca50425/6a8a9853-c115-4f9b-b14e-7eee344d2c98/Image/27c2be6cec0ae2e11c10c8426f5aa41c/sql_machine_512_trimmed.png" style="width: 512px; height: 353px; display: block; margin-left: auto; margin-right: auto;" /></p>
<p>So far we&#39;ve discussed storing JSON documents as-is in your database. But what if you&#39;re doing the good thing and storing your data relationally?</p>
<p>And you want to return the rows as JSON?</p>
<p>Converting query results to JSON is a non-trivial task. <a href="https://github.com/pljson/pljson">Many people</a> <a href="https://docs.oracle.com/cd/E59726_01/doc.50/e39149/apex_json.htm">have built libraries</a> to help you with this.</p>
<p>So Oracle Database 12.2 we added new functions to do this natively:</p>
<ul>
<li>json_array</li>
<li>json_arrayagg</li>
<li>json_object</li>
<li>json_objectagg</li>
</ul>
<p>How you use these is pretty self-explanatory.</p>
<p>The array functions return the arguments as an array (surrounded by square brackets).</p>
<p>The object functions return the attribute/value pairs as an object (surrounded by curly braces).</p>
<p>The non-AGG versions return a document for each row in the input. Whereas the AGG varieties can combine many rows into one document or array. According to your GROUP BY.</p>
<p>So if you want to create the document we started with, from the <a href="https://github.com/oracle/db-sample-schemas/tree/master/human_resources">standard HR schema</a>, you can do this:</p>
select json_object (
&#39;department&#39; value d.department_name,
&#39;employees&#39; value json_arrayagg (
json_object (
&#39;name&#39; value last_name || &#39;, &#39; || first_name,
&#39;job&#39; value job_title,
&#39;hireDate&#39; value hire_date
)
)
)
from hr.departments d
join hr.employees e
on d.department_id = e.department_id
join hr.jobs j
on e.job_id = j.job_id
where d.department_id = 110
group by d.department_name;
<p>To understand what&#39;s going on, it&#39;s best to work from the inside out.</p>
<p>First, we create a JSON object for each employee.</p>
json_object (
&#39;name&#39; value last_name || &#39;, &#39; || first_name,
&#39;job&#39; value job_title,
&#39;hireDate&#39; value hire_date
)
<p>Then combine these into an array for each department:</p>
json_arrayagg (
json_object (
&#39;name&#39; value last_name || &#39;, &#39; || first_name,
&#39;job&#39; value job_title,
&#39;hireDate&#39; value hire_date
)
)
<p>Finally we add the department attributes, returning the finished document:</p>
json_object (
&#39;department&#39; value d.department_name,
&#39;employees&#39; value json_arrayagg (
json_object (
&#39;name&#39; value last_name || &#39;, &#39; || first_name,
&#39;job&#39; value job_title,
&#39;hireDate&#39; value hire_date
)
)
)
<p>Note that on the first release these functions had limited data type support: just VARCHAR2, NUMBER, and DATE.</p>
<p>In 18c we&#39;ve finished off the implementation. So they support almost all the <a href="https://docs.oracle.com/en/database/oracle/oracle-database/18/adjsn/changes.html#GUID-8C9B00DD-45D8-499B-919C-8632E034B664">data types in Oracle Database</a>.</p>
<p>But like JSON_table, these can be tricky to work with. Converting a row to a JSON object is tedious if you have many columns.</p>
<p>So in 19c, we&#39;ve made it easier!</p>
<p>Pass * to JSON_object and it&#39;ll generate an object from all the columns returned by your query:</p>
select json_object ( * ) jdoc
from hr.departments
where department_id = 110;
JDOC
{
&quot;DEPARTMENT_ID&quot; : 110,
&quot;DEPARTMENT_NAME&quot; : &quot;Accounting&quot;,
&quot;MANAGER_ID&quot; : 205,
&quot;LOCATION_ID&quot; : 1700
}
<p>So these functions make it easy to convert your data to JSON. And it&#39;s simple to use JSON_table to receive documents and store the data relationally.</p>
<p>If you do this, it&#39;s inevitable that at some point you&#39;ll want to generate back out the same document you stored in the first place.</p>
<p>Which brings the important question:</p>
<p>How do you ensure the JSON you received and generated are the same?</p>
<a name="json-equal">How to Compare JSON Documents using SQL</a>
<p>At first glance this seems like a simple problem. JSON is just text. So you can just check if one document equals the other.</p>
<p>Right?</p>
<p>Well, according to the <a href="https://json.org/">JSON standard</a>:</p>
<ul>
<li>Insignificant whitespace is irrelevant</li>
<li>Insignificant attribute order is irrelevant</li>
</ul>
<p>So when comparing documents, first you need to strip out all the whitespace surrounding attributes and their values. Tricky, but doable.</p>
<p>But comparing documents which have their attributes in a different order?! Well that&#39;s another matter.</p>
<p>One way to do this, is to convert the documents to their relational form. Then you can do set difference operations in SQL to find any mismatches:</p>
with doc1_rows as (
select t.*
from departments_json, json_table ( department_data
columns
department,
nested employees[*]
columns (
name , job
)
) t
where department_id = 110
), doc2_rows as (
select *
from json_table ( &#39;{
&quot;employees&quot; :
[ {
&quot;name&quot; : &quot;Gietz, William&quot;,
&quot;job&quot; : &quot;Public Accountant&quot;,
&quot;hireDate&quot; : &quot;2002-06-07T00:00:00&quot;
},
{
&quot;hireDate&quot; : &quot;2002-06-07T00:00:00&quot;,
&quot;name&quot; : &quot;Higgins, Shelley&quot;,
&quot;job&quot; : &quot;Accounting Manager&quot;
}
],
&quot;department&quot; : &quot;Accounting&quot;
}&#39; columns
department,
nested employees[*]
columns (
name , job
)
)
), all_rows as (
select d.*, -1 tab
from doc1_rows d
union all
select d.*, 1 tab
from doc2_rows d
)
select department, name, job
from all_rows
group by department, name, job
having sum ( tab ) &lt;&gt; 0;
no rows selected
<p>Yuck.</p>
<p>So we&#39;ve addressed this in 18c. There&#39;s a new condition, JSON_equal. This takes two documents as input. And returns true if they&#39;re the same, false if they&#39;re different. According to JSON standard rules.</p>
<p>So you can replace all the code above with:</p>
select case
when json_equal ( department_data,
&#39;{&quot;employees&quot; :
[
{
&quot;name&quot; : &quot;Gietz, William&quot;,
&quot;job&quot; : &quot;Public Accountant&quot;,
&quot;hireDate&quot; : &quot;2002-06-07T00:00:00&quot;
},
{
&quot;hireDate&quot; : &quot;2002-06-07T00:00:00&quot;,
&quot;name&quot; : &quot;Higgins, Shelley&quot;,
&quot;job&quot; : &quot;Accounting Manager&quot;
}
],
&quot;department&quot; : &quot;Accounting&quot;
}&#39; ) then &#39;EQUAL&#39; else &#39;DIFFERENT&#39; end matching
from departments_json
where department_id = 110;
MATCHING
EQUAL
<p>Nice!</p>
<p>Note this doesn&#39;t tell you which parts are different. Just that they are. So if you need to show which attributes are different, you&#39;ll need to go down the relational shredding approach.</p>
<p>One final thing before we finish. I promised it would be easy to convert JSON stored as BLOB to text.</p>
<p>How?</p>
<a name="json-serialize">How to Pretty Print JSON</a>
<p><img alt="" src="https://cdn.app.compendium.com/uploads/user/e7c690e8-6ff9-102a-ac6d-e4aebca50425/6a8a9853-c115-4f9b-b14e-7eee344d2c98/Image/36c3e0fb38849ce4888db943c9644a29/nature_2834078_1280.png" style="width: 1280px; height: 756px;" /></p>
<p>Simple.</p>
<p>The JSON_query function includes a PRETTY clause. Include this at the end and it&#39;ll return the document as pretty-printed text:</p>
select json_query (
department_data, &#39;$&#39; pretty
)
from departments_json;
<p>Or, from 19c, you can use the new JSON_serialize function. This allows you to convert JSON from and to binary by specifying the return type as VARCHAR2, CLOB or BLOB.</p>
<p>And it includes a PRETTY clause :)</p>
<p>So you can get a nicely formatted document like so:</p>
select json_serialize (
department_data returning varchar2 pretty
)
from departments_json;
<p>So that was a quick tour of JSON in Oracle Database. So far we&#39;ve concentrated on working with JSON in SQL.</p>
<p>But if you&#39;re building PL/SQL APIs, chances are you&#39;ll want to work with JSON there too. The good news is we&#39;ve created some <a href="https://www.oracle.com/pls/topic/lookup?ctx=dblatest&amp;id=GUID-C0C2A8C0-99BD-4770-9EA2-B7D53804FC18">object types to do just that</a>.</p>
<p>If you want to try out the example in this post, you can find them all in <a href="https://livesql.oracle.com/apex/livesql/file/content_HZ0745GCRAOQYKGBJBP3LUH47.html">this Live SQL script</a>.</p>
<p>Or if you want to get into the details of how this works or any of the other options discussed above, read the <a href="https://docs.oracle.com/en/database/oracle/oracle-database/19/adjsn/index.html">JSON Developer&#39;s Guide</a>.</p>
<p>So now we&#39;d like to know:</p>
<p>Are you using JSON in your database? How? Any other options you&#39;d like to see to make this easier to work with?</p>
<p>Let us know in the comments!</p>
Thu, 21 Feb 2019 11:29:53 +0000https://blogs.oracle.com/sql/how-to-store-query-and-create-json-documents-in-oracle-databaseChris SaxonAnnouncing the 2018 Oracle Dev Gym Championship for Database Designhttps://blogs.oracle.com/sql/announcing-the-2018-oracle-dev-gym-championship-for-database-design
<p>The dust has settled. The scores are in. It&#39;s time to announce who will be taking part in the Database Design Annual Championship on the <a href="https://devgym.oracle.com">Oracle Dev Gym</a>!</p>
<p>The following players will be invited to take part. This is scheduled to take place on 14th February 2019.</p>
<p>The number in parentheses after their names are the number of championships in which they have already participated.</p>
<p>Congratulations to all listed below on their accomplishment. And best of luck in the upcoming competition.</p>
<p>May the best database designer win!</p>
<p>Now, to write some fiendishly hard quizzes to stretch all the players ;)</p>
Name
Rank
Stelios Vlasopoulos (4)
1
mentzel.iudith (4)
2
Andrey Zaytsev (4)
3
Ludovic Szewczyk (1)
4
Chad Lee (4)
5
siimkask (4)
6
Ivan Blanarik (4)
7
Maxim Borunov (4)
8
Jan &Scaron;er&aacute;k (4)
9
NielsHecker (4)
10
li_bao (4)
11
seanm95 (4)
12
Chase Mei (4)
13
_tiki_4_ (4)
14
tonyC (3)
15
Rytis Budreika (4)
16
pjas (1)
17
Sachi (2)
18
NickL (3)
19
Michal P. (4)
20
Vyacheslav Stepanov (4)
21
Aleksei Davletiarov (2)
22
Talebian (1)
23
mcelaya (2)
24
Henry_A (4)
25
Sartograph (2)
26
HSteijntjes (1)
27
swesley_perth (2)
28
RalfK (1)
29
ted (2)
30
PZOL (3)
31
msonkoly (2)
32
JustinCave (4)
33
Karel_Prech (3)
34
MarkusId (0)
35
richdellheim (1)
36
Rakesh Dadhich (3)
37
Cor van Berkel (1)
38
Mike Tessier (1)
39
K&ouml;teles Zsolt (0)
40
umir (3)
41
whab@tele2.at (2)
42
Kias (1)
43
Joaquin_Gonzalez (4)
44
Sandra99 (4)
45
syukhno (1)
46
JeroenR (3)
47
Hertha Rettinger (1)
48
Vyzvalda (0)
49
Alexandre (1)
50
Tue, 15 Jan 2019 14:42:45 +0000https://blogs.oracle.com/sql/announcing-the-2018-oracle-dev-gym-championship-for-database-designChris SaxonHow to Create, Alter, and Drop Tables in SQLhttps://blogs.oracle.com/sql/how-to-create-alter-and-drop-tables-in-sql
<p>You&#39;re building a new application. So you need somewhere to store your data. It&#39;s time to create a table in your database!</p>
<p>In this post you&#39;ll find out how to:</p>
<ul>
<li><a href="#create">Create a new table with CREATE TABLE</a></li>
<li><a href="#alter">Change an existing table with ALTER TABLE</a></li>
<li><a href="#drop">Remove a table with DROP TABLE</a></li>
</ul>
<a name="create"></a>
How to Create a Table
<p>The basic create table statement takes the form:</p>
create table &lt;table_name&gt; (
&nbsp; &lt;column1&gt; &lt;data type&gt;,
&nbsp; &lt;column2&gt; &lt;data type&gt;,
&nbsp; &lt;column3&gt; &lt;data type&gt;,
&nbsp; ...
);
<p>So to create a table called toys, with the columns toy_name, weight, and colour, run:</p>
create table toys (
toy_name varchar2(10),
&nbsp; weight number,
&nbsp; colour varchar2(10)
);
<p>Oracle Database has <a href="https://www.oracle.com/pls/topic/lookup?ctx=dblatest&amp;id=GUID-A3C0D836-BADB-44E5-A5D4-265BA5968483">many data types</a>&nbsp;to choose from. Common data types are:</p>
<ul>
<li>Number - stores numeric data: prices, weights, distances, etc.</li>
<li>Date - holds date and time information</li>
<li>Varchar2 - use for general purpose text; names, descriptions, etc.</li>
</ul>
<p>Pick the most appropriate type for the values you&#39;ll store in the column. Choosing the wrong type can lead to <a href="https://www.youtube.com/watch?v=OgyxXFIYmOc">slow queries</a>, <a href="https://blogs.oracle.com/oraclemagazine/on-implicit-conversions-and-more">wrong results</a>, and <a href="https://asktom.oracle.com/Misc/all-about-security-sql-injection.html">security holes</a>.</p>
<p><em>NOTE: Create table is a form of data-definition language (DDL) statement. These change the objects in your database. Oracle Database runs a commit before and after DDL. So if the create works, it&#39;s saved to your database.</em></p>
<p>You can also create a table based on a select statement. This makes a table with the same columns and rows as the source query. This operation is known as create-table-as-select (CTAS).</p>
<p>This is a handy way to copy one table to another. For example, the following creates toys_clone from toys:</p>
create table toys_clone as
&nbsp; select * from toys;
<p>Easy, right?</p>
<p>Yes. But, as always, there&#39;s more to it than this. You&#39;ll want to <a href="https://www.oracle.com/pls/topic/lookup?ctx=dblatest&amp;id=GUID-6A89FF39-AD42-4399-BD1B-E51ECEE50B4E">add some constraints</a> to your table. And there are many types of table available in Oracle Database, including:</p>
<ul>
<li>Table Organization:
<ul style="list-style-type:circle;">
<li><a href="#create-heap">Heap</a></li>
<li><a href="#create-iot">Index</a></li>
<li><a href="#create-external">External</a></li>
</ul>
</li>
<li><a href="#create-temporary">Temporary Tables</a></li>
<li><a href="#create-partition">Partitioning</a></li>
<li><a href="#create-cluster">Table Clusters</a></li>
</ul>
<p>These affect how the database physically stores the data. Which can have a big impact on performance.</p>
<p>Database tables tend to last a long time. And it&#39;s tricky to change the type of a table storing millions of rows. So it&#39;s worth spending a few minutes deciding which you need.</p>
<p>For an overview of these types, watch this video, taken from the first module of <a href="https://devgym.oracle.com/pls/apex/dg/class/databases-for-developers-foundations.html">Databases for Developers: Foundations</a>:</p>
<p></p>
<a name="create-heap"></a>
Heap Organized Tables
<p>This is the default for tables in Oracle Database. But if you want to be explicit, add the &quot;organization heap&quot; clause at the end:</p>
create table toys (
toy_name varchar2(10),
&nbsp; weight number,
&nbsp; colour varchar2(10)
) organization heap;
<p>Heaps are good general purpose tables. They are the most common type you&#39;ll see in Oracle Database installations.</p>
<p>With these, the database is free to store new rows wherever there is space. So if you read ten rows, they could be anywhere on disk.</p>
<p>If you&#39;re lucky, they&#39;re all in the same place. So the query can get them all in one trip to disk.</p>
<p>But it&#39;s not guaranteed. Each row could be in a different location. Meaning you need ten I/O operations to read them all.</p>
<p>This is bad news if you want the query to be as fast as possible. The more disk reads your SQL does, the slower it will be.&nbsp;Even if the rows are cached in memory, accessing ten memory addresses is slower than hitting one.</p>
<p>Luckily you can force the database to store rows with similar values in the same place. This can reduce work your query does to get them. Making your SQL faster!</p>
<p>To force this physical order, you need to change your table&#39;s properties. The first we&#39;ll look at is an index-organized table.&nbsp;&nbsp;</p>
<a name="create-iot"></a>
Index-Organized Tables (IOTs)
<p>Indexes are ordered data structures. So an IOT stores rows physically sorted according to its primary key.</p>
<p><em>NOTE: A primary key (PK) is a constraint. Each set of values in its columns can only appear once. So you can&#39;t have duplicates. It also has a not null constraint. And creates a unique index in the background.</em></p>
<p>To create one, add the organization index clause to the table definition:</p>
create table toys (
toy_name varchar2(10) primary key,
&nbsp; weight number,
&nbsp; colour varchar2(10)
) organization index;
<p>So why would you use this instead of a default heap table?</p>
<p>A couple of reasons.</p>
<p>First up, when using heap organization, the table and its primary key index are separate data structures. An IOT combines these into one.</p>
<p>This can reduce some overheads.</p>
<p>You no longer need an extra index for the primary key. Which can save you some space.</p>
<p>And SQL accessing a row using the primary key only has to access one structure. Instead of two. So these queries are that tiny bit faster.</p>
<p>But the biggest advantage (IMO) comes for tables with a multi-column primary key. For example many-to-many join tables. Such as customers to their addresses:</p>
create table customer_addresses (
customer_id integer,
&nbsp; address_id integer,
&nbsp; primary key ( customer_id, address_id )
) organization index;
<p>This stores the values sorted by customer_id, then address_id. So all the addresses for customer 1 are next to each other on disk.&nbsp;</p>
<p>So if you search for all the addresses for this customer, like so:</p>
select *
from customers
where customer_id = 1;
<p>You know all the rows will be in the same few locations. Making your SQL that tiny bit faster.</p>
<p>Whereas with a heap table, the database could store them anywhere.</p>
<p>You can extend this principle to any table with a composite PK. And you (almost always) search for rows where the first column of the PK equals some value.</p>
<p>This is common for tables in a master-detail relationship. For example:</p>
<ul>
<li>Orders and order items</li>
<li>Invoices and invoice lines</li>
<li>Flight itineraries and their flights</li>
</ul>
<p>Here you usually get the rows from the detail table matching a row in the parent. Such as all products in an order.</p>
<p>Remember, to use an IOT the table must have a primary key. So the values you want to sort by must be unique. If they&#39;re not, you can get around this by <a href="https://mwidlake.wordpress.com/2011/08/17/iot-part-5-primary-key-drawback-and-workaround/">creating a fake primary key</a>. But this is a niche technique. Only use if you&#39;re sure what you&#39;re doing!</p>
<p>You can also use partitioning or table clusters to impose order on your data. But let&#39;s cover off the organization clause first.</p>
<a name="create-external"></a>
External Tables
<img alt="" src="https://cdn.app.compendium.com/uploads/user/e7c690e8-6ff9-102a-ac6d-e4aebca50425/6a8a9853-c115-4f9b-b14e-7eee344d2c98/Image/ec2a3ff2df69004464d7228e26d8431a/grand_canyon_1865386_1920.jpg" />
<p style="text-align: center;"><a href="https://pixabay.com/en/grand-canyon-landscape-scenic-rock-1865386/">Pixabay</a></p>
<p>The final option for the organization clause is external. You use this to read text files stored on the database&#39;s file system. This enables you to read CSV or other formatted files into your database using SQL.</p>
<p>To create one you must have a directory object in place. This points to the folder where the file is:</p>
create or replace directory ext_files as &#39;/path/to/files&#39;;
<p><span data-offset-key="fmck6-0-0">To read the file toys.csv in /path/to/files, use this directory and define the file like so:</span></p>
create table toys_csv (
toy_name varchar2(10),
&nbsp; weight number,
&nbsp; colour varchar2(10)
) organization external (
default directory ext_files
&nbsp; location ( &#39;toys.csv&#39; )
);
<p>Now, when you query toys_csv, you&#39;re reading the records in the file toys.csv.</p>
<p><span data-offset-key="ahfmf-0-0">There are many options for <a href="https://www.oracle.com/pls/topic/lookup?ctx=dblatest&amp;id=GUID-F6948F0E-0557-4C42-9145-1897DE974CC3">creating external tables</a>. If the file you&#39;re reading isn&#39;t a straightforward CSV, set the external clause as needed.</span></p>
<p><span data-offset-key="enro8-0-0">External tables are great for loading files using SQL. But sometimes you may need to massage the data before saving it in your real tables. Or need to read the same record many times during the load.</span></p>
<p>When doing this, it&#39;s handy to stage the data in a private working area. Somewhere you can store rows that only you can see.</p>
<p>Enter:</p>
<a name="create-temporary"></a>
Temporary Tables
<p>Rows in temporary tables are private to your session. Only you can view them. And, once you disconnect, the database removes any rows you added.</p>
<p>Oracle Database has two types of temporary table: global (GTT) and private (PTT).</p>
Global Temporary Tables (GTT)
<p>The syntax to create a global temporary table is:</p>
create global temporary table toys_gtt (
toy_name varchar2(10),
&nbsp; weight number,
&nbsp; colour varchar2(10)
);
<p>By default the database will auto-delete all the rows at the end of each transaction. So as soon as you commit, it&#39;s empty!</p>
<p>If you need to keep the rows across transactions, set the on commit clause to preserve rows:</p>
create global temporary table toys_gtt_keep (
toy_name varchar2(10),
&nbsp; weight number,
&nbsp; colour varchar2(10)
) on commit preserve rows;
<p>Global temporary tables have a standard definition across the database. You create it once and leave it in your application. This is in sharp contrast to:</p>
Private Temporary Tables (PTT)
<p>PTTs are new in Oracle Database 18c. With these, not only are the rows hidden from other sessions, so is the table itself!</p>
<p>The following creates a PTT called ora$ptt_toys:</p>
create private temporary table ora$ptt_toys (
toy_name varchar2(10),
&nbsp; weight number,
&nbsp; colour varchar2(10)
);
<p><em>NOTE: the cryptic ora$ptt_ prefix for the table name. This must match whatever your database&#39;s private_temp_table_prefix parameter is set to. Otherwise it won&#39;t work! Also,&nbsp;unlike all other forms of DDL, create private temporary table does NOT commit!</em></p>
<p>By default, these only last for the duration of a transaction. As soon as you commit (or rollback) the table is gone.</p>
<p>If you need it to persist for the length of your connection, set the on commit clause to preserve definition:</p>
create private temporary table ora$ptt_toys_keep (
toy_name varchar2(10),
&nbsp; weight number,
&nbsp; colour varchar2(10)
) on commit preserve definition;
<p>Unlike every other type of table, two people can create a PTT at the same time with the same name. But different columns! Handy if you need to stage values from dynamic data sources.</p>
<a name="create-partition"></a>
Partitioning
<img alt="" src="https://cdn.app.compendium.com/uploads/user/e7c690e8-6ff9-102a-ac6d-e4aebca50425/6a8a9853-c115-4f9b-b14e-7eee344d2c98/Image/22fc26f23629c29c850618461461fe21/empty_tables_1040519_1280.jpg" style="width: 1280px; height: 853px;" />
<p style="text-align: center;"><a href="https://pixabay.com/en/empty-tables-gastronomy-empty-chairs-1040519/">Pixabay</a></p>
<p>As the number of rows in a table grows, it gets harder to manage them. Queries can become sluggish, adding indexes takes longer, and archiving off old data is slow.</p>
<p>Partitioning addresses these issues by allowing you to logically split a table into several smaller tables. But still access these as one table.</p>
<p>You can even subdivide each of these partitions further with subpartitions.</p>
<p><em>NOTE: to partition a table, you must purchase the <a href="https://www.oracle.com/pls/topic/lookup?ctx=dblatest&amp;id=GUID-AB56CEE3-955E-4E56-8B44-6075E889C283">Partitioning option</a>. Check you have this before diving in!</em></p>
<p>To partition a table, choose your partitioning column(s) and method. Supported ways include:</p>
<ul>
<li>Range &ndash; <span data-offset-key="90mua-0-0">each partition has an upper bound. Rows with values less than this and greater than or equal to the previous boundary go in this partition</span></li>
<li>List &ndash; <span data-offset-key="br8ku-0-0">states exactly which values go in each partition</span></li>
<li>Hash &ndash; uses an internal function to choose which partition to place rows in</li>
</ul>
<p>When you add rows to the table, they will go in the relevant partition.</p>
<p>For example, you may want to split up your toys by colour. To do so, list partition it by this column. And define which values will go in each partition:</p>
create table toys_partitioned (
toy_name varchar2(10),
&nbsp; weight number,
&nbsp; colour varchar2(10)
) partition by list ( colour ) partitions (
partition p_green values ( &#39;green&#39; ),
partition p_red values ( &#39;red&#39; ),
partition p_blue values ( &#39;blue&#39; )
);
<p>Each partition only stores rows where the row&#39;s colour value matches the partition value. And queries searching for a given colour will now only access that partition.</p>
<p>So if you search for all the red rows, the database knows they&#39;re all located in the same place. Which can make your SQL that bit faster.</p>
<p>You can also quickly load rows into a partition from another table with <a href="https://www.akadia.com/services/ora_exchange_partition.html">partition exchange</a>. But to use this the two tables must have identical structures. Which can be <a href="https://blogs.oracle.com/oraclemagazine/open-for-exchange">hard to validate</a>.</p>
<p>So Oracle Database 12.2 introduced a new create table option. Create-table-for-exchange:</p>
create table toys_stage
for exchange with table toys_partitioned;
<p>This makes a non-partitioned table with the same structure as the source. Ensuring you&nbsp;can exchange it with the partitioned table.</p>
<p>Partitioning can bring huge benefits. But get it wrong and you can make your database an unmaintainable mess! Read up on the pros and cons before jumping in.</p>
<p>To learn more, read my colleague Connor&#39;s <a href="https://asktom.oracle.com/partitioning-for-developers.htm">guide to partitioning for developers</a>.</p>
<p>So far we&#39;ve dealt with making queries against one table faster. But what if you want to get rows from two tables at the same time?</p>
<p>Consider:</p>
<a name="create-cluster"></a>
Table Clusters
<p>Joining tables is one of the most common operations in a database. To do this, first the database reads rows from one table. Then finds rows matching the join criteria in the other.</p>
<p><span data-offset-key="41j8o-0-0">So you need at least two lots of reads get the data. One for each table. In practice joins can do many more.</span></p>
<p>Table clusters avoid this &quot;join penalty&quot; by storing rows from different tables in the same place. <span data-offset-key="a3s7e-0-0">Rows with the same value for the cluster key from each table go in the same location. This means you get &quot;two-for-the-price-of-one&quot; access. Instead of an I/O operation per table you query, you can get all the rows in one shot.</span></p>
<p>Say you have a lookup table for colours. And you often join the toy table to this on the colour. Clustering these tables by colour ensures the database stores rows with the same colour in the same place.</p>
<p>To use table clusters, first you need to create the cluster. This can be a hash or index cluster. The following creates a hash cluster:</p>
create cluster colour_clus (
colour varchar2(10)
) hashkeys 1024;
<p>The cluster&#39;s columns are its key. Place the tables in this by adding the cluster clause to your create table. Here you state the cluster columns. These must have the same data type as in the cluster.<span style="display: none;">&nbsp;</span></p>
<p>The following adds colours_clustered and toys_clustered to the colour_clus:</p>
create table colours_clustered (
colour varchar2(10),
rgb_hex_value varchar2(6)
) cluster colour_clus ( colour );
create table toys_clustered (
toy_name varchar2(10),
weight number,
colour varchar2(10)
) cluster colour_clus ( colour );
<p>Now, if you insert a row&nbsp;with the colour red in both colours_clustered and toys_clustered, the database will stick them in the same place.&nbsp;</p>
<p>Table clusters are an advanced feature. While they can make joins faster, they come with several caveats. <a href="https://www.oracle.com/pls/topic/lookup?ctx=dblatest&amp;id=CNCPT608">Read up on them</a> before diving in!</p>
<a name="create-choice"></a>
So Which Type Should I Create?!
<p>Phew!</p>
<p>That was a lot of options! And there are few other, more specialized types available too. For a full list of options, read up on <a href="https://www.oracle.com/pls/topic/lookup?ctx=dblatest&amp;id=GUID-096986C4-9AD7-401D-BA6D-EF6CD4B494FE">tables and table clusters</a> in the Concepts Guide. Or view the full <a href="https://www.oracle.com/pls/topic/lookup?ctx=dblatest&amp;id=GUID-F9CE0CC3-13AE-4744-A43C-EAC7A71AAAB6">create table</a> syntax.</p>
<p>So now you may be wondering: which type should I use?</p>
<p>In most cases, a heap table is the way to go. These are the most versatile.</p>
<p>But think about how you&#39;ll access the rows in the table. Will you have one or two queries that need to be fast as possible? If so, an index-organized or partitioned table may be the way to go.</p>
<p>Or will it need to support many queries on different columns? In which case a default heap table is the better option.</p>
<p>And bear in mind you can combine some options. For example, you can have a partitioned IOT. Knowing which to use will come from experience. But more importantly testing!</p>
<p>So get familiar with the table types available. Play around with them to see how they work. When building new functionality, try different table settings. And test to see how they perform.</p>
<p>But making new tables is a small part of database development. Often you&#39;ll need to change existing tables. It&#39;s time to find out how to:</p>
<a name="alter"></a>
How to Alter Tables
<img alt="" src="https://cdn.app.compendium.com/uploads/user/e7c690e8-6ff9-102a-ac6d-e4aebca50425/6a8a9853-c115-4f9b-b14e-7eee344d2c98/Image/ef1327cc0e657d9536dec64a53790dc6/tools_2423826_1920.jpg" style="width: 1920px; height: 1384px;" />
<p style="text-align: center;"><a href="https://pixabay.com/en/tools-carpenter-wood-2423826/">Pixabay</a></p>
<p>So you&#39;ve created a shiny new table. But the chances are most of your development will be against existing tables. Tables you&#39;ll need to extend to store new information. So you need to add extra columns.</p>
<a name="alter-add-columns"></a>
How to Add Columns
<p>To add columns to your table, you need an alter table statement. This lists the table you&#39;re changing, with the names and data types of the columns you want to add. So to add a price column to toys, run:</p>
alter table toys add ( price number );
<p><span data-offset-key="2co8e-0-0">If you want to add more than one column, you can list them out in the add clause:</span></p>
alter table toys add (
&nbsp; cuddliness_factor&nbsp; &nbsp; integer,
&nbsp; quantity_of_stuffing integer
);
<p>So that&#39;s easy enough. But what about columns you no longer need? How do you get rid of them?</p>
<a name="alter-drop-columns"></a>
How to Drop Columns
<p>Every now and then you may want to remove a column from a table. Maybe the business has realised each toy has many colours. So there&#39;s no point storing this on the toys table. Thus you want remove this column.</p>
<p>You can do so with an alter table drop command:</p>
alter table toys drop ( weight );
<p>But beware!</p>
<p>This is an expensive operation. On tables storing millions of rows it will take a long time to run. And it blocks other changes to the table. So your application may be unusable while this runs!</p>
<p>A better option is to set the column unused:</p>
alter table toys set unused column weight;
<p>This is an instant operation. No matter how many rows are in the table, it will take the same amount of time.</p>
<p>This is because it doesn&#39;t physically remove the columns from the database. It marks them as unavailable. The data still exists. You just can&#39;t get to it!</p>
<p>If you&#39;re hoping to reclaim the space these columns used, you need to wipe them from the table. Do this with the following command:</p>
alter table toys drop ununsed columns;
<p>As you&#39;re now doing the work of deleting the data, this can take a long time. The key difference here is dropping unused columns is non-blocking. Your application can continue to run as normal.</p>
<p>Whichever method you use, take care when removing columns. Dropping columns or setting them unused are both one-way operations. There is no &quot;undrop column&quot; command. If you made a mistake, you&#39;ll have to recover the table from a backup!</p>
<p>So that helps if you&#39;re removing some of the columns. But what if you want to scrap the whole table?</p>
<p>It&#39;s time to drop it!</p>
<a name="drop"></a>
How to Drop Tables
<p>Sometimes you may want to delete a whole table. To do this, drop the table like so:</p>
drop table toys;
<p>This removes the whole table and its data. Like dropping columns, this is a one-way operation.</p>
<p>But, luckily, if you have <a href="https://www.oracle.com/pls/topic/lookup?ctx=dblatest&amp;id=GUID-03D1CAAE-D940-444A-8771-B1BC636D105D">Oracle Database&#39;s Flashback features</a> enabled, <a href="https://blogs.oracle.com/sql/how-to-recover-data-without-a-backup">you can </a><a href="https://blogs.oracle.com/sql/how-to-recover-data-without-a-backup">undrop</a><a href="https://blogs.oracle.com/sql/how-to-recover-data-without-a-backup"> a table</a>! So if you accidentally run your rollback script in production, you can get going again :)</p>
<p>Starting your SQL journey and want to know more? The video above is from the first module of <a href="https://devgym.oracle.com/pls/apex/dg/class/databases-for-developers-foundations.html">Databases for Developers: Foundations</a>. This beginner&#39;s course teaches you SQL.</p>
<p>The best part?</p>
<p>It&#39;s 100% FREE!</p>
<p>Head to the <a href="https://devgym.oracle.com/pls/apex/dg/class/databases-for-developers-foundations.html">registration page</a> to join.</p>
Mon, 15 Oct 2018 13:04:00 +0000https://blogs.oracle.com/sql/how-to-create-alter-and-drop-tables-in-sqlChris SaxonHow to Create Users, Grant Them Privileges, and Remove Them in Oracle Databasehttps://blogs.oracle.com/sql/how-to-create-users-grant-them-privileges-and-remove-them-in-oracle-database
<p>So, you&rsquo;ve got your shiny, brand new Oracle Database up and running. It&rsquo;s time to start creating users!</p>
<p>But how do you do this?</p>
<img alt="" src="https://cdn.app.compendium.com/uploads/user/e7c690e8-6ff9-102a-ac6d-e4aebca50425/6a8a9853-c115-4f9b-b14e-7eee344d2c98/Image/b27afdd0d1f0b3d8f69487b77252f975/gratisography_346h_small.jpg" style="width: 1482px; height: 988px;" />
<p style="text-align: center;">Ryan McGuire <a href="https://gratisography.com/">Gratisography</a></p>
<p>First you&rsquo;ll need login as system or sys. Once you&rsquo;re in, the basic create user command is:</p>
create user &lt;username&gt; identified by &quot;&lt;password&gt;&quot;;
<p>So to create the user data_owner with the password Supersecurepassword!, use:</p>
create user data_owner identified by &quot;Supersecurepassword!&quot;;
<p>Now you&rsquo;ve got your user. The next step is to connect to it. But try to do so and you&rsquo;ll hit:</p>
conn data_owner/Supersecurepassword!
ORA-01045: user DATA_OWNER lacks CREATE SESSION privilege; logon denied
<p>What&rsquo;s going on?</p>
<p>The problem is you haven&rsquo;t given the user any permissions! By default a database user has no privileges. Not even to connect.</p>
Granting User Privileges
<p>You give permissions with the grant command. For system privileges this takes the form:</p>
grant &lt;privilege&gt; to &lt;user&gt;
<p>To allow your user to login, you need to give it the create session privilege. Let&rsquo;s do that:</p>
grant create session to data_owner;
<p>There are a <a href="https://docs.oracle.com/en/database/oracle/oracle-database/18/sqlrf/GRANT.html#GUID-20B4E2C0-A7F8-4BC8-A5E8-BE61BDC41AC3__BABEFFEE">whole raft of other permissions</a> you can give your users. And some <a href="https://docs.oracle.com/en/database/oracle/oracle-database/18/dbseg/configuring-privilege-and-role-authorization.html#GUID-A5B26A03-32CF-4F5D-A6BE-F2452AD8CB8A">rather powerful roles</a> that grant them all.</p>
<p>So what should you enable?</p>
<p>At this point, keen to get developing, you may be tempted to give your user a bucket of powerful permissions.</p>
<p>Before you do, remember a key security concept:</p>
<p><a href="https://en.wikipedia.org/wiki/Principle_of_least_privilege">The Principle of Least Privilege</a>.</p>
<p>Only give your users the smallest set of privileges they need to do their job. For a basic data schema that&rsquo;s simply create table:</p>
grant create table to data_owner;
<p>This allows you to make tables. As well as indexes and constraints on them. But critically, not store data in them!</p>
<p>Which is could lead to embarrassing errors when deploy your brand new application:</p>
conn data_owner/Supersecurepassword!
create table customers (
customer_id integer not null primary key,
customer_name varchar2(100) not null
);
insert into customers values ( 1, &#39;First customer!&#39; );
ORA-01950: no privileges on tablespace &#39;USERS&#39;
<p>To avoid this, you need to give your user a tablespace quota. You&#39;ll want to do this on their default tablespace. Which you can find with:</p>
select default_tablespace from dba_users
where username = &#39;DATA_OWNER&#39;;
DEFAULT_TABLESPACE
USERS
<p>Assign the quota by altering the user, like so:</p>
alter user data_owner quota unlimited on users;
<p>These privileges will get you far. <span data-offset-key="ev76k-0-0">But to build an application there are a few other privileges you&rsquo;re likely to need:</span></p>
<ul>
<li>create view &ndash; Allows you to create views</li>
<li>create procedure &ndash; Gives the ability to create procedures, functions and packages</li>
<li>create sequence &ndash; The ability to make sequences</li>
</ul>
<p>You can give many system privileges in one go. <span data-offset-key="9u5vb-0-0">Grant these to data_owner by chaining them together like so:</span></p>
grant create view, create procedure, create sequence to data_owner;
<p>Notice the lack of &ldquo;drop &lt;object type&gt;&rdquo; access. That&rsquo;s because database users always have full privileges on their own objects. Meaning you can run any queries against your own tables. And insert, update, and delete rows however you like. And drop them!</p>
<p>Which brings a possible security loophole.</p>
<p>If your application connects to the database as the user which owns the tables, if you have any <a href="https://blogs.oracle.com/sql/what-is-sql-injection-and-how-to-stop-it">SQL injection vulnerabilities</a>&nbsp;you&rsquo;re in trouble!</p>
<p>To avoid this, separate the connection user and the data schema. Ideally with a PL/SQL API between your tables and the users.</p>
<img alt="" src="https://cdn.app.compendium.com/uploads/user/e7c690e8-6ff9-102a-ac6d-e4aebca50425/6a8a9853-c115-4f9b-b14e-7eee344d2c98/Image/ccf131579cb220731629c8a33d05fb59/least_privilege.png" style="width: 1125px; height: 626px;" />
<p>To learn more about protecting your database behind a PL/SQL API, head to the&nbsp;<a href="https://stevenfeuersteinonplsql.blogspot.com/2018/05/the-smartdb-resource-center.html">SmartDB resource center</a>.</p>
<p>So to secure your data, you need to create another user. The only system privilege you should give it is create session.</p>
<p>Great, another two statements you&#39;re thinking.</p>
<p>Luckily there&rsquo;s a shortcut. You can create a user and grant it system privileges in one go!</p>
<p>Just add the identified by clause to grant:</p>
grant create session to app_user identified by &quot;theawesomeeststrongestpassword&quot;;
<p>If the user already exists this will grant the privileges. And reset the password. So take care when running this, or you may change their password!</p>
Password Management
<p>A brief note on password rules. By default the password will expire every 180 days. Which can lead to <a href="https://blogs.oracle.com/sql/how-to-fix-ora-28002-the-password-will-expire-in-7-days-errors">ORA-28002 errors</a> on login.&nbsp;</p>
<p>Not only is this kinda annoying, it goes against <a href="https://www.troyhunt.com/passwords-evolved-authentication-guidance-for-the-modern-era/#donotmandateregularpasswordchanges">current password guidelines</a>. You can get around this by <a href="https://blogs.oracle.com/sql/how-to-fix-ora-28002-the-password-will-expire-in-7-days-errors#database-user-profiles">changing the password_life_time</a> for the user&#39;s profile.</p>
<div data-block="true" data-editor="ej0eg" data-offset-key="3pbqu-0-0">
<div class="public-DraftStyleDefault-block public-DraftStyleDefault-ltr" data-offset-key="3pbqu-0-0"><span data-offset-key="3pbqu-0-0">While you&#39;re at it, you </span><span class="adverb">probably</span><span data-offset-key="3pbqu-2-0"> want to stop people picking short, easy to crack passwords. You can define a <a href="http://www.oracle.com/pls/topic/lookup?ctx=dblatest&amp;id=GUID-00BED08C-E064-4824-9697-C7219CB8D3AB">password complexity function</a> to do this.&nbsp;</span></div>
</div>
<p>So you&rsquo;ve created your application user.</p>
<p>But you still need to assign it permissions on data_owner&rsquo;s objects. For table level access, you can give access to query and change the rows with:</p>
grant select, insert, update, delete on data_owner.customers to app_user;&nbsp;
<p>There is a &quot;grant all&quot; option for tables. But before you reach for this, be aware that not only does it include the DML permissions above, it also gives:</p>
<ul>
<li>alter</li>
<li>debug</li>
<li>flashback</li>
<li>index</li>
<li>on commit refresh</li>
<li>query rewrite</li>
<li>read</li>
<li>references</li>
</ul>
<p>Ouch!</p>
<p>Remember: only give out the exact permissions users need. No more!</p>
<p><span data-offset-key="2cju3-0-0">If you have done the good thing and protected your data behind a PL/SQL API, grant execute to allow app_user to call it. Like so:</span></p>
grant execute on data_owner.customers_api to app_user;
<p>You can only grant permissions on one object at a time. So you&rsquo;ll need to repeat this for each thing app_user needs access to.</p>
<p>To give these object privileges, you need to either:</p>
<ul>
<li>Own the object in question</li>
<li>Have the <em>grant any object privilege</em> privilege</li>
<li>Have been granted the permission using the <em>with grant option</em></li>
</ul>
<p>As a rule you should avoid giving out &quot;any&quot; privileges. So in most cases you should only grant object privileges when connected as the object owner.</p>
<p><span data-offset-key="ci6qe-0-0">But you may want to have a low-level admin user. You&#39;ll use this to grant permissions to other users. Such as the ability to query some of data_owner&#39;s tables for reporting. If you&#39;re feeling lazy, grant allows you to create many users in one go:</span></p>
grant create session to reporting_admin, report_user_1
identified by &quot;theadminpassword&quot;, &quot;theuserpassword&quot;;
<p>Now, to allow&nbsp;reporting_admin to give query privileges on data_owner&#39;s objects to report_user_1, you can:</p>
<ul>
<li>Connect to data_owner</li>
<li>Grant query permissions with grant option</li>
<li>Connect to reporting_admin to pass these permissions onto others</li>
</ul>
<p>Like so:</p>
conn data_owner/Supersecurepassword!
grant read on customers to reporting_admin with grant option;
conn reporting_admin/theadminpassword
grant read on data_owner.customers to report_user_1;
<p>Note the grant of read instead of select. This is a <a href="https://oracle-base.com/articles/12c/read-object-privilege-12cr1">new privilege in Oracle Database 12c</a>. Granting select allows users to lock tables. Read doesn&#39;t. So you should give this privilege to read-only users instead of select.</p>
<p>So you&#39;ve given your application users the smallest set of privileges they need.</p>
<p>You&#39;ve locked the front door. But there&rsquo;s still a backdoor!</p>
<img alt="" src="https://cdn.app.compendium.com/uploads/user/e7c690e8-6ff9-102a-ac6d-e4aebca50425/6a8a9853-c115-4f9b-b14e-7eee344d2c98/Image/41421017c60c6e79c2204684eb5745c9/gratisography_328h_small.jpg" />
<p style="text-align: center;">Ryan McGuire <a href="https://gratisography.com/">Gratisography</a></p>
<p>Anyone with access to your network can connect as data_owner. At which point they&rsquo;re free to wreak havoc in your database.</p>
<p>This is a tricky problem to avoid. You can stop people getting in by locking the account with:</p>
alter user data_owner account lock;
<p>But this brings a couple of issues.</p>
<p>First up, it&rsquo;s easy to overlook this step. If you want to connect to data_owner, say to release some changes, you&rsquo;ll need to unlock it. And remember to lock it again afterwards! A step easily forgotten when dealing with emergency releases.</p>
<p>But there&rsquo;s another problem. It allows hackers to easily discover the names of your database users. When you try and connect to a locked account, you&rsquo;ll get the following message:</p>
conn data_owner/random_password
ORA-28000: The account is locked.
<p>If I&rsquo;m phishing around your database, I now know it contains the user data_owner. Even though I don&rsquo;t know the password!</p>
<p>Now, hopefully(!), your network security is good enough that hackers can&rsquo;t scan through possible usernames to find the names of your accounts.</p>
<p><span data-offset-key="49h0h-0-0">But this trick is a quick way for them to see if your database has Oracle supplied users installed. Things like Oracle Text or Oracle Spatial. If you have, this increases the options for a hacker to get in.</span></p>
<p>So what do you do?</p>
<p>Luckily Oracle Database 18c offers another way around this problem: schema-only accounts!</p>
Schema vs. User
<p>At this point it&rsquo;s worth noting the difference between schemas and users. Officially a schema is a collection of tables. Whereas a user is an account you use to connect to the database. Some databases allow you to make a distinction between these with separate create schema and create user commands.</p>
<p>But in Oracle Database, there&rsquo;s no difference between a schema and a user. All tables belong to one user.</p>
<p>While the&nbsp;<a href="https://blogs.oracle.com/sql/creating-multiple-tables-in-a-single-transaction">create schema command exists</a>, you can only use it to <a href="https://blogs.oracle.com/sql/how-to-create-alter-and-drop-tables-in-sql">create tables</a> within an existing user.</p>
<p>So &quot;schema-only&quot; accounts are users which have no password. To create one, use the no authentication clause instead of identified by:</p>
create user data_owner no authentication;
<p>Now there is literally no way to login to this account. Any attempts to do so will hit:</p>
conn data_owner/random_password
ORA-01017: invalid username/password; logon denied
<p>So you no longer know if data_owner is a valid account.</p>
<p>Is the user missing? Or are they present, but you&rsquo;ve got the password wrong? You don&rsquo;t know.</p>
<p>So you&rsquo;ve stopped hackers learning about your database. Great. But.</p>
<p>You&rsquo;re probably thinking:</p>
<p>How do <em>I</em> connect to data_owner?</p>
<p>From time-to-time it&rsquo;s likely you&rsquo;ll want to connect to do things like run release scripts.</p>
<p>Sure, you can assign a temporary password with:</p>
alter user data_owner identified by &quot;Supersecurepassword!&quot;;
<p>And remove it again when you&rsquo;re done with:</p>
alter user data_owner no authentication;
<p>But this is a repeat of the lock problem again. What if you forget to remove authentication when you&rsquo;re done?</p>
<p>Luckily, there&rsquo;s a solution: proxy users.</p>
<img alt="" src="https://cdn.app.compendium.com/uploads/user/e7c690e8-6ff9-102a-ac6d-e4aebca50425/6a8a9853-c115-4f9b-b14e-7eee344d2c98/Image/04268a08668f5f36f28388a9859ad6be/gratisography_207h.jpg" style="width: 1440px; height: 960px;" />
<p style="text-align: center;">Ryan McGuire <a href="https://gratisography.com/">Gratisography</a></p>
Proxy Users
<p>Proxy users are low privilege accounts. With the ability to connect to higher powered users.</p>
<p>To use them, you need to create the user. And give it the power to connect through another account:</p>
grant create session to proxy_user identified by &quot;proxy_user_password&quot;;
alter user data_owner grant connect through proxy_user;
<p>With this in place, you can now connect to proxy_user. But run with the privileges of data_owner. Do so with:</p>
conn proxy_user[data_owner]/proxy_user_password
<p>Using this method, you can leave your <span class="adverb">schema-only</span><span data-offset-key="7qf5e-2-0"> accounts with no password.</span></p>
Removing Access
<p>Over time applications get decommissioned. Or rewritten to access different information. But usually the data remains.</p>
<p>Leaving the user with access to unneeded data is a security risk. Stay on top of this and remove access when it&rsquo;s no longer needed.</p>
<p>To do this, use the revoke command. This states what you&rsquo;re removing from who. For system privileges this is:</p>
revoke create table from data_owner;
<p>For object privileges, include the thing you&#39;re removing access from:</p>
revoke select on data_owner.important_stuff from app_user;
<p>Remember: if your release scripts have grants for existing objects you&#39;ll need to undo these if you have to rollback<span data-offset-key="6d1ad-1-0">. So ensure you include the corresponding revoke in your rollback scripts!</span></p>
Dropping Users
<p>Getting rid of unwanted users is easy. Drop them with:</p>
drop user &lt;username&gt;;
<p>You can only do this if the user is not connected to the database. So ensure you clear up any sessions it has before you do so.</p>
<p>And there&rsquo;s another step you need to watch for. Run this for data_owner and you&rsquo;re likely to hit this error:</p>
drop user data_owner;
ORA-01922: CASCADE must be specified to drop &#39;DATA_OWNER&#39;
<p>Why?</p>
<p>You can&rsquo;t remove users that own objects!</p>
<p>So you need to go in and drop all its tables, views, etc. Or do it in one shot with:</p>
drop user data_owner cascade;
<p>This is an easy way to wipe <em>all</em> your data. So use with care!</p>
<p>Want to know more?</p>
<p>Read up on <a href="http://www.oracle.com/pls/topic/lookup?ctx=dblatest&amp;id=GUID-F0246961-558F-480B-AC0F-14B50134621C">create user</a>, <a href="http://www.oracle.com/pls/topic/lookup?ctx=dblatest&amp;id=GUID-F766E1A2-6686-4734-89BA-0C5B4120B90E">drop user</a>,&nbsp;<a href="http://www.oracle.com/pls/topic/lookup?ctx=dblatest&amp;id=GUID-20B4E2C0-A7F8-4BC8-A5E8-BE61BDC41AC3">grant</a>, and <a href="http://www.oracle.com/pls/topic/lookup?ctx=dblatest&amp;id=GUID-BAAD2331-40A5-4366-86CA-BAA6B957E866">revoke</a> in the documentation.&nbsp;</p>
<p>Got any questions about creating users or managing their permissions?</p>
<p>Ask in the comments!</p>
SQLMon, 30 Jul 2018 10:10:00 +0000https://blogs.oracle.com/sql/how-to-create-users-grant-them-privileges-and-remove-them-in-oracle-databaseChris SaxonHow to Fix ORA-28002 The Password Will Expire in 7 Days Errorshttps://blogs.oracle.com/sql/how-to-fix-ora-28002-the-password-will-expire-in-7-days-errors
<p>The weekend is over. You&rsquo;re back at work. You login to the database to start coding.</p>
<p>Only to be hit with:</p>
<p><img alt="ORA-28002: the password will expire in 7 days" src="https://cdn.app.compendium.com/uploads/user/e7c690e8-6ff9-102a-ac6d-e4aebca50425/6a8a9853-c115-4f9b-b14e-7eee344d2c98/Image/11cea1649e181b5e44f7e29f548187af/ora_28002_pwd_expiry.png" style="width: 428px; height: 212px; border-style: solid; margin:0px auto;display:block;" /></p>
<p>Gaaaaaahh!</p>
<p>Not again. It&rsquo;s time for the bi-annual trip to the DBA to get them to change the password.</p>
<p>It&rsquo;d be kinda nice to avoid this and use the same password forever.</p>
<p>But how?</p>
Database User Profiles
<p>When you create a database user it <a href="https://docs.oracle.com/en/database/oracle/oracle-database/18/cncpt/topics-for-database-administrators-and-developers.html#GUID-364F918B-B228-4A42-9F5F-4E430290651E">has a profile</a>. Which, by default, is the &ldquo;DEFAULT&rdquo; profile. This defines many things. Such as CPU, I/O and memory limits (though we recommend you use the <a href="https://www.oracle.com/pls/topic/lookup?ctx=en/database/oracle/oracle-database/18/sqlrf&amp;id=ADMIN027">Database Resource Manager</a> for these instead).</p>
<p>And the password policy.</p>
<p>Which expires passwords every 180 days.</p>
<p>To stop this and allow users to keep their password forever, alter the profile like so:</p>
alter profile &quot;DEFAULT&quot; limit
password_life_time unlimited;
<p>Or create a new policy and assign that to your users:</p>
create profile unlimited_pwd_prof limit
password_life_time unlimited;
alter user &lt;username&gt; profile unlimited_pwd_prof;
<p>Once you&rsquo;ve done this you may think everything&rsquo;s good. So you&rsquo;re surprised when you login again and <em>still</em> hit the ORA-28002 error.</p>
<p>Or, if enough time has elapsed, you now see ORA-28001:</p>
<p><img alt="ORA-28001: the password has expired" src="https://cdn.app.compendium.com/uploads/user/e7c690e8-6ff9-102a-ac6d-e4aebca50425/6a8a9853-c115-4f9b-b14e-7eee344d2c98/Image/e90135b61939828eeaaa7f20a7c9c660/ora_28001.png" style="width: 415px; height: 212px; border-style: solid; margin:0px auto;display:block;" /></p>
<p>What&rsquo;s going on here?</p>
<p>The problem is you&#39;ve entered the grace period. This starts after password_life_time days have elapsed since the last password change. By default it runs&nbsp;for seven days. If you want to increase this time to say, two weeks, run:</p>
alter profile unlimited_pwd_prof limit
password_grace_time 14;
<p>During this time you can still login, but will get &quot;the password will expire&quot; warnings. After the number of days defined in the&nbsp;<span>password_grace_time have passed, the password expires.</span></p>
<p>The only way out of either situation is to reset the password!</p>
<p>To do so, run:</p>
alter user &lt;username&gt; identified by &lt;password&gt;;
<p>Note you can &ldquo;change&rdquo; the password back to itself. Which is generally considered a bad idea. Luckily you can force new passwords by setting the password reuse limit and days.</p>
<ul>
<li>password_reuse_max - the number of new passwords you must use before you return to an earlier one</li>
<li>password_reuse_time - the number of days that must elapse before you can reuse a password</li>
</ul>
<p>If you set both of these to a value other than unlimited, users must hit both criteria to change it. So to force at least ten new passwords <strong>and </strong>one year before you can repeat a password, update the profile like so:</p>
alter profile unlimited_pwd_prof limit
password_reuse_max 10
password_reuse_time 365;
<p>While you&rsquo;re sorting this out, you may want to ensure people choose &ldquo;strong&rdquo; passwords. Oracle Database 12.2 supplies some&nbsp;<a href="https://docs.oracle.com/en/database/oracle/oracle-database/18/dbseg/configuring-authentication.html#GUID-00BED08C-E064-4824-9697-C7219CB8D3AB">password complexity functions</a>:</p>
<ul>
<li>ora12c_verify_function&nbsp;<span><span>(also in 12.1)</span> </span></li>
<li>ora12c_strong_verify_function&nbsp;<span><span>(also in 12.1)</span> </span></li>
<li>ora12c_stig_verify_function</li>
<li>verify_function_11G (now deprecated)</li>
</ul>
<p>You can enable these by setting the password_verify_function in the profile.&nbsp;</p>
<p>Or you can create your own complexity function. This must accept three parameters (username, password, and old_password) and return Boolean. You must also create this function in the sys schema. One of the rare cases where you should create <em>anything</em> in sys!</p>
<p>So to ensure that all passwords are at least twenty characters long, do the following:</p>
create or replace function verify_password_length (
username varchar2, password varchar2, old_password varchar2
) return boolean as
begin
return ( length ( password ) &gt;= 20 );
end verify_password_length;
/
grant execute on verify_password_length to public;
You can then assign it to an existing profile or create a new one like so:
create profile long_password limit
password_verify_function verify_password_length;
<p>Note this only affects new passwords. If your database is full of users with weak passwords you need to reset them. If you want someone else to own the password for a user, you can force them to choose a new one by expiring the current:</p>
alter user &lt;username&gt; password expire;
<p>And they&#39;ll be faced with an ORA-28001 error when they next login!</p>
<p>But managing all these password settings is a hassle. And, chances are, you already have a central place to manage users and password policies.</p>
<p>Active Directory.</p>
<p>It&rsquo;d be kinda nice to authenticate against that. Which you can. Using Oracle Internet Directory.</p>
<p>This sits between the database and AD, mapping users between the two.</p>
<p>But this is another thing to setup, manage and is generally a faff.</p>
<p>Luckily in Oracle Database 18c this is a lot easier.</p>
<p>You can cut out the middleman and map database users <a href="https://docs.oracle.com/en/database/oracle/oracle-database/18/dbseg/integrating_mads_with_oracle_database.html#GUID-9739D541-FA9D-422A-95CA-799A4C6F488D">directly to Active Directory users</a>!</p>
<p><img alt="" src="https://cdn.app.compendium.com/uploads/user/e7c690e8-6ff9-102a-ac6d-e4aebca50425/6a8a9853-c115-4f9b-b14e-7eee344d2c98/Image/3d8ef7fd57e5ba1d75d1795dac04fab5/ad_authentication.png" style="width: 1280px; height: 648px;" /></p>
<p>This allows you to use AD as your central user management system. So you have one place to define password policies such as how many failed logins you allow.</p>
<p>How do you manage passwords in Oracle Database? Any other gotchas to be aware of? Let us know in the comments!</p>
SQLThu, 29 Mar 2018 14:12:00 +0000https://blogs.oracle.com/sql/how-to-fix-ora-28002-the-password-will-expire-in-7-days-errorsChris SaxonAnnouncing the 2017 SQL and Database Design Championships Winners on Oracle Dev Gymhttps://blogs.oracle.com/sql/announcing-the-2017-sql-and-database-design-championships-winners-on-oracle-dev-gym
<p>The dust has settled. The results are in. It&#39;s time to announce the winners of the <a href="https://devgym.oracle.com/">Oracle Dev Gym</a> annual championships for SQL and Database Design!</p>
<p><img alt="" src="https://cdn.app.compendium.com/uploads/user/e7c690e8-6ff9-102a-ac6d-e4aebca50425/6a8a9853-c115-4f9b-b14e-7eee344d2c98/Image/d9b12210f064d52a159353e5d1dfebfb/chess_1483735_1920.jpg" style="width: 1920px; height: 1313px;" /></p>
<p>Both competitions took place over a series of five quizzes. These pushed players to the limits of their knowledge of Oracle Database, covering a wide range of topics. These included <a href="https://docs.oracle.com/en/database/oracle/oracle-database/18/dwhsg/sql-pattern-matching-data-warehouses.html">SQL pattern matching</a>, <a href="https://docs.oracle.com/en/database/oracle/oracle-database/18/adobj/index.html">object-relational features</a>, and <a href="https://docs.oracle.com/en/database/oracle/oracle-database/18/adjsn/index.html">JSON</a>&nbsp;in the database.</p>
<p>A huge thanks to ACED <a href="http://www.kibeha.dk/">Kim Berg Hansen</a>, author of the SQL quizzes. For over seven years he&#39;s created weekly, high-quality quizzes. These have helped thousands of people learn how to write better SQL.</p>
<p>And now, onto the winners!</p>
Database Design Winners
<p style="margin-left: 40px;">1st Place: <a href="https://devgym.oracle.com/pls/apex/f?p=10001:PROFILE:107353309110309::NO::P26_USER_ID:4294&amp;cs=1DsRzN0lU-TIsvLNIHVbFmb3pEVZ7ilGXlEwb786wysggYN7qfT_sYn5qruchnLXNfEKm_OsP-8rjAb1U9NKPOQ">mentzel</a><a href="https://devgym.oracle.com/pls/apex/f?p=10001:PROFILE:107353309110309::NO::P26_USER_ID:4294&amp;cs=1DsRzN0lU-TIsvLNIHVbFmb3pEVZ7ilGXlEwb786wysggYN7qfT_sYn5qruchnLXNfEKm_OsP-8rjAb1U9NKPOQ">.iudith</a> of Israel<br />
2nd Place: <a href="https://devgym.oracle.com/pls/apex/f?p=10001:PROFILE:103972693580129::NO::P26_USER_ID:487&amp;cs=18hyqy7W0tdf3bgdyfPTS2PywjFd3UAgvN8KbOrCvzVDwXExXbjkqP6-RQ4_Fae5Dw5A4kU07VqVQ0uxPDaWlaQ">NielsHecker</a> of Germany<br />
3rd Place: <a href="https://devgym.oracle.com/pls/apex/f?p=10001:PROFILE:103972693580129::NO::P26_USER_ID:3996&amp;cs=1IWDyG86inlf0ROJlAr7eTS66HXRP7sSOgVYzbtjc-5OkR-FFFu2W949iKpHTjTggKg9FJ99GT_9JecFtVgDz0A">Ivan Blanarik</a> of Slovakia</p>
SQL Winners
<p style="margin-left: 40px;">1st Place: <a href="https://devgym.oracle.com/pls/apex/f?p=10001:PROFILE:107353309110309::NO::P26_USER_ID:4294&amp;cs=1DsRzN0lU-TIsvLNIHVbFmb3pEVZ7ilGXlEwb786wysggYN7qfT_sYn5qruchnLXNfEKm_OsP-8rjAb1U9NKPOQ">mentzel</a><a href="https://devgym.oracle.com/pls/apex/f?p=10001:PROFILE:107353309110309::NO::P26_USER_ID:4294&amp;cs=1DsRzN0lU-TIsvLNIHVbFmb3pEVZ7ilGXlEwb786wysggYN7qfT_sYn5qruchnLXNfEKm_OsP-8rjAb1U9NKPOQ">.iudith</a> of Israel<br />
2nd Place: <a href="https://devgym.oracle.com/pls/apex/f?p=10001:PROFILE:107353309110309::NO::P26_USER_ID:3390&amp;cs=1ngDnS2JSvS__o6rt1vA8nqJlRbwgIIOqQ-X-oBI-YWg9Gd2k5w-P5QFC8ZmVZ3kotnncGPec8vIyZNPpZXomTg">Pavel Zeman</a> of Czech Republic<br />
3rd Place: <a href="https://devgym.oracle.com/pls/apex/f?p=10001:PROFILE:107353309110309::NO::P26_USER_ID:24095&amp;cs=1Wva-gi2SINJN0nMWOu03jjgAJYTWuEPrxvY5PBf6DMwCer4DSbBPS_G93vgWUGjLxZt-dzwkjdojUOG8ZLccCg">Andrey Zaytsev</a> of Russia</p>
<p>Huge congratulations to long-time player Iudith Mentzel for taking the top spot in both competitions! Her dedication to learning how Oracle Database works and improving her skills is outstanding. She thoroughly deserves both awards.</p>
<p>I also want to give supreme thanks to Elic. His efforts in reviewing the quizzes once again ensured a top-quality competition to challenge the contestants.</p>
<p>Think you&#39;ve got what it takes to win next year&#39;s championships?</p>
<p>To qualify you need to be a top-ranking player in the ongoing weekly quizzes in 2018. Head over to the <a href="https://devgym.oracle.com/">Oracle Dev Gym</a>&nbsp;to take part, learn and join the fun!</p>
Full Database Design Championship Rankings
Rank
Name
Country
Total Time
% Correct
Score
1
mentzel.iudith (4)
Israel
30 m
88%
5177
2
NielsHecker (4)
Germany
34 m
81%
4811
3
Ivan Blanarik (4)
Slovakia
24 m
79%
4650
4
JustinCave (4)
United States
21 m
77%
4616
5
msonkoly (2)
Hungary
32 m
74%
4422
6
Sartograph (2)
Germany
34 m
70%
4361
7
JeroenR (3)
No Country Set
18 m
72%
4277
8
Joaquin_Gonzalez (4)
Spain
21 m
70%
4115
9
Maxim Borunov (4)
Russia
34 m
67%
4110
10
mcelaya (2)
Spain
33 m
67%
4015
11
Chase Mei (4)
Canada
21 m
67%
3965
12
Stelios Vlasopoulos (4)
No Country Set
34 m
67%
3960
13
Sandra99 (4)
Italy
20 m
65%
3919
14
Karel_Prech (3)
No Country Set
26 m
65%
3845
15
Mike Tessier (1)
Canada
28 m
63%
3836
16
Eric Levin (4)
United States
22 m
65%
3809
17
Kias (1)
No Country Set
29 m
65%
3781
18
Mehrab (4)
United Kingdom
27 m
63%
3740
19
Rytis Budreika (4)
Lithuania
15 m
63%
3740
20
Jo&atilde;o Borges Barreto (4)
Portugal
29 m
63%
3732
21
berkeso (1)
Hungary
10 m
63%
3708
22
RalfK (1)
Germany
18 m
60%
3526
23
swesley_perth (2)
Australia
11 m
60%
3504
24
Jan &Scaron;er&aacute;k (4)
Czech Republic
26 m
58%
3445
25
ted (2)
United Kingdom
32 m
58%
3421
26
Michal P. (4)
Poland
34 m
53%
3310
27
richdellheim (1)
United States
33 m
53%
3216
28
tonyC (3)
United Kingdom
25 m
53%
3149
29
Cor van Berkel (1)
Netherlands
28 m
51%
3037
30
siimkask (4)
Estonia
17 m
51%
2929
31
Chad Lee (4)
United States
31 m
49%
2826
32
Henry_A (4)
Czech Republic
11 m
44%
2755
33
Pavel Zeman (4)
Czech Republic
33 m
44%
2714
34
Ludovic Szewczyk (1)
No Country Set
34 m
47%
2664
35
NickL (3)
United Kingdom
35 m
44%
2460
36
umir (3)
Italy
34 m
35%
2111
&nbsp;
Full SQL Championship Rankings
<span style="font-weight: 400;">Rank</span>
<span style="font-weight: 400;">Name</span>
<span style="font-weight: 400;">Country</span>
<span style="font-weight: 400;">Total Time</span>
<span style="font-weight: 400;">% Correct</span>
<span style="font-weight: 400;">Score</span>
1
mentzel.iudith (5)
Israel
44 m
86%
5371
2
Pavel Zeman (4)
Czech Republic
44 m
84%
5222
3
Andrey Zaytsev (4)
Russia
39 m
79%
4941
4
Aleksei Davletiarov (3)
Russia
44 m
79%
4923
5
NielsHecker (5)
Germany
44 m
79%
4922
6
JustinCave (5)
United States
35 m
77%
4809
7
Sartograph (2)
Germany
44 m
77%
4772
8
Chase Mei (5)
Canada
24 m
74%
4703
9
Chad Lee (5)
United States
30 m
74%
4677
10
Rytis Budreika (5)
Lithuania
06 m
72%
4625
11
Tony Winn (3)
Australia
24 m
72%
4552
12
Maxim Borunov (4)
Russia
34 m
72%
4513
13
li_bao (5)
Russia
22 m
70%
4411
14
Hertha Rettinger (4)
Germany
32 m
70%
4372
15
mcelaya (2)
Spain
41 m
70%
4335
16
richdellheim (2)
United States
42 m
67%
4180
17
Sandra99 (4)
Italy
21 m
65%
4115
18
Andrii Dorofeiev (4)
No Country Set
19 m
63%
3971
19
Rimantas Adomauskas (3)
Lithuania
21 m
63%
3964
20
seanm95 (5)
United States
24 m
60%
3800
21
NickL (2)
United Kingdom
35 m
60%
3759
22
Eric Levin (5)
United States
16 m
58%
3682
23
Ivan Blanarik (5)
Slovakia
30 m
58%
3626
24
Kias (1)
No Country Set
34 m
58%
3613
25
msonkoly (1)
Hungary
44 m
58%
3571
26
swesley_perth (4)
Australia
15 m
56%
3539
27
berkeso (3)
Hungary
08 m
53%
3415
28
Henry_A (4)
Czech Republic
12 m
53%
3399
29
VictorD (4)
Russia
16 m
53%
3385
30
Narendra Reddy (2)
India
18 m
53%
3375
31
Mike Tessier (2)
Canada
27 m
53%
3342
32
Michal P. (3)
Poland
42 m
53%
3279
33
Karel_Prech (3)
No Country Set
36 m
51%
3155
34
Stelios Vlasopoulos (5)
No Country Set
44 m
51%
3122
35
whab@tele2.at (1)
Austria
25 m
47%
2897
37
Mehrab (3)
United Kingdom
04 m
9%
582
SQLMon, 26 Mar 2018 13:59:30 +0000https://blogs.oracle.com/sql/announcing-the-2017-sql-and-database-design-championships-winners-on-oracle-dev-gymChris SaxonHow to Find the Next Business Day and Add or Subtract N Working Days with SQLhttps://blogs.oracle.com/sql/how-to-find-the-next-business-day-and-add-or-subtract-n-working-days-with-sql
<p>The weekend. It&rsquo;s the time that, hopefully, you get to leave your work behind. And, of course, those days thacreate t throw a spanner in your &ldquo;next business day&rdquo; calculations.</p>
<img alt="" src="https://cdn.app.compendium.com/uploads/user/e7c690e8-6ff9-102a-ac6d-e4aebca50425/6a8a9853-c115-4f9b-b14e-7eee344d2c98/Image/ccfdb286ee3ad5720409a640dbb206a8/442h_medium.jpg" style="width: 1988px; height: 1326px;" />
<p style="text-align: right;">Ryan McGuire / <a href="https://gratisography.com/">Gratisography</a></p>
<p>In this post we&#39;ll find out how to</p>
<ul>
<li>Get the <a href="#next-working-day">next working day</a>, stopping by to look at <a href="#to_char">to_char format masks</a> along the way</li>
<li><a href="#add-n">Add N working days</a> to a date and <a href="#add-n-sql">do this in SQL</a></li>
<li>Find the working days <a href="#days-between-loop">between two dates</a> and the <a href="#days-between-sql">SQL version</a></li>
<li>Deal with <a href="#holidays">public holidays</a>&nbsp;and other non-working days</li>
</ul>
How to Find Next Working Day
<p>Basic date arithmetic in Oracle Database is easy. The number of days between two dates is an integer. So to get the next day, add one to your date. Or, if you&rsquo;re feeling adventurous, you <a href="https://oracle-base.com/articles/misc/oracle-dates-timestamps-and-intervals#interval">can use intervals</a>.</p>
<p>The problem comes when the result is a weekend. In most countries the next working day after Friday is Monday. But adding one to a Friday results in a Saturday.</p>
<p>In this case to advance to the next working day, you need to find the current day of the week and:</p>
<ul>
<li>If it&rsquo;s Friday, add three to the date</li>
<li>If it&rsquo;s Saturday add two to the date</li>
<li>Otherwise add one</li>
</ul>
<p>So first you need to find the day of the week. You can extract this in Oracle Database using to_char with the &ldquo;day&rdquo; format mask.</p>
<p>Plugging this into the above logic gives a case expression like this:</p>
case
when to_char(calendar_day, &#39;fmday&#39;) = &#39;friday&#39; then
calendar_day + 3
when to_char(calendar_day, &#39;fmday&#39;) = &#39;saturday&#39; then
calendar_day + 2
else
calendar_day + 1
end next_day
<p>If weekends fall on different days in your country, update the logic above to skip the days as needed.</p>
TO_CHAR Day Formats
<p>To_char has several <a href="https://livesql.oracle.com/apex/livesql/file/content_GCEY1DN2CN5HZCUQFHVUYQD3G.html">day related format masks</a>, including:</p>
<ul>
<li>DAY &ndash; Returns the name of the day</li>
<li>DY &ndash; Gives the three letter abbreviation of the day name</li>
<li>D &ndash; returns the day of week as a number 1-7</li>
</ul>
<p>You control the case of the output by the case of the input.</p>
select to_char(date&#39;2018-01-01&#39;, &#39;Day&#39;) initcap,
to_char(date&#39;2018-01-01&#39;, &#39;day&#39;) lower,
to_char(date&#39;2018-01-01&#39;, &#39;DAY&#39;) upper
from dual;
INITCAP LOWER UPPER
Monday monday MONDAY
<p>Also note that without the FM prefix, the string is right-padded with spaces to match the length of the longest day.</p>
<p>Obviously the values returned by the name of day formats depend on the language. This is determined by the client&rsquo;s setting for the NLS_DATE_LANGUAGE parameter. So servers in different parts of the world can return different values.</p>
<p>For example:</p>
alter session set nls_date_language = SPANISH;
select to_char(date&#39;2018-01-01&#39;, &#39;Day&#39;) dy
from dual;
DY
Lunes
alter session set nls_date_language = ENGLISH;
select to_char(date&#39;2018-01-01&#39;, &#39;Day&#39;) dy
from dual;
DY
Monday
<p>You may think you can avoid this using to_char(&lt;dt&gt;, &lsquo;d&rsquo;), which returns the day number of the week.</p>
<p>But this varies by NLS_TERRITORY!</p>
<p>For some reason, the US believes Sunday is the first day of the week. Whereas most of the rest of the world and the ISO standard consider Monday the start. So for Monday the output could flip between 1 and 2:</p>
alter session set nls_territory = AMERICA;
select to_char(date&#39;2018-01-01&#39;, &#39;d&#39;) day_number
from dual;
DAY_NUMBER
2
alter session set nls_territory = &quot;UNITED KINGDOM&quot;;
select to_char(date&#39;2018-01-01&#39;, &#39;d&#39;) day_number
from dual;
DAY_NUMBER
1
<p>This could cause you all sorts of headaches as your application conquers the globe.</p>
<p>Luckily you can avoid these problems by fixing the language for the conversion. Do this by passing the desired language in the third parameter of to_char:</p>
alter session set nls_date_language = ENGLISH;
select to_char(date&#39;2018-01-01&#39;, &#39;fmDay&#39;, &#39;NLS_DATE_LANGUAGE = SPANISH&#39;) day_name
from dual;
DAY_NAME
Lunes
<p>You can only do this with the language, not the territory. So you&rsquo;re best off avoiding the &lsquo;D&rsquo; format in your code.</p>
How to Add N Business Days
<p>The logic above works well enough when you want the next day. But sometimes you need to add a variable number of working days to a date, such as five.</p>
<p>In this case you need to think carefully. If you want to add one working day and it&rsquo;s currently Saturday, there are two ways to do the calculation:</p>
<ul>
<li>Add one day to the date. If the result is a non-working day, move to the next date. The resulting date is a Monday.</li>
<li>Move to the next working day. Then add one working day. The resulting date is a Tuesday!</li>
</ul>
<p>As <a href="https://twitter.com/chrisrsaxon/status/953317369434247168">my Twitter poll</a> shows, people are evenly split in which approach they take:</p>
<p><img alt="" src="https://cdn.app.compendium.com/uploads/user/e7c690e8-6ff9-102a-ac6d-e4aebca50425/6a8a9853-c115-4f9b-b14e-7eee344d2c98/Image/f80ec2d970aa95199550addc860e6544/twitter_working_day.png" style="width: 627px; height: 464px;" /></p>
<p>It also brings up the interesting question of what happens when you add zero days to a non-working day.</p>
<p>Mathematically, adding zero is the identity operation. i.e. you get what you started with.</p>
<p>But if you&rsquo;re adding working days, shouldn&rsquo;t the result <em>always</em> be a working day?!</p>
<p>There&rsquo;s no fixed answer to this. For the calculations below I&rsquo;m assuming that:</p>
<ul>
<li>Adding zero returns the current date, whether it&rsquo;s a working day or not</li>
<li>To add N working days, you first move to the next working day. Then add the days.</li>
</ul>
<p>Whatever you do, make sure you confirm these assumptions with your business! If they have a different understanding of this you could end up with lots of off-by-one errors. Which could lead to lots of unhappy people if these calculations relate to payment dates&hellip;</p>
<p>OK, so we&rsquo;ve got that out of the way. How do you to the calculation?</p>
<p>At first glance, if the input date is a weekend you may be tempted to bump it to the next Monday. Then add the days. But this only works if N is less&nbsp;than a working week. As soon as it spans a weekend, it falls apart. You need to bypass an unknown number of non-working days.</p>
<p>The simple method to do this is:</p>
<ul>
<li>Start a loop, beginning with the initial date. If this is a weekend, move it to the next Monday</li>
<li>On each iteration increment the date</li>
<li>If the date is a weekday, also increment a counter</li>
<li>Stop the loop when the counter equals the requested number of days</li>
</ul>
<p>As you&rsquo;re looping until a condition is true, a while loop is a great candidate. So you can build a function like this:</p>
create or replace function add_n_working_days (
start_date date, working_days pls_integer
) return date as
end_date date := start_date;
counter pls_integer := 0;
begin
if working_days = 0 then
&nbsp; end_date := start_date;
&nbsp; elsif to_char(start_date, &#39;fmdy&#39;) in (&#39;sat&#39;, &#39;sun&#39;) then
end_date := next_day(start_date, &#39;monday&#39;);
end if;
while (counter &lt; working_days) loop
end_date := end_date + 1;
if to_char(end_date, &#39;fmdy&#39;) not in (&#39;sat&#39;, &#39;sun&#39;) then
counter := counter + 1;
end if;
end loop;
return end_date;
end add_n_working_days;
/
<p>At this point you may be worried. Loops are slow,&nbsp;right? Where&rsquo;s the <a href="https://explainextended.com/2009/07/12/double-thinking-in-sql/">set-based SQL goodness</a> we&rsquo;re always <a href="http://www.oracle.com/technetwork/testcontent/o27asktom-084983.html">banging on about</a>?!</p>
<p>True, as you increase N the function will get proportionally slower:</p>
<p><img alt="" src="https://cdn.app.compendium.com/uploads/user/e7c690e8-6ff9-102a-ac6d-e4aebca50425/6a8a9853-c115-4f9b-b14e-7eee344d2c98/Image/e7c14eed9103612d537461fa77718f38/add_n_working_days_loop.png" style="width: 728px; height: 412px;" /></p>
<p>But, before you get scared, the scale is in <em>microseconds</em>. So even if you&rsquo;re adding hundreds of days it&rsquo;ll still complete before you blink.</p>
<p>But in some cases every CPU cycle counts, right?</p>
<p>Luckily you can take shortcuts.</p>
<p>Say you&rsquo;re adding a hundred working days. You don&rsquo;t need to loop 100 times plus however many weekends the range spans. Instead you can add the number of weeks you&rsquo;re spanning. <em>Then</em> loop over the last few days to get the result.</p>
<p>So if the timeframe is longer than a working week (five days) you can:</p>
<ul>
<li>Find the number of weeks by taking the floor of the N / 5</li>
<li>Multiply it by seven to give the number of weeks spanned</li>
<li>Move the start date to the next Monday, then add the calculated weeks</li>
<li>Loop from this point for another ( N mod 5 ) days</li>
</ul>
<p>Which in PL/SQL looks like:</p>
create or replace function add_n_working_days_optimized (
start_date date, working_days pls_integer
) return date as
end_date date;
counter pls_integer := 0;
remaining_days pls_integer;
weeks pls_integer;
begin
if working_days = 0 then
end_date := start_date;
&nbsp; elsif to_char(start_date, &#39;fmdy&#39;) in (&#39;sat&#39;, &#39;sun&#39;) then
end_date := next_day(start_date, &#39;monday&#39;);
else
end_date := start_date;
end if;
if working_days &lt;= 5 then
remaining_days := working_days;
else
weeks := floor ( working_days / 5 ) ;
end_date := end_date + ( weeks * 7 );
remaining_days := mod ( working_days, 5 ) ;
end if;
while (counter &lt; remaining_days) loop
end_date := end_date + 1;
if to_char(end_date, &#39;fmdy&#39;) not in (&#39;sat&#39;, &#39;sun&#39;) then
counter := counter + 1;
end if;
end loop;
return end_date;
end add_n_working_days_optimized;
/
<p>So now, worst case, you loop through five days.</p>
<p>This is great. But what if you want to subtract N days, going back in time?</p>
<p>The good news is this all works in reverse.</p>
<p>Subtracting is the same as adding a negative amount. So by adding the weeks multiplied by the sign of N, you decrement the start date. Then loop until the working day counter hits the absolute value of the remaining days.</p>
create or replace function add_n_working_days_optimized (
start_date date, working_days pls_integer
) return date as
end_date date;
counter pls_integer := 0;
remaining_days pls_integer;
weeks pls_integer;
begin
if working_days = 0 then
end_date := start_date;
&nbsp; elsif to_char(start_date, &#39;fmdy&#39;) in (&#39;sat&#39;, &#39;sun&#39;) then
if sign(working_days) = 1 then
end_date := next_day(start_date, &#39;monday&#39;);
else
end_date := next_day(start_date-7, &#39;friday&#39;);
end if;
else
end_date := start_date;
end if;
if abs(working_days) &lt;= 5 then
remaining_days := working_days;
else
weeks := floor ( abs(working_days) / 5 ) * sign(working_days);
end_date := end_date + ( weeks * 7 );
remaining_days := mod ( working_days, 5 );
end if;
while (counter &lt; abs(remaining_days)) loop
end_date := end_date + sign(working_days);
if to_char(end_date, &#39;fmdy&#39;) not in (&#39;sat&#39;, &#39;sun&#39;) then
counter := counter + 1;
end if;
end loop;
return end_date;
end add_n_working_days_optimized;
/
<p>So now we can find the next working day after any date. But what if you want to count the working days in a date range?</p>
How to Find the Number of Working Days Between Two Dates
<p>When figuring out whether you can deliver your code by the ludicrously optimistic perfectly reasonable release date your manager you gave you, it&rsquo;s useful to know how many working days there are between now and then.</p>
<img alt="" src="https://cdn.app.compendium.com/uploads/user/e7c690e8-6ff9-102a-ac6d-e4aebca50425/6a8a9853-c115-4f9b-b14e-7eee344d2c98/Image/c0c2e1d8c7a2d5ffe9e99adbb1c2c6d5/439h_medium.jpg" style="width: 1364px; height: 909px;" />
<p style="text-align: right;">Ryan McGuire / <a href="https://gratisography.com/">Gratisography</a></p>
<p>So how do you do this?</p>
<p>Again, you have to be careful. Do you include the end date or not?</p>
<p>In any case, there&rsquo;s a simple solution available:</p>
<ul>
<li>Start a loop from one up to the number of days between the start and end dates. Minus one if you exclude the end date.</li>
<li>On each iteration, add the current index to the start date</li>
<li>If the result is a weekday, increment a counter of working days</li>
</ul>
<p>Which in code looks like:</p>
create or replace function get_working_days_between (
start_date date, end_date date
) return pls_integer as
counter pls_integer := 0;
date_range pls_integer;
begin
date_range := end_date - start_date;
for dys in 1 .. abs( date_range ) loop
if to_char (
start_date + ( dys * sign ( date_range ) ), &#39;fmdy&#39;
) not in (&#39;sat&#39;, &#39;sun&#39;) then
counter := counter + 1;
end if;
end loop;
return counter;
end get_working_days_between;
/
<p>Hmmm. More loops. It&rsquo;s going to be slow if the dates are far apart, right?</p>
<p>Well, yes, it will take longer.</p>
<p>But there&rsquo;s a bigger problem looming. And at last we&rsquo;re going to see some SQL!</p>
Public Holidays
<p>Dates are funny things. A working day isn&rsquo;t a fixed concept. Different businesses and <a href="https://www.timeanddate.com/holidays/">different countries</a> assign it in different ways.</p>
<p>Many businesses consider public holidays to be non-working days. Which brings a couple of problems.</p>
<p>First up, for the most part they&rsquo;re not fixed days. The date of Easter changes every year.</p>
<p>And even those that fall on a fixed date, such as Christmas day, often have the public holiday assigned to the next weekday when they fall on a weekend.</p>
<p>To handle these cases you can use logic like:</p>
<ul>
<li>Find the next working day</li>
<li>If this is a public holiday, skip to the next workday</li>
</ul>
<p>To do this you&rsquo;ve got to go with the basic loop. The optimized method adding weeks will incorrectly skip holidays. So you need to extend the non-working day test to include these dates, like so:</p>
if to_char(end_date, &#39;fmdy&#39;) not in (&#39;sat&#39;, &#39;sun&#39;) and
end_date not in (
date&#39;2018-01-01&#39;, date&#39;2018-03-30&#39;, date&#39;2018-04-01&#39;,
date&#39;2018-05-07&#39;, date&#39;2018-05-28&#39;, date&#39;2018-08-07&#39;,
date&#39;2018-12-25&#39;, date&#39;2018-12-26&#39;
) then
counter := counter + 1;
end if;
<p>Filling this out for all non-working days is hugely messy. Particularly if you want to include every holiday for the foreseeable future. And even that doesn&rsquo;t remove all code maintenance. If the Queen fancies a day off after working for sixty years, a <a href="http://news.bbc.co.uk/1/hi/8441972.stm">new holiday appears</a>.</p>
<p>And the problem gets worse if you need to handle holidays for different countries.</p>
<p>Luckily, there&rsquo;s a solution:</p>
<p>Make it data-driven.</p>
Why Every Application Needs a Dates Table
<p>Instead of programmatically figuring out if each is a working day or not, store the results in a table!</p>
<p><a href="https://blogs.oracle.com/sql/how-to-create-alter-and-drop-tables-in-sql">Create a table</a> with a row for each day for the foreseeable future:</p>
create table calendar_dates (
&nbsp; calendar_day&nbsp; &nbsp;date not null primary key,
&nbsp; is_working_day varchar2(1) not null
);
<p>Oracle Database doesn&#39;t have a SQL <a href="https://community.oracle.com/ideas/2633">Boolean data type</a>. So is_working_day column stores Y/N, 1/0 or however you want to represent true/false in your database. You can then use it bypass non-working days as needed.</p>
<p>This allows you to give control over what counts as a working day or not to whoever in the business makes these decisions. All you need is a simple admin app where staff can set the non-working days.</p>
<p>Then, with a bit of SQL, your code will adapt as the data changes. Without you needing to write any more code!</p>
<p>But before we get to that, a quick note on dates in Oracle Database.</p>
Day Data Type in Oracle Database
<p>Sadly there&rsquo;s no &ldquo;day with no time&rdquo; data type in Oracle Database. Dates and timestamps include times. Always.</p>
<p>So the convention for calendar days is to store them as dates with a time of midnight. To ensure you can only store midnight, you need a check constraint:</p>
alter table calendar_dates add constraint is_midnight&nbsp;
&nbsp; check ( calendar_day = trunc ( calendar_day ) );
<p>By default, <a href="https://docs.oracle.com/en/database/oracle/oracle-database/18/sqlrf/TRUNC-date.html#GUID-BC82227A-2698-4EC8-8C1A-ABECC64B0E79">trunc() returns a date at midnight</a>. So this check will fail if you try and pass a date at a time other than 00:00:00.</p>
<p>With this in place we can - at last! &ndash; get to some SQL.</p>
Find Next Working Day with SQL
<p><img alt="" src="https://cdn.app.compendium.com/uploads/user/e7c690e8-6ff9-102a-ac6d-e4aebca50425/6a8a9853-c115-4f9b-b14e-7eee344d2c98/Image/da37e68636416459683e7533921ded95/db_disk_sql.png" style="width: 512px; height: 256px; border-style: solid; margin:0px auto;display:block;" /></p>
<p>The first thing you need to do is restrict the data set to working days on or after the start date.</p>
<p>At this point you&rsquo;ve only got working days. So you want the next row in the set, ordered by calendar date. The <a href="https://docs.oracle.com/en/database/oracle/oracle-database/18/sqlrf/LEAD.html">analytic function lead</a> does this for you.</p>
<p>This will find the next day for every row. You only want the first of these, which you can get with min:</p>
with dates as (
select * from calendar_dates
where calendar_day &gt;= to_date(:start_date, &#39;dd/mm/yyyy&#39;)
and is_working_day = &#39;Y&#39;
), plus_n_days as (
select lead(calendar_day)
over (order by calendar_day) dt
from dates
)
select min(dt) from plus_n_days;
<p>At this point it&rsquo;s worth noting: this will process all rows in the table after your initial date. To rely on this method, you&rsquo;ll need at least six months to a year&rsquo;s worth of days after the current date. Possibly more. And the SQL will access them all!</p>
<p>That&rsquo;s a lot of wasted effort.</p>
<p>To limit this you need an upper bound for the end date. Which is the date you&rsquo;re looking for!</p>
<p>So you&rsquo;re in a chicken-and-egg situation.</p>
<p>As a clumsy hack elegant workaround, you can use an upper bound of the start date plus the largest number of consecutive non-working days. Plus one.</p>
<p>In the UK Good Friday and Easter Monday are both public holidays. So you can easily have a run of four non-working days. So the next working day could be as many as five after the supplied date.</p>
<p>To restrict the data and improve performance, only access the working days between the requested date plus five days:</p>
where&nbsp; calendar_day between to_date(:start_date, &#39;dd/mm/yyyy&#39;)
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; and to_date(:start_date, &#39;dd/mm/yyyy&#39;) + 5
<p>Or, if you&rsquo;re risk-averse, six days. Or, even better, make this fudge factor safety margin data driven too!</p>
Find the Nth Working Day with SQL
<p>You can use this as the basis for adding N working days to a date. First, remember that adding one working day may not be the same as getting the next working day. Sticking with &ldquo;move to next working day, then add N&rdquo; approach, change the where clause above to only select working days on or after the supplied date.</p>
<p>Then, to look N rows ahead, use the second parameter of lead! This states how many rows forward to inspect.</p>
<p>Finally, to aid performance, you&rsquo;ve got to consider how to define an upper bound for the end date.</p>
<p>First you need to cover the complete calendar weeks N spans. So convert working weeks to this by dividing N by five working days and multiplying by seven calendar days.</p>
<p>Then you need an extra buffer on top of this to account for public holidays or any shutdowns your business may have. Speak with your users to find what the longest run of non-working days your business expects.</p>
<p>Altogether this gives:</p>
with dates as (
&nbsp; select * from calendar_dates
&nbsp; where&nbsp; calendar_day between to_date(:start_date, &#39;dd-mm-yyyy&#39;)
and to_date(:start_date, &#39;dd-mm-yyyy&#39;) +
&nbsp; ( :working_days / 5 * 7 ) + 10
&nbsp; and&nbsp; &nbsp; is_working_day = &#39;Y&#39;&nbsp;
), plus_n_days as (
&nbsp; select lead(calendar_day, :working_days)&nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;over (order by calendar_day) dt
&nbsp; from&nbsp; &nbsp;dates
)
&nbsp; select min(dt) from plus_n_days;
<p>Of course, using an upper bound like this makes me uneasy.</p>
<p>What if you set it too low, leading to incorrect results?</p>
<p>To avoid this, you can pick large buffer such as 50. But do this and you&rsquo;re losing the performance benefit of culling unnecessary rows.</p>
<p>Luckily there&rsquo;s another way: a cursor for loop</p>
<p>This is like the original looped solution. But now we&rsquo;re data-driven. And it&rsquo;s easy to write too:</p>
&nbsp; for dates in (
&nbsp; &nbsp; select * from calendar_dates
&nbsp; &nbsp; where&nbsp; calendar_day &gt; start_date
&nbsp; &nbsp; and&nbsp; &nbsp; is_working_day = &#39;Y&#39;
&nbsp; &nbsp; order&nbsp; by calendar_day
&nbsp; ) loop
&nbsp; &nbsp;&nbsp;
&nbsp; &nbsp; end_date := dates.calendar_day;
&nbsp; &nbsp; counter := counter + 1;
&nbsp; &nbsp;&nbsp;
&nbsp; &nbsp; exit when counter &gt; working_days;
&nbsp; &nbsp;&nbsp;
&nbsp; end loop;
<p>But loops are slow, right?</p>
<p>True, but there&rsquo;s a great way to speed up SQL loops: bulk collection. Load all the data into an array, then process that.</p>
<p>So how does the performance of these SQL methods compare?</p>
<p>To find out, I loaded calendar_dates with twenty years&#39; worth of rows. Ten in the past, ten in the future. Then tested the various SQL methods for increasing numbers of working days, as in <a href="https://livesql.oracle.com/apex/livesql/file/content_F975V7F8DGK3YKDEWOREJQ31A.html">this LiveSQL script</a>.</p>
<p>Here&#39;s the results for increasing values of N:</p>
<p><img alt="" src="https://cdn.app.compendium.com/uploads/user/e7c690e8-6ff9-102a-ac6d-e4aebca50425/6a8a9853-c115-4f9b-b14e-7eee344d2c98/Image/f1ec3aa187e4e402ede336adc83f85bd/add_working_days_sql_methods.png" style="width: 786px; height: 462px;" /></p>
<p>Clearly the unbounded SQL method is by far the slowest. Even worse it takes more-or-less the same time, however many days you&#39;re adding. Luckily, the optimized version capping the maximum number of days you inspect is the fastest. As long as you can define a safety margin large enough to cover the longest run of consecutive holidays you&#39;re fine.&nbsp;</p>
<p>If you can&#39;t, the bulk approach is best. My tests had a collection limit of 100. If you normally expect add few days, you could squeeze more performance out of this by using a smaller collection limit.&nbsp;</p>
<p>In any case, you need to ensure your dates table has enough days in the future for the working days to add!</p>
<p>To stop you running out of dates, create a job. Set it to run every week or month&nbsp;it and&nbsp;insert the next batch of dates.</p>
How to Subtract N Working Days in SQL
<p>The previous methods work great if you want to add N days. But what if you want to subtract instead?</p>
<p>You&#39;ve got to work backwards through the dates. So you need to flip the operators around. Swap greater than for less than and sort descending.</p>
<p>So the cursor for looping through the data becomes:</p>
select * from calendar_dates
where calendar_day &lt; to_date(:start_date, &#39;dd/mm/yyyy&#39;)
and is_working_day = &#39;Y&#39;
order by calendar_day desc;
<p>And for the set method you need to make more changes. Lead only accepts positive values for rows to jump ahead. Change it to lag to look back and get the max value this returns</p>
<p>Altogether this gives a SQL statement like:</p>
with dates as (
select * from calendar_dates
where calendar_day between to_date(:start_date, &#39;dd-mm-yyyy&#39;) +
&nbsp; ( :working_days / 5 * 7 ) - 10
and to_date(:start_date, &#39;dd-mm-yyyy&#39;)
and is_working_day = &#39;Y&#39;
), minus_n_days as (
select lag(calendar_day, abs(:working_days))
over (order by calendar_day) dt
from dates
)
select max(dt) from minus_n_days
<p>Now change your add N function so it picks the appropriate query based on whether N is positive or negative.</p>
<p>One final note.&nbsp;</p>
<p>If you want these pure SQL methods to include the current date in the calculation (so adding one working day to Saturday returns Monday instead of Tuesday) change the where clause to:</p>
where (
is_working_day = &#39;Y&#39; or
calendar_day = to_date(:start_date, &#39;dd/mm/yyyy&#39;)
)
<p>Phew! We got there at last!</p>
Count Working Days Between Two Dates with SQL
<p>Now we have a dates table, the great news is counting business days is simple. Select the count of working days after the start date, up to the end date. If you&#39;re excluding the end date, make this strictly less than otherwise less than or equal to:</p>
select count(*)
from&nbsp; &nbsp;calendar_dates
where&nbsp; is_working_day = &#39;Y&#39;
and&nbsp; &nbsp; calendar_day &gt; to_date(:start_date, &#39;dd-mm-yyyy&#39;)
and calendar_day &lt;= to_date(:end_date, &#39;dd-mm-yyyy&#39;);
<p>Easy!</p>
<p>Armed with this knowledge, you write your next business day functions.</p>
<p>Then the CEO announces the company is launching operations all around the EU. Now you need to support holidays in many countries. Aaaaargh!</p>
<p><img alt="" src="https://cdn.app.compendium.com/uploads/user/e7c690e8-6ff9-102a-ac6d-e4aebca50425/6a8a9853-c115-4f9b-b14e-7eee344d2c98/Image/509f66a4a75058d2f97d26e7cb5e1ed4/binary_1254502.png" style="width: 1920px; height: 1132px;" /></p>
<p>The best method is to extend the table to handle holidays in different countries. You may be tempted to do this by adding a flag for each&nbsp;country you need to support. For example:</p>
create table calendar_dates (
&nbsp; calendar_day&nbsp; &nbsp; &nbsp; date not null primary key,
&nbsp; is_working_day_gb varchar2(1) not null,
&nbsp; is_working_day_fr varchar2(1) not null,
&nbsp; is_working_day_de varchar2(1) not null,
&nbsp; is_working_day_es varchar2(1) not null,
&nbsp; -- and so on for every other country you support...
&nbsp; constraint is_midnight check ( calendar_day = trunc ( calendar_day ) )
);
<p>This can work if you know you only need to support a small number of countries. But to choose dates for a given country, either you need to use dynamic SQL, like this:</p>
begin
stmt := &#39;select count(*)
from calendar_dates
where is_working_day_&#39; || country_code || q&#39;| = &#39;Y&#39;
and calendar_day between :start_date and :end_date|&#39;;
execute immediate stmt
into working_days
using start_date, end_date ;
return working_days;
end;
<p>Or come up with a big case expression like this:</p>
select count(*)
from calendar_dates
where case
when :country_code = &#39;GB&#39; then
is_working_day_gb
when :country_code = &#39;FR&#39; then
is_working_day_fr
when :country_code = &#39;DE&#39; then
is_working_day_de
when :country_code = &#39;ES&#39; then
is_working_day_es
...
end = &#39;Y&#39;
and calendar_day between :start_date and :end_date
<p>Neither of these is particularly appealing. And if you need to support a new country, you still need to run an alter table to add it. And update any code which maintains this.</p>
<p>A better option is to make the country a column in the working dates table.</p>
<p>For example:</p>
create table country_working_calendar_dates_iot (
&nbsp; country_code&nbsp; &nbsp;varchar2(2) not null,
&nbsp; calendar_day&nbsp; &nbsp;date not null,
&nbsp; is_working_day varchar2(1) not null,
&nbsp; primary key ( country_code, calendar_day )
&nbsp; constraint is_midnight check ( calendar_day = trunc ( calendar_day ) )
) organization index;
<p>If you go down this route you&rsquo;ll always query by country. So to aid performance I recommend making this an <a href="https://mwidlake.wordpress.com/2011/07/18/">index-organized table</a> with country as the leading column of the primary key.</p>
<p>To use this, extend the examples above to accept the country too. And everything is fully data driven!</p>
<p>Expanding into Mongolia?</p>
<p>Your admin team can add the data with that lovely app you built!</p>
<p>So you can focus on delivering more functionality. Or enjoying some work-free weekends ;)</p>
<p>Over to you!</p>
<p>How do you handle next working day calculations in your application? Any other options you prefer to these?</p>
<p>Let us know in the comments!</p>
<p><strong>UPDATE 26 Feb 2019</strong>: Removing duplicate create in calendar_dates creation</p>
SQLFri, 02 Mar 2018 10:07:00 +0000https://blogs.oracle.com/sql/how-to-find-the-next-business-day-and-add-or-subtract-n-working-days-with-sqlChris SaxonHow to Create and Use Indexes in Oracle Databasehttps://blogs.oracle.com/sql/how-to-create-and-use-indexes-in-oracle-database
.boxed { border: 2px solid red ; }
<p>Indexes. They&#39;re one of the most powerful and misunderstood aspects of SQL performance. In this post we&#39;ll look at the <a href="#why">purpose of an index</a>, <a href="#create">how to create</a> and <a href="#choose">choose choose your index type</a>.&nbsp;Then finish with a discussion of <a href="#decide">how to decide what to index</a> and <a href="#use">how to see if it&#39;s useful</a>.</p>
<a name="why"></a>Why Index?
<p>Database tables can get big. Stupendously, colossally big. For example the Large Hadron Collider at CERN generates <a href="http://www.oracle.com/us/corporate/profit/features/092310-cern-176399.html">10 TB of summary data</a> <em>per day</em>.&nbsp;</p>
<img alt="A giant table!" src="https://cdn.app.compendium.com/uploads/user/e7c690e8-6ff9-102a-ac6d-e4aebca50425/6a8a9853-c115-4f9b-b14e-7eee344d2c98/Image/ea19860593d3323ee591fb59de6383ea/chair_1791455_1280.jpg" style="width: 1280px; height: 856px;" />
<em>Image&nbsp;<a href="https://pixabay.com/en/chair-table-park-tree-monza-1791455/">Pixabay</a></em>
<p>Scanning through millions, billions or trillions of rows to return just two or three is a huge waste. Waste and index can help you avoid.</p>
<p>An index stores the values in the indexed column(s). And for each value the locations of the rows that have it. Just like the index at the back of a book. This enables you to hone in on just the data that you&#39;re interested in.</p>
<p>They&#39;re most effective when they enable you to find a &quot;few&quot; rows. So if you have a query that fetches a handful of rows, there&#39;s a good chance you&#39;ll want it to use an index. The first thing you need to do is create it!</p>
<a name="create"></a>How to Create an Index
<p>Creating an index is easy. All you need to do is identify which column(s) you want to index and give it a name!</p>
create index &lt;index_name&gt; on &lt;table_name&gt; ( &lt;column1&gt;, &lt;column2&gt;, &hellip; );
<p>So if you want to index the color column of your toys table and call it toys_color_i, the SQL is:</p>
create index toys_color_i on toys ( color );
<p>Simple, right?</p>
<p>But, as always, there&#39;s more to it than that. You can place many columns in the same index. For example, you could also include the type of the toys in the index like so:</p>
create index toys_color_type_i on toys ( color, type );
<p>This is known as a composite or compound index. Which order you place columns in your index has a big effect on whether the optimizer will use it. We&rsquo;ll discuss the&nbsp;<a href="#decide">ins and outs of this later</a>.</p>
<p>But first, let&rsquo;s delve into the different types of indexes available in Oracle Database.</p>
<a name="choose"></a>How to Choose the Index Type
<p>Oracle Database offers many different types of index to improve your SQL. One of the key decisions you need to make is whether to go with a bitmap or B-tree index.</p>
B-tree vs. Bitmap
<img alt="Tree vs. Map of bits" src="https://cdn.app.compendium.com/uploads/user/e7c690e8-6ff9-102a-ac6d-e4aebca50425/6a8a9853-c115-4f9b-b14e-7eee344d2c98/Image/e82c8a9a739ebb0f6cb8dfb0cad52a67/tree_vs_bitmap.png" style="width: 1280px; height: 640px;" />
<em>Images&nbsp;<a href="https://pixabay.com/en/tree-branches-aesthetic-tribe-log-51358/">Pixabay</a>&nbsp;and <a href="https://pixabay.com/en/matrix-code-data-networking-1735640/">Pixabay</a></em>
<p>By default indexes are B-tree. These are balanced. This means that all the leaf nodes are at the same depth in the tree. So it&#39;s the same amount of work (O(log n)) to access any value.</p>
<img alt="" src="https://cdn.app.compendium.com/uploads/user/e7c690e8-6ff9-102a-ac6d-e4aebca50425/6a8a9853-c115-4f9b-b14e-7eee344d2c98/Image/9d2ad5d12b4fb11a6b552d33fc09865f/btree_hot.png" style="width: 512px; height: 512px;" />
&nbsp;&nbsp;
<p>Each leaf index entry points to exactly one row.</p>
<p>Bitmaps couldn&#39;t be more different. As with B-trees, they store the indexed values. But instead of one row per entry, the database associates each value with a range of rowids. It then has a series of ones and zeros to show whether each row in the range has the value (1) or not (0).</p>
<strong>Value Start Rowid End Rowid Bitmap</strong>
VAL1 AAAA ZZZZZ 001000000...
VAL2 AAAA ZZZZZ 110000000...
VAL3 AAAA ZZZZZ 000111100...
...
<p>Note that the start and end rowid ranges cover all the rows in the table. But large tables may have to split the rows into several ranges. So each indexed value has many entries.</p>
<p>This brings about a key difference between bitmaps and B-trees:</p>
<p>Rows where all the indexed values are null are NOT included in a B-tree. But they are in a bitmap! So the optimizer can use a bitmap to answer queries like:</p>
where indexed_column is null;
<p>But normally it won&#39;t for a B-tree. You can get around this with B-trees by adding a constant to the end of an index. This makes the following composite index:</p>
create index enable_for_is_null_i on tab ( indexed_column, 1 );
<p>Another benefit of bitmaps is it&#39;s easy to compress all those ones and zeros. So a bitmap index is typically smaller than the same B-tree index.</p>
<p>For example, using a table of Olympic medal winners. Creating indexes on edition, sport, medal, event and athlete gives the following sizes (in blocks)</p>
<p><strong>Column</strong></p>
<p><strong>B-tree</strong></p>
<p><strong>Bitmap</strong></p>
<p>Edition</p>
<p>61</p>
<p>6</p>
<p>Sport</p>
<p>83</p>
<p>1</p>
<p>Athlete</p>
<p>117</p>
<p>115</p>
<p>Medal</p>
<p>71</p>
<p>3</p>
<p>Event</p>
<p>111</p>
<p>5</p>
<p>Gender</p>
<p>64</p>
<p>1</p>
<p>As you can see, in most cases the bitmap indexes are substantially smaller. Though this <a href="https://richardfoote.wordpress.com/2014/10/31/index-advanced-compression-vs-bitmap-indexes-candidate/">advantage diminishes</a> as the number of different values in the index increases.</p>
<p>If you&#39;re familiar with Boolean logic truth tables, you may spot another big advantage for bitmaps:</p>
<p>It&#39;s trivial for the optimizer to combine them.</p>
<p>By overlaying the rowid ranges of two indexes, you can find which rows match the where clause in both. Then go to the table for just those rows.&nbsp;</p>
<p>This means indexes which point to a large number of rows can still be useful. Say you want to find all the female gold medal winners in the 2000 Athens Olympics. Your query is:</p>
select * from olym_medals where gender = &#39;Women&#39; and medal = &#39;Gold&#39; and edition = 2000;
<p>Each medal is about one third of the rows. And you&#39;d think there&#39;d be roughly a 50/50 split between men and women. Sadly it&#39;s more like 75/25 men to women. But in any case it&#39;s unlikely an index on just gender or medal will be useful. The table has 26 Olympic games. Which is getting close to &quot;few&quot; rows. But there&#39;s no guarantee the optimizer will choose this index.</p>
<p>But combined these conditions will identify roughly:</p>
(1/2) * (1/3) * (1/26) = 1/156 ~ 0.64% of the rows
<p>That&#39;s definitely getting into the realm of &quot;few&quot; rows. So it&#39;s likely the query will benefit from some sort of index.</p>
<p>If you create single column bitmaps, the database can combine using a BITMAP AND like so:</p>
---------------------------------------------------------------
| Id | Operation | Name |
---------------------------------------------------------------
| 0 | SELECT STATEMENT | |
| 1 | TABLE ACCESS BY INDEX ROWID BATCHED| OLYM_MEDALS |
| 2 | BITMAP CONVERSION TO ROWIDS | |
| 3 | BITMAP AND | |
|* 4 | BITMAP INDEX SINGLE VALUE | OLYM_EDITION_BI |
|* 5 | BITMAP INDEX SINGLE VALUE | OLYM_MEDAL_BI |
|* 6 | BITMAP INDEX SINGLE VALUE | OLYM_GENDER_BI |
---------------------------------------------------------------
<p>This gives a lot of flexibility. You only need to create single column indexes on your table. If you have conditions against several columns the database will stitch them together!</p>
<p>B-trees don&#39;t have this luxury. You can&#39;t just plonk one on top of the other to find what you&#39;re looking for. While Oracle Database can combine B-trees (via a &quot;bitmap conversion from rowids&quot;), this is relatively expensive. In general to get the same performance as the three bitmaps, you need to place all three columns in a single index. This affects how reusable an index is, which we&#39;ll come to later.</p>
<p>At this point it looks like a slam dunk victory for bitmaps over B-trees.</p>
<ul>
<li>Smaller? Check</li>
<li>More flexible? Check</li>
<li>Include null values? Check</li>
</ul>
<p>So you may be wondering:</p>
Why are B-trees the default instead of bitmaps?
<p>Well, bitmap indexes come with a massive drawback:</p>
<p>Killing write concurrency.</p>
<p>They&#39;re one of the few situations in Oracle Database where an insert in one session can <a href="https://asktom.oracle.com/pls/apex/f?p=100:11:::NO:RP:P11_QUESTION_ID:9533571800346380617">block an insert in another</a>. This makes them questionable for most OLTP applications.</p>
<p>Why?</p>
<p>Well, whenever you insert, update or delete table rows, the database has to keep the index in sync. This happens in a B-tree by walking down the tree, changing the leaf entry as needed. You can see how this works with this <a href="https://www.cs.usfca.edu/~galles/visualization/BTree.html">visualization tool</a>.</p>
<p>But bitmap locks the entire start/end rowid range! So say you add a row with the value RED. Any other inserts which try to add another row with the value RED to the same range are blocked until the first one commits!</p>
<p>This is an even bigger problem with updates. An update from a B-tree is really a delete of the old value and insert of the new one. But with a bitmap Oracle Database has to lock the affected rowid ranges for both the old and new values!</p>
<p>As a result, bitmap indexes are best suited for tables that will only have one process at a time writing to them. This is often the case on reporting tables or data warehouses. But not your typical application.</p>
<p>So if more than one session will change the data in a table at the same time, think carefully before using bitmap indexes.</p>
<p>And of course, there&#39;s the cost reason. Bitmap indexes are an Enterprise Edition only feature. Or, if you like your database in the cloud, an <a href="https://cloud.oracle.com/en_US/database/pricing">Enterprise Package DBaaS</a> or better.</p>
<p>So, great as bitmaps may be, for most applications B-trees are the way to go!</p>
<p>If you&#39;d like to run these comparisons yourself, use <a href="https://livesql.oracle.com/apex/livesql/file/content_FB7W07TH4IGZB4YQ79HYDBRVU.html">this LiveSQL script</a>. That covers one of the biggest differences in index types. Here are some other common index types.</p>
Function-based Indexes
<img alt="" src="https://cdn.app.compendium.com/uploads/user/e7c690e8-6ff9-102a-ac6d-e4aebca50425/6a8a9853-c115-4f9b-b14e-7eee344d2c98/Image/377b2b6a4eb51086826ce4fb38f0b0ca/mathematics_757566_1280.jpg" style="width: 1280px; height: 896px;" />
<em>Image&nbsp;<a href="https://pixabay.com/en/mathematics-formula-physics-school-757566/">Pixabay</a></em>
<p>These are simply indexes where one or more of the columns have a function applied to them. The index stores the result of this calculation. For example:</p>
create index date_at_midnight_i on table ( trunc ( datetime ) );
<p>or</p>
create index upper_names_i on table ( upper ( name ) );
<p>You can use functions in bitmap or B-tree indexes.</p>
<p>Bear in mind if you have a function-based index, to use it the function in your where clause must match the definition in the index exactly(*). So if your index is:</p>
create index dates_i on dates ( trunc (datetime) );
<p>Your where clause must be:</p>
where&nbsp; trunc (datetime) = :some_date
<p>This reduces the reusability of your index. So it&#39;s better to avoid function-based indexes if you can. Instead move the calculation off the column to the variable. Examples include:</p>
<p><strong>Simple formulas</strong></p>
<p>Using standard math, rearrange the formula so there are no functions on the column:</p>
<span style="font-family: monospace;">column + 10 = val</span>
becomes
<span style="font-family: monospace;">column = val &ndash; 10</span>
<span style="font-family: monospace;">column * 100 = val</span>
becomes
<span style="font-family: monospace;">column = val / 100</span>
<strong>Findings rows on a given day</strong>
<p>The date data type in Oracle Database always includes the time of day. So to guarantee you have all the rows that fall on a given day, you can normalize the date to midnight. Then compare the result to a date, like so:</p>
trunc( datetime_col ) = :dt
<p>But there is another way to do this. Check that the column is greater than or equal to the variable and strictly less than the variable plus one day:</p>
datetime_col &gt;= :dt and datetime_col &lt; :dt + 1
<p>Changing your SQL like this makes means you can create regular indexes.</p>
<p>Note that the reverse isn&#39;t always true. You can have a normal index on a column. Then apply a function to that column in your where clause. Sticking with the dates example, you can index:</p>
create index date_i on table ( datetime_col );
<p>And a where clause like:</p>
where trunc( datetime_col ) = :dt
<p>And the database can still use it. This is because it can filter the rows in the index. This is often faster than full scanning the table.</p>
<p>*Starting in 11.2.0.2, Oracle Database can use function-based indexes to process queries without the function in the where clause. This happens in a special case where the function preserves the leading part of the indexed values. For example, trunc() or substr().</p>
<p>I still think it&#39;s better to create the regular index. But at least this helps if you must use the function-based index for some reason. For worked examples, head <a href="https://livesql.oracle.com/apex/livesql/file/content_FDV56FM68ZMGHQW5EPEL0NNR6.html">over to LiveSQL</a>.&nbsp;</p>
Unique Indexes
<img alt="" src="https://cdn.app.compendium.com/uploads/user/e7c690e8-6ff9-102a-ac6d-e4aebca50425/6a8a9853-c115-4f9b-b14e-7eee344d2c98/Image/c4f428d7ac30bdc6d837798dbdc496af/censorship_610101_1280.jpg" />
<em>Image&nbsp;<a href="https://pixabay.com/en/censorship-limitations-610101/">Pixabay</a></em>
<p>A unique index is a form of constraint. It asserts that you can only store a given value once in a table. When you create a primary key or unique constraint, Oracle Database will automatically create a unique index for you (assuming there isn&#39;t <a href="https://asktom.oracle.com/pls/asktom/f?p=100:11:2017491281965674::::P11_QUESTION_ID:36858373078604">an index already available</a>). In most cases you&#39;ll add the constraint to the table and let the database build the index for you.</p>
<p>But there is one case where you need to manually create the index: &nbsp;</p>
<p>Function-based unique constraints.</p>
<p>You can&#39;t use functions in unique constraints. For example, you might want to build a &quot;dates&quot; table that stores one row for each calendar day. Unfortunately, Oracle Database doesn&#39;t have a &quot;day&quot; data type. Dates always have a time component. To ensure one row per day, you need lock the time to midnight. You apply the trunc() function to do this. But a constraint won&#39;t work:</p>
alter table dates add constraint date_u unique ( trunc ( calendar_date ) );
<p>So you need to resort to a unique function-based index:</p>
create unique index date_ui on dates ( trunc ( calendar_date ) );
<p>Or you can hide the function in a virtual column. Then index that. For example:</p>
alter table dates add cal_date_no_time as ( trunc(calendar_date) );
alter table dates add constraint date_u unique ( cal_date_no_time );
<p>(HT to Stew Ashton for pointing this out).</p>
<p>Note you can&#39;t create unique bitmap indexes.</p>
Descending Indexes
<p>B-tree indexes are ordered data structures. New entries have to go in the correct location, according to the logical order imposed by the columns. By default these sort from small to large.</p>
<p>But you can change this. By specifying desc after the column, Oracle Database sorts the values from large to small.</p>
<p>Usually this makes little difference. The database can walk up or down the index as needed. So it&#39;s rare you&#39;ll want to use them.</p>
<p>But there is a case where they can help. If your query contains &quot;order by descending&quot;, the descending index can avoid a sort operation. The simplest case is where you&#39;re searching a range of values, sorting these descending and another column.</p>
<p>For example, finding the orders for customer_ids in a given range. Return these ids in reverse order. Then sort by sale date:</p>
select * from orders
where customer_id between :min_cust_id and :max_cust_id
order by customer_id desc, order_datetime;
<p>If you use a regular composite index on ( customer_id, order_datetime ), the plan will look like:</p>
-------------------------------------------------
| Id | Operation | Name |
-------------------------------------------------
| 0 | SELECT STATEMENT | |
| 1 | SORT ORDER BY | |
| 2 | TABLE ACCESS BY INDEX ROWID| ORDERS |
| 3 | INDEX RANGE SCAN | ORDERS_I |
-------------------------------------------------
<p>But create it with ( customer_id desc, order_datetime ) and the sorting step disappears!</p>
------------------------------------------------
| Id | Operation | Name |
------------------------------------------------
| 0 | SELECT STATEMENT | |
| 1 | TABLE ACCESS BY INDEX ROWID| ORDERS |
| 2 | INDEX RANGE SCAN | ORDERS_I |
------------------------------------------------
<p>This could be a big time saver if the sort is &quot;expensive&quot;.</p>
JSON Search Indexes
<img alt="{JSON}" src="https://cdn.app.compendium.com/uploads/user/e7c690e8-6ff9-102a-ac6d-e4aebca50425/6a8a9853-c115-4f9b-b14e-7eee344d2c98/Image/3d7f43ad2e53b5466d2e28cff5a646a2/json.png" style="width: 512px; height: 350px;" />
&nbsp;&nbsp;
<p>Storing data as JSON documents is (sadly) becoming more common. All the attributes are part of the document, instead of a&nbsp;column in their own right. This makes indexing the values tricky. So searching for documents that have specific values can be agonizingly slow. You can overcome this by creating function-based indexes on the attributes you search.</p>
<p>For example, say you store the Olympic data as JSON. You regularly look for which medals an athlete&#39;s won. You can index this like so:</p>
create index olym_athlete_json_i on olym_medals_json ( json_value ( jdoc, &#39;$.athlete&#39; ) );
<p>And your queries with that json_value expression should get a nice boost.</p>
<p>But you may want to do ad-hoc searching of the JSON, looking for any values in the documents. To help with this you can create an index specifically for JSON data from Oracle Database 12.2. It&#39;s easy to do:</p>
create search index olym_medals_json_i on olym_medals_json (jdoc) for json;
<p>The database can then use this for any SQL using the various JSON functions. You can read more about this in the <a href="https://docs.oracle.com/database/122/ADJSN/indexes-for-json-data.htm">JSON Developer&#39;s indexes guide</a>&nbsp;or try these out <a href="https://livesql.oracle.com/apex/livesql/file/content_FBP0D2WB6CADVAYBMTRCJW3P2.html">over on LiveSQL</a>.</p>
<p>If you&#39;re wondering how the JSON search index works, underneath the covers it uses Oracle Text. This brings us nicely to:</p>
Oracle Text Indexes
<p>Maybe you have large bodies of free text in your database. The kind that you want to do fuzzy searches, semantic analysis on and so on. For these you can create a Text Index. These come in three flavors:</p>
<ul>
<li>Context</li>
<li>Category</li>
<li>Rule</li>
</ul>
<p>Using these indexes is a huge topic in itself. So if you want to do this kind of search, I recommend starting with the <a href="http://docs.oracle.com/database/122/CCAPP/toc.htm">Text Developer&#39;s Guide</a>.</p>
Advanced Index Types
<p>There are a couple of other index types available. You&#39;ll probably never need to create these, but I&#39;ve included them just in case ;)</p>
Application Domain Indexes
<p>From time-to-time, you want create a specialized index, customized to your data. Here you specify how the database indexes and stores your data.</p>
<p>If you want to do this, check out the <a href="https://docs.oracle.com/database/122/ADDCI/toc.htm">Data Cartridge Developer&#39;s Guide</a>.&nbsp;</p>
Reverse Key Indexes
<img alt="" src="https://cdn.app.compendium.com/uploads/user/e7c690e8-6ff9-102a-ac6d-e4aebca50425/6a8a9853-c115-4f9b-b14e-7eee344d2c98/Image/b6116c2b996cd4278f25385996b49b34/arrow_145764_1280.png" style="width: 1280px; height: 934px;" />
<em>Image&nbsp;<a href="https://pixabay.com/en/arrow-green-glossy-left-previous-145764/">Pixabay</a>&nbsp;(recolored)</em>
<p>If you have an index on a column that stores ever-increasing values, all the new entries must go on the right hand edge of the index. Sequence based primary keys and insert timestamps are common examples.</p>
<p>If you have a large number of sessions doing this at the same time, this can lead to problems. They all need to access the same index block to add their data. As only one can change it at a time, this can lead to contention. A term called &quot;hot blocks&quot;.</p>
<p>Reverse key indexes avoid this problem by flipping the byte order of the indexed values. So instead of storing 12345, you&#39;re effectively storing 54321. The net effect of this is it spreads new rows throughout the index.</p>
<p>It&#39;s rare you&#39;ll need to do this. And because the entries are not stored in their natural sort order, you can only use these indexes for equality queries. So it&#39;s often better to solve this problem in other ways, such as hash partitioning the index.</p>
<p>OK, so that covers the different types of index available. But we still haven&#39;t got to the bottom of which indexes you should create for your application. It&#39;s time to figure out:</p>
<a name="decide"></a>How to Decide What to Index
<p>First up, it&#39;s important to know there&#39;s no &quot;right&quot; answer here (though there are some wrong ones ;). It&#39;s likely you&#39;ll have many different queries against a table. If you create an index tailored to each one, you&#39;ll end up with a mass of indexes. This is a problem because each extra index adds storage and <a href="http://use-the-index-luke.com/sql/dml/insert">data maintenance overheads</a>.</p>
<p>So for each query you need to make a compromise between making it as fast as possible and overall system impact. Everything is a trade-off!</p>
<p>The most important thing to remember is that for the database to use an index, the column(s) must be in your query! So if a column <em>never</em> appears in your queries, it&#39;s not worth indexing. With a couple of caveats:</p>
<ul>
<li>Unique indexes are really <em>constraints</em>. So you may need these to ensure data quality. Even if you don&#39;t query them columns themselves.</li>
<li>You should <a href="https://richardfoote.wordpress.com/2014/04/02/indexing-foreign-keys-helden/">index any foreign key columns</a> where you delete from the parent table or update its primary key.</li>
</ul>
<p>Beyond this, an index is most effective when its columns appear in your where clause to locate &quot;few&quot; rows in the table. I give a brief overview of what &quot;few&quot; means in this video series:</p>
<p style="text-align: center;"></p>
<p>A key point is it&rsquo;s not so much about how many rows you get, but how many <a href="http://docs.oracle.com/database/122/CNCPT/logical-storage-structures.htm#CNCPT250">database blocks</a> you access. Generally an index is useful when it enables the database to touch fewer blocks than a full table scan reads. A query may return &ldquo;lots&rdquo; of rows, yet access &ldquo;few&rdquo; database blocks.</p>
<p>This typically happens when logically sorting the rows on the query column(s) closely matches the physical order database stores them in. Indexes store the values in this logical order. So you normally want to use indexes where this happens. The <a href="http://www.oracle.com/technetwork/issue-archive/2012/12-sep/o52asktom-1735913.html">clustering factor</a> is a measure of how closely these logical and physical orders match. The lower this value is, the more effective the index is likely to be. And thus the optimizer to use it.</p>
<p>But contrary to popular belief, you don&#39;t have to have the indexed column(s) in your where clause. For example, if you have a query like this:</p>
select indexed_col from table
<p>Oracle Database can full scan the index instead of the table. Which is good because indexes are usually smaller than the table they&rsquo;re on. But remember: nulls are excluded from B-trees. So it can only use these if you have a not null constraint on the column!</p>
<p>If your SQL only ever uses one column of a table in the join and where clauses then you can have single column indexes and be done.</p>
<p>But the real world is more complicated than that. Chances are you have relatively few basic queries like:</p>
select * from tab where col = &#39;value&#39;;
<p>And a whole truckload with joins and filters on many columns, like:</p>
select * from tab1
join tab2
on t1.col = t2.col
join tab3
on t2.col2 = t3.col1
where t1.other_col = 3
and t3.yet_another_col
order by t1.something_else;
<p>As discussed earlier, if you&rsquo;re using bitmaps you can create single column indexes. And leave the optimizer to combine them as needed. But with B-trees this may not happen. And it&rsquo;s more work when it does.</p>
<p>So it&#39;s likely you&rsquo;re going to want some composite indexes. This means you need to think about which order to place columns in the index.</p>
<p>Why?</p>
<p>Because Oracle Database reads an index starting with its leftmost (&ldquo;first&rdquo;) column. Then it goes onto the second, third, etc. So if you have an index on:</p>
create index i on tab ( col1, col2, col3 );
<p>And your where clause is:</p>
where col3 = &#39;value&#39;
<p>To use the index the database either has to wade through all the values in col1 and col2. Or, more likely, read the whole thing to find those matching your conditions.</p>
<p>Bearing this in mind, here&#39;s a few guidelines for composite indexes:</p>
<ul>
<li>Columns with equality conditions (=) should go first in the index</li>
<li>Those with range conditions (&lt;, &gt;=, between, etc.) should go towards the end</li>
<li>Columns only in the select or order by clauses should go last</li>
</ul>
<p>To put this into context, let&#39;s return to the Olympic medals table and investigate some queries.</p>
<div class="boxed"><strong>Important disclaimer</strong>: The findings for column order here are NOT cast-iron rules. The specifics of your data and requirements may lead you to different conclusions and thus indexes. The important part to grasp is the process of analyzing the trade-offs so you can find the &quot;best&quot; indexes for your application. If you&#39;d like to follow along, use <a href="https://livesql.oracle.com/apex/livesql/file/content_FB76JSIMP2TH6OAMSGB28DSW8.html">these LiveSQL scripts</a>.&nbsp;</div>
<p>Say you&#39;ve got three common queries on this data:</p>
select listagg(athlete,&#39;,&#39;) within group (order by athlete)
from olym_medals where medal = &#39;Gold&#39;;
select listagg(athlete,&#39;,&#39;) within group (order by athlete)
from olym_medals where event = &#39;100m&#39;;
select listagg(athlete,&#39;,&#39;) within group (order by athlete)
from olym_medals where medal = &#39;Gold&#39; and event = &#39;100m&#39;;
<p>What do you index?</p>
<p>With just three values for medal and an even spread between them an index on this is unlikely to help. Queries on event return few rows. So this is definitely worth indexing.</p>
<p>But what about the queries against both columns?</p>
<p>Should you index:</p>
( event, medal )
<p>OR</p>
( medal, event )
<p>For the query searching both columns it will make little difference. Both options will only hit the table for the rows matching the where clause.</p>
<p>But the order affects which of the other queries will use it.</p>
<p>Remember Oracle Database reads index columns from left to right. So if you index (medal, event), your query looking for all the 100m medal winners must first read the medal values.</p>
<p>Normally the optimizer won&rsquo;t do this. But in cases where there are few values in the first column, it can bypass it in what&rsquo;s called an <a href="http://docs.oracle.com/database/122/TGSQL/optimizer-access-paths.htm#GUID-B7C62F0F-EB7C-422C-919D-D86456A74A60">index skip scan</a>.</p>
<p>There are only three different medals available. So this is few enough to enable a skip scan. Unfortunately this is more work than when event is first. And, as discussed, SQL looking for the winners of a particular medal are unlikely to use an index anyway. So an index with medal first probably isn&#39;t worth it.</p>
<p>Using ( event, medal ) has the advantage of a more targeted index compared to just ( event ). And it gives a tiny reduction to the work the query on both columns compared to an index on event alone.</p>
<p>So you&#39;ve got a choice. Do you index just event, or event and medal?</p>
<p>The index on both columns reduces the work for queries on event and medal slightly.</p>
<p>But is it worth having two indexes for this saving?</p>
<p>As well as consuming more disk space, adding to DML overheads, etc. there&#39;s another problem with the composite index. Compare the clustering factor for the indexes on event:</p>
<p align="center"><strong>INDEX_NAME</strong></p>
<p align="center"><strong>CLUSTERING_FACTOR</strong></p>
<p>OLYM_EVENT_I</p>
<p>916</p>
<p>OLYM_EVENT_MEDAL_I</p>
<p>2219</p>
<p>Notice that the clustering factor for event_medal is nearly 3 times&nbsp;higher than for medal!</p>
<p>So what&#39;s so bad about that?</p>
<p>The clustering factor is one of the key drivers determining how &quot;attractive&quot; an index is. The higher it is, the less likely the optimizer is to choose it. If you&#39;re unlucky this could be enough to make the optimizer think a full table scan is cheaper...</p>
<p>Sure, if you really need the SQL looking for the gold winners in given events to be ultra speedy, go for the composite index. But in this case it may be better to go one step further. Add athlete to the index too. That way Oracle Database can answer the query by accessing just the index. Avoiding the table access can save you some work. In most other cases I&#39;d stick with the single column event index.</p>
<p>Let&#39;s take another example. Say your dominant query is to find the winners for given events in a particular year:</p>
select listagg(athlete,&#39;,&#39;) within group (order by athlete)
from&nbsp;&nbsp; olym_medals where&nbsp; edition = 2000 and event = &#39;100m&#39;;
<p>In most cases this will return three or six rows. Even with those pesky team events with lots of athletes getting medals you&#39;re looking at around 100 rows tops. So a composite index is a good idea.</p>
<p>But which order to you put the columns in?!</p>
<p>If you have no other queries on these columns, it seems to make no difference.</p>
<p>But there is another dimension to consider:</p>
Index compression
<img alt="Now that's compression!" src="https://cdn.app.compendium.com/uploads/user/e7c690e8-6ff9-102a-ac6d-e4aebca50425/6a8a9853-c115-4f9b-b14e-7eee344d2c98/Image/99a76fe600d6e49e532156bd3fd0a3f1/stocksnap_a75bc84c4b.jpg" style="width: 1122px; height: 747px;" />
<em>Image <a href="https://stocksnap.io/photo/A75BC84C4B">Stocksnap.io</a></em>
<p>You can compress an index. This effectively removes duplicates in its leading column(s). Provided there are &quot;many&quot; rows for each value, this can shrink the size of an index drastically. This saves disk space. And means you have less data to scan when querying. This can reduce the work your SQL does. The uncompressed index sizes (in blocks) are:</p>
<p align="center"><strong>INDEX_NAME</strong></p>
<p align="center"><strong>LEAF_BLOCKS</strong></p>
<p>OLYM_EVENT_YEAR_I</p>
<p>127</p>
<p>OLYM_YEAR_EVENT_I</p>
<p>127</p>
<p>So which column should you put first and compress? Year or event?</p>
<p>There&#39;s many more rows for each year than each event. So this should go first, right?</p>
<p>Let&#39;s see. Rebuilding an index with &ldquo;compress N&rdquo; compresses its first N columns. So to compact the leading column, you run:</p>
alter index olym_event_year_i rebuild compress 1;
alter index olym_year_event_i rebuild compress 1;
<p>Which brings the index sizes down to:</p>
<p align="center"><strong>INDEX_NAME</strong></p>
<p align="center"><strong>LEAF_BLOCKS</strong></p>
<p>OLYM_EVENT_YEAR_I</p>
<p>64</p>
<p>OLYM_YEAR_EVENT_I</p>
<p>111</p>
<p>Whoa!</p>
<p>Placing event first nearly halves the size of the index compared to year. Even though on average there are three times as many rows per year as per event.</p>
<p>So why&#39;s that?</p>
<p>It&#39;s all to do with the length of the values.</p>
<p>Some of the events have lengthy descriptions, such as the weightlifting &quot;75 - 82.5kg, one-two hand 3 e. (light-heavyweight)&quot;. This leads to an average event length of 15 bytes. Nearly four times more than the 4 bytes needed to store the years.</p>
<p>So you need to look at how compressible the values are, not just how many you have.</p>
<p>Of course, in this case you can go further and compress both columns of both indexes with</p>
alter index olym_event_year_i rebuild compress 2;
alter index olym_year_event_i rebuild compress 2;
<p>And they&rsquo;re back down to the same size:</p>
<p align="center"><strong>INDEX_NAME</strong></p>
<p align="center"><strong>LEAF_BLOCKS</strong></p>
<p>OLYM_EVENT_YEAR_I</p>
<p>59</p>
<p>OLYM_YEAR_EVENT_I</p>
<p>59</p>
<p>So it&#39;s a toss-up for which column to place first. Inspect your other SQL to see if this pushes you to place one or the other first.</p>
<p>So far we&#39;ve only talked about equality checks. Often in SQL you&#39;re searching for a range of values in one of the columns. For example, finding all the 100m medal winners after 2000:</p>
select * from olym_medals
where event = &#39;100m&#39; and edition &gt; 2000;
<p>In this case it&#39;s better to make an index on (event, edition) as opposed to the other way around. This is because the database can first identify only the entries for the 100m. Then filter those for Olympic Games this century. As opposed to finding all the post-2000 entries. Then filtering those to find the 100m winners.</p>
<p>Remember: the recommendations for column order here are specific to this data set. What&#39;s more important to understand is the process and the questions you need to ask. Such as:</p>
<ul>
<li>Which column(s) appear in the where clause of the most queries?</li>
<li>Do the conditions on these columns return a small enough set of rows and access few enough blocks for an index to be useful?</li>
<li>Which columns in the index are most compressible?</li>
<li>Which SQL statements do you want to prioritize the performance of?</li>
</ul>
<p>So you&#39;ve gone out and created your indexes. And you want to assess whether they&#39;re helping or not. So that leaves an important question:</p>
<a name="use"></a>How Do I Know Which Indexes a Query Used?
<p>To figure this out, you need to look at the <a href="https://blogs.oracle.com/sql/how-to-create-an-execution-plan">query&#39;s execution plan</a>. This shows you all the steps the database took to process the query. If the optimizer chose an index it will appear as a step in the plan.</p>
<p>For example, this plan shows a query using the index orders_i to access the orders table:</p>
------------------------------------------------
| Id | Operation | Name |
------------------------------------------------
| 0 | SELECT STATEMENT | |
| 1 | TABLE ACCESS BY INDEX ROWID| ORDERS |
| 2 | INDEX RANGE SCAN | ORDERS_I |
------------------------------------------------
<p>At this point you may discover the optimizer didn&#39;t choose your index. You&#39;re certain it&rsquo;s the &quot;right&quot; index and the SQL should use it. So you may be wondering:</p>
How Can I Force a SQL Statement to Use an Index?
<p>I&#39;m sure some of you are now tempted to reach for the <a href="http://docs.oracle.com/database/122/SQLRF/Comments.htm#GUID-EC0D9F8A-20E7-4281-A16A-6B9C993F2930">index hint</a>. This instructs the database to use the index. This can be useful for seeing if it really is more efficient than the plan the optimizer chose instead.</p>
<p>But I strongly discourage you from using these in your production SQL. Although they&#39;re called hints, they&#39;re really directives. The database has to use the hinted index if it&#39;s able to. Your data may change, making the index slower than a full table scan. But the query is still forced to use it. This can create maintenance problems for you in the future.</p>
<p>But what if your index really did enable a faster plan? You want some way to force or encourage the database down the right path, right?</p>
<p>In this case I&#39;d point you towards <a href="http://www.oracle.com/technetwork/database/bi-datawarehousing/twp-sql-plan-mgmt-12c-1963237.pdf">SQL Plan Management</a> instead. Using SQL Baselines ensures queries use the plan you want, with an in built process for moving to better plans if one is available.</p>
Closing Words
<p>A final piece of advice:</p>
<img alt="" src="https://cdn.app.compendium.com/uploads/user/e7c690e8-6ff9-102a-ac6d-e4aebca50425/6a8a9853-c115-4f9b-b14e-7eee344d2c98/Image/49b9f313c40fedb001b044baac4782f8/231h.jpg" style="width: 1440px; height: 960px;" />
<em>Image <a href="http://www.gratisography.com/">Gratisography</a></em>
<p>Keep the number of indexes you create to a minimum.</p>
<p>It&#39;s hard to prove that your application doesn&#39;t need an existing index. The reasons for creating rarely used indexes tend to get forgotten over time. People will come and go on your team. And no one will know why Dave made that five column function-based monstrosity. You don&#39;t know if it&#39;s never used or critical for year-end reporting and nothing else.</p>
<p>So once you create an index in production, dropping it is risky. This can lead to the bizarre case where a table has more indexes than columns!</p>
<p>Advances such as <a href="https://oracle-base.com/articles/11g/invisible-indexes-11gr1">invisible indexes</a> in 11.1 and <a href="https://blog.dbi-services.com/12cr2-new-index-usage-tracking/">better index monitoring</a> in 12.2 help reduce this risk. But it never fully goes away. If torn between creating two &quot;perfect&quot; indexes and one &quot;good enough&quot; index, I&#39;d normally go with the one that&#39;s good enough.</p>
<p>After all, if it turns out you did need a more targeted index I&#39;m sure your customers will be quick to tell you how slow your app is!</p>
<p>But above all, test!</p>
Want to Know More?
<p>Phew! That was a lot. And it only scratches the surface of indexes!</p>
<p>Indexing is a large and important topic. If you want to learn more about these, I recommend reading the following:</p>
<ul>
<li style="margin-left: 36pt;"><a href="https://richardfoote.wordpress.com/">Richard Foote&#39;s</a> blog&nbsp;</li>
<li style="margin-left: 36pt;"><a href="http://use-the-index-luke.com/">Use the Index Luke</a> a vendor agnostic description of indexes from Markus Winand.</li>
<li style="margin-left: 36pt;">To know more about how the cost-based optimizer works, check out <a href="https://jonathanlewis.wordpress.com/">Jonathan Lewis&#39; blog</a> or books on this topic. The <a href="https://blogs.oracle.com/optimizer/">optimizer blog</a> also gives great insight into how various parts of a plan work.</li>
<li style="margin-left: 36pt;">And of course you can always put your questions to us over on <a href="https://asktom.oracle.com/">Ask TOM</a></li>
</ul>
<p>Got burning questions about indexing? Let us know in the comments!</p>
<p><strong>UPDATED </strong>14 Aug 2017: Clarified section about descending indexes refers to a composite index (thanks&nbsp;to Jonathan Lewis) and some minor formatting edits.</p>
<p><strong>UPDATED </strong>16 Aug 2017: Added virtual column note for function-based unique indexes.</p>
<p><strong>UPDATED&nbsp;</strong>28 Sep 2017: Correction: &quot;The B stands for balanced&quot;: the B in B-tree doesn&#39;t stand for anything! But these indexes are balanced.</p>
SQLThu, 10 Aug 2017 15:25:00 +0000https://blogs.oracle.com/sql/how-to-create-and-use-indexes-in-oracle-databaseChris SaxonAnnouncing the 2016 PL/SQL Challenge Championship for Database Designhttps://blogs.oracle.com/sql/announcing-the-2016-plsql-challenge-championship-for-database-design
<p>In 2016, 340 people answered 6,185 quizzes in the Database Design quiz on the PL/SQL Challenge. I&#39;m pleased to see so many people taking an interest in how to build database applications!</p>
<p>And I&#39;m delighted to announce the 50 players below have qualified for the Database Design Annual Championship 2016. They&#39;ve all shown grit and a solid understanding of Oracle Database.</p>
<p>This is currently scheduled to take place on 15 March 2017.</p>
<p>The number in parentheses after their names are the number of championships in which they have already participated.</p>
<p>Congratulations to all listed below on their accomplishment and best of luck in the upcoming competition!</p>
Name
Rank
Qualification
Country
SteliosVlasopoulos (2)
1
Top 50
Belgium
Pavel Zeman (2)
2
Top 50
Czech Republic
mentzel.iudith (2)
3
Top 50
Israel
Maxim Borunov (2)
4
Top 50
Russia
NielsHecker (2)
5
Top 50
Germany
Vyacheslav Stepanov (2)
6
Top 50
No Country Set
Marek Sobierajski (1)
7
Top 50
Poland
Joaquin_Gonzalez (2)
8
Top 50
Spain
Andrey Zaytsev (2)
9
Top 50
Russia
Jo&atilde;o Borges Barreto (2)
10
Top 50
Portugal
siimkask (2)
11
Top 50
Estonia
seanm95 (2)
12
Top 50
United States
krzysioh (2)
13
Top 50
Poland
li_bao (2)
14
Top 50
Russia
Chad Lee (2)
15
Top 50
United States
ivan_blanarik (2)
16
Top 50
Slovakia
Rytis Budreika (2)
17
Top 50
Lithuania
Chase (2)
18
Top 50
Canada
Eric Levin (2)
19
Top 50
United States
Pavel_Noga (2)
20
Top 50
Czech Republic
JustinCave (2)
21
Top 50
United States
Sandra99 (2)
22
Top 50
Italy
yonderboi (1)
23
Top 50
Russia
coba (2)
24
Top 50
Netherlands
Karel_Prech (1)
25
Top 50
No Country Set
Michal P. (2)
26
Top 50
Poland
Kuvardin Evgeniy (2)
27
Top 50
Russia
_tiki_4_ (2)
28
Top 50
Germany
Henry_A (2)
29
Top 50
Czech Republic
JasonC (2)
30
Top 50
United Kingdom
Aleksei Davletiarov (1)
31
Top 50
Russia
PZOL (2)
32
Top 50
Hungary
Mehrab (2)
33
Top 50
United Kingdom
Enrico Rebecchi (1)
34
Top 50
Italy
Alexandre (0)
35
Top 50
Brazil
JeroenR (1)
36
Top 50
Netherlands
mcelaya (0)
37
Top 50
Spain
tonyC (1)
38
Top 50
United Kingdom
NeilC (0)
39
Top 50
United Kingdom
kbentley1 (1)
40
Top 50
United States
syukhno (0)
41
Top 50
Ukraine
umir (1)
42
Top 50
Italy
Rakesh Dadhich (1)
43
Top 50
India
dannyg64 (1)
44
Top 50
United States
whab@tele2.at (1)
45
Top 50
Austria
Sherry (2)
46
Top 50
Czech Republic
NickL (1)
47
Top 50
United Kingdom
Vladimir13 (0)
48
Top 50
Russia
ratte2k4 (0)
49
Top 50
Germany
Sartograph (0)
50
Top 50
Germany
PL/SQLThu, 19 Jan 2017 16:28:00 +0000https://blogs.oracle.com/sql/announcing-the-2016-plsql-challenge-championship-for-database-designChris Saxon12 Things Developers Will Love About Oracle Database 12c Release 2 INFOGRAPHIChttps://blogs.oracle.com/sql/12-things-developers-will-love-about-oracle-database-12c-release-2-infographic
<p>It's here! Oracle Database 12c Release 2 (12.2) is available on <a href="https://cloud.oracle.com/home" target="_blank">Oracle Cloud</a>!</p> <p>Here's an infographic summarizing my 12 favorite features. If you want to know about these in detail, read <a href="https://blogs.oracle.com/sql/entry/12_things_developers_will_love">12 Things Developers Will Love About Oracle Database 12c Release 2</a>.</p> <p><img src="http://cdn.app.compendium.com/uploads/user/e7c690e8-6ff9-102a-ac6d-e4aebca50425/f4a5b21d-66fa-4885-92bf-c4e81c06d916/Image/bd65f709a518379f59180f84934b1a0b/12_2_features_infographic.png" /> </p>Thu, 10 Nov 2016 13:54:08 +0000https://blogs.oracle.com/sql/12-things-developers-will-love-about-oracle-database-12c-release-2-infographicChris Saxon12 Things Developers Will Love About Oracle Database 12c Release 2https://blogs.oracle.com/sql/12-things-developers-will-love-about-oracle-database-12c-release-2
.Code { font-family: monospace; white-space: pre; padding-left: 10px;<span></span> }
<p>It&#39;s here! Oracle Database 12c Release 2 (12.2) is available on <a href="https://cloud.oracle.com/home" target="_blank">Oracle Cloud</a>&nbsp;and on-premises!</p>
<p><img src="http://cdn.app.compendium.com/uploads/user/e7c690e8-6ff9-102a-ac6d-e4aebca50425/f4a5b21d-66fa-4885-92bf-c4e81c06d916/Image/d4ea4dab9eae28e41fe5791c17ffb092/badge_new_12dot2.png" /></p>
<p>With it comes a whole host of new features to help you write better, faster applications.</p>
<p>Here&#39;s my rundown of the top 12 new features to help you when developing against Oracle Database.</p>
<ol>
<li><a href="#json">Easier, Better, Faster, Stronger JSON</a></li>
</ol>
<ul style="margin-left:40px;">
<li><a href="#json-sql">JSON from SQL</a></li>
<li><a href="#json-plsql">JSON in PL/SQL</a></li>
</ul>
<ol start="2">
<li><a href="#long-names">Looooooooooong Names</a></li>
<li><a href="#expressions">Robust Code using Constants for Data Type Lengths</a></li>
<li><a href="#listagg">Listagg Improved On Overflow</a></li>
<li><a href="#real-time-mv">Lightning Fast SQL with Real Time Materialized Views</a></li>
<li><a href="#approx-queries">Fast Estimates with Approximate Query Enhancements</a></li>
<li><a href="#validate-conversion">Verify Data Type Conversions</a></li>
<li><a href="#cast-default">Handle Casting Conversion Errors</a></li>
<li><a href="#partition-online">Single Statement Table Partitioning</a></li>
<li><a href="#auto-list-partition">Automatic List Partitioning</a></li>
<li><a href="#deprecate">Mark Old Code as &quot;Not for Use&quot;</a></li>
<li><a href="#code-coverage">PL/SQL Code Coverage</a></li>
</ol>
<a name="json"></a>Easier, Better, Faster JSON
<p>12.1.0.2 brought JSON support to Oracle Database. This helped you work with JSON documents stored in clobs or varchar2s.</p>
<p>These are fantastic. But storing raw JSON should be the exception, not the norm. Most of the time you should shred your JSON documents into relational tables.</p>
<p>This leaves you with a problem though. Getting the data back out in JSON format!</p>
<p>Trying to write your own JSON generator is hard. So in 12.2 we offer a whole host of options to help you get the job done.</p>
<a name="json-sql"></a>JSON From SQL
<p>12.2 provides four key functions to help you write SQL that returns data in JSON format:</p>
<ul>
<li>JSON_object</li>
<li>JSON_objectagg</li>
<li>JSON_array</li>
<li>JSON_arrayagg</li>
</ul>
<p>You use the JSON_object* functions to create series of key-value pair documents. i.e. the output has curly braces {}. The JSON_array* functions take a list of values and return it as an array i.e. in square brackets [].</p>
<p>For each row in the input, the non-agg versions of these functions output a row. The agg versions combine multiple rows into a single document or array.</p>
<p>OK. So how do these work?</p>
<p>Let&#39;s look at an example.</p>
<p>Say you&#39;re using the classic employees and departments tables. For each department you want a JSON document that contains:</p>
<ul>
<li>The department name</li>
<li>An array of its employees</li>
<li>Each element of this array should be its own document, listing the employee&#39;s name and their job title.</li>
</ul>
<p>For example:</p>
{
&quot;department&quot;: &quot;Accounting&quot;,
&quot;employees&quot;: [
{
&quot;name&quot;: &quot;Shelley,Higgins&quot;,
&quot;job&quot;: &quot;Accounting Manager&quot;
},
{
&quot;name&quot;: &quot;William,Gietz&quot;,
&quot;job&quot;: &quot;Public Accountant&quot;
}
]
}
<p>How do you create this using the new functions?</p>
<p>Let&#39;s work from the inside out:</p>
<ul>
<li>First you need a document for each employee. This has two attributes, name and job. Pass these into a JSON_object call.</li>
<li>Then you need to turn these into an array. So wrap the JSON_object in a JSON_arrayagg. Group by department to split out the employees for each one into a separate array.</li>
<li>Finally you have a document per department. So you need another JSON_object with department and employees attributes. The values for these are the department name and the results of the JSON_arrayagg call in the previous step.</li>
</ul>
<p>Put it all together and you get:</p>
select json_object (
&#39;department&#39; value d.department_name,
&#39;employees&#39; value json_arrayagg (
json_object (
&#39;name&#39; value first_name || &#39;,&#39; || last_name,
&#39;job&#39; value job_title )))
from hr.departments d, hr.employees e, hr.jobs j
where d.department_id = e.department_id
and e.job_id = j.job_id
group by d.department_name;
<p>And voila! You have your JSON!</p>
<p><img align="middle" src="http://cdn.app.compendium.com/uploads/user/e7c690e8-6ff9-102a-ac6d-e4aebca50425/f4a5b21d-66fa-4885-92bf-c4e81c06d916/Image/3d7f43ad2e53b5466d2e28cff5a646a2/json.png" /></p>
<a name="json-plsql"></a>JSON in PL/SQL
<p>So now you have your JSON document. But what if you want to edit it?</p>
<p>Say you want to change the names to uppercase. And add a title element. So the previous document becomes:</p>
{
&quot;department&quot;: &quot;Accounting&quot;,
&quot;employees&quot;: [
{
&quot;name&quot;: &quot;SHELLEY,HIGGINS&quot;,
&quot;job&quot;: &quot;Accounting Manager&quot;,
&quot;title&quot;: &quot;&quot;
},
{
&quot;name&quot;: &quot;WILLIAM,GIETZ&quot;,
&quot;job&quot;: &quot;Public Accountant&quot;,
&quot;title&quot;: &quot;&quot;
}
]
}
<p>If you&#39;re generating the document it&#39;s easiest to add these in the SQL! So this assumes you want to change a JSON document from an external source.</p>
<p>To help with this, there are new PL/SQL objects. These enable you to access, modify and add elements to a JSON document with get/put calls.</p>
<p>The key object types are:</p>
<ul>
<li>json_element_t &ndash; a supertype for documents and arrays</li>
<li>json_object_t &ndash; for working with JSON documents</li>
<li>json_array_t &ndash; for working with JSON arrays</li>
</ul>
<p>The first thing you need to do is create the JSON object. Do this by parsing the document:</p>
doc := json_object_t.parse(&#39;
{
&quot;department&quot;: &quot;Accounting&quot;,
&quot;employees&quot;: [
{
&quot;name&quot;: &quot;Shelley,Higgins&quot;,
&quot;job&quot;: &quot;Accounting Manager&quot;
},
{
&quot;name&quot;: &quot;William,Gietz&quot;,
&quot;job&quot;: &quot;Public Accountant&quot;
}
]
}
&#39;);
<p>You can then access the employees array using get:</p>
emps := treat(doc.get(&#39;employees&#39;) as json_array_t) ;
<p>The treat function casts the element to the appropriate type (JSON_array_t here).</p>
<p>Once you have the array, you can loop through the employees. Put adds a new key if it&#39;s not present. Otherwise it overwrites the existing value.</p>
for i in 0 .. emps.get_size - 1 loop
emp := treat(emps.get(i) as json_object_t);
emp.put(&#39;title&#39;, &#39;&#39;);
emp.put(&#39;name&#39;, upper(emp.get_String(&#39;name&#39;)));
end loop;
<p>The get functions return a reference to the original object. So if you get some JSON and modify it, the original document also changes!</p>
<p>If you don&#39;t want this, clone the element when you get it. For example:</p>
emps := treat(doc.get(&#39;employees&#39;) as json_array_t).clone
<p>So the complete PL/SQL block to transform the JSON is:</p>
declare
doc json_object_t;
emps json_array_t;
emp json_object_t;
begin
doc := json_object_t.parse(&#39;{
&quot;department&quot;: &quot;Accounting&quot;,
&quot;employees&quot;: [
{
&quot;name&quot;: &quot;Shelley,Higgins&quot;,
&quot;job&quot;: &quot;Accounting Manager&quot;
},
{
&quot;name&quot;: &quot;William,Gietz&quot;,
&quot;job&quot;: &quot;Public Accountant&quot;
}
]
}&#39;);
emps := treat(doc.get(&#39;employees&#39;) as json_array_t) ;
for i in 0 .. emps.get_size - 1 loop
emp := treat(emps.get(i) as json_object_t);
emp.put(&#39;title&#39;, &#39;&#39;);
emp.put(&#39;name&#39;, upper(emp.get_String(&#39;name&#39;)));
end loop;
dbms_output.put_line(doc.to_String);
end;
/
{
&quot;department&quot;: &quot;Accounting&quot;,
&quot;employees&quot;: [
{
&quot;name&quot;: &quot;SHELLEY,HIGGINS&quot;,
&quot;job&quot;: &quot;Accounting Manager&quot;,
&quot;title&quot;: &quot;&quot;
},
{
&quot;name&quot;: &quot;WILLIAM,GIETZ&quot;,
&quot;job&quot;: &quot;Public Accountant&quot;,
&quot;title&quot;: &quot;&quot;
}
]
}
<p>Now you can generate JSON from SQL and change it in PL/SQL you have powerful options to work with it.</p>
<p>And there&#39;s a raft of other improvements to JSON functionality in 12.2. Other enhancements include:</p>
<ul>
<li>JSON_exists function</li>
<li>Support for In-Memory, Partitioning and Materialized Views</li>
<li>Search indexes</li>
<li>GeoJSON</li>
<li>JSON Data Guide</li>
</ul>
<p>If you&#39;re desperate to work with JSON, I recommend <a href="https://docs.oracle.com/database/122/ADJSN/changes.htm#ADXDB6162" target="_blank">checking these out</a>.</p>
<a name="long-names"></a>Loooooooooooooooong Names
<p><img src="http://cdn.app.compendium.com/uploads/user/e7c690e8-6ff9-102a-ac6d-e4aebca50425/f4a5b21d-66fa-4885-92bf-c4e81c06d916/Image/dc108f70a4efce82f933b5c506241a0a/there_are_only_two_hard.jpg" /></p>
<p>Oracle Database handles cache invalidation for you. So as a developer you don&#39;t have to worry about this.</p>
<p>But when it comes to naming things, we&#39;ve made it harder than it ought to be.</p>
<p>Why?</p>
<p>Take the following example:</p>
alter table customer_addresses add constraint
customer_addresses_customer_id_fk
foreign key ( customer_id )
references customers ( customer_id );
<p>Looks like a standard foreign key creation, right?</p>
<p>But there&#39;s a problem. Run it and you&#39;ll get:</p>
SQL Error: ORA-00972: identifier is too long
<p>Aaarrghh! The constraint name is just a tiny bit too long :(.</p>
<p>Staying within the 30 byte limit can be tricky. Particularly if you have naming standards you have to follow. As a result, many people have asked for us to allow longer names.</p>
<p>Starting in 12.2 we&#39;ve increased this limit. The maximum is now 128 bytes. So now you can create objects like:</p>
create table with_a_really_really_really_really_really_long_name (
and_lots_and_lots_and_lots_and_lots_and_lots_of int,
really_really_really_really_really_long_columns int
);
<p>Remember: the limit is 128 <em>bytes</em>. Not characters. So if you&rsquo;re using a multi-byte character set, you&rsquo;ll find you can&rsquo;t create:</p>
create table tabl&eacute;&eacute;&eacute;&eacute;&eacute;&eacute;&eacute;&eacute;&eacute;&eacute;&eacute;&eacute;&eacute;&eacute;&eacute;&eacute;&eacute;&eacute;&eacute;&eacute;&eacute;&eacute;&eacute;&eacute;&eacute;&eacute;&eacute;&eacute;&eacute;&eacute;&eacute;&eacute;&eacute;&eacute;&eacute;&eacute;&eacute;&eacute;&eacute;&eacute;&eacute;&eacute;&eacute;&eacute;&eacute;&eacute;&eacute;&eacute;&eacute;&eacute;&eacute;&eacute;&eacute;&eacute;&eacute;&eacute;&eacute;&eacute;&eacute;&eacute;&eacute;&eacute;&eacute; (
is_67_chars_but_130_bytes int
);
<p>This is because &eacute; uses two bytes in character sets such as UTF8. So even though the string above is only 67 characters, it needs 130 bytes!</p>
<p>I know some of you are desperate to startCreatingTablesWithRidiculouslyLongNames. But before you rush out to do so, check your code. If your dev team uses sloppy coding practices, there may be some traps waiting for you...</p>
<a name="expressions"></a>Robust Code using Constants for Data Type Lengths
<img src="http://cdn.app.compendium.com/uploads/user/e7c690e8-6ff9-102a-ac6d-e4aebca50425/f4a5b21d-66fa-4885-92bf-c4e81c06d916/Image/0013c996b95554e4b3eda33d5f556aa6/178h_600px.jpg" />
<em>Ryan McGuire / <a href="http://www.gratisography.com/" target="_blank">Gratisography</a></em>
<p style="margin-top:15px;">Most applications have at least one piece of PL/SQL that selects from the data dictionary. For example:</p>
begin
select table_name
into tab
from user_tables
where ...
<p>Because the maximum length of a table name has been 30 bytes <em>forever</em>, some developers took to declaring the variable like so:</p>
declare
tab varchar2(30);
<p>Because who needs more than 30 characters, right?</p>
<p>But as we just saw, upgrade to 12.2 and the limit is now 128 bytes! So it&#39;s only a matter of time before people create tables with longer names. Eventually this code will fail with:</p>
ORA-06502: PL/SQL: numeric or value error: character string buffer too small
<p>So what to do?</p>
<p>It&#39;d be nice if you could change the maximum length of a varchar2 dynamically. So instead of combing through your PL/SQL, changing varchar2 ( 30 ) -&gt; varchar2 ( 128 ), you could increase the size in a single place.</p>
<p>Fortunately, in 12.2 you can!</p>
<p>The new release enables you to declare a variable length using a constant. So you could create a constants package:</p>
create or replace package constants as
tab_length constant pls_integer := 128;
end constants;
/
<p>Then use this when declaring your variables:</p>
declare
tab varchar2( constants.tab_length );
<p>Now if we ever increase the length of names again, you only need to make one change: the constant&#39;s value!</p>
<p>Note these aren&#39;t fully dynamic. The PL/SQL compiler has to know the value for the variable size at compile time. This means you can&#39;t base it on the results of a query. User-defined functions are also out. So to enable the variable to hold longer strings, you need to increase the value of constants.tab_length and recompile your code.</p>
<p>Now you may be thinking: for something as common as object names, surely Oracle provides something stating their max length?</p>
<p>The good news is: we do :)</p>
<p>In DBMS_standard you&#39;ll find a new constants. This includes ora_max_name_len. As the name suggests, this states the maximum length for object names. So you can change your table name variable declarations to:</p>
declare
tab varchar2( ora_max_name_len );
begin
<p>The best part is you can make your code future proof now! By using conditional compilation you can change your data dictionary based variable declarations to:</p>
declare
$if DBMS_DB_VERSION.VER_LE_12_1 $then
tab varchar2( 30 );
$else
tab varchar2( ora_max_name_len );
$end
<p>Then when you come to upgrade the tab variable will automatically have the larger limit!</p>
<p>You may be thinking that all sounds like a lot of work. And you&#39;re right.</p>
<p>You can also make your variables 12.2 compatible now with type anchoring:</p>
declare
tab user_tables.table_name%type;
<p>Whichever method you use, start preparing your code now. It may be a long time until you upgrade. But the more robust your code is, the easier it&#39;ll be for you to use the new features.</p>
<p>Variable declarations are one of the more obvious problems you&#39;ll meet with longer names. Let&#39;s look at a more subtle issue:</p>
<a name="listagg"></a>Listagg Improved on Overflow
<p>The following query returns a comma-separated list of indexes for each table in your schema:</p>
select table_name,
listagg(index_name, &#39;,&#39;) within group (order by index_name) inds
from user_indexes
group by table_name;
<p>This is all very well and good. But there&#39;s a potential problem with it. <a href="http://docs.oracle.com/database/121/SQLRF/functions101.htm" target="_blank">Listagg</a>() returns a varchar2. This is limited to 4,000 bytes (32,767 if you&rsquo;re using <a href="https://oracle-base.com/articles/12c/extended-data-types-12cR1" target="_blank">extended data types</a>).</p>
<p>So in 12.1 and 11.2, you needed 130 or more indexes on a table before you start running into issues. And if you have that many indexes on one table, you&#39;ve got bigger problems than hitting this limit!</p>
<img src="http://cdn.app.compendium.com/uploads/user/e7c690e8-6ff9-102a-ac6d-e4aebca50425/f4a5b21d-66fa-4885-92bf-c4e81c06d916/Image/c147f3ce0408694f80a87a931bdc8f2a/281h_600px.jpg" />
<em>Ryan McGuire / <a href="http://www.gratisography.com/" target="_blank">Gratisography</a></em>
<p style="margin-top:15px;">But this changes in 12.2. With longer names, you could hit this limit at just over 30 indexes on a table. While still a large number, this is plausible. Particularly in reporting databases and data warehouses. And you can be sure that someone, somewhere will start creating &quot;self-documenting&quot; indexes like:</p>
create index
reducing_the_monthly_invoice_run_
from_four_hours_to_three_minutes_
PROJ12345_make_everything_faster_
csaxon_thanks_everyone_yeah_baby on ...
<p>Create too many of these and your listagg query will throw frustrating ORA-01489 errors.</p>
<p>To <a href="https://blogs.oracle.com/datawarehousing/entry/managing_overflows_in_listagg" target="_blank">get around this is tricky</a>. So in 12.2 we&#39;ve added an overflow clause. To use it, place &quot;on overflow truncate&quot; after the separator:</p>
select table_name,
listagg(index_name, &#39;,&#39;
on overflow truncate
) within group (order by index_name) inds
from user_indexes
group by table_name;
<p>With this in place, instead of an exception your output will now look something like:</p>
...lots_and_lots_and_lots,of_indexes,...(42) <span> </span>
<p>The &quot;&hellip;&quot; at the end indicates that the output is larger than Oracle can return. The number in brackets how many characters Oracle trimmed from the results. So not only can you see there is more data, you get an indication of how much there is.</p>
<p>The full syntax of this is:</p>
listagg (
things, &#39;,&#39;
[ on overflow (truncate|error) ]
[ text ] [ (with|without) count ]
) within group (order by cols)
<p>Now you can explicitly say whether you want error or truncation semantics. There&#39;s a good chance you&#39;ve already written code to handle the ORA-1489 errors. So to keep the behaviour of your code the same, the default remains error.</p>
<p>The text and count clauses control what appears at the end of the string. If you want to replace &quot;...&quot; with &quot;more&quot;, &quot;extra&quot; or a &quot;click for more&quot; hyperlink, just provide your new string!</p>
select table_name,
listagg(index_name, &#39;,&#39; on overflow truncate
&#39;&lt;a href=&quot;http://www.showfulldetails.com&quot;&gt;click here&lt;/a&gt;&#39;
) within group (order by index_name) inds
from user_indexes
group by table_name;
<p>You can also remove the number of trimmed characters by specifying &quot;without count&quot;.</p>
<a name="real-time-mv"></a>Lightning Fast SQL with Real Time Materialized Views
<p>Materialized views (MVs) can give amazing performance boost. Once you create one based on your query, Oracle can get the results direct from the MV instead of executing the statement itself. This can make SQL significantly faster. Especially when the query processes millions of rows but there are only a handful in the output.</p>
<p>There&#39;s just one problem.</p>
<p>The data in the MV has to be fresh. Otherwise Oracle won&#39;t do the rewrite.</p>
<p>You could of course query the MV directly. But the data will still be old.</p>
<p>So you need to keep the materialized view up-to-date. The easiest way is to declare it as &quot;fast refresh on commit&quot;.</p>
<p>But this is easier said than done. Doing this has a couple of issues:</p>
<ul style="margin-top: 0cm;" type="disc">
<li>Only some queries support on commit refreshes</li>
<li>Oracle <a href="http://rwijk.blogspot.co.uk/2010/01/enq-ji-contention.html" target="_blank">serializes MV refreshes</a></li>
</ul>
<p>So if you have complex SQL you may not be able to use query rewrite. And even if you can, on high transaction systems the refresh overhead may cripple your system.</p>
<p>So instead of &quot;fast refresh on commit&quot;, you make the MV &quot;fast refresh on demand&quot;. And create a job to update it. Which runs every second!</p>
<p>But no matter how frequently you run the job, there will always be times when the MV is stale. So query performance could switch between lightning fast and dog slow. A guaranteed way to upset your users!</p>
<p>So how do you overcome this?</p>
<p>With real time materialized views!</p>
<p>These give the best of both worlds. You can refresh your MV on demand. But still have it return up-to-date information.</p>
<p>To do this, create the MV with the clause:</p>
<span>on query computation </span> <span> </span>
<p>For example:</p>
create table t (x not null primary key, y not null) as
select rownum x, mod(rownum, 10) y from dual connect by level &lt;= 1000;
create materialized view log on t with rowid (x, y) including new values;
create materialized view mv
refresh fast on demand
enable on query computation
enable query rewrite
as
select y , count(*) c1
from t
group by y;
<p>With this, you can add more data to your table:</p>
<span>insert into t </span>
insert into t
select 1000+rownum, 1 from dual connect by level &lt;= 100;
commit;
<p>And Oracle can still use the MV to rewrite. <em>Even though the MV is stale</em>!</p>
select /*+ rewrite */y , count(*) from t
group by y;
<p>It does this by:</p>
<ul style="margin-top: 0cm;" type="disc">
<li>Querying the stale MV</li>
<li>Then applying the inserts, updates and deletes in the MV log to it</li>
</ul>
<p>This can lead to some scary looking execution plans!</p>
<p><img src="http://cdn.app.compendium.com/uploads/user/e7c690e8-6ff9-102a-ac6d-e4aebca50425/f4a5b21d-66fa-4885-92bf-c4e81c06d916/Image/bc069e14b610fef5375fb9b8e2af3458/realtime_mv_exec_plan.png" /></p>
<p>The point to remember is Oracle is reading the materialized view log. Then applying the changes to the MV. So the longer you leave it between refreshes, the more data there will be. You&#39;ll need to test to find the sweet spot to balancing the refresh process and applying MV change logs on query rewrite.</p>
<p>You can even get the up-to-date information when you query the MV directly. To do so, add the fresh_mv hint:</p>
select /*+ fresh_mv */* from mv; <span> </span>
<p>The really cool part?</p>
<p>You can convert your existing MVs to real time with the following command:</p>
alter materialized view mv enable on query computation;
<p>This makes MVs much easier to work with, opening up your querying tuning options!</p>
<a name="approx-queries"></a>Approximate Query Enhancements
<p>If you do data analysis, you often need to answer questions such as:</p>
<ul>
<li>How many customers visited our website yesterday?</li>
<li>How many different products did we sell last month?</li>
<li>How many unique SQL statements did the database execute last week?</li>
</ul>
<p>OK, maybe that last one is just me ;)</p>
<p>In any case, often these questions are simply the starting point for further analysis. So you just want a quick estimate.</p>
<p><img src="http://cdn.app.compendium.com/uploads/user/e7c690e8-6ff9-102a-ac6d-e4aebca50425/f4a5b21d-66fa-4885-92bf-c4e81c06d916/Image/fc168bc5547b3066a15fbd6488666d35/an_approximate_answer_to.jpg" /></p>
<p>Answering these questions normally needs a count distinct along the lines of:</p>
select count ( distinct customer_id ) from website_hits;
But these queries can take a long time to run!
<p>Waiting for the answer is frustrating. But it&#39;s worse if you&#39;re getting the figures for someone else. Like your boss. And they need the figures for a meeting. That starts in a minute. And your query takes at least ten minutes.</p>
<p>Your boss can&#39;t wait that long. They need an answer now!</p>
<p>In cases like this you just need an quick estimate. After all, your boss will round your figure to one or two significant digits anyway.</p>
<p>So in 12.1.0.2 we introduced <a href="http://docs.oracle.com/database/121/SQLRF/functions013.htm#SQLRF56900" target="_blank">approx_count_distinct</a>. This returns an estimate of how many different values there are in the target column. This is typically over 99% accurate and could be significantly faster than exact results.</p>
<p>This is cool. But to take advantage of it, you need to change your code! This could be a time consuming task. Particularly because most of the time you&#39;ll want to be able to switch between exact and approximate results. So a simple find+replace is out. Instead you&#39;ll have to pass in a flag to toggle between modes.</p>
<p>If you&#39;re a big user of distinct counts this could be a lot of work...</p>
<p>So in 12.2 we&#39;ve introduced a new parameter, approx_for_count_distinct. Set this to true like so:</p>
alter session set approx_for_count_distinct = true; <span> </span>
<p>and Oracle implicitly converts all count distincts to the approximate version.</p>
<p>While playing with this you may notice a couple of other new parameters:</p>
<ul>
<li>approx_for_aggregation</li>
<li>approx_for_percentile</li>
</ul>
<p>So what are these all about?</p>
<p>Well in 12.2 we&#39;ve created a new function, approx_percentile. This is the approximate version of the percentile_disc and percentile_cont functions. It&#39;s the same concept as approx_count_distinct, just applied to these functions.</p>
<p>The syntax for it is:</p>
approx_percentile (
&lt;expression&gt; [ deterministic ],
[ (&#39;ERROR_RATE&#39; | &#39;CONFIDENCE&#39;) ]
) within group ( order by &lt;expression&gt;)
<p>As you can see, this has a couple of extra clauses over approx_count_distinct.</p>
Deterministic
<p>This defines whether you get the same results each time you run it on the same data set. Non-deterministic is the default. Meaning you could get different answers each time.</p>
<p>Now you may be wondering, &quot;But why would I ever want non-deterministic results?!&quot;. Well, a couple of reasons:</p>
<ul style="margin-top: 0cm;" type="disc">
<li>Non-deterministic results are faster.</li>
<li>You can only get deterministic results on numeric values.</li>
</ul>
<p>So if you want the 10th percentile in a range of dates, you have to go non-deterministic.</p>
<p>But is the time saving for non-deterministic results worth it?</p>
<p>To find out I created a 16 million row table Exadata Express Cloud Service. Then compared the run time of the following exact, deterministic and non-deterministic percentiles:</p>
select percentile_disc(0.1) within group (order by y)
from super_massive;
select approx_percentile(0.1 deterministic) within group (order by y)
from super_massive;
select approx_percentile(0.1) within group (order by y)
from super_massive;
<p>Averaging the time for three runs of each gave the following results:</p>
<img src="https://cdn.app.compendium.com/uploads/user/e7c690e8-6ff9-102a-ac6d-e4aebca50425/6a8a9853-c115-4f9b-b14e-7eee344d2c98/Image/52178df00ae4232c6efb789a609710f6/approx_percentile_run_comparison.png" />
<em>The figures show the average run time in hundredths of a second.</em>
<p style="margin-top:15px;">Non-deterministic results are around 5x faster than deterministic. And nearly 15x faster than exact results!</p>
<p>So if an estimate is all you need, you can save yourself a lot of time using approx_percentile.</p>
<p>If you want to check this for yourself, <a href="" target="_blank">here&#39;s the script</a> I used.</p>
ERROR_RATE and CONFIDENCE
<p>If you&#39;re getting estimated figures, it does beg the question, just how accurate <em>are</em> the results? If it&#39;s 99.9999% that&#39;s almost certainly &quot;good enough&quot;. But what if they&#39;re only 98% accurate? Or 95%? At some point the error is too large for you to rely on the estimate and you&#39;ll want to switch back to exact calculations.</p>
<p>But to do this you need to know what the error is!</p>
<p>To find this, pass ERROR_RATE or CONFIDENCE as the second parameter. Then you&#39;ll get the accuracy figures instead of the function result. Confidence is how certain we are that the answer is correct. Error rate gives the level of inaccuracy.</p>
<p>Perfect for finding out how good the approximation is.</p>
<p>And there&#39;s more.</p>
<p>The stats geeks among you will know that median is a special case of percentile. So there&#39;s also an approx_median function available. This works in the same way as approx_percentile.</p>
<p>So how does these functions relate to the parameter approx_for_percentile?</p>
<p>Well there are two percentile functions in Oracle, percentile_disc and percentile_cont. So you have options to convert either of these or both of them. And whether to do so in a deterministic manner or not. The values this takes are:</p>
<ul>
<li>all deterministic</li>
<li>percentile_disc deterministic</li>
<li>percentile_cont deterministic</li>
<li>all</li>
<li>percentile_disc</li>
<li>percentile_cont</li>
<li>none</li>
</ul>
<a name="validate-conversion"></a>Verify Data Type Conversions
<p>It&rsquo;s one of those all too common problems. <a href="https://asktom.oracle.com/pls/apex/f?p=100:11:0::::P11_QUESTION_ID:9528368000346452927" target="_blank">Validating a date is indeed a date</a>.</p>
<p>A prime cause of this is the terrible practice of storing dates as strings. One of the biggest issues it is enables people to store things that clearly aren&#39;t dates in &quot;date&quot; columns:</p>
create table dodgy_dates (
id int,
is_this_a_date varchar2(20)
);
insert into dodgy_dates
values (1, &#39;abc&#39;);
<p>Along with a whole bunch of values that might be dates. If you supply the correct format mask:</p>
insert into dodgy_dates
values (2, &#39;20150101&#39;);
insert into dodgy_dates
values (3, &#39;01-jan-2016&#39;);
insert into dodgy_dates
values (4, &#39;01/01/2016&#39;);
<p>Returning only the valid dates is tricky. If you try to convert everything using to_date(), you&#39;ll get exceptions:</p>
select t.*
from dodgy_dates t
where to_date(is_this_a_date) &lt; sysdate;
ORA-01858: a non-numeric character was found where a numeric was expected
<p>or maybe:</p>
ORA-01861: literal does not match format string <span> </span>
<p>or</p>
ORA-01843: not a valid month <span> </span>
<p><img src="http://cdn.app.compendium.com/uploads/user/e7c690e8-6ff9-102a-ac6d-e4aebca50425/f4a5b21d-66fa-4885-92bf-c4e81c06d916/Image/0c199d1d1fdd37402230974d60a8249f/boy_666803_1280.jpg" width="600" /></p>
<p>You could get around this by writing your own is_date() function. Or, if you&#39;re really stupid brave, use a regular expression.</p>
<p>Either way, it&#39;s a lot of unnecessary work.</p>
<p>So to make your life easier, we&#39;ve created a new function, validate_conversion. You pass this a value and a data type. Then Oracle will tell you whether it can do the conversion. If it can, it returns one. Otherwise you get zero.</p>
<p>To return the rows in the table that can be real dates, place this in your where clause:</p>
select t.*
from dodgy_dates t
where validate_conversion(is_this_a_date as date) = 1;
ID IS_THIS_A_DATE
---------- --------------------
3 01-jan-2016
There&#39;s no error. But where did rows 2 and 4 go? They&#39;re possible dates too!
<p>Validate_conversion only tests one date format at a time. By default this is your NLS_date_format</p>
<p>Each client can set their own format. So if you rely on this, you may get unexpected results&hellip;</p>
<p>To avoid this, I strongly recommend you pass the format as a parameter. For example:</p>
select t.*
from dodgy_dates t
where validate_conversion(is_this_a_date as date, &#39;yyyymmdd&#39;) = 1;
ID IS_THIS_A_DATE
---------- --------------------
2 20150101
<p>So to return all the possible dates, you&#39;ll need to call this multiple times:</p>
select t.*
from dodgy_dates t
where validate_conversion(is_this_a_date as date, &#39;yyyymmdd&#39;) = 1 or
validate_conversion(is_this_a_date as date, &#39;dd/mm/yyyy&#39;) = 1 or
validate_conversion(is_this_a_date as date, &#39;dd-mon-yyyy&#39;) = 1;
ID IS_THIS_A_DATE
---------- --------------------
2 20150101
3 01-jan-2016
4 01/01/2016
<p>And this isn&#39;t just for dates. You can use validate_conversion with any of the following data types:</p>
<ul style="margin-top: 0cm;">
<li>binary_double</li>
<li>binary_float</li>
<li>date</li>
<li>interval day to second</li>
<li>interval year to month</li>
<li>number</li>
<li>timestamp</li>
<li>timestamp with time zone</li>
</ul>
<p>If you want to convert strings to dates, you&#39;ll need similar logic in the select. This will test the expression against various format masks. If it matches, call to_date with the relevant mask:</p>
case
when validate_conversion(is_this_a_date as date, &#39;yyyymmdd&#39;) = 1
then to_date(is_this_a_date, &#39;yyyymmdd&#39;)
when validate_conversion(is_this_a_date as date, &#39;dd/mm/yyyy&#39;) = 1
then to_date(is_this_a_date, &#39;dd/mm/yyyy&#39;)
when validate_conversion(is_this_a_date as date, &#39;dd-mon-yyyy&#39;) = 1
then to_date(is_this_a_date, &#39;dd-mon-yyyy&#39;)
end
<p>But this is clunky. Fortunately, 12.2 has more functionality to support data type conversions:</p>
<a name="cast-default"></a>Handle Casting Conversion Errors
<p>From time-to-time you&#39;ll want to cast a value to a different data type. This can bring problems if your values are incompatible with desired the type.</p>
<p>You could overcome this with the validate_conversion function we discussed above. But there is another way. Cast now has a &ldquo;default on conversion error&rdquo; clause.</p>
<p>This specifies which value Oracle returns if it can&rsquo;t convert the expression to the type you wanted. For example, say you&#39;re attempting to cast a varchar2 column to a date. But it happens to include the value &quot;not a date&quot;. You&#39;d get a nasty error:</p>
select cast ( &#39;not a date&#39; as date )
from dual;
ORA-01858: a non-numeric character was found where a numeric was expected
<p>With the new clause you can tell Oracle to return a &quot;magic date&quot; instead of throwing an exception. For example:</p>
select cast (
&#39;not a date&#39; as date
default date&#39;0001-01-01&#39; on conversion error
) dt
from dual;
DT
--------------------
01-JAN-0001 00:00:00
<p>You can then add checks to your code for this magic value.</p>
<p>Note that the default value has to match the data type you&#39;re converting to. So if you&#39;re casting to a date, you can&#39;t return a string:</p>
select cast (
&#39;01012010&#39; as date
default &#39;not a date&#39; on conversion error
) dt
from dual;
ORA-01858: a non-numeric character was found where a numeric was expected
<p>And, as with validate_conversion, cast uses your NLS settings for the default format. If you want to override these, pass the format as a parameter:</p>
select cast (
&#39;01012010&#39; as date
default &#39;01010001&#39; on conversion error,
&#39;ddmmyyyy&#39;
) dt
from dual;
DT
--------------------
01-JAN-2010 00:00:00
<p>This is neat. But at first glance it seems, well, limited. After all, how often do you use cast? If you&#39;re like me, the answer is &quot;rarely&quot;.</p>
<p>But there&#39;s more to it than that!</p>
<p>The conversion error clause also applies to other casting functions. Such as:</p>
<ul style="margin-top: 0cm;" type="disc">
<li>to_date()</li>
<li>to_number()</li>
<li>to_yminterval()</li>
<li>etc.</li>
</ul>
<p>Now that&#39;s <em>really</em> useful! These are functions you use all the time.</p>
<p>So you can write data type conversions like this:</p>
select to_date(
&#39;not a date&#39; default &#39;01010001&#39; on conversion error,
&#39;ddmmyyyy&#39;
) dt
from dual;
DT
--------------------
01-JAN-0001 00:00:00
<p>Combine this with validate_conversion makes changing expressions to a new data type <em>much </em>easier.</p>
<a name="partition-online"></a>Single Statement Table Partitioning
<p>It&rsquo;s a question that frequently comes up on Ask Tom:</p>
<p>&ldquo;How do I convert a non-partitioned table to a partitioned one?&rdquo;</p>
<p>Before 12.2 this was a convoluted process. You had to create a partitioned copy of the table and transfer the data over.</p>
<p>You could use DBMS_redefinition to do this online. But it was a headache and easy to get wrong.</p>
<p>In Oracle Database 12c Release 2 it&#39;s easy. All you need is a single alter table command:</p>
create table t ( x int, y int, z int );
alter table t modify partition by range (x) interval (100) (
partition p1 values less than (100)
) online;
<p>And you&rsquo;re done!</p>
<p>&ldquo;But what about all the indexes?&rdquo; I hear you cry. Well, you can convert them too!</p>
<p>Just add an &ldquo;update indexes&rdquo; clause and state whether you want them to be local or global after the conversion:</p>
create index iy on t (y);
create index iz on t (z);
alter table t modify partition by range (x) interval (100) (
partition p1 values less than (100)
) update indexes (
iy local,
iz global
);
<p>If you really want, you can give your global indexes different partitioning scheme.</p>
<p>While you can change from a non-partitioned table to partitioned, you can&rsquo;t go back again. You also can&rsquo;t change the partitioning scheme, e.g. to go from list to range. Try to do so and you&rsquo;ll get:</p>
ORA-14427: table does not support modification to a partitioned state DDL
<p>But if you want to get <em>really</em> fancy, you can go direct from a normal table to one with subpartitions too!</p>
alter table t modify partition by range (x) interval (100)
subpartition by hash (y) subpartitions 4 (
partition p1 values less than (100)
) online;
<p>And there&#39;s even more improvements to partitioning. Such as:</p>
<a name="auto-list-partition"></a>Automatic List Partitioning
<p>List partitions are great when you have a column with a specific set of values you want to carve into separate partitions. Things like states, countries and currencies are all good examples.</p>
<p>Reference data like these change rarely. But they do change. For example, South Sudan came into being in 2011.</p>
<p><img src="http://cdn.app.compendium.com/uploads/user/e7c690e8-6ff9-102a-ac6d-e4aebca50425/f4a5b21d-66fa-4885-92bf-c4e81c06d916/Image/641d8e2e667b1d46daaf2b82c8ba86c3/640px_southsudanstates_svg.png" /></p>
<p>If you list partitioned your data by country you need to keep your partitions up-to-date. Particularly if you let customers provide their own values. Or you could end up with embarrassing errors such as:</p>
SQL Error: ORA-14400: inserted partition key does not map to any partition
<p>Which of course will happen at 2am. A great way to incur the wrath of the on-call DBA!</p>
<p>To avoid this you could create a default partition. Any new values would then go into this. This prevented inserts throwing exceptions. But all new values go into the default partition. Over time this would fill up with all the new values.</p>
<p>So you need a regular maintenance task to split values out as needed.</p>
<p>12.2 resolves this with Automatic List Partitioning. Every time you insert new values, Oracle will create the new partition on the fly!</p>
<p>To use it, simply place the automatic keyword after the partition column:</p>
create table orders (
customer_id integer not null,
order_datetime date not null,
country_iso_code varchar2(2) not null
) partition by list (country_iso_code) automatic (
partition pUS values (&#39;US&#39;),
partition pGB values (&#39;GB&#39;),
partition pDE values (&#39;DE&#39;),
partition pFR values (&#39;FR&#39;),
partition pIT values (&#39;IT&#39;)
);
insert into orders values (1, sysdate, &#39;ZA&#39;);
select partition_name
from user_tab_partitions
where table_name = &#39;ORDERS&#39;;
PARTITION_NAME
--------------
PDE
PFR
PGB
PIT
PUS
SYS_P1386
<p>Note the new partition will have a system-generated name. So you may want to change them to meaningful names. You can do this with:</p>
alter table orders rename partition SYS_P1386 to pZA;
<p>Be aware. The default partition and automatic list partitioning are mutually exclusive options:</p>
create table orders (
customer_id integer not null,
order_datetime date not null,
country_iso_code varchar2(2) not null
) partition by list (country_iso_code) automatic (
partition pUS values (&#39;US&#39;),
partition pGB values (&#39;GB&#39;),
partition pDE values (&#39;DE&#39;),
partition pFR values (&#39;FR&#39;),
partition pIT values (&#39;IT&#39;),
partition pDEF values (default)
);
SQL Error: ORA-14851: DEFAULT [sub]partition cannot be specified for AUTOLIST [sub]partitioned objects.
<p>Which makes sense when you think about it. But if you want to migrate list partitions with a default to automatic, you&#39;ll need to go through a process. First split everything out of the default partition, then drop it:</p>
create table orders (
customer_id integer not null,
order_datetime date not null,
country_iso_code varchar2(2) not null
) partition by list (country_iso_code) (
partition pUS values (&#39;US&#39;),
partition pGB values (&#39;GB&#39;),
partition pDE values (&#39;DE&#39;),
partition pFR values (&#39;FR&#39;),
partition pIT values (&#39;IT&#39;),
partition pDEF values (default)
);
insert into orders values (1, sysdate, &#39;ZA&#39;);
insert into orders values (2, sysdate, &#39;JP&#39;);
alter table orders split partition pDEF into (
partition pZA values (&#39;ZA&#39;),
partition pJP values (&#39;JP&#39;),
partition pDEF
);
alter table orders drop partition pDEF;
alter table orders set partitioning automatic;
<p>Note this does leave a brief time when there&#39;s no default partition. And automatic partitioning isn&#39;t ready. So you may want to take a short outage to do this!</p>
<a name="deprecate"></a>Mark Old Code as &quot;Not For Use&quot;
<p>Times change. New code quickly becomes legacy code. And legacy code is often superseded by better, faster code. So you deprecate the old code.</p>
<p><img src="http://cdn.app.compendium.com/uploads/user/e7c690e8-6ff9-102a-ac6d-e4aebca50425/f4a5b21d-66fa-4885-92bf-c4e81c06d916/Image/e1de5d9917c2adc428216e19f9503c4d/legacy_code.png" /></p>
<p>But this brings a problem:</p>
<p>How do you stop people using the legacy modules?</p>
<p>People tend to stick with what they know. Even after you&#39;ve repeatedly told everyone to move to the new module there&#39;s always (at least) one developer who insists on using the deprecated procedure instead of the newer, shinier option.</p>
<p>And in complex applications it&#39;s tough to keep track of what&#39;s obsolete.</p>
<p>This is tough to solve.</p>
<p>So to help you with the deprecation process, we&#39;ve introduced a new pragma for this.</p>
<p>To use it, place</p>
pragma deprecate ( deprecated_thing, &#39;Message to other developers&#39; );
<p>below the retired section.</p>
<p>Great. But how does it help?</p>
<p>We&#39;ve added a bunch of new PL/SQL warnings: PLW-6019 to PLW-6022. Enable these and Oracle will tell you if you&#39;re using deprecated code:</p>
alter session set plsql_warnings = &#39;enable:(6019,6020,6021,6022)&#39;;
create or replace procedure your_old_code is
pragma deprecate (
your_old_code, &#39;This is deprecated. Use new_code instead!&#39;
);
begin
null;
end your_old_code;
/
show err
Warning(2,3): PLW-06019: entity YOUR_OLD_CODE is deprecated
<p>This is great. But we&#39;ve all been ignoring the &quot;AUTHID DEFINER&quot; warning forever! If code is truly obsolete, it would be good if you could stop people using it all together.</p>
<p>Fortunately you can!</p>
<p>Here&#39;s the great thing about warnings. You can upgrade them to be errors! PLW-6020 is thrown when you write code calling a deprecated item. Set this to error and the offending code won&#39;t compile:</p>
alter session set plsql_warnings = &#39;error:6020&#39;;
create or replace procedure calling_old_code is
begin
your_old_code();
end calling_old_code;
/
sho err
3/3 PLS-06020: reference to a deprecated entity: This is deprecated. Use new_code instead!
<p>Of course, if you turn PLW-6020 into an error system wide, a lot of stuff might break! So you can selectively upgrade it on given objects:</p>
alter procedure calling_old_code compile plsql_warnings = &#39;error:6020&#39; reuse settings;
<p>So now you have the power to force others to stop using pre-historic code.</p>
<a name="code-coverage"></a>PL/SQL Code Coverage
<p>We&#39;ve covered a lot of new functionality. Some if it you&#39;ll use straight away. Other bits you&#39;ll wait a while.</p>
<p>In any case, when you upgrade to 12.2 you&#39;ll want to test all your code to ensure it works as expected. Which brings the question:</p>
<p>&quot;How much of my code did the tests actually run?&quot;</p>
<p>Coverage metrics will help immensely with this.</p>
<p><img src="http://cdn.app.compendium.com/uploads/user/e7c690e8-6ff9-102a-ac6d-e4aebca50425/f4a5b21d-66fa-4885-92bf-c4e81c06d916/Image/5894f8b9b8fc24e414fc4323423383d6/code_coverage.png" /></p>
<p>Simple line level analysis of the tests isn&#39;t good enough. To see why, consider the following code:</p>
<p>We have a basic function that returns its argument and calls dbms_output.</p>
<p>The procedure calls the function twice in a single if statement:</p>
create or replace function f (p int)
return int as
begin
dbms_output.put_line(&#39;Executed: &#39; || p);
return p;
end;
/
create or replace procedure p is
begin
if f(1) = 1 or f(2) = 2 then
dbms_output.put_line(&#39;this&#39;);
else
dbms_output.put_line(&#39;that&#39;);
end if;
end p;
/
<p>Due to short-circuit evaluation, f(2) is never executed! You can see this from the output:</p>
SQL&gt; exec p;
Executed: 1
this
<p>Anything working at the line level will incorrectly report this as fully covered.</p>
<p>To overcome this, you need to details of &quot;basic block&quot; executions.</p>
<p>So what is a &quot;basic block&quot;?</p>
<p>It&#39;s a piece of code that you either runs completely or not at all. Code always belongs to exactly one basic block. For example:</p>
if f(1) = 1 or f(2) = 2 then
dbms_output.put_line(&#39;this&#39;);
else
dbms_output.put_line(&#39;that&#39;);
end if;
<p>has four basic blocks. One for each call to f and two for the calls to dbms_output.put_line.</p>
<p>The new code coverage functionality measures and reports on these basic blocks.</p>
<p>Using it is easy. First you need to create coverage tables to store the metrics:</p>
exec dbms_plsql_code_coverage.create_coverage_tables;
<p>Then call start_coverage before your test and stop_coverage after:</p>
declare
run_id pls_integer;
begin
run_id := dbms_plsql_code_coverage.start_coverage(&#39;TEST&#39;);
p;
dbms_plsql_code_coverage.stop_coverage;
end;
/
<p>You can then get metrics by querying the dbmspcc* tables that hold these details:</p>
select owner, name, type,
round( ( sum(covered)/count(*) * 100), 2) pct_covered
from dbmspcc_runs r
join dbmspcc_units u
on r.run_id = u.run_id
join dbmspcc_blocks b
on r.run_id = b.run_id
and u.object_id = b.object_id
where r.run_comment = &#39;TEST&#39;
group by owner, name, type;
OWNER NAME TYPE PCT_COVERED
----- ----- ---------- -----------
CHRIS P PROCEDURE 50
CHRIS F FUNCTION 100
<p>This is all well and good. But there&#39;s always some code which your tests don&#39;t cover. Maybe it&#39;s deprecated, so you don&#39;t need test it. Or it&#39;s &quot;just-in-case&quot; code to cover theoretically possible but practically impossible cases. Such as the infamous &quot;when others&quot; exception handler.</p>
<p>You want to exclude these sections from your reports. Fortunately you can with the coverage pragma. By marking lines as &quot;NOT_FEASIBLE&quot; you can filter these out of your reports:</p>
create or replace procedure p is
begin
if f(1) = 1 or f(2) = 2 then
dbms_output.put_line(&#39;this&#39;);
else
pragma coverage (&#39;NOT_FEASIBLE&#39;);
dbms_output.put_line(&#39;that&#39;);
end if;
end p;
/
<p>Rerun the tests and you can hide the untestable parts in your report!</p>
select owner, name, type,
round( ( sum(covered)/count(*) * 100), 2) pct_covered
from dbmspcc_runs r
join dbmspcc_units u
on r.run_id = u.run_id
join dbmspcc_blocks b
on r.run_id = b.run_id
and u.object_id = b.object_id
where r.run_comment = &#39;TEST&#39;
and b.not_feasible = 0
group by owner, name, type;
OWNER NAME TYPE PCT_COVERAGE
----- ---- --------- ------------
CHRIS P PROCEDURE 66.67
CHRIS F FUNCTION 100
<p>If you really want, you can exclude whole sections of code by wrapping it in two coverage pragmas. The first starting NOT_FEASIBLE_START, the second NOT_FEASIBLE_END:</p>
begin
pragma coverage (&#39;NOT_FEASIBLE_START&#39;);
a_section();
of_untestable_code();
pragma coverage (&#39;NOT_FEASIBLE_END&#39;);
end;
/
Wrap Up
<p>Phew! We&#39;ve covered a huge amount of functionality here. And this is far from a complete list of cool, new features. So head over to the docs to check out the <a href="http://docs.oracle.com/cloud/latest/dbcs_dbaas/whatsnewindb.htm" target="_blank">full list</a>.</p>
<p>Want to get your hands on Oracle Database 12c Release 2? Head over to the <a href="https://cloud.oracle.com/en_US/database" target="_blank">database options</a> on Oracle Cloud.</p>
<p>There you can get access with Oracle Database Cloud Service or Exadata Cloud Service.</p>
<p>Or you can let us look after your database with the Exadata Express Cloud Service.</p>
<p>Alternatively you can <a href="http://www.oracle.com/technetwork/database/enterprise-edition/downloads/index-092322.html" target="_blank">download it from OTN</a>.</p>
<p>Want a handy quick reference for these features? Then check this <a href="https://blogs.oracle.com/sql/entry/12_things_developers_will_love1" target="_blank">infographic</a>.</p>
Over to You
<p>What do you think? Which of these features are your favorites? How will they make your life easier?</p>
<p>Let us know in the comments!</p>
<p><strong>UPDATED</strong>: 28 July 2017 Fixed labels on count distinct approx comparisons graph</p>
Data WarehousingPartitioningPL/SQLThu, 10 Nov 2016 13:52:00 +0000https://blogs.oracle.com/sql/12-things-developers-will-love-about-oracle-database-12c-release-2Chris SaxonHow to Convert Rows to Columns and Back Again with SQL (Aka PIVOT and UNPIVOT)https://blogs.oracle.com/sql/how-to-convert-rows-to-columns-and-back-again-with-sql-aka-pivot-and-unpivot
<p><img src="http://cdn.app.compendium.com/uploads/user/e7c690e8-6ff9-102a-ac6d-e4aebca50425/f4a5b21d-66fa-4885-92bf-c4e81c06d916/Image/c6db1cf9bf00cda9997aeb3d2d5c24b2/medal_1622549_640.png" width="200" /> <img src="http://cdn.app.compendium.com/uploads/user/e7c690e8-6ff9-102a-ac6d-e4aebca50425/f4a5b21d-66fa-4885-92bf-c4e81c06d916/Image/91b887c915645b838c3fb58c62d51ef6/medal_1622523_640.png" width="200" /> <img src="http://cdn.app.compendium.com/uploads/user/e7c690e8-6ff9-102a-ac6d-e4aebca50425/f4a5b21d-66fa-4885-92bf-c4e81c06d916/Image/8e5c71b9d1742651307f04aec3945c71/medal_1622529_640.png" width="200" /></p>
<p>The Olympics is over for another year. But there&#39;s still plenty of time for SQL-style data wrangling of the results!</p>
<p>To do this, I&#39;ve compiled a <a href="" target="_blank">table of medal winners</a> from Rio for each sport:</p>
OLYMPIC_YEAR SPORT GENDER EVENT MEDAL NOC ATHLETE
2016 Archery M Men&#39;s Individual Gold KOR KU Bonchan
2016 Archery M Men&#39;s Individual Silver FRA VALLADONT Jean-Charles
2016 Archery M Men&#39;s Individual Bronze USA ELLISON Brady
2016 Archery W Women&#39;s Individual Gold KOR CHANG Hyejin
2016 Archery W Women&#39;s Individual Silver GER UNRUH Lisa
2016 Archery W Women&#39;s Individual Bronze KOR KI Bobae
<p>This is great when looking for a specific result. But what everyone <em>really</em> wants to know how their country fared overall. To get this you need to convert the table above to the final medal table:</p>
Country
Gold
Silver
Bronze
United States
46
37
38
Great Britain
27
23
17
China
26
18
26
Russia
19
18
19
Germany
17
10
15
&nbsp;
<p>To do this, you need to count the number of gold, silver and bronze rows for each country. Then create new columns to hold the results.</p>
<p>The question is, how?</p>
<p>This post will teach you. You&#39;ll also learn various row and column transformations with SQL including:</p>
<ul>
<li><a href="#pivot">Converting Rows to Columns</a></li>
<li><a href="#unpivot">Converting Columns to Rows</a></li>
<li><a href="#transpose">Swapping Rows and Columns</a></li>
</ul>
<a name="pivot"></a>
<p>If you want to play along you can access the scripts in <a href="https://livesql.oracle.com/apex/livesql/file/content_DT38BTKONDYSUSJYEUPQ7V1QR.html" target="_blank">LiveSQL</a>. Or you can nab the <a href="#scripts">create table scripts</a> at the bottom of this post.</p>
<p>Ready? Let&#39;s begin!</p>
Convert Rows to Columns (PIVOT)
<p><img src="http://cdn.app.compendium.com/uploads/user/e7c690e8-6ff9-102a-ac6d-e4aebca50425/f4a5b21d-66fa-4885-92bf-c4e81c06d916/Image/1394158a78f62132e4167f8474064b9d/rose_to_column.png" /></p>
<p>Oracle Database 11g introduced the <a href="http://docs.oracle.com/database/121/SQLRF/statements_10002.htm#CHDFAFIE" target="_blank">pivot operator</a>. This makes switching rows to columns easy. To use this you need three things:</p>
<ol>
<li>The column that has the values defining the new columns</li>
<li>What these defining values are</li>
<li>What to show in the new columns</li>
</ol>
<p>The value in the new columns must be an aggregate. For example, count, sum, min, etc. Place a pivot clause containing these items after the table name, like so:</p>
<span>select * from table
</span><span>pivot ( 3 for 1 in (2, 2, 2) );</span>
<p>So to create the final medal table from the raw data, you need to plug in:</p>
<ol>
<li>You want the medals to become columns. So this is medal.</li>
<li>The values defining the columns are Gold, Silver and Bronze</li>
<li>You need how many rows there are for each colour. i.e. a count(*)</li>
</ol>
<p>Stick it all together and you get:</p>
select * from olympic_medal_winners
pivot (
count(*) for medal in ( &#39;Gold&#39; gold, &#39;Silver&#39; silver, &#39;Bronze&#39; bronze )
)
order by noc
fetch first 6 rows only;
OLYMPIC_YEAR SPORT GENDER EVENT NOC ATHLETE GOLD SILVER BRONZE
2016 Athletics M Men&#39;s 1500m ALG MAKHLOUFI Taoufik 0 1 0
2016 Athletics M Men&#39;s 800m ALG MAKHLOUFI Taoufik 0 1 0
2016 Hockey M Men ARG Argentina 1 0 0
2016 Judo W Women -48 kg ARG PARETO Paula 1 0 0
2016 Sailing X Nacra 17 Mixed ARG Carranza Saroli 1 0 0
2016 Sailing X Nacra 17 Mixed ARG Lange 1 0 0
<p>Hmmm, that&#39;s not right! You wanted the total medals for each country. This is giving the results per athlete!</p>
<p>This is because Oracle adds an <a href="http://docs.oracle.com/database/121/SQLRF/statements_10002.htm#SQLRF55304" target="_blank">implicit group by</a> for all the columns not in the pivot clause. To avoid this, use an inline view that selects just the columns you want in the results:</p>
select * from (
select noc, medal from olympic_medal_winners
)
pivot (
count(*) for medal in ( &#39;Gold&#39; gold, &#39;Silver&#39; silver, &#39;Bronze&#39; bronze )
)
order by 2 desc, 3 desc, 4 desc
fetch first 5 rows only;
NOC GOLD SILVER BRONZE
USA 47 40 40
CHN 31 19 29
GBR 30 26 19
RUS 21 19 19
GER 20 11 17
<p>And voila!</p>
<p>This is looking promising. But it&#39;s still not right. China didn&#39;t finish second. Team GB did! And all countries have too many medals.</p>
<p>So what&#39;s going on?</p>
<p>Doubles events.</p>
<p>In these cases multiple people win a medal. And each person has their own entry in the winners table. So for sports like badminton, tennis and rowing there are multiple rows for some events. But the medal table is counted per event, not per person!</p>
<p>To overcome this, instead of counting all rows you need to get the number of different events. Do this by changing the count(*) to a count (distinct). The distinct contains the columns defining a particular event. In this case that&#39;s sport, gender and event.</p>
<p>Include these in inline view and update the pivot clause, giving:</p>
select * from (
select noc, medal, sport, event, gender
from olympic_medal_winners
)
pivot (
count(distinct sport ||&#39;#&#39;|| event ||&#39;#&#39;||gender )
for medal in ( &#39;Gold&#39; gold, &#39;Silver&#39; silver, &#39;Bronze&#39; bronze )
)
order by 2 desc, 3 desc, 4 desc
fetch first 5 rows only;
NOC GOLD SILVER BRONZE
USA 46 37 38
GBR 27 23 17
CHN 26 18 26
RUS 19 18 19
GER 17 10 15
<p>That&#39;s better!</p>
Creating Multiple Columns
<p><img src="http://cdn.app.compendium.com/uploads/user/e7c690e8-6ff9-102a-ac6d-e4aebca50425/f4a5b21d-66fa-4885-92bf-c4e81c06d916/Image/709fb26ed5e4a57b04fa9a1bbbc154f0/citadel_hill_2972_640.jpg" /></p>
<p>Now you&#39;ve got your medal table. But if you&#39;re a true data geek like me you&#39;ll want to include other things in the total. For example:</p>
<ul>
<li>How many different sports did each country win their medals in?</li>
<li>What were the names of the athletes winning each medal?</li>
<li>etc.</li>
</ul>
<p>Including these in the pivot is easy. All you need to do is add more aggregate functions to your SQL!</p>
<p>For the count of sports per medal, this is count(distinct sport). For the athlete names you can use listagg().</p>
select * from (
select noc, medal, sport, event, gender, athlete
from olympic_medal_winners
)
pivot (
count( distinct sport ||&#39;#&#39;|| event ||&#39;#&#39;|| gender ) medals,
count( distinct sport ) sports,
listagg( athlete, &#39;,&#39;) within group (order by athlete) athletes
for medal in ( &#39;Gold&#39; gold )
)
where gold_medals &gt; 1
order by gold_medals, gold_sports, noc
fetch first 5 rows only;
NOC GOLD_MEDALS GOLD_SPORTS GOLD_ATHLETES
RSA 2 1 SEMENYA Caster,VAN NIEKERK Wayde
THA 2 1 SRISURAT Sukanya,TANASAN Sopita
BEL 2 2 THIAM Nafissatou,VAN AVERMAET Greg
DEN 2 2 BLUME Pernille,Denmark
GEO 2 2 KHINCHEGASHVILI Vladimer,TALAKHADZE Lasha
<p>Note that for the athlete&#39;s name, you get an entry per medal won. So people like Michael Phelps and Jason Kenny who won multiple medals will appear several times!</p>
<p>To overcome this you can slap a regular expression around the gold_athletes column. For example:</p>
regexp_replace(gold_athletes, &#39;([^,]+)(,\1)+&#39;, &#39;\1&#39;)
<p>This has to go in the select clause. Not the pivot clause!</p>
<p>When pivoting multiple functions, be aware that Oracle prefixes each generated column with the alias you provide in the &quot;in&quot; clause. Without the alias Oracle uses the values from the source column. So with many aggregates the new columns will all have the same name. This leads to nasty</p>
<span>ORA-00918: column ambiguously defined</span>
<p>errors. So make sure you alias the functions!</p>
Filtering Pivoted Rows
<p>The SQL above only shows countries with more than one gold medal. Note the position of the where clause. This is after the pivot. This rule applies even if you&#39;re filtering on columns in the original table. For example, if you want to find gold medals won by countries starting with D, stick the where at the end:</p>
select * from (
select noc, medal, sport, event, gender, athlete
from olympic_medal_winners
)
pivot (
count( distinct sport ||&#39;#&#39;|| event ||&#39;#&#39;|| gender ) medals,
count( distinct sport ) sports,
listagg( athlete, &#39;,&#39;) within group (order by athlete) athletes
for medal in ( &#39;Gold&#39; gold )
)
where noc like &#39;D%&#39;
order by gold_medals;
NOC GOLD_MEDALS GOLD_SPORTS GOLD_ATHLETES
DOM 0 0
DEN 2 2 BLUME Pernille,Denmark
Enter the Matrix
<img src="http://cdn.app.compendium.com/uploads/user/e7c690e8-6ff9-102a-ac6d-e4aebca50425/f4a5b21d-66fa-4885-92bf-c4e81c06d916/Image/32602a1e786beafc6dba969fb1d1cbe6/matrix_356024_320.jpg" />
Maybe you don&#39;t want to sum, count or otherwise aggregate anything. Perhaps you want a matrix, showing which sports each country won a medal in. Sports across the top, countries down the side. And an X in each cell where that country was a medalist in that sport:
NOC ARC ATH HOC JUD SAI WRE
ALG X
ARG X X X
ARM X
AUS X X X
AUT X
AZE X X
BAH X
How do you do this? Return to the three point list at the start:
<ol>
<li>You want the sports as columns. So this is sport</li>
<li>A list of all the sports you want to display</li>
<li>An X?</li>
</ol>
<p>&nbsp;</p>
<p>Steps 1 and 2 are easy. But how do you get the X? Remember you have to use an aggregate. So the solution is simply:</p>
min(&#39;X&#39;)
<p>Plug those values in and you get:</p>
select * from (
select noc, sport
from olympic_medal_winners
)
pivot (min(&#39;X&#39;) for sport in (
&#39;Archery&#39; as arc, &#39;Athletics&#39; as ath, &#39;Hockey&#39; as hoc,
&#39;Judo&#39; as jud, &#39;Sailing&#39; as sai, &#39;Wrestling&#39; as wre
)
)
order by noc
fetch first 7 rows only;
NOC ARC ATH HOC JUD SAI WRE
ALG X
ARG X X X
ARM X
AUS X X X
AUT X
AZE X X
BAH X
<p>Note you don&#39;t have to list out all the sports. Just those you want in your matrix. Rows from other sports are excluded from the final table. This rule applies to all forms of pivot. Only rows which match values in the in list appear in the results.</p>
Common problems
<p>So far so good. But there are a few issues you may hit when pivoting. The first is that this feature is only available in Oracle Database 11g and above. So if you&#39;re on earlier versions you need a different approach.</p>
Manual Pivoting
<p>If pivot isn&#39;t available in your database you&#39;ll have to do it the old-school way: manually. For each column you want to create in the final results you need to:</p>
<ol>
<li>Check whether the current value of the pivoting column equals the value you want</li>
<li>If it does, return the value you want to aggregate. Otherwise null</li>
<li>Apply you aggregation to the result of this</li>
</ol>
<p>This means you&#39;ll have several columns in the form:</p>
aggregate_function ( case when pivoting_column = &#39;VALUE&#39; then pivoted_value end )
<p>For example, the old fashioned way to define columns for the medals table is:</p>
count ( case when medal = &#39;Gold&#39; then 1 end ) gold_medals,
count ( case when medal = &#39;Silver&#39; then 1 end ) silver_medals,
count ( case when medal = &#39;Bronze&#39; then 1 end ) bronze_medals
<p>To split the values out, just add a group by for the columns you want to count each value by. The final SQL statement then looks like:</p>
select noc,
count ( case when medal = &#39;Gold&#39; then 1 end ) gold_medals,
count ( case when medal = &#39;Silver&#39; then 1 end ) silver_medals,
count ( case when medal = &#39;Bronze&#39; then 1 end ) bronze_medals
from olympic_medal_winners
group by noc
order by 2 desc, 3 desc, 4 desc
fetch first 5 rows only;
NOC GOLD_MEDALS SILVER_MEDALS BRONZE_MEDALS
USA 47 40 40
CHN 31 19 29
GBR 30 26 19
RUS 21 19 19
GER 20 11 17
<p>Convoluted, isn&#39;t it? And to fix the double counting athletes problem you need an even more obscure:</p>
count ( distinct case when medal = &#39;Gold&#39; then sport ||&#39;#&#39;|| event ||&#39;#&#39;|| gender end )
<p>I&#39;m glad the pivot function exists now!</p>
Dynamic Column Lists
<p>Another hurdle that often comes up is changing the column list dynamically. The Olympic committee has a habit of tweaking which sports are contested each time. So the matrix of sports to country winners may have different columns each time.</p>
<p>For the Olympics this happens at most every four years. So changing your SQL is low effort.</p>
<p>But this could become a pain if the values often change. Fortunately there are ways around this.</p>
Dynamic SQL
<p>If you know the columns will change regularly, it may be best to go with dynamic SQL. With this you generate the pivot values on the fly. All you need is a SQL query providing these in a comma-separated list.</p>
<p><a href="http://docs.oracle.com/database/121/SQLRF/functions101.htm#SQLRF30030" target="_blank">Listagg</a> is perfect here. For example, to generate a list of all the contested sports, use:</p>
select listagg(&#39;&#39;&#39;&#39; || sport || &#39;&#39;&#39; as &#39; || sport, &#39;,&#39;) within group (order by sport)
from (select distinct sport from olympic_medal_winners);
<p>You can then pop this into a dynamic SQL statement, such as:</p>
declare
sql_stmt clob;
pivot_clause clob;
begin
select listagg(&#39;&#39;&#39;&#39; || sport || &#39;&#39;&#39; as &quot;&#39; || sport || &#39;&quot;&#39;, &#39;,&#39;) within group (order by sport)
into pivot_clause
from (select distinct sport from olympic_medal_winners);
sql_stmt := &#39;select * from (select noc, sport from olympic_medal_winners)
pivot (count(*) for sport in (&#39; || pivot_clause || &#39;))&#39;;
execute immediate sql_stmt;
end;
/
<p>Like pivot, listagg is only available in 11g and up. So if you&#39;re stuck in the dark ages on an earlier version, check out <a href="https://asktom.oracle.com/pls/asktom/f?p=100:11:0::::p11_question_id:2196162600402" target="_blank">Tom Kyte&#39;s stragg function</a>.</p>
<p>This works. But it can be fiddly to write. Especially if you have a complex pivot. And the number of columns can change on each execution. Which makes fetching the results tricky.</p>
<p>On top of all this you&#39;ve got an extra query to find the columns. So this could slow you down.</p>
<p>Fortunately there&#39;s a method that&#39;s far easier to write, fetch the results from and is a single query!</p>
XML PIVOTing
<p>To use this, just specify XML after the pivot keyword. Then place a subquery in the values clause. And you&#39;re done!</p>
<p>This gives you the results as an XML document. Each column is in an item tag:</p>
select * from (
select noc, sport
from olympic_medal_winners
)
pivot xml (count(*) medal_winners for sport in (
select sport
from olympic_medal_winners
where sport like &#39;A%&#39;)
)
where rownum = 1;
NOC SPORT_XML
ALG &lt;?xml version=&quot;1.0&quot; encoding=&quot;UTF-8&quot;?&gt;
&lt;PivotSet&gt;
&lt;item&gt;
&lt;column name=&quot;SPORT&quot;&gt;Archery&lt;/column&gt;
&lt;column name=&quot;MEDAL_WINNERS&quot;&gt;1&lt;/column&gt;
&lt;/item&gt;
&lt;item&gt;
&lt;column name=&quot;SPORT&quot;&gt;Artistic Gymnastics&lt;/column&gt;
&lt;column name=&quot;MEDAL_WINNERS&quot;&gt;1&lt;/column&gt;
&lt;/item&gt;
&lt;item&gt;
&lt;column name=&quot;SPORT&quot;&gt;Athletics&lt;/column&gt;
&lt;column name=&quot;MEDAL_WINNERS&quot;&gt;2&lt;/column&gt;
&lt;/item&gt;
&lt;/PivotSet&gt;
<img src="http://cdn.app.compendium.com/uploads/user/e7c690e8-6ff9-102a-ac6d-e4aebca50425/f4a5b21d-66fa-4885-92bf-c4e81c06d916/Image/6a0eba556c351299fb8054fbd3de952b/arrow_37061_640.png" width="275" />
<p>At this point the Olympic geeks among you may be shouting &quot;But Algeria didn&#39;t win any medals in archery or gymnastics in 2016! Why is this giving them one in each?&quot;</p>
<p>Count(*) returns how many rows there are. There&#39;s one row per country. So this will give every country at least one medal in each sport!</p>
<p>To avoid this you need to count a specific column. Crucially, this needs to be empty if a country has no medals in a given event. This is because count(column) only returns the number of non-null rows. So if a country has no medals, in a sport it will appear as zero. For example:</p>
select * from (
select noc, sport, athlete
from olympic_medal_winners
)
pivot xml (count(athlete) medal_winners for sport in (
select sport
from olympic_medal_winners
where sport like &#39;A%&#39;)
)
where rownum = 1;
NOC SPORT_XML
ALG &lt;?xml version=&quot;1.0&quot; encoding=&quot;UTF-8&quot;?&gt;
&lt;PivotSet&gt;
&lt;item&gt;
&lt;column name=&quot;SPORT&quot;&gt;Archery&lt;/column&gt;
&lt;column name=&quot;MEDAL_WINNERS&quot;&gt;0&lt;/column&gt;
&lt;/item&gt;
&lt;item&gt;
&lt;column name=&quot;SPORT&quot;&gt;Artistic Gymnastics&lt;/column&gt;
&lt;column name=&quot;MEDAL_WINNERS&quot;&gt;0&lt;/column&gt;
&lt;/item&gt;
&lt;item&gt;
&lt;column name=&quot;SPORT&quot;&gt;Athletics&lt;/column&gt;
&lt;column name=&quot;MEDAL_WINNERS&quot;&gt;2&lt;/column&gt;
&lt;/item&gt;
&lt;/PivotSet&gt;
<p>If you want the XML to include all the sports you can use the any keyword instead:</p>
pivot xml (count(*) medal_winners for sport in ( any )
<p>The great part about this the results will automatically update if the query output changes. So when Armchair Sitting becomes an Olympic sport the XML will include this as an element :)</p>
<p>The terrible part is you get the output in XML! So you&#39;re going to have to parse this to extract the values...</p>
Custom Types
<p>But there is another way! Anton Scheffer put together a <a href="https://technology.amis.nl/2006/05/24/dynamic-sql-pivoting-stealing-antons-thunder/" target="_blank">solution using custom types</a>. This enables you to pivot the results of a query. For example:</p>
select *
from table ( pivot (
&#39;select noc,
medal,
count(distinct sport ||&#39;&#39;#&#39;&#39;|| event ||&#39;&#39;#&#39;&#39;|| gender ) count_medal
from olympic_medal_winners
group by noc, medal&#39;
)
);
<p>This is really cool. But it comes with notable parsing overheads. And if your query creates a large number of columns the SQL it generates may be too large for the variables it uses!</p>
<p>Use with care.</p>
<p>So far we&#39;ve been working with the raw event results. But what if you&#39;ve only got the final medal table, like this:</p>
NOC GOLD_MEDALS SILVER_MEDALS BRONZE_MEDALS
USA 46 37 38
GBR 27 23 17
CHN 26 18 26
RUS 19 18 19
GER 17 10 15
...
<p>And you want to write SQL converting the gold, silver and bronze columns into rows?</p>
<p>It&#39;s time for the unpivot!</p>
<a name="unpivot"></a>
Converting Columns To Rows (UNPIVOT)
<p><img src="http://cdn.app.compendium.com/uploads/user/e7c690e8-6ff9-102a-ac6d-e4aebca50425/f4a5b21d-66fa-4885-92bf-c4e81c06d916/Image/9a0a024259cfb7a54def52751d9862f1/unpivot.png" /></p>
<p>Unpivot works in a similar way to pivot. You need to supply three things:</p>
<ol>
<li>The name of a new column that will store the values from the old columns</li>
<li>Another new column showing the source of these values</li>
<li>The list of columns that hold the values you want to become rows.</li>
</ol>
<p>So for the medal table this is:</p>
<ol>
<li>You want the count of medals. So medal_count</li>
<li>You want the colour of each medal: medal_colour</li>
<li>The columns to become rows are Gold, Silver and Bronze</li>
</ol>
<div>This gives an unpivot like:</div>
select * from olympic_medal_tables
unpivot (medal_count for medal_colour in (
gold_medals as &#39;GOLD&#39;,
silver_medals as &#39;SILVER&#39;,
bronze_medals as &#39;BRONZE&#39;
))
order by noc
fetch first 6 rows only;
NOC MEDAL_COLOUR MEDAL_COUNT
ALG GOLD 0
ALG BRONZE 0
ALG SILVER 2
ARG GOLD 3
ARG BRONZE 0
ARG SILVER 1
<p>Nice and easy.</p>
Unpivot Multiple Groups
<p>But what if there are multiple sets of columns you want to unpivot? For example, medals won and distinct sports they were won in? e.g.</p>
NOC GOLD_MEDALS GOLD_SPORTS SILVER_MEDALS SILVER_SPORTS BRONZE_MEDALS BRONZE_SPORTS
USA 46 14 37 13 38 17
GBR 27 16 23 15 17 11
CHN 26 10 18 11 26 13
RUS 19 10 18 11 19 11
GER 17 9 10 8 15 13
<p>As with pivot, this is easy. If a little cumbersome to write!</p>
<p>First name the two columns you want in the output. Then specify each pair of columns that provide the values for the rows:</p>
select * from olympic_medal_tables
unpivot ((medal_count, sport_count) for medal_colour in (
(gold_medals, gold_sports) as &#39;GOLD&#39;,
(silver_medals, silver_sports) as &#39;SILVER&#39;,
(bronze_medals, bronze_sports) as &#39;BRONZE&#39;
))
fetch first 9 rows only;
NOC MEDAL_COLOUR MEDAL_COUNT SPORT_COUNT
USA GOLD 46 14
USA SILVER 37 13
USA BRONZE 38 17
GBR GOLD 27 16
GBR SILVER 23 15
GBR BRONZE 17 11
CHN GOLD 26 10
CHN SILVER 18 11
CHN BRONZE 26 13
<p>Note the aliases for the column pairings. Without these medal column contains the concatenation of the source column names. For example GOLD_MEDALS_GOLD_SPORTS.</p>
<p>Unfortunately you can&#39;t split the values back out. So if all you have is the final medal table, you can&#39;t go back and get the rows by sport, event and so on.</p>
<p>Unless your source columns are a comma separated list. Say, the names of the athletes who won each medal!</p>
<p>In this case, you can unpivot to get the medals and athlete lists as rows:</p>
select * from olympic_medal_tables
unpivot ((medal_count, athletes) for medal_colour in (
(gold_medals, gold_athletes) as &#39;GOLD&#39;,
(silver_medals, silver_athletes) as &#39;SILVER&#39;,
(bronze_medals, bronze_athletes) as &#39;BRONZE&#39;
))
where medal_colour = &#39;GOLD&#39;
and medal_count = 2
order by noc
fetch first 3 rows only;
NOC MEDAL_COLOUR MEDAL_COUNT ATHLETES
BEL GOLD 2 THIAM Nafissatou,VAN AVERMAET Greg
DEN GOLD 2 BLUME Pernille,Denmark
GEO GOLD 2 KHINCHEGASHVILI Vladimer,TALAKHADZE Lasha
<p>You can now split the names into separate rows. There are several ways to do this. Here I&#39;ve used Stew Ashton&#39;s <a href="https://stewashton.wordpress.com/2016/08/01/splitting-strings-surprise/" target="_blank">XQuery tokenizing solution</a>:</p>
with rws as (
select * from olympic_medal_tables
unpivot ((medal_count, athletes) for medal_colour in (
(gold_medals, gold_athletes) as &#39;GOLD&#39;,
(silver_medals, silver_athletes) as &#39;SILVER&#39;,
(bronze_medals, bronze_athletes) as &#39;BRONZE&#39;
))
where medal_colour = &#39;GOLD&#39;
and medal_count = 2
)
select noc, athlete
from rws, xmltable (
&#39;if (contains($X,&quot;,&quot;)) then ora:tokenize($X,&quot;\,&quot;) else $X&#39;
passing athletes as X
columns athlete varchar2(4000) path &#39;.&#39;
)
order by 1, 2
fetch first 6 rows only;
NOC ATHLETE
BEL THIAM Nafissatou
BEL VAN AVERMAET Greg
DEN BLUME Pernille
DEN Denmark
GEO KHINCHEGASHVILI Vladimer
GEO TALAKHADZE Lasha
<p>We&#39;ve seen how to convert rows into columns. And columns back into rows. But what if you want to both at the same time?</p>
<a name="transpose"></a>
Swapping Row and Columns (Transpose)
<p>Let&#39;s revisit the medals won by each country in each sport table:</p>
NOC ATH GYM CYC BOX SAI
BRA 1 3 0 1 2
CHN 6 2 2 4 1
DEN 1 0 2 0 3
ESP 2 0 0 0 0
ETH 8 0 0 0 0
GRE 1 1 0 0 2
<p>Your boss has asked you to switch it over, so countries are across the top and sports down the side. Like so:</p>
SPORT BRA CHN DEN ESP ETH GRE
Athletics 1 6 1 2 8 1
Artistic Gym 3 2 0 0 0 1
Boxing 1 4 0 0 0 0
Sailing 2 1 3 0 0 2
Track Cycling 0 2 2 0 0 0
<p>Aka a transpose.</p>
<p>But there isn&#39;t an in-built transpose function in SQL.<span> </span>So what&#39;s the solution?</p>
<p>Pivot and unpivot!</p>
<p>You can chain these together like so:</p>
select * from olympic_country_sport_medals
pivot (
sum(ath) ath, sum(box) box, sum(gym) gym, sum(sai) sai, sum(cyc) cyc
for noc in (&#39;BRA&#39; BRA, &#39;CHN&#39; CHN, &#39;DEN&#39; DEN, &#39;ESP&#39; ESP, &#39;ETH&#39; ETH, &#39;GRE&#39; GRE )
)
unpivot (
(BRA, CHN, DEN, ESP, ETH, GRE ) for sport in (
(BRA_ATH, CHN_ATH, DEN_ATH, ESP_ATH, ETH_ATH, GRE_ATH) as &#39;Athletics&#39;,
(BRA_GYM, CHN_GYM, DEN_GYM, ESP_GYM, ETH_GYM, GRE_GYM) as &#39;Artistic Gym&#39;,
(BRA_BOX, CHN_BOX, DEN_BOX, ESP_BOX, ETH_BOX, GRE_BOX) as &#39;Boxing&#39;,
(BRA_SAI, CHN_SAI, DEN_SAI, ESP_SAI, ETH_SAI, GRE_SAI) as &#39;Sailing&#39;,
(BRA_CYC, CHN_CYC, DEN_CYC, ESP_CYC, ETH_CYC, GRE_CYC) as &#39;Track Cycling&#39;
)
);
SPORT BRA CHN DEN ESP ETH GRE
Athletics 1 6 1 2 8 1
Artistic Gym 3 2 0 0 0 1
Boxing 1 4 0 0 0 0
Sailing 2 1 3 0 0 2
Track Cycling 0 2 2 0 0 0
<p>Hmmm, that looks painful. For each extra column you want to transpose, you need to add a sum to the pivot. Then a big list of countries in unpivot&#39;s sport list. This is tedious.</p>
<p>Fortunately there is an easier way. Unpivot first, then pivot!</p>
<p>Do this and your SQL is:</p>
select * from olympic_country_sport_medals
unpivot (
(medals) for sport in ( ath, box, gym, sai, cyc )
)
pivot (
sum(medals) for noc in (
&#39;BRA&#39; BRA, &#39;CHN&#39; CHN, &#39;DEN&#39; DEN, &#39;ESP&#39; ESP, &#39;ETH&#39; ETH, &#39;GRE&#39; GRE
)
);
SPORT BRA CHN DEN ESP ETH GRE
CYC 0 2 2 0 0 0
BOX 1 4 0 0 0 0
GYM 3 2 0 0 0 1
SAI 2 1 3 0 0 2
ATH 1 6 1 2 8 1
<p>Much better :)</p>
<p>Adding more sports is simply a matter of adding a value to the unpivot. And for more countries you just extend the pivot country list.</p>
Conclusion
<p>You&#39;ve seen how the SQL pivot and unpivot operators can enable powerful data transformations. Between these two you can swap your rows and columns over however you like!</p>
<p>Just remember the three step process for each. For pivot it&#39;s the column(s) you&#39;re pivoting on, the values defining the new columns and the functions giving the values for these.</p>
<p>With unpivot you need the column which states the source of the values, what these sources are and the list of columns you want to convert to rows.</p>
<p>All the examples so far are with a single table. Often you&#39;ll want to use pivots with joins. You can pivot or unpivot the result of a join, for example:</p>
select * from t1 join t2 on ...
pivot ( ... )
<p>Or</p>
select * from t1 join t2 on ...
unpivot ( ... )
<p>So you can embed these in much larger SQL queries.</p>
<a name="scripts"></a>
Scripts
<p>If you want the scripts in this post, head over to <a href="https://livesql.oracle.com/apex/livesql/file/content_DT38BTKONDYSUSJYEUPQ7V1QR.html" target="_blank">LiveSQL</a>. There you can find and run the demos on a sample of the data. Or you can use this table:</p>
create table olympic_medal_winners (
olympic_year int,
sport varchar2( 30 ),
gender varchar2( 1 ),
event varchar2( 128 ),
medal varchar2( 10 ),
noc varchar2( 3 ),
athlete varchar2( 128 )
);
<p>And load the medal winners into it from <a href="https://app.compendium.com/api/post_attachments/8222a67a-b896-43e9-bfcf-1eb7995b0392/view">this CSV</a>. You can then create the tables for the unpivot and transpose examples from this like so:</p>
create table olympic_medal_tables as
select * from (
select noc, medal, sport, event, gender
from olympic_medal_winners
)
pivot ( count(distinct sport ||&#39;#&#39;|| event ||&#39;#&#39;||gender ) for medal in (
&#39;Gold&#39; gold_medals, &#39;Silver&#39; silver_medals, &#39;Bronze&#39; bronze_medals
))
order by 2 desc, 3 desc, 4 desc;
-- Medal table with distinct sports
drop table olympic_medal_tables purge;
create table olympic_medal_tables as
select * from (
select noc, medal, sport, event, gender
from olympic_medal_winners
)
pivot ( count(distinct sport ||&#39;#&#39;|| event ||&#39;#&#39;||gender ) medals,
count(distinct sport) sports
for medal in (
&#39;Gold&#39; gold, &#39;Silver&#39; silver, &#39;Bronze&#39; bronze
))
order by 2 desc, 4 desc, 6 desc;
-- Medal table with athlete names
drop table olympic_medal_tables purge;
create table olympic_medal_tables as
select * from (
select noc, medal, sport, event, gender, athlete
from olympic_medal_winners
)
pivot ( count(distinct sport ||&#39;#&#39;|| event ||&#39;#&#39;||gender ) medals,
listagg(athlete, &#39;,&#39;) within group (order by athlete) athletes
for medal in (
&#39;Gold&#39; gold, &#39;Silver&#39; silver, &#39;Bronze&#39; bronze
))
order by 2 desc, 4 desc, 6 desc;
create table olympic_country_sport_medals as
select * from (
select noc, sport
from olympic_medal_winners
)
pivot (count(sport) for sport in (
&#39;Athletics&#39; as ath, &#39;Artistic Gymnastics&#39; as gym, &#39;Cycling Track&#39; as cyc,
&#39;Boxing&#39; as box, &#39;Sailing&#39; as sai
)
)
order by 1;
Still stuck?
<p><img src="http://cdn.app.compendium.com/uploads/user/e7c690e8-6ff9-102a-ac6d-e4aebca50425/f4a5b21d-66fa-4885-92bf-c4e81c06d916/Image/0d203b8bdfe5e472d3ec2c0bd871412d/210h.jpg" /></p>
<em>Ryan McGuire / <a href="http://www.gratisography.com/" target="_blank">Gratisography</a></em>
<p style="margin-top:15px;">Check the docs for <a href="http://docs.oracle.com/database/121/SQLRF/statements_10002.htm#CHDFIIDD" target="_blank">more examples</a> of <a href="http://docs.oracle.com/database/121/DWHSG/analysis.htm#DWHSG0209" target="_blank">how to use pivot and unpivot</a>. Or you can head over to <a href="https://asktom.oracle.com" target="_blank">Ask Tom</a> where Connor and I can help you write your SQL. Just remember to provide a test case!</p>
<p><em>Note: the examples use the &quot;fetch first&quot; syntax to get the first N rows. This is only available in Oracle Database 12c and above. If you want to do top-N queries in releases before this, you need to use the <a href="http://www.oracle.com/technetwork/issue-archive/2006/06-sep/o56asktom-086197.html" target="_blank">rownum trick</a>. </em></p>
<p><strong>UPDATE 25 Feb, 2019</strong>: Fixed link to CSV data file</p>
Analytical SQLThu, 08 Sep 2016 11:12:00 +0000https://blogs.oracle.com/sql/how-to-convert-rows-to-columns-and-back-again-with-sql-aka-pivot-and-unpivotChris SaxonHow to Recover Data (Without a Backup!)https://blogs.oracle.com/sql/how-to-recover-data-without-a-backup
<img src="http://cdn.app.compendium.com/uploads/user/e7c690e8-6ff9-102a-ac6d-e4aebca50425/f4a5b21d-66fa-4885-92bf-c4e81c06d916/Image/6ffa008d0ca2c4c694cdc9246a8a7658/145h.jpg" />
<em>Ryan McGuire / <a href="http://www.gratisography.com/" target="_blank">Gratisography</a></em>
<p>It&#39;s the classic career-limiting maneuver: accidentally deleting data you weren&#39;t meant to. It&#39;s easy to do this as the result of mistakes such as:</p>
<ul>
<li>Running a test script on production</li>
<li>Getting the where clause wrong for a delete</li>
</ul>
<ul>
</ul>
<p>Doing this in test or dev is likely to bring the ire of your colleagues in IT. Do this in production and key parts of the business may come to a stop!</p>
<p>In either case you&#39;ll want to get your data back as quickly as possible.</p>
<p>Restoring from backup can be a time-consuming process. Time you don&#39;t have in extreme cases.</p>
<p>Luckily Oracle can help you recover from many mistakes quickly - without needing a backup! In this post we&#39;ll look at how to undo the damage in the following cases:</p>
<ul style="margin-top: 0cm;" type="disc">
<li><a href="#Restore-a-Whole-Table">Restore a Whole Table</a></li>
<li><a href="#Recover-a-Few-Rows">Recover a Few Rows</a></li>
<li><a href="#Recover-a-Few-Rows++">Recover a Few Rows++</a></li>
<li><a href="#Restore-Dropped-Tables">Restore Dropped Tables</a></li>
<li><a href="#Revert-the-Whole-Database">Revert the Whole Database!</a></li>
</ul>
<p>Ready? Let&#39;s begin!</p>
<a name="Restore-a-Whole-Table"> </a>
How to Restore a Whole Table
<p>It&#39;s a classic rookie mistake: running a delete without a where clause. And then committing it!</p>
<p>Here you need to recover all the data. Using <a href="http://docs.oracle.com/database/121/SQLRF/statements_9012.htm#SQLRF01802" target="_blank">Flashback Table</a>, you can return a whole table to an earlier state. All you need to do is run:</p>
flashback table &lt;table&gt; to timestamp &lt;when it was good&gt;;
<p>For example, execute:</p>
flashback table orders to timestamp systimestamp - interval &#39;1&#39; hour;
<p>And Oracle restores the table its state one hour ago. Handy if you&rsquo;ve just deleted all the rows!</p>
<p>To use this, you must enable row movement:</p>
alter table &lt;table&gt; enable row movement;
<p>If you haven&rsquo;t done this, you&rsquo;ll get the following error:</p>
ORA-08189: cannot flashback the table because row movement is not enabled
<p>This is great if you&rsquo;ve accidentally deleted or updated the whole table. But if there are only a handful of rows you need to recover it&rsquo;s excessive. You&rsquo;ve used stick of dynamite to kill an ant.</p>
<p>Even if you need to recover a large section of a table, flashing it back loses any changes made after the time you&rsquo;re restoring it to. In most production systems there will be new rows you want to keep!</p>
<p>So this is handy for worst-case scenarios. It&rsquo;s also useful for returning a table to a known state after testing. But for small finger trouble issues, it&rsquo;s more likely you need to recover a handful of rows.</p>
<a name="Recover-a-Few-Rows"> </a>
How to Recover a Few Rows
<img src="http://cdn.app.compendium.com/uploads/user/e7c690e8-6ff9-102a-ac6d-e4aebca50425/f4a5b21d-66fa-4885-92bf-c4e81c06d916/Image/18dd0fd80fe56b7cf86d741759182a50/117h.jpg" />
<em>Ryan McGuire / <a href="http://www.gratisography.com/" target="_blank">Gratisography</a></em>
<p>So what do you do if there are a small number of rows you need to restore? Clearly Flashback Table is overkill. You need something more nuanced.</p>
<p>Enter <a href="http://docs.oracle.com/database/121/ADFNS/adfns_flashback.htm#ADFNS01003" target="_blank">Flashback Query</a>. This enables you see the contents of table at a point in time in the past. To do this you just add the &ldquo;as of&rdquo; clause after your table.</p>
<p>To see it at a given time, use &quot;as of timestamp&quot;. For example, to see how it looked one hour ago, use:</p>
select * from &lt;table&gt; as of timestamp systimestamp - interval &#39;1&#39; hour;
<p>Or you can use a database SCN with:</p>
select * from &lt;table&gt; as of scn 1234567;
Salvaging the Deleted Rows
<p>If you know which rows were removed, add the appropriate where clause to your SQL. Then pass the result of this to an insert. For example:</p>
insert into table
select * from &lt;table&gt; as of timestamp sysdate &ndash; interval &#39;1&#39; hour
where &lt;conditions to find the rows&gt;;
<p>And you&rsquo;ll have your missing data back!</p>
<p>If you&rsquo;re not sure which rows are gone, you can find the deleted ones using minus. This enables you to compare the current state of the table how it looked before the time of the disaster. The SQL find rows which were in the table an hour ago, but not anymore is:</p>
select * from &lt;table&gt; as of timestamp sysdate &ndash; interval &#39;1&#39; hour
minus
select * from &lt;table&gt;;
<p>To recover these, insert the result of this query!</p>
insert into &lt;table&gt;
select * from &lt;table&gt; as of timestamp sysdate &ndash; interval &#39;1&#39; hour
minus
select * from &lt;table&gt;;
<p>Note this will include <strong>all</strong> rows deleted in the past hour. If there are genuine deletions, you&rsquo;ll need to remove them again.</p>
Recover Overwritten Values
<p>What if the rows weren&rsquo;t deleted, just updated? And you need to restore the original values, but don&rsquo;t know what they are?</p>
<p>Then you can use Flashback Query in an update too:</p>
update &lt;table&gt; cur
set (col1, col2, col3) = (
select col1, col2, col3 from &lt;table&gt;
as of timestamp systimestamp &ndash; interval &#39;1&#39; hour old
where cur.primary_key = old.primary_key
)
where &lt;rows to update&gt;;
<p>This is fantastic. But Flashback Query has a couple of limitations:</p>
<ul style="margin-top: 0cm;" type="disc">
<li>Oracle only ensures you can query as far back as the value of your &ldquo;undo_retention&rdquo; parameter.</li>
<li>Oracle is unable to query across many forms of DDL. So if you change a table&#39;s structure there&#39;s a good chance Flashback Query will fail.</li>
</ul>
<p>By default the undo retention is 900 seconds. That&rsquo;s just 15 minutes. Unless you&rsquo;re quick there&rsquo;s a good chance you&rsquo;ll miss this window in busy production systems.</p>
<p>You can overcome this by increasing the retention time. For example, to increase it to one day, run:</p>
alter system set undo_retention = 86400 scope = both;
<p>Take care before doing this. Oracle uses the undo tablespace to run flashback queries. So increasing the retention means you&rsquo;ll need to make this larger to support flashback. This could lead to a big jump in storage requirements.</p>
<p>The second problem normally rears its head after releases. If you&rsquo;ve run DDL against a table, there&#39;s a good chance you&rsquo;ll get:</p>
ORA-01466: unable to read data - table definition has changed
<p>This is frustrating. It&rsquo;s useful to be able to compare a table before and after a release. Particularly when something&rsquo;s gone wrong! If you can compare the before and after states of the table it can make diagnosis simple.</p>
<p>Also note that truncate is DDL in Oracle. So if you&rsquo;ve used this method to wipe a table, there&rsquo;s no way back!</p>
<p>Fortunately, you can overcome both of these issues.</p>
<a name="Recover-a-Few-Rows"> </a>
<a name="Recover-a-Few-Rows++"></a>
How to Recover a Few Rows++
<p>Flashback Data Archive powers up Flashback Query. It does this by storing the changes in tables. This means you have a permanent store instead of relying on undo.</p>
<p>To use this, first you need to create an archive. You can do this with the following SQL:</p>
create flashback archive &lt;archive&gt; tablespace &lt;tablespace&gt; retention 1 year;
<p>The retention clause states how long you want to keep your history. You can specify this as a number of days, months or years.</p>
<p>Once you have the archive in place, simply alter your tables to use it:</p>
alter table &lt;table&gt; flashback archive &lt;archive&gt;;
<p>And you&rsquo;re done!</p>
<p>You can then recover data using the same method described above. But with the bonuses of:</p>
<ul>
<li>Being able to query across DDL</li>
<li>Having a larger window of time to recover data</li>
</ul>
<p>Best of all, as of 11.2.0.4, Flashback Data Archive is free*!</p>
<p>One word of caution: when you enable it, Oracle creates history tables. It does this in a background process. So this setup can fail <a href="https://blogs.oracle.com/sql/entry/why_you_can_get_ora" target="_blank">without giving you an error</a>. Be sure to check this is working before you rely on it!</p>
<p>So far we&rsquo;ve looked at recovering rows. But what happens if you <a href="https://blogs.oracle.com/sql/how-to-create-alter-and-drop-tables-in-sql#drop">drop a table</a>?! Can flashback help here?</p>
<a name="Restore-Dropped-Tables"> </a>
How to Restore Dropped Tables
<p>In 10g, Oracle introduced the recyclebin. Just like the recyclebin in your &ldquo;favourite&rdquo; OS, you can recover objects placed in here.</p>
<p>To do so, simply run:</p>
flashback table &lt;table&gt; to before drop;
<p>And you&rsquo;ll have your table back!</p>
<p>You can view the contents of the recyclebin with this SQL:</p>
select * from recyclebin;
<p>To see the names of tables you&rsquo;ve dropped, check original_name.</p>
<p>This only contains objects your user owned. If you have the appropriate permissions, you can query dba_recyclebin. This enables you to see dropped objects for all users.</p>
<p><img src="http://cdn.app.compendium.com/uploads/user/e7c690e8-6ff9-102a-ac6d-e4aebca50425/f4a5b21d-66fa-4885-92bf-c4e81c06d916/Image/c602391403de3d06b8c95feccf14a9b4/256px_recycle001.png" /></p>
<em>By Users Cbuckley, Jpowell on en.wikipedia [Public domain], <a href="https://commons.wikimedia.org/wiki/File%3ARecycle001.svg" target="_blank">via Wikimedia Commons</a></em>
<p>Tables in the recyclebin still consume space. If you&rsquo;re sure you want to permanently drop a table, use the purge option:</p>
drop table &lt;table&gt; purge;
<p>And the table is gone for good. Or you if you want a safety net, drop it normally. Then remove it from the recyclebin with:</p>
purge table &lt;table&gt;;
<p>If you want to recover all the space the recyclebin is using, clear it out with:</p>
purge recyclebin;
<p>And it&rsquo;s empty!</p>
<p>Note that the recyclebin only applies when you use drop table. If you take other actions that remove tables, e.g. drop user or drop tablespace, they are gone for good.</p>
<p>These solutions are all great if you&rsquo;re dealing with a single table. But what if something more serious has happened? What if someone managed to run a truncate cascade wiping out your whole database?</p>
<p>Individually recovering tables could take some time.</p>
<a name="Revert-the-Whole-Database"> </a>
How to Revert the Whole Database
<p>If someone has accidentally or maliciously has trashed your data, figuring out what to restore could be a long process. With <a href="http://docs.oracle.com/database/121/SQLRF/statements_9011.htm#SQLRF01801" target="_blank">Flashback Database</a>, you can restore a whole database back to an earlier time.</p>
<p>Oracle enables this with flashback logs. It stores these in the fast recovery area.</p>
<p>To use Flashback Database you need to do some initial setup. There are two ways to enable this:</p>
<ul style="margin-top: 0cm;">
<li>Enable Flashback Database</li>
<li>Create a restore point</li>
</ul>
Enable Flashback Database
<p>Firstly, your database must be running in archivelog mode. Assuming this is the case, the process for enabling it is:</p>
<ol style="padding: 0px;">
<li>Configure the fast recovery area</li>
<li>Set the DB_flashback_retention_target parameter (optional)</li>
<li>Enable Flashback Database</li>
</ol>
<p>For step one, you need to set a couple of parameters. These are DB_RECOVERY_FILE_DEST_SIZE and DB_RECOVERY_FILE_DEST. These control how much space there is available for the logs and where they go respectively. For example:</p>
alter system set DB_RECOVERY_FILE_DEST = &#39;/u01/oradata/recovery_area&#39; scope=both;
alter system set DB_RECOVERY_FILE_DEST_SIZE = 10G scope=both;
<p>Set DB_flashback_retention_target to give the upper limit for how far back you can flashback the database. This is specified in minutes. So to set a maximum duration of one week, run:</p>
alter system set DB_flashback_retention_target = 10080 scope=both;
<p>Just ensure that you set the DB_RECOVERY_FILE_DEST_SIZE large enough to support this! For further discussion about this, <a href="http://docs.oracle.com/database/121/BRADV/rcmconfb.htm#BRADV89422" target="_blank">check the docs</a>.</p>
<p>The final step is simple. Just run:</p>
alter database flashback on;
<p>And you&#39;re done!</p>
<p>Note that enabling this adds some overhead. Oracle must log all changes you make to the data. So the more inserts, updates and deletes you have the greater the overhead.</p>
Create a Restore Point
<p>Doing this is easy. Simply run:</p>
create restore point &lt;restore_point&gt; guarantee flashback database;
<p>The guarantee clause is optional.</p>
<p>Without this, Oracle will age out old restore points. So you may be unable to go back to the time of a particular restore point.</p>
<p>With it, Oracle ensures you can always recover back to the time you created it. As with Flashback Database, Oracle stores the logs to support this in the fast recovery area.</p>
<p>To recover the space you must drop the restore point manually.</p>
<p>If you forget to do this then you can run out of space. And your database may grind to a halt!</p>
<p>This seems risky. So why would you want to create a guaranteed restore point?</p>
<p>Two key use cases for this are:</p>
<ul style="margin-top: 0cm;" type="disc">
<li>An extra safety net for database releases</li>
<li>Reverting test databases to a known state</li>
</ul>
<p>Database releases are notoriously difficult to undo. Especially if you&rsquo;ve dropped columns or other breaking schema changes. Flashing back is quicker and easier than unpicking the errors if you have unexpected failures.</p>
<p>Using a guaranteed restore point ensures you have this fall back. With a normal restore point, you may find it was aged out in the release process. Just ensure you have <a href="http://docs.oracle.com/database/121/BRADV/rcmmaint.htm#BRADV89614" target="_blank">monitoring on your Fast Recovery Area</a>.</p>
<p>Reverting a database after running tests another great use case. As with releases, undoing the changes can be time consuming and tricky.</p>
<p>Restore points make it easy to reset after test runs. Instead of worrying about how to get back to the original state, just flashback once they&rsquo;re complete!</p>
<p>This is super handy when it comes to preparing and testing release scripts. Writing the scripts for complex upgrades can take a few tries to get right. Being able to flashback the database after each try is a huge time saver.</p>
<p>Flashback Database could also form part of your Continuous Integration and DevOps strategies. Just build a script to flash the database back after each test run.</p>
<p>In any case, ensure you have a process for removing guaranteed restore points. Without this you may find your job finishes as quickly as your database does!</p>
Flashing Back a Database
<p>With a restore point in place or flashback enabled you&rsquo;re all set! If disaster strikes, you can return to a point in the past by:</p>
<ol style="padding: 0px;">
<li>
<p>Shutting down the database:</p>
startup mount
</li>
<li>
<p>Starting it up in mount mode</p>
shutdown immediate
</li>
<li>
<p>Running flashback:</p>
<ol style="padding: 0px;" type="a">
<li>To an SCN
flashback database to scn 12345;
</li>
<li>To a point in time
flashback database to time
&quot;to_date(&#39;01/01/2016&#39;, &#39;dd/mm/yyyy&#39;)&quot;;
</li>
<li>To a restore point
flashback database to restore point &lt;restore_point&gt;
</li>
</ol>
</li>
<li>Reopen the database with resetlogs
alter database open resetlogs;
</li>
</ol>
<p>And you&rsquo;re done!</p>
<img align="middle" src="http://cdn.app.compendium.com/uploads/user/e7c690e8-6ff9-102a-ac6d-e4aebca50425/f4a5b21d-66fa-4885-92bf-c4e81c06d916/Image/bc56a4dd7f6788ca3a6e4e3ec36824c8/lightning_bolt.png" />
<em>benluna12 / <a href="https://pixabay.com" target="_blank">Pixabay</a></em>
<p>If you want to take extra care, you can open the database in read only mode. Do this between steps three and four above. If you do, you must restart the database in mount mode before opening it with resetlogs.</p>
Summary
<p>The term Flashback in Oracle covers a number of technologies. All these serve the goal of enabling you to return to a previous state quickly and easily. If you want more details on these, check out the following sections of the docs:</p>
<ul>
<li><a href="http://docs.oracle.com/database/121/ADFNS/adfns_flashback.htm#ADFNS1008" target="_blank">Using Oracle Flashback Technology</a></li>
<li><a href="http://docs.oracle.com/database/121/BRADV/flashdb.htm#BRADV71000" target="_blank">Using Flashback Database and Restore Points</a></li>
<li><a href="http://docs.oracle.com/database/121/BRADV/rcmflash.htm#BRADV80055" target="_blank">Performing Flashback and Database Point-in-Time Recovery</a></li>
</ul>
<p>For examples of Flashback Table, Query and Drop in action, check this <a href="https://livesql.oracle.com/apex/livesql/file/content_C8QM31V4O543MILYWY724EF44.html" target="_blank">script in LiveSQL</a>.</p>
<p>Flashback in its various forms has saved me many times in my career. The situations ranged from restoring rows mistakenly deleted by users to recovering tables dropped by accidentally running test scripts against production!</p>
<p>Flashback is great for overcoming these accidental mishaps. But there are some things it can&rsquo;t recover from. For example, people deleting database files in the OS. There&rsquo;s no substitute for full protection.</p>
<p>Remember, always have a backup!</p>
<p>How about you? Has flashback has saved you or your users at any time?</p>
<p>If so, please let us know in the comments!</p>
<p><em>* Applies to Basic Flashback Archive only. Optimization requires EE license and the Advanced Compression Option. To use Flashback Data Archive in 11.2.0.3 and earlier you had to purchase the relevant option.</em></p>
<p><strong>UPDATED</strong>&nbsp;1 Sep 2018: Fixed formatting</p>
Big Data SQLThu, 05 May 2016 08:56:00 +0000https://blogs.oracle.com/sql/how-to-recover-data-without-a-backupChris SaxonHow to Create an Execution Planhttps://blogs.oracle.com/sql/how-to-create-an-execution-plan
<p>When you&rsquo;re trying to get SQL to run faster there&rsquo;s one thing that&rsquo;s vital: an execution plan. In this post we&rsquo;ll investigate four ways to create these:</p>
<ul>
<li><a href="#autotrace">Autotrace</a></li>
<li><a href="#sqlmonitor">SQL Monitor</a></li>
<li><a href="#tkprof">TKPROF</a></li>
<li><a href="#dbmsxplan">DBMS_XPlan</a></li>
</ul>
<p>But before we begin let&#39;s answer the question:</p>
What is an Execution Plan?
<p>The plan for a SQL statement is a set of instructions. This tells the database how to access the data and join it together.</p>
<p>Plans come in two varieties:</p>
<ul style="margin-top: 0cm;" type="disc">
<li>Explain</li>
<li>Execution</li>
</ul>
<p>These sound similar. But they&rsquo;re different. To understand why consider this example.</p>
<p>You&rsquo;re making a car journey. You plan it beforehand. This is the route you expect to take.</p>
<p>But just as you&rsquo;re going to leave, you hear on the news that there&rsquo;s been an accident on your chosen route. This will make it much slower. So you go a different way.</p>
<p><img src="http://cdn.app.compendium.com/uploads/user/e7c690e8-6ff9-102a-ac6d-e4aebca50425/f4a5b21d-66fa-4885-92bf-c4e81c06d916/Image/154f9c395fa579f8a218eb9e786b9b69/directions_plan_vs_actual.png" /></p>
<p>There are two routes here. The one you expected to take and the one you actually took.</p>
<p>After you arrive you wonder whether you could have completed the journey faster. To figure this out, you need to look at where you went. Not where you planned to go.</p>
<p>An explain plan <em>predicts</em> how Oracle will process your query.</p>
<p>An execution plan describes the steps it <em>actually took</em>.</p>
<p>Just as in the driving example above, Oracle may use a different route than the one it predicted. As <a href="http://tkyte.blogspot.co.uk/2007/04/when-explanation-doesn-sound-quite.html" target="_blank">Tom Kyte discusses</a>, there are several reasons this could happen. So to diagnose your queries, you need to know what Oracle did. Not what it guessed it might do!</p>
Side note
<p>&quot;Explain plan&quot; sounds a lot like &quot;execution plan&quot;. This can lead to confusion. Many times when I ask people to supply an execution plan they provide one of the explain variety instead.</p>
<p>I think it would be better if we gave these very different names. For example, calling execution plans &ldquo;described paths&rdquo; or similar. This would help alert people to the fact these are different things (though they look similar) and reduce the chance of confusion.</p>
<p>Top tip: If someone asks you for &ldquo;the plan&rdquo; they&rsquo;re usually looking for one of the execution variety. I.e. what really happened.</p>
<p>So with that cleared up, how do you go about gathering them?</p>
<a name="autotrace"></a>Autotrace
<p>First up, autotrace. This is a freely available tool. You can enable it in SQL*Plus using the &ldquo;set autotrace&rdquo; command.</p>
<p>With this on, after each query Oracle displays its output. Below this is the plan along with its performance stats:</p>
<p><img src="http://cdn.app.compendium.com/uploads/user/e7c690e8-6ff9-102a-ac6d-e4aebca50425/f4a5b21d-66fa-4885-92bf-c4e81c06d916/Image/84d58e23c7b18006670b39af82658cc9/autotrace_sql_plus.png" /></p>
<p>If your query returns hundreds or thousands of rows you have to wait for Oracle to display them all. This can be a pain.</p>
<p>Luckily you can suppress the results with the trace[only] option. You can also choose to show just the plan or the stats. For example, to hide the query output and show just the stats, use:</p>
set autotrace trace stat
<p>To include the output and the plan, enter</p>
set autotrace on exp
<p>Once you&#39;re finished you can switch it off with:</p>
set autotrace off
<p>While using SQL*Plus is better than nothing, it has several limitations:</p>
<ul>
<li><span>It doesn&rsquo;t show the stats (ouput rows, buffer gets, etc.) for each step in the plan</span></li>
<li><span>It uses explain plans, so what you see may not be <a href="http://kerryosborne.oracle-guy.com/2010/02/autotrace-lies/" target="_blank">what Oracle did</a>!</span></li>
</ul>
<p>So while it&rsquo;s useful for a quick finger in the air, there are better versions available.</p>
<p>Such as:</p>
Autotrace in SQL Developer
<p>This is a step up over the SQL*Plus version. It displays the stats for every step of the plan. So for each operation you can see metrics such as:</p>
<ul>
<li>How many buffer gets it used</li>
<li>How long it took to run</li>
<li>How many disk reads and writes it did</li>
</ul>
<p>This makes it much easier to which points are consuming the most resources.</p>
<p>To see how to configure and run it, check out this video:</p>
<p></p>
<p>Besides the step-by-step breakdown, autotrace in <a href="http://www.oracle.com/sqldeveloper" target="_blank">SQL Developer</a> has more advantages over SQL*Plus. These include:</p>
<ul>
<li><span>You can expand and collapse parts of the plan. This makes it much easier to see what&rsquo;s going on</span></li>
<li><span>You can compare execution plans. A fantastic tool when you&rsquo;re trying to find the difference between two large plans.</span></li>
</ul>
<p>This is a big step up over the SQL*Plus version. So from now on when I refer to autotrace I mean the SQL Developer version.</p>
<p>A quick word of warning. By default SQL Developer only fetches the number of rows according to the &quot;<a href="http://www.thatjeffsmith.com/archive/2014/09/30-sql-developer-tips-in-30-days-day-7-the-array-fetch-size-preference/" target="_blank">SQL array fetch size</a>&quot; parameter. For queries that return a large number of rows this could miss a big chunk of the work. Ensure you set it to &quot;<a href="http://www.thatjeffsmith.com/archive/2013/07/explain-plan-and-autotrace-enhancements-in-oracle-sql-developer-4/" target="_blank">Fetch all rows</a>&quot;.</p>
<p>Often when you&#39;re tuning queries you want to save the plan for later reference or to share with others. To do this, right click the plan and you&#39;ll get an option to &quot;Export HTML&quot;:</p>
<p><img src="http://cdn.app.compendium.com/uploads/user/e7c690e8-6ff9-102a-ac6d-e4aebca50425/f4a5b21d-66fa-4885-92bf-c4e81c06d916/Image/31c1f907117aa1f2ba77d1d78d3aa8fb/sql_dev_export_autotrace.png" /></p>
<p>Save this and you&#39;ll get a file like this:</p>
OPERATION
OBJECT_NAME
CARDINALITY
LAST_CR_BUFFER_GETS
LAST_OUTPUT_ROWS
<img border="0" hheight="16" src="http://cdn.app.compendium.com/uploads/user/e7c690e8-6ff9-102a-ac6d-e4aebca50425/f4a5b21d-66fa-4885-92bf-c4e81c06d916/Image/c72e32d2e46615e5cc516094f3519cdc/minus.gif" wwidth="16" /><img border="0" hheight="16" src="http://cdn.app.compendium.com/uploads/user/e7c690e8-6ff9-102a-ac6d-e4aebca50425/f4a5b21d-66fa-4885-92bf-c4e81c06d916/Image/6ce12f68cd8a747d2d4452ce49d01636/gray.png" wwidth="16" />
SELECT STATEMENT
&nbsp;
&nbsp;
&nbsp;
&nbsp;
&nbsp;
<img border="0" hheight="16" src="http://cdn.app.compendium.com/uploads/user/e7c690e8-6ff9-102a-ac6d-e4aebca50425/f4a5b21d-66fa-4885-92bf-c4e81c06d916/Image/c72e32d2e46615e5cc516094f3519cdc/minus.gif" wwidth="16" /><img border="0" hheight="16" src="http://cdn.app.compendium.com/uploads/user/e7c690e8-6ff9-102a-ac6d-e4aebca50425/f4a5b21d-66fa-4885-92bf-c4e81c06d916/Image/ca0ba3e93f1ec51de3aca3ce0ada3b42/sortascending_ena.png" wwidth="16" />
SORT
&nbsp;
1
35
1
&nbsp;
&nbsp;
<img border="0" hheight="16" src="http://cdn.app.compendium.com/uploads/user/e7c690e8-6ff9-102a-ac6d-e4aebca50425/f4a5b21d-66fa-4885-92bf-c4e81c06d916/Image/c72e32d2e46615e5cc516094f3519cdc/minus.gif" wwidth="16" /><img border="0" hheight="16" src="http://cdn.app.compendium.com/uploads/user/e7c690e8-6ff9-102a-ac6d-e4aebca50425/f4a5b21d-66fa-4885-92bf-c4e81c06d916/Image/ad4b5bb28232452105afcdbb2de2d219/table.png" wwidth="16" />
TABLE ACCESS
TARGET
1
35
3
&nbsp;
&nbsp;
&nbsp;
<img border="0" hheight="16" src="http://cdn.app.compendium.com/uploads/user/e7c690e8-6ff9-102a-ac6d-e4aebca50425/f4a5b21d-66fa-4885-92bf-c4e81c06d916/Image/c72e32d2e46615e5cc516094f3519cdc/minus.gif" wwidth="16" /><img border="0" hheight="16" src="http://cdn.app.compendium.com/uploads/user/e7c690e8-6ff9-102a-ac6d-e4aebca50425/f4a5b21d-66fa-4885-92bf-c4e81c06d916/Image/da1af1fbeb84de8ef03496077b343fac/sigmafilter.png" wwidth="16" />
Filter Predicates
&nbsp;
&nbsp;
&nbsp;
&nbsp;
<img border="0" hheight="16" src="http://cdn.app.compendium.com/uploads/user/e7c690e8-6ff9-102a-ac6d-e4aebca50425/f4a5b21d-66fa-4885-92bf-c4e81c06d916/Image/c72e32d2e46615e5cc516094f3519cdc/minus.gif" wwidth="16" /><img border="0" hheight="16" src="http://cdn.app.compendium.com/uploads/user/e7c690e8-6ff9-102a-ac6d-e4aebca50425/f4a5b21d-66fa-4885-92bf-c4e81c06d916/Image/c6f4a70667d6445c66435e50fafdc9c5/empty.png" wwidth="16" />
T.SORT_CODE=:B1
&nbsp;
<img border="0" hheight="16" src="http://cdn.app.compendium.com/uploads/user/e7c690e8-6ff9-102a-ac6d-e4aebca50425/f4a5b21d-66fa-4885-92bf-c4e81c06d916/Image/c72e32d2e46615e5cc516094f3519cdc/minus.gif" wwidth="16" /><img border="0" hheight="16" src="http://cdn.app.compendium.com/uploads/user/e7c690e8-6ff9-102a-ac6d-e4aebca50425/f4a5b21d-66fa-4885-92bf-c4e81c06d916/Image/ad4b5bb28232452105afcdbb2de2d219/table.png" wwidth="16" />
TABLE ACCESS
SOURCE
10
8
10
<p style="margin-top:15px;">This all sounds great.</p>
<p>But there is one big drawback to autotrace: you have to wait for the query to finish!</p>
<p>If your SQL takes minutes or hours to complete waiting for this is a pain. It would be awesome if you could see query progress in real time.</p>
<p>Which brings us to the next method:</p>
<a name="sqlmonitor"></a>SQL Monitor
<p>The <a href="http://www.oracle.com/technetwork/database/manageability/sqlmonitor-084401.html" target="_blank">SQL Monitor</a> levels up autotrace. It provides similar operation-level stats. But with an added bonus. You can view the progress of execution plans in real time!</p>
<p>Got a troublesome full tablescan? You can watch while Oracle churns away at it. Instead of having to wait for the query to finish, you can see the plan immediately. This makes identifying bottlenecks easy.</p>
<p>Even better, unlike autotrace which you need to run manually, Oracle will capture the plan for you automatically.</p>
<p>So how do you view the plans?</p>
<p>They&#39;re available in SQL Developer by going to Tools -&gt; Monitor SQL&hellip;</p>
<p>Or you can view them in the Performance tab of Enterprise Manager:</p>
<p><img src="http://cdn.app.compendium.com/uploads/user/e7c690e8-6ff9-102a-ac6d-e4aebca50425/f4a5b21d-66fa-4885-92bf-c4e81c06d916/Image/e33e6a8c85b0e35483a31cc3e02f04bf/em_sql_monitor.png" /></p>
<p>Clicking on the SQL ID for your statement brings up the full details:</p>
<p><img src="http://cdn.app.compendium.com/uploads/user/e7c690e8-6ff9-102a-ac6d-e4aebca50425/f4a5b21d-66fa-4885-92bf-c4e81c06d916/Image/2a1e614e365976e7ff413aa96fc62240/em_sql_monitor_detail.png" /></p>
<p>As with SQL Developer, you can expand and collapse the sections of your plan. For more details on reading these reports see this post on <a href="https://blogs.oracle.com/datawarehousing/entry/monitoring_parallel_execution_using_real" target="_blank">monitoring parallel execution</a>.</p>
<p>For those of you who prefer text, you can output the plans using SQL. To do so, use the following query:</p>
select dbms_sqltune.report_sql_monitor(
sql_id =&gt; &#39;4htx5uyx0gxxx&#39;,
type =&gt; &#39;TEXT&#39;,
report_level =&gt; &#39;ALL&#39;
) as report
from dual;
<p>You&#39;ll need to replace SQL ID parameter with the ID of your query. You can find this with the following statement:</p>
select sql_id, sql_text from v$sql
where sql_text like &#39;%some text from your query%&#39;
and sql_text not like &#39;%not this%&#39;;
<p>If there are many matches, check the SQL text to see which one is your statement.</p>
<p>Note SQL_text is limited to 1,000 characters. So for really large statements, you may need to include the SQL_fulltext column too!</p>
<p>If it&rsquo;s so great, why isn&rsquo;t this my favourite approach?</p>
<p>Well, there are a couple of caveats:</p>
<ul>
<li>Not all queries appear by default. Oracle only captures those lasting longer than five seconds or those running in parallel.</li>
<li><span>You need to have Diagnostics and Tuning packs licenses to use it</span></li>
</ul>
<p>The first limitation is easy to get around. One way to do this is by adding the monitor hint:</p>
select /*+ monitor */&hellip; from &hellip;
<p>For licensing you&rsquo;ll need to speak to you local sales rep ;)</p>
<p>Assuming you are licensed, the SQL Monitor is a fantastic way to do early troubleshooting on those hour-long queries. Often you can spot which parts of the plan are doing the most damage within a few minutes. This enables early diagnosis. You can plan new approaches without waiting for the statement to finish!</p>
<p>Both of these methods are great. But they work on a single statement at a time. What if you want to analyze the performance of several pieces of SQL in a transaction?</p>
<p>Enter:</p>
<a name="tkprof"></a>TKPROF
<p>TKPROF is a command line utility that analyzes trace files and turns them into readable form. It gives you the execution stats for all the SQL in the file. Which begs the question:</p>
How do I generate a trace file?
<p>There are several methods. The easiest is to set SQL tracing on. The command to do this is:</p>
alter session set sql_trace = true;
<p>Oracle will capture all statements you execute after this in the trace file. To stop this, either disconnect or turn tracing off with:</p>
alter session set sql_trace = false;
<p>This method is easy but limited. For example, it only traces your session.</p>
<p>A more powerful method is to call <a href="http://docs.oracle.com/database/121/ARPLS/d_monitor.htm#ARPLS67178" target="_blank">DBMS_monitor.session_trace_enable</a>. This has five parameters:</p>
<ul style="margin-top: 0cm;" type="disc">
<li>Session_id</li>
<li>Serial_num</li>
<li>Waits</li>
<li>Binds</li>
<li>Plan_stat</li>
</ul>
<p>Pass in the relevant session_id and serial_num to trace another session. If you leave these null, Oracle will trace your current session. Setting waits and binds to true includes information about these in the file.</p>
<p>To stop tracing, call <a href="http://docs.oracle.com/database/121/ARPLS/d_monitor.htm#ARPLS67176" target="_blank">DBMS_monitor.session_trace_disable</a>. As with the enable procedure, pass the relevant session_id and serial_num. Or leave blank if tracing your current session.</p>
<p>So to generate a trace file for your current session, including waits and bind details, do the following:</p>
exec DBMS_monitor.session_trace_enable ( null, null, true, true );
***your code here***
exec DBMS_monitor.session_trace_disable;
<p>DBMS_monitor also includes procedures to trace all statements:</p>
<ul>
<li><a href="http://docs.oracle.com/database/121/ARPLS/d_monitor.htm#ARPLS67166" target="_blank">Across the database</a></li>
<li><a href="http://docs.oracle.com/database/121/ARPLS/d_monitor.htm#ARPLS67162" target="_blank">For given clients </a></li>
<li>Specific combinations of <a href="http://docs.oracle.com/database/121/ARPLS/d_monitor.htm#ARPLS67170" target="_blank">service name, module and action</a>.</li>
</ul>
<p>Note that tracing adds overhead. So avoid enabling it for the whole database. And remember to disable it when you&#39;ve finished!</p>
<p>Once you&#39;ve traced your code you need to get the file so you can analyze it.</p>
How to get the trace file
<p>&ldquo;But where can I find the trace file?&rdquo; I hear you ask.</p>
<p>It lives on the database server. This means you need access to it (or help from someone who does!). You also need to know where it is and what it&rsquo;s called!</p>
<p>Connor explains how to find them in this video:</p>
<p>Getting the files can be fiddly, especially if you don&rsquo;t have access to the server. You can get around this by <a href="https://oracle-base.com/articles/misc/sql-trace-10046-trcsess-and-tkprof#query_contents_of_trace_files" target="_blank">configuring a file reader</a> enabling you to query their contents (and thus save to your local machine).</p>
<p>With the trace file in hand you can parse it with TKPROF. The basic syntax for this is:</p>
<blockquote style="margin: 0px 0px 0px 40px; border: none; padding: 0px;">
tkprof &lt;trace_file_name&gt; &lt;output_file_name&gt;
</blockquote>
<p>For example:</p>
<blockquote style="margin: 0px 0px 0px 40px; border: none; padding: 0px;">
tkprof ORCL_ora_27883.trc trace.log
</blockquote>
<p>This parses the contents of ORCL_ora_27883.trc into trace.log. You&#39;re now ready to start analyzing the queries!</p>
<p>Quick note: TKPROF also includes an explain option. This will show you explain, not execution plans. Use this with caution.</p>
<p>Compared to autotrace and the SQL Monitor this is a lot of work.</p>
<p>So you may be wondering: is this extra hassle worth it?</p>
<p>Well TKPROF has some of advantages over these:</p>
<ul style="margin-top: 0cm;" type="disc">
<li>It includes all the SQL statements run between you starting and stopping tracing. This includes recursive SQL i.e. statements inside triggers, functions, etc.</li>
<li>It breaks down the execution time into parse, execution and fetch times</li>
</ul>
<p>The first benefit is great if you have a slow transaction that includes several statements. You can sort them in the output file from slowest to fastest. This helps you spot which is takes the longest. To do this, use the sort option:</p>
<blockquote style="margin: 0px 0px 0px 40px; border: none; padding: 0px;">
tkprof &lt;trace_file_name&gt; &lt;output_file_name&gt; sort=prsela,exeela,fchela
</blockquote>
<p>This is particularly useful for spotting fast queries that you execute many times. This could be because you&#39;re doing row-by-row processing. Or you have a SQL statement with many calls PL/SQL which itself calls SQL (something you should <a href="https://blogs.oracle.com/sql/the-problem-with-sql-calling-plsql-calling-sql">generally avoid</a>). Or trigger logic you weren&#39;t aware of. Though, as Jonathan Lewis notes in the comments, be aware that a SQL statement could have a different plan for each exection. TKPROF will only report one of these.&nbsp;</p>
<p>The parse, execution and fetch breakdown helps you spot issues unrelated to the plan itself. For example, in an overloaded system parse times can be higher. If a significant fraction of a statement&rsquo;s runtime is parsing then you should start looking for issues other than the plan itself.</p>
<p>So TKPROF enables you to see information not visible in the previous two tools. Sometimes the extra hassle is worth it!</p>
<a name="dbmsxplan"></a>DBMS_XPlan
<p><img src="http://cdn.app.compendium.com/uploads/user/e7c690e8-6ff9-102a-ac6d-e4aebca50425/f4a5b21d-66fa-4885-92bf-c4e81c06d916/Image/d7c289cfea1d848fbce97f256896ed40/dbms_xplan_output.png" /></p>
<p>The final method we&rsquo;ll look at uses the package <a href="http://docs.oracle.com/database/121/ARPLS/d_xplan.htm" target="_blank">DBMS_XPlan</a>. This includes several display functions. These are pipelined, so you can query these like normal tables using the table() operator.</p>
<p>Using these, you can get the performance stats for each step of the plan: actual rows, time taken, etc.</p>
<p>To see this information, you need to increase Oracle&rsquo;s stats level. You can do this by:</p>
<ul style="margin-top: 0cm;">
<li>Setting statistics_level to all in the session (or database!) using:</li>
</ul>
<blockquote style="margin: 0px 0px 0px 40px; border: none; padding: 0px;">
alter session set statistics_level = all;
</blockquote>
<ul style="margin-top: 0cm;" type="disc">
<li>Adding the /*+ gather_plan_statistics */ hint to your query</li>
</ul>
<p>Once you&rsquo;ve done one of these and run the statement to completion you can get your plan!</p>
<p>Call display_cursor to get the plan immediately after executing your SQL (taking care to ensure you&rsquo;ve <a href="https://connormcdonald.wordpress.com/2016/01/29/common-gather_plan_statistic-confusion/" target="_blank">set serveroutput off</a>):</p>
<blockquote style="margin: 0px 0px 0px 40px; border: none; padding: 0px;">
select * from table(dbms_xplan.display_cursor);
</blockquote>
<p>This fetches the plan for the last statement in your session. This is good but misses many of the juicy details. To include these, use the format parameter. This enables you to include or exclude parts of the plan. Some of my favourite options are:</p>
<ul style="margin-top: 0cm;" type="disc">
<li>ALLSTATS - Shorthand for IOSTATS MEMSTATS. This includes details such as how many rows and consistent gets each step used.</li>
<li>LAST - Only display the stats for the last execution. Otherwise this default to giving you the information for every execution.</li>
<li>PARTITION - Include the Pstart and Pstop columns. This tells you which partitions Oracle accessed. Only needed if you&#39;re querying partitioned tables!</li>
<li>PARALLEL - Again, only relevant for parallel statements!</li>
<li>NOTE - includes the note section. This provides extra information, such as whether Oracle used features like dynamic statistics, adaptive plans, etc.</li>
</ul>
<p>If you want to exclude parts of the plan, just prefix the format with the minus sign &#39;-&#39;.</p>
<p>You can also use this to get the plans for previous statements. You just need to find the SQL ID and pass it in:</p>
<blockquote style="margin: 0px 0px 0px 40px; border: none; padding: 0px;">
select * from table(dbms_xplan.display_cursor(&#39;sql_id&#39;, null, &#39;ALLSTATS LAST&#39;));
</blockquote>
<p>The plan must still be in the cursor cache to use this. Head to <a href="https://livesql.oracle.com/apex/livesql/file/content_FAQU2DYYI7N1RYDB5CR2Y98DD.html">LiveSQL for an example</a> of how to do this. If you&#39;re using the <a href="https://docs.oracle.com/cd/E11882_01/server.112/e41573/autostat.htm#PFGRF02601" target="_blank">Automatic Workload Repository</a>, you can get older plans using display_AWR:</p>
<blockquote style="margin: 0px 0px 0px 40px; border: none; padding: 0px;">
select * from table(dbms_xplan.display_awr(&#39;sql_id&#39;, null, null, &#39;ALLSTATS LAST&#39;));
</blockquote>
<p>This is all well and good. But the output is static text. You can&rsquo;t expand and collapse the plan like you can in SQL Developer. This can make large plans tricky to read.</p>
<p>So why would you choose this over SQL Developer?</p>
<p>An obvious answer is because this is part of a scripted process. You need pure SQL for these!</p>
<p>Another is if you want to post the plan somewhere (*cough* <a href="https://asktom.oracle.com" target="_blank">Ask Tom</a> *cough*) to get help tuning help from others. The simple text format makes it great for sharing.</p>
<strong>Summary</strong>
<p>We&rsquo;ve looked at four methods of creating execution plans.</p>
<p>These all come with their own pros and cons. In the end which you use is largely a matter of personal taste.</p>
<p>For short running queries, the ability to expand, collapse and compare execution plans makes autotrace in SQL Developer the winner for me. For longer queries &ndash; where licensed &ndash; I&rsquo;d go for the SQL monitor.</p>
<p>But according to my recent Twitter poll, DBMS_XPlan is the most popular in my followers:</p>
<p><img src="http://cdn.app.compendium.com/uploads/user/e7c690e8-6ff9-102a-ac6d-e4aebca50425/f4a5b21d-66fa-4885-92bf-c4e81c06d916/Image/e1b5a5217a8b6e696a51be58f0388f80/twitter_xplan_poll_results.png" /></p>
<p>There are many other ways to generate Oracle execution plans. Notably <a href="https://carlos-sierra.net/2012/04/03/what-is-sqltxplain/" target="_blank">SQLTXPLAIN</a>. This is a powerful tool that also includes lots of supporting details, such as the current table statistics. This is most useful if need help from other people (e.g. support!). This ensures they have all the information they need to diagnose the problem.</p>
<p>The most important thing is to ensure that you are getting <i>execution</i> plans, <b>not</b> <i>explain </i>plans. If the output is missing details such as actual rows and time taken then there&#39;s a good chance you&#39;ve got the wrong type. At the very least, you&#39;re missing vital information to help you tune your SQL.</p>
<p>How about you? Which do you like best? Are there any other tools you use I haven&rsquo;t mentioned? I&rsquo;ve heard there are tools from other vendors for creating execution plan tools too ;) Do any of these have significant benefits?</p>
<p>Let us know in the comments!</p>
<p><strong>UPDATED</strong>: 19 July 2017: added Jonathan Lewis&#39; comment about multiple plans, link to LiveSQL demo for DBMS_XPlan and fixed some formatting</p>
PartitioningTue, 22 Mar 2016 16:10:00 +0000https://blogs.oracle.com/sql/how-to-create-an-execution-planChris Saxon