3204 Guilhem Bichot 2010-09-20
a Python script to validate that the traces found in a result file are JSON-compliant
(currently they are not because " and newlines in queries are not escaped)
added:
WL4800_validate_json.py
modified:
WL4800_Doxyfile
WL4800_extract_plans.py
WL4800_select_to_explain_test.py
sql/opt_trace.cc
sql/opt_trace.h
unittest/gunit/opt_trace-t.cc
3203 Guilhem Bichot 2010-09-18
1) implementing the main remaining pieces of WL#5257:
1.1) @@optimizer_trace_offset, @@optimizer_trace_limit.
@@optimizer_trace_offset is a signed variable, so I had to add support for
this in MySQL (so far all integer system variables were unsigned, it was impossible
to set a system variable to a negative value, it would be rounded to 0).
1.2) using stored routines implies that one statement has sub-statements; so far
all of the statement's and sub-statements' traces were concatenated in one
single row of INFORMATION_SCHEMA.OPTIMIZER_TRACE. This prevented
@@optimizer_trace_offset/limit to function. Fixed by making each statement and
sub-statement generate a new row in INFORMATION_SCHEMA.OPTIMIZER_TRACE.
1.3) @@optimizer_trace_max_mem_size. If allocating memory for extending a trace
fails because of optimizer_trace_max_mem_size, report missing bytes in
INFORMATION_SCHEMA.OPTIMIZER_TRACE.MISSING_BYTES_BEYOND_MAX_MEM_SIZE.
If OS' malloc()/realloc() fail, report failure in
INFORMATION_SCHEMA.OPTIMIZER_TRACE.OS_MALLOC_ERROR.
1.4) mtr tests of the new variables
2) test of optimizer_trace vs {query with a stored function, or a stored procedure,
or a trigger}.
3) test of optimizer_trace vs {a view, or an INFORMATION_SCHEMA table}.
4) test that first and second EXECUTE of a prepared statement give the same trace.
@ include/my_getopt.h
utility function
@ include/mysql/plugin.h
comments
@ include/mysql/plugin_audit.h.pp
layout of space changed in .h, so changes in .h.pp
@ include/mysql/plugin_ftparser.h.pp
layout of space changed in .h, so changes in .h.pp
@ mysql-test/include/have_optimizer_trace.inc
optimizer_trace tests must not fail if binary doesn't support it
@ mysql-test/include/optimizer_trace.inc
new tests
@ mysql-test/lib/My/SafeProcess/safe_process.cc
temporary fix (should not be pushed to main trees);
without it, some valgrind "at shutdown" errors (memory leaks)
are missed because kill-9 prevents valgrind from finding/printing them
@ mysql-test/r/bug42620.result
update result due to new columns in I_S.OPTIMIZER_TRACE
@ mysql-test/r/mysqld--help-notwin.result
update result due to new variables
@ mysql-test/r/optimizer_trace_ps_prot.result
result is identical to no_prot, with the exception of 3 subquery
transformations as explained at the start of optimizer_trace.inc
@ mysql-test/suite/sys_vars/t/optimizer_trace_limit_basic.test
basic test of reading/writing the optimizer_trace_limit variable
@ mysql-test/suite/sys_vars/t/optimizer_trace_max_mem_size_basic.test
basic test of reading/writing the optimizer_trace_max_mem_size variable
@ mysql-test/suite/sys_vars/t/optimizer_trace_offset_basic.test
basic test of reading/writing the optimizer_trace_offset variable
@ mysql-test/suite/sys_vars/t/optimizer_trace_offset_max-master.opt
this should cap "SET optimizer-trace-offset" at values between -50 and 50
@ mysql-test/suite/sys_vars/t/optimizer_trace_offset_max.test
test that --maximum-optimizer-trace-offset caps values as intended
@ mysys/my_getopt.c
reducing code size by using a utility function, max_of_int_range()
@ sql/CMakeLists.txt
to-be-unit-tested object files should be linked into sqlgunitlib
@ sql/item_func.cc
support for SHOW_SIGNED_LONG. The "unsigned_flag" is what makes
-1 be displaid as -1 and not a huge positive number in "SELECT @@var"
@ sql/item_subselect.cc
unused variable
@ sql/mysqld.cc
unimportant change (get_tail() is now safe to call even
if trace is not started)
@ sql/opt_range.cc
more disabling of range optimizer's tracing. Fixing compiler warnings.
Removing unneeded todo comments.
@ sql/opt_trace.cc
see comments in opt_trace.h
@ sql/set_var.cc
see explanation in commit comment of sql/sys_vars.h
@ sql/sql_class.h
new variables optimizer_trace_offset, optimizer_trace_limit,
optimizer_trace_max_mem_size.
@ sql/sql_plugin.h
support for SHOWing _signed_ long integer variables (in SHOW VARIABLES,
in SELECT @@thevar): request this by using SHOW_SIGNED_LONG.
@ sql/sql_select.cc
removing unneeded todo comments
@ sql/sql_show.cc
the first change is not needed today (there are no SHOW_SIGNED_LONG variables
in SHOW STATUS) but good for the future.
The second change is what makes -1 be displaid as -1 and not a huge positive number
in "SHOW VARIABLES"
@ sql/sys_vars.cc
new variables optimizer_trace_offset, optimizer_trace_limit,
optimizer_trace_max_mem_size.
@ sql/sys_vars.h
Support for _signed_ integer system variables. This makes do_check()
more complicated, as code cannot be the same as for unsigned.
The third parameter of throw_bounds_warning(), named 'fixed', was
here 'var->save_result.ulonglong_value != uv' which tests whether
there was rounding due to the allowed range defined with VALID_RANGE()
(see sys_vars.cc); that rounding is done by getopt_ull_limit_value().
But there is another 'fixing', which is that a negative value is rounded
to 0; this is done earlier with
if (var->value->unsigned_flag)
...
else
uv= (ulonglong) (v < 0 ? 0 : v);
The passed value of 'fixed' didn't reflect this rounding, so
throw_bounds_warning() had to explicitely detect it with the
part after || :
if (fixed || (!is_unsigned && v < 0))
By changing the parameter to be
var->save_result.ulonglong_value != (ulonglong)v
which detects whether any change happened between input value
(v) and final value (var->save_result.ulonglong_value) we can
just test 'fixed' in throw_bounds_warning().
@ unittest/gunit/CMakeLists.txt
build opt_trace unit test
@ unittest/gunit/Makefile.am
build opt_trace unit test
@ unittest/gunit/opt_trace-t.cc
sql/ unit tests live should be put in unittest/gunit nowadays
added:
mysql-test/include/have_optimizer_trace.inc
mysql-test/suite/sys_vars/r/optimizer_trace_limit_basic.result
mysql-test/suite/sys_vars/r/optimizer_trace_max_mem_size_basic.result
mysql-test/suite/sys_vars/r/optimizer_trace_offset_basic.result
mysql-test/suite/sys_vars/r/optimizer_trace_offset_max.result
mysql-test/suite/sys_vars/t/optimizer_trace_limit_basic.test
mysql-test/suite/sys_vars/t/optimizer_trace_max_mem_size_basic.test
mysql-test/suite/sys_vars/t/optimizer_trace_offset_basic.test
mysql-test/suite/sys_vars/t/optimizer_trace_offset_max-master.opt
mysql-test/suite/sys_vars/t/optimizer_trace_offset_max.test
renamed:
sql/opt_trace-t.cc => unittest/gunit/opt_trace-t.cc
modified:
WL4800_TODO.txt
include/my_getopt.h
include/mysql/plugin.h
include/mysql/plugin_audit.h.pp
include/mysql/plugin_ftparser.h.pp
mysql-test/include/optimizer_trace.inc
mysql-test/lib/My/SafeProcess/safe_process.cc
mysql-test/r/bug42620.result
mysql-test/r/mysqld--help-notwin.result
mysql-test/r/optimizer_trace_no_prot.result
mysql-test/r/optimizer_trace_ps_prot.result
mysql-test/suite/sys_vars/t/optimizer_trace_basic.test
mysql-test/t/bug42620.test
mysql-test/t/optimizer_trace_bugs.test
mysys/my_getopt.c
sql/CMakeLists.txt
sql/item_func.cc
sql/item_subselect.cc
sql/mysqld.cc
sql/opt_range.cc
sql/opt_trace.cc
sql/opt_trace.h
sql/set_var.cc
sql/sql_class.h
sql/sql_plugin.h
sql/sql_select.cc
sql/sql_show.cc
sql/sys_vars.cc
sql/sys_vars.h
unittest/gunit/CMakeLists.txt
unittest/gunit/Makefile.am
unittest/gunit/opt_trace-t.cc
=== modified file 'WL4800_Doxyfile'
--- a/WL4800_Doxyfile 2010-09-05 16:08:07 +0000
+++ b/WL4800_Doxyfile 2010-09-20 08:12:18 +0000
@@ -1,5 +1,20 @@
# Doxyfile 1.6.1
+# Copyright (c) 2010, Oracle and/or its affiliates. All rights reserved.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; version 2 of the License.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software
+# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA */
+
# This file describes the settings to be used by the documentation system
# doxygen (www.doxygen.org) for a project
#
=== modified file 'WL4800_extract_plans.py'
--- a/WL4800_extract_plans.py 2010-05-24 18:38:10 +0000
+++ b/WL4800_extract_plans.py 2010-09-20 08:12:18 +0000
@@ -1,5 +1,20 @@
#! /usr/bin/python
+# Copyright (c) 2010, Oracle and/or its affiliates. All rights reserved.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; version 2 of the License.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software
+# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA */
+
import sys
usage = """
=== modified file 'WL4800_select_to_explain_test.py'
--- a/WL4800_select_to_explain_test.py 2010-09-04 17:40:51 +0000
+++ b/WL4800_select_to_explain_test.py 2010-09-20 08:12:18 +0000
@@ -1,5 +1,20 @@
#! /usr/bin/python
+# Copyright (c) 2010, Oracle and/or its affiliates. All rights reserved.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; version 2 of the License.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software
+# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA */
+
import sys, re
usage = """
=== added file 'WL4800_validate_json.py'
--- a/WL4800_validate_json.py 1970-01-01 00:00:00 +0000
+++ b/WL4800_validate_json.py 2010-09-20 08:12:18 +0000
@@ -0,0 +1,65 @@
+#! /usr/bin/python
+
+# Copyright (c) 2010, Oracle and/or its affiliates. All rights reserved.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; version 2 of the License.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software
+# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA */
+
+import sys
+
+usage = """
+Usage:
+ %s <a_file>
+
+It will verify that all optimizer traces of a_file (usually a_file
+is a .result or .reject file which contains
+SELECT * FROM OPTIMIZER_TRACE; ) are JSON-compliant
+""" % sys.argv[0]
+
+if len(sys.argv) != 2:
+ print usage
+ sys.exit(1)
+
+import json, re
+
+def check(trace, first_trace_line):
+ s = "".join(trace)
+ try:
+ json.loads(s)
+ except:
+ print "error at line", first_trace_line
+ print sys.exc_info()
+ print s
+ else:
+ print "ok at line", first_trace_line
+ print
+
+all = open(sys.argv[1]).readlines()
+
+first_trace_line = trace_line = 0
+trace = None
+for l in all:
+ trace_line += 1
+ if l == "{\n":
+ assert first_trace_line == 0
+ trace = []
+ first_trace_line = trace_line
+ if l.startswith("}\t") or l.startswith("}\n"): # end of current trace
+ assert first_trace_line != 0
+ trace.append("}") # eliminate any following columns of table (OS_MALLOC_ERROR etc)
+ check(trace, first_trace_line)
+ first_trace_line = 0
+ if first_trace_line != 0:
+ # eliminate /* */ (not valid JSON)
+ no_comment = re.sub("/\*.*\*/", "", l)
+ trace.append(no_comment)
=== modified file 'sql/opt_trace.cc'
--- a/sql/opt_trace.cc 2010-09-18 16:25:43 +0000
+++ b/sql/opt_trace.cc 2010-09-20 08:12:18 +0000
@@ -1,4 +1,4 @@
-/* Copyright (c) 2000, 2010, Oracle and/or its affiliates. All rights reserved.
+/* Copyright (c) 2009, 2010, Oracle and/or its affiliates. All rights reserved.
This program is free software; you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
=== modified file 'sql/opt_trace.h'
--- a/sql/opt_trace.h 2010-09-18 16:25:43 +0000
+++ b/sql/opt_trace.h 2010-09-20 08:12:18 +0000
@@ -1,4 +1,4 @@
-/* Copyright (c) 2000, 2010, Oracle and/or its affiliates. All rights reserved.
+/* Copyright (c) 2009, 2010, Oracle and/or its affiliates. All rights reserved.
This program is free software; you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
=== modified file 'unittest/gunit/opt_trace-t.cc'
--- a/unittest/gunit/opt_trace-t.cc 2010-09-18 16:25:43 +0000
+++ b/unittest/gunit/opt_trace-t.cc 2010-09-20 08:12:18 +0000
@@ -1,4 +1,4 @@
-/* Copyright (c) 2000, 2010, Oracle and/or its affiliates. All rights reserved.
+/* Copyright (c) 2010, Oracle and/or its affiliates. All rights reserved.
This program is free software; you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
Attachment: [text/bzr-bundle] bzr/guilhem@mysql.com-20100920081218-3r0d16cbtwuup7hp.bundle

Content reproduced on this site is the property of the respective copyright holders. It is not reviewed in advance by Oracle and does not necessarily represent the opinion of Oracle or any other party.