[ https://issues.apache.org/jira/browse/PHOENIX-4625?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vikas Vishwakarma updated PHOENIX-4625:
---------------------------------------
Description:
We have two different code path
# In ConnectionQueryServicesImpl RenewLeaseTasks is scheduled based on the following checks if
renew lease feature is supported and if the renew lease config is enabled supportsFeature(ConnectionQueryServices.Feature.RENEW_LEASE)
&& renewLeaseEnabled
# In PhoenixConnection for every scan iterator is added to a Queue for lease renewal based
on just the check if the renew lease feature is supported services.supportsFeature(Feature.RENEW_LEASE)
In PhoenixConnection we however miss the check whether renew lease config is enabled (phoenix.scanner.lease.renew.enabled)
Now consider a situation where Renew lease feature is supported but phoenix.scanner.lease.renew.enabled
is set to false in hbase-site.xml . In this case PhoenixConnection will keep adding the iterators
for every scan into the scannerQueue for renewal based on the feature supported check but
the renewal task is not running because phoenix.scanner.lease.renew.enabled is set to false,
so the scannerQueue will keep growing as long as the PhoenixConnection is alive and multiple
scans requests are coming on this connection.
We have a use case that uses a single PhoenixConnection that is perpetual and does billions
of scans on this connection. In this case scannerQueue is growing to several GB's and ultimately
leading to Consecutive Full GC's/OOM
Add iterators for Lease renewal in PhoenixConnection
=============================================
{code}
public void addIteratorForLeaseRenewal(@Nonnull TableResultIterator itr) {
if (services.supportsFeature(Feature.RENEW_LEASE))
{ checkNotNull(itr); scannerQueue.add(new WeakReference<TableResultIterator>(itr));
}
}
{code}
Starting the RenewLeaseTask
=============================
checks if Feature.RENEW_LEASE is supported and if phoenix.scanner.lease.renew.enabled is
true and starts the RenewLeaseTask
{code}
ConnectionQueryServicesImpl {
....
this.renewLeaseEnabled = config.getBoolean(RENEW_LEASE_ENABLED, DEFAULT_RENEW_LEASE_ENABLED);
.....
@Override
public boolean isRenewingLeasesEnabled()
{ return supportsFeature(ConnectionQueryServices.Feature.RENEW_LEASE) && renewLeaseEnabled;
}
private void scheduleRenewLeaseTasks() {
if (isRenewingLeasesEnabled()) {
renewLeaseExecutor =
Executors.newScheduledThreadPool(renewLeasePoolSize, renewLeaseThreadFactory);
for (LinkedBlockingQueue<WeakReference<PhoenixConnection>> q : connectionQueues)
{ renewLeaseExecutor.scheduleAtFixedRate(new RenewLeaseTask(q), 0, renewLeaseTaskFrequency,
TimeUnit.MILLISECONDS); }
}
}
.......
}
{code}
To solve this We must add both checks in PhoenixConnection if the feature is supported and
if the config is enabled before adding the iterators to scannerQueue
ConnectionQueryServices.Feature.RENEW_LEASE is true && phoenix.scanner.lease.renew.enabled
is true
instead of just checking if the feature ConnectionQueryServices.Feature.RENEW_LEASE is supported
was:
We have two different code path
# In ConnectionQueryServicesImpl RenewLeaseTasks is scheduled based on the following checks if
renew lease feature is supported and if the renew lease config is enabled supportsFeature(ConnectionQueryServices.Feature.RENEW_LEASE)
&& renewLeaseEnabled
# In PhoenixConnection for every scan iterator is added to a Queue for lease renewal based
on just the check if the renew lease feature is supported services.supportsFeature(Feature.RENEW_LEASE)
In PhoenixConnection we however miss the check whether renew lease config is enabled (phoenix.scanner.lease.renew.enabled)
Now consider a situation where Renew lease feature is supported but phoenix.scanner.lease.renew.enabled
is set to false in hbase-site.xml . In this case PhoenixConnection will keep adding the iterators
for every scan into the scannerQueue for renewal based on the feature supported check but
the renewal task is not running because phoenix.scanner.lease.renew.enabled is set to false,
so the scannerQueue will keep growing as long as the PhoenixConnection is alive and multiple
scans requests are coming on this connection.
We have a use case that uses a single PhoenixConnection that is perpetual and does billions
of scans on this connection. In this case scannerQueue is growing to several GB's and ultimately
leading to Consecutive Full GC's/OOM
Add iterators for Lease renewal in PhoenixConnection
=============================================
public void addIteratorForLeaseRenewal(@Nonnull TableResultIterator itr) {
if (services.supportsFeature(Feature.RENEW_LEASE)) {
checkNotNull(itr);
scannerQueue.add(new WeakReference<TableResultIterator>(itr));
}
}
Starting the RenewLeaseTask
=============================
checks if Feature.RENEW_LEASE is supported and if phoenix.scanner.lease.renew.enabled is
true and starts the RenewLeaseTask
ConnectionQueryServicesImpl {
....
this.renewLeaseEnabled = config.getBoolean(RENEW_LEASE_ENABLED, DEFAULT_RENEW_LEASE_ENABLED);
.....
@Override
public boolean isRenewingLeasesEnabled() {
return supportsFeature(ConnectionQueryServices.Feature.RENEW_LEASE) && renewLeaseEnabled;
}
private void scheduleRenewLeaseTasks() {
if (isRenewingLeasesEnabled()) {
renewLeaseExecutor =
Executors.newScheduledThreadPool(renewLeasePoolSize, renewLeaseThreadFactory);
for (LinkedBlockingQueue<WeakReference<PhoenixConnection>> q : connectionQueues)
{
renewLeaseExecutor.scheduleAtFixedRate(new RenewLeaseTask(q), 0,
renewLeaseTaskFrequency, TimeUnit.MILLISECONDS);
}
}
}
.......
}
To solve this We must add both checks in PhoenixConnection if the feature is supported and
if the config is enabled before adding the iterators to scannerQueue
ConnectionQueryServices.Feature.RENEW_LEASE is true && phoenix.scanner.lease.renew.enabled
is true
instead of just checking if the feature ConnectionQueryServices.Feature.RENEW_LEASE is supported
> memory leak in PhoenixConnection if scanner renew lease thread is not enabled
> -----------------------------------------------------------------------------
>
> Key: PHOENIX-4625
> URL: https://issues.apache.org/jira/browse/PHOENIX-4625
> Project: Phoenix
> Issue Type: Bug
> Affects Versions: 4.13.0
> Reporter: Vikas Vishwakarma
> Priority: Major
> Attachments: QS.png
>
>
> We have two different code path
> # In ConnectionQueryServicesImpl RenewLeaseTasks is scheduled based on the following
checks if renew lease feature is supported and if the renew lease config is enabled supportsFeature(ConnectionQueryServices.Feature.RENEW_LEASE)
&& renewLeaseEnabled
> # In PhoenixConnection for every scan iterator is added to a Queue for lease renewal
based on just the check if the renew lease feature is supported services.supportsFeature(Feature.RENEW_LEASE)
> In PhoenixConnection we however miss the check whether renew lease config is enabled
(phoenix.scanner.lease.renew.enabled)
>
> Now consider a situation where Renew lease feature is supported but phoenix.scanner.lease.renew.enabled
is set to false in hbase-site.xml . In this case PhoenixConnection will keep adding the iterators
for every scan into the scannerQueue for renewal based on the feature supported check but
the renewal task is not running because phoenix.scanner.lease.renew.enabled is set to false,
so the scannerQueue will keep growing as long as the PhoenixConnection is alive and multiple
scans requests are coming on this connection.
>
> We have a use case that uses a single PhoenixConnection that is perpetual and does billions
of scans on this connection. In this case scannerQueue is growing to several GB's and ultimately
leading to Consecutive Full GC's/OOM
>
> Add iterators for Lease renewal in PhoenixConnection
> =============================================
> {code}
> public void addIteratorForLeaseRenewal(@Nonnull TableResultIterator itr) {
> if (services.supportsFeature(Feature.RENEW_LEASE))
> { checkNotNull(itr); scannerQueue.add(new WeakReference<TableResultIterator>(itr));
}
> }
> {code}
>
> Starting the RenewLeaseTask
> =============================
> checks if Feature.RENEW_LEASE is supported and if phoenix.scanner.lease.renew.enabled
is true and starts the RenewLeaseTask
> {code}
> ConnectionQueryServicesImpl {
> ....
> this.renewLeaseEnabled = config.getBoolean(RENEW_LEASE_ENABLED, DEFAULT_RENEW_LEASE_ENABLED);
> .....
> @Override
> public boolean isRenewingLeasesEnabled()
> { return supportsFeature(ConnectionQueryServices.Feature.RENEW_LEASE) && renewLeaseEnabled;
}
> private void scheduleRenewLeaseTasks() {
> if (isRenewingLeasesEnabled()) {
> renewLeaseExecutor =
> Executors.newScheduledThreadPool(renewLeasePoolSize, renewLeaseThreadFactory);
> for (LinkedBlockingQueue<WeakReference<PhoenixConnection>> q : connectionQueues)
> { renewLeaseExecutor.scheduleAtFixedRate(new RenewLeaseTask(q), 0, renewLeaseTaskFrequency,
TimeUnit.MILLISECONDS); }
> }
> }
> .......
> }
> {code}
> To solve this We must add both checks in PhoenixConnection if the feature is supported
and if the config is enabled before adding the iterators to scannerQueue
> ConnectionQueryServices.Feature.RENEW_LEASE is true && phoenix.scanner.lease.renew.enabled
is true
> instead of just checking if the feature ConnectionQueryServices.Feature.RENEW_LEASE
is supported
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)