Adding this doesn’t work unfortunately. After (retry x retry_wait) seconds pass by, the test fails. I verified that also with higher retry values. Adding a puts inside shows that the code is executed retry times, but it seems more like should be_listening is not re-evaluated. Is this possible?

You could test the code directly with inspec. What I did was simply putting the code block I posted to some file (/tmp/test_port_spec.rb) and running chef exec inspec exec /tmp/test_port_spec.rb.
Maybe even test the different scenarios with different ports to test if the code itself works:

Run the test locally with against your local machine

Run the test in the kitchen VM. You might need to install inspec into the chef installation (/opt/chef/embedded/bin/gem install inspec) and then you should find the inspec executable in /opt/chef/embedded/bin

Run the test from your local machine in the VM with inspec exec test.rb -t ssh://user@hostname -i /path/to/key (key should be somewhere in the .kitchen directory if you are using Vagrant).

In case someone else is looking for this; and this was the only item that came up for me in a search; it is possible as follows:

gem install rspec-retry

then add a test as below (retrying every second for 60 seconds):

require 'rspec/retry'
describe 'Port 8080' do
it 'should be listening', retry: 60, retry_wait: 1 do
expect(port(8080).listening?).to eq true
end
end

debug by adding the following to your test

RSpec.configure do |config|
# show retry status in spec process
config.verbose_retry = true
# show exception that triggers a retry if verbose_retry is set to true
config.display_try_failure_messages = true
end