I'm using latest versions of community and document server. I install them on different machines and both running as expected.
I switch them to use https with self signed certificates (openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout /etc/ssl/priv .... ) and I can call/access every machine separately (work with community normally, and with document server I get welcome page - all normal).
Now when I try to connect community and document server I receive following error:

So, if I understand it correctly document server does not trust self signed cert from document server. And that's normal.
But, how can I tell community server that it's OK to use that certificate? Or, I expect that I need to copy something (key, pem) from document server to community server, but where and do I need to change some config files?

Hello!
1. Please specify what instruction you followed to install servers.
2. All self-signed certificates do not have CA certs that's why they are not safe, but Let'sEncrypt has CA certs.
3. So, you can use Certbot to enable HTTPS on your website or you can install Enterprise Edition on the domain name you need and generate SSL certs (Let'sEncrypt) and use them on your servers.

But, how can I tell community server that it's OK to use that certificate?

You can't because it depends on the OS, not on the Document or Community server, OS doesn't trust SSL certs.

You can't because it depends on the OS, not on the Document or Community server, OS doesn't trust SSL certs.

I do not agree about that
It's ubuntu and I communicate on OS level between servers without any problem, and I also use CURL without problem (of course after I convince him to trust that cert), so, my wild guess is that your library require approval for that cert. For example, in my JAVA applications I need to use TrustSelfSignedStrategy() to override that problem.