flink-user mailing list archives

Re: Docker-Flink Project: TaskManagers can't talk to JobManager if they are on different nodes

Date

Mon, 06 Nov 2017 13:37:38 GMT

Hi Till,
Thanks a lot for your answer.
Is the taskmanager.data.port unique per TaskManager? The documentation says it is assigned
at runtime by the OS. My thinking here is that you would need to know what that is at service
creation time, which would go against the whole idea of how services are scaled in Docker
Swarm.
When you create a Swarm service using 'docker stack deploy' or 'docker service create', the
configuration that is used at that point is the same that will be used by all instances of
the service. If you then scale TaskManager to 8 or 10 containers, each of them gets the same
service configuration(the one used to create the service).
I have in fact tried to map specific ports in the TaskManager service configuration, but then
I got "port already in use" when I tried to scale up the service.
I wonder if there is a way around it.
Maybe the people who developed the create-docker-swarm-service.sh script in the docker-flink
project would be able to shed some light?
________________________________
From: Till Rohrmann <trohrmann@apache.org>
Sent: 06 November 2017 12:40:33
To: Piotr Nowojski
Cc: Vergilio, Thalita; user@flink.apache.org; Patrick Lucas
Subject: Re: Docker-Flink Project: TaskManagers can't talk to JobManager if they are on different
nodes
Hi Thalita,
in order to make Flink work, I think you have to expose the JobManager RPC port, the Blob
server port and make sure that the TaskManager can talk to each other by exposing the `taskmanager.data.port`.
The query server port is only necessary if you want to use queryable state.
I've pulled in Patrick who has more experience with running Flink on top of Docker. He'll
definitely be able to provide more detailed recommendations.
Cheers,
Till
On Mon, Nov 6, 2017 at 9:22 AM, Piotr Nowojski <piotr@data-artisans.com<mailto:piotr@data-artisans.com>>
wrote:
Till, is there somewhere a list of ports that need to exposed that’s more up to date compared
to docker-flunk README?
Piotrek
On 3 Nov 2017, at 10:23, Vergilio, Thalita <t.vergilio4822@student.leedsbeckett.ac.uk<mailto:t.vergilio4822@student.leedsbeckett.ac.uk>>
wrote:
Just an update: by changing the JOB_MANAGER_RPC_ADDRESS to the public IP of the JobManager
and exposing port 6123 as {{PUBLIC_IP}}:6123:6123, I manged to get the TaskManagers from different
nodes and even different subnets to talk to the JobManager.
This is how I created the services:
docker network create -d overlay overlay
docker service create --name jobmanager --env JOB_MANAGER_RPC_ADDRESS={{PUBLIC_IP}} -p 8081:8081
-p{{PUBLIC_IP}}:6123:6123 -p 48081:48081 -p 6124:6124 -p 6125:6125 --network overlay --constraint
'node.hostname == ubuntu-swarm-manager' flink jobmanager
docker service create --name taskmanager --env JOB_MANAGER_RPC_ADDRESS={{PUBLIC_IP}} -p 6121:6121
-p 6122:6122 --network overlay --constraint 'node.hostname != ubuntu-swarm-manager' flink
taskmanager
However, I am still encountering errors further down the line. When I submit a job using the
Web UI, it fails because the JobManager can't talk to the TaskManager on port 35033. I presume
this is the taskmanager.data.port, which needs to be set to a range and this range exposed
when I create the service?
Are there any other ports that I need to open at service creation time?
Connecting the channel failed: Connecting to remote task manager + '/{{IP_ADDRESS_OF_MANAGER}}:35033'
has failed. This might indicate that the remote task manager has been lost.
at org.apache.flink.runtime.io<http://org.apache.flink.runtime.io>.network.netty.PartitionRequestClientFactory$ConnectingChannel.waitForChannel(PartitionRequestClientFactory.java:196)
at org.apache.flink.runtime.io<http://org.apache.flink.runtime.io>.network.netty.PartitionRequestClientFactory$ConnectingChannel.access$000(PartitionRequestClientFactory.java:131)
at org.apache.flink.runtime.io<http://org.apache.flink.runtime.io>.network.netty.PartitionRequestClientFactory.createPartitionRequestClient(PartitionRequestClientFactory.java:83)
at org.apache.flink.runtime.io<http://org.apache.flink.runtime.io>.network.netty.NettyConnectionManager.createPartitionRequestClient(NettyConnectionManager.java:59)
at org.apache.flink.runtime.io<http://org.apache.flink.runtime.io>.network.partition.consumer.RemoteInputChannel.requestSubpartition(RemoteInputChannel.java:112)
at org.apache.flink.runtime.io<http://org.apache.flink.runtime.io>.network.partition.consumer.SingleInputGate.requestPartitions(SingleInputGate.java:433)
at org.apache.flink.runtime.io<http://org.apache.flink.runtime.io>.network.partition.consumer.SingleInputGate.getNextBufferOrEvent(SingleInputGate.java:455)
at org.apache.flink.streaming.runtime.io.BarrierTracker.getNextNonBlocked(BarrierTracker.java:91)
at org.apache.flink.streaming.runtime.io<http://runtime.io>.StreamInputProcessor.processInput(StreamInputProcessor.java:213)
at org.apache.flink.streaming.runtime.tasks.OneInputStreamTask.run(OneInputStreamTask.java:69)
at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:263)
at org.apache.flink.runtime.taskmanager.Task.run(Task.java:702)
at java.lang.Thread.run(Thread.java:748)
________________________________
From: Piotr Nowojski <piotr@data-artisans.com<mailto:piotr@data-artisans.com>>
Sent: 02 November 2017 14:26:32
To: Vergilio, Thalita
Cc: user@flink.apache.org<mailto:user@flink.apache.org>
Subject: Re: Docker-Flink Project: TaskManagers can't talk to JobManager if they are on different
nodes
Did you try to expose required ports that are listed in the README when starting the containers?
https://github.com/apache/flink/tree/master/flink-contrib/docker-flink
Ports:
• The Web Client is on port 48081
• JobManager RPC port 6123 (default, not exposed to host)
• TaskManagers RPC port 6122 (default, not exposed to host)
• TaskManagers Data port 6121 (default, not exposed to host)
Piotrek
On 2 Nov 2017, at 14:44, javalass <t.vergilio4822@student.leedsbeckett.ac.uk<mailto:t.vergilio4822@student.leedsbeckett.ac.uk>>
wrote:
I am using the Docker-Flink project in:
https://github.com/apache/flink/tree/master/flink-contrib/docker-flink
I am creating the services with the following commands:
docker network create -d overlay overlay
docker service create --name jobmanager --env
JOB_MANAGER_RPC_ADDRESS=jobmanager -p 8081:8081 --network overlay
--constraint 'node.hostname == ubuntu-swarm-manager' flink jobmanager
docker service create --name taskmanager --env
JOB_MANAGER_RPC_ADDRESS=jobmanager --network overlay --constraint
'node.hostname != ubuntu-swarm-manager' flink taskmanager
I wonder if there's any configuration I'm missing. This is the error I get:
- Trying to register at JobManager akka.tcp://flink@jobmanager:6123/
user/jobmanager (attempt 4, timeout: 4000 milliseconds)
--
Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/
To view the terms under which this email is distributed, please go to:-
http://disclaimer.leedsbeckett.ac.uk/disclaimer/disclaimer.html
To view the terms under which this email is distributed, please go to:-
http://disclaimer.leedsbeckett.ac.uk/disclaimer/disclaimer.html