flink-user mailing list archives

Great then, I will look into my configuration. Thanks for your help!
Cheers,
Theofilos
On 6/15/2016 2:00 PM, Maximilian Michels wrote:
> You should also see TaskManager output in the logs. I just verified
> this using Flink 1.0.3 with Hadoop 2.7.1. I executed the Iterate
> example and it aggregated correctly including the TaskManager logs.
>
> I'm wondering, is there anything in the Hadoop logs of the
> Resourcemanager/Nodemanager that could indicate a transfer failure?
>
> On Wed, Jun 15, 2016 at 10:18 AM, Theofilos Kakantousis <tkak@kth.se
> <mailto:tkak@kth.se>> wrote:
>
> Hi,
>
> By yarn aggregated log I mean Yarn log aggregation is enabled and
> the log I'm referring to is the one returned by `yarn logs
> -applicationId <id>`. When running a Spark job for example on the
> same setup, the yarn aggregated log contains all the information
> printed out by the application.
>
> Cheers,
> Theofilos
>
>
> On 6/15/2016 10:14 AM, Maximilian Michels wrote:
>> Please use the `yarn logs -applicationId <id>` to retrieve the
>> logs. If you have enabled log aggregation, this will give you all
>> container logs concatenated.
>>
>> Cheers,
>> Max
>>
>> On Wed, Jun 15, 2016 at 12:24 AM, Theofilos Kakantousis
>> <tkak@kth.se <mailto:tkak@kth.se>> wrote:
>>
>> Hi Max,
>>
>> The runBlocking(..) problem was due to a Netty dependency
>> issue on my project, it works fine now :)
>>
>> To pinpoint the logging issue, I just ran a single flink job
>> on yarn as per the documentation "./bin/flink run -m
>> yarn-cluster -yn 2 ./examples/streaming/Iteration.jar" and I
>> have the same issue.During the job I can see in the
>> containers the taskmanager logs, and a sample output from the
>> taskmanager.out is the following:
>> "cat
>> /srv/hadoop/logs/userlogs/application_1465901188070_0037/container_1465901188070_0037_01_000002/taskmanager.out
>>
>> 2> ((49,1),3)
>> 2> ((25,11),4)
>> 2> ((46,44),2
>> .."
>>
>> However, the yarn aggregated log contains only the jobmanager
>> output. Is this expected or could it indicate a problem with
>> my hadoop logging configuration not picking up taskmanager logs?
>>
>> Cheers,
>> Theofilos
>>
>>
>> On 6/13/2016 12:13 PM, Maximilian Michels wrote:
>>
>> Hi Theofilos,
>>
>> Flink doesn't send the local client output to the Yarn
>> cluster. I
>> think this will only change once we move the entire
>> execution of the
>> Job to the cluster framework. All output of the actual
>> Flink job
>> should be within the JobManager or TaskManager logs.
>>
>> There is something wrong with the network communication
>> if the Client
>> doesn't return from `runBlocking(..)`. Would be
>> interesting to take a
>> look at the logs to find out why that could be.
>>
>> Cheers,
>> Max
>>
>>
>> On Sat, Jun 11, 2016 at 1:53 PM, Theofilos Kakantousis
>> <tkak@kth.se <mailto:tkak@kth.se>> wrote:
>>
>> Hi Robert,
>>
>> Thanks for the prompt reply. I'm using the
>> IterateExample from Flink
>> examples. In the yarn log I get entries for the
>> YarnJobManager and
>> ExecutionGraph, but I was wondering if there is a way
>> to push all the
>> logging that the client produces into the yarn log.
>> Including the System.out
>> calls. Is there a way to modify the example to use a
>> logging framework to
>> achieve it?
>>
>> Also when I submit the program using the Client
>> runBlocking method, although
>> I see in the taskmanager and jobmanager log that the
>> application has
>> finished, the runBlocking method does not return.
>> Should I call it in a
>> separate thread?
>>
>> Cheers,
>> Theofilos
>>
>> On 2016-06-10 22:12, Robert Metzger wrote:
>>
>> Hi Theofilos,
>>
>> how exactly are you writing the application output?
>> Are you using a logging framework?
>> Are you writing the log statements from the open(),
>> map(), invoke() methods
>> or from some constructors? (I'm asking since
>> different parts are executed on
>> the cluster and locally).
>>
>> On Fri, Jun 10, 2016 at 4:00 PM, Theofilos
>> Kakantousis <tkak@kth.se <mailto:tkak@kth.se>> wrote:
>>
>> Hi all,
>>
>> Flink 1.0.3
>> Hadoop 2.4.0
>>
>> When running a job on a Flink Cluster on Yarn,
>> the application output is
>> not included in the Yarn log. Instead, it is only
>> printed in the stdout from
>> where I run my program. For the jobmanager, I'm
>> using the log4j.properties
>> file from the flink/conf directory. Yarn log
>> aggregation is enabled and the
>> YarnJobManager log is printed in the yarn log.
>> The application is submitted
>> by a Flink Client to the FlinkYarnCluster using a
>> PackagedProgram.
>>
>> Is this expected behavior and if so, is there a
>> way to include the
>> application output in the Yarn aggregated log?
>> Thanks!
>>
>> Cheers,
>> Theofilos
>>
>>
>>
>>
>
>