1 2# The contents of this file are subject to the Mozilla Public License 3# (MPL) Version 1.1 (the "License"); you may not use this file except 4# in compliance with the License. You may obtain a copy of the License 5# at http://www.mozilla.org/MPL/ 6# 7# Software distributed under the License is distributed on an "AS IS" 8# basis, WITHOUT WARRANTY OF ANY KIND, either express or implied. See 9# the License for the specific language governing rights and 10# limitations under the License. 11# 12# The Original Code is LEPL (http://www.acooke.org/lepl) 13# The Initial Developer of the Original Code is Andrew Cooke. 14# Portions created by the Initial Developer are Copyright (C) 2009-2010 15# Andrew Cooke (andrew@acooke.org). All Rights Reserved. 16# 17# Alternatively, the contents of this file may be used under the terms 18# of the LGPL license (the GNU Lesser General Public License, 19# http://www.gnu.org/licenses/lgpl.html), in which case the provisions 20# of the LGPL License are applicable instead of those above. 21# 22# If you wish to allow use of your version of this file only under the 23# terms of the LGPL License and not to allow others to use your version 24# of this file under the MPL, indicate your decision by deleting the 25# provisions above and replace them with the notice and other provisions 26# required by the LGPL License. If you do not delete the provisions 27# above, a recipient may use your version of this file under either the 28# MPL or the LGPL License. 29 30''' 31Stream support for lexers. 32''' 33 34 35fromlepl.stream.iterimportbase_iterable_factory 36fromlepl.stream.coreimportOFFSET,s_delta,s_line,HashKey,s_key,s_next 37fromlepl.stream.facadeimportHelperFacade 38fromlepl.support.libimportfmt,LogMixin 39 40

42''' 43 This wraps a sequence of values generated by the lexer. The sequence 44 is a source of (tokens, stream) instances, where the stream was generated 45 from the source. 46 47 It follows that the `value` returned by s_next is also (tokens, stream). 48 This is interpreted by `Token` which forwards `stream` to sub-matchers. 49 50 Implementation is vaguely similar to `IterableHelper`, in that we use 51 a `Cons` based linked list to allow memory handling. However, instead 52 of a "line" of data, each node contains, again, (tokens, stream) and 53 there is no need to store the line_stream explicitly in the state. 54 ''' 55

80''' 81 This doesn't have much meaning in terms of tokens, but might be 82 used for some debug output, so return something vaguely useful. 83 ''' 84try: 85# implement in terms of next so that filtering works as expected 86((_,line_stream),_)=self.next(cons) 87returns_line(line_stream,empty_ok) 88exceptStopIteration: 89ifempty_ok: 90raiseTypeError('Token stream cannot return an empty line') 91else: 92raise

111'''112 Used by `RestrictTokensBy` to filter tokens from the delegate.113114 This filters a list of token IDs in order. If the entire list does115 not match then then next token is returned (even if it appears in the116 list).117 '''118

125126defadd_self(response):127'''128 Replace the previous helper with this one, which will then 129 delegate to the previous when needed.130 '''131((tokens,token),(state,_))=response132self._debug(fmt('Return {0}',tokens))133return((tokens,token),(state,self))