WINNER-TAKE-ALL ABSTRACT

Transcription

1 703 WINNER-TAKE-ALL NETWORKS OF O(N) COMPLEXITY J. Lazzaro, S. Ryckebusch, M.A. Mahowald, and C. A. Mead California Institute of Technology Pasadena, CA ABSTRACT We have designed, fabricated, and tested a series of compact CMOS integrated circuits that realize the winner-take-all function. These analog, continuous-time circuits use only O(n) of interconnect to perform this function. We have also modified the winner-take-all circuit, realizing a circuit that computes local nonlinear inhibition. Two general types of inhibition mediate activity in neural systems: subtractive inhibition, which sets a zero level for the computation, and multiplicative (nonlinear) inhibition, which regulates the gain of the computation. We report a physical realization of general nonlinear inhibition in its extreme form, known as winner-take-all. We have designed and fabricated a series of compact, completely functional CMOS integrated circuits that realize the winner-take-all function, using the full analog nature of the medium. This circuit has been used successfully as a component in several VLSI sensory systems that perform auditory localization (Lazzaro and Mead, in press) and visual stereopsis (Mahowald and Delbruck, 1988). Winnertake-all circuits with over 170 inputs function correctly in these sensory systems. We have also modified this global winner-take-all circuit, realizing a circuit that computes local nonlinear inhibition. The circuit allows multiple winners in the network, and is well suited for use in systems that represent a feature space topographically and that process several features in parallel. We have designed, fabricated, and tested a CMOS integrated circuit that computes locally the winner-take-all function of spatially ordered input.

2 704 Lazzaro, Ryckebusch, Mahowald and Mead THE WINNER-TAKE-ALL CmCUIT Figure 1 is a schematic diagram of the winner-take-all circuit. A single wire, associated with the potential Vc, computes the inhibition for the entire circuit; for an n neuron circuit, this wire is O(n) long. To compute the global inhibition, each neuron k contributes a current onto this common wire, using transistor T2 a.' To apply this global inhibition locally, each neuron responds to the common wire voltage Vc, using transistor Tla.' This computation is continuous in time; no clocks are used. The circuit exhibits no hysteresis, and operates with a time constant related to the size of the largest input. The output representation of the circuit is not binary; the winning output encodes the logarithm of its associated input. Figure 1. Schematic diagram of the winner-take-all circuit. Each neuron receives a unidirectional current input 11;; the output voltages VI.. VB represent the result of the winner-take-all computation. If II; = max(ii I B ), then VI; is a logarithmic function of 11;; if Ii <: 11;, then Vi ~ O. A static and dynamic ana.lysis of the two-neuron circuit illustrates these system properties. Figure 2 shows a schematic diagram of a two-neuron winner-take-all circuit. To understand the beha.vior of the circuit, we first consider the input condition II = 12 = 1m. Transistors TIl ~d T12 have identical potentials at gate and source, and are both sinking 1m; thus, the drain potentials VI and V2 must be equal. Transistors T21 and T22 have identical source, drain, and gate potentials, and therefore must sink the identical current ICI = IC2 = Ic/2. In the subthreshold region of operation, the equation 1m = 10 exp(vc/vo) describes transistors Til and T12, where 10 is a fabrication parameter, and Vo = kt/qlt. Likewise, the equation Ic/2 = 10 exp((vm - T22. Solving for Vm(Im, Ie) yields Vel/Volt where Vm = VI = V2, describes transistors T21 and Vm = Voln(~:) + Voln(:;). (1)

3 Winner-Take-All Networks ofo(n) Complexity 705 Thus, for equal input currents, the circuit produces equal output voltages; this behavior is desirable for a winner-take-all circuit. In addition, the output voltage V m logarithmically encodes the magnitude of the input current 1m. Figure 2. Schematic diagram of a two-neuron winner-take-all circuit. The input condition II = 1m + Oi, 12 = 1m illustrates the inhibitory action of the circuit. Transistor Til must sink 0, more current than in the previous example; as a result, the gate voltage of Til rises. Transistors Tit and TI2 share a common gate, howeverj thus, TI2 must also sink 1m + 0,. But only 1m is present at the drain of T 12 To compensate, the drain voltage of T 12, V2, must decrease. For small OiS, the Early effect serves to decrease the current through T h, decreasing V 2 linearly with 0,. For large o's, TI2 must leave saturation, driving V2 to approximately 0 volts. As desired, the output associated with the smaller input diminishes. For large OiS, Ie2 $!:::f 0, and Iel $!:::f Ie. The equation 1m + 0, = 10 exp(ve/vo) describes transistor Til' and the equation Ie = 10 exp((vi - Vel/Yo) describes transistor T21 Solving for VI yields The winning output encodes the logarithm of the associated input. The symmetrical circuit topology ensures similar behavior for increases in 12 relative to II. Equation 2 predicts the winning response of the circuit; a more complex expression, derived in (Lazzaro et.al., 1989), predicts the losing and crossover response of the circuit. Figure 3 is a plot of this analysis, fit to experimental data. Figure 4 shows the wide dynamic range and logarithmic properties of the circuitj the experiment in Figure 3 is repeated for several values of 12, ranging over four orders of magnitude. The conductance of transistors Til and T1:a determines the losing response of the circuit. Variants of the winner-take-all circuit shown in (Lazzaro et. ai., 1988) achieve losing responses wider and narrower than Figure 3, using circuit and mask layout techniques. (2)

4 706 Lazzaro, Ryckebusch, Mahowald and Mead WINNER-TAKE-ALL TIME RESPONSE A good winner-take-all circuit should be stable, and should not exhibit damped oscillations ("ringing") in response to input changes. This section explores these dynamic properties of our winner-take-all circuit, and predicts the temporal response of the circuit. Figure 8 shows the two-neuron winner-take-all circuit, with capacitances added to model dynamic behavior. o T 102 Vo Ie Figure 8. Schematic diagram of a two-neuron winner-take-all circuit, with capacitances added for dynamic analysis. 0 is a large MOS capacitor added to each neuron for smoothingj 0., models the parasitic capacitance contributed by the gates of Tu and T12, the drains of T21 and T22, and the interconnect. (Lazzaro et. al., 1988) shows a small-signal analysis of this circuit. The transfer function for the circuit has real poles, and thus the circuit is stable and does not ring, if 10 > 41(Oe/O), where 11 RlI2 Rl 1. Figure 9 compares this bound with experimental data. H Ie > 41(00 /0), the circuit exhibits first-order behavior. The time constant OVo/I sets the dynamics of the winning neuron, where Vo = A:T /qk. Rl 40 mv. The time constant OVE/I sets the dynamics of the losing neuron, where VE Rl 50 v. Figure 10 compares these predictions with experimental data.

5 Winner-Take-All Networks ofo(n) Complexity 707 Vl,V, (V) I.S ~I----t~--t---f S II/I, Figure 8. Experimental data (circles) and theory (solid lines) for a two-neuron winner-take-all circuit. II, the input current of the first neuron, is swept about the value of 12, the input current of the second neuron; neuron voltage outputs VI and V2 are plotted versus normalized input current. 2.6 I.S IdA) Figure 4. The experiment of Figure 3 is repeated for several values of 12; experimental data of output voltage response are plotted versus absolute input current on a log scale. The output voltage VI = V 2 is highlighted with a circle for each experiment. The dashed line is a theoretical expression confirming logarithmic behavior over four orders of magnitude (Equation 1).

6 708 Lazzaro, Ryckebusch, Mahowald and Mead 1 Figure 9. Experimental data (circles) and theoretical statements (solid line) for a two-neuron winner-take-all circuit, showing the smallest 1 0, for a given I, necessary for a first-order response to a small-signal step input. Figure 10. Experimental data (symbols) and theoretical statements (solid line) for a two-neuron winner-take-all circuit, showing the time constant of the first-order response to a small-signal step input. The winning response (filled circles) and losing response (triangles) of a winner-take-a.ll circuit are shownj the time constants differ by several orders of magnit ude.

7 Winner~Take~AlI Networks ofo(n) Complexity 709 THE LOCAL NONLINEAR INHIBITION CIRCUIT The winner-take-all circuit in Figure 1, as previously explained, locates the largest input to the circuit. Certain applications require a gentler form of nonlinear inhibition. Sometimes, a circuit that can represent multiple intensity scales is necessary. Without circuit modification, the winner-take-all circuit in Figure 1 can perform this task. (Lazzaro et. al., 1988) explains this mode of operation. Other applications require a local winner-take-all computation, with each winner having inhuence over only a limited spatial area. Figure 12 shows a circuit that computes the local winner-taite-all function. The circuit is identical to the original winner-take-all circuit, except that each neuron connects to its nearest neighbors with a nonlinear resistor circuit (Mead, in press). Each resistor conducts a current Ir in response to a voltage ~V across it, where Ir = I.tanh(~V/(2Vo)). 1., the saturating current of the resistor, is a controllable parameter. The current source, 10, present in the original winner-take-all circuit, is distributed between the resistors in the local winner-take-all circuit. Figure 11. Schematic diagram of a section of the local winner-take-all circuit. Each neuron i receives a unidirectional current input Iii the output voltages Vi represent the result of the local winner-take-all computation. To understand the operation of the local winner-take-all circuit, we consider the circuit response to a spatial impulse, defined as 1" :> 1, where 1 == h~". 1,,:> 1"-1 and 1,,:> 1"+1, so Ve:,. is much larger than Ve:,._l and Ve:lI+l' and the resistor circuits connecting neuron 1: with neuron 1: - 1 and neuron 1: + 1 saturate. Each resistor sinks 1. current when saturatedj transistor T2,. thus conducts Ie: current. In the subthreshold region of operation, the equation 1" = 10 exp(ve:,. /Vo) describes transistor T I,., and the equation Ie = Ioexp((V" - Ve:,.)/Vo) describes transistor

8 710 Lazzaro, Ryckebusch, Mahowald and Mead T2,.. Solving for VA: yields VA: = voln((2i )/10 ) + voln(ia:/lo). (4) As in the original winner-take-all circuit, the output of a winning neuron encodes the logarithm of that neuron's associated input. As mentioned, the resistor circuit connecting neuron Ie with neuron Ie - 1 sinks 1. CUlTent. The current sources 10 associated with neurons Ie -1, Ie - 2,... must supply this current. If the current source 10 for neuron Ie - 1 supplies part of this current, the transistor T2,._1 carries no current, and the neuron output VA:-l approaches zero. In this way, a winning neuron inhibits its neighboring neurons. This inhibitory action does not extend throughout the network. Neuron Ie needs only 1. current from neurons Ie - 1, Ie - 2,... Thus, neurons sufficiently distant from neuron Ie maintain the service of their current source 10, and the outputs of these distant neurons can be active. Since, for a spatial impulse, all neurons Ie - 1, Ie - 2,... have an equal input current I, all distant neurons have the equal output Similar reasoning applies for neurons Ie + 1, Ie + 2,... The relative values of 1. and 10 determine the spatial extent of the inhibitory action. Figure 12 shows the spatial impulse response of the local winner-take-all circuit, for different settings of 1./1 0, (5) I I I I I o Ie (Pollition) Figure 12. Experimental data showing the spatial impulse response of the local winner-take-all circuit, for values of 1./10 ranging over a factor of Wider inhibitory responses correspond to larger ratios. For clarity, the plots are vertically displaced in 0.25 volt increments.

9 Winner-Take-All Networks ofo(n) Complexity 711 CONCLUSIONS The circuits described in this paper use the full analog nature of MOS devices to realize an interesting class of neural computations efficiently. The circuits exploit the physics of the medium in many ways. The winner-take-all circuit uses a single wire to compute and communicate inhibition for the entire circuit. Transistor TI,. in the winner-take-all circuit uses two physical phenomena in its computation: its exponential current function encodes the logarithm of the input, and the finite conductance of the transistor defines the losing output response. As evolution exploits all the physical properties of neural devices to optimize system performance, designers of synthetic neural systems should strive to harness the full potential of the physics of their media. Acknow ledgments John Platt, John Wyatt, David Feinstein, Mark Bell, and Dave Gillespie provided mathematical insights in the analysis of the circuit. Lyn Dupre proofread the document. We thank Hewlett-Packard for computing support, and DARPA and MOSIS for chip fabrication. This work was sponsored by the Office of Naval Research and the System Development Foundation. References Lazzaro, J. P., Ryckebusch, S., Mahowald, M.A., and Mead, C.A. (1989). Winner Take-All Networks of O(N) Oomplexity, Caltech Computer Science Department Technical Report Caltech-CS-TR Lazzaro, J. P., and Mead, C.A. {in press}. Silicon Models of Auditory Localization, Neural Oomputation. Mahowald, M.A., and Delbruck, T.I. (1988). An Analog VLSI Implementation of the Marr-Poggio Stereo Correspondence Algorithm, Abstracts of the First Annual INNS Meeting, Boston, 1988, Vol. I, Supplement I, p Mead, C. A. (in press). Analog VLSI and Neural Systems. Reading, MA: Addison Wesley.

PHYSCS 111 LABORATORY Experiment #3 Current, Voltage and Resistance in Series and Parallel Circuits This experiment is designed to investigate the relationship between current and potential in simple series

Supply voltage Supervisor TL77xx Series Author: Eilhard Haseloff Literature Number: SLVAE04 March 1997 i IMPORTANT NOTICE Texas Instruments (TI) reserves the right to make changes to its products or to

0. Bode Plots Introduction Each of the circuits in this problem set is represented by a magnitude Bode plot. The network function provides a connection between the Bode plot and the circuit. To solve these

Op-Amp Simulation EE/CS 5720/6720 Read Chapter 5 in Johns & Martin before you begin this assignment. This assignment will take you through the simulation and basic characterization of a simple operational

Laboratory 4: Feedback and Compensation To be performed during Week 9 (Oct. 20-24) and Week 10 (Oct. 27-31) Due Week 11 (Nov. 3-7) 1 Pre-Lab This Pre-Lab should be completed before attending your regular

5/16 Capacitance-1/5 CAPACITANCE IN A RC CIRCUIT PURPOSE: To observe the behavior of resistor-capacitor circuit, to measure the RC time constant and to understand how it is related to the time dependence

Low Power Low Offset Voltage Quad Comparators General Description The LM139 series consists of four independent precision voltage comparators with an offset voltage specification as low as 2 mv max for

Measuring Electric Phenomena: the Ammeter and Voltmeter 1 Objectives 1. To understand the use and operation of the Ammeter and Voltmeter in a simple direct current circuit, and 2. To verify Ohm s Law for

5. Sequential CMOS Logic Circuits In sequential logic circuits the output signals is determined by the current inputs as well as the previously applied input variables. Fig. 5.1a shows a sequential circuit

Basic Op Amp ircuits Manuel Toledo INEL 5205 Instrumentation August 3, 2008 Introduction The operational amplifier (op amp or OA for short) is perhaps the most important building block for the design of

THAT Corporation Design Note 5. Channel Volume Control The circuits within this application note feature THAT8x to provide the essential function of voltage-controlled amplifier (VCA). Since writing this

Voltage/current converter opamp circuits This worksheet and all related files are licensed under the Creative Commons Attribution License, version 1.0. To view a copy of this license, visit http://creativecommons.org/licenses/by/1.0/,

Oscillations and Regenerative Amplification using Negative Resistance Devices Ramon Vargas Patron rvargas@inictel.gob.pe INICTEL The usual procedure for the production of sustained oscillations in tuned

CMOS Power Consumption and C pd Calculation SCAA035B June 1997 1 IMPORTANT NOTICE Texas Instruments (TI) reserves the right to make changes to its products or to discontinue any semiconductor product or

Chapter 6 Capacitors and RC Circuits Name: Lab Partner: Section: 6.1 Purpose The purpose of this experiment is to investigate the physics of capacitors in circuits. The charging and discharging of a capacitor

Small Signal Analysis of a PMOS transistor Consider the following PMOS transistor to be in saturation. Then, 1 I SD = µ pcox( VSG Vtp)^2(1 + VSDλ) 2 From this equation it is evident that I SD is a function

Physics 3330 Experiment #7 Fall 1999 Transistor Amplifiers Purpose The aim of this experiment is to develop a bipolar transistor amplifier with a voltage gain of minus 25. The amplifier must accept input

Storing And Releasing Charge In A Circuit Topic The characteristics of capacitors Introduction A capacitor is a device that can retain and release an electric charge, and is used in many circuits. There

Loop Bandwidth and Clock Data Recovery (CDR) in Oscilloscope Measurements Application Note 1304-6 Abstract Time domain measurements are only as accurate as the trigger signal used to acquire them. Often

C esonant Circuits Andrew McHutchon April 20, 203 Capacitors and Inductors There is a lot of inconsistency when it comes to dealing with reactances of complex components. The format followed in this document

Table of Contents Part 1: Audio Power Amplifier Basics 1. Introduction 1.1 Organization of the book 1.2 The role of the power amplifier 1.3 Basic performance specifications 1.4 Additional performance specifications

Building the AMP Amplifier Introduction For about 80 years it has been possible to amplify voltage differences and to increase the associated power, first with vacuum tubes using electrons from a hot filament;

Creating a Usable Power Supply from a Solar Panel An exploration in DC- DC converters By Kathleen Ellis Advised by Dr. Derin Sherman Department of Physics, Cornell College November 21, 2012 Introduction

Inductors This worksheet and all related files are licensed under the Creative Commons Attribution License, version 1.0. To view a copy of this license, visit http://creativecommons.org/licenses/by/1.0/,

J Indian Inst Sci, July Aug 2006, 86, 315 325 Indian Institute of Science Application of network analyzer in measuring the performance functions of power supply B SWAMINATHAN* AND V RAMANARAYANAN Power

SIMULATIONS OF PARALLEL RESONANT CIRCUIT POWER ELECTRONICS COLORADO STATE UNIVERSITY Page 1 of 25 PURPOSE: The purpose of this lab is to simulate the LCC circuit using MATLAB and ORCAD Capture CIS to better

Analog to Digital, A/D, Digital to Analog, D/A Converters An electronic circuit to convert the analog voltage to a digital computer value Best understood by understanding Digital to Analog first. A fundamental

CHAPTER 8 ELECTRIC CIRCUITS 1. Sketch a circuit diagram for a circuit that includes a resistor R 1 connected to the positive terminal of a battery, a pair of parallel resistors R and R connected to the

Notes about Small Signal Model for EE 40 Intro to Microelectronic Circuits 1. Model the MOSFET Transistor For a MOSFET transistor, there are NMOS and PMOS. The examples shown here would be for NMOS. Figure

Using the Impedance Method The impedance method allows us to completely eliminate the differential equation approach for the determination of the response of circuits. In fact the impedance method even

1 Lab 03: Differential Amplifiers (BJT) (20 points) NOTE: 1) Please use the basic current mirror from Lab01 for the second part of the lab (Fig. 3). 2) You can use the same chip as the basic current mirror;

NOTE: This project does not include a complete parts list. In particular, the IC described here does not come in a dual-inline-package (DIP), and so a gull-wing package has to be soldered to an adaptor

Physics 623 Transistor Characteristics and Single Transistor Amplifier Sept. 8, 1997 1 Purpose To measure and understand the common emitter transistor characteristic curves. To use the base current gain

DESIGN CHALLENGES OF TECHNOLOGY SCALING IS PROCESS TECHNOLOGY MEETING THE GOALS PREDICTED BY SCALING THEORY? AN ANALYSIS OF MICROPROCESSOR PERFORMANCE, TRANSISTOR DENSITY, AND POWER TRENDS THROUGH SUCCESSIVE

Lab 7: Operational Amplifiers Part I Objectives The objective of this lab is to study operational amplifier (op amp) and its applications. We will be simulating and building some basic op amp circuits,

CMOS, the Ideal Logic Family INTRODUCTION Let s talk about the characteristics of an ideal logic family. It should dissipate no power, have zero propagation delay, controlled rise and fall times, and have

Technical Note #3 Error Amplifier Design and Applications Introduction All regulating power supplies require some sort of closed-loop control to force the output to match the desired value. Both digital

GT Sensors Precision Gear Tooth and Encoder Sensors NVE s GT Sensor products are based on a Low Hysteresis GMR sensor material and are designed for use in industrial speed applications where magnetic detection

LAB 7 MOSFET CHARACTERISTICS AND APPLICATIONS Objective In this experiment you will study the i-v characteristics of an MOS transistor. You will use the MOSFET as a variable resistor and as a switch. BACKGROUND

1: (ta initials) 2: first name (print) last name (print) brock id (ab13cd) (lab date) Experiment 1 Capacitance In this Experiment you will learn the relationship between the voltage and charge stored on

page 1 of 7 MOSFET DIFFERENTIAL AMPLIFIER (TWO-WEEK LAB) BACKGROUND The MOSFET is by far the most widely used transistor in both digital and analog circuits, and it is the backbone of modern electronics.