System And Method For Memory Hub-based Expansion Bus - Patent 7610430 by Patents-57

The present invention relates generally to a memory system for a processor-based computing system, and more particularly, to a hub-based memory system providing expansion capabilities for computer components.BACKGROUND OF THE INVENTIONComputer systems use memory devices, such as dynamic random access memory ("DRAM") devices, to store data that are accessed by a processor. These memory devices are normally used as system memory in a computer system. In a typical computersystem, the processor communicates with the system memory through a processor bus and a memory controller. The memory devices of the system memory, typically arranged in memory modules having multiple memory devices, are coupled through a memory bus tothe memory controller. The processor issues a memory request, which includes a memory command, such as a read command, and an address designating the location from which data or instructions are to be read. The memory controller uses the command andaddress to generate appropriate command signals as well as row and column addresses, which are applied to the system memory through the memory bus. In response to the commands and addresses, data are transferred between the system memory and theprocessor. The memory controller is often part of a system controller, which also includes bus bridge circuitry for coupling the processor bus to an expansion bus, such as a PCI bus.In memory systems, high data bandwidth is desirable. Generally, bandwidth limitations are not related to the memory controllers since the memory controllers sequence data to and from the system memory as fast as the memory devices allow. Oneapproach that has been taken to increase bandwidth is to increase the speed of the memory data bus coupling the memory controller to the memory devices. Thus, the same amount of information can be moved over the memory data bus in less time. However,despite increasing memory data bus speeds, a corresponding increase in bandwidth does not result

United States Patent: 7610430
&nbsp;
( 1 of 1 )
United States Patent
7,610,430
Jeddeloh
October 27, 2009
System and method for memory hub-based expansion bus
Abstract
A system memory includes a memory hub controller, a memory module
accessible by the memory hub controller, and an expansion module having a
processor circuit coupled to the memory module and also having access to
the memory module. The memory hub controller is coupled to the memory hub
through a first portion of a memory bus on which the memory requests from
the memory hub controller and memory responses from the memory hub are
coupled. A second portion of the memory bus couples the memory hub to the
processor circuit and is used to couple memory requests from the
processor circuit and memory responses provided by the memory hub to the
processor circuit.
Inventors:
Jeddeloh; Joseph M. (Shoreview, MN)
Assignee:
Micron Technology, Inc.
(Boise,
ID)
Appl. No.:
12/075,424
Filed:
March 10, 2008
Related U.S. Patent Documents
Application NumberFiling DatePatent NumberIssue Date
11715517Mar., 20077370134
11399873Apr., 20067206887
10810229Mar., 20047120723
Current U.S. Class:
710/311 ; 365/230.01
Current International Class:
G06F 13/36&nbsp(20060101)
Field of Search:
710/311,312,316 711/119,5 365/230.01,230.03 370/423,912
References Cited [Referenced By]
U.S. Patent Documents
4045781
August 1977
Levy et al.
4240143
December 1980
Besemer et al.
4245306
January 1981
Besemer et al.
4253144
February 1981
Bellamy et al.
4253146
February 1981
Bellamy et al.
4443845
April 1984
Hamilton et al.
4608702
August 1986
Hirzel et al.
4707823
November 1987
Holdren et al.
4724520
February 1988
Athanas et al.
4809232
February 1989
Baumbaugh et al.
4813772
March 1989
Kowel et al.
4891808
January 1990
Williams
4930128
May 1990
Suzuki et al.
4953930
September 1990
Ramsey et al.
5241506
August 1993
Motegi et al.
5243703
September 1993
Farmwald et al.
5251303
October 1993
Fogg, Jr. et al.
5269022
December 1993
Shinjo et al.
5307381
April 1994
Ahuja
5317752
May 1994
Jewett et al.
5319755
June 1994
Farmwald et al.
5327553
July 1994
Jewett et al.
5355391
October 1994
Horowitz et al.
5379382
January 1995
Work et al.
5432823
July 1995
Gasbarro et al.
5432907
July 1995
Picazo, Jr. et al.
5442770
August 1995
Barratt
5461627
October 1995
Rypinski
5465229
November 1995
Bechtolsheim et al.
5479370
December 1995
Furuyama et al.
5497476
March 1996
Oldfield et al.
5502621
March 1996
Schumacher et al.
5544319
August 1996
Acton et al.
5566325
October 1996
Bruce, II et al.
5577220
November 1996
Combs et al.
5581767
December 1996
Katsuki et al.
5606717
February 1997
Farmwald et al.
5608264
March 1997
Gaul
5623534
April 1997
Desai et al.
5638334
June 1997
Farmwald et al.
5659798
August 1997
Blumrich et al.
5706224
January 1998
Srinivasan et al.
5715456
February 1998
Bennett et al.
5729709
March 1998
Harness
5748616
May 1998
Riley
5787475
July 1998
Pawlowski
5808897
September 1998
Miller, Jr. et al.
5818844
October 1998
Singh et al.
5819304
October 1998
Nilsen et al.
5822255
October 1998
Uchida
5831467
November 1998
Leung et al.
5832250
November 1998
Whittaker
5875352
February 1999
Gentry et al.
5875454
February 1999
Craft et al.
5928343
July 1999
Farmwald et al.
5966724
October 1999
Ryan
5973935
October 1999
Schoenfeld et al.
5973951
October 1999
Bechtolsheim et al.
5978567
November 1999
Rebane et al.
5987196
November 1999
Noble
6023726
February 2000
Saksena
6026226
February 2000
Heile et al.
6029250
February 2000
Keeth
6031241
February 2000
Silfvast et al.
6033951
March 2000
Chao
6061263
May 2000
Boaz et al.
6061296
May 2000
Ternullo, Jr. et al.
6067262
May 2000
Irrinki et al.
6073190
June 2000
Rooney
6076139
June 2000
Welker et al.
6078451
June 2000
Ioki
6079008
June 2000
Clery, III
6098158
August 2000
Lay et al.
6101151
August 2000
Watanabe et al.
6105075
August 2000
Ghaffari
6111757
August 2000
Dell et al.
6125431
September 2000
Kobayashi
6131149
October 2000
Lu et al.
6134624
October 2000
Burns et al.
6137709
October 2000
Boaz et al.
6144587
November 2000
Yoshida
6167465
December 2000
Parvin et al.
6167486
December 2000
Lee et al.
6175571
January 2001
Haddock et al.
6185352
February 2001
Hurley
6186400
February 2001
Dvorkis et al.
6191663
February 2001
Hannah
6201724
March 2001
Ishizaki et al.
6208180
March 2001
Fisch et al.
6219725
April 2001
Diehl et al.
6226729
May 2001
Stevens et al.
6229727
May 2001
Doyle
6233376
May 2001
Updegrove
6243769
June 2001
Rooney
6243831
June 2001
Mustafa et al.
6246618
June 2001
Yamamoto et al.
6247107
June 2001
Christie
6249802
June 2001
Richardson et al.
6256692
July 2001
Yoda et al.
6266730
July 2001
Perino et al.
6272609
August 2001
Jeddeloh
6285349
September 2001
Smith
6286083
September 2001
Chin et al.
6294937
September 2001
Crafts et al.
6301637
October 2001
Krull et al.
6327642
December 2001
Lee et al.
6330205
December 2001
Shimizu et al.
6347055
February 2002
Motomura
6349363
February 2002
Cai et al.
6356573
March 2002
Jonsson et al.
6366529
April 2002
Williams et al.
6367074
April 2002
Bates et al.
6370068
April 2002
Rhee
6373777
April 2002
Suzuki
6381190
April 2002
Shinkai
6392653
May 2002
Malandain et al.
6401213
June 2002
Jeddeloh
6405280
June 2002
Ryan
6421744
July 2002
Morrison et al.
6434639
August 2002
Haghighi
6434654
August 2002
Story et al.
6434696
August 2002
Kang
6434736
August 2002
Schaecher et al.
6438622
August 2002
Haghighi et al.
6438668
August 2002
Esfahani et al.
6449308
September 2002
Knight, Jr. et al.
6453393
September 2002
Holman et al.
6457116
September 2002
Mirsky et al.
6462978
October 2002
Shibata et al.
6463059
October 2002
Movshovich et al.
6470422
October 2002
Cai et al.
6473828
October 2002
Matsui
6477592
November 2002
Chen et al.
6477614
November 2002
Leddige et al.
6477621
November 2002
Lee et al.
6479322
November 2002
Kawata et al.
6490188
December 2002
Nuxoll et al.
6493803
December 2002
Pham et al.
6496909
December 2002
Schimmel
6501471
December 2002
Venkataraman et al.
6502161
December 2002
Perego et al.
6505287
January 2003
Uematsu
6523092
February 2003
Fanning
6523093
February 2003
Bogin et al.
6526498
February 2003
Mirsky et al.
6539490
March 2003
Forbes et al.
6552564
April 2003
Forbes et al.
6553479
April 2003
Mirsky et al.
6564329
May 2003
Cheung et al.
6570429
May 2003
Hellriegel
6584543
June 2003
Williams et al.
6587912
July 2003
Leddige et al.
6590816
July 2003
Perner
6594713
July 2003
Fuoco et al.
6594722
July 2003
Willke, II et al.
6598154
July 2003
Vaid et al.
6615325
September 2003
Mailloux et al.
6622227
September 2003
Zumkehr et al.
6628294
September 2003
Sadowsky et al.
6629220
September 2003
Dyer
6631440
October 2003
Jenne et al.
6636110
October 2003
Ooishi et al.
6636957
October 2003
Stevens et al.
6646929
November 2003
Moss et al.
6651139
November 2003
Ozeki et al.
6658509
December 2003
Bonella et al.
6662304
December 2003
Keeth et al.
6665202
December 2003
Lindahl et al.
6667895
December 2003
Jang et al.
6681292
January 2004
Creta et al.
6681301
January 2004
Mehta et al.
6697926
February 2004
Johnson et al.
6711652
March 2004
Arimilli et al.
6715018
March 2004
Farnworth et al.
6718440
April 2004
Maiyuran et al.
6721187
April 2004
Hall et al.
6721195
April 2004
Brunelle et al.
6724685
April 2004
Braun et al.
6728800
April 2004
Lee et al.
6735679
May 2004
Herbst et al.
6735682
May 2004
Segelken et al.
6745275
June 2004
Chang
6751113
June 2004
Bhakta et al.
6751703
June 2004
Chilton
6751722
June 2004
Mirsky et al.
6754117
June 2004
Jeddeloh
6754812
June 2004
Abdallah et al.
6756661
June 2004
Tsuneda et al.
6760833
July 2004
Dowling
6771538
August 2004
Shukuri et al.
6772261
August 2004
D'Antonio et al.
6775747
August 2004
Venkatraman
6789173
September 2004
Tanaka et al.
6792059
September 2004
Yuan et al.
6792496
September 2004
Aboulenein et al.
6793408
September 2004
Levy et al.
6795899
September 2004
Dodd et al.
6799246
September 2004
Wise et al.
6799268
September 2004
Boggs et al.
6804760
October 2004
Wiliams
6804764
October 2004
LaBerge et al.
6807630
October 2004
Lay et al.
6811320
November 2004
Abbott
6816931
November 2004
Shih
6816947
November 2004
Huffman
6820181
November 2004
Jeddeloh et al.
6821029
November 2004
Grung et al.
6823023
November 2004
Hannah
6845409
January 2005
Talagala et al.
6889304
May 2005
Perego et al.
6910109
June 2005
Holman et al.
6950956
September 2005
Zerbe et al.
6956996
October 2005
Gordon et al.
6961259
November 2005
Lee et al.
6961834
November 2005
Weber
6982892
January 2006
Lee et al.
7000062
February 2006
Perego et al.
7016213
March 2006
Reeves et al.
7016606
March 2006
Cai et al.
7024547
April 2006
Kartoz
7035212
April 2006
Mittal et al.
7047351
May 2006
Jeddeloh
7062595
June 2006
Lindsay et al.
7102907
September 2006
Lee et al.
7106611
September 2006
Lee et al.
7120723
October 2006
Jeddeloh
7120727
October 2006
Lee et al.
7174409
February 2007
Jeddeloh
7181584
February 2007
LaBerge
7200024
April 2007
Taylor
7206887
April 2007
Jeddeloh
7222210
May 2007
Jeddeloh
7222213
May 2007
James
7234070
June 2007
James
7242213
July 2007
Pax et al.
7245145
July 2007
Pax et al.
7254331
August 2007
Murphy
7266633
September 2007
James
7282947
October 2007
Pax et al.
7289347
October 2007
Taylor
2001/0023474
September 2001
Kyozuka et al.
2001/0034839
October 2001
Karjoth et al.
2001/0039612
November 2001
Lee
2002/0038412
March 2002
Nizar et al.
2002/0112119
August 2002
Halbert et al.
2002/0116588
August 2002
Beckert et al.
2002/0144064
October 2002
Fanning
2003/0005223
January 2003
Coulson et al.
2003/0043158
March 2003
Wasserman et al.
2003/0043426
March 2003
Baker et al.
2003/0093630
May 2003
Richard et al.
2003/0149809
August 2003
Jensen et al.
2003/0163649
August 2003
Kapur et al.
2003/0177320
September 2003
Sah et al.
2003/0193927
October 2003
Hronik
2003/0217223
November 2003
Nino, Jr. et al.
2003/0227798
December 2003
Pax
2003/0229734
December 2003
Chang et al.
2003/0229770
December 2003
Jeddeloh
2004/0022094
February 2004
Radhakrishnan et al.
2004/0044833
March 2004
Ryan
2004/0123088
June 2004
Poisner et al.
2004/0126115
July 2004
Levy et al.
2004/0128421
July 2004
Forbes
2004/0144994
July 2004
Lee et al.
2004/0148482
July 2004
Grundy et al.
2004/0230718
November 2004
Polzin et al.
2004/0236885
November 2004
Fredriksson et al.
2005/0044304
February 2005
James
2005/0044327
February 2005
Howard et al.
2005/0071542
March 2005
Weber et al.
2005/0162882
July 2005
Reeves et al.
2005/0210216
September 2005
Jobs et al.
2005/0228939
October 2005
Janzen
2005/0268060
December 2005
Cronin et al.
2006/0047891
March 2006
James et al.
2006/0200598
September 2006
Janzen
2006/0204247
September 2006
Murphy
2006/0206667
September 2006
Ryan
2006/0206742
September 2006
James
2006/0271720
November 2006
James et al.
2007/0025133
February 2007
Taylor
2007/0035980
February 2007
Taylor
2007/0143553
June 2007
LaBerge
2008/0036492
February 2008
Pax et al.
2008/0162861
July 2008
Jobs et al.
Foreign Patent Documents
0849685
Jun., 1998
EP
2001265539
Sep., 2001
JP
WO 93/19422
Sep., 1993
WO
WO 02/27499
Apr., 2002
WO
Other References
Mike Clugston, Definition of `cache`, 2004, www.xreferplus.com. cited by other
.
Depari, A. et al, "USB Sensor Network for Industrial Applications", Instrumentation and Measurement Technology Conference, 2004, IMTC 04, Proceedings of the 21.sup.st IEEE, vol. 2, May 2004, Abstract only. One page. cited by other
.
"Free On-Line Dictionary of Computing" entry Flash Erasable Programmable Read-Only Memory, online May 17, 2004 [http://foldoc.doc.ic.ac.uk/foldoc/foldoc.cgi?flash+memory]. cited by other
.
Gillett, R. et al., "Using The Memory Channel Network", Micro IEEE, vol. 17, Issue 1, Jan.-Feb. 1997, Abstract Only, Two pages. cited by other
.
Gommans, L. et al., "Applications drive secure lightpath creation across heterogeneous domains", Communications Magazine, IEEE, vol. 44, Issue 3, Mar. 2006. Abstract only. cited by other
.
Hsieh, M. et al., "On The Design of A Local Switch For Heterogenous Multi-Subsystem Interconnect", TENCON '93, Proceedings, Computer, Communications, Control and Power Engineering, 1993, IEEE Region 10 Conference, vol. 3, Oct. 1993. Abstract Only.
One page. cited by other
.
Intel, "Flash Memory PCT Add-In Card for Embedded Systems", Application Note AP-758, Sep. 1997, pp. i-13. cited by other
.
Intel, "Intel 840 Chipset: 82840 Memory Controller Hub (MCH)", Datasheet, www.intel.com/design/chipsets/datashts/298020.htm, Oct. 1999, pp. 1-178. cited by other
.
Micron Technology, Inc., "Synchronous DRAM Module 512MB/1GB (x72, ECC) 168-PIN Registered FBGA SDRAM DIMM", Micron Technology, Inc., 2002, pp. 1-23. cited by other
.
Shanley, T. et al., "PCI System Architecture", Third Edition, Mindshare, Inc., 1995, pp. 24-25. cited by other.
Primary Examiner: Knoll; Clifford H
Attorney, Agent or Firm: Dorsey & Whitney LLP
Parent Case Text
CROSS REFERENCE TO RELATED APPLICATIONS
This application is a continuation of U.S. patent application No.
11/715,517, filed Mar. 7, 2007, which is a continuation of U.S. patent
application No. 11/399,873, filed Apr. 7, 2006, issued Apr. 17, 2007 as
U.S. Pat. No. 7,206,887, which is a divisional of U.S. patent application
No. 10/810,229, filed Mar. 25, 2004, issued Oct. 10, 2006 as U.S. Pat.
No. 7,120,723.
Claims
What is claimed is:
1. A memory system, comprising: a memory hub controller; a memory bus coupled to the memory hub controller; a plurality of memory modules each having a memory hub coupled
to the memory bus, each of the memory modules further having a plurality of memory devices coupled to the respective memory hub and the memory hub configured to issue memory requests to only the memory devices coupled thereto in response to memory
requests from the memory hub controller; and at least one expansion module each having a processor unit coupled to the memory bus, each of the at least one expansion module further having a plurality of memory devices coupled to the respective processor
unit, each processor unit configured to provide memory requests to the memory devices of the respective expansion module and at least one of the memory devices of one of the memory modules and receive memory responses from the memory devices to which the
memory requests are provided.
2. The memory system of claim 1 wherein the processor unit of one of the at least one expansion module is a graphics controller.
3. The memory system of claim 1 wherein the processor unit of each of the at least one expansion module is for a respective purpose different than the respective purpose of the processor unit of another of the at least one expansion module.
4. The memory system of claim 3 wherein the processor unit of one of the at least one expansion module is a graphics controller, and wherein the processor unit of another of the at least one expansion module is an input/output processor.
5. The memory system of claim 1 wherein memory requests from each of the at least one expansion module are primarily serviced by a respective one of the memory modules and the memory devices of the respective expansion module.
6. An electronic device, comprising: a processor; a memory hub controller coupled to the processor; a plurality of memory modules each having a memory hub coupled to the memory hub controller, each of the memory modules having a plurality of
memory devices coupled to the respective memory hub and configured to issue local memory requests in response to memory requests issued by the memory hub controller to only the memory devices coupled to the memory hub and provide memory responses to the
memory hub controller; an expansion module having a processor unit coupled to at least one of the memory modules, the expansion module further having a plurality of memory devices coupled to the processor unit and configured to provide memory responses
to the processor unit responsive to receiving memory requests from the processor unit, the processor unit configured to provide memory requests to and receive memory responses from at least one of the memory devices of the at least one of the memory
modules.
7. The electronic device of claim 6, further comprising first and second devices, the first device coupled to receive data from the memory hub controller but not the expansion module, the second device coupled to receive data from the expansion
module but not the memory hub controller.
8. The electronic device of claim 6 wherein the expansion module comprises a plurality of expansion modules each having a processor unit and a plurality of memory devices coupled to the respective processor unit, each expansion module coupled
to at least a respective one of the memory modules, memory requests from each processor unit being serviced by the respective memory devices of the respective expansion module and at least one memory device of the respective one of the memory modules.
9. The electronic device of claim 6 wherein at least one memory device of the at least one of the memory modules coupled to the expansion module primarily services memory requests from the expansion module while memory devices of at least one
other memory module primarily service memory requests from the memory hub controller.
10. The electronic device of claim 6 wherein the memory hub controller and the expansion module each accesses the memory modules independently.
11. A method of accessing data stored in a plurality of memory devices in a plurality of memory modules coupled to each other and to a memory controller, comprising: issuing a memory request from the memory controller to at least one of the
memory modules; receiving the memory request from the memory controller by a memory hub for the at least one of the memory modules and issuing a local memory request to memory devices coupled thereto, the memory hub limited to issuing local memory
requests to the coupled memory devices; issuing a memory request from a first expansion module coupled to a first of the memory modules to the first of the memory modules; providing a memory response from the at least one of the memory modules to the
memory controller responsive to the memory request from the memory controller; and providing a memory response from the first of the memory modules to the first expansion module responsive to the memory request from the first expansion module.
12. The method of claim 11 wherein the at least one of the memory modules providing the memory response to the memory controller is not the first of the memory modules.
13. The method of claim 11 wherein the at least one of the memory modules providing the memory response to the memory controller includes the first of the memory modules.
14. The method of claim 11, further comprising: issuing a memory request from a second expansion module coupled to a second of the memory modules to the second of the memory modules; and providing a memory response from the second of the
memory modules to the second expansion module responsive to the memory request from the second expansion module.
15. The method of claim 14 wherein at least one of the memory devices in the first of the memory modules primarily services memory requests from the first expansion module, and wherein at least one of the memory devices in the second of the
memory modules primarily services memory requests from the second expansion module.
16. A method of receiving memory requests and providing memory responses to and from memory devices in a module in a memory system, comprising: receiving a memory request on a system bus; in response to receiving the memory request, issuing a
local memory request from a memory hub coupled to memory devices of a module, the memory hub limited to issuing local memory requests to the memory devices of the module; locally accessing data on a local bus responsive to the local memory request and
providing a first memory response; locally generating and processing a memory request independently of receiving the memory request on the system bus; locally accessing data on the local bus responsive to the locally generated memory request and
providing a second memory response; and coupling the first and second memory responses from the local bus to the system bus.
17. The method of claim 16 wherein locally generating and processing a memory request comprises generating and processing a memory request for graphics data.
18. The method of claim 16 wherein locally generating and processing a memory request comprises generating command packets that include identification data for identify the memory module.
19. The method of claim 16 further comprising coupling input and output devices to the memory module through a high-speed input/output bus.
20. The method of claim 16 wherein the system bus comprises a downstream bus and an upstream bus.
21. A method of providing memory requests to a plurality of memory modules of a memory system, comprising: coupling a subprocessor independently from a memory system controller of the memory system to a memory hub of at least one of the memory
modules, the memory hub having a plurality of memory devices and configured to issue local memory requests to only the plurality of memory devices; providing memory requests generated by the subprocessor to the memory hub of the at least one of the
memory modules; and receiving memory responses from the plurality of memory devices and from the memory hub of the at least one of the memory modules independent of the memory system controller.
22. The method of claim 21, further comprising receiving memory requests from the system controller and providing memory responses from the plurality of memory devices responsive to receiving the memory requests from the memory system
controller.
23. The method of claim 21 wherein the memory requests comprise command packets that include information identifying the subprocessor or the memory hub of the at least one of the modules to which the command packets are directed towards.
24. The method of claim 21 wherein the subprocessor is directly coupled to input and output devices through a high-speed input/output bus.
25. The method of claim 21 wherein receiving memory responses from the plurality of memory devices and the memory hub of the at least one of the memory modules comprises receiving graphics data. Description
TECHNICAL FIELD
The present invention relates generally to a memory system for a processor-based computing system, and more particularly, to a hub-based memory system providing expansion capabilities for computer components.
BACKGROUND OF THE INVENTION
Computer systems use memory devices, such as dynamic random access memory ("DRAM") devices, to store data that are accessed by a processor. These memory devices are normally used as system memory in a computer system. In a typical computer
system, the processor communicates with the system memory through a processor bus and a memory controller. The memory devices of the system memory, typically arranged in memory modules having multiple memory devices, are coupled through a memory bus to
the memory controller. The processor issues a memory request, which includes a memory command, such as a read command, and an address designating the location from which data or instructions are to be read. The memory controller uses the command and
address to generate appropriate command signals as well as row and column addresses, which are applied to the system memory through the memory bus. In response to the commands and addresses, data are transferred between the system memory and the
processor. The memory controller is often part of a system controller, which also includes bus bridge circuitry for coupling the processor bus to an expansion bus, such as a PCI bus.
In memory systems, high data bandwidth is desirable. Generally, bandwidth limitations are not related to the memory controllers since the memory controllers sequence data to and from the system memory as fast as the memory devices allow. One
approach that has been taken to increase bandwidth is to increase the speed of the memory data bus coupling the memory controller to the memory devices. Thus, the same amount of information can be moved over the memory data bus in less time. However,
despite increasing memory data bus speeds, a corresponding increase in bandwidth does not result. One reason for the non-linear relationship between data bus speed and bandwidth is the hardware limitations within the memory devices themselves. That is,
the memory controller has to schedule all memory commands to the memory devices such that the hardware limitations are honored. Although these hardware limitations can be reduced to some degree through the design of the memory device, a compromise must
be made because reducing the hardware limitations typically adds cost, power, and/or size to the memory devices, all of which are undesirable alternatives. Thus, given these constraints, although it is easy for memory devices to move "well-behaved"
traffic at ever increasing rates, for example, sequel traffic to the same page of a memory device, it is much more difficult for the memory devices to resolve "badly-behaved traffic," such as bouncing between different pages or banks of the memory
device. As a result, the increase in memory data bus bandwidth does not yield a corresponding increase in information bandwidth.
In addition to the limited bandwidth between processors and memory devices, the performance of computer systems is also limited by latency problems that increase the time required to read data from system memory devices. More specifically, when
a memory device read command is coupled to a system memory device, such as a synchronous DRAM ("SDRAM") device, the read data are output from the SDRAM device only after a delay of several clock periods. Therefore, although SDRAM devices can
synchronously output burst data at a high data rate, the delay in initially providing the data can significantly slow the operating speed of a computer system using such SDRAM devices. Increasing the memory data bus speed can be used to help alleviate
the latency issue. However, as with bandwidth, the increase in memory data bus speeds do not yield a linear reduction of latency, for essentially the same reasons previously discussed.
Although increasing memory data bus speed has, to some degree, been successful in increasing bandwidth and reducing latency, other issues are raised by this approach. For example, as the speed of the memory data bus increases, loading on the
memory bus needs to be decreased in order to maintain signal integrity since traditionally, there has only been wire between the memory controller and the memory slots into which the memory modules are plugged. Several approaches have been taken to
address the memory bus loading issue. For example, reducing the number of memory slots to limit the number of memory modules that contribute to the loading of the memory bus, adding buffer circuits on a memory module in order to provide sufficient
fanout of control signals to the memory devices on the memory module, and providing multiple memory device interfaces on the memory module since there are too few memory module connectors on a single memory device interface. The effectiveness of these
conventional approaches are, however, limited. A reason why these techniques were used in the past is that it was cost-effective to do so. However, when only one memory module can be plugged in per interface, it becomes too costly to add a separate
memory interface for each memory slot. In other words, it pushes the system controllers package out of the commodity range and into the boutique range, thereby, greatly adding cost.
One recent approach that allows for increased memory data bus speed in a cost effective manner is the use of multiple memory devices coupled to the processor through a memory hub. A computer system 100 shown in FIG. 1 uses a memory hub
architecture. The computer system 100 includes a processor 104 for performing various computing functions, such as executing specific software to perform specific calculations or tasks. The processor 104 includes a processor bus 106 that normally
includes an address bus, a control bus, and a data bus. The processor bus 106 is typically coupled to cache memory 108, which, is typically static random access memory ("SRAM"). Finally, the processor bus 106 is coupled to a system controller 110,
which is also sometimes referred to as a bus bridge. The system controller 110 serves as a communications path to the processor 104 for a variety of other components. For example, as shown in FIG. 1, the system controller 110 includes a graphics port
that is typically coupled to a graphics controller 112, which is, in turn, coupled to a video terminal 114. The system controller 110 is also coupled to one or more input devices 118, such as a keyboard or a mouse, to allow an operator to interface with
the computer system 100. Typically, the computer system 100 also includes one or more output devices 120, such as a printer, coupled to the processor 104 through the system controller 110. One or more data storage devices 124 are also typically coupled
to the processor 104 through the system controller 110 to allow the processor 104 to store data or retrieve data from internal or external storage media (not shown). Examples of typical storage devices 124 include hard and floppy disks, tape cassettes,
and compact disk read-only memories (CD-ROMs).
The system controller 110 includes a memory hub controller 128 that is coupled to the processor 104. The system controller 110 is further coupled over a high speed bi-directional or unidirectional system controller/hub interface 134 to several
memory modules 130a-n. As shown in FIG. 1, the controller/hub interface 134 includes a downstream bus 154 and an upstream bus 156 which are used to couple data, address, and/or control signals away from or toward, respectively, the memory hub controller
128. Typically, the memory modules 130a-n are coupled in a point-to-point or daisy chain architecture such that the memory modules 130a-n are connected one to another in series. Thus, the system controller 110 is coupled to a first memory module 130a,
with the first memory module 130a connected to a second memory module 130b, and the second memory module 130b coupled to a third memory module 130c, and so on in a daisy chain fashion. Each memory module 130a-n includes a memory hub 140 that is coupled
to the system controller/hub interface 134, and is further coupled a number of memory devices 148 through command, address and data buses, collectively shown as local memory bus 150. The memory hub 140 efficiently routes memory requests and responses
between the memory hub controller 128 and the memory devices 148.
The memory devices 148 on the memory modules 130a-n are typically capable of operating at high clock frequencies in order to facilitate the relatively high speed operation of the overall memory system. Consequently, computer systems employing
this architecture can also use the high-speed system controller/hub interface 134 to complement the high clock speeds of the memory devices 148. Additionally, with a memory hub based system, signal integrity can be maintained on the system
controller/hub interface 134 since the signals are typically transmitted through multiple memory hubs 140 to and from the memory hub controller 128. Moreover, this architecture also provides for easy expansion of the system memory without concern for
degradation in signal quality as more memory modules are added, such as occurs in conventional memory bus architectures.
Although the memory hub architecture shown in FIG. 1 provides improved memory system performance, the advantages may not directly benefit the various components of the computer system 100. As previously described, the components, such as the
graphics controller 112, the input and output devices 118, 120, and the data storage 124 are coupled to the system controller 110. It is through the system controller 110 that the components 112, 118, 120, 124 access the memory modules 130a-n. As a
result of the memory requests necessarily being coupled through the system controller 110, a "bottleneck" can often result since the system controller 110 can handle only a finite number of memory requests, and corresponding memory responses from the
memory modules 130a-n, at a given time. The graphics port through which the graphics controller 112 is coupled to the system controller 110 provides some relief to the bottleneck issue, since the graphics port typically provides direct memory access
(DMA) to the memory modules 130a-n, as well known in the art. That is, the graphics controller 112 is able to access the memory modules 130a-n directly, with limited intervention by the system controller 110.
As well known, arbitration schemes are implemented by the system controller 110 in order to prioritize memory requests it receives from the various components 112, 118, 120, 124, as well as memory requests received from the processor 104. The
arbitration schemes that are implemented attempt to provide efficient memory access to the various components 112, 118, 120, 124, and processor 104 in order to maximize processing capabilities. Some memory requests are given priority over others
regardless of the order in which the requests are received by the system controller 110, for example, the processor 104 is often given highest priority to access the memory modules 130a-n to avoid the situation where processing is halted while the
processor 104 is waiting for a memory request to be serviced. As sophisticated as arbitration techniques have become, it is still unlikely that bottlenecks at the system controller 110 can be completely avoided. Even where a component is given direct
memory access to the memory modules 130a-n, such as the graphics controller 112, it is nevertheless subject to the arbitration routine that is implemented by the system controller 110, and consequently, the component does not have unlimited access
privileges to the memory modules 130a-n. It is by the nature of the architecture used in the computer system 100, namely, providing access to the memory modules 130a-n through the single point of the system controller 110, that makes bottlenecks at the
system controller 110 inevitable. Therefore, there is a need for an alternative system and method for providing components of a processing system, such as a computer system, access to memory resources.
SUMMARY OF THE INVENTION
A system memory in one aspect of the invention includes a memory hub controller, a memory module accessible by the memory hub controller, and an expansion module coupled to the memory module having a processor circuit also having access to the
memory module. The memory hub controller provides memory requests to access memory devices, and the memory module includes a plurality of memory devices coupled to a memory hub. The memory hub receives the memory requests, accesses the memory devices
according to the memory requests, and provides memory responses in response to the memory requests. The processor circuit of the expansion module provides memory requests to the memory hub of the memory module to access the memory devices, and processes
data returned in the memory responses from the memory hub. The memory hub controller is coupled to the memory hub through a first portion of a memory bus on which the memory requests and the memory responses are coupled. A second portion of the memory
bus couples the memory hub to the processor circuit and is used to couple memory requests from the processor circuit and memory responses provided by the memory hub to the processor circuit.
BRIEF DESCRIPTION OF THE DRAWINGS
FIG. 1 is a partial block diagram of a conventional processor-based computing system having a memory hub-based system memory.
FIG. 2 is a partial block diagram of a processor-based computing system having a memory hub-based memory system according to an embodiment of the present invention providing peripheral component expansion capabilities.
FIG. 3 is a partial block diagram of a memory hub of the hub-based memory system of FIG. 2.
FIG. 4 is a partial block diagram of a processor-based computing system having a memory hub-based memory system according to an alternative embodiment of the present invention providing peripheral component expansion capabilities.
DETAILED DESCRIPTION OF PREFERRED EMBODIMENTS
FIG. 2 illustrates a processor based computing system 200 according to an embodiment of the present invention. The system 200 includes many of the same functional blocks as previously described with reference to FIG. 1. As such, the same
reference numbers will be used in FIG. 2 as in FIG. 1 to refer to the same functional blocks where appropriate. The system 200 includes a processor 104 coupled to a system controller 110 through a processor bus 106. As in FIG. 1, the processor performs
various computing functions, for example, executing software to perform specific calculations or tasks, and the processor bus 106 typically includes an address bus, a control bus, and a data bus. A cache memory 108 is also coupled to the processor bus
106 to provide the processor 104 with temporary storage of frequently used data and instructions. As previously discussed with respect to FIG. 1, the system controller 110 serves as a communications path to the processor 104 for a variety of other
components. Typically, this includes one or more input devices 118, such as a keyboard or a mouse, to allow an operator to interface with the system 200, one or more output devices 120, such as a printer, and one or more data storage devices 124 to
allow the processor 104 to store data or retrieve data from internal or external storage media (not shown).
As shown in FIG. 2, the system controller 110 includes a memory hub controller 128 to which several memory modules 130a-c are coupled over a high speed bi-directional or unidirectional system controller/hub interface 134. The controller/hub
interface 134 includes a downstream bus 154 and an upstream bus 156 which are used to couple data, address, and/or control signals away from or toward, respectively, the memory hub controller 128. As shown in FIG. 2, the memory modules 130a-c are
coupled in a point-to-point architecture such that the memory modules 130a-c are connected one to another in series. Each memory module 130a-c in the system 200 includes a memory hub 240 that is coupled to the system controller/hub interface 134, and is
further coupled to a number of memory devices 148 through command, address and data buses, collectively shown as bus 150. As previously mentioned, the memory hub 240 efficiently routes and arbitrates memory requests and responses between the memory hub
controller 128 and the memory devices 148. As will be explained in further detail below, the memory hub 240 can receive memory requests and provide memory responses in both downstream and upstream directions over the downstream and upstream buses 154,
156, respectively.
In contrast to the computer system 100 of FIG. 1, the system 200 includes a component expansion module 230 coupled to the controller/hub interface 134. As shown in FIG. 2, the component expansion module 230 includes a graphics controller 234
coupled to local memory devices 248 over a local graphics/memory bus 250. The graphics controller 234, the local graphics/memory bus 250, and the local memory devices 248 can be of conventional design and operation, as well known in the art. The
graphics/memory bus 250 includes command, data, and address buses as well known in the art. A video bus 260 can be used for coupling video data from the graphics controller 234 to a video terminal (not shown) as known in the art. It will be appreciated
that the component expansion module 230 replaces the graphics controller 112 of the computer system 100. That is, the component expansion module 230 can provide the computer graphics capabilities and functionality of the graphics controller 112.
Although the component expansion module 230 is shown in FIG. 2 as having local memory devices 248, access to data stored in the system memory, such as memory modules 130a-c, is often required for processing by the graphics controller 234. For
example, the memory provided by the local memory devices 248 may not be sufficient to store all of the graphics data necessary for rendering a scene. As a result, the bulk of the graphics data is typically loaded into system memory, with the graphics
controller 234 retrieving the portion of graphics data necessary for rendering the current scene from the system memory. Additionally, since access to the local memory devices 248 is typically limited to the graphics controller 234, data that has been
first processed elsewhere, for example, by the processor 104, must be stored to a location in the system memory for retrieval by the graphics controller 234 before being stored in the local memory devices 248 for further processing. Thus, access to the
memory modules 130a-c by the component expansion module 230 is often necessary.
The arrangement of the system 200 allows for access to the memory modules 130a-c by the component expansion module 230 without intervention by the system controller 110. As previously discussed, the memory hubs 240 can receive memory requests
and provide memory responses in both the downstream and upstream directions. By adopting a consistent communication protocol with the memory hubs 240 of the memory modules 130a-c, communication with the memory hubs 240 of the memory modules 130a-c can
be performed directly by the component expansion module 230, thereby eliminating the need for intervention by the system controller 110. As a result, access to the memory modules 130a-c is not limited to going through the system controller 110, but the
component expansion module 230 can access the memory modules 130a-c directly. In contrast, the graphics controller 112 in the computer system 100 (FIG. 1) is typically coupled to the system controller 110 through an advanced graphics port, and although
the graphics controller 112 has DMA access to the memory, it is still nevertheless subject to the memory request and memory response loading issues of the system controller 110. In the system 200, however, the graphics controller 234 is not subject to
the loading issues of the system controller 110.
Many suitable communication protocols are known in the art, including the use of command packets that include appropriate information for making memory requests to particular memory modules 130a-c in the system 200 and providing memory responses
in return. For example, command packets can include information such as identification data for uniquely identifying the particular memory request, address information for identifying a particular memory module 130a-c to which the memory request is
directed, and memory device command information, including memory addresses, command type, and where a write operation is requested, data can be included as well. Other protocols can be used as well, and it will be appreciated by those ordinarily
skilled in the art that the present invention is not limited by the particular protocol implemented.
Additionally, the arrangement of the system 200 reduces the memory request and response load on the system controller 110 since it is relieved from handling the memory requests from a requesting entity, namely the graphics controller 112 (FIG.
1). For these reasons, the likelihood that a memory request and response bottleneck occurring at the system controller 110 is also reduced. Moreover, by coupling the component expansion module 230 to the controller/hub interface 134 rather than to the
system controller 110, the number of buses in the system 200 can be reduced.
FIG. 3 illustrates a portion of the memory hub 240 (FIG. 2). The memory hub 240 includes four link interfaces 302, 304, 306, 308 coupled to a cross bar switch 310 by respective local link buses 312, 314, 316, 318. Memory controllers 324a, 324b
are further coupled to the cross bar switch 310 through respective local memory controller buses 326a, 326b. The cross bar switch 310, which may be of a conventional or hereinafter developed design, can couple any of the link interfaces 302, 304, 306,
308 to each other. The link interfaces 302, 304, 306, 308 may be either unidirectional or duplex interfaces, and the nature of the memory accesses coupled to or from the link interfaces 302, 304, 306, 308 may vary as desired, including communication
protocols having conventional memory address, control and data signals, shared address and control signals and packetized memory access signals. As shown in FIG. 3, the link interfaces 302 and 304 are coupled to the downstream bus 154 and the link
interfaces 306 and 308 are coupled to the upstream bus 156.
The cross bar switch 310 can also couple any of the link interfaces 302, 304, 306, 308 to either or both of the memory controllers 324a, 324b, each of which is coupled to a plurality of memory devices 148 (not shown in FIG. 3) over respective
local memory buses 150 (FIG. 2). The memory controllers 324a, 324b may be conventional memory controllers or some hereinafter developed design for a memory controller. The specific structure and operation of the memory controllers 324a, 324b will, of
course, depend on the nature of the memory devices 148 used in the memory modules 130a-c. The cross bar switch 310 couples the link interfaces 302, 304, 306, 308 to the memory controllers 324a, 324b to allow any of a plurality of memory access devices to
write data to or read data from the memory devices 148 coupled to the memory controllers 324a, 324b. The cross bar switch 310 further couples the link interfaces 302, 304, 306, 308 to the memory controllers 324a, 324b to allow any data to be transferred
to or from the memory devices 148 coupled to the memory controllers 324a-324b from or to, respectively, other memory modules 130a-c containing a memory hub 240. Thus, as previously discussed, the memory hub 240 is capable of receiving memory requests
and providing memory responses in both downstream and upstream directions over the downstream and upstream buses 154, 156.
It will be appreciated by those ordinarily skilled in the art that FIG. 3 illustrates merely a portion of the memory hub 240, and that the memory hub 240 will generally include components in addition to those shown in FIG. 3. For example, a
cache memory for each of the memory controllers 324a, 324b can be included for storing recently or frequently accessed data retrieved from or stored in the memory devices 148. Additionally, a write buffer can also be included for accumulating write
addresses and data directed to the memory devices 148 serviced by a respective one of the memory controllers 324a, 324b if the memory devices 148 are busy servicing a read memory request or other read requests are pending. Such components are
conventional and known in the art. These components have been omitted from FIG. 3 in the interest of brevity and clarity. It will further be appreciated by those ordinarily skilled in the art that in some applications, components shown in FIG. 3 may be
omitted. For example, although the memory hub 240 shown in FIG. 3 includes two memory controllers 324a, 324b the number of memory controllers may vary as desired.
FIG. 4 illustrates a processor-based computing system 400 according to another embodiment of the present invention. The system 400 includes many of the same functional blocks as previously described with reference to FIGS. 1 and 2. As such, the
same reference numbers will be used in FIG. 4 as in FIGS. 1 and 2 to refer to the same functional blocks where appropriate. The system 400 includes a processor 104 coupled to a memory hub controller 428 through a processor bus 106. A cache memory 108
is also coupled to the processor bus 106 to provide the processor 104 with temporary storage of frequently used data and instructions. The memory hub controller 428 is further coupled to a system controller 110, which serves as a communications path to
the processor 104 for a variety of other components. As shown in FIG. 4, data storage device 124 is coupled to the system controller 110 to allow the processor 104 to store data or retrieve data from internal or external storage media (not shown).
The memory hub controller 428 is coupled over a high speed bi-directional or unidirectional system controller/hub interface 134 to several memory modules 130a-c. The controller/hub interface 134 includes a downstream bus 154 and an upstream bus
156 which are used to couple data, address, and/or control signals away from or toward, respectively, the memory hub controller 428. Each memory module 130a-c in the system 400 includes a memory hub 240 that is coupled to the system controller/hub
interface 134, and which is further coupled to a number of memory devices 148 through command, address and data buses, collectively shown as bus 150. The memory hub 240 efficiently routes memory requests and responses between the memory hub controller
128 and the memory devices 148. As with the memory hub 240 shown in FIG. 2, memory requests and memory responses can be provided in both downstream and upstream directions over the downstream and upstream buses 154, 156, respectively, by the memory hub
240.
Coupled in series with the memory modules 130a-c over the downstream and upstream buses 154, 156 are component expansion modules 230 and 430. The component expansion module 230, as previously described with reference to FIG. 2, includes a
graphics controller 234 coupled to local memory devices 248 over a local graphics/memory bus 250. The component expansion module 230 provides video data over a video bus 260 to a video terminal (not shown), as known in the art. In contrast to the
system 200 of FIG. 2, the system 400 further includes the component expansion module 430. The component expansion module 430 includes an input/output (IO) processor 434 coupled to local memory devices 448 over a local memory device bus 450. Although
the-component expansion module 430 includes local memory devices 448, the IO processor 434 has access to system memory, for example, memory modules 130a-c, as well.
Unlike the systems 100 and 200, where the input and output devices 118, 120 are coupled to the system controller 110, input and output devices (not shown in FIG. 4) can be coupled to the system 400 through the component expansion module 430 and a
high-speed IO bus 460. By including the component expansion module 430, memory request and response loading on the system controller 410 can be reduced compared to the configuration of systems 100 and 200. Using a consistent communication protocol with
the memory hub 240 over the downstream and upstream buses 154, 156, the memory hub controller 428, the IO processor 434, and the graphics controller 234, can each access the memory modules 130a-c independently. As shown in FIG. 4, the memory modules
130a-c and the component expansion modules 230, 430 are series coupled in an arrangement that takes advantage of the point-to-point architecture provided by the downstream and upstream buses 154, 156. The memory hub controller 428, the IO processor 434
and the graphics controller 234 each have a respective memory module 130a-c which can be used primarily for servicing memory requests by the respective component. That is, the memory module 130a can be used primarily by the memory hub controller 428 for
servicing memory requests from the processor 104 and the system controller 410, the memory module 130b can be used primarily by the component expansion module 430 for servicing memory requests from the IO processor 434, and the memory module 130c can be
used primarily by the component expansion module 230 for servicing memory requests from the graphics controller 234. Thus, although the memory hub controller 428, the component expansion module 430, and the component expansion module 230 have access to
any of the memory modules 130a-c, memory requests from each of the requesting entities can be primarily serviced by a respective memory module 130a-c. As a result, the memory request and response loading that is conventionally handled by the system
controller 110 is distributed throughout the memory system, thereby reducing the likelihood of memory requests and response being bottlenecked through one access point.
It will be appreciated by those ordinarily skilled in the art that the embodiments shown in FIGS. 2 and 4 have been provided by way of example, and are not intended to limit the scope of the present invention. Modifications can be made to the
previously described embodiments without departing from the scope of the present invention. For example, the system 400 has been described as providing each of the requesting components, the memory hub controller 428, the component expansion module 430,
and the component expansion module 230, with a respective memory module 130a-c for primarily servicing memory requests. However, only portions of the memory available on a memory module 130a-c can be used for one requesting entity, with the remaining
memory of the same memory module 130a-c allocated for primarily servicing the memory requests of another requesting entity. That is, the allocation of memory is not limited to a per module basis, but can be allocated as desired. Additionally, the order
in which the memory modules 130a-c and the requesting entities are coupled, namely the memory hub controller 428, the component expansion module 430, and the component expansion module 230, can be changed and remain within the scope of the present
invention. Although the order of the requesting entities can be arranged advantageously with respect to the memory modules 130a-c, as previously described with respect to having a primary memory for servicing memory requests, the present invention is
not limited to any specific order of coupling of the memory modules and requesting entities.
From the foregoing it will be appreciated that, although specific embodiments of the invention have been described herein for purposes of illustration, various modifications may be made without deviating from the spirit and scope of the
invention. Accordingly, the invention is not limited except as by the appended claims.
* * * * *