±³¾°

½üÄêÀ´£¬Ëæ×ÅÈ˹¤ÖÇÄܼ¼ÊõµÄѸÃÍ·¢Õ¹£¬»úÆ÷ѧϰģÐ͵ĹæÄ£³ÊÏÖ³ö±¬·¢Ê½Ôö³¤Ì¬ÊÆ¡£ÓÈÆäÔÚÉî¶ÈѧϰÁìÓò£¬Ä£Ð͹æÄ£µÄÏÔÖøÌá¸ß³ÉΪÍƶ¯Öî¶à¼¼ÊõÍ»ÆÆÓëÐÔÄÜÌáÉýµÄ¹Ø¼üÒªËØ¡£µ±Ä£ÐͲÎÊýÊýÁ¿´ïµ½ÍòÒÚ¼¶±ðʱ£¬ÊÜÏÞÓÚµ¥Ì¨É豸µÄÎïÀí×ÊÔ´¶øÎÞ·¨ÊµÏÖµ¥»úѵÁ·¡£Îª´Ë£¬·Ö²¼Ê½ÑµÁ·¼¼ÊõÓ¦Ô˶øÉú£¬²¢Ñ¸ËÙ³ÉΪѵÁ·³¬´ó¹æģģÐ͵ĺËÐÄÊֶΡ£·Ö²¼Ê½ÑµÁ·Äܹ»ÏÔÖøÌá¸ßѵÁ·Ð§ÂÊ£¬Í»ÆƵ¥»úÄÚ´æºÍ¼ÆËãÄÜÁ¦µÄÆ¿¾±¡£ÈÚºÏÄ£ÐͲ¢ÐÐÓëÊý¾Ý²¢Ðеȼ¼ÊõµÄ·Ö²¼Ê½»ìºÏ²¢ÐÐѵÁ·²ßÂÔ³ÉΪѵÁ·³¬´ó¹æÄ£´óÓïÑÔÄ£Ð͵ÄÖ÷Òª¼¼Êõ;¾¶£¬¸Ã·½·¨²»½öÄÜÓÐЧÀûÓôó¹æÄ£Ó²¼þ×ÊÔ´£¬»¹ÄÜÔÚÈ·±£ÑµÁ·Îȶ¨ÐÔºÍÄ£ÐÍÐÔÄܵÄÇ°ÌáÏ£¬ÊµÏÖ´óÓïÑÔÄ£Ð͵ĸßЧѵÁ·Ó벿Êð¡£

Ò»¡¢ÍøÂç¹ÊÕÏ

·Ö²¼Ê½ÑµÁ·ÊÇÖ¸ÔÚ¶à¸ö¼ÆËã½ÚµãÉÏЭͬ×÷Òµ£¬¹²Í¬Íê³É»úÆ÷ѧϰģÐ͵ÄѵÁ·¹ý³Ì¡£Í¨¹ý½«ÑµÁ·ÈÎÎñ²ð·ÖÖÁ¶à¸öÉ豸Éϲ¢ÐÐÖ´ÐУ¬²»½öÄܺÏÀí·ÖÅä¼ÆËãºÍ´æ´¢×ÊÔ´£¬Ò²ÄÜÏÔÖøÌáÉýѵÁ·Ëٶȣ¬²¢ÇÒÄܹ»´¦Àí¸ü´ó¹æÄ£µÄÊý¾ÝºÍÄ£ÐÍ¡£

È»¶ø£¬ÑµÁ·¹ý³ÌÖгöÏÖ¹ÊÕϵĿÉÄÜÐÔ»áËæ×ÅѵÁ·¹æÄ£ºÍ³ÖÐøʱ¼äµÄÔö¼Ó¶øÉý¸ß¡£Ò»µ©·¢Éú¹ÊÕÏ£¬½«Ê¹ËùÓвÎÓëµÄÉ豸´¦ÓÚÏÐÖÃ״̬£¬Ö±ÖÁ¹ÊÕÏÉ豸»Ö¸´Õý³££¬µ¼Ö´óÁ¿ËãÁ¦ÎÞ·¨µÃµ½³ä·ÖÀûÓá£À´×ÔMeta¡¢HuggingFaceºÍLAIONµÄÍŶӾù±¨¸æÁËÔÚѵÁ·´óÐÍÄ£ÐÍʱÒòʧ°Üµ¼ÖÂÀûÓÃÂÊÑÏÖØϽµµÄÇé¿ö¡£

¸ù¾ÝMetaÍŶÓѵÁ·Llama3.1µÄ±¨¸æ£¬ÔÚѵÁ·ÆÚ¼äƽ¾ùÿÈý¸öСʱ¾Í»á·¢ÉúÒ»´Î¹ÊÕÏ£¬ÆäÖÐÔ¼8.4%µÄ¹ÊÕÏÊÇÓÉÍøÂçÉ豸¹ÊÕϵ¼ÖµÄ¡£²»Í¬ÓÚÆäËûÓÉÈí¼þÒý·¢µÄ¹ÊÕÏ£¬ÍøÂçÉ豸¹ÊÕÏ´ó¶àΪӲ¼þ¹ÊÕÏ£¬ÀýÈçÍøÂçÏßÀ»òÊÇÍø¿¨¹âÄ£¿éµÈÓ²¼þÉ豸³öÏÖ¹ÊÕÏ¡£Ïà½ÏÓÚÈí¼þ¹ÊÕÏ£¬Ó²¼þ¹ÊÕϵĴ¦ÀíÄѶȴó¡¢»Ö¸´Ê±¼ä³¤£¬¿ÉÄÜÔì³É¸ü´óµÄËãÁ¦Ëðʧ¡£

½»»»»úÉ豸¼äÁ´Â·¹ÊÕÏÄ¿Ç°ÒÑÄܹ»ÊµÏÖºÁÃ뼶´¦ÀíÓëÊÕÁ²£¬µ«¶Ë²àÍøÂçÉ豸¹ÊÕÏÄ¿Ç°ÉÐÎÞ½ÏΪ¸ßЧµÄ´¦Àí¹«º£²Ê´¬¡¤6600¹ÙÍø¡£¶Ë²àÉ豸¹ÊÕÏÖ÷Òª°üÀ¨·þÎñÆ÷Íø¿¨¹ÊÕÏ¡¢·þÎñÆ÷Óë½»»»»úµÄÁ¬½ÓÏßÀÂËð»µÒÔ¼°ÏßÀ¹âÄ£¿é¹ÊÕϵÈ¡£Ëæ׿ÆË㼯Ⱥ¹æÄ£³ÖÐøÀ©´ó£¬¶Ë²àÍøÂçÉ豸¹ÊÕÏÒѳÉΪؽ´ý½â¾öµÄÎÊÌâ¡£

¶þ¡¢¹ÊÕÏ´¦Àí

Ä¿Ç°£¬³ýÈí¼þ¹ÊÕÏÍ⣬Ӳ¼þ¹ÊÕϵĴ¦Àí²ßÂÔͨ³£²»Çø·Ö¾ßÌåµÄ¹ÊÕÏÀàÐÍ£¬·¢ÉúÓ²¼þ¹ÊÕÏʱ£¬Í¨³£²ÉÓÃÏàͬµÄ¹«º£²Ê´¬¡¤6600¹ÙÍø½øÐд¦Àí¡£

¼ì²éµã£¨Checkpoint£©ÊÇÒ»ÖÖ³£¼ûµÄ³Ö¾Ã»¯»úÖÆ£¬ÓÃÓÚ±£´æѵÁ·½ø¶È¡£ÔÚ¹ÊÕÏ·¢Éúºó£¬ÏµÍ³»áÁ¢¼´¶Ô¹ÊÕÏÉ豸½øÐÐÐÞ¸´£¬»òÕß½«¹ÊÕÏÉ豸´Ó¼¯ÈºÖÐÌÞ³ý£¬È»ºó´Ó×î½üµÄ¼ì²éµã¼ÓÔØÄ£ÐͲ¢¼ÌÐøѵÁ·¡£Ä¿Ç°£¬Ðí¶àÓÅ»¯²ßÂÔΧÈƼì²éµã»úÖÆÕ¹¿ª£º

1.EasyCkpt ²ÉÓÃÒì²½»¯¡¢²ã´Î»¯µÄ±£´æ·½Ê½£¬½áºÏÖصþÄ£ÐÍ¿½±´Óë¼ÆËã¡¢ÍøÂç¸ÐÖªµÄÒì²½´æ´¢²ßÂÔ£¬ÊµÏÖÁ˽üºõÁ㿪ÏúµÄÄ£Ðͱ£´æ»úÖÆ£¬²¢±£Ö¤ÁË´óÄ£ÐÍѵÁ·¹ý³ÌÖÐÄ£Ðͱ£´æÓë»Ö¸´µÄ¾«¶ÈÎÞËð¡£

2.Gemini Äܹ»½« Checkpoint ±£´æÔÚ¾ßÓиü´ó¾ÛºÏ´ø¿íµÄ CPU ÄÚ´æÖУ¬²¢Í¨¹ýһϵÁй«º£²Ê´¬¡¤6600¹ÙÍøʵÏÖÁË´óÐÍÄ£ÐÍѵÁ·µÄ¿ìËÙ¹ÊÕϻָ´¡£

¾¡¹ÜÈç´Ë£¬ÈÔÎÞ·¨ÍêÈ«±ÜÃâÒò¹ÊÕ϶¨Î»ºÍÄ£ÐÍѵÁ·ÖØÆôËù´øÀ´µÄËãÁ¦Ëðʧ¡£

ÈßÓà¼ÆË㣨Redundant computation£©¿ÉÒÔ±ÜÃâÖØÐÂÅäÖúÍÖØÐÂÆô¶¯µÄ¿ªÏú£¬ÔÚ½øÐÐÄ£ÐÍѵÁ·Ê±Ê¹Óò»Í¬µÄ½Úµã½øÐÐÈßÓà¼ÆËã¡£µ±Ä³¸ö½Úµã·¢Éú¹ÊÕÏʱ£¬ÆäËû½ÚµãÄܹ»´úÌæ¹ÊÕϽڵã½øÐмÆË㣬µ«ÊÇÕâÑùÒýÈëÁ˹̶¨µÄÄڴ濪ÏúºÍ¼ÆË㿪Ïú£¬½øÒ»²½Ôö¼ÓÁËѵÁ·³É±¾¡£

Èý¡¢¹«º£²Ê´¬¡¤6600ÍøÂçµÄ¶Ë²àÁ´Â·¹ÊÕÏÌÓÉú

¹«º£²Ê´¬¡¤6600ÍøÂçµÄ¶Ë²àÁ´Â·¹ÊÕÏÌÓÉúÊÇÒ»ÖÖ»ùÓÚNCCL£¨NVIDIA Collective Communications Library£©ÊµÏֵĶ˲àÁ´Â·¹ÊÕÏ¿ìËÙ¹æ±Ü¹«º£²Ê´¬¡¤6600¹ÙÍø£¬Äܹ»ÔÚÉϲãѵÁ·¿ò¼ÜÎÞ¸ÐÖªµÄÇé¿öÏ£¬×Ô¶¯¶Ô¶Ë²àÁ´Â·¹ÊÕϽøÐÐʶ±ð²¢´¦Àí£¬ÔÚÄ£ÐÍѵÁ·¹ý³ÌÖз¢Éú¶Ë²àÁ´Â·¹ÊÕÏʱÄܹ»±£³ÖѵÁ·²»ÖжÏ£¬ÔÚ¹ÊÕÏÉ豸Íê³ÉÐÞ¸´ºóѵÁ·ÐÔÄÜ×Ô¶¯»Ö¸´¡£

¶Ë²à¹ÊÕÏÌÓÉú¹«º£²Ê´¬¡¤6600¹ÙÍøͨ¹ýÔÚNCCL¼ÓÈë¹ÊÕÏʶ±ðÒÔ¼°¹ÊÕϹæ±Ü»úÖÆ£¬²¢ÔÚ´´½¨Êý¾ÝÁ´Â·Ê±Í¬²½´´½¨±¸·ÝÁ´Â·À´´¦ÀíÓÉÓÚÓ²¼þ¹ÊÕϵ¼Öµ¥ÌõÁ´Â·ÎÞ·¨Í¨ÐŵÄÎÊÌâ¡£

±¸·ÝÁ´Â·ÔÚÖ÷Á´Â·×´Ì¬Õý³£Ê±²»»á½øÐÐÊý¾Ý´«Ê䣬¶Ô´«ÊäЧÂʲ»²úÉúÈκÎÓ°Ïì¡£¶øÔÚ³öÏֶ˲àÁ´Â·¹ÊÕÏÖ®ºó£¬Í¨¹ýһϵÁÐÇл»»úÖƽ«Ô­Á´Â·ÉϵÄͨÐÅÈÎÎñתÒƵ½±¸·ÝÁ´Â·½øÐд«Êä¡£

¹«º£²Ê´¬¡¤6600ÍøÂçµÄ¶Ë²àÁ´Â·¹ÊÕÏÌÓÉú¹«º£²Ê´¬¡¤6600¹ÙÍø¾ßÓÐÒÔÏÂÖ÷ÒªÌØÐÔ£º

1.·ÇÇÖÈëʽ

¸Ã¹«º£²Ê´¬¡¤6600¹ÙÍø²»ÊÜÌض¨ÑµÁ·¿ò¼ÜµÄÏÞÖÆ£¬Ò²ÎÞÐè¶ÔÉϲã¿ò¼Ü½øÐÐÐ޸ġ£¹ÊÕÏ·¢Éúºó£¬ÓÉNCCL×ÔÐд¦Àí£¬Éϲã¿ò¼ÜÎÞ¸ÐÖª¡£

2.¿ìËÙÇÒ¿É¿¿

Äܹ»¶Ô¶Ë²àÁ´Â·¹ÊÕϽøÐкÁÃ뼶µÄʶ±ðÓ붨룬²¢¶Ô¹ÊÕÏÁ´Â·ÉϵÄͨÐÅÈÎÎñ½øÐпìËÙǨÒÆ£¬¿ÉʵÏÖÃ뼶ÊÕÁ²£¬²¢Äܱ£ÕÏͨÐÅÈÎÎñ²»ÖжÏ¡£

3.¿É»Ö¸´ÐÔ

ʵʱ¼à²âÉ豸״̬£¬Èô¹ÊÕÏÉ豸״̬»Ö¸´Õý³££¬Äܹ»ÔÚ»Ö¸´µÄÉ豸ÉÏÖؽ¨Í¨ÐÅÁ´Â·£¬²¢½«ÆäÖØÐÂÄÉÈëͨÐÅÉ豸¼¯Èº¡£

¹«º£²Ê´¬¡¤6600ÍøÂçµÄ¶Ë²àÁ´Â·¹ÊÕÏÌÓÉú¹«º£²Ê´¬¡¤6600¹ÙÍø¾ßÓÐÒÔϼ¸µã¹Ø¼üÊÕÒ棺

1.ÌáÉýϵͳ¿É¿¿ÐÔ

ʹÓö˲à¹ÊÕÏÌÓÉú¹«º£²Ê´¬¡¤6600¹ÙÍøÄܹ»ÓÐЧ±ÜÃâÒò¶Ë²àÍø¿¨¹âĤÏßÀµÈÉ豸·¢Éú¹ÊÕ϶øÒýÆðµÄѵÁ·ÒµÎñÖжÏ£¬ÊµÏÖ¶ÏÁ´²»¶Ïѵ£¬Í¨ÐŵĿɿ¿ÐÔÌáÉý10±¶¡£

2.½µµÍѵÁ·³É±¾

Äܹ»ÓÐЧ±ÜÃâѵÁ·¼¯ÈºÒò¶Ë²àÁ´Â·¹ÊÕ϶øµ¼ÖµÄËãÁ¦×ÊÔ´ÀË·Ñ£¬Í¬Ê±»ùÓÚ×Ô¶¯¹ÊÕϻָ´´¦Àí»úÖÆÈ·±£ÍøÂçÉ豸ʵÏÖ×î´óÀûÓÃÂÊ£¬´Ó¶øÏÔÖø½µµÍѵÁ·³É±¾¡£ ÒÀ¾ÝMeta¹«²¼µÄѵÁ·ÈÕÖ¾£¨Llama 3.1£¬405BÄ£ÐÍ£©£¬ÔÚΪÆÚ54ÌìµÄԤѵÁ·½×¶Î£¬¹²³öÏÖ419´ÎÒâÍâÖжÏÇé¿ö£¬ÆäÖÐÔ¼8.4%ÊÇÓÉÍøÂç¹ÊÕÏËùÖ¡£H100µÄËãÁ¦½¨Éè·ÑÓÃԼΪ10Ôª/¿¨/Сʱ£¬Èôÿ´ÎÖжϻָ´ÐèºÄʱ1Сʱ£¬ÄÇôÍò¿¨¼¯ÈºÃ¿´ÎѵÁ·ÒòÍøÂç¹ÊÕÏÔì³ÉµÄËãÁ¦ËðʧԼΪ350ÍòÔª¡£Í¨¹ý¶Ë²àÁ´Â·¹ÊÕÏÌÓÉú»úÖÆ£¬Ã¿Äê¿ÉÍì»ØÉÏǧÍòÔªµÄËðʧ¡£

×ܽá

´óÄ£ÐÍ·Ö²¼Ê½ÑµÁ·¹ý³ÌÖеĶ˲àÍøÂç¹ÊÕÏ£¬ÕâÊÇÓ°ÏìÄ£ÐÍѵÁ·µÄÖØ´ó×è°­¡£¹¹½¨¾«×¼ÇÒ¸ßЧµÄ¹ÊÕÏ´¦Àí»úÖÆ£¬ÊǸ÷´óÐÍÄ£ÐÍѵÁ·ÍŶӵijÖÐø×·Çó£¬Ò²ÊDZ£ÕÏ´óÄ£ÐÍѵÁ·µÄÖØÒª»·½Ú¡£

ͨ¹ýÔÚͨÐÅ¿âÖÐÌí¼Ó¹ÊÕÏ´¦ÀíÄ£¿é£¬Äܹ»´ó·ù¶È¼õÉٶ˲àÁ´Â·¹ÊÕϵ¼ÖµÄËãÁ¦Ëðʧ¡£Ëæ׏«º£²Ê´¬¡¤6600¹ÙÍøµÄ³ÖÐøµü´úÓëÍêÉÆ£¬ÎÒÃǼáÐŶ˲à¹ÊÕÏÌÓÉú¹«º£²Ê´¬¡¤6600¹ÙÍøÄܹ»ÔÚ´ó¹æÄ£¼¯ÈºÑµÁ·¹ÊÕÏ´¦ÀíÖз¢»Ó¸üΪÖØÒªµÄ×÷Óã¬Îª¿Í»§´´Ôì¸ü´óµÄ¼ÛÖµ¡£¹«º£²Ê´¬¡¤6600ÍøÂ磬ÖÂÁ¦×ö×¶Ë²àµÄÍøÂ繩ӦÉÌ£¡