New Method of PCIe Testing Offers Quick Insights

Introduction

Traditionally, transmitter (Tx) testing requires the use of an oscilloscope to make eye height and eye width measurements of a transmitted signal from a device under test (DUT) in order to assess signal quality. As a consistently specified tool for validation and compliance systems, oscilloscopes are heavily used in development to determine the readiness of a DUT to undergo compliance certification for many high speed I/O (HSIO) standards. While critical for development and debug, there has been a growing need for other equipment that can reduce the time required by oscilloscopes to gather insights into Tx link health.

Test systems for validation and compliance can be complex and time consuming to set up properly and can often take hours or days of testing to compile the results needed to fully assess the health of a DUT - especially for wide communication buses like those found in PCI Express (PCIe). A tool that puts an increased focus on time to insights and ease of use would complement oscilloscope functionality by allowing engineers to do more regular evaluations of their DUTs during development with less impact on schedules. For this reason, Tektronix has introduced the TMT4 Margin Tester. Modern engineers need another instrument to support PCIe testing; one that is simple to setup, easy to use, and that improves their ability to identify bugs that typically are not found until much later in development.

Tx Testing with the TMT4 Margin Tester

The TMT4 Margin Tester takes a different approach to Tx testing than traditional testing setups utilizing oscilloscopes. In PCIe compliance testing today, Tx Link Equalization (LEQ) suites use oscilloscopes as a simulated receiver to evaluate the transmitted signal quality of the DUT. Tektronix has opted to embed the TMT4 tester in the link, using a real receiver and real traffic to evaluate the link performance of the DUT and the Margin Tester. With this approach, the TMT4 is able to use PCIe protocol to either naturally link train with the DUT and test the negotiated link (Quick Scans), or to force specific lane and preset combinations for a more thorough evaluation of the link possibilities (Custom Scans).

This approach is fundamentally different than other products on the market currently being used for physical layer testing and allows the Margin Tester to generate different insights than existing equipment. With the TMT4 Margin Tester, Tektronix has set out to re-imagine how PCIe testing could be conducted to complement existing test equipment. By taking a different testing approach, the TMT4 can offer faster time to insight with greater simplicity into many of the problems engineers currently experience that are not being fully addressed by existing test systems.

This application note highlights several real-world examples that demonstrate how the TMT4 Tx testing approach provides actionable insights into PCIe board and system designs.

BIOS Settings and Interoperability

Working with customers, Tektronix has found in multiple cases the TMT4 Margin Tester complements traditional testing systems and compliance testing. In this first example, a motherboard manufacturer was attempting to optimize their signal quality by changing BIOS setting as measured by traditional compliance testing. The table below shows that the eye width and eye height measurements for both the default and updated BIOS settings for lanes 0 through 4 showed passing results, with little variation in the measured results.

		Compliance Test Results
		Original BIOS Settings			Updated BIOS Settings			Delta (Old to New)
Lane	Preset	Eye Height	Eye Width	Result	Eye Height	Eye Width	Result	Eye Height	Eye Width
4	8	68.843 mV	28.394 ps	Pass	68.737 mV	27.586 ps	Pass	-0.106 mV	-0.808 ps
2	8	71.164 mV	27.968 ps	Pass	73.628 mV	28.575 ps	Pass	2.464 mV	0.606 ps
2	7	79.190 mV	22.086 ps	Pass	80.360 mV	21.812 ps	Pass	1.170 mV	-0.273 ps
1	8	82.233 mV	29.086 ps	Pass	77.147 mV	29.218 ps	Pass	-5.086 mV	0.132 ps
1	7	86.544 mV	22.721 ps	Pass	85.649 mV	23.772 ps	Pass	-0.894 mV	1.051 ps
1	6	50.349 mV	32.812 ps	Pass	49.421 mV	33.261 ps	Pass	-0.927 mV	0.449 ps
0	8	71.097 mV	29.982 ps	Pass	70.482 mV	29.697 ps	Pass	-0.614 mV	-0.285 ps
0	7	78.126 mV	24.648 ps	Pass	77.035 mV	25.551 ps	Pass	-1.091 mV	0.902 ps
0	6	41.237 mV	33.345 ps	Pass	40.164 mV	32.964 ps	Pass	-1.073 mV	-0.381 ps

Table 1: Compliance Test Results of Original and Updated BIOS settings

Despite this, the motherboard manufacturer began experiencing interoperability issues with the updated BIOS settings that did not previously exist with the default settings. With a minimal difference between the two compliance test results, the manufacturer was not sure what was causing the issues, nor why they were achieving passing results with both configurations using their testing systems. Tektronix partnered with this manufacturer to see if there was any insight that could be drawn from the TMT4 Margin Tester that was not showing up in the results from their test system. To begin, the TMT4 ran a Quick Scan with the default BIOS settings, which initiates a natural link negotiation to see - in just 5 minutes of testing - what could be found. The eye plots, link training parameters, and range of possible DFE tap values used to format the table can be seen in the figures below:

LIV curves — **Figure 1: Quick Scan Plot of Default BIOS Settings**

Data from the first four lanes were extracted from the link training parameters table to compare to the subset of BIOS settings results above, and the DFE taps were conditionally formatted in an Excel spreadsheet according to the limits of each DFE tap. Immediately apparent is that the Margin Tester did not train to any of the presets that were eventually changed from the default settings. This indicated that the negotiated presets between TMT4 and the DUT were not amongst the presets of interest to change in the BIOS.

Lane	Preset	eye_width	eye_height	att	vga	ctle	dfe1	dfe2	dfe3	dfe4	dfe5
4	6	22.32	67.03	-6.58	2.28	6.61	26.4	-8.25	9.28	2.75	4.29
2	9	22.15	85.72	-10	1.71	5.77	22.34	-14.44	6.53	3.75	3.43
1	9	20.99	83.69	-8.86	2.28	7.45	24.37	-5.5	6.18	2.75	2.92
1	9	22.86	92.63	-8.86	2.28	6.61	28.43	-10.32	7.9	3	3.95

	dfe1	dfe2	dfe3	dfe4	dfe5
Low	-55.0 mV	-44.0 mV	-22.0 mV	-16.0 mV	-11.0 mV
High	55.0 mV	43.0 mV	21.7 mV	15.8 mV	10.8 mV

Table 2: Link Training Parameters and DFE Tap Ranges

Additionally, it can clearly be seen that the DFE Tap 1 is being heavily used to open the eye, which indicates that the receiver is using lot of equalization to form the link. So, while a link was able to form, another DUT with poor receiver margins could potentially result in interoperability issues.

Since the DUT didn’t naturally train to the lane and preset settings denoted in the BIOS table, a Custom Scan was run using the TMT4 to look at all of the lane-preset combinations across the DUT. In about 30 minutes, TMT4 compiled the eye plots and link training parameters table shown below:

lane	preset	eye_width	eye_height	att	vga	ctle	dfe1	dfe2	dfe3	dfe4	dfe5
4	8	18.85	54.84	-8.86	2.85	3.67	30.46	-14.44	7.9	6	6.18
2	8	21.73	68.66	-10	2.85	3.67	28.43	-17.19	3.78	6.75	4.98
2	7	21.58	71.5	-10	4	2.83	44.68	-16.5	2.75	8.25	6.01
1	8	16.74	53.63	-10	2.28	3.67	22.34	-12.38	5.5	6	4.29
1	7	15.77	52	-10	2.85	2.83	30.46	-13.75	4.81	6.5	4.98
1	6	21.03	71.91	-8.86	1.71	7.87	22.34	-10.32	3.78	2.75	1.89
0	8	22.32	67.44	-10	1.71	3.67	30.46	-15.82	4.46	4	6.01
0	7	23.62	67.44	-10	2.85	2.83	36.56	-16.5	3.78	5.75	8.07
0	6	25.05	62.97	-8.86	1.71	6.19	30.46	-13.75	5.15	3.75	3.26

Table 3: Link Training Parameters for Custom Scan Plots of Default BIOS Settings

As before with the Quick Scan results, the link training parameters table has been pared-down for just a comparison with the subset of BIOS settings we are evaluating. Again, it is quite easy to see how heavily the DFE is being used to open the eyes, especially from DFE taps 1, 2 and 5. Also of interest is the difference in measured eye height and eye width from the compliance results and the TMT4 results, which is shown in the table below.

		Compliance Test		TMT4		Delta
Lane	Preset	Eye Height	Eye Width	Eye Height	Eye Width	Eye Height	Eye Width
4	8	68.843 mV	28.394 ps	54.84 mV	18.85 ps	-14.003	-9.544
2	8	71.164 mV	27.968 ps	68.66 mV	21.73 ps	-2.504	-6.238
2	7	79.190 mV	22.086 ps	71.5 mV	21.58 ps	-7.69	-0.506
1	8	82.233 mV	29.086 ps	53.63 mV	16.74 ps	-28.603	-12.346
1	7	86.544 mV	22.721 ps	52 mV	15.77 ps	-34.544	-6.951
1	6	50.349 mV	32.812 ps	71.91 mV	21.03 ps	21.561	-11.782
0	8	71.097 mV	29.982 ps	67.44 mV	22.32 ps	-3.657	-7.662
0	7	78.126 mV	24.648 ps	67.44 mV	23.62 ps	-10.686	-1.028
0	6	41.237 mV	33.345 ps	62.97 mV	25.05 ps	21.733	-8.295

Table 4: Comparison of Eye Height and Width Measurements - Compliance and TMT4

Again, using conditional formatting in an Excel spreadsheet, the table visually depicts meaningful differences between the compliance test results and the TMT4 results. In particular, lane one using presets seven and eight looked much worse in the TMT4 results using real link traffic than in the compliance test results using test signals for both eye height and eye width. Overall, due to the heavy reliance on equalization to open the eyes, in both the Quick Scan and Custom scan results, there is a high chance of interoperability issues with this DUT.

Next, The TMT4 was used to evaluate the updated BIOS settings to see what could be learned. As previously was the case, the manufacturer found that when the settings were updated the TMT4 was unable to form a link with the DUT, and thus was unable to scan the results as shown in the screenshot below.

While this result was expected based on the interoperability issues this manufacturer had already seen after the BIOS settings updates, when you look at how heavily the DFE was used in the default settings, it is not a surprising result. It appears that the BIOS settings update made the link bad enough to cause interoperability issues, especially if the DUTs they were attempting to interoperate with did not have good receiver margins.

In just 30 minutes of testing, and with some basic Excel formatting of the exported results, the TMT4 was able to provide additional information to this manufacturer on the level of equalization needed to open the eyes of their transmitted signals and replicated the interoperability issues they were experiencing. This is a simple example of how TMT4 can nicely complement your traditional compliance testing, especially if you are getting passing compliance results and still seeing interoperability issues with your design.

Re-Driver Gain Design and Interoperability

Another motherboard manufacturer was attempting to resolve an issue they were seeing with their re-driver gain setup. They had used the reference design from their silicon vendor and were seeing passing results using Tektronix’s TekExpress software and oscilloscope as shown in the diagrams below for lanes 0, 8, and 15.

Despite these passing results, the manufacturer has been experiencing interoperability issues that they were unable to root-cause. Tektronix offered them an opportunity to use a TMT4 to help debug this issue, and the TMT4 experienced the interoperability issues that they were seeing at Gen 4 speeds with their DUT as shown in the screenshot below.

After failing to form a link at Gen 4 speeds, the TMT4 was adjusted to test their DUT at Gen 3 speeds. After adjusting to Gen 3 speeds, TMT4 was able to link with their DUT, and with some basic analysis in an Excel spreadsheet, the manufacturer was able to easily visualize where the problem was occurring. As shown in the table below, there is little equalization needed to open the eye on the top 8 lanes of the slot, but the bottom 8 lanes had a few troublesome lanes as indicated in red text.

lane	preset	eye_width	eye_height	att	vga	ctle	dfe1	dfe2	dfe3	dfe4	dfe5
0	2	82.6	190.3	-8.86	2.28	3.67	-2.04	0	7.21	1	1.71
1	2	82.3	150.3	-8.86	1.71	4.51	-12.19	8.25	5.15	0.25	0.68
2	2	79.9	153.0	-10	1.14	4.09	-6.1	0.68	2.06	1	0.85
3	2	81.5	152.3	-8.86	1.14	4.09	-10.16	1.37	1.03	-2	2.75
4	2	80.9	132.0	-7.72	1.14	3.25	4.06	4.12	2.4	1.5	3.78
5	2	83.0	149.6	-8.86	1.14	3.67	-2.04	3.43	3.78	2	3.26
6	2	89.3	150.3	-8.86	1.14	3.25	2.03	1.37	2.06	-0.25	3.26
7	2	82.2	156.4	-7.72	1.14	3.67	0	2.06	3.09	-0.25	2.92
8	2	88.5	182.1	-7.72	0	2	-18.29	-4.82	-1.04	-3.25	1.03
9	2	87.5	148.3	-5.43	0.57	2	-12.19	-4.13	-0.69	-3.25	-0.69
10	2	85.9	144.2	-7.72	0	2	-14.22	-2.75	-1.38	-1.75	0.34
11	2	83.9	172.0	-7.72	0	2	-20.32	-3.44	-3.44	-1.25	-2.07
12	2	87.3	149.0	-7.72	0.57	2	-2.04	0.68	4.12	-2.25	1.2
13	2	92.3	103.6	-8.86	0	2	-2.04	3.43	-2.41	-1.5	-0.52
14	2	87.0	165.9	-7.72	0	2	-8.13	-2.75	-1.04	-0.75	0.51
15	2	86.3	157.1	-7.72	0.57	2	-8.13	6.87	2.4	-1.25	1.03

You will notice that lanes 8-11 are using much more DFE tap one equalization than any of the other lanes. This quickly indicated to the manufacturer that the receiver at the other end of the link is having to generally use more equalization to open the eyes for lanes 8-15 than 0-7. On lane thirteen, you’ll notice that not much equalization was needed to open the eye, but that it is noticeably smaller than all other lanes - especially compared to lane zero, where it is almost half the size. In just a quick 5-minute test, the TMT4 indicated to this manufacturer that their interoperability issues likely lie in the back eight lanes of their re-driver design given the higher DFE equalization and smaller eyes found in lanes 8-15.

Development Phase Improvements

In an evaluation of the TMT4, a PCIe device vendor wanted to determine how easily they could have performed regular performance checks of their device quickly throughout development had they had a TMT4. In the screenshot below, this vendor took their first-phase development board and hooked it up to the Margin Tester to get a baseline for their performance. These results can be seen in the screenshot below:

In a 30-minute test, the vendor was able to quickly assess the health of their first-phase development board to set their baseline. As can be seen above, the performance of their first-stage board was poor when using presets 0-5, and much better in presets 6-9, but still 136/156 failures based on their user-defined limits with no ability to generate an eye for four of the lane-preset combinations.

After this baseline, the manufacturer evaluated their second-phase board to see if the Margin Tester could quickly show the improvements in their designs. Again, in just 30 minutes, the vendor was able to compare their second-phase board to their first-phase to evaluate if performance improvements were shown using the TMT4. The second test can be seen in the screenshot below:

Even just a visual comparison shows the improvement from their first-phase to their second-phase design in minutes. You can easily see the improvements across presets 4 and 5, which now look much more open, but you also see the improvements in presets 6-9, which have opened up even further in their second-phase design. Using the same limits, we now see only 110/158 failures, with only two lane-preset combinations unable to generate an eye. With these two tests, this vendor was able to evaluate the progress of their development from their first-phase to their second-phase in under an hour of testing.

Quick Product Comparisons

A system vendor was interested in evaluating which of two add-in cards might be more optimal to use in their system. Using a TMT4 Margin Tester, the vendor could see how TMT4 receivers performed with each DUT. Using the Quick Scan on each DUT results were ready to export and assess in about 10 minutes. The link training parameters were imported into Excel to be formatted, and are shown below in the table, along with each eye diagram plot from the data export:

DUT #1
lane	preset	eye_width	eye_height	att	vga	ctle	dfe1	dfe2	dfe3	dfe4	dfe5
0	6	31.3	103.6	-10	2.85	12.06	34.53	0	1.37	1.37	-2.41
1	6	31.5	109.7	-10	2.28	11.64	30.46	6.18	0.68	-1.25	-0.18
2	6	26.1	90.1	-10	3.42	12.9	42.65	1.37	0	-2.25	-1.04
3	9	27.9	101.6	-10	2.28	10.8	34.53	-0.69	0	-2	0.17
4	9	29.5	101.6	-10	2.28	10.8	32.5	2.75	0.68	-1.5	0.17
5	9	31.4	100.2	-10	2.28	10.8	34.53	-1.38	3.43	0.5	1.03
6	9	26.8	89.4	-10	2.28	10.8	22.34	-1.38	1.03	-0.5	-0.86
7	9	31.8	115.1	-10	1.14	9.54	30.46	-2.75	1.37	0.75	0.51
8	9	29.6	121.9	-10	2.28	10.8	26.4	-3.44	2.06	-1	0.51
9	9	32.1	122.6	-10	2.28	9.54	22.34	-2.07	4.12	-0.5	0.85
10	9	29.4	90.1	-10	2.28	9.54	20.31	-0.69	2.4	1.75	-0.18
11	6	32.1	130.0	-10	2.28	10.38	28.43	-2.75	1.71	0.75	-2.75
12	6	29.9	91.4	-8.86	2.28	9.12	26.4	0	3.78	2.25	-0.18
13	6	36.0	79.9	-10	2.28	9.96	26.4	0	0	1.5	-1.04
14	9	31.4	119.2	-10	2.28	9.54	32.5	-5.5	1.37	2	-1.72
15	9	31.9	121.9	-10	2.28	9.54	32.5	0.68	6.87	2	0

	<100 m

DUT #2
lane	preset	eye_width	eye_height	att	vga	ctle	dfe1	dfe2	dfe3	dfe4	dfe5
0	6	31.3	138.1	-10	2.85	9.96	26.4	0	4.12	1	-0.35
1	9	29.8	137.4	-10	2.28	9.96	18.28	4.81	2.4	-0.25	-1.9
2	9	29.1	107.0	-10	3.42	9.96	30.46	0	2.4	0	-1.72
3	6	32.7	105.6	-10	1.14	8.7	22.34	-3.44	1.71	-1.75	1.03
4	9	28.5	100.2	-8.86	2.28	9.54	20.31	0.68	3.78	-1.75	0
5	9	31.3	121.2	-10	2.28	9.54	24.37	0.68	4.46	-2.25	1.71
6	9	28.8	100.2	-10	2.28	9.12	12.18	0	2.4	0	-0.18
7	9	32.1	130.7	-8.86	2.28	9.96	22.34	2.06	3.43	-1.25 -	-1.55
8	9	29.9	135.4	-10	2.28	9.12	18.28	0	3.09	0	-0.18
9	9	32.7	115.1	-10	1.14	7.03	14.21	0	3.78	2.5	-0.52
10	9	29.5	123.2	-10	2.28	8.7	10.15	1.37	3.43	0.25	0.17
11	9	28.8	136.1	-10	1.71	8.7	18.28	-2.07	2.75	-1	-0.69
12	6	29.1	116.5	-8.86	2.28	8.7	16.25	2.06	3.43	-0.5	0
13	9	32.2	98.9	-10	3.42	11.22	22.34	4.12	-2.07	-0.75	-2.93
14	9	33.7	110.4	-10	2.28	9.96	24.37	-2.07	0.34	1.75	-2.93
15	9	33.0	121.2	-10	2.28	9.96	26.4	5.5	4.81	0	-0.69

	<100 m

While the TMT4 is not fully representative of their complete system, the vendor was greatly interested in seeing how the two different products trained and performed with a real receiver. Looking at the eye diagram plots, it was not immediately clear which device offered better link performance with the Margin Tester, though DUT #2 does seem to have slightly more open eyes. Looking more deeply into the link training parameters, formatted to their ranges of possible values, it becomes much clearer that DUT #2 offers better performance than DUT #1 when a link is formed with the Margin Tester. The DFE was not as heavily used in DUT #2 as DUT #1, and only had one eye below 100 mV compared to five below 100 mV in DUT #1.

Even though the performance with TMT4 does not fully represent how a device would perform within their system, as a data point in their evaluation process, it did give the system vendor a good idea of how each device performed with a real receiver and real traffic. It is important to note that different preset selections in negotiation, or a signal path with less loss than the TMT4, can influence each DUTs performance in their system. But even with these nuances, in about 10 minutes of testing, the vendor had an idea of the real-world performance of each DUT after undergoing natural link training and negotiation and was able to quickly see that DUT #2 offered better performance when linked with the TMT4 Margin Tester than DUT #1.

Riser Card Impacts

At Tektronix, an experiment was performed to investigate what impact a riser card has on the performance of a link, and how well the TMT4 could detect the extra loss introduced by riser cards during testing. To do this, a Gen 4 motherboard with a 16-lane PCIe slot was used to test the slot with and without a riser card in the signal path. The results from the first Custom Scan, without a riser card, took about 30 minutes to test, and the results can be seen in the screenshot below:

As you can see, the link performance shows open eyes across all presets, with only five failures based the limits used, across only presets 1 and 2. All lane-preset combinations were able to generate an eye, even if there were a few failures at the lower presets.

For the next test, a riser card was added between the motherboard and the TMT4 to introduce more loss into the system. After another 30 minutes test, the Margin Tester produced the following results:

Even at a quick glance, you can clearly see the impact of the riser card on the link. An additional twenty-one eyes failed using the same user-defined limits, across four more presets than without the riser card. In general, all eyes shrunk across all presets, even for those that indicated passing results per the limits put in place. All lane-preset combinations were still able to generate eyes, but they were noticeably degraded from the test results without the riser card. This experiment quickly indicates that the sensitivity of the Margin Tester in detecting and indicating noticeable performance differences between a link formed with and without a riser card in the signal path.

Conclusion

Each example in this paper has shown how the TMT4 provides actionable results and valuable insight into PCIe Gen 3 and Gen 4 designs far faster and more easily than is possible with any other test equipment on the market. These are just some of the ways in which the TMT4 can be used to provide insight or direction into board or system design decisions, and Tektronix is excited to solve some of the testing challenges centered around ease of use and speed of test. The results gathered with the Margin Tester nicely complement those of existing equipment. As compliance systems gather data about the DUT using simulated receivers and test signals, the TMT4 gathers data using a real receiver and live link traffic. Using an oscilloscope-based system together with a TMT4 enables engineers to gather a truly wholistic view of the physical layer performance of their PCIe Gen 3 and Gen 4 devices. Learn more about our entire portfolio of PCIe test solutions.

Find more valuable resources at TEK.COM

Copyright © Tektronix. All rights reserved. Tektronix products are covered by U.S. and foreign patents, issued and pending. Information in this publication supersedes that in all previously published material. Specification and price change privileges reserved. TEKTRONIX and TEK are registered trademarks of Tektronix, Inc. All other trade names referenced are the service marks, trademarks or registered trademarks of their respective companies.

100622 55W-73961-0