You do not need a statistics background to run a valid significance test. You need two numbers from each post: the number of impressions and the number of engagements (or clicks, depending on your primary metric). Those four numbers go into a proportion significance calculator.

The test to use: A chi-square test for two proportions. It compares the engagement rate of version A against version B and tells you whether the difference is likely real or likely due to random variation.

Free calculators: AB Testguide and VWO's significance calculator both accept raw numbers and return a p-value instantly.

Worked example:

Version A: 1,200 impressions, 84 engagements. Engagement rate: 7.0%
Version B: 1,150 impressions, 115 engagements. Engagement rate: 10.0%
Enter those four numbers into AB Testguide. The calculator returns p = 0.008.

What p = 0.008 means in plain language: If there were actually no difference between the two versions, you would see a gap this large by random chance only 0.8% of the time. A p-value below 0.05 is the standard threshold for calling a result statistically significant. This result clears that bar.

What p = 0.34 means: There is a 34% chance the observed difference is random noise. You cannot conclude version B is better. You need more data points.

Important caveat: Statistical significance tells you the difference is probably real. It does not tell you the difference is large enough to matter. A 0.5% improvement in engagement rate that clears p = 0.05 is technically significant but practically irrelevant. Always check the absolute size of the difference alongside the p-value.