Why did it seem to affect everyone differently?
The Mind Stone passes through a lot of hands: Loki, Hydra, Ultron, Vision and Thanos. Here is every stop it makes on screen, what the films and series actually show it doing, and which part of the popular 'it amplifies you' idea is backed up.
Levels marked "Sourced facts" are checked against the sources listed at the bottom. Levels marked "Fan theory" are interpretation, not confirmed by the films or their makers.
In The Avengers (2012) the Mind Stone sits inside the scepter Loki carries. He uses it to take over the minds of Clint Barton and Erik Selvig, and at the end Natasha Romanoff uses the scepter to close the wormhole over New York. After Loki's defeat the scepter goes to S.H.I.E.L.D. and then to Hydra's Baron Strucker, who is seen with it in the mid-credits scene of Captain America: The Winter Soldier (2014). Age of Ultron (2015) explains that Strucker experimented on people with it, and that Wanda and Pietro Maximoff were the only survivors. In the same film the Avengers take the scepter back, Tony Stark and Bruce Banner find an artificial intelligence inside the gem and use it for the Ultron program, and Ultron takes the stone for a new body that becomes Vision. In Avengers: Infinity War (2018) Thanos tears the stone out of Vision's forehead in Wakanda.
Loki: Wikipedia's Loki article (citing the films) says the scepter controls minds and, without Loki knowing, amplified his hatred for Thor and for the people of Earth. The Avengers itself never spells this out. Wanda: Age of Ultron shows the experiments giving her powers. WandaVision (2021, episode 'Previously On') goes further: the stone amplified magic she already had, and during the experiment she glimpsed herself as the Scarlet Witch. Vision: the stone powers the body Ultron built; Stark and Banner upload J.A.R.V.I.S. into it and Thor's lightning finishes the job. Thanos: for him it is one of six stones he needs for the Infinity Gauntlet.
This part is fan theory, not something the films confirm. A popular reading ties the stone's effects together: it doesn't hand out one fixed power, it boosts what is already inside a person. Two pieces have on-screen support: Loki's hatred being amplified (as the Loki character summary puts it) and Wanda's existing magic being amplified (shown in WandaVision). The idea that the stone shaped Vision's kindness, or worked like a 'mirror' for everyone who touched it, is interpretation only. We could not find a film scene or a statement from Marvel that says so.